Tests and Problem Definitions: Perspectives From Six Sigma

In most business organizations, quality departments or staff conduct tests aimed at improving the company’s bottom line. Various methods exist for designing such tests and some produce useful results more than others depending on the type of process. In many cases, test designers make the mistake of designing tests without first clearly defining the problem that they are attempting to address.

At my company, I find that many leaders and project managers design tests using assumed problem definitions. As the problem definitions are usually unspoken, they are also usually vague. Unfortunately, this puts the test designer at a significant disadvantage.

For example, in one case a leader assigned a project manager on his team to test how providing tiered customer support would save the company money. The manager assumed that the current (non-tiered) support model was more expensive than a tiered model but did not define the problem more clearly. As a result, the project manager floundered for several weeks before the so-called “test” was abandoned. The main issue was that the project manager could not design an effective test because he was not clear on what needed to be tested.

In addition to leaving test objectives unclear, not formally defining the problem up front often leaves the test design team in a muddle over which input and output variables the test should use. This was one of the major stumbling blocks for the project manager that attempted to test the tiered support model. If it is unclear which variables the test will employ, it is impossible to design effective analysis methods and overall test designs.

Another pitfall associated with not defining the problem is that the test team is at greater risk of losing track of the original goal. This became clear in the example above when a process analyst investigated the project and began asking questions. The frustrated project manager explained that the test had morphed to a test to simply determine what issues might arise when they employed a tiered support model. He also had to admit that he did not receive a clear objective from his manager in terms of the perceived problem and was severely confused.

The test also suffered from scope creep. That is to say, the project manager quickly realized that there were several questions he could attempt to answer with the test and planned to address each question in turn. Scope creep can quickly move a project or test into a dangerous state of affairs where little is accomplished because the staffer or team responsible spends all their time simply managing the test requirements.

The bottom line: Before you attempt to design a test, be sure to clearly define the problem in one or two sentences. This will dramatically improve the chances of a successful test from the perspective of test design, analysis methods, focus on the original objective, and avoidance of scope creep.