Hypothesis Testing


Hypothesis testing is the method by which a theory is tested with statistical tests using observed data. Typically the investigator will have an idea or theory as to how a process works. They will create a null hypothesis which is the theory stated in the null (e.g., if my hypothesis is that applying a new process to technical support will improve customer satisfaction by 3 points, my null hypothesis is that application of this new process will not change customer satisfaction scores.)

Input and output data for the process are collected and analyzed using a statistical test. The statistical test assesses the likelihood that any differences, changes, or patterns were due to chance. If the p value for the test is less than the pre-determined target (often p < .05 or .01) the investigator will reject the null hypothesis and accept an alternative hypothesis instead. If the p value is greater than the pre-determined value, the investigator does not reject the null hypothesis–in practical terms this means that the sampled data do not support the alternative hypothesis.

Example: My alternative hypothesis is that having technical support agents collect specific data on each support inquiry will correlate with a difference in the satisfaction scores of our customers. The null hypothesis, then, is that implementing this new procedure will not relate with any difference in customer satisfaction scores.

I implement the new process in a test group and also observe a control group that continues to operate without the new process. I then collect customer satisfaction scores for each group prior to and during the test period. I run an ANOVA model that results in a p-value of .01.

In this case I reject the null hypothesis because the chances that the difference between the test and control groups was by chance is less than one percent.