Bar Charts Illuminated by Regression and ANOVA

Today I encountered what is perhaps one of the more common mistakes made even by those that have a decent background in statistics: Unfounded conclusions drawn from simple, one dimensional bar charts.

Here was the scenario: One of our departments produces automated charts for use by the call center and training organizations. The charts break phone agents into two groups–those that have been out of training for less than 50 days and those that have been out of training for 50 days or more. Each bar represents one of these two groups. Then, each of the charts shows the average of a key metric (e.g., average handle time, member satisfaction score, etc) for each of the groups.

What prompted me to view these charts (which have always seemed a bit one dimensional/almost useless) is a current debate in our organization over average handle time and why it increased by over 60 seconds last fall and has remained elevated since then. One hypothesis that has almost ascended to the status of lore and is accepted by a significant contingent is that we have more new agents in the call center. The thinking continues that new agents are less experienced and take longer to complete technical support calls as a result.

One of the bar charts I described are often cited as proof of this hypothesis (never mind that hypotheses are more easily rejected than proven according to Popper). The chart shows the average handle time of agents that have been taking calls for less than 50 days as having an average handle time that is almost two minutes higher than agents that have been taking calls for more than 50 days.

Proof! See? Not so fast.

Dismay struck me at first. All those scatter plots and regression analyses I had run in the past showed no relationship between agent tenure and agent average handle time. How could this be?
I ran a regression test again using a sample of well over 1000. Lo and behold there was a statistically significant relationship with an R Square value of…3.7%. The scatter and fitted line plots were as unimpressive as those I had seen in the past. Sample sizes this size usually provide enough power to find statistically significant results but there are outstanding questions one must ask, like is this simply an artifact of my sampling technique? Or, did I truly meet all of the assumptions of a regression test if it only explains 3.7% of the variance? Recall that one of the assumptions of a regression test is that you have included all of the factors that relate to the output variable. Evidently that is not the case here or we might expect to explain a larger chunk of the variance in handle time.

I started to wonder what might happen if I broke the agents into categories based on how long they have taken calls and run an ANOVA. It occurred to me that taking a continuous variable and putting it into categories would tidy things up and remove some variance. Sometimes that is helpful and sometimes that is deceiving. The jury was still out in this case.

The ANOVA also found a statistically significant difference between the agent tenure categories (e.g., 120 days tenure group. There was my answer and there was the flaw in the thinking of that contingent of people in my organization! While those groups did appear to have a higher handle time on average, there were very few of them in relation to the overall call center. Thus, when I ran a quick weighted average to combine the handle time of the two groups, I saw that the newbies only contributed a few seconds to the overall average handle time.

Ok, Dean, don’t let those silly, out of context, bar charts fool you or anyone else again.

An Example of the Pain Associated with an IT Upgrade

Last blog entry I discussed why IT upgrades or system changes often result in pain. This time I will give an example to illustrate the common issues that companies face when they attempt to upgrade a system.

In this example, the vendor’s legacy system provides tabular reports in flat text files and XML based data extracts. This worked well given the structure of the system’s database (mainframe based) until the volume processed by the system grew dramatically–a reflection of the customer’s growing business. The customer made a contractual commitment to upgrade to the next generation system provided by the same vendor which is capable, foremost, of handling the larger volume.

The newer system’s ability to process larger volumes of transactions is based on a three tier client-server architecture with a standard relational database management system and application servers in the middle tier. The new database was designed to handle large volumes of data with very low latency. As such, the database design is optimized for transaction processing.

To support reporting, the system replicates transactional data to an online data store (ODS) which is essentially a copy of the transactional database. To improve reporting response times for a select number of the reports, the vendor added a data mart which pre-aggregates some of the data that requires substantial processing prior to display in reports. The reporting system pulls data from the ODS and data mart and displays the results in a web browser interface. The developers that produced the reports took advantage of the new more flexible browser interface and designed many of the reports so that their structure was no longer tabular. For example, many reports in the new system display summary rows amongst detail rows and allows users to collapse the report to display only summary rows, or open select details.

To replace the data extracts from the legacy system, the vendor offers customers the ability to query the ODS directly on the new system. Because the ODS is hosted on an industry standard database system, the opportunities to extract and load the system’s data into other systems are much greater than with the XML extracts on the legacy system. To take advantage of these changes, however, the customer must change their approach to extracting data from the system. Some customers embrace such change. Others prefer that the new system maintain strict compatibility with the legacy system, perhaps even at the cost of performance.

One customer expected that they would obtain the best of both worlds: Dramatically improved performance and 100% (or very near) compatibility.

The resulting pain is based on their somewhat unrealistic expectation. This expectation may have resulted from invalid assumptions on the part of the customer, misleading impressions from the vendor, or more likely, both.

In particular, the issue arose when the customer realized that the new reports produced output differently than the legacy system and thus all of their internal processes that pulled information from reports and loaded the data to other systems would not longer work. Rather than investigate the potential benefits of the ODS, the customer opted to rely on contractual terms in an effort to force the vendor to enhance the functionality of the new system. As you might imagine, this approach proved time-consuming and painful.

IT Systems Upgrades: How to Avoid the Pain

Over the course of 10 years in Information Technology, I have found that the process of upgrading or migrating to new systems is almost always painful for most of the people involved. In particularly challenging cases, that pain extends also to customers. The way to mitigate or avoid some of this pain is to ensure that an organization makes a solid case for what they gain and lose from the upgrade/migration and that they set realistic expectations for how their internal processes will need to change to integrate the new system.

Why is upgrading usually painful? In my experience, the pain is usually felt due to two sets of circumstances. The first is that newer versions of a product usually implement the same functionality differently. Often this is because a newer version attempts to solve additional problems or it uses a different or newer technology. The result is that any processes that a business has developed using the old system, such as reporting or workflow, will be disrupted. A similar issue often occurs when the replacement system comes from a different vendor.

The second set of circumstances relates to the business processes that evolve around a system. Often, the new system will offer a different set of interfaces or produces output differently. Because of this, an organization’s internal processes will not interface seamlessly with the replacement system.

When should organizations upgrade or migrate to new IT systems? Often decisions to upgrade will be made based on a somewhat superficial evaluation. For example, one company I worked for decided to upgrade one of it’s systems almost immediately following the vendor’s announcement that they would be producing a new version based on the newer platform. Management made the assumption that the older platform would be phased out quickly. There may also have been a little machismo in the decision (e.g., “We’re always first, always pushing the envelope…”) In fact, the vendor did not end up phasing out the older product quickly and early releases of the new product lagged behind in terms of stability and functionality in some areas.

An organization should conduct a reasonably thorough systems analysis that assesses the benefits and costs of moving to the new system. This analysis should include information related to how existing internal business processes and systems will have to change. Where possible, the organization should quantify or at least clarify the expected revenue generating or cost saving benefits that result from the new system. I do not suggest that such assessments are ever particularly objective, but clarity is a good first step and can help identify potential pitfalls. If, for example, the analysis cannot point to any compelling reasons to move to a new platform (e.g., an existing platform provided by a vendor has a very specific sunset date, the new system provides opportunities to dramatically increase revenue or decrease costs) perhaps a move is not in the best interest of the organization.

I have also seen some circumstances where organizations have had to come to grips with their expectations for a system’s useful lifetime. At my current company, we have a customer that is using our last generation product at volumes well beyond what it was designed to handle. Because their business is growing rapidly and they rely on our system to generate revenue, they must upgrade to our latest generation system. While rapid growth is not always predictable, in their case we could have planned a little more carefully so that the upgrade would taken on such an acute flavor.

What can organizations do to avoid some of the pain associated with an upgrade? I have found that the ideal situation is for a systems analyst to outline the benefits and costs of the new system in terms of revenue and capital/operation costs. If you can articulate the top and bottom line impacts of a new system and the net result is positive, your basis for migrating or upgrading is simplified. You can convey this vision to your organization and it can help to smooth out the bumps that you will surely encounter. It is not a panacea–change is still difficult for most people. But you will not likely have to fight the subversive challenges that come along when people realized 70% of the way through a migration that the new system costs more and does little to increase revenues.

In some cases you cannot tie clear lines between the system upgrade, changes in revenue generation and operating expenditures. In these cases, you best bet may be to clearly articulate the cost savings over and above the current system.

In less glamorous cases, you are essentially forced to upgrade just to keep pace. In these cases, you will probably want to investigate changing to systems that will simultaneously reduce operating costs. If there don’t seem to be a lot of options out there to gain at least minor cost reductions, it may be time to revisit the organizations business model. I realize that’s easy to say but hard to do. However, these cases usually work out best when you work with executives to develop a solution rather than just ask for more capital or bigger operating budgets.

Assess The Value of a Business Initiative

My former manager had posted on his computer monitor a simple note that caught my attention one day several months back. The note was a one line type-written equation that read as follows:

Value = Acceptance x Impact x Execution

I asked him what this equation meant and he explained it roughly as I describe below.

Value can refer to many things but in our case, let’s apply it to a Six Sigma project or program. Value, in this case, is a measure of the benefits that the project or program brings to the company.

Acceptance is the extent to which employees, leaders, and team members of the Six Sigma project support the project. Like it or not, the extent to which others at your organization support a Six Sigma project determines whether it will succeed or fail. Having a sense for this at the outset of the project can help the team determine whether they should proceed with the project.

Impact is the extent to which the project or program decreases costs, increases revenue, or positively impacts business metrics used by your organization. If an initiative does not impact one of these important metrics, the initiative provides no more than subtle, immeasurable benefits to the organization.

Execution refers to how well the business initiative, a Six Sigma project in this case, is implemented. If your project actually fails to effectively find efficiencies that benefit the business in a tangible way, it is likely because your execution was not effective. This can take many forms with a Six Sigma project including, but not limited to, a poor problem definition, failure to effectively analyze the problem to ensure it exists, use of poor measurement systems such that your experiments result in no improvement in your output variables, etc.

Now this may seem obvious but I must note that if any one of the three input factors in this equation are zero, the business initiative will provide no value.

This little equation can prove helpful in assessing which projects are most worthwhile before you even begin implementation. You can simply ask yourself whether acceptance, impact, or execution run any significant risk of equaling zero. If so, you might be wise to work on another initiative first.

I have found this equation useful in determining how to use my time. For example, a colleague came to me and asked for help analyzing some data from a test his team had conducted. I quickly realized that they had not controlled their input variables and suggested to him that instead of analyzing their original hypothesis (which they couldn’t given the lack of control), that instead they spend their time analyzing why they had trouble controlling their input variables. In this case, the question of why they had trouble controlling the input was a useful exercise because it helped them identify a potential opportunity to address a part of the process that was not working well.

Corporate Information Assets: Why Are They Disconnected and Messy?

I had a few second thoughts about my original notes on corporate information assets. These second thoughts came when I took on a recent project to assemble a data set from several other databases.

I collected the requirements for the database from the requestor and then identified where we might obtain each of the data elements. I then identified the groups from which I would need to obtain access for each of these data sources. This is when the issue revealed itself.

Several of the groups that owned data I needed for the project held exclusive control of their databases. Furthermore, the groups were not inclined to give me or any other analysts access to their data.

In each case, their overt reasoning for withholding access was that their data were complex and required specialized knowledge to use the data appropriately. They had experienced problems in the past where other groups had misused their data in the past and drawn invalid conclusions, etc. They opt, therefore, to avoid having to trust anyone with a rather restrictive policy of no access for anyone.

I suspect that in some cases they also have a covert reason for withholding access. They may, for example, prefer to maintain a monopoly on their data so that they are in on the credit for any analysis that uses their data.

Given these circumstances, I have to question or at least qualify my previous assertion that each business should operate independently. In the case I cite above, separate departments hold exclusive hold on data sources that could be useful in concert with other data sources for improving quality of products or processes. As such, a policy of everyone owning their own data may not serve an organization well in cases where some departments refuse to share their data.

What, then, is the solution? In my case I have to appeal to executives to clear the way. This is not my preferred way of working because it usually leaves the department holding the data with bad feelings.

Ideally, a ranking executive in the organization would specify that all departments are to share data so that analysts and black belts would not have to resort to appeals for help from executives after rebuffs from data-holding departments. That is, unfortunately, rather optimistic and usually unrealistic.

Variance

Variance is the difference in values of data points in a sample or population. Many often refer to variance as the spread or dispersion of the data in a sample or population.

Technically, varinace is the square of the standard deviation of a sample or population. It is calculated as follows:

variance = average(sum((mean-X)^2)))

Where:

  • X = the individual data points in the sample/population
  • mean = the sample or population mean

 

1.5 Sigma Shift

Sigma shift is a method used to estimate long-term process capability from short term process capability. This method assumes that a process will drift 1.5 standard deviations (sigma) on a long term basis as compared to a short-term basis. Thus, a process that is within six standard deviations with a short-term sample, would be 4.5 sigma on a long-term basis.

Hypothesis Testing

Hypothesis testing is the method by which a theory is tested with statistical tests using observed data. Typically the investigator will have an idea or theory as to how a process works. They will create a null hypothesis which is the theory stated in the null (e.g., if my hypothesis is that applying a new process to technical support will improve customer satisfaction by 3 points, my null hypothesis is that application of this new process will not change customer satisfaction scores.)

Input and output data for the process are collected and analyzed using a statistical test. The statistical test assesses the likelihood that any differences, changes, or patterns were due to chance. If the p value for the test is less than the pre-determined target (often p < .05 or .01) the investigator will reject the null hypothesis and accept an alternative hypothesis instead. If the p value is greater than the pre-determined value, the investigator does not reject the null hypothesis–in practical terms this means that the sampled data do not support the alternative hypothesis.

Example: My alternative hypothesis is that having technical support agents collect specific data on each support inquiry will correlate with a difference in the satisfaction scores of our customers. The null hypothesis, then, is that implementing this new procedure will not relate with any difference in customer satisfaction scores.

I implement the new process in a test group and also observe a control group that continues to operate without the new process. I then collect customer satisfaction scores for each group prior to and during the test period. I run an ANOVA model that results in a p-value of .01.

In this case I reject the null hypothesis because the chances that the difference between the test and control groups was by chance is less than one percent.

Statistical Process Control (SPC)

Statistical Process Control is a methodology for improving business processes. One of the most widely known tools used with the methodology are statistical control charts such as X-bar and Individual-Moving Range. These charts use statistical methods to assess whether a business process operates within a stable range of variance. The charts are also often used to identify special or common cause variation in a process.

A statistical control chart typically sets upper and lower control limits (often at the third standard deviations above and below the mean for the sample). When data points exceed these control limits, the process is considered to be out of statistical control (and experiencing special cause variation.) Often, analysts will also employ other tests to determine whether a process is out of control by looking for various non-random patterns in the process data.

Special Cause Variation

Special cause variation refers to variation in an output or process that result from factors outside of the process. Special cause variation is often detected when output measures exceed the upper or lower control limits in a statistical process control chart. Special cause can also be detected when non-random patterns develop within the control limits.

An example of special cause of variation: Variation in the supply inputs for a process may change over time resulting in an process that goes out of statistical control.