Type II Error

When testing a hypothesis, this sort of error occurs when the null hypothesis is NOT rejected but should be rejected. The probability that this type of error occurred is represented by the Greek letter beta.

Examples:

  • A quality engineer passes a product when in fact it is defective
  • A defendant is acquitted but in truth, they are guilty
  • Employees of a company assume that the company does NOT have a particular problem but in fact the problem exists

 

Type I Error

When testing a hypothesis, this sort of error occurs when the null hypothesis is rejected but should not be rejected. The probability that this type of error occurred is represented by the Greek letter alpha.

Examples:

  • A quality engineer rejects a product as defective when in fact it is NOT defective
  • A defendant is convicted but in truth, they are innocent
  • Employees of a company assume that the company has a problem but in fact the problem does not exist

 

Tactics Vs Strategy in Business Processes

I find it almost amusing that many business leaders struggle to understand the difference between strategy and tactics. One situation that illustrates this that you have likely experienced follows.

I once attended a two day conference for a division of my company that was called a “strategic summit.” On the first day we reviewed what our organization had accomplished (in some cases taking credit for success where credit was not necessarily due) and reviewed some of the new technologies we had implemented in order to improve our ability to measure our success. The second day consisted of a few additional such presentations and then we all attended the plenary session which was hosted by the director of the organization.

The director took the podium and presented the metric targets that our organization would be responsible to meet the following year. He then stepped down from the podium, asked that one of the attendees close the door to the conference room, and said, “None of us are leaving here today until we decide how we’re going to achieve these targets…” I almost laughed aloud.

The (rather weak and half hearted) discussion that followed started with very narrow suggestions on how we should change some of our processes. Then the “discussion” degraded into a debate as to which changes would be most likely to ensure we would meet our targets. Since I was feeling particularly punchy this day, I asked the director whether we intended to conduct any sort of analysis to determine whether the changes we implemented were actually responsible for the increases or decreases in the key metrics. He responded that we would not. No analysis. No data. Almost no logic. Today or in the future.

Wow.

Fortunately, in light of the necessity to maintain professional propriety, I laughed in my head and then started to wonder about the technical definition of “strategy.” I wondered because what we were doing did not seem particularly strategic. Lame management style aside, this conversation did not seem productive and may have even served to create more chaos.

While wondering, the following came to mind: I considered a chess game. When a decent chess player plays a game, they will typically use a strategy which includes an overall goal and a set of principles for reaching those goals. The principles, however, are not cookbook types of moves. They are principles–higher order rules for how to outwit the opponent.

The player then translates those principles into actual moves on the gameboard–these are tactics. Tactics are meant to accomplish a near-term aim. Tactics are specific in nature (e.g., move the knight from A3 to C4). Several different tactical moves may stem from a single strategic principle.

Thus, the plenary session was not really a strategic planning session at all. The strategy had already been set–fortunately, given the incompetence of this particular leader, the strategy had already been set by higher level leaders. What we were really enduring was a tactical planning session. A poorly managed one, but a tactical planning session nonetheless.

Why is it important to make the distinction between strategy and tactics? One of the main reasons is that overall business management is far more likely to succeed when a leader first sets a strategy and then develops or refines their tactics to implement that strategy.

Keep the two clear in your mind so that you do not make a fool of yourself as did this organization leader.

Project Team Size: Smaller is Often Sweeter

How often have you seen managers make a decision–or more likely, fail to make an effective decision–by calling a meeting and inviting lots of people? At least two people from each of several cross functional groups are invited even though that may be redundant. The meeting consists of at least ten people. The meeting is set to discuss a process and the problem is not explicitly defined.

The meeting lasts for sixty or ninety minutes during which time possible strategies are discussed. Others point out the weaknesses and risks of strategies presented by others. The conversation loses focus of the issue at hand several times. At the end of the meeting, the meeting organizer or leader of the functional group responsible states something along the lines of, “This was a good discussion. Let’s meet again next week and discuss further.”

Then next week comes around and, if the leader actually followed through, the meeting reconvenes. Several of the previous week’s attendees are out on vacation or sick leave and others have joined or filled in for those that are missing. The meeting follows the same track as before except that some participants become a little more frustrated than the previous time because of the lack of progress. Many of the same topics and issues are discussed. The meeting ends the same as the previous meeting with the leader noting, “Good discussion. Frank, will you please set up a meeting for next week so we can continue our discussion?”

And so this dismal pattern continues until finally a higher level leader cites a different issue as the most important item for the organization’s focus. Or, the leader that owns the functional group responsible either forgets to set up a new meeting or forgets to assign a team member the task.

Or, in an information technology organization, how often have you observed a company assemble a large group to develop software? Because they are a large group (after all, the projects they will deliver are large projects) they will enjoy efficiencies as a result of economies of scale. But, 18 months later, the group is disbanded when higher level leaders find that the group has not produced any tangible deliverables. I have seen this a few times from both the inside and outside.

Large groups are often inefficient at certain types of work. This flies in the face of what you learned in operations management class where economies of scale associated with large, pooled-resource work groups are the mantra. Six Sigma projects almost always work better with relatively small project teams of between two and four core members. Likewise, many types of software development projects (not necessarily all), even those that seem relatively large, often work better with smaller development teams.

In the example of the strategy meeting above, there are several issues that prevented the group from achieving success. Putting aside things like a lack of agenda, ineffective meeting management, etc., one of the main issues was the size of the group. Why is it that larger groups, which arguably consist of a greater number of skills when you consider each of the participants, seem to have a harder time delivering results?

The problem lies between the people. And I mean more than just communication. Lots of communication, as demonstrated above, does not always result in everyone agreeing on how to improve a process–in fact, larger groups tend to have a wider array of conflicting opinions. It is, therefore, helpful to have a leader that has final decision making authority. Then, if the group is stuck, the leader can take input from each team member and then make a decision for the group.

Communication is also a factor. It is much easier for a smaller group to stay in sync during a project simply because there are fewer people to inform regarding changes. There also tends to be more self-enforcing accountability in a smaller group. If I do some work that creates a problem for the project, I have less chance of hiding or passing blame for that work. I have more vested interest in resolving the issue because if I do not, the colleague with which I work every day must. Because I rely on my teammates (smaller teams must, of necessity, rely on one another more than larger groups) I am much more likely to feel obligated to resolve any issues associated with my work that might trip up my colleague.

I have worked on several different software development teams and the one that was most effective in terms of delivering a product that customers used and appreciated was a small 2 person team that eventually grew to three people. The odd thing was that we delivered a working product much more quickly than the larger groups with which I worked even though the projects were similar in terms of size. The smaller team was able to make changes and enhancements much more quickly as well.

Six Sigma projects tend to be similar–if you have to spend significant time working out internal team process issues (as you are more likely to face with larger groups), you are less likely to make tangible progress.

Obviously, you have to have a large enough group to handle the task at hand. A high ranking (or at least very effective) champion that truly supports your process improvement goal is a necessity to remove barriers so that you do not need several additional team members to handle this work. And you need to cover all of the basic skills necessary to complete a project (e.g., someone that can access the necessary data, an analyst capable of valid statistical tests, etc.)

Summary: Keep your core project team small enough to operate efficiently and just large enough to have all the skills necessary.

Tests and Problem Definitions: Perspectives From Six Sigma

In most business organizations, quality departments or staff conduct tests aimed at improving the company’s bottom line. Various methods exist for designing such tests and some produce useful results more than others depending on the type of process. In many cases, test designers make the mistake of designing tests without first clearly defining the problem that they are attempting to address.

At my company, I find that many leaders and project managers design tests using assumed problem definitions. As the problem definitions are usually unspoken, they are also usually vague. Unfortunately, this puts the test designer at a significant disadvantage.

For example, in one case a leader assigned a project manager on his team to test how providing tiered customer support would save the company money. The manager assumed that the current (non-tiered) support model was more expensive than a tiered model but did not define the problem more clearly. As a result, the project manager floundered for several weeks before the so-called β€œtest” was abandoned. The main issue was that the project manager could not design an effective test because he was not clear on what needed to be tested.

In addition to leaving test objectives unclear, not formally defining the problem up front often leaves the test design team in a muddle over which input and output variables the test should use. This was one of the major stumbling blocks for the project manager that attempted to test the tiered support model. If it is unclear which variables the test will employ, it is impossible to design effective analysis methods and overall test designs.

Another pitfall associated with not defining the problem is that the test team is at greater risk of losing track of the original goal. This became clear in the example above when a process analyst investigated the project and began asking questions. The frustrated project manager explained that the test had morphed to a test to simply determine what issues might arise when they employed a tiered support model. He also had to admit that he did not receive a clear objective from his manager in terms of the perceived problem and was severely confused.

The test also suffered from scope creep. That is to say, the project manager quickly realized that there were several questions he could attempt to answer with the test and planned to address each question in turn. Scope creep can quickly move a project or test into a dangerous state of affairs where little is accomplished because the staffer or team responsible spends all their time simply managing the test requirements.

The bottom line: Before you attempt to design a test, be sure to clearly define the problem in one or two sentences. This will dramatically improve the chances of a successful test from the perspective of test design, analysis methods, focus on the original objective, and avoidance of scope creep.

The Power of Process Mapping

One of the challenges I often face is that of explaining the current state of our processes to leaders at my company. Sometimes my goal is to convince them that the current process is “in statistical control” and the seeming irregularities they saw last week were not anything that they need to worry about. Other times, they do not seem to think that some processes are problematic and my goal is to convince them that the same processes are out of control, or trending in the wrong direction.

The causes of these issues usually occur in sub-processes that the leaders do not regularly monitor. Who can blame them? They have far too many metrics to watch as it is. In truth, many of these processes go unmonitored or are completely misunderstood by most employees.

For example, our technical support group uses a knowledge management tool to help them track and resolve customers’ technical issues. The business has spent tremendous effort attempting to prove that this tool either helps or hurts the satisfaction of our customers. The test is usually constructed as follows:

- Collect data on the extent to which each representative uses the knowledge management tool
- Collect data on the extent to which customers handled by each representative are satisfied
- Run a regression analysis using knowledge management tool usage as a predictor of customer satisfaction

Obviously, there are a number of issues with this simplistic analysis. After watching the business run through several iterations of analyses similar to this, it occurred to me that one of the factors ignored, among many others, was the diagnostic questions that the agents used to decide which search terms to put into the knowledge management tool to find the appropriate content.

The focus of the operations group (who do not particularly like the tool) has generally been on the quality of the content. They essentially point their fingers at the content management group and blame them for the poor performance of the tool. On the other hand, the content management group point thier fingers at the operations group as the problem, noting that because the agents do not provide feedback on content that needs improvement, the content does not improve.

When my teammates and I mapped out the technical support process at a high level, it quickly became evident that a major part of the process has been ignored. It occurred to us that the company does not have any measurement systems that assess the quality of the diagnosis performed by the agents. So now we have ourselves a little project to conduct. This may not necessarily be a full fledged Six Sigma project. At the very least, we have an assignment to create a new measurement system that assesses the quality of diagnosis performed by agents. Interestingly, this may require a new knowledge management tool. It is a daunting task but if we are to make progress, we must assess where we stand.

The bottom line: Process mapping will often quickly help you identify areas where your business does not have information that tells them which parts of a process to improve. Once you have this information, it is often clear where you need to develop or refine measurement systems so that you can improve process outputs.

Developing New Measurement Systems

As a Six Sigma Black Belt at a company that does not embrace Six Sigma as part of its culture, data are sometimes hard to come by–useful data, at least. As I described earlier, some types of data are hard to measure and collect and many black belts move on to projects that can be completed using existing data. I am guilty again: I used this very tactic for my Black Belt project.

I have bumped up against this issue enough times that it led me to think and where I started thinking was as follows.

In my college years, I decided that I wanted to study psychology with an emphasis on family and couple dynamics. I completed an undergraduate degree in family psychology and entered a doctoral program with the intent to become either a research clinician or family therapist. I have the utmost respect for those that perform this worthwhile work but found that my skill set did not lend itself to either of these careers. After completing my master’s coursework, I discontinued my studies and focused my efforts on my then fledgling career as a business analyst.

The work that most impressed me during my graduate studies was that conducted by Dr. John Gottman. He built what is known as “The Love Lab” in Washington state where he and his research team observe couples interacting. Unlike many social scientists, however, Gottman and his team make more than the typical social science types of observations. They use heart rate monitors, measures of pulse amplitude, jitteriness, and skin conductivity. This is all in addition to a very detailed behavioral observation system he developed to characterize behaviors and emotions of couples as they interact.

But Gottmans approach to measuring behavior and emotions is not the entire story. He and his team have made significant discoveries using these data. They are able to predict with incredible accuracy–around 90%–whether a couple will still be together a year from the date that they observe them in their laboratory!

How does this relate to Six Sigma and data measurement systems in business? I suspect that many of us find ourselves stuck using the same old types of measurement systems. I believe that one of the keys to breakthrough improvements lies in developing fundamentally different ways of collecting business data.

When I say fundamentally different, I mean that the type of data or method of collecting data should be very different than what/how your business and my business currently collect–second order change.

An example of second order change might be to invest effort in text mining the written comments you receive from customers (rather than or in addition to just using likkert-scale surveys). A first order change would be to simply add more questions to your customer surveys. First order change can result in improvements but when a process is stuck, second order change is the often the only thing that will allow a breakthrough result.

Measurement and Collection of Data

In the service industry, many companies struggle to measure and collect data. The reasons for this are varied but the impact is the same–if you don’t have reliable, valid data, it will be nearly impossible to carry a Six Sigma project through to completion. The obvious reason is that the measure, analyze, and improve steps of the DMAIC model require quantifiable data.

One of the main reasons that many service industry companies do not measure many of their inputs and outputs is because these variables are often difficult to measure. For example, how does a service company measure the satisfaction level of their customers? Customer surveys jumps to mind but these efforts are time consuming and costly. Further, even if a company commits to the costs associated with surveying customers, it is very difficult to measure customer satisfaction in a valid, reliable fashion. Consider how often you, personally, fill out customer satisfaction surveys. If you filled one out recently, you most likely received some sort of incentive to do so (think big dollars for the company that wanted that survey from you). Then if you did receive a survey, did that incentive alter the ratings you gave the company?

Another reason that service companies opt not to measure and collect data on some of their important inputs or outputs is that they do not believe that investing in the collection of such data will prove cost effective. In some cases, managers that make these decisions may be right. It does not make sense to measure every possible input or output variable. Key inputs and outputs, however, should be measured and tracked if a company intends to succeed.

Though I have not worked in the manufacturing industry, I suspect there is often less deliberating over whether to measure most input and output variables because most of them are relatively simple to measure. There are obviously still costs involved but many of the variables are a little more tangible than things like customer satisfaction.

I find the mental exercise of listing key input and output variables for a process useful. Then I review the list and identify which of the variables are already measured and collected. This way I can quickly identify projects with the greatest potential for success.

Making a business case to start measuring a key metric (that is not currently measured or tracked) can also be a useful strategy to create potential Six Sigma project opportunities.

While the service industry faces the challenge of measuring and collecting business data, there is hope and plenty of innovation opportunities for those that are persistent.

A Jack Handy Moment – Randomness and Infinity

Hang on to your socks for this one… ’cause it hurts my brain just trying to type it.

I had a Jack Handy moment the other day. Remember Jack Handy? The character from Saturday Night Live? Anyway…

I was sitting in a classroom half listening to a fellow student attempt to kiss up to the teacher by droaning on and on about financial theories and randomness and unpredictability. My mind started turning over the idea of an unpredictable stock market and all the variables that would have to go into being able to predict its behavior – “chaos theory” popped into my head as a term. Now whether or not “chaos theory” is actually applicable to this topic is beyond me. Sometimes I don’t know if my brain is really good at picking things up and applying them correctly or if my brain just likes to throw things into the mix to make me think I’m smart. My brain likes to mess with me.

I digress…

So as I sat there contemplating how many variables would have to go into predicting the stock market, it dawned on me that the size of the regression equation needed to predict such a thing would just go on and on and on… to infinity! That made me wonder if things we call “random” are really just variables to infinity? Which then made me think of all the times at work I watch people hand wave past “hard problems” as they spout “random” rather than digging into the process improvement – because THEY don’t see it as a process that can be improved. THEY see it as random when in reality, it might just be a really big set of variables but it’s NOT random. THEY are just lazy.

So, if we were to plot the number of variables in a process on the X axis and the loudness of an executive yelling “random” on the Y axis… I suspect we’d get an inelastic curve at around 3-4 variables. Everything beyond that on the X axis would result in one, giant, RANDOM scream from executives.

Fishbones and Post-It notes

It was time I actually wrote something useful. πŸ™‚

I thought talking about an easy way to do fishbone (cause & effect) diagrams might do the trick. I’ve used this technique many times and it works like a charm. No muss, no fuss. And more importantly, no fighting!

What you’ll need:
— a bunch o’ square Post-It notes (not the tiny kind you can barely write on)
— pens
— a giant, wall-sized white board (if you don’t have those, tape up butcher paper to a wall so you can write on it)
— Dry Erase markers (or regular magic markers if using paper)
— your project team πŸ™‚

Sequester your team in a conference room with the white board or wall o’ butcher paper. Write your output measure(s) on the white board to refresh everyone’s memory. Explain that you all are about to do some ‘silent brainstorming’. Pass a small hunk of Post-Its around to each team member and tell them to start brainstorming ANY input variable they can think of that might affect the outputs on the board. Reinforce the ‘brainstorming’ part and the ‘silent’ part. One idea per Post-It.

GO!

Let them scribble and think and scribble and think some more. Let them keep going until you hear practically no more scribbling. This part usually lasts around 5 minutes, maybe 10 if they are feeling extra creative.

Then have EVERYONE go stick their notes all over the white board. No particular order. Just stick ’em up there.

Now, if you have a small team, keep them all at the white board and have them start grouping like items together. If you have a large team, ask a couple of them to stay while the others sit down. I try to pick people who have been really quiet. This get them participating and involved.

As they group them, have them read the notes out loud. This stimulates discussion and more Post-Its might be thought of – or some of the Post-Its might be removed upon second thought. BUT, here’s a rule: only the author of a Post-It is allowed to remove it!

You might have to help them a bit here or there. The goal here is to start getting the ‘bones’ of your fishbone. You are aiming for 4-6 major ‘bones’ with as many ‘little bones’ as you’d like.

Once the groups/bones have started to take shape, take a dry erase marker and write the category (or bone) name above/below each group of notes. Connect the groups into bones and join it to the output measures! You have a fishbone!

The teams really love this exercise. It involves them all. The ‘silent’ portion of it allows them all a voice via the Post-It notes. And you have instant buy-in because the entire team helped create it.