After reading Joe De Feo’s July 8, 2011, Quality Digest Daily article, “A Positive Prognosis: Transforming Health Care in America,” I took another look at the wonderful book, Escape Fire (Jossey-Bass, 2003), a compendium of Dr. Donald Berwick’s inspiring plenary speeches at the Institute for Healthcare Improvement’s (IHI) 1992–2002 annual forum. Berwick is probably the leading health care-improvement thinker in the world. He is the former CEO of IHI and, as some of you know, a controversial Obama appointee as head of the Centers for Medicare and Medicaid Administration. In my opinion, he is most definitely the person for the job. As if it wasn’t difficult enough to deal only with health care cultures, he now has the thankless job of integrating messy political agendas into the very serious business of health improvement.

But let’s get back to when all Berwick had to worry about was the current state of health care itself. I’ve extracted three lists that appeared in separate speeches of his and added context from his text. Note that the most recent one is from 1997. It’s all inspiring rhetoric, but truly, what has really changed these past 15 to 20 years?

I will not deny the nontrivial improvements cited by De Feo, but in the great scheme of things, it’s all a mere drop in the bucket. All it proves is if you put focused attention on anything, it improves. Regarding the alleged improvements, my experience is in line with Tripp Babbitt’s Jan. 4, 2011, Quality Digest Daily article, “That Giant Sucking Sound from Missed Opportunity”:

“Missed opportunities for improvement represent a 20–60 percent chunk carved out of the bottom line. Scores of programs and projects that claim improvement but never materialize in the financials are a travesty.”

Lest you think I sound cynical, I’ll let you draw your own conclusions as to how close the current state of health care is to what Berwick envisioned—sort of like the No Child Left Behind Act in education. It all seems so logical, but ask yourself: “Why the glacial progress on such basic issues?”

The following three excerpts are from Berwick’s speeches at the National Forum on Quality Improvement in Health Care, the annual forum of the IHI now in its 23rd year. 

From “Buckling Down to Change” (1993)

“Now, in health care, among the people at this Forum, we have made the needed preparations for change. Our preparations are sufficient. We have studied enough. We have reviewed our cultures enough. We have spent the time we needed, enough time, in training and planning and filling our kit with new and useful tools and methods. We know how. Now, we must remember why….

“I propose 11 aims for our work over the next two years—11 needed results that, if achieved, would represent the first solid steps toward the systemic change that is worthy of the name ‘health care reform’:

“1. Reduce the use of inappropriate surgery, hospital admissions, and diagnostic tests
2. Improve health status through reduction in underlying root causes of illness
3. Reduce cesarean section rates to below 10 percent without compromise in maternal or fetal outcomes
4. Reduce the use of unwanted and ineffective medical procedures at the end of life
5. Adopt simplified formularies and streamline pharmaceutical use
6. Increase the frequency with which patients participate in decision making about medical interventions
7. Decrease uninformative waiting of all type
8. Reduce inventory levels
9. Record only useful information only once
10. Reduce the total supply of high-technology medical and surgical care, and consolidate high-technology services into regional and communitywide centers
11. Reduce the racial gap in health status, beginning with infant mortality and low birth weight”

Two years, eh? 

From “Run to Space” (1995)

In my opinion this speech was his all-time best. Berwick applies an analogy of coaching grade-school girls’ soccer to improving medicine. In a thinly veiled poke at the medical world, he talks about his five strategies for motivating his “team” the Angels:

“1. I began with laissez-faire. Empowerment.

‘You’re professionals. You know what to do. Go for it.’

Angels 1, Pixies 4.

2. I elected to switch my strategy. Perhaps, I thought, these girls are not as motivated as I had initially believed. We need a results orientation.

I gave them feedback. When they scored a goal, I yelled from the sidelines, ‘You scored a goal! You scored a goal!’ When they missed, I yelled, ‘You missed! Next time, don’t miss!’

We lost.

3. No more Mr. Nice Guy.

We were in crisis. It was time for performance pay. No score, no cookies. Score: Hershey Bar.

Simple, direct, informative.

We lost.

4. The team rewards were of course insufficient motivation. Report cards became individualized. I posted the scores by individual players, protecting their anonymity, of course.

They insisted that I case-mix adjust the individual scores according to the competence of the other team, the weather, and so forth. I said, ‘You don’t get the point: We have to beat them no matter who they are.’

Angels 0, Gerbils 6.

5. Guidelines were the answer. The real problem was lack of standards for plays. We soon produced our first soccer guideline:

Do you have the ball? If not, get it. Shoot. Score.

We then lost again.

“Rebecca came up to me at halftime and said, ‘I’m sick of losing.’

“‘Oh, yeah?’ I said, sipping my cappuccino. ‘If you’re so sick of losing, why not win? I point out the scoreboard, I motivate, I make guidelines, I tell you pass-pass-pass-shoot. That’s my job.’

“‘You don’t get it. It doesn’t help me when you yell, “Pass-pass-pass-shoot.” You have to tell me how. How do you play soccer?’”

Berwick’s point? Leaders have to be able to coach on methods. These were his six ideas that represent the “appropriate foundations of design for the era of change that will be responsive to the new context of care:”
1. Reduce waste in all of its forms
2. Study and apply the principle of continuous flow
3. Reduce demand
4. Plot measurements related to aims over time
5. Match capacity to demand
6. Cooperate

How are we doing 16 years later? There has been lots of activity, but, impact—I mean real, deep, fundamental impact?  

From “Why the Vasa Sank” (1997)

Background: The Vasa was a Swedish warship built in 1628. It was supposed to be the grandest, largest, and most powerful warship of its time. King Gustavus Adolphus himself took a keen personal interest in it and insisted on an entire extra deck above the waterline to add to the majesty and comfort of the ship, and to make room for the 64 guns he wanted it to carry. This innovation went beyond the shipbuilder knowledge of the time—and would make it unstable. No one dared tell him. On its maiden voyage, it sailed less than a mile and sank to the bottom of Stockholm harbor.

“I want to see health care become world class,” said Berwick. “I want us to promise our patients and their families things that we have never before been able to promise them. I am not satisfied with what we give them today. And as much respect as I have for the stresses and demoralizing erosion of trust in our industry, I am getting tired of excuses.

“To get there we must become bold. We are never going to get there if timidity guides our aims. Marginal aims can be achieved with marginal change, but bold aims require bold changes. The managerial systems and culture that support progress at the world-class level don’t look like business as usual:
1. Bold aims, with tight deadlines
2. Improvement as the strategy
3. Signals and monitors—providing evidence of commitment to aim, giving visible evidence of strategy via management of monitors
4. Idealized designs
5. Insatiable curiosity and incessant search
6. Total relationships with customers
7. Redefining productivity and throughput
8. Understanding waste
9. Cooperation
10. Extreme levels of trust”

“The lesson about the Vasa is not about the risk of ambition. It is about the risk of ambition without change, ambition without method.”

Standing ovation, and we all say, “So true, so true,” then go back to work… and the phone rings….

Are any of the issues from these three speeches any less relevant today? Think about the last 14 years regarding quality improvement. Are quality professionals unwittingly helping their organizations get top-heavy regarding improvement? If progress is glacial, maybe organizations are perfectly designed to have it be glacial? How do quality improvement professionals need to change? How do they help organizations to change and not sink under an excruciatingly formal improvement process?

Have quality improvement professionals become all-too-willing accomplices in management’s efforts to change a culture by focusing on the structure, rewards, or roles and core competencies? It all seems so logical, but as those of us in quality improvement know, when human psychology is involved, logic is almost never persuasive.

I think Tripp Babbitt, in the same article quoted above, nails the real root cause of it all:

“It’s possible to discover new, counterintuitive truths, but focusing on process or traditional approaches to improvement won’t get you there. That giant sucking sound from the bottom line is the missed opportunity to address the big problem—management thinking.”

Maybe when we choose to stop building Vasas, we’ll see some real progress.

Any article about control charts leads to inevitable (and torturous) discussions of special cause tests—all nine of them. No wonder confused people continue to use things like trend lines. But I’m getting ahead of myself.

First of all, before you take another tools seminar or read another book—except, perhaps, Brian Joiner’s Fourth Generation Management (McGraw-Hill, 1994)—please try Dr. Donald Berwick’s admonition at the end of my Aug. 2, 2011, article, “A New Conversation for Quality Management”: Find something important, and plot it over time. This is probably the best way to learn the most important lesson of quality improvement: That everything is a process, and effective improvement means having new conversations around the crucial distinction between common and special causes. As I have relentlessly tried to make clear, you are swimming in everyday opportunity.

Most of my articles are concerned with statistical thinking in the context of an overall improvement process. This is much different from control charting, say, a machine on the manufacturing floor, which could indeed show a trend as in the classic case of tool wear. I’m sure there are others, but once again, that’s an application for the 1 to 2 percent of people who need advanced statistics. I’m trying to counteract the effects of Rule No. 4 of Deming’s Funnel Experiment, as it has manifested in the process of statistical training for the masses.

Treating every special-cause signal as… a special cause?

Many people have the misconception that each special-cause signal on a chart must be treated as a “special” cause—i.e., needs to be uniquely investigated. Have you ever thought, “Might there be one underlying explanation generating all of these signals?”

How ironic that a lot of people teaching control charts as a tool don’t understand this more subtle manifestation of common vs. special cause. As I continually emphasize, quality improvement is a mind-set that knows how to ask the right questions.

“But the test for trend is statistically significant…”

Someone once presented me with the graph in figure 1. (Yes, the y-scale started at zero.) It almost convinces you there is a trend, eh? (p-value < 0.001.)

Fig. 1: Percent conformance to goal

Some of you could be wondering, “What insight might a control chart give?” Figure 2 shows you, with the resulting tests for special causes:

Fig. 2: Percent conformance to goal showing tests for special causes (y-axis expanded)

Test 1: One point more than 3.00 standard deviations from center line.
Test failed at points 9, 50, 51, 52.

Test 2: Nine points in a row on same side of center line.
Test failed at points 9, 1.

Test 5: Two out of 3 points more than 2 standard deviations from center line (on one side of CL).
Test failed at points 4, 6, 7, 9, 15, 48, 49, 50, 51, 52.

Test 6: Four out of 5 points more than 1 standard deviation from center line (on one side of CL).
Test failed at points 5, 6, 7, 8, 9, 10, 17, 18, 20, 50, 51, 52.

Test 8: Eight points in a row more than 1 standard deviation from center line (above and below CL).
Test failed at points 9, 10.

Yikes! Sixteen of the 52 data points generate special-cause signals. Some data points even generate multiple signals. That’s 30 special-cause signals total. Where does one begin?

Unfortunately, the way I see control charts generally taught, “obviously” one should initially investigate, individually, the four points outside the three standard deviation limits, in this case observations points 9 and 50–52… not!

As time goes on, I have developed an increasing affection for the much-neglected run chart, a time plot of your process data with the median drawn in as a reference line. It is filter No. 1 for any process data, and it answers the question, “Did this process possibly have at least one shift during this time period?” This is generally signaled by:

• A clump of eight consecutive points either all above or below the median
• Less often, six consecutive increases or decreases. This is sometimes called a “trend” but more correctly, it indicates a transition to a new process level. (Any good software package should do this analysis and let you effortlessly toggle between run charts and control charts.)

Here’s the rationale for using the median: If special causes are observed in the run chart, then it makes no sense to do a control chart at this time because the average of all these data doesn’t exist. Sort of like, “If I put my right foot in a bucket of boiling water and my left foot in a bucket of ice water, on average I’m pretty comfortable.”

One of the healthiest things that a run chart can do is get you thinking in terms of “process needle(s),” i.e., focusing on the process’s central tendency.

Most of the time, run charts are glossed over and taught as the boring prerequisite to learning control charts. Isn’t it far more exciting to jump right to the control chart with all its bells and whistles, look at the special-cause signals, and try to find reasons for each individual signal?

The run chart does not find individual special-cause observations because that is not its purpose. That is one of the objectives of the control chart—call it filter No. 2. One plots the data incorporating the shifts detected via the run chart. This usually reduces the number of subsequent special-cause signals, resulting in a lot less confusion. The control chart also has an additional power to detect more subtle shifts neither obvious nor detectable in the run chart.

So, what light might a run chart shed on the current situation? Take a look at figure 3:

Fig. 3: Run chart of percent conformance to goal

With the y-axis scale a lot healthier and no control limits as a distraction, doesn’t it look like the “needle” shifted twice—around observation points 21 and 47? In fact, when I asked the clients about those two particular points and their corresponding dates, they looked at me like I was a magician and asked, “How did you know?” Those dates coincided with two major interventions to improve this process.

As the chart in figure 4 shows, they worked—two needle bumps—not a continuously increasing improvement “trend.” Making only those two adjustments, the correct resulting control chart is shown below. There’s not a special cause to be found, but there is a possible improvement/transition in the making, as evidenced by the last four data points. Time will tell.

Fig. 4: Control chart of percent conformance to goal

 Take “the challenge”

1. Can you think of one or two applications like this in your everyday work or meetings? Plot it.

2. Can I also challenge you to take one routinely “trended” data display and plot it as a run chart?

3. Will you promise me that, before you take yet another course on more tools, you will do a (mere) run chart of a number that makes you sweat?

4. What part of “never”…. If someone asks you to “trend” some data, will you flat out refuse? If so, be prepared: You probably will also have to refuse his valiant attempt to keep the trend monster alive via the dreaded “two-headed transplant” technique—i.e., when he predictably insists, “OK, then, just put them both on the same page.”

5. And beware of this monster’s cousin (figure 5):

Fig. 5: “Trend” control chart for percent conformance to goal.

P-value for trend: < 0.001, R-squared: 51.1 percent—and total rubbish!

A final clarification

For those of you who may still be slightly confused, ponder this: Suppose you’re trying to lose weight. You make a hefty cut in your calories. You start weighing yourself every day and plot it. During the first two weeks, you will probably lose 5 to 7 pounds of “water weight.” For the next two weeks, you will probably lose 1 to 1 1/2 pounds a week. After that, physiologically, the body adjusts to your decreased caloric intake to the weight that it is “perfectly designed to get” and levels off. Then, by all means, go ahead and fit a trend line; I guarantee a statistically significant regression. And if your weight loss proceeds perfectly linearly according to the line, please let me know—the day before your weight goes to zero.

For purposes of improvement: trend = transition to the new process you are “perfectly designed to get,” given your new inputs vis-à-vis your old status quo.

My March 30, 2011 article ended with wisdom from Yogi Berra as a warning to the quality profession. Some prickly reactions to it got me thinking about the last 30 years or so of quality improvement.

The 1980 NBC television show, “If Japan Can, Why Can’t We?” introduced the teachings of W. Edwards Deming to U.S. viewers and caused a quantum leap in awareness of the potential for quality improvement in industry. During the late 1980s, the movement also caught fire in health care. Those of you familiar with Deming’s funnel rules (which shows that a process in control delivers the best results if left alone) will smile to realize that his rule No. 4—making, doing, or basing your next iteration based on the previous one—also known as a “random walk,” has been in operation for the last 30 years.

Jeff Liker, professor of industrial and operations engineering at the University of Michigan, beautifully describes the random walks that have taken place within the time spans of Six Sigma and lean. In a private correspondence with leadership expert Jim Clemmer, Liker writes:

“Originally Six Sigma was derived from Toyota Quality Management (TQM) by Motorola to achieve six sigma levels of quality, and then through Allied Signal and GE it morphed to projects by Black Belts based on statistics to become a cost-reduction program—every project needs a clear ROI. In other words, we denigrated the program from a leadership philosophy to a bunch of one-off projects to cut costs. It was a complete bastardization of the original, and it rarely led to lasting, sustainable change because the leadership and culture were missing.

“A similar thing happened to lean when it got reduced to a toolkit (e.g., value-stream mapping, KPI boards, cells, kanban).

“Lean and Six Sigma in no way reflect the original thinking of excellent Japanese companies or their teachers like Deming.”

Clemmer also cites multiple studies from 1996–2007 concluding that about 18 to 24 months after these various quality systems are launched, 50–70 percent of them fail. Liker concurs and feels that the four key failure factors, in this order, are:

• Leadership lacking deep understanding and commitment
• Focus on tools and techniques without understanding the underlying cultural transformation required
• Superficial program instead of deep development of processes that surface problems solved by thinking people
• Isolated process improvements instead of creating integrated systems for exceptional customer value

Virtually everyone agrees that the No. 1 barrier to improvement is still top management’s inability to be visibly committed to quality. Is this the “elephant in the living room” or as Clemmer calls it, “the moose on the table”? The longer I’m in improvement, the more I realize the wisdom of Deming’s statement, “If I could reduce my message to management to just a few words, I’d say it all has to do with reducing variation.” Why reduce variation? Because it affords better prediction. He said it so often: “Management is prediction!”

Deming also says in point No. 2 of his famous 14 Points: “Adopt the new philosophy.”

Unfortunately, Deming’s philosophy seems to have morphed into a training mill turning out “belts” by the thousands with statistical training that makes my palms sweat. I’ve said it before: People don’t need statistics; they need to know how to solve their problems. All that’s needed is a few simple tools and a working knowledge of variation to be able to distinguish between common and special causes. Only 1–2 percent of people need advanced statistical knowledge. Deming would roll over in his grave if he could see the statistical subculture of “hacks” (his term) that have been turned out in his name.

In Deming’s words

I think the best book on design of experiments (DOE) is Quality Improvement Through Planned Experimentation, by Ronald Moen, Thomas Nolan, and Lloyd Provost (McGraw-Hill Professional, 1999). It is the only book I’ve seen that uses a process-oriented approach, which is so sorely needed in the real world.

The foreword was written by none other than W. Edwards Deming, and in it he explains the approach to statistics needed:

“Prediction is the problem, whether we are talking about applied science, research and development, engineering, or management in industry, education, or government,” he says. “The question is, ‘What do the data tell us? How do they help us to predict?’

“Unfortunately, the statistical methods in textbooks and in the classroom do not tell the student that the problem in data use is prediction. What the student learns is how to calculate a variety of tests (t-test, F-test, chi-square, goodness of fit, etc.) in order to announce that the difference between the two methods or treatments is either significant or not significant. Unfortunately, such calculations are a mere formality. Significance or the lack of it provides no degree of belief—high, moderate, or low—about prediction of performance in the future, which is the only reason to carry out the comparison, test, or experiment in the first place.

“… [I]nterchange of any two numbers in the calculation of the mean of a set of numbers, their variance or their fourth moment does not change the mean, variance, or fourth moment.

“In contrast, interchange of two points in a plot of points may make a big difference in the message that the data are trying to convey for prediction.

“The plot of points conserves the information derived from the comparison or experiment.”

And, in addition to the process output being measured, determining the sample itself to be measured is its own separate process. The concepts of “randomness” and “sample size for significance” go out the window.

Deming coined the term “analytic” to describe studies to improve a product or process in the future:

• Prediction is the aim.
• There is a need to conduct multiple plan-do-study-act (PDSA) cycles over a wide range of conditions.
• There are limitations of commonly used statistical methods such as analysis of variance to address the important sources of uncertainty in analytic studies.
• Graphical methods of analysis are primary.

Confirmation of the results of exploratory analysis comes primarily from prediction rather than from using formal statistical methods such as confidence intervals. Satisfactory prediction of the results of future studies conducted over a wide range of conditions is the means to increase the degree of belief that the results provide a basis for action.

When planning to test a change, people are making a prediction that the change will be beneficial in the future. What people don’t realize is that a limited set of conditions will be present during the test; the conditions in the past, during the test, and in the future could all be different. Circumstances unforeseen or not present at the time of the test will arise in the future. Will the change still result in an improvement under these new, future conditions?

Knowledge about the change is based on the specific subject matter on which the change itself is based, as well as knowledge about the environment in which the change will be implemented. Extrapolating the test results to the future is the primary source of uncertainty when a change is tested. The question then becomes, “How does one randomly sample the future?” Easy: One can’t.

The connection between knowledge of the subject matter from which the change is developed and analysis of the data from a test of the change is essential to effective improvement. This cannot happen in a statistical vacuum.

Integrating statistics’ role into leadership philosophy

The fact that most leadership is clueless to the power of statistical thinking in everyday management certainly doesn’t help quality professionals’ efforts. That said, quality practitioners need to start by improving the  process of teaching statistics, especially before they attempt to bring current seminars into the “C-suite.” Much of what is currently taught shouldn’t be applied to daily management—or probably most anything else (except maybe manufacturing product quality). The wrong things continue to be taught: p-values, confidence intervals, normal distribution, sample size, and regression, to name a few.

I once gave a talk following an ASQ Fellow who tried to make a case for bringing a quincunx into the board room—and passing out three pages of statistical definitions. I could feel the tension in the room rising. I then began my talk by saying, “If I brought a quincunx into a board room, they’d throw me out on my ear,” and the room erupted in laughter.

Where to start? Here is a quote from Dr. Donald Berwick, a pioneer in health care improvement:

“Plotting measurements over time turns out, in my view, to be one of the most powerful devices we have for systemic learning…. Several important things happen when you plot data over time. First, you have to ask what data to plot. In the exploration of the answer, you begin to clarify aims, and also to see the system from a wider viewpoint. Where are the data? What do they mean? To whom? Who should see them? Why? These are questions that integrate and clarify aims and systems all at once…. If you follow only one piece of advice from this lecture when you get home, pick a measurement you care about and begin to plot it regularly over time, you won’t be sorry.”

Until the culture at large appreciates the concept of “process” and eradicates blame, true improvement will not take place. To “solve” their problems everyone in a culture truly committed to improvement must work from perspectives of:

• Customer orientation
• Continuous improvement
• Elimination of waste
• Prevention, not detection
• Reduction of variation
• Statistical thinking and use of data
• Adherence to best-known methods
• Use of best available tools
• Respect for people and their knowledge
• Results-based personal feedback

Creating this culture is far, far more important than teaching a bunch of statistical techniques.

There are four statements regarding control charts that are myths and in my experience, just refuse to die. The next time you’re sitting in a seminar and someone tries to teach you how to transform data to make them normally distributed, or at any point during the seminar says, “Normal distribution” twice within 30 seconds, leave. You’ve got better things to do with your time.

The four myths

When you attend statistical seminars, do some statistical calculations seem like this? (2 minutes)

Are you “taught” these four things about control charts?

1. Data must be normally distributed before they can be placed on a control chart.
2. Control charts work because of the central limit theorem.
3. Data must be in control before you can plot them on a control chart.
4. Three standard deviation limits are too conservative.

April fool!

If you happen to be in a seminar where someone tries to teach this nonsense, invite them to click this link. It is a complimentary brain scan that will assess what “color” belt a person’s knowledge warrants. (Take it yourself; it’s a hoot.)

And then, just to make sure they’re certified as claimed, give them the “Certification Activity Book” obtainable through the tab in the left margin of that page.

I’ll leave it to your judgment whether you say “April fool” or not.

OK, time to get serious for a few minutes

As I like to say, the I-chart (a control chart for individual values, i.e., not subgrouped) is the Swiss army knife of control charts. During the early 1990s, statistical process control methods were the favored tool for medical quality improvement. When I used it, I came up against a lot of resistance from the entrenched “randomized double-blind clinical trial” cultural mindset that had been the norm in medicine—and provided the perfect smoke screen not to change.

Let’s consider these four myths in greater detail. (The first three are courtesy of Donald J. Wheeler, Ph.D. in a column 15 years ago.)

Myth No. 1: Data must be normally distributed before they can be placed on a control chart.
Reality: Although the control chart constants were created under the assumption of normally distributed data, the control chart technique is essentially insensitive to this assumption. The normality of the data is neither a prerequisite nor a consequence of statistical control.

Myth No. 2: Control charts work because of the central limit theorem.
Reality: The central limit theorem does indeed apply to subgroup averages. Because many statistical techniques use the central limit theorem, it’s only natural to assume that it’s the basis of the control chart. Ready for a shocker? I don’t even teach it.

It does have some justification in the case of X-bar-R and X-bar-S charts, but, especially in manufacturing, people usually miss the point and superimpose specification limits on the chart. (Wrong!) As I’ve said, these are rarely used in medicine because one generally does not have the luxury of subgrouping.

Actually, the central limit theorem is pretty much irrelevant to the I-chart. This myth has been one of the greatest barriers to the effective use of the I-chart with management and service-industry data, where data obtained one-value-per-time-period is the norm.

Believing this myth to be true and having no doubt endured a lengthy lecture or demonstration of the central limit theorem, people feel compelled to average something to make use of it. As Wheeler says, “The rationality of the data analysis will be sacrificed to superstition.” As a decision criterion, an I-chart with three standard deviation limits—calculated correctly—is very robust to almost any data distribution.

Myth No. 3: Data must be in control before you can plot them on a control chart.
Reality: I find that people generally make this conclusion only from computing limits incorrectly. Among the blunders that have been made in the name of this myth are getting rid of “obvious” outliers prior to charting them, and using limits that aren’t three standard deviations (see myth No. 4).

The purpose of the chart is to detect lack of control. It’s a very, very valuable initial diagnostic tool for a process. So tell me: If a control chart can’t detect lack of control, why use it?

Myth No. 4: Three standard deviation limits are too conservative.
Reality: Walter Shewhart, the originator of the control chart, deliberately chose three standard deviation limits. He wanted limits wide enough so that people wouldn’t waste time interpreting noise as signals (a Type I error). He also wanted limits narrow enough to detect an important signal that people shouldn’t miss (avoiding a Type II error). In years of practice he found, empirically, that three standard deviation limits provided a satisfactory balance between these two mistakes. My experience has borne this out as well.

I’ve seen two standard deviation limits commonly used because people, especially in medicine, are obsessed that they might “miss something.” There are two major reasons people do this:

1. The “two standard deviations” criterion for (alleged) significance has been drummed into peoples’ heads as the gold standard for decision making. This reasoning is based on the central limit theorem and making only one decision. (See my newsletter, “Why Three Standard Deviations?”)

2. They have performed an incorrect calculation of the standard deviation that has (unknowingly) resulted in an inflated estimate.

Novices continually think that they know better and invent shortcuts that are wrong. I once had a chart where my three standard deviation limits, calculated correctly, were equivalent to 1 1/2 standard deviations of the proposed analysis (needless to say, calculated incorrectly).

You almost never use the calculation of standard deviation taught in your “basic” statistics class, which, unfortunately, is so readily available in most spreadsheet programs. If the very special causes you are trying to detect are present, they will seriously inflate the estimate. Not knowing this, people will even try to use one standard deviation as an outlier criterion.

So, in the spirit of baseball season starting, let’s tap into the wisdom of “The Ol’ Perfesser.” If you ever ask a question in a statistical seminar, and the answer in any way resembles the following, leave:

“Well, I will tell you I got a little concerned yesterday in the first three innings when I saw the three players I had gotten rid of, and I said when I lost nine what am I going to do, and when I had a couple of my players I thought so great of that did not do so good up to the sixth inning, I was more confused but I finally had to go and call on a young man in Baltimore that we don’t own and the Yankees don’t own him, and he is doing pretty good, and I would actually have to tell you that I think we are more like a Greta Garbo-type now from success.”

This is how legendary baseball manager Casey “The Ol’ Perfesser” Stengel testified before a special Congressional House subcommittee on July 8, 1958. The committee was studying monopoly power as it applied to baseball’s antitrust exemption, and Stengal was asked if his team would keep on winning. (This is just a fraction of Stengel’s 45-minute discourse, the rest of which is just as priceless, along with Mickey Mantle’s followup.)

In summary

Trying to teach fancy theory does no one any good. W. Edwards Deming emphasized a basic understanding of variation and taught few techniques in his seminars. To do the type of work required to improve everyday culture, only 1–2 percent of people need advanced statistical knowledge.

Deming is probably rolling over in his grave at the subculture of “hacks” (his term) that has been created in the name of quality. Will the 80/20 rule inevitably apply to quality professionals? I answered that question here. And if the following is how your role is perceived, consider yourself forewarned (3 minutes):

As quality professionals, we must be careful not to perpetuate deeply embedded stereotypes of “sadistics” by making seminars nothing short of legalized torture and keeping our roles self-serving. Take it as a given: The people whom we teach will never like statistics as much as we do. So don’t teach people statistics—teach them how to solve their problems.

I will close with some more baseball wisdom. Like the Ol’ Perfesser, Yogi Berra is another beloved baseball icon who tends to unintentionally misspeak. Given the economy, many of us might unexpectedly face a career crossroads during the next few years. In fact, Yogi warns us, “It gets late early out there,” so I’m sure he would advise: “When you come to the fork in the road, take it.” Because (and I paraphrase) if people ain’t gonna go to statistics classes, how we gonna stop ‘em?

When teaching the I-chart, I’m barely done describing the technique (never mind teaching it) when, as if on cue, someone will ask, “When and how often should I recalculate my limits?” I’m at the point where this triggers an internal “fingernails on the blackboard” reaction. So, I smile and once again say, “It depends.” By the way…

… Wrong question!

I made a point in Part 1 of this article that I feel is so important, I’m going to make it again: Do not bog down in calculation minutiae. If you feel the instinct to ask that question, pause and think of how you would answer these from me instead:

1. Could you please show me the data (or describe an actual situation) that are making you ask me this question?

2. Please tell me why this situation is important.

3. Please show me a run chart of these data plotted over time.

4. What ultimate actions would you like to take with these data?

And since writing Part 1, I’ve thought of a fifth question I’d like to add:

5. What “big dot” in the board room are these data and chart going to affect? Or less tactfully,

5a. Who cares whether the limits are correct or not?

When you supply me with the answers to questions 1 and 2, then we can begin a dialogue, during the course of which I would be happy to answer your question about limits.

OK, I’ll answer the question now… sort of

The purpose of the limits is to give a reasonable range of expected performance due to common cause. For the I-chart, as long as the limits are computed correctly—via the moving range between consecutive observations in time order—and “three sigma” are used, then they are “correct limits.” As Donald J. Wheeler likes to say, “Notice that the definite article is missing.” They are just “correct limits,” not “the correct limits.”

Ready for a blinding flash of the obvious? The time to recompute the limits for your charts comes when, in your best judgment, they no longer adequately reflect your experience with the process. There are no hard and fast rules. It is mostly a matter of deep thought analyzing the way the process behaves, the way the data are collected, and the chart’s purpose.

If the process has shifted to a new location, and you don’t think there will be a change in its common-cause variability, then you could use the former measure of variation in conjunction with the new measure of location to obtain temporarily useful limits. Meanwhile, it would probably be a good idea to keep track of the moving range on an MR-chart to note any obvious changes. There is no denying that you will need to ponder the issue of recalculating the limits. With today’s computers, as mentioned below, it’s less of an issue; however, it still requires good judgment.

Wheeler wrote a column 15 years ago that is every bit as relevant today. So, let’s have him ask you three questions:

1. Do the limits need to be revised for you to take the proper action on the process?

2. Do the limits need to be revised to adequately reflect the voice of the process?

3. Were the current limits computed using the proper formulas?

Still not sure? Look at the chart and ask these additional questions Wheeler added from Perry Regier of Dow Chemical Co.:

1. Do the data display a distinctly different kind of behavior than in the past?

2. Is the reason for this change in behavior known?

3. Is the new process behavior desirable?

4. Is it intended and expected that the new behavior will continue?

If the answer to all four questions is yes, then it is appropriate to revise the limits based on data collected since the change in the process.

If the answer to question 1 is no, then there should be no need for new limits.

If the answer to question 2 is no, then you should look for the special cause instead of tinkering with the limits.

If the answer to question 3 is no, then why aren’t you working to remove the detrimental special cause instead of tinkering with the limits?

If the answer to question 4 is no, then you should again be looking for the special cause instead of tinkering with the limits.

The objective is to discover what the process can do or can be made to do.

Yes, indeed: It depends.

Wait for it…

Frustrated by my lack of a concise answer and now trying to distract me from pressing for answers to all these questions, I then get asked, “Well, even though I can’t think of a situation, how many data points are needed to compute accurate limits?”

I generally answer, “How much data have you got?” (It’s usually not very much.)

In my experience, useful limits may be computed with small amounts of data. Even as few as seven to 10 observations are sufficient to start computing limits, especially if, as frequently happens to me, it’s all you’ve got. What else are you going to do? I dare you to find a more accurate way to assess the situation. I chuckle when I think of how many times executives have told me, “Your way of doing things has too much uncertainty.” I’ve been so tempted to answer, “So exactly what are you going to do instead?”

The limits do begin to solidify when 15 to 20 individual values are used in the computation. To argue semantics, when fewer data are available, the limits can be considered “temporary limits,” subject to revision as additional data become available. When more than 50 datum are used in computing limits, there will be little point to further revise the limits.

However, who does charts by hand anymore? Given today’s computer packages, limits are automatically updated as new data are added, so what’s the problem? You might have to make a decision about what period to aggregate for the appropriate moving range statistic, but it’s a somewhat minor point. Frankly, it’s a question I rarely consider; I generally have far too many questions regarding the process being improved. After those are settled, the calculation process always somehow seems to sort itself out. Rest assured, by focusing on the process, you will get “correct limits.”

So stop getting sucked into the swamp of calculation minutiae. Instead, spend all that energy using your charts to understand and improve your processes. And the first time you say, “It depends” in answer to someone’s question, let me know, and we’ll both smile.

Quality improvement people sure love those tools. A particular favorite, of course, is the control chart, of which, I think, seven are usually taught. Two questions I’m always asked are, “Which chart do I use for which situation?” and “When and how often should I recalculate my limits?”

Wrong questions!

Regarding the first (we’ll deal with second question in part 2), I’ve seen many flowcharts in books to help you determine which chart to use for which situation. I find them far too confusing for the average user. (They even give me sweaty palms.) I don’t even teach this in my work.

As my respected colleague Donald Wheeler likes to say, “The purpose is not to have charts. The purpose is to use the charts…. You get no credit for computing the right number—only for taking the right action. Without the follow-through of taking the right action, the computation of the right number is meaningless.”

So let’s get back to basics… again

As I said in my Feb. 21 newsletter, “Making Variation Disappear on Paper Doesn’t Make It Disappear in Reality,” after you know how your data were defined and collected, the next step is to assess the process that produced the data. This is done via a run chart (or as I like to say, “filter No. 1”) and most of the time, a subsequent control chart (“filter No. 2”). This rationale was explained in my Jan. 24 newsletter, “What Part of ‘Never’ Don’t People Understand?”

Because the data are in a time sequence (or should be), the control chart of choice with which to start is the individuals chart; it uses the moving range (MR) between consecutive points to determine the limits. The individuals chart is the “Swiss army knife” of control charts. (In some of the figures in the newsletters mentioned above, it is advertised as an I-MR chart combination. Don’t worry too much about the MR chart for the moment; I will address this in a future column.) It usually approximates the “correct” chart under most conditions.

I can hear the chorus, “So, what are the conditions when it isn’t correct?” Remember Dr. Donald M. Berwick’s recommendation, “If you follow only one piece of advice from this lecture when you get home, pick a measurement you care about and begin to plot it regularly over time.” Note that he doesn’t say, “Plot a control chart of that data.”

The bottom line is that before you ask, “Which chart should I use for which situation?” or challenge me with a “what if” doomsday scenario (I’m truly amazed at the creative hypothetical situations thrown at me during my teaching), let me request:

• Could you please show me the data (or describe an actual situation) that are making you ask me this question?
• Please tell me why this situation is important.
• Please show me a run chart of these data plotted over time.
• What ultimate actions would you like to take with these data?

If you have the patience to answer these and follow them through with a data set to an appropriate action, you will have probably answered the question yourself—solving a major problem in the process—and saved yourself a major side trip into the “swamp” of calculation minutiae.

Let’s consider the various charts

X-bar/R chart and X-bar/S chart. Since a lot of readers are in health care, I’ll tell you right now: You virtually never use these. They were designed for manufacturing processes where thousands of parts are made per day, and it’s no big deal to grab, say, four to five parts consecutively produced every hour (which takes seconds). People, even at manufacturing facilities, seem to find these confusing and hard to use because the limits are based on averages.

In health care clinical applications, you don’t have patients coming through in numbers similar to an assembly line. I literally cannot remember the last time I’ve used these charts in my health care work. They might lend themselves to high-volume administrative processes, but once again, “it depends,” and people still find them confusing.

So I never formally teach them and would do so only if needed in the context of solving an issue.

C-chart (for counts). These are easily approximated by the individuals chart, especially if theaverage is at least five. Remember, the process’s stability is the key question and determines whether you subsequently use a common- or special-cause strategy.

Small numbers (and rare events) get very tricky and usually require guidance by a statistical expert to get the “right number.” Regardless, the run chart and individuals chart will generally lead you to the right initial action.

P-chart and u-chart. When p-charts (percentages) and u-charts (rates) are plotted over time, pretty much everyone gets confused by the stair-step limits caused by the varying denominator sizes. This confusion only escalates by well-meaning attempts to explain them. Occasionally, the chart might come in handy for finding an individual outlier.

More important, these also become especially problematical, especially for p-charts, when data are aggregated monthly, quarterly, or even annually. The resulting large denominators (as in hundreds or thousands) create many artificial out-of-control signals (i.e., above or below the limits).

Donald J. Wheeler, Ph.D., whose books are very practical, well-written, and offer good examples (you can peruse them at www.spcpress.com), is of the opinion that true independence of occurrence of events in reality is rarely encountered. So, he feels it’s correct to use only the individuals chart. My experience has pretty much borne this out. But, “it depends,” and gets tricky with small denominators.

Np chart. As far as the np chart goes, it’s a marginal pedagogical classroom exercise—and that’s about it. I virtually never use it because having equal sample sizes in the denominator is a rare luxury indeed. Or the machinations to create equal denominators then explain the resulting chart to one’s puzzled audience far outweigh any benefits.

In defense of p-charts and u-charts

I have found p-charts and u-charts to be helpful in stratification (a common-cause strategy). One uses them to compare, statistically, individual rates that each have been obtained by:

• Plotting a run chart
• Following up with a control chart
• Determining the most recent stable history, and only then,
• Aggregating the data from No. 3 into summed numerators and denominators for statistical comparison

In these cases, the horizontal axis is not “time.” It could be, for example, individual doctors or locations. Many of you have no doubt encountered these via the fancy euphemism “funnel plots,” where the results are sorted horizontally by increasing denominator size.

Always start with process data gathered over time and plot a run chart

First, find out what you’re “perfectly designed” to get, and second, see whether common or special-cause strategies are needed to further solve the problem. This gives you a baseline with which to assess the current state as well as your subsequent intervention efforts.

Many projects fail because they lack a baseline. So you see, there’s never an escape from plotting your process data over time. As Berwick says at the end of his quote, “You won’t be sorry.”

Part 2 will look at “the question that drives me nuts” and explode some common myths about control charts.

You know what the third-quarter review meeting means: a packet will be handed out  with bar graphs and, no doubt, trend lines on each of about a zillion “key performance indicators” that show:

• This month vs. last month vs. 12 months ago (maybe year-to-date as well)
• The three months’ performance of the current quarter
• The first three quarters of the year
• This quarter vs. last quarter vs. third quarter a year ago

(Of course, tables are included with red, yellow, and green cells measuring performances and variances from targets, for which you are already preparing your explanations as to why you didn’t achieve them, right?)

Data insanity as a source of waste

Many of you know me as a statistician, yet I’ve been talking about organizational transformation in my last two articles and the need to take improvement to the next level—“built-in” vs. “bolt on.” And I do feel that a key catalyst to accelerate this process is my “data sanity” concept.

In case you aren’t familiar with my definition of data sanity: It’s the everyday use of data in a process-oriented context so as to react appropriately to its inherent variation and achieve improvement and better prediction.

Or put another way: Don’t treat all variation as special cause and take actions that increase complexity.

Most days, doesn’t it feel like you’ve been hired to ride in a car with four flat tires going down the highway at 70 miles per hour with your job being to change the tires? And you’re given only one small caveat: You can’t stop the car under any circumstances. However, you are empowered to lecture the driver on improvement or even give her appropriate audio books for self-study (and hope the car doesn’t have an ejector-seat button).

Mark Graham Brown, for whom I’ve developed a respect because of his excellent writings on balanced scorecards, has said:

• 50 percent of the meetings that executives attend involving data are wasted time
• Middle managers waste one hour a day poring over useless data
• 80 percent of published financial data is waste
• 60 percent of routine published operational data is waste

These meetings also result in subsequent work for you and other people looking for reasons why some key indicators went “up” or “down” or didn’t achieve established targets. But because you are very smart people—that’s the problem—you find them. In process parlance, every deviation from a target is treated as a special cause.

The time-chasing special cause explanations could be spent communicating vision and strategy, and be devoted to improvement as the “learning and growth” arm of the business (learning and growth is one of the categories from the balanced scorecard’s original theory).

But anyway… back to the quarterly review meeting

Let’s see, how could three numbers look… and be interpreted?

Isn’t it amazing that given three different numbers, there are six different ways they can manifest? And each one has its “special cause” explanation. But, is it really a special cause?

Also, two of the six patterns fit a widespread preconception of the word—trend: Either all the points go up or all the points go down.

Let’s talk about that ubiquitous word “trend.”

In this case, with three different data points, given the fact that these are two out of six possibilities, there could be a 33-percent potential risk of being wrong by arbitrarily declaring three points a trend, i.e., calling something a trend (special cause) when it is merely a common cause.

Did you know that the statistical rule of thumb (based in theory) to declare a true trend is a cluster of seven data points either all going up or all going down, i.e., six successive increases or decreases?  If there are 20 or less points being plotted, one can then use six consecutive points, or five successive increases or decreases.

I give you this rule to mainly tell you what a trend is not, because its occurrence is relatively rare.

What does trend mean anyway?

Putting it in a process context, I like to think of the trend signal as showing a process in transition: You previously had a process that was perfectly designed to get the results it was getting and you have done an intervention (tried to create a beneficial special cause). The process is transitioning to what it is perfectly designed to get given its new inputs (your transition).

And here is the key point:  transitioning to what it is perfectly designed to get—at which point it will level off.

Here is a wonderful analogy:  The holiday season will soon be here and for many us, it will be time to make our yearly pledge to reduce our weight (sort of like the goal setting organizations do at annual budget time). OK, so we set a target and plan to weigh ourselves every day.

Next, we cut calories (some of us throw in exercise). Throughout the next four weeks or so (especially during the first two weeks), some of us will no doubt see the five or six successive decreases in a row. (Don’t lose heart if you don’t. There is another test to show progress).

In the reading I’ve done, right at the beginning of such an effort, there are about 5 to 10 lb of “water weight” just waiting to be lost if your calorie intake differs significantly from the norm. And if you are able to stay on track, the results in weeks three and four are less dramatic, but still effective: maybe 1 to 1.5 lb a week.

Then what happens? Your body has adjusted to the weight that it is “perfectly designed” to reach given your diet regimen. Any further loss will require another new process (usually more exercise, in my case). And if you haven’t reached your target weight after week four, do not despair. Take your data and put in the trend line. Not only will you be able to tell when you will reach your target weight, you can then project when your weight will go to zero, right?

Of course not, so if this nonsense of doing trend analysis is apparent for your weight, why don’t people make the analogy to organizational processes?  This is assuming they have first accepted the fact that they do indeed manage processes.

So, trend = transition.

Here is a more important question: At what value does the trend stop—if there was even a trend in the first place?

Bottom line: Once you account for the number of meeting hours multiplied by salaries, multiplied by benefits, or as in the case of an all-day meeting, an entire day multiplied by everyone’s salary and benefits… how much would data sanity save?

I attended a talk in 2006 given by a world leader in quality that contained a bar graph summary ranking 21 U.S. counties from best to worst (see figure 1). The counties were ranked from 1 to 21 for 10 different indicators, and these ranks were summed to get a total score for each county (e.g., minimum 21, maximum 210, average 110. Smaller score = better). Data presentations such as this usually result in discussions where terms like “above average,” “below average,” and  “who is in what quartile” are bandied about. As W. Edwards Deming would say, “Simple… obvious… and wrong!” Any set of numbers needs a context of variation within which to be interpreted.

Rank Sum             County
42                                  1
76                                  2
84                                  3
87                                  4
92                                  5
99                                  6
101                                  7
102                                  8
105                                  9
105                                10
107                                11
108                                12
112                                13
113                                14
114                                15
121                                16
128                                17
131                                18
145                                19
157                                20
181                                21

Figure 1: Summary of 21 U.S. counties

I asked for the original data (i.e., the individual sets of rankings for each of the 10 characteristics), and the presenter was kind enough to supply it. My analysis showed there was one “above average” county (No. 21) and one “below average” county (No. 1). Counties 2–20 were indistinguishable. (If you’re interested in the statistics involved, you can view them here.)

I then shared my analysis with him. Our e-mail correspondence follows:

World quality leader: A subtle issue you did not tackle is the political-managerial issue of communicating such insights to [the two special-cause counties] and the counties that thought they were “different” but, statistically, aren’t. I wonder what framework one could use to approach that psychological challenge?

Balestracci: As I say to my audiences, “Hey, I’m just the statistician, man!” I think the issue is how people and leaders like you are going to facilitate these difficult conversations. This is the leadership that quality gurus keep alluding to and seems to be in very short supply.

My job is to keep you all out of the “data swamp,” but I would be a willing participant. I would love to pilot some of these analyses with you or other leaders. We need to figure out what this process should be. This is potentially very exciting and could quantum-leap the quality improvement movement.

My point is that this “language” needs to be a fundamental piece of any improvement process and led by leaders who understand it and are promoted into leadership positions only if they understand it. If this could become culturally inculcated, then the rampant shoot-from-the-hip analyses and resulting defensiveness would stop, period. The discussion would then focus, as it should, on process. We need new conversations, and this could be a key catalyst.

World quality leader: Nope. I don’t buy it. Yes, I am a leader and need to carry the message. But I know you too well to let you off the hook. I’d love to see you try to lead these conversations and experiment with approaches. You’re a leader, too.

Balestracci: Give me an opportunity, and I will do my best to lead that conversation. Have you fathomed the potential of this?

Real root causes?

That last e-mail of mine has never been answered. I’m still waiting for the promised opportunity. I try to remind him every once in awhile but have given up. During the past four years, further e-mails from me have not been responded to. At his insistence, I even sent the analysis with explanation to the original executive group that collected and summarized the data. No reply.

Many of this example’s statistical principles are what Deming demonstrated during his seminars. After more than 20 years of trying to teach similar concepts, I am still amazed at the abject cowardice of (yes, cowardice) and fierce resistance from (alleged) leaders who abdicate responsibility to comprehend the power of a simple understanding of variation. As a lot of us know, Deming had zero tolerance for such ignorance or arrogance.

Let me tie this reaction into the current hot topic of root cause analysis. An excellent article by John Dew, “The Seven Deadly Sins of Quality Management” (Quality Progress, 2003) considers the true root causes to quality problems. They are entrenched in a “quality as a bolt-on” culture, of which the conversation I had above is symptomatic. These root causes include:

1. Placing budgetary considerations ahead of quality
2. Placing schedule considerations ahead of quality
3. Placing political considerations ahead of quality
4. Being arrogant
5. Lacking fundamental knowledge, research, or education about improvement
6. Pervasively believing in entitlement
7. Practicing autocratic behaviors that result in “endullment” rather than empowerment

Regarding items 4 and 5, I believe quality professionals have made huge strides in speaking the language of senior management. In fact, maybe too good; I’m seeing an increasing emphasis on “bottom line results.” In many organizations, senior management still does not know the fundamental lessons of quality and, frankly, shows no interest in learning them other than insisting, “Get to the punchline and give me the 10-minute overview and bottom-line results.”

Promotions self-perpetuate the status quo. Could it be that few quality managers make it into senior management positions because senior management does not really believe in quality concepts?

Am I the only one who sees the potential implications of this simple example?

Mark Graham Brown, a balanced scorecard and measurement expert, thinks that 50 percent of executive meetings where data are involved are a waste of time—as is middle management spending an hour a day poring over useless operational data. (Put that into a dollar figure.)

Why is it the only people who truly don’t seem to get it, or want to get it, tend to:

• Look at tables of raw data and draw circles around numbers they don’t like
• Look at data summarized by smiley faces, bar graphs, trend lines, and traffic lights
• Compare a number to an arbitrary goal and throw a tantrum
• Brag about reading the latest airport best-seller, leadership-fad book

Sigh. Passionate lip service continues to be alive and well.

What can you do?

Herein lies the opportunity for quality professionals: Getting the respect we deserve by bringing “data sanity” to organizations, which would free up precious time to consider and make quality an organizational “build in.” People in quality must stop seeing themselves as victims or being complacent because they are “so busy.” Activity is not impact. (See my 2009 column on this subject here.)

Join me and watch like a hawk for opportunities to convert everyday executive data presentations into this “funny statistical way” of doing things. This will keep you from doing yet another self-sabotaging seminar simulating Deming’s red bead experiment. We need to stop whining that people “just don’t get it” and think more formally about how to stop boring execs to death.

Getting mad and focusing that energy wouldn’t hurt, either.

During my recent travels, I have noticed an increasing tendency toward formalizing organizational quality improvement (QI) efforts into a separate silo. Even more disturbing is an increasing (and excruciating) formality. Expressions such as “saving dark-green dollars” are creeping into justifications for such “programs,” usually referred to as Six Sigma, lean, or lean Six Sigma. As always, Jim Clemmer pinpoints this trend perfectly:

“The quality movement [has given] rise to a new breed of techno-manager—the qualicrat. These support professionals see the world strictly through data and analysis, and their quality improvement tools and techniques. While they work hard to quantify the ‘voice of the customer,’ the face of current customers (and especially potential new customers) is often lost. Having researched, consulted, and written extensively on quality improvement, I am a big convert to, and evangelist for, the cause. But some efforts are getting badly out of balance as customers, partners, and team members are reduced to numbers, charts, and graphs.”

“Doing” QI vs. transforming an organization

Contrast Clemmer’s description with my view of how a quality focus should transform an organization:

QI: “One-shot” skills training via courses
Transformation: Routine, continuous education through daily work

QI: Many teams of key personnel focused on routine, daily operational issues
Transformation: A few top management-led teams focused on key strategic issues

QI: Heavy emphasis on tools
Transformation: Entire work culture educated in QI theory

QI: Focus on obvious, current problems—representing 3–15 percent of opportunity—through:

• Formal problem identification
• Problem-solving tools
• Management guidance teams
• Formal team reviews
• Storyboards
• QI coordinator and formal quality structure

Transformation: Focus on hidden problems—representing 85–97 percent of opportunity—through:

• Appreciation of systems and interactions
• Cultural and individual psychology
• Deep understanding of variation
• Use of data to test improvement theories
• Continuous establishment and documentation of routine processes important to customers

QI: Team facilitators with QI tools skills
Transformation: Change agents with formal cultural change skills in addition to problem-solving skills

QI: Arbitrary numerical goals and traffic-light reporting
Transformation: Understanding variation and process capability through targets, runs, and control charts. Establishing an integrated measurement system via a balanced scorecard

QI: Management behavior is:

• Comfortable with maintaining the status quo
• Shortsighted, so solves problems only as they crop up
• Reactive to variation and treats each as unique with special causes
• Task-oriented, so chooses projects and reviews progress
• Distancing, so sends people to courses

Transformation: Management behavior is:

• Engaged and seeks to understand and improve processes
• Constantly facilitating problem solving and removing cultural barriers
• Proactively responsive to variation, so asks, “Is this common or special cause?”
• Able to exhibit QI skills through behavior
• Interested in teaching QI through routine daily work and meetings

QI: Quality is a “certain percent” of the job and explicit
Transformation: Quality is 100 percent of the job and implicit

Built in, not bolted on

Until quality concepts permeate an organization’s culture to the point where the words “statistical” and “quality” are dropped as qualifiers because they are givens, any “dark-green dollar” savings will be nickels and dimes compared to what is truly possible.

Many organizations are still locked into the mindset of quality as a “bolt-on” program rather than the strategy for developing a strategy. And the “guru vs. guru wars” continue, which means that people still don’t get it.

I once received some feedback from a concerned reader who thought I tended to poke fun at quality programs, especially lean Six Sigma (LSS). I don’t (well, maybe a little), but he said it so well: “I suppose it is valid to poke at those who would market [lean Six Sigma] as a quick fix (and deliver nothing but high-priced training), but I do believe that when viewed as a culture, infrastructure, methodology, and metric, LSS is a disciplined way to organize for quality and make improvements project by project. However, it pains me to have our initiative questioned by folks who, when I referred them to your site (to order your book), come back even more suspicious about LSS.”

In essence, he and I agree. This person does indeed get it, and I apologize for any others I may also be offending.

But the comment got me thinking. There is a message that always bears repeating: Total quality management (TQM), continuous quality improvement (CQI), Six Sigma, LSS, and the Toyota Production System all come out of the same theory, which truly hasn’t changed in the last 22 years.

And, by the way, did you notice that my reader’s comment included the term “project by project?”  I’m not sure that approach (on which Juran’s success was based during the 1970s and 1980s) is going to be effective any more. As I’ve tried to show, this thinking must also infiltrate the everyday management of any organization.

Quality, when integrated into a business strategy, is present in virtually every aspect of every employee’s everyday work. Process-oriented thinking is the anchoring concept of any good improvement framework and creates a common organizational language that will reduce defensiveness. It’s not the problems that march into your office that are important: It’s the ones of which no one is aware.

And then there’s that statistical conundrum: Some lean purists argue that statistics have absolutely no place in the discussion, while the Six Sigma pros contend that it’s all about statistics. It’s become a sort of modern-day “How many angels can dance on the head of a pin?” debate. Don’t you think your time would be better spent considering how you can use your knowledge and experience to help your company move away from quality as a bolt-on program to a built-in culture change?

Have you become a “qualicrat?”

Customer satisfaction data resulting in various quality indexes abound. The airline industry is particularly watched. The April 10 Quality Digest Daily had an article with the title “Study: Airline Performance Improves” and the subtitle “Better on-time performance, baggage handling, and customer complaints.”

The analysis method? In essence, a bunch of professors pored over some tables of data and concluded that some numbers were bigger than others…and gave profound explanations for the (alleged) differences. If I’m not mistaken, W. Edwards Deming called this “tampering:” They treated all differences (variation) as special cause.

How much information like this gets published and how much of this type of (alleged) “analysis” are we subjected to in meeting after meeting…everyday?

“Released during a news conference at the National Press Club, the rankings show that of the 17 carriers rated in 2008 and 2009, all but Alaska Airlines had improved AQR scores for 2009.”

So, given 17 carriers, 16 had numbers bigger than last year. It sounds pretty impressive.

However, there is a deeper question: Is the process that produced the 2009 number different from the process that produced the 2008 number? Was there a formal method in place for improvement or was it just exhortation to “get better?” To paraphrase one saying of Joseph Juran’s, “There is no such thing as ‘improvement in general.’” And to paraphrase two saying of Deming’s, “A goal without a method is nonsense!” and “Statistics on the number of accidents does not improve the number of accident occurrences.” In other words, statistics on performance don’t improve performance.

So was it just a matter of work harder, work smarter?

Actually, in defense of the intuitive conclusion that things had indeed gotten better, statistics can be applied to this situation: Using the simple nonparametric technique called the Sign Test, given 17 pairs of numbers, the p-value of 16 out of 17 paired numbers being bigger just due to chance is 0.0003. In other words, based on this data, there is a 0.03 percent chance of being wrong making this statement, which is a pretty small chance.

Now, if 13 out of the 17 had gotten better, would you intuitively feel that things had improved? Probably. The p-value for that is almost exactly 0.05 (5% risk). For 12 improvements, the p-value now is 0.14 (14% risk). Surprised? This conclusion with the original data was pretty obvious, but sometimes things that “seem” obvious aren’t…and you’re probably just as well, if not better off, using a Ouija board (Well…you are using a statistical Ouija board of sorts).

The article had a reference where I was able to track down each key indicator’s 24 monthly numbers of 2008-2009. So, I did a “seasonality” analysis (Model: Year-over-year analysis as well as a seasonality analysis trying to determine whether certain months—regardless of the year—had a statistical tendency to always be “high” or “low”). I used an appropriate regression model and found the significant terms (via subset selection).

I then did the traditional regression diagnostics. They all analyze the residuals—the actual data values minus the model’s predicted values. The residuals of any model contain all of the information one needs to assess the model. Three diagnostics are usually taught in any good regression course:

1) the residuals vs. predicted value plot (should look like a shotgun scatter—this one was reasonable)
2) a Normal plot of the residuals (should look like a straight line and you get a p-value—this passed)
3) some type of lack-of-fit test to see whether the model is a reasonable one. This last test is based on having repeated x-values, which wasn’t the case here; however, many packages contain a proxy test using clusters of near neighbors as an approximation.

However, with process-oriented statistics, there is one additional diagnostic, which tends not to be taught in regression courses: I also plotted a run chart (time ordered plot with the median of the data drawn in as a reference) of the residuals, which should exhibit random behavior. This can help to find additional special causes due to interventions made at various times, which would invalidate the model as described above even if it passed all three of those diagnostics. Special causes require a model adjustment via dummy variables, which then requires a new subset selection with retesting of diagnostics.

When a reasonable model was found, I then used the model’s predicted values as the centerline of a control chart plot of the data.

The figure below shows the overall quality index as well as the individual plots for 14 of the airlines (Of the original airlines analyzed, I left American Eagle, Atlantic South East, and Comair out of the this and subsequent analyses. These were small regional airlines and there were issues with missing data and wide variation that would have clouded discussion of the major points of this article).

Oh, so how did they determine that the 2009 number was greater than the 2008 number? In other words, how were the 2008 number and 2009 numbers literally calculated (operational definition)? My best guess is that it was just comparing one 12-month average to another…and concluding that any difference was a special cause.

(click on any image to enlarge)

If one sees a distinct “shift” upward in the 2009 data vis-à-vis the 2008 data, that’s statistical evidence that the 2009 result was higher than the 2008 result. This appears in the overall score. One can also see the distinct drop for Alaska Airlines (AS). However, due to special causes for which the model was adjusted, the alleged increases for F9 (Frontier), NW (Northwest), SW (Southwest) aren’t necessarily “real” (consistent?), given the data. So, out of these 14 airlines, 10 got better, one got worse, and three stayed the same. Applying the Sign Test: p = 0.0117, still a good indicator of “overall” improvement. But, then again, what does “overall improvement” mean? The aggregation of all scores into an overall indicator is like saying, “If I stick my right foot in a bucket of boiling water and my left foot in a bucket of ice water, on the average, I’m pretty comfortable.” I don’t fly an “average” airline, I fly a specific airline.

So, I’m curious. Are you intrigued by this presentation? Oh, and, by the way, in the data report cited in the article, this is all presented (overall and for each airline) in individual line graphs, with a trend line automatically drawn in because things should “somehow” be getting better (I wonder if they tested this “model” with the four diagnostics I used. I doubt it.)

Computer packages just love to delight their customers—who want to be able to draw in trend lines willy-nilly. The packages are only too happy to oblige! Ask about “diagnostics” and you’ll get met with blank stares—from the people using the package and the people who wrote the package. And that’s not all.

The overall and individual airlines each had their own bar graph as well. The horizontal axis was “month” and the two years’ results for each month were side-by-side bars. Here they are for the overall quality metric:

(click on any image to enlarge)

In fairness, the model was statistically significant (Of course it was: You’re fitting, in essence, the “two” different points of 2008 and 2009!). This “trend” model also passed the three basic diagnostics—the residuals vs. predicted value plot looked reasonable, the residuals were normally distributed, and, even though there weren’t any true repeated points, a proxy test didn’t declare the possibility that the model could be wrong. BUT…the last, rarely taught, diagnostic—a run chart of the residuals plotted in their time order—makes the alleged trend’s house of cards come crashing down:

(click on image to enlarge)

As previously mentioned, the residuals of any model contain all of the information you need to assess your model. And, in this case, this plot tells several stories. Notice the eight-in-a-row (observations 4-11) all above the median in 2008. Notice also that the January and December residuals (Observations 1, 12, 13, and 24) for both years are consistently low—indicating seasonality.

As the saying attributed to George Box (or maybe it was Deming) goes, “All models are wrong. Some, however, are quite useful.” My seasonality model passed all four diagnostics and you can even see the seasonality of the January and December observations.

They say “never say never,” but I am about to make an exception: In my 30 years as a statistician, I have never seen an appropriate use of a trend line on sets of data like this…never. If I had my way, trend lines would be banished from every management statistical package.

And, speaking of “trend,” here is another similar conclusion:

“For the second consecutive year, the performance of the nation’s leading carriers improved, according to the 20th annual national Airline Quality Rating (AQR). It was the third best overall score in the 19 years researchers have tracked the performance of airlines.”

Think about it: Given three (different) numbers, there are six possible combinations, two of which we would call “trend”—all going up or all going down. See www.qualitydigest.com/feb05/departments/spc_guide.shtml.

So, the odds of all three data points denoting a “trend” going up or down is two out of six or 0.33. It might be nicer to have a time series plot of all 19 numbers or at least a context of variation for interpreting the three numbers. I know, I know…I can hear some of you saying, “Well, a lot has changed over 19 years, so the plot might not be valid.” OK…so why compare the current number to the previous 18 then—the same issues apply, don’t they? And, isn’t it amazing: Given 19 numbers, one is indeed the third highest.

Let’s move on.

“The industry improved in three of the four major elements of the AQR: on-time performance, baggage handling, and customer complaints. Denied boarding is the only element where the performance declined.”

Yes, indeed, and, once again, as the title of this article implies, given two numbers, one will be larger. And, yes, as you will see, I agree with this conclusion. But, I think my theory and analysis is slightly more solid than just noticing that two numbers are different, then jumping directly to a conclusion.

I performed the seasonality model and diagnostics as previously described. Here are the results for the overall industry data for the four indicators data aggregated for all airlines listed above (Denied Boarding was collected quarterly—eight data points vs. 24 for the others). So, which do you prefer and which gives you more confidence: A lucky guess with pretty graphs or an analysis based in theory?

(click on image to enlarge)

And then there is what some people consider the “bottom line”—the ultimate rankings: “Given a set of numbers, one will be the biggest…one will be the smallest…25 percent will be the top quartile…and 25 percent will be the bottom quartile.” But that’s a whole other article.

My point here is that it’s amazing how nonrandom randomness can look—and those bar graphs and trend lines are quite pretty aren’t they? And you are then at the mercy of the human variation in perception of the people in the room. How many meetings do you sit in with data presented this way? It reminds me of a quote I heard once, “When I die, let it be in a meeting. The transition from life to death will be barely perceptible.”

That’s why one should always begin by plotting the data in its naturally occurring time order, ask questions, and proceed cautiously to resist temptation to explain any (alleged) differences.