Attending the 20th International Forum on Quality & Safety in Healthcare arranged by no other than the British Medical Journal and the Institute for Healthcare Improvement one expects to see high quality presentations of high quality improvement work.

However, a tour around the poster area is a depressing experience. So much good work go to waste because of poor data presentation.

Don’t get me wrong. Many posters show data in nice graphs and tables. But a significant proportion of posters use ineffective or even misleading graphs to present their good work.

In the following I give examples of bad graph design found in the conference poster area and give suggestions on how to make graphs communicate better.

It is not my intention to point fingers at anybody. These examples represent important improvement work that fail to communicate great messages due to bad graph design. Should anyone feel bad about me using their data for this purpose, please let me know, and I will remove them from this post immediately.

Mealtimes Matter

A poster titled “Mealtimes Matter” reports a successful improvement project targeting elderly patients at risk of malnutrition. The results are presented in these two graphs placed away from each other on the poster.

Figure 1

Figure 2

What’s wrong with these graphs?

  • In order to get the message, you have to read the poster text explaining that the measure of interest is the percent of patients who replied “yes” to the question “Did you get enough help from staff to eat your meals?” Always make a graph self explanatory.

  • Placing before and after measurements in different graphs each with individually scaled y axes makes it hard, if not impossible, to se the improvement that actually happened. Always place before and after measurements in the same graph and on the same scale.

  • The Y axis in the after graph does not start at zero. Always start bar graphs at zero.

  • A bar graph is not the most effective way to present change over time. Always use a line graph to display change over time. Bars are good for nominal and ordinal comparisons of relatively few values but are ineffective to show trends and patterns over time. Furthermore, the Y axis in a line graph does not necessarily have to start at zero, which allows for better visual resolution of variation over time.

  • The bars are 3-D. Never use 3-D graphs to present 2-D data. By the way, never use 3-D graphs at all. The third dimension adds nothing but confusion and makes it impossible to visually decode the information hidden in the graph.

  • The information given in the legends is redundant. It is already present in the X axis labels. Always remove unnecessary elements from your graphs.

  • The meaning of the colours, red, yellow, green, is not clear. From reading the poster text, I assume that green bars mean that the results in these months are above a target of 90%. Never corrupt your quality improvement data with red-yellow-green approaches. Instead, use run or control charts to identify improvement or degradation over time.

A suggestion

Always begin analysis of your quality data with a run chart. A run chart is a line graph of a measure plotted over time with the median as a horizontal line. The main purpose of the run chart is to identify process improvement or degradation, which can be detected by statistical tests for non-random patterns in the data sequence.

If the process of interest shows only random variation, the data points will be randomly distributed around the median. “Random” meaning that we cannot know if the next data point will fall above or below the median, but that the probability of each event is 50%, and that the data points are independent. If the process shifts, these conditions are no longer true and patterns of non-random variation may be detected by two simple statistical tests. We call these patterns signals:

  • The shift signal is present if any run of consecutive data points on the same side of the median is longer than its prediction limit.

  • The crossings signal is present if the number of times the graph crosses the median is smaller than its prediction limit.

Prediction limits for the shift and crossings signals can be calculated or looked up in a statistical table. If you want to know more about run charts, read my two recent papers Run Charts Revisited and Diagnostic Value of Run Chart Analysis.

Send in the run charts

A run chart of the data presented in the mealtime poster shows clearly that improvement has occurred. The curve alone tells the story. But the run chart analysis adds “significance” to our subjective visual analysis. Using before data as baseline to establish the median, we find an “unusually” long run of data points above the median and that the curve crosses the median “unusually” few times.

For a detailed explanation of the run chart, read my previous blog post on Run Charts with R.

Providing Safer Care for Hip Fracture Patients

The next graph comes from a poster reporting an improvement programme to reduce length of stay for hip fracture patients.

The poster text concludes: “The mean time of hospital stay dropped from 15 to 5 days, as shown in the chart below.”

hospital stay

What is wrong with this graph?

  • The authors use a bar chart to show improvement over time. Always use line graphs for this purpose.

  • There is a trend line. Never use trend lines to show change over time A trend line suggests a linear relation between the outcome and time, which is an extremely rare situation. Quality moves in shifts rather than drifts. And even if a linear relation may be assumed, a run or control chart will detect it quickly.

  • The conclusion, “from 15 to 5”“, uses the actual first value of 15 as the starting point and the last trend line value as the ending point. Never mix apples and oranges.

  • Finally and most importantly, the conclusion, that length of stay has dropped is not supported by data, which is seen clearly from the next run chart. Always analyse improvement data with a run chart.

Run chart to the rescue

With 12 data points, we would expect the longest run to be no longer than 7 and the number of crossings to be at least 3. Both conditions are fulfilled and the run charts displays only random variation. The apparent trend is supported by the first data point alone.

However, the first data point does indeed seem to be out of tune with the rest. This calls for further investigation. It may well be random. But one could suspect that earlier on these patients actually did stay longer.

This hypothesis would be easy to investigate if we had more data. While the authors may have more data, we don’t. But we can use computer simulated random numbers to imagine this situation.

In the next run chart I added simulated random numbers mimicking another year of baseline data with an average length of stay of 15 days and a standard deviation of 2 (which is the actual SD in the provided data).

Now, this is improvement. I urge the authors to do this analysis with true data.

From the litter box

For a bad graph lover like me, International Forum is always a joy to visit. Here are a few more examples.

Fractional counts

counts

Why speak clearly if you don’t have to

counts

Save the pies for desert

pies

Improving improvement graphs

What would happen if BMJ and IHI started an improvement programme to improve graphical display of improvement data? In my experience, things usually happen very fast when these big boys decide to move.

I have a suggestion for a small test of change: At the International Forum 2016 in Göteborg, offer a full or half day seminar on Graphical display of quantitative information.

More on graphical display of quantitative information