Improvement: Testing and Implementing Change
Run Chart
A run chart helps a QI team track such improvements over time. It is simply a graph where quality is on the vertical axis and time is on the horizontal axis. Data points representing a measurement of quality at a particular point in time are plotted and connected with a line.
An annotated run chart has comments with arrows pointing to the moments in time when different ideas for improvement were tested. This helps explain any sudden changes in quality that may have occurred.
Run charts should be set up at the start of a QI project and be constantly updated with new data as the project unfolds.
Often, in response to a quality problem, teams will decide to implement a solution and measure quality at only two points in time, before and after. They then show the data on a bar chart with two columns. We strongly encourage teams using run charts instead, and to use PDSA cycles to test different ways of implementing solutions rather than planning on doing an implementation of only one way of improving care.
Directions
Q: My measure is an adverse event that occurs infrequently. If I plot the adverse event rate by month then I get a lot of zeros and ones. It’s difficult to see changes in quality over time. What should I do?
A: Use the same time scale but instead of putting adverse event rate or count on the y-axis, plot the time between adverse events, or number of error-free cases performed (or number of patients seen) in between adverse events.
Rules for Identifying Statistically Significant Changes in Quality on a Run Chart
Visit the Institute for Healthcare Improvement (IHI)’s “Improvement Tracker”, where organizations from around the world report run charts from a variety of different QI projects: http://www.ihi.org/ihi/workspace/tracker/
Statistical Explanation of Run Chart Rules
Imagine flipping a coin 5 times in a row. The probability that you will get 5 consecutive heads is 1 / 25 = 3.1%. We can be certain to a p-value of 0.03 that this did not occur by random chance.
Note that different quality improvement experts recommend slightly different thresholds for number of consecutive points you need in the same direction to declare a significant change. This is because of different opinions about how certain you need to be about the change. If you want to have a 99% certainty (i.e. p-value of <0.01, one-tailed test) that this is a true change and didn’t occur just by random chance, then insist on 7 points in a row (see table below).
- Change Concepts and Ideas
- Change Concepts: Improving Effectiveness of Chronic Disease Prevention and Management
- Change Concepts: Improving Flow
- Change Concepts: Mistake Proofing (Poka Yoke)
- PDSA Cycles
- PDSA Ramps
- Run Charts
A run chart helps a QI team track such improvements over time. It is simply a graph where quality is on the vertical axis and time is on the horizontal axis. Data points representing a measurement of quality at a particular point in time are plotted and connected with a line.
An annotated run chart has comments with arrows pointing to the moments in time when different ideas for improvement were tested. This helps explain any sudden changes in quality that may have occurred.
Run charts should be set up at the start of a QI project and be constantly updated with new data as the project unfolds.
Often, in response to a quality problem, teams will decide to implement a solution and measure quality at only two points in time, before and after. They then show the data on a bar chart with two columns. We strongly encourage teams using run charts instead, and to use PDSA cycles to test different ways of implementing solutions rather than planning on doing an implementation of only one way of improving care.
Directions
- Identify how frequently you want to be collecting and showing data (e.g. daily, weekly, monthly). This will depend on the project, and how quickly you think you can get results from the changes you test out.
- Collect some baseline data about quality, if possible. As a rule of thumb, try to have about measurements of quality at ten points in time prior the start of testing any improvements. For example, if you plan on tracking quality every month, then look at data going back the preceding ten months. This step helps you identify when you’ve made a significant change in improvement later. However, if it is impossible to get previous data, then don’t delay the start of the QI project just to get baseline data.
- Arrange your data in a table in chronological order. For example:
Month % of Diabetes Patients in Dr. Jones’ Family Practice with Good Blood Sugar Control (HbA1C <0.07) Jan 49% Feb 48% Mar 50% Apr 50% May 49% Jun 53% Jul 55% Aug 55% Sep 56% Oct 59% Nov 62% Dec 63% - Create a graph with this data, where the measure of quality is on the vertical axis and time is on the horizontal axis, and each data point is connected by a line. (If you are using Excel, highlight the data in the above table, then use Insert -> Chart -> Standard Types tab -> Line.)
- Calculate the median (i.e. the middle value) of your data points. (In excel, use the =MEDIAN ( , , ) function to calculate this, where the values in the parentheses are your baseline data.) Plot this as a horizontal line across the graph, labelled baseline.
Suggested embellishments: - If you have a target for improvement (strongly recommended), plot it as a horizontal line across the graph, labelled target. (To add a line, use Insert -> Picture -> Autoshapes and select the icon for lines. Use the Text Box feature to insert a label describing the target.)
- Annotate the chart with comments and arrows pointing to key points in time along the line of the run chart when different improvements were tried. (A simple way to do this in Excel: go to Insert -> Picture -> Autoshapes and select the cartoon-style balloons called ‘callouts’. Or, use lines and text boxes as described above.)
- To make your run chart stylish, it should be length to width ratio should be about 3:2 (or, the same proportion as an 8.5” x 11” landscape sheet of paper). The top of the vertical axis should be at least a bit higher then the highest value in the chart or your target (whichever is higher).
Q: My measure is an adverse event that occurs infrequently. If I plot the adverse event rate by month then I get a lot of zeros and ones. It’s difficult to see changes in quality over time. What should I do?
A: Use the same time scale but instead of putting adverse event rate or count on the y-axis, plot the time between adverse events, or number of error-free cases performed (or number of patients seen) in between adverse events.
Rules for Identifying Statistically Significant Changes in Quality on a Run Chart
- “5 in a row”: If you have five consecutive points all going up (or down) in a row, then you have a statistically significant change in quality. (If you have two points consecutive points that are equal to each other, ignore one of these points when counting how many points you have in a row.)
- “6 a side”: If you have six consecutive points above the median line, then you have a statistically significant change in quality. (If a point sits on the median, ignore that point when counting how many points you have in a row.)
Visit the Institute for Healthcare Improvement (IHI)’s “Improvement Tracker”, where organizations from around the world report run charts from a variety of different QI projects: http://www.ihi.org/ihi/workspace/tracker/
Statistical Explanation of Run Chart Rules
Imagine flipping a coin 5 times in a row. The probability that you will get 5 consecutive heads is 1 / 25 = 3.1%. We can be certain to a p-value of 0.03 that this did not occur by random chance.
Note that different quality improvement experts recommend slightly different thresholds for number of consecutive points you need in the same direction to declare a significant change. This is because of different opinions about how certain you need to be about the change. If you want to have a 99% certainty (i.e. p-value of <0.01, one-tailed test) that this is a true change and didn’t occur just by random chance, then insist on 7 points in a row (see table below).
| Number consecutive in a row | Probability of this occurring by random chance |
| 4 | 6.2% |
| 5 | 3.1% |
| 6 | 1.6% |
| 7 | 0.8% |
| 8 | 0.4% |
| 9 | 0.2% |
About HQO | News | Public Reporting | Information for the Public | Quality Improvement | Contact Us | Sitemap
© Queen's Printer for Ontario, 2011 | Privacy | Important Notices | Last modified: 2007-12-11