System Review: A Method for Investigating Medical Errors in Health Care Settings (Continued)

Additional Statement, Submitted by Gregory L. Alexander M.H.A., B.S.N., R.N., and Tamara T. Stone Ph.D., M.B.A.

Statistical Analysis

Several statistical measures were used to compare data from system occurrences during July-December 1998 (n=38) prior to implementation of the system review team to data after implementation of the system review team January-June 1999 (n=61). Descriptive statistics were used to compare system occurrences by type, severity rating by type of system occurrence, frequency of severity ratings, means of severity rating scale, system occurrence rate with severity >3 per unit, and total recommendations made by the system review committee.

A two sample paired comparison t-test was utilized to determine if their were significant differences in severity rating means for sample populations rated by the system review team. A two sample paired comparison t-test was also used to determine if there were significant differences in the severity rating means between occurrences rated by system review team and those rated by PCC nurses. This statistic would provide an estimation of the inter-rater reliability of severity scale mean scores for the population of system occurrences. To obtain this final statistic a sample (n=55) of system occurrences were rated by PCC nurses using the same severity scale used by the system review team. PCC nurses were provided an equal number of system occurrences to rate from data collected before and after system review team implementation.

Descriptive Statistics

Figure 1 indicates System Occurrence by Type. During the monitoring period in 1998, 15/38 or 40% of the system occurrences occurred in the work up category followed by medication events that occurred in 8/38 or 21% of the occurrences. In the work up category of the 15 events reported 4 of these involved physical therapy evaluations that were delayed. Of the 4 two of the physical therapy evaluations had been ordered for >10 days. During the monitoring period for 1999, 19/61 or 31% of the occurrences were medication events followed by treatment occurrences that were 14/61 or 23% of total occurrences. Of the 19 medication errors 4 were identified which involved a delay in antibiotic administration with the greatest delay occurring over a period of 3 days and 1 involved hanging and administering blood with an incompatible solution of IV fluid.

Figure 2 and Figure 3 identify the percentage of system occurrences by severity rating. During the second period of data collection (Jan-July 1999), there was an increase in the system occurrences reported of nearly 38%. However, the combined severity ratings or those determined to have marginal deviation from standard of care or greater decreased from 79% in the first 6 months to 64% in the second six months. While the combined severity ratings <2 or those determined to have acceptable patient care management or better increased from 21% in the first 6 months to 36% in the second 6 months.

Figure 4 indicates the frequency distribution for the severity scale during the two periods of data collection. During the first 6 months of data collection the frequency of severe occurrences at level 4 was 17.86% higher than in the latter 6-month period. The frequency of severe occurrences in the last 6 months that were >3 were 15.01% lower than the first 6 months.

Figure 5 shows the means of the severity scale populations. The mean for the population sample collected in the first 6 months is in the level 3 marginal deviation of standard of care category. The mean for the sample collected in the last 6 months is in the level 2 acceptable patient care management category.

Figures 6 and 7 indicate the total system occurrences per department and frequency of system occurrences with a severity score >3 by department. Select to access Appendix B (PDF file, 11 KB; Text Version) and Appendix C (PDF file, 8 KB; Text Version) for a breakdown of the frequencies by department and description of system occurrences for data periods 1998 and 1999 respectively.

During 1998, 6 months of data indicate physical therapy had a total of five system occurrences, each rated >3. As indicated previously, these deviations were related to delayed evaluation work up. The skilled nursing unit had the most system occurrences at 7 with a total of 5 of these being rated in the marginal deviation category or higher. Two significant deviations occurred involving patient falls. Three marginal deviations occurred including omission of IV fluids for >24 hours, IV heparin infusing without orders following transfer, and antibiotics being given prior to obtaining a UA culture. General surgery was second in the category of total system occurrences with a total of 6. General surgery had a total of 5 in the marginal category or greater during the first 6 months. Two of these involved significant deviations of level 4.

During 1999, 6 months of data indicates that general surgery, medical/oncology, and the skilled nursing unit had the most system occurrences with 18, 13, and 7 noted on each unit respectively. Of the 18 on general surgery 10 occurrences were identified as having marginal deviation standard of care or higher. Of these 10 occurrences 3 were determined to be significant deviations, 2 involved a dosage of medication given at the wrong time and 1 involved a patient not being seen by a physician within 24 hours admission which is part of health system policy and Joint Commission regulations.

Figure 8 indicates the total number of actions recommended by the system review committee following review of each of the system occurrences. A total of 34 letters were generated and distributed to nurse managers or department heads regarding specific system occurrences in their area of responsibility. No action was taken on 19 system occurrences. On 15 different occasions policy and procedure was reviewed. Policies and/or procedures that were revised or reviewed as a direct result of committee recommendations include: 1) more specific criteria created to complete 24 hour chart checks performed by staff to catch missed orders, 2) labeling of blood bags with a bright colored sticker that identifies compatible solutions with which to administer blood, and 3) review of a standing orders to determine if a specific medication should be included in a specific treatment regimen.

Paired Comparison T-Test

Statistical tests were used to determine if there were variances in the mean between the two sample populations from July-Dec 1998 and Jan-July 1999. Levene's test for Equality of Variances was performed to determine if the variances of the population samples are equal. The result indicates that equal variances are assumed with an F-statistic equal to 1.73. The t-test for equality of means was performed with a t value equal to 2.072, degrees of freedom 97, and a probability of .041. Confidence intervals at the 95% level were calculated to range from a low of .019 to a high of .874

Another two sample equal variance t-test was performed to test inter-rater reliability between system occurrence severity ratings performed by the system review team and severity ratings performed by PCC nurses who collected the data. The result was a t-statistic of .526.

The open coding system appeared to be beneficial in identifying categories in which to measure, benchmark, and trend system occurrences reported by PCC nurses. From the categorical information we were able to trend specific units, procedures, and processes which were causing variations in the quality of care provided. For example the delayed physical therapy evaluations created longer periods of immobility for patients. Immobility can lead to increased pulmonary problems, skin breakdown, and depression for patients who are unable to ambulate without assistance. These types of complication lead to longer lengths of stay, increased costs to the system, and poor utilization of resources. By measuring, benchmarking, and trending this data the system review team was able to recommend changes in processes, procedures, etc. to improve coordination and sequencing of care.

The increased reporting of system occurrences and reduction in severity levels between the two sets of data is clinically significant. The data indicates that even though there were more system occurrences reported during the second period of data collection the level of severity of systems occurrences did not increase. In fact, the means of the two sample populations shifted from a level 3 marginal deviation of care in the first 6 months to a level 2 acceptable level of care in the second 6 months of data collection. This could be attributable to some process and procedure changes that occurred during the second six months as a result of benchmarking, trending, and recommendations proposed. It could also be attributable to the increasing awareness of system problems discussed in numerous communications sent to patient care areas.

In the statistical analysis completed Levene's Test for Equality of Variances indicated an F-statistic of 1.733. This is significant because if all groups have the same mean in the population the F-statistic approaches 1. Since the F-statistic in the research was found to be close to 1 this implies that equal variances can be assumed within the population samples. The t-statistic was then calculated and found to have a significance level of .041. This also implies that there is sufficient evidence to suspect the variances are equal between the sample populations. This implies that the population means of for the two groups are equal and the observed data can be considered to be two samples from the same population.

The t-test performed on the two sample populations made up of severity scale ratings from the system review team and severity scale ratings from the PCC nurses showed a t-statistic value of 0.527. Once again this large value implies that the variances are equal and that the data observed can be considered to come from the same population samples. This is important because it is a measure of inter-rater reliability between the two sample populations. Because the variances in the sample populations can be assumed to be equal, there is sufficient evidence to indicate inter-rater reliability is strong between the two groups.

Solutions for decreasing system error and variation in process occur as a result of increased communication and identification of breakthrough opportunities. Establishing an interdisciplinary team to review system events and refine system processes is one method to curtail system occurrences. Being a part of the team will promote ownership and breakdown barriers that interrupt the continuum of care. Increased research in system variation is necessary to assist in the development of patient safety standards and reporting mechanisms for medical errors.¹³ This research can be used by health care facilities to establish safer health care environments.

References:

1. Adair, Charlene B., & Murray, Bruce A. (1994). Breakthrough in Process Redesign. New York, NY: American Management Association.

2. Reed, Laura MSN, RN, Blegen, Mary A. PhD, RN, Goode, Colleen S. PhD, RN, FAAN. (1998). Adverse Patient Occurrences as a Measure of Nursing Care Quality. Journal of Nursing Administration. May, Volume 28(5), 62-69.

3. Lepley, Cyndi J. PhD, RN. (1998). Problem-Solving Tools for Analyzing System Problems: The Affinity Map and the Relationship Diagram. Journal of Nursing Administration, December, Volume 28(12), 44-50.

4. Lawler, Edward E. (1986). High Involvement Management. San Francisco, CA: Jossey- Bass Inc., Publishers.

5. Juran, Joseph M. (1995). Managerial Breakthrough. (2nd edition.). New York, NY: McGraw-Hill, Inc.

6. Berwick, Donald M. MD, MPP. (1991). Controlling Variation in Health Care: A consultation from Walter Shewhart. Medical Care, December, 29(12), 1212-1225.

7. Localio, Russell A., JD, MPH, MS, Weaver, Susan L. MS, Landis, Richard J. PhD, Lawthers, Ann G. ScD, Brennan, Troyen, A. MD, JD, Hebert, Liesi ScD, Sharp, Tonya J. MS. (1996). Identifying Adverse Events Caused by Medical Care: Degree of Physician Agreement in Retrospective Chart Review. Annals of Internal Medicine, September 15, 125(6), 457-464.

8. Wu, Albert W. MD, MPH, Folkman, Susan PhD, McPhee, Stephen J. MD, Lo, Bernard MD. (1991). Do House Officers Learn from their Mistakes? JAMA, April 24, 265(16), 2089- 2094.

9. Audet, Anne-Marie MD MSc, SM, Scott, Denman, MD, MPH, (1994). The Oversight of Medical Care: A Proposal for Reform. Annals of Internal Medicine, March, 120(5), 423- 431.

10. Burns, Nancy & Grove, Susan K. (1995). Understanding Nursing Research. Philadelphia, PA: W.B. Saunders/Harcourt Brace & Co.

11. Stern, P.M., (1980). Grounded Theory Methodology: its uses and processes. Image, 12(1), 20-23.

12. Brooks, Juliana MD, Renz, Kathleen, Richardson, Steven MD, MS, White, Susan PhD, Hosey, Janet RN. (1992). Systems Versus Performance Problems: A Peer Review Organization's Perspective. Quality Review Bulletin, May, 172-176.

13. Simpson, Roy L. (2000). Medical Errors, Airplanes, and Information Technology. Nursing Management, June, 31(6), 14-15.

Author Biographies:

Greg Alexander RN, MHA is the Director of Education for Freeman Health System in Joplin, Missouri. Mr. Alexander is a graduate of the University of Missouri Columbia school of Health Management and Informatics. Mr. Alexander has over 20 years experience in health care and is interested in improving processes that impact patient care.

Tamara T. Stone, PhD, MBA, is an Assistant Professor in the Department of Health Management and Informatics and a Research Associate in the Center for Family Medicine Science at the University of Missouri-Columbia, School of Medicine. Dr. Stone's research focuses on clinical process design and quality management in health care settings.

Return to Top

Current as of September 2000

Internet Citation:

System Review: A Method for Investigating Medical Errors in Health Care Settings. Additional Statement. National Summit on Medical Errors and Patient Safety Research. September 2000. http://www.quic.gov/summit/aalexander1.htm

Return to Additional Statement Directory
National Summit on Medical Errors and Patient Safety Research
QuIC Home Page
Department of Health and Human Services