JULY 2007 Data Brief COMMISSION ON A HIGH PERFORMANCE HEALTH SYSTEM Measuring Hospital Performance: The Importance of Process Measures Anthony Shih and Stephen C. Schoenbaum The Commonwealth Fund For more information about this ABSTRACT: Providers and patients agree that high-quality care is an essential study, please contact: component of a high performance health system, a position supported by Anthony Shih, M.D., M.P. H. The Commonwealth Fund’s Commission on a High Performance Health Assistant Vice President, Quality System. However, recent debate has focused on how best to define “high qual- Improvement and Efficiency ity” in the hospital setting. Recent journal articles have found only a “modest The Commonwealth Fund Tel 212.606.3856 relationship” between performance on Hospital Quality Alliance (HQA) Fax 212.606.3500 process measures and short-term mortality. Despite the lack of a more robust E-mail ts@cmwf.org connection, the authors of this data brief contend that these HQA measures represent actionable items that can have an impact on quality and health out- comes and identify specific activities hospitals can work on to improve per- formance. Developing new process measures to reduce mortality and other poor outcomes, while improving performance on the current measures, can help move the nation toward achieving higher-quality health care and a high performance health system. * * * * * Introduction While most providers and patients agree that high-quality care is an essential component of a high performing health system, there has been much recent debate regarding how to best define “high quality” in the hospital setting. This and other Commonwealth Much of the discussion focuses on whether the process measures of the Fund publications are online at www.commonwealthfund.org. To Hospital Quality Alliance (HQA)—a national public–private collaboration learn more about new publications designed to encourage hospitals to voluntarily collect and report hospital when they become available, visit quality performance information—are valuable. This data brief reviews the the Fund’s Web site and register to receive e-mail alerts. findings of recent studies on the HQA measures, and finds that while addi- Commonwealth Fund pub. 1046 tional quality measures are necessary, providers should not hesitate to take Vol. 6 action to improve performance on the current measures. The Commonwealth 2 The Commonwealth Fund Commission on a High Performance Health System Fund’s Commission on a High Performance Health Table 1. Current Hospital Quality System—launched in 2005 to develop strategies Alliance Process Measures to promote a health system that provides all Heart attack Americans with affordable access to high-quality, G Aspirin at arrival safe care while maximizing efficiency—has identi- G Aspirin at discharge fied high-quality care as one of the core goals of G ACE inhibitor or ARB for LVS dysfunction a high performing health system.1 G Beta blocker at arrival Since 2005, the Centers for Medicare and Medicaid Services (CMS) has been publicly G Beta blocker at discharge reporting hospital-specific HQA performance G Fibrinolytic medication within 30 minutes of arrival measures on processes of care. These process meas- ures assess whether certain care processes recom- G PCI received within 90 minutes of hospital arrival mended in clinical guidelines are administered, such the administration of an aspirin within 24 G Smoking cessation advice/counseling hours of a heart attack (Table 1). As demonstrated Heart failure in the Commission’s national and state scorecards, G Evaluation of LVS function there is wide variation on these measures across G ACE inhibitor or ARB for LVS dysfunction the country (Figure 1).2,3 G Discharge instrucctions This variation in performance indicates an opportunity for providers to improve, and also G Smoking cessation advice/counseling allows consumers to identify higher- and lower- Pneumonia performing hospitals. However, the benefit is G Oxygenation assessment dependent on the validity of the measures—that is, G Initial antibiotic timing whether they truly reflect quality of care. In con- G Pneumococcal vaccination trast to mortality rates, which directly measure a G Influenza vaccination G Blood culture performed in the emergency department prior to initial antibiotic received in hospital G Appropriate initial antibiotic selection G Smoking cessation advice/counseling Surgical care improvement/ surgical infection prevention G Prophylactic antibiotic received within one hour prior to surgical incision G Prophylactic antibiotics discontinued within 24 hours after surgery end time G Prophylactic antibiotic selection Notes: Starter set of 10 measures are italicized. ACE = angiotensin-converting enzyme; ARB = angiotensin- receptor blocker; LVS = left ventricular systolic; PCI = percutaneous coronary intervention. Source: Centers for Medicare & Medicaid Services Hospital Compare Web site: http://www.hospitalcompare.hhs.gov. Measuring Hospital Performance: The Importance of Process Measures 3 vital patient outcome, it is more difficult to tell measures in the field, as they become widely whether a process measure is an appropriate meas- adopted for public reporting programs such as ure of quality of care. Hospital Compare, as well as for pay-for-perform- In June 2007, CMS and HQA began pub- ance programs. Over the past year, there have been licly reporting hospital performance on 30-day two journal articles that examined the relationship risk-adjusted mortality for patients admitted for between the HQA measures and mortality. heart attacks and heart failure, publishing this The most recent article, written by Ashish information on the Medicare Hospital Compare Jha and colleagues and supported by The Web site (www.hospitalcompare.hhs.gov). The Commonwealth Fund, was published in Health appeal of mortality rate data is clear: it is easy to Affairs in July 2007.4 The article compared per- understand that a patient is better off at a hospital formance on HQA process measures for heart with a low mortality rate than a high one. attack, heart failure, and pneumonia with risk- However, when CMS separated hospitals into per- adjusted mortality for those areas. The authors cre- formance categories of: “better than U.S. national ated an aggregate performance score for each rate” (i.e., better than the national average), “worse condition for each hospital. Examining the full than U.S. national rate,” and “no different than range of performance by quartiles, their analysis U.S. national rate,” more than 98 percent of the revealed a consistent relationship of higher per- hospitals fell into the “no different” category formance on the process measures with lower risk- (Figure 2). adjusted mortality (Table 2). In the Journal of the American Medical Association, Rachel Werner and Eric Bradlow per- formed a similar but more limited analysis, also using the HQA database, with findings consistent with Jha and colleagues.5 Across all three condi- tions, the authors found that hospitals performing in the top quartile on each composite score had lower risk-adjusted mortality rates than those per- forming in the lowest quartile. Two other recent studies examined the rela- tionship between process measures and patient outcomes, using measures closely related to the HQA measures. Elizabeth Bradley and colleagues examined the relationship between heart attack process measures from the National Registry of Myocardial Infarction and mortality, and found that the measures explained a small proportion of the variation in mortality.6 Gregg Fonarow and colleagues examined the relationship between the Examining the Hospital Quality Alliance American College of Cardiology/American Heart Process Measures Association heart failure performance measures Although the evidence supporting the individual and mortality, and found no relationship with in- care processes captured in the HQA quality meas- hospital mortality risk for each individual measure. ures is strong, there is a need to evaluate these Only the use of angiotensin-converting enzyme 4 The Commonwealth Fund Commission on a High Performance Health System Table 2. Adjusted Mortality Rates, Stratified by Hospital Performance on Hospital Quality Alliance (HQA) Summary Scores Predicted mortality rate (95% confidence interval) HQA performance AMI CHF Pneumonia First quartile 10.0% 4.6% 7.1% (9.7, 10.4) (4.4, 4.8) (6.9, 7.4) Second quartile 10.2% 4.9% 7.4% (10.0, 10.5) (4.8, 5.1) (7.2, 7.6) Third quartile 10.6% 5.0% 7.5% (10.3, 10.9) (4.8, 5.2) (7.2, 7.7) Fourth quartile 10.8% 5.0% 7.9% (10.5, 11.2) (4.8, 5.1) (7.6, 8.1) p value for trend <0.001 0.005 <0.001 Notes: Adjusted for patient age, sex, race, and the presence or absence of each of 30 comorbidities. AMI = acute myocardial infarction; CHF = congestive heart failure. Source: A. K. Jha, E. J. Orav, Z. Li et al., “The Inverse Relationship Between Mortality Rates and Performance in the Hospital Quality Alliance Measures,” Health Affairs, July/Aug. 2007 26(4):1104–10. inhibitor/angiotensin-receptor blocker for left programs.8 We believe this line of argument is mis- ventricular systolic dysfunction and beta-blocker guided. Instead, we offer the following response: use at discharge were associated with mortality or rehospitalization at 60 to 90 days.7 Fonorow found G Although the relationship between the no relationship between other measures, such as HQA measures and mortality is modest and discharge instructions or smoking cessation coun- there are clearly other factors that predict seling (both of which are part of the current set of mortality, the HQA measures represent HQA measures), with the outcomes they studied. actionable items that can have an impact. As per Jha et al., if hospitals in the lowest quar- Commentary tile of performance had the mortality rates Across these studies, it appears that performance of the top quartile, approximately 2,200 on the initial set of HQA process measures has a deaths could have been avoided. Moving all modest relationship to short-term mortality, and hospitals to the top decile of performance that some individual measures included in the cur- would improve this rate substantially. Even rent HQA data set are not related to short-term though the measures included in the HQA mortality at all. Some of the more vocal responses do not represent all the process steps that to these findings are troubling, particularly the crit- might reduce mortality, they are an impor- icism that since the HQA measures do not have a tant foundation for reductions in mortality. large effect on mortality, they also have limited Although we should continue to search for usefulness for informing consumers about quality of additional performance measures, we believe care, helping providers improve outcomes, or guid- that these measures should continue to be ing payers seeking value in pay-for-performance used by providers, consumers, and payers. Measuring Hospital Performance: The Importance of Process Measures 5 G Demonstrating that a process measure has a as the Institute for Healthcare Improvement or relationship to reduction in short-term mor- through government-sponsored programs such as tality is desirable, but it not the only criteria Medicare’s Quality Improvement Organizations, by which we should judge whether it is use- which operate in each state. ful. For instance, Fonarow et al. found no The real “bottom line” means not taking a association between mortality and indicators simplistic approach to measurement and improve- that measure whether heart failure patients ment. We must continue to generate evidence to received smoking cessation counseling. evaluate new process measures that may help However, we would argue that these reduce mortality and other poor outcomes; work processes should nevertheless be performed. hard to improve performance in the current suite Even minimal counseling (less than three of HQA measures; and periodically check to minutes) by a physician has been shown to assess the impact of improved implementation of be effective in reducing smoking,9 and existing and new measures on outcomes. This smoking cessation is related to risk reduction approach will put us well along the path to achieving for heart disease, lung cancer, and stroke.10 higher-quality health care and a high performance Decreased mortality is not the only desired health system. outcome of good medical care. Other important outcomes for patients and payers are decreased morbidity, including fewer in-hospital and post-hospital complications; fewer rehospitalizations; improved under- standing of conditions, leading to better NOTES longer-term care and outcomes; and decreased costs. 1 The Commonwealth Fund Commission on a High Performance Health System, Framework for a High Conclusion Performance Health System for the United States (New Both outcomes measures, such as mortality rates, York: The Commonwealth Fund, Aug. 2006). 2 and process measures are useful for improving The Commonwealth Fund Commission on a High quality of care.11 Mortality rates may seem to rep- Performance Health System, Why Not the Best? resent “the bottom line,” but there are numerous Results from a National Scorecard on U.S. Health System Performance (New York: The Commonwealth Fund, factors, many beyond providers’ control, that con- Sept. 2006). tribute to mortality. Risk-adjustment methods to 3 account for these factors are improving, but as the J. C. Cantor, C. Schoen, D. Belloff, S. K. H. How, and D. McCarthy, Aiming Higher: Results from a State recent public reporting effort for heart attack and Scorecard on Health System Performance (New York: heart failure mortality demonstrates, it is difficult The Commonwealth Fund, June 2007). to discriminate among providers using only mor- 4 A. K. Jha, E. J. Orav, Z. Li et al., “The Inverse tality rates. In addition, mortality rates alone do Relationship Between Mortality Rates and Performance not point to the specific actions providers must in the Hospital Quality Alliance Measures,” Health undertake to improve care. Process measures Affairs, July/Aug. 2007 26(4):1104–10. derived from clinical guidelines, such as those in 5 R. M. Werner and E. T. Bradlow, “Relationship the HQA, identify specific activities hospitals can Between Medicare’s Hospital Compare Performance work to improve. Guidance for improving these Measures and Mortality Rates,” Journal of the American processes is available through private entities such Medical Association, Dec. 13, 2006 296(22):2694–2702. 6 The Commonwealth Fund Commission on a High Performance Health System 6 E. H. Bradley, J. Herrin, B. Elbel et al, “Hospital Quality for Acute Myocardial Infarction: Correlation Among Process Measures and Relationship with Short-Term Mortality,” Journal of the American Medical Association, July 5, 2006 296(1):72–78. 7 G. C. Fonarow, W. T. Abraham, N. M. Albert et al., “Association Between Performance Measures and Clinical Outcomes for Patient Hospitalized with Heart Failure,” Journal of the American Medical Association, Jan. 3, 2007 297(1):61–70. 8 S. D. Horn, “Commentary: Performance Measures and Clinical Outcomes,” Journal of the American Medical Association, Dec. 13, 2006 296(22):2731–32. 9 M. C. Fiore, D. W. Wetter, W. C. Bailey et al., Smoking Cessation Clinical Practice Guideline (Rockville, Md.: Agency for Health Care Policy and Research, U.S. Dept. of Health and Human Services, 1996). 10 The Health Benefits of Smoking Cessation: A Report of the Surgeon General. DHHS Publication No. (CDC) 90-8416 (Rockville, Md.: Centers for Disease Control, U.S. Department of Health and Human Services, 1990). 11 S. C. Schoenbaum, “Outcomes Measurement: The End or the Beginning?” in S. C. Schoenbaum, ed., Measuring Clinical Care: A Guide for Physician Executives (Tampa, Fla.: American College of Physician Executives, 1995), pp. 169–74. Measuring Hospital Performance: The Importance of Process Measures 7 ABOUT THE AUTHORS Anthony Shih, M.D., M.P. H., is assistant vice president at The Commonwealth Fund, overseeing the Program on Quality Improvement and Efficiency. Dr. Shih came from IPRO, an independent not-for- profit health care quality improvement organization (QIO), where he held a variety of positions since 2001, most recently as vice president, quality improvement and medical director, managed care. In this position he developed and managed large-scale quality improvement projects for the Medicare popula- tion and designed quality measures and quality improvement studies for Medicaid managed care mar- kets. Previously, Dr. Shih was the assistant medical director for a community-based mental health clinic in Northern California serving immigrant and refugee populations. He is board-certified in public health and preventive medicine, and has expertise in epidemiology, health services research, and in the principles and practice of health care quality improvement. Dr. Shih holds a B.A. in economics from Amherst College, an M.D. from New York University School of Medicine, and an M.P.H. from Columbia University Mailman School of Public Health. Stephen C. Schoenbaum, M.D., M.P. H., is executive vice president for programs of The Commonwealth Fund and executive director of The Commonwealth Fund Commission on a High Performance Health System with responsibility for coordinating the development and management of the Fund’s program areas. He is also a member of the Fund’s executive management team. Prior to joining the Fund in February 2000, he was president of Harvard Pilgrim Health Care of New England and senior vice president of Harvard Pilgrim Health Care, responsible for delivery system operations in a mixed staff and network model HMO with approximately 150,000 members. Prior to joining Harvard Community Health Plan in 1981, Dr. Schoenbaum was a member of the Department of Medicine at Brigham and Women’s Hospital and did epidemiologic research in obstetrics and infectious diseases. He is a lecturer in the Department of Ambulatory Care and Prevention, Harvard Medical School, the author of more than 125 scientific articles and papers, and the editor of a book on measuring clinical care. Dr. Schoenbaum received an A.B. from Swarthmore College with honors, an M.D. from Harvard Medical School (cum laude), and an M.P.H. from Harvard School of Public Health. He also completed the Program for Management Development at Harvard Business School. The mission of The Commonwealth Fund is to promote a high performing health care system. The Fund carries out this mandate by supporting independent research on health care issues and making grants to improve health care practice and policy. The views presented here are those of the authors and not necessarily those of The Commonwealth Fund or its directors, officers, or staff, or of The Commission on a High Performance Health System or its members.