C A L I FOR N I A H EALTH C ARE F OU NDATION Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures Prepared for California HealthCare Foundation by Bruce Spurlock, M.D. Rebecca Abravanel, Ph.D. Sarah Spurlock Convergence Health Consulting, Inc. December 2008 About the Authors Convergence Health Consulting, Inc. is a multi-disciplinary team of professionals supporting health care organizations to advance patient safety, improve quality, strengthen leadership, promote sustainable change, and facilitate a solution-oriented dialogue among physicians, hospitals, health plans, purchasers, the public and other health care stakeholders. Bruce Spurlock, M.D., President and Chief Executive Officer of Convergence Health, is a national expert in quality and patient safety and serves as the Executive Director for the California Hospital Assessment and Reporting Taskforce (CHART) and as the Executive Director, Clinical Acceleration, for the Bay Area Patient Safety Collaborative (Beacon). Rebecca Abravanel, Ph.D., is a statistician, demographer, and sociologist with research clients in several industries, including health care. Sarah Spurlock is a research assistant and student of political science and history at the University of California, Davis. About the Foundation The California HealthCare Foundation is an independent philanthropy committed to improving the way health care is delivered and financed in California. By promoting innovations in care and broader access to information, our goal is to ensure that all Californians can get the care they need, when they need it, at a price they can afford. For more information, visit www.chcf.org. ©2008 California HealthCare Foundation Contents 2 Introduction 3 Methodology 3 Key Findings 6 Study Limitations 7 Implications 9 Endnotes 1 0 Appendices A: Data Sources B: Measures C: Hospital Performance by Condition D: Hospital Improvement Over Time F: Statistical Results from Regression Models Introduction various measures. The project’s goal was to ascertain With the advent of the “transparency movement” quantitatively whether standardized, publicly in health care, a limited but nonetheless substantial disclosed hospital characteristics were independently amount of standardized information has become associated with performance. available to compare the clinical performance of physicians, hospitals, health plans, and skilled Few studies have quantitatively examined this nursing facilities. National organizations such as question in depth: The relationship between the Hospital Quality Alliance (HQA), the AQA, measured performance and hospital size or teaching the National Quality Forum (NQF), The Joint status is inconsistent.1 Other structural characteristics Commission (TJC), and the National Committee associated with Leapfrog Group standards have a for Quality Assurance (NCQA) promulgate modest but significant relationship with clinical detailed measures, data collection rules, and performance.2 Ownership (tax-exempt, investor- recommendations for reporting results gleaned from owned) and rural/urban characteristics also have clinical performance information. mixed results.3 A number of studies have examined common quality process measures supported by As information on hospital performance the Centers for Medicare & Medicaid Services accumulates, issues emerge that go beyond the basics (CMS) and The Joint Commission (TJC), but these of which hospitals perform better or worse for a found only a tenuous connection between hospital given clinical condition. If it were understood which performance on process measures and clinical strategies or characteristics lead to higher clinical outcomes.4 performance in some hospitals, other hospitals might adopt them in an effort to provide better care to This project took a different approach. It used their patients. CHART data to assess the impact of various hospital characteristics, taking advantage of new measures The purpose of this project was to understand of performance — especially clinical outcomes — to which, if any, hospital characteristics were associated test hypotheses and identify patterns. Likewise, the with hospital performance as measured by CHART. project used operational, financial, and utilization Since CHART has data on several aspects of information regarding California hospitals, available clinical care across a range of hospital services, through the Office of Statewide Health Planning this project sought to identify any patterns that and Development (OSHPD), to provide additional could describe high and low performers among the variables in testing for patterns. Data from CHART In California, stakeholders from health plans, purchasers, hospitals, consumer advocates, physicians, and other groups have created a voluntary hospital performance measurement and reporting program called the California Hospital Assessment and Reporting Taskforce (CHART). Initially sponsored by the California HealthCare Foundation (CHCF), CHART produced its first public report on California hospital clinical performance in March 2007. This report captured data from several sources, much of which was publicly available but dispersed among a number of entities and Web sites. It also published previously unreported measures of care such as patient experience, mortality rates for patients in intensive care units, and rates of hospital-acquired pressure ulcers. As of June 2008, 220 hospitals representing 82 percent of the state’s daily census (CHART does not include children’s hospitals) participate in CHART. The CHART Web site (www.calhospitalcompare.org) is designed for easy consumer access and use but also provides information appropriate for hospitals undertaking improvement activities, health plans assessing their networks, and purchasers considering health benefits design. 2  |  C alifornia H ealth C are F oundation Methodology Once the overall structure of the data and the relationships between hospital attributes and clinical Data Sources and Structure outcomes were understood, the analysis continued CHART captures data from a number of sources by building regression models to determine (see Appendix A) and collates the information for whether any variables that previously had shown no each participating California hospital. This project relationship now emerged (having been suppressed used five data sets from CHART’s initial launch in in the individual analysis by another variable) or March 2007, plus four of CHART’s quarterly data variables that had demonstrated a relationship now updates (see Appendix B). CHART measures were disappeared (were not independently associated with the dependent variables in the analysis. the clinical outcome). Summaries of these regression models are presented in Appendix F. The project also used publicly available data from OSHPD: the Hospital Annual Financial and rolling Finally, once the impact of the variables on outcomes quarterly data sets. The OSHPD data — which was understood, the project tested whether the include structural, financial, and operational variables had any impact on improvement. This measures — were selected to best match the time part of the analysis examined the impact of hospital periods covered by the CHART data (see Appendix attributes on selected core measures and on two A, Table 2). Data also were gathered from other outcome measures: CABG mortality and patient sources, including the Accreditation Council for experience. Individual hospital performance for Graduate Medical Education (ACGME) and each CHART measure was analyzed to determine researchers who developed a report on California if performance was improving or declining. Because hospitalists.5 In all, 30 hospital attributes were the CHART data for each core measure and patient evaluated against CHART results. experience cover a rolling 12-month period, both the earliest and latest results were examined. For the Analysis specific time periods, see Appendix A. The project The project’s analysis initially examined the also examined the characteristics of the hospitals impact of various hospital attributes on clinical with the greatest improvement and the hospitals outcomes. Figures on coronary artery bypass graft whose performance declined, using logistic regression (CABG) mortality, intensive care unit (ICU) to predict the likelihood of membership in each of mortality, hospital-acquired pressure ulcer (HAPU) these groups. rates, and patient experience were examined to determine if a relationship existed for any of Key Findings the 30 hospital attributes. The project explored Of the 30 hospital characteristics the project whether the relationship met basic statistical criteria examined against 11 primary performance and and whether it was linear or non-linear (e.g., improvement measures, only four were suggestive —  U-shaped, asymptotic). Such a relationship, or lack using presence in at least three statistical models of relationship, did not determine the final effect as the threshold — as independent predictors of within a regression model, where other factors are performance: controlled to establish whether the relationship is K Disproportionate Share Hospital (DSH) status independent of other variables. But it is important to (meaning a significant proportion of indigent care understand patterns in the data to more accurately and Medi-Cal patients); build the models. For correlations of individual hospital characteristics with clinical outcomes from K Membership in a large system (more than ten this analysis, see Appendix E. hospitals); Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  3 K Percentage of gross revenue collected; and hospital setting are more widespread than previously thought. This finding requires more in-depth K Initial starting value (in the improvement analysis using other data sets that include socio- analyses). demographic measures. DSH Status Membership in a Large System DSH designation was a significant factor in five Membership in a health system of more than models: CABG mortality, ICU mortality, patient ten hospitals was significantly related to the experience, and improvement in both heart failure improvement measure for heart failure care and to and one surgical infection prevention measure (SIP two surgical improvement measures. In addition, 3). In all but one of these cases, being a DSH facility membership in a medium-to-large system (four produced worse results or had a negative impact or more hospitals) was positively associated with on improvement. The exception was observed improvement in the pneumonia quality measure. with respect to one surgical infection prevention However, system size was not in the final model for measure (SIP 3), where DSH status was significantly either performance or improvement in any of the positively associated with improvement. outcome measures. This suggests that large health systems may be better able to improve process DSH status is a marker for high levels of indigent measures over time, but that large system size has no and Medi-Cal-funded care, and reflects a patient effect on overall clinical outcomes. It is noteworthy population with socio-demographics that differ that the profitability of large health systems was not from populations in non-DSH facilities. The a factor regarding either improvement or clinical patients who make up the DSH population may outcomes. have a higher severity of illness, but presumably the risk-adjustment models control for a large portion The explanation for this modest relationship of such clinical severity. DSH status also might be a between large system size and a few improvement marker regarding gross revenue and profitability, but measures is not apparent from the data but could these factors disappeared when controlled for in all stem from several factors, including: 1) making of the models, as did DSH revenue as a percentage improved performance a priority (several health of gross revenue. In other words, DSH status as systems include core measure performance in a source of revenue or as an indicator of financial executive compensation formulas, but the data health does not factor into clinical performance. on this are not publicly available); 2) allocating Further, the proportion of minority discharges was corporate resources toward improvement at the not a factor in any of the 11 models, suggesting expense of other priorities; 3) sharing effective that race alone cannot account for the observed improvement practices among system hospitals; results. DSH status, then, is probably a marker for and 4) using technology and corporate information socio-demographic or clinical factors that are not systems to drive improvement, as well as other represented by any of the 30 variables tested in this factors that were not publicly available. These are study. hypotheses only, however, and these and other possible factors require further study and analysis. The available data do not reveal whether the mostly negative performance and weaker improvement Percentage of Gross Revenue Collected associated with DSH status is a result of poorer The percentage of gross revenue collected is a hospital care, more challenging and unmeasured calculated variable in the OSHPD financial data patient characteristics, or both. The fact that set, reflecting the actual amount received from DSH status was prominent in several performance patients and third-party payers as a percentage domains suggests that disparities of care in the 4  |  C alifornia H ealth C are F oundation of all patient revenue. Essentially, it represents Non-factors in Hospital Performance the ability of hospitals to collect money for the While this study found a few hospital attributes that services they provide. It comes as no surprise that appear modestly related to performance, the large the higher percentage of gross revenue hospitals majority of factors that never or rarely appear in the collect, the better their performance on some quality models may be more significant. In other words, the indicators — in this study, ICU mortality, patient bigger story may be the many hospital characteristics satisfaction, and improvement in patient satisfaction that have no bearing on hospital performance or and one surgical infection prevention composite (SIP improvement. 3). Presumably, these hospitals are able to use the collected revenue to improve quality of care. One Nine major domains (representing 15 variables) did exception should be noted, however, with regard not factor in any of the 11 hospital performance or to the otherwise positive impact of percentage of improvement models. These hospital attributes are: gross revenue collected: This variable is negatively K Teaching status; associated with improvement in one surgical infection prevention measure (SIP 1), but the reason K Rural hospital status; is unclear. K Measures of hospital value (net property, plant, and equipment [PPE], net PPE/bed, and bad It is noteworthy that while several other measures debt); of income also were examined — gross inpatient K Length of stay; revenue, pre-tax net income, and net income — the gross revenue collected measure had the only K Presence of hospitalists; consistently significant effect in the models. K CABG volume; That is, the percentage of gross revenue collected was important with respect to performance and K Proportion of managed care revenue and managed performance improvement, but neither the level care days; of revenue (gross or net) nor profitability was K Proportion of minority discharges; significantly correlated with clinical performance. K Performance on the Leapfrog computerized provider order entry standard; Initial Starting Level of Performance During the analysis of improvement measures, the K Performance on the Leapfrog intensivist physician project evaluated the impact of the initial starting standard; value as an independent predictor of improvement. K System membership (two or more hospitals); For all of the change-over-time measures, this had the greatest impact on improvement. In several K Profitability (total margin, operating margin, and cases, this factor alone explained between 25 percent cost-per-charge ratio); and 50 percent of the variance in improvement. K Staffing measures; While initial starting value is not, strictly speaking, a K Occupancy measures; and hospital attribute, it was the dominant element in all of this study’s models exploring improvement, and K Pre-tax net income. has been described elsewhere.6 This is not surprising, since the absolute opportunity to improve is the Another five major hospital domains (representing greatest with the weakest performers. Also, since the 12 variables) were present only in one of the 11 highest level of performance for the process measures models — a relationship much too tenuous to is 100 percent, the ability of the strongest performers consider them performance predictors. to improve is capped. Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  5 Three major hospital domains were present in two are available, this latter issue is ripe for study to models but were inconsistent in the direction or type determine how compensation formulas contribute to of measure: clinical improvement. K Region was a factor in HAPU rates and ICU mortality but different regions were involved The level of engagement and participation by in each and no discernable pattern existed with hospital medical staff is generally recognized as regional groups; playing a role in clinical performance. Anecdotal reports from hospital leadership routinely mention K Ownership status was a factor, but with strikingly physician engagement as a strategic imperative when different results: investor-owned hospitals had talking about transparency and public reporting, but less improvement in one surgical improvement this issue is much harder to assess since standardized measure and tax-exempt hospitals were weakly tools and approaches are not available. associated with improved patient experience; and K Gross system revenue was related to better Hospitals consider staffing level in the quality performance in CABG mortality and CABG department as a driver of improved clinical improvement but is not involved in any other performance. As with all hospital staffing discussions, clinical area. however, the exact makeup and responsibilities of the quality department are difficult to categorize. Some These three hospital characteristics are only weak hospitals have begun to move quality department predictors of performance in a limited number of responsibility for specific clinical measures toward models and do not display any consistent patterns. the operational unit involved. For example, surgical Statistically, they have some role in each of the infection prevention is the responsibility of the clinical conditions where they were present in the operating room along with the surgery department, model but may well be markers for other, unreported while ICU process measures reside with the ICU aspects of hospital performance. In sum, it is difficult staff. The differences in these practices would make to make the case that they are important predictors, comparisons among hospitals difficult. and they probably are not factors in the larger view of hospital performance. Specific departmental structures and practices within operational units at hospitals were not studied in Study Limitations this project (except for the Leapfrog group standard Although the study included a large number of involving intensivist physicians). With regard to hospital characteristics, the list of attributes it hospitalist programs, a hospital-wide structure examined is not exhaustive. Most quality experts this project evaluated, they did not appear to be believe other attributes are also important. With independent drivers of the clinical areas examined. regard to improvement, in particular, many There are now standardized measures with which experts suggest that leadership participation plays to examine specific practices and structures across a role. In this regard, future study could include a large group of hospitals, so other organizational a survey instrument recently adopted by CMS to structures and practices should be studied in the assess leadership practices associated with better future. performance and safety (the Healthcare Leadership and Quality Assessment Tool). Participation in Testing in this study was very limited regarding the an executive compensation program that rewards impact of health information technology. Only one improved performance on items like CHART such proxy (from Leapfrog), related to computerized measures also may be a marker of executive physician order entry, was studied, and it was not an engagement. Now that standardized measures independent driver of performance on the measures 6  |  C alifornia H ealth C are F oundation examined. Since health information technology is a famous quality triad — with structure, process, and major part of discussions in both the hospital market outcomes in equipotent roles — does not seem to and health policy circles, it should be studied against hold up based on the data analyzed in this study. actual performance. Of course, certain structural variables that were not tested in this study (primarily because data for them Implications are not available) may conceivably play significant The findings from the study’s analysis of various roles, so this conclusion about Donabedian’s triad hospital characteristics on a range of clinical remains a tentative one. performance measures may have significant implications regarding public reporting, performance Based on this study’s findings, however, it can improvement, and public policy. be postulated that clinical performance and improvement are predominantly driven not by Transparency and Improvement structural hospital attributes but by the reliable Initially, the transparency movement in health implementation of effective practices across a care attempted simply to shine some light on diverse group of patients with a given condition. clinical performance by hospitals, physicians, other This includes attention to execution, and an providers, and health plans so that consumers understanding of care-related work flow abetted and purchasers could make more informed by the growing field of improvement science.8 decisions regarding their health care choices. Over Some organizational structures may yield higher time, however, the notion that publicly reported or lower levels of reliable implementation, but it is information might be widely used to make choices likely these structures will be directly related to the has waned, replaced by hope that it could be used actual practices rather than broad constructs such to spur improvement in clinical performance. Some as hospital size, teaching status, ownership, hospital recent studies suggest that public reporting has had financial health, or the many other attributes some limited effect on motivating work on quality evaluated in this study. improvement, but for the most part the drivers of improvement are poorly understood.7 (It is also There are several implications of the project’s analysis thought that publicly reported information may for public reporting and performance: promote the development and deployment of new 1.With regard to improvement measures, financing structures, such as pay-for-performance several are nearly “topped off” and so provide and value-based purchasing, but the scope of little information to consumers and other this project did not permit an evaluation of such stakeholders. Also, since improvement is often structures.) most strongly predicted by a low initial starting value, by definition significant improvement will The data from this study suggest that only a few slow as starting value rises. As a result, public out of a large number of mostly structural hospital reporting entities may want to alter the use of attributes are associated with more than one or topped-off measures such as “all-or-nothing” two performance measures. Somewhat surprisingly, reporting (in which, for a given condition, a hospital’s or health system’s financial health is if any individual measure is not performed, not broadly associated with either performance or care is deemed to be unreliable and a score of improvement. While this study did not assess all zero — “nothing”— is rendered), or simply report structural hospital characteristics, the fact that so those hospitals that perform below a chosen many had limited or no impact begins to challenge threshold. long-held assumptions about the role of such characteristics in quality improvement. Donabedian’s Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  7 2.Since this study found no relationship between with poorer performance because of a weaker most hospital attributes — such as teaching financial picture or insufficient funds overall for the status, hospital size, ownership, and other hospitals. characteristics — and clinical performance, stratifying hospitals by these categories tends A different explanation for poorer performance to mislead the intended audience. DSH status focuses on the patient population. DSH status may be an exception, but there should be no is likely to reflect patients with particular socio- stratification by DSH status until it is clear demographic features — patients who tend to have that performance differences are due primarily poor follow-up after discharge, who often experience to poorer care and not to unmeasured patient delays in care for a variety of reasons, and who may characteristics. not respond to treatments in the same ways as other patients. If the explanation for poorer outcomes 3.Because of the lack of correlation between and less improvement in DSH facilities is found structural characteristics and performance to result mostly from such unmeasured patient measures, hospitals should emphasize the characteristics, then most risk-adjustment models execution of improvement strategies rather than meant to normalize various patient factors will need major structural or transformational strategies. modification. Also, public reporting programs might 4.All hospital types have the opportunity to consider altering their stratification of DSH facilities improve, regardless of their organizational in order to account for the differences in patient structures, finances, or initial level of population. performance. Different types of hospitals and hospital systems can share effective improvement There is also, however, a need to investigate whether strategies with a fair amount of confidence that and to what extent DSH facilities perform worse the strategies are not unique to their facility. as the result of poorer care delivery. If additional research uncovers actionable deficiencies in care Public Policy practices, specific policies will need to attend to those Policy discussions regarding hospital quality often deficiencies. From a policy perspective, poorer care revolve around ownership type, region, or some at DSH facilities needs to be addressed regardless other organizational structure. This study uncovered of its sources, whether unmeasured differences in that only in a small number of models did these patient factors, deficient care practices, or other types of attributes drive performance. That is, they undetermined causes. do not serve as broad proxies for overall quality of care at California hospitals. Consequently, policymakers should adjust their approach regarding matters of performance, ignoring these structural characteristics when creating legislation or regulation to improve California hospital quality. In the specific case of DSH facilities, the findings of this study suggest an urgent need to explore why this status often correlates with poor performance. DSH facilities receive revenue from the federal government based on a complicated formula using matching funds sent by state and local governments. But this study determined that DSH status is not associated 8  |  C alifornia H ealth C are F oundation Endnotes 1. Kroch, E.A., M. Duan, S. Silow-Carroll, and J. Meyer. April 2007. “Hospital Performance Improvement: Trends in Quality and Efficiency.” The Commonwealth Fund. www.commonwealthfund.org/ publications/publications_show.htm?doc_id=471264; Thornlow, D.K., and G. Stukenborg. March 2006. “The Association Between Hospital Characteristics and Rates of Preventable Complications and Adverse Events.” Medical Care 44(3): 265 – 269. 2. Jha, A.K. June 2008. “Does the Leapfrog Program Help Identify High-Quality Hospitals?” Journal on Quality and Patient Safety 34(6): 318 – 325. 3. Hines, S., and M. Joshi. June 2008. “Variation in Quality of Care Within Health Systems.” Journal on Quality and Patient Safety 34(6): 326 – 332; Kahn, C. January/February 2006. “Snapshot of Hospital Quality Reporting and Pay-for-Performance Under Medicare.” Health Affairs 25(1): 148 – 162. 4. Bradley, E.H., et al. July 2006. “Hospital Quality for Acute Myocardial Infarction: Correlation Among Process Measures and Relationship with Short-term Mortality.” Journal of the American Medical Association 296: 72 – 78. 5. Auerbach, A., and E. Vasilevskis. University of California, San Francisco. Personal communication. 6. Kroch, E.A. et al. “Hospital Performance Improvement.” 7. Fung, C.H., et al. January 2008. “Systematic Review: The Evidence That Publishing Patient Care Performance Data Improves Quality of Care.” Annals of Internal Medicine 148(2): 111 – 123; Hibbard, J. January 2008. “What Can We Say About the Impact of Public Reporting? Inconsistent Execution Yields Variable Results.” Annals of Internal Medicine 148(2): 160 – 161. 8. Berwick, D.M. March 2008. “The Science of Improvement.” Journal of the American Medical Association 299(10): 1182 – 1184. Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  9 Appendix A: Data Sources This study relied upon data from CHART concerning hospital performance across various measures. CHART collects data from a number of sources: 1.Core measures (CMS and TJC) are submitted from a hospital’s data vendor who submits the same measures to national program Web sites. 2.Clinical outcomes regarding CABG and pneumonia mortality are derived from OSHPD reports. 3.ICU mortality variables and process measures are submitted by hospitals. 4.Patient experience data are submitted from a hospital’s data vendor, specific to that measure domain, through the National CAHPS Benchmarking Database. 5.Hospital-acquired pressure ulcer (HAPU) results are submitted to the California Nursing Outcomes Coalition (CalNOC). 6.Leapfrog performance standards for participating California hospitals are obtained from the Leapfrog Group (www.leapfroggroup.org). Performance data from CHART are refreshed quarterly; this project used five data sets, from March 2007 through May 2008. Some measures, such as HAPU rates and ICU mortality, were added in January 2008. Table 1 shows the sample period for the various measures in CHART. Table 1. Project Sample Periods for CHART Data D ata R e fre sh Se t March 2 0 0 7 July 2007 Octob er 2007 January 2008 May 2008 CABG 2003 2003 2004 2005 2005 Maternity 2004 2004 2004 2006 2006 Patient Experience December 2005 March through March through July 2006 through July 2006 through through December 2006 December 2006 June 2007 June 2007 February 2006 TJC/CMS Core Measures Q3 2005 through Q1 through Q2 2006 through Q3 2006 through Q4 2006 through Q1 2006 Q4 2006 Q1 2007 Q2 2007 Q3 2007 Leapfrog June 2006 June 2006 June 2007 Not directly Not directly reported reported OSHPD Pneumonia 2002 to 2004 2002 to 2004 2002 to 2004 2002 to 2004 ICU Process and Outcomes Q1 through Q1 through Q3 2007 Q4 2007 CalNOC HAPU Q1 through Q1 through Q3 2007 Q4 2007 10  |  C alifornia H ealth C are F oundation The study also used financial, operational, and structural data on California hospitals from OHSPD. Other measures are taken from OSHPD reports, some of which are updated annually, others less frequently. Table 2 describes the time periods for OSHPD data that were used as a portion of the independent variables in the analysis. Table 2. Project Sample Periods for OSHPD Data Data Source and Time Period P e rformanc e Me asure C HART OSHPD CABG Q1 through Q4 2005 2005 annual financial data Patient Experience Q3 2006 through Q2 2007 Rolling quarterly financial data covering Q3 2006 through Q2 2007 ICU Q1 through Q3 2007 Rolling quarterly financial data covering Q4 2006 through Q3 2007 HAPU Q1 through Q3 2007 Rolling quarterly financial data covering Q4 2006 through Q3 2007 The project augmented OSHPD data on hospital characteristics with information from other sources: K System size (number of hospitals) from the Web sites of each health system with hospitals in California; K Teaching program designation from the Accreditation Council for Graduate Medical Education (ACGME — www.acgme.org/adspublic) list of residency programs in California (a more extensive list than the OSHPD teaching hospital designation and the Council of Teaching Hospitals [COTH] lists, which include only major teaching programs); K System revenue for tax-exempt organizations from each organization’s IRS Form 990 for 2006, found at www.guidestar.org; and K Identification of hospitalist programs in California generated in the report “The Rise of the Hospitalist in California,” Vasilevskis et al., July 2007, published by the California HealthCare Foundation (www.chcf.org/documents/policy/RiseHospitalistCalifornia.pdf ). Additional conversations with the report’s authors validated information regarding hospitalist programs in California in 2007. Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  11 Appendix B: Measures CHART Measures (Dependent Variables) The hospital performance measures reported by CHART can be found at www.calhospitalcompare.org, with detailed data definitions at chart.ucsf.edu. Briefly, CHART produces measures for the following: K ICU mortality, using the MPM II risk-adjustment methodology (S. Lemeshow, D. Teres, J. Klar, J. S. Avrunin, S. H. Gehlbach, and J. Rapoport. November 1993. “Mortality Probability Models [MPM II] Based on an International Cohort of Intensive Care Unit Patients.” Journal of the American Medical Association 270[20]: 2478 – 2486.) A sample of hospitals participates in an external audit. ICU mortality is reported on a rolling 12-month basis. K Hospital-acquired pressure ulcers (HAPU), using a prevalence methodology (www.calnoc.org). Hospitals receive training and certification to conduct the study. HAPU data are reported on a rolling 12-month basis. K CABG mortality, using a customized risk-adjustment methodology with medical record-based clinical variables calculated by OSHPD (www.oshpd.ca.gov/HID/Products/Clinical_Data/CABG/2005/ CCORP2005_Web.pdf ). CABG mortality is reported annually by OSHPD for calendar year periods. K Patient experience, using a modified form of the HCAHPS® tool with four additional experience questions. Patient experience results are reported on a rolling 12-month basis. K ICU process measures, which are self-reported, related to reducing ventilator-associated pneumonia. A subset of measures is audited by an external firm in a sample of hospitals. ICU process measures are reported on a rolling 12-month basis. K Core measures, including acute myocardial infarction, heart failure, surgical improvement, and pneumonia, using CMS/TJC specifications. A weighted composite of the individual results for each domain is created (chart.ucsf.edu). Core measures are reported on a rolling 12-month basis. 12  |  C alifornia H ealth C are F oundation Hospital Characteristics (Independent Variables) Table 3 groups the hospital characteristics used as independent variables in this study into five basic categories. Table 3. Hospital Characteristics by Category I ncom e Value Profitability S tr ucture Oth er Pre-tax net income Net PPE Total margin Size Region (property, plant, and equipment [PPE] and construction-in-progress) Net income Net PPE per bed Operating margin Teaching status Rural Gross inpatient revenue Bad debt Cost-per-charge Ownership Total nursing level ratio (city/county, district, investor, non-profit) Percentage of gross System membership Staffed or licensed bed revenue collected occupancy Disproportionate Share System size Adjusted length of stay Hospital (DSH) status (number of hospitals) System gross revenue Leapfrog computerized Presence of hospitalists physician order entry standard Proportion of managed Leapfrog intensivist Proportion of… care revenue physician standard • minority discharges • managed care days CABG volume Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  13 Appendix C: Hospital Performance by Condition Coronary Artery Bypass Graft Mortality DSH facilities tend to have an almost 1 percent The study included CABG performance data for higher level of CABG mortality. This is due to DSH both CHART hospitals (N=100) and non-CHART status alone: This difference remains when various hospitals (N=20). While no significant differences financial factors are controlled for, regardless of were found between the CHART and non-CHART hospital size (number of discharges). groups, there was significant variation in CABG mortality (range 0 to 11.5 percent) among individual Total nurse staffing levels also significantly affect hospitals. CABG mortality: On average, the higher the nursing level, the lower the CABG mortality. More Large hospitals fare worse when it comes to CABG specifically, for each additional paid registered nurse mortality: Hospitals with annual discharges of or licensed vocational nurse hour per patient day, 18,000 or more tend to have CABG mortality rates CABG mortality decreases by about 0.34 percent. about 1.2 percent higher than hospitals with fewer discharges. System revenue also has a significant, ICU Mortality though small, effect on CABG mortality: The larger With regard to performance regarding ICU the system revenue, the lower the CABG mortality. mortality, there is a fair amount of variation among The decrease in mortality is 0.067 percent for each all hospitals (range 9.5 percent to 25 percent), and additional billion dollars in system revenue. there are significant differences among hospital systems, as shown in Figure 1. Figure 1. ICU Mortality by Health System Catholic Healthcare West* 15.0% Providence Health & Services 14.8% Memorial Health Services 14.7% Kaiser Permanente 14.5% Not part of system with 3+ hsps 14.3% Tenet Healthcare 13.9% University of California 13.5% Adventist Health System 13.5% Sutter Health* 12.9% Sharp HealthCare 12.6% Scripps Health 12.2% St. Joseph Health System† 11.9% * ifference between this system and all other hospitals statistically significant at the 95 percent level of confidence. D †Difference between this system and all other hospitals statistically significant at the 90 percent level of confidence. 14  |  C alifornia H ealth C are F oundation When other variables are controlled using a role in determining the outcome than hospital regression model, five factors play a role in the infrastructure or finance. mortality rate associated with ICU care. (The high performance noted in Figure 1 for St. Joseph Hospital-Acquired Pressure Ulcers Health System and Sutter Health disappears in Analysis of HAPU performance showed that HAPU the regression model and so is not independently rates differ by region, with northern California associated with better ICU mortality performance.) hospitals in rural areas and in the extended Each of the five ICU mortality factors has a nearly Sacramento area having much lower — better —  equal role in the model, but the model only weakly rates. This relationship did not hold for the San predicts mortality (R2=0.126). These five factors are: Francisco Bay Area, or for central or southern California regions. The reason for this relationship K Central region (predominantly the Central Valley) is unknown — it is not explained by membership in hospitals have worse performance; a health system in general or in a particular health K DSH hospitals have worse performance; system, or by being a rural hospital. The results may reflect the fact that these hospitals have addressed K Hospitals that do not submit any information to this subject longer than other hospitals: Some of the Leapfrog Group on their implementation of these hospitals have participated for years in a the intensivist physician standard are associated program with CalNOC to measure and internally with worse performance; and report HAPU rates, so perhaps a greater number of K Hospitals in the CHW and Kaiser Permanente hospitals in this region are CalNOC participants. systems as a whole have worse performance (although individual hospitals may have average Hospital size also is associated with HAPU rates: or better-than-average performance). Small hospitals (fewer than 6,500 discharges per year) consistently have better rates. Better The study saw no financial or other structural performance in small hospitals was not correlated hospital characteristics independently associated with with being a rural hospital, since this factor ICU mortality rates. was not in the final model. And while smaller hospitals are often financially challenged because It is unclear why the factors noted above are of high fixed costs and the lack of economies of important, except that DSH status might reflect scale found in larger hospitals, it turned out that unmeasured clinical socio-demographic or income, profitability, and value were not associated other factors, poorer care for a more challenging with HAPU rates. This might be because smaller population, or a combination of both (as discussed hospitals may be able to provide better attention to in the body of this report). Clearly, poorer fewer patients or to fewer higher-risk patients. For performance for CHW and Kaiser systems is worth example, these hospitals may tend to transfer long addressing by the particular hospitals and corporate term ICU patients to other settings. structures involved. A higher proportion of hospital days paid by Another important issue is that ICU mortality is managed care organizations is associated with worse almost certainly affected by pre-ICU care, perhaps HAPU performance. This was the only financial even care provided before the patient is seen in variable involved in the regression model. Why the the hospital. This might explain why the statistical proportion of managed care hospital days is a feature relationship between the model and mortality rates in only this one model is unclear. is so weak: Overall actual care plays a much greater Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  15 Figure 2. Hospital-Acquired Pressure Ulcer Rates by Health System St. Joseph Health System 0.049 Providence Health & Services 0.042 Not part of system with 3+ hsps 0.039 Memorial Health Services 0.039 University of California 0.038 Scripps Health 0.038 Sutter Health 0.035 Catholic Healthcare West 0.033 Tenet Healthcare 0.032 Kaiser Permanente 0.031 Adventist Health System 0.027 Sharp HealthCare* 0.012 * ifference between this system and all other hospitals statistically significant at the 95 percent level of confidence. D It is worth noting that when specific health systems DSH facilities also tend to have slightly lower with more than three hospitals were examined (see (4 percent) rates of patient satisfaction, an effect not Figure 2, above), Sharp HealthCare stood out as related to financial or other factors in the model. a high performer and independent predictor of The study explored whether these differences might success regarding HAPU rates. The practices Sharp be driven by the racial diversity of the DSH patient HealthCare has employed to lower HAPU rates population, but such differences did not explain this probably should be studied, including the feasibility finding. of implementing them at other hospitals. Hospitals that have a higher percentage of gross Patient Experience revenue collected, have higher total margins, or The initial model confirms many of the findings are nonprofit tend to have higher rates of patient from the individual variable evaluations. For satisfaction. An increase in the percentage of gross example, medium-size hospitals (between 4,000 and revenue collected yields a very small (0.2 percent) 16,500 annual discharges) tend to have lower rates increase in patient satisfaction, while a 1 percent of patient satisfaction than other hospitals — an increase in total margin yields the largest effect effect not traceable to other factors such as nonprofit on performance, a 20 percent increase in patient status, DSH status, hospital financial health, or satisfaction scores. On average, nonprofit hospitals occupancy rates. The effect on patient experience score 3 percent higher than other hospitals. As seen scores, however, is small (4 percent). A higher in Figure 3 on the next page, however, this nonprofit licensed bed occupancy rate also has a negative factor disappears when specific systems with four impact on patient satisfaction, independent of the or more hospitals are included, suggesting that other variables included in the model, with a modest nonprofit status is a proxy for specific, but not all, effect (9 percent). tax-exempt health systems. 16  |  C alifornia H ealth C are F oundation Figure 3. Patient Satisfaction Rates by Health System Sharp HealthCare* 0.71 Scripps Health † 0.69 St. Joseph Health System* 0.69 Sutter Health* 0.68 Providence Health & Services 0.68 University of California† 0.67 Daughters of Charity 0.63 Catholic Healthcare West 0.61 Memorial Health Services 0.60 Not part of system with 3+ hsps 0.60 Tenet Healthcare 0.59 Adventist Health System 0.58 Kaiser Permanente* 0.54 * ifference between this system and all other hospitals statistically significant at the 95 percent level of confidence. D †Difference between this system and all other hospitals statistically significant at the 90 percent level of confidence. Initial findings of system performance are shown in Figure 3. When individual systems with four or more hospitals were added to the initial model, four health systems retained independent associations with patient satisfaction (Sharp HealthCare, Sutter Health, St. Joseph Health System, and Scripps Health). Status as a tax-exempt (not-for-profit) hospital disappeared in the final model, which strongly suggests that it is these four health systems that actually explain performance, rather than simply the fact of being a nonprofit. Sharp HealthCare had the highest effect, with a 10 percent higher level of performance. Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  17 Appendix D: Hospital Improvement Over Time CABG Mortality Improvement the risk-adjustment model annually to provide The study examined CABG mortality change in the best statistical fit for the data, irrespective of performance between 2003 and 2005 as reported whether there is any change in clinical practices or by OSHPD. There was no overall improvement performance. This tends to normalize performance in mortality during this time, and fewer than 20 over time, even if a hospital was improving or percent of hospitals continuously improved from worsening. 2003 through 2005. Other factors that independently support high The likelihood of improvement depended heavily or low CABG improvement include size — large on the initial starting level of performance. Figure 4, hospitals (more than 18,000 annual discharges) show below, demonstrates improvement in mortality, in less improvement — and higher gross system revenue, relation to initial level of mortality, between 2003 which yields greater improvement. and 2005. It is notable that CABG volume was not a factor Fifty percent of the variance in CABG mortality rate in any model, nor was it present in the overall change over time was explained by the initial starting performance analysis. rate. As discussed in the body of this report, this can be partially explained by the fact that mortality is The study also analyzed mortality improvement capped at 0 percent, and therefore high-performing using a broader definition: the presence of hospitals have much less room to improve. Another improvement, rather than the amount, using logistic reason initial value is the strongest driver among the regression. In this broader definition, the only improvement models is that OSHPD recalibrates additional factor besides initial starting mortality rate Figure 4. Initial Rate and Rate Change for CABG Mortality 8 7 6 2 R = 0.4496 5 4 3 2 1 Percentage Point Decrease In CABG Mortality 0 0 1 2 3 4 5 6 7 8 9 -1 Initial Risk-Adjusted Mortality Rate (Improving Hospitals Only) 18  |  C alifornia H ealth C are F oundation was that lower total nurse staffing is associated with words, it is not topped out: Both relatively low and less improvement. Notably, two factors associated relatively high performers still have plenty of room with increased improvement rates — large hospital for improvement. size and higher gross system revenue — are not correlated with whether hospitals actually improved, With respect to patient experience, the initial level even by very small amounts. of performance demonstrates an independent relationship as a predictor of improvement, but with Finally, the study assessed whether good-to-better an explanatory value of 5 percent, it is the lowest of hospitals (initial mortality rate below the median, all the improvement models. This is at least partially and improvement over time) increased their due to the fact that the measure is far from being improvement over time, and poor-to-worse (initial topped out. mortality rate above the median, and mortality increased) worsened over time. The study also used The only other variable affecting patient experience a stricter definition of “good-to-better,” looking improvement over time was the percentage of gross only at the hospitals that initially had below-median revenue collected. Hospitals that do a better job of mortality rates (better performance) and that reduced collecting on their billed charges seem to improve mortality further by at least 0.9 percent. While more on this measure, while those that do not the number of hospitals in these groups was small seem to improve the least. This may be a function (nine in the looser definition and five in the stricter of contracting rather than improvement, but since definition), the relationship with the initial starting the charge-to-cost ratio variable is not present in rate and total nurse staffing remains significant. The the model, this is not a result of a lower relative result is further evidence that total nurse staffing is level of charges. In other words, the prices charged, related to CABG mortality improvement, but not as a function of total costs, do not affect patient necessarily to large improvement. Although this experience improvement. measure reflects hospital-wide staffing levels while the specific activities of nursing in CABG programs Logistic regression offers another way to examine are often quite circumscribed (i.e., CABG nursing hospital attributes affecting the probability of staff are mostly exclusive to the CABG program and improvement. This method can indicate how patients), the relationship is nonetheless present and hospitals with the most improvement are different deserves further analysis. from the others. Since many hospitals improved patient satisfaction by a very small amount, this ICU Mortality and HAPU Improvement study evaluated the likelihood of being in the top 12 No analysis was performed on improvement with percent of hospitals (having at least a 4 percentage ICU mortality or HAPU rates because of insufficient point improvement in patient satisfaction score). available data (only one point in time from the The only key drivers of higher improvement are the January update of the www.calhospitalcompare.org initial level of patient satisfaction (the higher the Web site). initial score, the less likely the hospital is to be in the high-improvement group) and being an investor- Patient Experience Improvement owned hospital (rather than nonprofit, county, or Overall patient experience in California hospitals district hospital), which makes it significantly less showed virtually no overall change (median 0.601 to likely to be in the high-improvement group. 0.604) during the two periods analyzed in the study. Of the CHART measures this project evaluated, The study also analyzed the probability of a patient experience had the least amount of clustering hospital’s patient experience score worsening by near the highest levels of performance. In other at least 3 percent over the period, which would Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  19 put it in the bottom 9 percent of hospitals with The amount of improvement depended heavily respect to improvement. These effects turned on the starting AMI quality score, which explains out to mirror those for predicting the likelihood about 33 percent of the variation. No other of being a high-performing hospital. For each variables — e.g., financial factors, DSH status, system percentage point in initial patient satisfaction score, size, region, teaching status — had a significant effect the likelihood of being among the least-improving on the AMI quality composite. Similarly, when using hospitals increases by a small amount. Being an logistic regression to analyze the probability of being investor-owned hospital also increases a hospital’s among above-average-improving hospitals, the only chances of being in the worst-improving group, by a driver is the initial score. factor of more than two. Taking both logistic model results together, investor-owned hospitals are twice Congestive Heart Failure Quality as likely to fall into the worst hospital group with Overall heart failure quality composite scores among respect to improvement, and half as likely to place California hospitals improved significantly but among the high-performing hospitals with respect to modestly, by 4.5 percent, during the time periods improvement, regardless of their initial score. evaluated (median 82.6 to 87.1 percent). This measure is moving toward topping out and, as with CHART Core Measure Improvement other such measures, the initial level of performance The study analyzed a subset of the CHART explains a large amount (25 percent) of the variance measures related to nationally reported core in change over time. measures. Of the CHART-weighted composites for various groups of core measures, the study examined: Two variables, in addition to initial level, appear to have small but statistically significant effects on K Acute myocardial infarction (AMI) quality; change in heart failure improvement. DSH facilities K Congestive heart failure quality; tended to have declining scores over the period K Pneumonia quality; studied (by 1.7 points, on average), while hospitals in relatively large systems (10 or more hospitals) K Surgical Infection Prevention 1 (SIP 1); and tended to improve more than average (by about K Surgical Infection Prevention 3 (SIP 3). 1.7 points). Other variables were found to have no significant net effects in the model. Three composite CHART measures were not evaluated in this study: AMI timeliness, heart failure In a logistic regression analysis, the study looked prevention, and pneumonia prevention. These at the likelihood of achieving above-average were excluded because the number of patients improvement. In this analysis, the key drivers — in and hospitals in the AMI timeliness measure was addition to initial level of performance — appear to small, and because the prevention measures have no be large size (more than 18,000 annual discharges), immediate effect on hospital outcomes. reflecting less likelihood to have above-average improvement, and higher pre-tax net income, which Acute Myocardial Infarction Quality has a small but positive effect. There was a small but significant overall improvement in AMI quality scores (median Pneumonia Quality 94.8 to 96.2 percent) in California hospitals over CHART hospitals overall experienced a small but the time span analyzed in this study. Since this significant improvement in pneumonia quality of measure is close to topping out, with over half the 2.3 percent (median 89.1 to 91.4 percent) over the hospitals above 96 percent performance, only small time period studied. This measure is also moving improvement would be expected. toward topping out in the near future. 20  |  C alifornia H ealth C are F oundation As with other core measures, the degree of point. While small, the effect is statistically pneumonia quality improvement depends heavily significant at the 90 percent level of confidence. on the starting score. However, the initial level is not K Ownership type. Investor hospitals were more as important for the pneumonia quality composite likely to have their scores worsen over the as for the AMI or heart failure quality scores (11 period — by about 4.4 percentage points, on percent compared with 33 percent and 25 percent, average. respectively). Two other variables also appear to have statistically significant negative effects on pneumonia K System size. Hospitals in large systems tended composite improvement: the cost-to-charge ratio to improve scores by about 3.4 percentage points and membership in a small system (fewer than three more than other hospitals. hospitals, or independent). No other variables were found to have significant net effects in the model. No other tested variables were found to have significant net effects in the model. When the likelihood of achieving above-average improvement was examined, the key drivers The study also examined the likelihood of achieving appeared to be: initial level; a higher proportion above-average improvement. In this model, the key of patient days covered by managed care payers, drivers appear to be the initial level of SIP 1 (the which has a positive effect; and location in the state’s higher the initial score, the less likely the hospital is central region, which has a significant negative effect. to be in the above-average-improvement group) and The proportion of managed care days has a fairly being part of a large hospital system (which makes it substantial impact on above-average improvement; more likely to be in the high-improving group). this raises the interesting possibility that coordination with managed care entities, which in California often Surgical Infection Prevention 3 means capitated medical groups, explains some of CHART hospitals’ SIP 3 performance (associated this effect. with appropriate cessation of prophylactic antibiotics) showed the largest absolute improvement Surgical Infection Prevention 1 of all of the CHART measures studied, at During the period studied in this project, 10.3 percent (median 67.2 to 77.6 percent), over CHART hospitals experienced a moderate overall the time period evaluated in the study. As with other improvement in SIP 1 performance (related to core measures, the degree of improvement depends pre-surgery prophylactic antibiotic initiation) of heavily on the initial SIP 3 score, which accounts 6 percent (median 80.6 percent to 86.6 percent). for 32 percent of the variance in score change. Many hospitals are topping out on this measure, Four other factors also appear to have statistically but there is still room for substantial improvement significant effects on change with this measure: among low-performing hospitals. K Net income. For each $1 million increase in pre-tax net income, the SIP 3 score increases by As with other core measures, the amount of about three one-hundredths of a percent. For improvement depends heavily on the initial SIP 1 example, if one hospital has $100 million more in score, which accounts for 35 percent of the variance pre-tax net income than another hospital, its SIP in score change. Three other variables also appear to 3 score is likely to improve, on average, by about have statistically significant effects on change in the 3 percentage points more than the other hospital. SIP 1 score: While small, the effect is statistically significant at K Gross revenue collected. For each percentage the 90 percent level of confidence. point increase in gross revenue collected, the SIP 1 score decreases by about a tenth of a percentage Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  21 K DSH status. Perhaps surprisingly, DSH facilities were more likely to have their scores improve over the period — on average by about 3.4 percentage points more than non-DSH facilities. K System size. Hospitals in large systems tended to improve their scores more, on average by about 3.7 percentage points more than other hospitals. K Staffed beds. Staffed bed occupancy had a positive impact on SIP 3 improvement: With each unit increase in the staffed bed occupancy rate, the SIP 3 composite score increased by 4.7 points, on average. No other variables were found to have significant net effects in the model. 22  |  C alifornia H ealth C are F oundation Appendix E: Statistical Results of Major Hospital Attributes The following table displays statistically significant (of at least p # 0.10) performance measures for 14 major hospital attributes (which aggregate all 30 subcategories) analyzed in this study as independent variables. I n depe n d e n t Statis tica lly Significa nt Per forma nce Me a s ur e Var i abl e CABG CABG C hg I CU HAPU Pat S at C h g Pat S at AMI C h g HF Chg PN Qual Chg SIP 1 Chg SIP 3 Chg Region     Central – Northern               worse California performance‡ outside of the Bay Area – better performance* Hospital Size Large Large   Small   Medium           (18,000 or more (18,000 or more (fewer than (4,000 to 16,500 annual discharges) annual discharges) 6,500 annual annual discharges) – worse – worse discharges) – – worse performance* improvement* better performance† performance‡ Ownership           Nonprofit –       Investor-owned   better – worse performance* improvement* System     CHW – worse                 Membership performance* System Size               Large Small/none (10+ hospitals) (1– 3 hospitals) Large (10+ hospitals) – – better – worse better improvement† improvement* improvement* DSH Status DSH hospital –   DSH hospital –     DSH hospital –   DSH hospital –     DSH hospital – worse worse worse worse better performance performance performance* improvement improvement* Income     Higher         Higher Higher percentage of percentage of percentage of Higher percentage of gross gross revenue gross revenue gross revenue revenue collected – collected – collected – collected – better performance* better worse better performance* performance* performance* Profit           Higher total     Higher cost     margin – better to charge performance‡ ratio – worse improvement* *p # 0.05 †p # 0.005 ‡p # 0.0005 Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  23 I n dep e n d e n t Statis tica lly Significa nt Per forma nce Me a s ur e Var i a b l e CABG CABG C hg I CU HAPU Pat S at C h g Pat S at AMI C h g HF Chg PN Qual Chg SIP 1 Chg SIP 3 Chg Staffing Higher nurse                     staffing levels – better performance* Occupancy Rates           Higher licensed         Higher staffed bed occupancy bed occupancy rate – worse rate – better performance‡ improvement* Proportion       Higher               of Managed proportion Care-Days third-party managed care days – better performance* IPS     Non-submission                 of intensivists data to Leapfrog – worse performance* System Revenue Higher gross Higher gross                   system revenue system revenue – better – better performance improvement Initial Level   Higher mortality     Lower initial   rate in 2003 – level of better performance Lower initial level of performance – better improvement‡ improvement‡ – better improvement† *p # 0.05 †p # 0.005 ‡p # 0.0005 Note: The following attributes were not found in any of the models: rural location; teaching hospital designation; value (net PPE, bad debt); length of stay; presence of hospitalists; CABG volume; and computerized physician order entry. 24  |  C alifornia H ealth C are F oundation Appendix F: Statistical Results from Regression Models The specific statistical regression models for each dependent variable are listed below. The explanatory power of the model is indicated by the R2 value, with higher results indicating a better statistical fit with the data. Unstandardized coefficients describe the absolute impact of each variable in the model, using the unit of measurement (number of discharges, billions of dollars, etc.) specific to each variable. The standardized coefficients describe the relative impact of each variable and can be compared to one another. For example, in the CABG model, having 18,000 or more discharges (Beta=0.279) has a relatively larger impact on CABG mortality than does being a DSH hospital (Beta=0.181). Only variables with statistically significant effects are shown; if a variable is not shown, it means that it does not have a significant effect on the given measure once the other variables in the model are included. Table 4. CABG Mortality Unstandardized S tandardi ze d Coefficients C oefficients Variable B S td. Error B e ta S ig. (Constant) 5.749 1.138   0.000 Large Hospital (more than 18,000 annual discharges) 1.241 0.438 0.279 0.006 System Gross Revenue* (in billions of dollars) – 0.067 0.039 – 0.159 0.094 DSH Hospital Status 0.981 0.522 0.181 0.064 Nurse Staffing Level – 0.337 0.139 – 0.234 0.017 Notes: N=100 and R2=0.188 *System Gross Revenue is calculated from 2005 tax year forms for all nonprofit systems. For non-system hospitals and for-profit systems, gross system revenue is the sum of gross revenue across relevant hospitals listed in the full OSHPD 2005 annual financial data. Sources: CHART CABG Data (2005) and OSHPD Annual Financial Data (2005). Table 5. ICU Mortality Unstandardized S tandardi ze d Coefficients C oefficients Variable B S td. Error B e ta S ig. (Constant) 14.79 0.75   0.00 Central Region 1.98 0.68 0.21 0.00 CHW 1.19 0.49 0.18 0.02 DSH Hospital Status 0.96 0.50 0.14 0.06 Did Not Submit Leapfrog Intensivist Physician Standard 1.80 0.69 0.19 0.01 Percentage of Gross Revenue Collected – 0.06 0.03 – 0.16 0.03 Notes: N=166 and R2=0.184 Source: CHART ICU Data (Q1 – Q3 2007) and OSHPD Quarterly Financial Data (Q4 2006 – Q3 2007). Because the system membership performance varied as noted in the table above, the study tested all systems (not just CHW) for significant effects but had to remove the variable Percentage of Gross Revenue Collected since OSHPD does not collect that information for Kaiser Permanente hospitals. Tables 6, 7 and 8 present the results of that analysis. Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  25 Table 6. ICU Model Results Including Kaiser Permanente Hospitals Uns ta nd a r d ized Standardized Coeffic ients C oeffic ients Va r i a b l e B Std. Error B e ta Sig. (Constant) 13.191 0.257   0.000 Central 1.865 0.699 0.186 0.008 DSH Hospital Status 0.956 0.532 0.127 0.074 Did Not Submit Leapfrog Intensivist Physician Standard 1.660 0.725 0.160 0.023 Kaiser Permanente 1.210 0.549 0.156 0.029 CHW 1.207 0.524 0.162 0.022 Notes: N=195 and R2=0.126 Sources: CHART ICU Data (Q1 – Q3 2007) and OSHPD Quarterly Financial Data (Q4 2006 – Q3 2007). Table 7. HAPU Uns ta nd a r d ized S tandardized Coeffic ients C oeffic ients Va r i a b l e B Std. Error B e ta Sig. (Constant) 0.048 0.003   0.000 Northern California (excluding Bay Area) – 0.009 0.004 – 0.171 0.021 Small Hospital (fewer than 6,500 annual discharges) – 0.016 0.004 – 0.305 0.000 Proportion of Third-Party Managed Care Days – 0.028 0.009 – 0.208 0.004 Sharp HealthCare – 0.027 0.010 – 0.173 0.011 Notes: N=191 and R2=0.174 Sources: CHART HAPU Data (Q1 – Q3 2007) and OSHPD Quarterly Financial Data (Q4 2006 – Q3 2007). Table 8. Patient Experience Uns ta nd a r d ized S tandardized Coeffic ients C oeffic ients Va r i a b l e B Std. Error B e ta Sig. (Constant) 0.657 0.028   0.000 Medium-Size Hospital (4,000 to 16,500 annual discharges) -0.040 0.011 -0.221 0.001 DSH Hospital Status -0.042 0.014 -0.183 0.005 Percentage of Gross Revenue Collected 0.002 0.001 0.148 0.032 Total Margin 0.191 0.052 0.242 0.000 Licensed Bed Occupancy Rate -0.084 0.024 -0.226 0.001 St. Joseph Health System 0.071 0.031 0.140 0.024 Scripps Health 0.067 0.034 0.121 0.050 Sharp HealthCare 0.103 0.034 0.187 0.003 Sutter Health 0.044 0.016 0.174 0.007 Notes: N=184 and R2=0.362 Sources: CHART Patient Satisfaction Data (Q3 2006 – Q2 2007) and OSHPD Quarterly Financial Data (Q3 2006 – Q2 2007). 26  |  C alifornia H ealth C are F oundation CABG Mortality Improvement The first analysis looked at potential factors that affect the absolute amount of improvement over time. Table 9 displays the results. Table 9. ariables in the Equation: Percentage Point Improvement in CABG Mortality, 2003 – 2005 V (OLS Regression) Unstandardized Standardi ze d Coefficients C oefficients Variable B S td. Error B e ta S ig. (Constant) – 3.374 0.422   0.000 Initial Mortality Rate (2003) 1.064 0.112 0.708 0.000 Large Hospital (more than 18,000 annual discharges) – 0.895 0.416 – 0.157 0.034 Gross System Revenue (in billions of dollars) 0.069 0.039 0.132 0.078 Notes: N=91 and R2=0.548 Another way to describe improvement is whether any improvement occurred during the time period studied. Using logistic regression, the result is the model shown in Table 10. Table 10. Variables in the Equation: Any Improvement, 2003 – 2005 (Logistic Regression) Variable B S.E. Wald df S ig. Exp(B) (Constant) – 7.799 2.197 12.604 1 0.000 0.000 Initial Mortality Rate (2003) 1.026 0.217 22.461 1 0.000 2.791 Total Nursing Staff 0.520 0.230 5.125 1 0.024 1.683 Notes: N=91 and Nagelkerke R2=0.541 The project examined good-to-better hospitals (initial mortality rate below median, with improvement) and poor-to-worse (initial mortality rate above median, and worse mortality over time), which yielded the models shown in Tables 11 and 12. Table 11. Variables in the Equation: Good-to-Better Hospitals,* 2003 – 2005 (Logistic Regression) Variable B S.E. Wald df S ig. Exp(B) (Constant) – 6.159 1.935 10.131 1 0.001 0.002 Total Nursing Staff 0.454 0.210 4.664 1 0.031 1.574 Notes: N=91 and Nagelkerke R2=0.105 *Below median-level mortality in 2003 and improved by 2005. Table 12. Variables in the Equation: Poor-to-Worse Hospitals,* 2003 – 2005 (Logistic Regression) Variable B S.E. Wald df S ig. Exp(B) (Constant) – 3.821 0.928 16.949 1 0.000 0.022 Initial Mortality Rate (2003) 0.356 0.185 3.691 1 0.055 1.427 Large Hospital (more than 18,000 annual discharges) 1.273 0.673 3.572 1 0.059 3.571 Notes: N=91 and Nagelkerke R2=0.162 *Above median-level mortality in 2003 and no improvement by 2005. Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  27 Patient Experience Improvement Table 13. ariables in the Equation: Percentage Point Improvement in Patient Satisfaction, V July 2007 – January 2008 (OLS Regression) Unstandardized S tandardi zed C oefficients Coefficients Variable B S td. Error B e ta S ig. (Constant) 0.044 0.017   0.013 Initial Patient Satisfaction – 0.090 0.028 – 0.255 0.001 Percentage of Gross Revenue Collected 0.001 0.000 0.167 0.034 Notes: N=165 and R2=0.071 The study also analyzed the likelihood of being a top 12 percent hospital (improving by at least 4 percentage points) using logistic regression. Table 14. ariables in the Equation: Likelihood of Top 12 Percent Improvement re Patient Satisfaction, V July 2007 – January 2008 (Logistic Regression) Variable B S.E. Wald df S ig. Exp(B) (Constant) 2.967 1.091 7.391 1 0.007 19.428 Initial Patient Satisfaction – 0.042 0.018 5.666 1 0.017 0.959 Investor-Owned Hospital – 0.873 0.452 3.733 1 0.053 0.418 Notes: N=192 and Nagelkerke R2=0.060 Finally, the study looked at the likelihood of poor performance over time (being among the bottom 9 percent of hospitals — decreasing by at least 3 percentage points). Table 15. ariables in the Equation: Likelihood of Poor Performance re Patient Satisfaction Improvement, V July 2007 – January 2008 (Logistic Regression) Variable B S.E. Wald df S ig. Exp(B) (Constant) – 2.868 1.098 6.821 1 0.009 0.057 Initial Patient Satisfaction 0.038 0.018 4.666 1 0.031 1.039 Investor-Owned Hospital 0.818 0.448 3.331 1 0.068 2.265 Notes: N=165 and Nagelkerke R2=0.316 28  |  C alifornia H ealth C are F oundation Core Measure Improvement AMI Quality Composite Table 16. ariables in the Equation: Percentage Improvement in AMI Quality Composite, 2006 – 07 V (OLS Regression) Uns ta nd a rd ized Sta nd a rd ized C oefficients C oefficients Va r iab l e B Std. Error B e ta Sig . (Constant) 30.778 3.426   0.000 AMI QC, Time 1 – 0.311 0.036 – 0.573 0.000 Notes: N=184 and R2=0.329 Another way to assess this relationship is to examine whether a hospital had above-average improvement rather than the percentage point improvement. Table 17. ariables in the Equation: Above-Average AMI Quality Improvement, 2006 – 07 V (Logistic Regression) Va r iab l e B S.E. Wa l d df Sig . Exp(B) (Constant) 27.047 5.351 25.551 1 0.000 5.577E11 Initial AMI Quality Score – 0.287 0.056 26.212 1 0.000 0.750 Notes: N=153 and Nagelkerke R2=0.321 Heart Failure Quality Composite Table 18. ariables in the Equation: Percentage Improvement in Heart Failure Composite, 2006 – 07 V (OLS Regression) Uns ta nd a rd ized Sta nd a rd ized C oefficients C oefficients Va r iab l e B Std. Error B e ta Sig . (Constant) 26.440 2.537   0.000 Initial Heart Failure Value – 0.272 0.031 – 0.550 0.000 DSH Hospital Status – 1.658 0.899 – 0.114 0.067 Large System Hospital 1.663 0.693 0.150 0.017 (membership in system with 10+ hospitals) Notes: N=201 and R2=0.290 Table 19. ariables in the Equation: Above-Average Improvement in Heart Failure Composite, 2006 – 07 V (Logistic Regression) Va r iab l e B S.E. Wa l d df Sig . E x p(B) (Constant) 10.580 2.047 26.726 1 0.000 39358.722 Initial Heart Failure Value – 0.125 0.024 27.819 1 0.000 0.882 Large Hospital (more than 18,000 annual discharges) – 1.546 0.521 8.815 1 0.003 0.213 Pre-Tax Net Income (in millions of dollars) 0.012 0.005 5.426 1 0.020 1.012 Notes: N=170 and Nagelkerke R2=0.369 Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  29 Pneumonia Quality Composite Table 20. ariables in the Equation: Percentage Improvement in Pneumonia Composite, 2006 – 07 V (OLS Regression) Unstandardized Standardi zed C oefficients Coefficients Variable B S td. Error B e ta S ig. (Constant) 26.073 4.344   0.000 Initial Pneumonia Quality Value – 0.252 0.045 – 0.442 0.000 Cost-to-Charge Ratio – 5.331 2.380 – 0.171 0.026 Small System Hospital – 0.805 0.397 – 0.150 0.044 (membership in system with 1– 3 hospitals) Notes: N=201 and R2=0.157 Table 21. ariables in the Equation: Above-Average Improvement in Pneumonia Quality Composite, 2006 – 07 V (Logistic Regression) Variable B S.E. Wald df S ig. Exp(B) (Constant) 15.047 3.947 14.535 1 0.000 3425900.876 Initial Pneumonia Quality Value – 0.177 0.045 15.602 1 0.000 0.838 Proportion of Managed Care Days 2.493 0.987 6.385 1 0.012 12.100 Central Region – 1.146 0.588 3.799 1 0.051 0.318 Notes: N=171 and Nagelkerke R2=0.164 Surgical Improvement Prevention 1 (SIP 1) Table 22. ariables in the Equation: Percentage Improvement in SIP 1 Composite, 2006 – 07 V (OLS Regression) Unstandardized Standardi zed C oefficients Coefficients Variable B S td. Error B e ta S ig. (Constant) 38.613 4.118   0.000 Initial SIP 1 Value – 0.374 0.040 – 0.577 0.000 Percentage of Gross Revenue Collected – 0.126 0.074 – 0.111 0.091 Investor-Owned Hospital –4.398 1.605 – 0.182 0.007 Large System Hospital 3.390 1.048 0.203 0.001 (membership in system with 10+ hospitals) Notes: N=201 and R2=0.441 Table 23. ariables in the Equation: Above-Average Improvement in SIP 1 Composite, 2006 – 07 V (Logistic Regression) Variable B S.E. Wald df S ig. Exp(B) (Constant) 10.441 1.871 31.129 1 0.000 34231.565 SIP 1, Time 1 – 0.135 0.022 36.338 1 0.000 0.873 Large System Hospital 0.720 0.353 4.151 1 0.042 2.054 (membership in system with 10+ hospitals) Notes: N=197 and Nagelkerke R2=0.401 30  |  C alifornia H ealth C are F oundation Surgical Improvement Prevention 3 (SIP 3) Table 24. ariables in the Equation: Percentage Improvement in SIP 3 Composite, 2006 – 07 V (OLS Regression) Unstandardized S tandardi ze d Coefficients C oefficients Variable B S td. Error B e ta S ig. (Constant) 23.151 3.441   0.000 Initial SIP 3 Value – 0.321 0.035 – 0.579 0.000 Pre-Tax Net Income 0.031 0.014 0.141 0.027 (in millions of dollars) DSH Hospital Status 3.420 1.504 0.143 0.024 Large System Hospital 3.693 1.172 0.197 0.002 (membership in system with 10+ hospitals) Staffed Bed Occupancy 4.665 2.009 0.144 0.021 Notes: N=165 and R2=0.401 Table 25. ariables in the Equation: Above-Average Improvement in SIP 3 Composite, 2006 – 07 V (Logistic Regression) Variable B S.E. Wald df S ig. Exp(B) (Constant) 3.362 0.732 21.083 1 0.000 28.837 Initial SIP 3 value – 0.061 0.011 30.816 1 0.000 0.941 Large System Hospital 0.645 0.338 3.652 1 0.056 1.907 (membership in system with 10+ hospitals) Notes: N=19 and Nagelkerke R2=0.195 Table 26. ariables in the Equation: Decrease in Performance in SIP 3 Composite, 2006 – 07 V (Logistic Regression) Variable B S.E. Wald df S ig. Exp(B) (Constant) – 8.772 2.258 15.097 1 0.000 0.000 Initial SIP 3 Value 0.089 0.028 10.057 1 0.002 1.094 Pre-Tax Net Income – 0.032 0.015 4.720 1 0.030 0.969 (in millions of dollars) Notes: N=165 and Nagelkerke R2=0.316 Do Hospital Characteristics Drive Clinical Performance? An Analysis of Standardized Measures   |  31 C A L I FOR N I A H EALTH C ARE F OU NDATION 1438 Webster Street, Suite 400 Oakland, CA 94612 tel: 510.238.1040 fax: 510.238.1388 www.chcf.org