Evaluation of the Medi-Cal Plan/Practice Improvement Project: Executive Summary Prepared for California HealthCare Foundation Prepared by Rita M. Mangione-Smith, M.D., M.P.H., Department of Pediatrics, Child Health Institute, University of Washington and Clarissa Hsu, Ph.D., Center for Community Health and Evaluation, Group Health Cooperative January 2008 About the Authors Rita M. Mangione-Smith, M.D., M.P.H., department of pediatrics, Child Health Institute, Seattle WA. Mangione-Smith is associate professor of pediatrics and adjunct associate professor for the department of health services at the University of Washington. Her primary research interests are quality and appropriateness of care in pediatrics and the development of interventions to improve the care provided to children. Mangione- Smith has worked on several quality of care projects at University of California Los Angeles, RAND, and the National Committee for Quality Assurance (NCQA). Clarissa Hsu, Ph.D., senior program manager at the Center for Community Health and Evaluation, Group Health Cooperative, Seattle WA. Hsu is a medical anthropologist who holds a doctorate in sociocultural anthropology with specialized training and experience in qualitative research methods, research design, and participatory evaluation. Her evaluation and research projects include the Partnership for the Public’s Health (PPH) Initiative and an Ethnographic Study of Childbirth Education Classes. Hsu also is affiliate assistant professor at the University of Washington School of Public Health and Community Medicine. About the Foundation The California HealthCare Foundation, based in Oakland, is an independent philanthropy committed to improving California’s health care delivery and financing systems. Formed in 1996, our goal is to ensure that all Californians have access to affordable, quality health care. For more information about CHCF, visit us online at www.chcf.org. ©2008 California HealthCare Foundation Contents 4 I. Introduction A. P/PIP Background B. Evaluation Goal 5 II. Methods 6 III. Findings A. Summary of Key Findings B. Feedback on the Plan/Practice Improvement Model 10 IV. Lessons Learned, Recommendations, and Future Plans A. Relationship Building and Roles B. Overarching Design Issues C. Content and Logistics D. Insights and Lessons Learned 14 V. Summary I. Introduction A. P/PIP Background The Plan/Practice Improvement Project (P/PIP) was a new quality improvement collaborative focused on asthma care and funded by the California HealthCare Foundation (CHCF). P/PIP was implemented and led by a partnership of the Center for Health Care Strategies (CHCS), the Medi-Cal Managed Care Division (MMCD), the National Initiative for Children’s Healthcare Quality (NICHQ), and the Improving Chronic Illness Care (ICIC) program of the MacColl Institute for Healthcare Innovation. The major goal of the project was to help Medi-Cal health plans synchronize their quality improvement and chronic care management approaches at the health plan and provider levels. The 18-month collaborative’s overarching goal was a 50 percent reduction in emergency department use and hospital admissions for members with asthma in the practice intervention sites. This collaborative was innovative in using virtual learning sessions rather than traditional face-to-face meetings with quality improvement teams from participating health care organizations. This approach, if effective, would greatly enhance the financial feasibility of widely disseminating the collaborative approach to quality improvement. B. Evaluation Goal Our main objective was to evaluate the effectiveness of an innovative collaborative support infrastructure to improve asthma care at Medi-Cal practice sites. Specifically, we evaluated the advantages and disadvantages of several strategies. We address three main questions with this evaluation: n What were the pros and cons of key elements of the program’s design (e.g. recruitment strategies, methods of information transfer, materials, incentives, health plan support, methods of faculty-participant communication, etc.)? n What facilitators and barriers to implementing recommended quality improvement interventions did collaborative participants encounter? n What are the facilitators and barriers to successfully spreading the recommended quality improvement interventions to other practices?   |  C alifornia H ealth C are F oundation II. Methods The innovative nature of the P/PIP meant that standardized, quantitative data collection tools couldn’t fully capture the advantages, disadvantages, and lessons learned. A qualitative approach was chosen to provide descriptive data and fully explore these issues. Open-ended telephone interviews explored issues such as project design, recruitment, goals for participation, investment of resources, experiences with the virtual learning sessions (VLS) and spread collaboratives, changes made by practices and health plans, overarching recommendations, and feedback on the leadership organizations. An initial focus group was conducted with NICHQ staff to familiarize the evaluation team with the P/PIP and identify focus areas for the interviews. Table 1 summarizes the target number of respondents in each group and the actual number of respondents. To maximize the number of respondents without exceeding the resources provided for this evaluation, interviews were sometimes conducted with two respondents at a time. Table 1: Interview Sample Target Actual Leadership Team 7 7 Health Plans Medical Directors 8 6 Key Staff 8 8 Practices 14 10 Evaluation of the Medi-Cal Plan/Practice Improvement Project: Executive Summary   |  III. Findings A. Summary of Key Findings 1. Challenges in the Design Phase A major design change made the project difficult to implement. The initial idea for P/PIP was to teach health plans how to do practice site improvement and integrate the silos between practice-level quality improvement and health plan-level quality improvement. The plans would train and support the practices with help from the faculty. The full leadership team, once assembled, moved away from this core idea in favor of having practices and plans conduct separate but interrelated activities. The health plans’ role changed from one that was central to implementation of quality improvement activities to one that was more supportive in nature. 2. Practice Recruitment Issues Health plans were given few instructions regarding recruitment and neither health plans nor faculty had a clear understanding of California HealthCare Foundation’s strong interest in recruiting small practices. Health plans used a wide variety of approaches to recruit practice participants. Some identified high-volume providers or poor performers. Other plans engaged medical groups and independent practice associations to help recruit physicians. One health plan issued a request for applications. In general, most health plans identified potentially receptive practices and recruited them in person. The different recruitment techniques resulted in a set of participating practices that had different needs and expectations. Health plans that recruited poor performers found that these practices were the least ready and willing to engage in this type of project. The request for applications process was effective in identifying motivated early adopters. However, no small practices self-selected for participation. Some health plans worked very hard to recruit small practices, while others concentrated on practices they knew were receptive to quality improvement initiatives, regardless of size.   |  C alifornia H ealth C are F oundation The recruitment process also produced variable project overwhelming given the timeline and levels of understanding of the project. Practices that required too much work by the health plans and completed an application process felt less informed practices. Many reported they were just reaching about the work and time commitment than many of the point of being ready to spread changes at the the practices recruited through one-on-one meetings. project’s conclusion. The design for the spread component was also seen as impractical for large Several health plans said the project’s design was not health plans with contracted practices across well-suited to providers in small practices, which California. lacked the resources to participate. Overall, health plans that did not include small practices had a 5. Limited Degree of Change at the better experience with the project. Practice Level An important goal of P/PIP was changing actions 3. Virtual Learning Session Content at the practice level to produce successful models One of the strongest findings was that the health that could be spread to other practices. Some plans and practices all felt that the content of members of the leadership team were disappointed the virtual learning sessions was too theoretical with the degree of change. Also, a perceived lack and academic. Multiple respondents said it was of communication and engagement led to ongoing “overwhelming.” Participants wanted more practical questions about what had really happened in the information directly relevant to their clinical practices. practice. The practices valued concrete examples of how to make changes. The majority of changes came from two practices: Both are larger, community-based clinics with The key theoretical components of the P/PIP, the quality improvement experience. They are in a Plan-Do-Study-Act (PDSA) model and the chronic geographic region that has a number of other care model, received mixed responses. There was factors that support an increased focus on chronic very little uptake on the PDSA model. In some disease management and quality improvement in cases practices appeared to complete the PDSA asthma care. steps but did not realize or acknowledge that they had followed the model. Other practices felt that Most of the practices made changes during P/PIP. the model’s reporting materials were redundant However, several indicated that the changes were not and didn’t facilitate completion of the process. a direct result of their participation in the project, Participants valued the chronic care model and but instead were related to efforts under way before felt the presentation was good, but many wanted P/PIP or were changes they would have made to spend less time on it because they already regardless of their participation. Changes reported by understood and used the model. Several respondents multiple practices included: felt the chronic care model did not address the n Adoption of an asthma encounter form; practical barriers inherent in the everyday practice of n Use of asthma action plans; primary care medicine, especially for small practices. n Training in and increased use of spirometry; 4. Simultaneous VLS and Spread Collaboratives n Use of peak flow meters; The redesign meant that P/PIP had two major components that were implemented in parallel: n Increased use of controller medications; the virtual learning sessions collaborative focused n Flagging medical records for patients with asthma primarily on the practices, and the spread (with either a sticker or a flag on an electronic collaborative in which only the health plans medical record); and participated. Some of the faculty and health plan n Establishment of registries. medical directors felt that the redesign made the Evaluation of the Medi-Cal Plan/Practice Improvement Project: Executive Summary   |  Major facilitators to implementing changes asthma patients. The Plan/Practice model presented, were a high level of support from health plan particularly the spread component, was considered personnel, a physician champion for P/PIP in the unfeasible for health plans that cover practices practice, and the existence of regional activities to spanning large geographic regions or for large support improved asthma care and chronic disease health plans contracted with thousands of providers. management. Comments from these health plan representatives included: Major barriers to implementing changes at the practice level were a lack of time and resources and …when you start to work with [providers], you the difficulty of dealing with multiple health plans with different incentives and reporting requirements. start to realize all of the reasons why you are Most practices were unable to invest extra personnel not doing well, all the obstacles they encounter time in P/PIP, so the project activities were added every day and how they are different from the to existing workloads. This was especially true for next clinic…You also begin to learn what else small practices. The smaller practices interviewed often mentioned that there were major financial the health plan can do at a higher level to make disincentives for them to participate because time things easier for them. not spent seeing patients meant loss of revenue. n n n This is in contrast to salaried staff at the larger participating health centers. It’s hard in our area because we’re a small plan; B. Feedback on the Plan/Practice we’re geographically large so it’s a four-hour drive Improvement Model from one edge to the other edge of our service P/PIP participants had positive things to say about area. To have staff to actually be able to drive a the overarching goal of better integrating quality couple of hours to an office and do an interven- improvement work between the health plans and the tion and drive back is very time-consuming. practices. Participants also had concerns about the model that may have affected the manner in which it n n n was implemented. I think we learned a lot from our practice sites. The health plan participants cited many advantages to working with practices on quality improvement, I think we definitely learned how to approach including the chance to build relationships with them. We definitely learned it’s not a one-size-fits- contracted providers, learning the barriers and all program. And we have to figure out what it is challenges practices face when trying to do quality they want to accomplish, not what our goal is. improvement work, and learning what health plans can do to facilitate positive changes at the practice Several practice participants felt the quality level. One disadvantage is that many practices improvement resources provided by health plans contract with multiple health plans. If one health were very helpful. These resources included IT plan works on quality improvement at a practice, support, financial remuneration, and staff to assist other plans also contracted with that practice reap with data collection and entry. They were glad the the benefits of the health plan’s investment of time health plans had the opportunity to learn about and resources. The multiple contracts issue also can services that should be reimbursed, such as asthma result in conflicting messages from different health education, longer visits, and spirometry. A few plans regarding management of a practice’s   |  C alifornia H ealth C are F oundation providers were skeptical of the health plans’ motives for supporting practice-based quality improvement and suggested they were doing this primarily to improve their profit margin. …getting health plans who are our funders to understand the importance of these changes and reimburse them appropriately is very important. For example, they should allow us to bill for group asthma education visits. n n n I have to say we looked upon them with a great deal of suspicion as to what their motives were. A number of faculty respondents felt it was imperative that quality improvement efforts go beyond the health plan level to also include providers. In their view, P/PIP allowed them to begin to move in the right direction and they need to keep trying until they get it right. They would like to work toward identifying quality improvement strategies that work for practices of all sizes, i.e. “population-based quality improvement.” They also will try to work with larger provider entities such as medical groups or independent practice associations. Finally, they have plans to leverage support from other organizations such as medical societies and boards of medicine where quality improvement activities may be required for physician recertification. I think my one comment is I’d like to do it again…I think we learned so much from this that I think we could do it the right way this time. Evaluation of the Medi-Cal Plan/Practice Improvement Project: Executive Summary   |  IV. Lessons Learned, Recommendations, and Future Plans The majority of participants had positive feedback on one or more components of P/PIP. Most health plan participants said the organizations involved had done an excellent job and they would certainly consider participating in a similar project again. Practices, however, were more mixed in their reviews of the project and willingness to participate in a similar project in the future. In general, all member organizations of the leadership team felt this project was an excellent learning experience, and most of the faculty would collaborate again with team members from other organizations. However, P/PIP faces many challenges. Some key challenges and areas with recommendations for improvement are: relationship building and roles, overarching design issues, and collaborative content and logistics. A. Relationship Building and Roles Establish clear roles for all organizations involved in implementation. Clarifying roles was problematic for all key leadership organizations, starting with the role of the funder in influencing the project’s leadership and design. The directive role taken during the proposal and early funding phase created a need for ongoing involvement and guidance. The multiple organizations involved could have been more effective had roles been clearly outlined. Key to this was the need to identify a clear project leader and empower that leader with decision-making authority. Clarification and upfront negotiation also were needed regarding specific organizational agendas, responsibilities, and procedures to facilitate collaborative work. Issues such as data sharing, transparency of processes, flexibility of approaches, and organization culture and perspectives were seen as barriers to successful implementation of the project. The Medi-Cal Managed Care Division’s varying level of involvement and leadership was viewed as a missed opportunity. MMCD could have increased health plan accountability and follow-through, and functioned as the health plan advocate when inter-organizational conflicts arose. Allot extra time in the design phase when orchestrating collaborations among organizations that have no experience jointly implementing projects. Funders should be careful when orchestrating a partnership between two organizations. Extra time and effort should be given to developing mutual trust and 10  |  C alifornia H ealth C are F oundation understanding. One in-person meeting is probably Conducting the VLS and spread collaboratives not sufficient to build the relationships needed simultaneously was counterproductive. Although to jointly implement a complex project. A longer there was interest in fostering rapid spread of preliminary planning period is needed to clearly positive changes in asthma care, it is clear that define project goals and the roles of leadership team simultaneous collaboratives were overwhelming for members. health plan participants and did not take advantage of the lessons learned from the practice sites. Spread Build broader shared ownership in the project activities might have been more focused and effective from all stakeholders, including health plans. had they followed the asthma learning collaborative. Securing health plan “buy-in” from the very start is critical, especially if the plans are going to play a Develop clear practice site recruitment major role in implementation and ongoing support. procedures. Health plans should receive clear A number of respondents felt the health plans had guidance regarding recruitment methods and not been appropriately involved in the redesign criteria. A more uniform recruitment procedure and implementation of the intervention. The may have provided more shared understanding of evaluation also identified underlying disagreements expectations and the types of practices to recruit. about the appropriate role of health plans in quality Because incentives for participation varied greatly it improvement that undermined the plans’ ability to was difficult to assess the role incentives played in fully contribute. the overall commitment and success of practice site activities. Health plan data may be useful to identify B. Overarching Design Issues high-volume providers, which are more likely to be Expectations need to be communicated clearly at interested in participating. Such data should not be the beginning and should not be changed after used to identify poor performers, which typically are the project has launched. Changes in program the least ready to embark on a quality improvement requirements (such as recruitment goals and overall project. Several health plans found that recruiting redesign) after implementation created challenges at physicians face-to-face (at health plan-sponsored all levels. Funders need to ensure that expectations continuing medical education events or visits to are clearly communicated and documented with all their offices, for example) was the most effective member organizations. recruitment approach. Be flexible and maintain a willingness to Assess practice site capacities during recruitment. tailor approaches, activities, and materials It is important to determine if practices have the to participants’ needs. Avoid a one-size-fits-all necessary resources, such as IT capabilities, to approach. Find a balance between accommodating participate in a Web-based collaborative before they expectations of the health plans or practices and are recruited. This is especially important with small holding them accountable and maintaining the practices. Performing a practice needs assessment and integrity of the intervention. tailoring the quality improvement intervention based Design the intervention to accommodate practical on these needs probably would be far more effective. limitations for practice site participation. The Provide closure for practice participants. Several timeline for submitting reports was unrealistic for practice participants indicated they felt the project people running a full-time practice. Consider the lacked closure; some were unaware the project had budgetary implications of doing this type of project ended. A final project meeting where practices can at the practice level and adjust the time commitment share successes and future plans would be one way to and workload accordingly. Smaller practices felt the provide closure. program would be much more relevant to them if the faculty did some groundwork to determine what is and isn’t possible in small practice settings. Evaluation of the Medi-Cal Plan/Practice Improvement Project: Executive Summary   |  11 C. Content and Logistics Acknowledge and accommodate for the fact that Clarify project leadership and line of commu- quality improvement is a slow process at the nication. Participants found the complexity of the practice level. Practices, especially small practices, project leadership confusing. They felt it would have have a full-time job managing day-to-day operations. been easier if fewer organizations were involved. It While they may value quality improvement, was often difficult to determine who was in charge additional resources and time are needed to make and to whom they should direct their questions. such activities possible. Change often needs to happen in small steps that take time to manifest in Develop concrete, practical content. The content concrete process or systems changes. of both collaboratives was too theoretical and not practical enough. Only a small amount of time needs Identify a physician champion. Having a physician to be spent on theoretical concepts; after that, both champion who is deeply committed to quality health plans and practices want concrete examples improvement is key to the project’s success at the of specific changes to make and evidence that these practice level. changes would contribute to improving health Develop mechanisms for promoting sustainability outcomes. Participants were particularly interested in of positive changes. Given the clinical demands the effectiveness of this type of quality improvement practices face, it is essential to identify mechanisms work in terms of both patient outcomes and the for promoting sustainability. Key barriers to financial bottom line for health plans and providers. sustainability are the inability to engage all clinicians The change package should be smaller and focus on and staff in quality improvement efforts and high-impact items. Although the faculty eventually staff turnover. Mechanisms that might improve responded to this feedback, there was a feeling that sustainability include: they could have responded more quickly. n Processes that involve all staff in needs assessments Use Web-based conferencing strategically. The and decisions regarding changes; Web-based format for the collaborative workshops n Infrastructure to support training programs for received mixed reviews. Most participants felt the new staff (to do asthma education or to perform format was an asset to the project; however, the in-office spirometry, for example); and format also frustrated faculty and participants. When n Continued support and accountability through there is a need for community building, one face-to- face meeting for all participants would help establish health plan or state policies (such as support personal relationships and mitigate communication and development of disease-specific registries for problems. Also, online content needs to be carefully Medi-Cal recipients statewide, or institution of tailored so that it is high-impact. The virtual learning report card measures at the practice, independent sessions should be shorter (no more than three to practice association, or medical group level four hours). to assess progress and the degree to which improvements are sustained). D. Insights and Lessons Learned Promote partnership and resource sharing Do assessment work before developing among health plans. Health plans that cover the interventions for small practices. If there is a desire same practices should be encouraged to collaborate to do future quality improvement work with small and share resources. Partnering has a number of practices, the faculty should consider doing some advantages including capturing a larger share of the formative work (such as focus groups) to determine practices’ patients for the project, reinforcing the how the program’s structure should be modified to benefits of a culture of quality improvement, and be successful. encouraging health plan innovation that has the potential for systems change. 12  |  C alifornia H ealth C are F oundation Provide high-level infrastructure support. Small care interventions that probably had more to do practices have few resources to invest in quality with improved outcomes than did P/PIP. The area improvement and lack economies of scale for had reached a tipping point at which the cumulative investment in electronic patient records and data- impact of multiple factors coincided with P/PIP. tracking technologies. One example of higher-level Also, focusing interventions in geographic regions support mentioned by respondents was working that have multiple chronic care management projects with independent practice associations and medical occurring simultaneously may create important groups to facilitate registry building, data collection, synergies that allow for greater levels of change, in and quality improvement efforts. Government both organizational processes and health outcomes. entities also have the potential to provide this type of support. Support development of registries. Registries are critical to quality improvement for chronic conditions. To study a population of patients and track quality improvement outcomes requires that the population first be identified using uniform criteria. Until providers have registries for their patients, the ability to carry out Plan-Do-Study-Act cycles is limited. Registries are sometimes feasible for larger community clinics to develop internally, but for many small practices the only viable option will be registry development supported by health plans or the state. Advocate for change in state Medicaid policies. The state Medicaid offices have the potential to provide a level of accountability and standardization that could greatly enhance quality improvement efforts, especially in support of infrastructure such as registries, best practice guidelines, performance measurement, and reporting. Be cautious about linking health plan outcome data to focused, site-limited interventions. Goals and measures should be realistic and scaled to the intervention; otherwise they do not validly measure the effects of the intervention. For interventions that target a limited number of practices within a region, goals and outcome measures should be limited to the participating populations. Account for the influence of environmental and/or contextual factors. In San Francisco, where health plans saw a significant improvement in emergency room and hospitalization outcomes, there were several concurrent asthma and chronic Evaluation of the Medi-Cal Plan/Practice Improvement Project: Executive Summary   |  13 V. Summary P/PIP was an innovative intervention that has much to contribute to our understanding of improving chronic care management in primary care settings. The findings of this evaluation highlight a number of strengths, challenges, and lessons learned. This project revealed the importance of spending adequate time to set goals, establish clear roles, and develop an implementation plan acceptable to all participants. It also revealed the need for formative work to determine how best to approach quality improvement and implementation of the chronic care model in small practices. This will be of critical importance if such quality improvement efforts are to be broadly disseminated among U.S. health care providers since most care occurs in these small, under-resourced practices. The virtual learning sessions received mixed reviews, but this was primarily due to the content, not the format. Using Web- based conferencing to facilitate wide dissemination of quality improvement programs should be explored further. However, the content and presentation need to be improved. It is not clear that health plans are the best leverage point for supporting quality improvement activities at the practice level. Their motives were met with skepticism among a handful of providers, and the issue of providers contracted with multiple health plans can be problematic. It is possible that the Medi-Cal Managed Care Division, medical societies, and medical boards may be more positively viewed as supporters of quality improvement efforts; however, these groups are unlikely to be able to offer the level of resources and hands-on support provided by many health plans in this project. In most cases P/PIP was effective in fostering improved relationships between health plans and their practice participants. Most practices implemented changes in their approach to the management and treatment of asthma patients, but these changes fell short of true implementation of the chronic care model. It is also unclear whether the changes will be widely spread or sustained. However, this concern can be generalized to most quality improvement activities. Future work needs to focus on how to best enhance the spread and sustainability of improvements once they have been made. This in turn will require better understanding of what aspects of an organization’s culture are key to sustaining positive changes. 14  |  C alifornia H ealth C are F oundation