Evidence-Based State Health Policymaking: Researcher and Policymaker Perspectives June 2016 By Ellen O’Brien and Enrique Martinez-Vidal Evidence-Based State Health Policymaking: Researcher and Policymaker Perspectives Introduction Methods In the current environment, states face growing responsibilities for In August 2014, researchers from AcademyHealth fielded two health policy decision making, increasing complexity in the health web-based surveys—one of state government officials and one of care system, and heightened expectations for delivering improved researchers working on state policy issues—for the purpose of as- quality of care at better value. States have made innumerable policy sessing the role of research in guiding state health policy decision decisions related to implementation of the Affordable Care Act making. We identified potential survey respondents4 by drawing (ACA), and they also have developed Marketplaces, streamlined eli- from mailing lists developed by AcademyHealth, a not-for-profit gibility systems, and supported innovations in health care payment membership organization dedicated to improving health and the and delivery. As their responsibilities and influence have expanded, performance of the health system by supporting the production state policymakers have sought out information and analyses in and use of evidence to inform policy and practice. support of essential policy decisions. Aided by substantial federal funding for ACA implementation, states also have been able to en- The survey sought to collect information about the quality of the hance in-house data and analytic capacity, and they have contracted evidence base, the level and quality of communication between with outside vendors for those analytic services, including consult- researchers and policymakers, and strategies that offer the poten- ing firms and academically-based researchers. Some states rely on tial to close the gap between research and policy, thereby better policy intermediaries that serve a bridging and translating function aligning the needs and interests of researchers and policymakers. between the research and policy communities.1 We asked researchers outside state government to describe their research and its relevance to state health policy. More specifically, In order to inform AcademyHealth’s ongoing support of state re- we asked them about factors that facilitate evidence-informed searchers and policymakers, we sought to explore the general con- policymaking in states and inquired about strategies that might tours of the state policy landscape through surveys administered to help expedite better research and better dissemination of research researchers and policymakers. The goal of the project was to gather findings and, ultimately, more evidence-informed policymaking. perceptions of evidence-based health policymaking in the states, Similarly, we surveyed state officials for their views on the volume, to identify potential strategies for generating useful and relevant quality, and timeliness of research and on the factors that might information that facilitate evidence-based decision making, and to make research more useful for decision making. determine how best to support moving that evidence into action. Findings A general conceptual framework of how knowledge is produced n Survey respondents. Respondents to the survey totaled 79 and used in policymaking guided the design of the survey. Our state health policy researchers and 138 policymakers. Universi- starting assumption was that research knowledge flows between ties employed most of the researchers who responded to the defined groups of evidence users (policymakers) and evidence survey (63 percent); research or consulting firms employed 16 producers (researchers) and that the groups use the knowledge in percent of respondents. Among the policymakers, the major- various ways. Often the groups encounter barriers to the use of ity (88 percent) was employed in the executive branch of state information in policymaking. The barriers may be on either the government. Only 6 percent were employed in the legislative supply side or the demand side. On the supply side, the lack of branch. We sent surveys to state officials in both the executive relevant and timely research may impede policymaking. On the and legislative branches, including senior executives as well as demand side, state health policy and program staff lack of training mid-level policy and research professionals. Most respondents and skills in the use of data and research knowledge may ham- (69 percent) were mid-level agency staff (program direc- per decision making.2 At the same time, facilitators that expedite tors and program managers) or lower-level program staff (24 the use of evidence may include highly targeted research prod- percent); a relatively small proportion (7 percent) comprised ucts and messaging and the engagement of effective third-party high-level state executives (e.g., secretary, deputy secretary, and intermediaries in a bridging and translating function between Medicaid director). The policymaker respondents represented evidence producers and evidence users. the District of Columbia and 45 states. We set out to enhance our understanding of the state health poli- n Producing and using research at different stages of policy cymaking environment in order to gain insight into the strategies development. We asked researchers to assess the relevance of that could potentially “create cultural shifts that can facilitate the their research at five stages of the policy development pro- ongoing use of research knowledge in decision making” at the cess: (1) understanding the scope and extent of the problem; state level.3 2 Evidence-Based State Health Policymaking: Researcher and Policymaker Perspectives (2) developing policy options; (3) implementing reforms; (4) management, that is, the stages at which policymakers say they day-to-day program management; and (5) policy and program use evidence. Researchers, however, are less likely to focus their evaluation.5 Some 80 percent of researchers reported that their efforts on those policy development stages. research was extremely relevant at stage 1 (Figure 1). More than half reported that their research activities were highly n Communication channels.6 To assess the communication relevant at the stage of program or policy evaluation (stage 5). channels used by researchers and state policymakers, we asked A relatively small percentage reported that their research was researchers about the strategies they use to communicate their highly relevant for purposes of day-to-day program manage- research findings and guide policy development. We asked ment. Policymakers reported that they always or frequently use them to indicate whether they used each of eight communi- evidence to understand the scope of a problem (83 percent), cation strategies, including direct outreach to policymakers, identify policy options (82 percent), assist with policy imple- traditional publication and media routes, and social media.7 mentation (80 percent), assist with day-to-day program man- Researchers most frequently reported that they pursue direct agement (60 percent), and evaluate policy and program perfor- outreach to state policy decision makers and prefer to publish mance (71 percent) (Figure 2). The findings suggest a possible policy briefs. Only a handful of survey respondents reported gap between the demand for and supply of research evidence in that they rely on social media to disseminate research findings the intermediate stages of policy implementation and program (Figure 3). Figure1.Researchers’AssessmentsoftheRelevanceofTheirResearchtoFive Figure1.Researchers’AssessmentsoftheRelevanceofTheirResearchtoFive StagesofPolicyDevelopment StagesofPolicyDevelopment VeryRelevant VeryRelevant SomewhatRelevant SomewhatRelevant NotatAllRelevant NotatAllRelevant NoResponse NoResponse 1% 1% Stage1:Understandingthescopeandextentofthe Stage1:Understandingthescopeandextentofthe 80% 80% 14% 5% 14% 5% problem problem Stage2:Developingpolicyoptions Stage2:Developingpolicyoptions 49% 49% 30% 30% 6% 14% 6% 14% Stage3:Implementingreforms Stage3:Implementingreforms 39% 39% 34% 34% 19% 8% 19% 8% Stage4:Day-to-dayprogrammanagement Stage4:Day-to-dayprogrammanagement 14% 14% 34% 34% 41% 41% 11% 11% Stage5:Policyandprogramevaluation Stage5:Policyandprogramevaluation 54% 54% 37% 37% 5%5% 4% 4% Figure2.StateUsers’AssessmentsofTheirUseofResearchEvidence Figure2.StateUsers’AssessmentsofTheirUseofResearchEvidence atFiveStagesofPolicyDevelopment atFiveStagesofPolicyDevelopment Always/Frequently Sometimes Rarely/Never NoResponse Always/Frequently Sometimes Rarely/Never NoResponse 1% Stage1:UnderstandingtheScopeandExtentofthe 1% Stage1:UnderstandingtheScopeandExtentofthe 83% 15% 1% Problem 83% 15% 1% Problem 2% 2% 2% 2% Stage2:Developingpolicyoptions 82% 14% Stage2:Developingpolicyoptions 82% 14% Stage3:Implementingreforms 80% 15% 5% Stage3:Implementingreforms 80% 15% 5% Stage4:Day-to-dayprogrammanagement 60% 28% 12% Stage4:Day-to-dayprogrammanagement 60% 28% 12% Stage5:Policyandprogramevaluation 71% 22% 6% Stage5:Policyandprogramevaluation 71% 22% 6% 0% 50% 100% 0% 50% 100% 3 Evidence-Based State Health Policymaking: Researcher and Policymaker Perspectives Figure3.MethodsUsedbyResearcherstoDisseminateFindingstoPolicy Audiences Figure3.MethodsUsedbyResearcherstoDisseminateFindingstoPolicyAudiences Directoutreachtostatepolicymakers 66% Directoutreachtostatepolicymakers 66% Publishedapolicybrief 56% Publishedapolicybrief 56% Interviewedbyareporter 51% Interviewedbyareporter 51% Newsmediacoveredapublishedstudy 42% Newsmediacoveredapublishedstudy 42% Directlycontactedbyaninterestgroup 29% Directlycontactedbyaninterestgroup 29% Bloggedaboutresearchorpolicy Bloggedaboutresearchorpolicy 16% 16% UsedFacebookorTwittertopromoteresearch UsedFacebookorTwittertopromoteresearch 14% 14% Publishededitorialwithpolicyrecommendations Publishededitorialwithpolicyrecommendations 13% 13% 0% 50% 100% Figure4.SourcesofandMethodsforObtainingPolicy-RelevantResearch Figure4.SourcesofandMethodsforObtainingPolicy-RelevantResearch Frequently Frequently Sometimes Sometimes Healthpolicybriefs/reports Healthpolicybriefs/reports 68%68% 28% 28% Healthpolicyjournals Healthpolicyjournals 60%60% 30% 30% Internalstatedataandresearchdepartments Internalstatedataandresearchdepartments 54%54% 40% 40% Webinars Webinars 46%46% 46% 46% Emails/newslettersofevidenceproducers(e.g.,think … Emails/newslettersofevidenceproducers(e.g.,think … 41%41% 35% 35% Emails/newsletterofevidencedisseminators Emails/newsletterofevidencedisseminators 37%37% 43% 43% Media Media 34%34% 38% 38% Stateorlocaladvocacygroups 28% 51% Stateorlocaladvocacygroups 28% 51% In-stateconferencesandmeetings 20% 47% In-stateconferencesandmeetings 20% 47% Workshops(withpeersfromotherstates) 19% 42% Workshops(withpeersfromotherstates) Out-of-stateconferencesandmeetings 19%19% 42% 39% Directcontactwithlocalresearchers 19% Out-of-stateconferencesandmeetings 18% 39%41% Directcontactwithlocalresearchers Nationaladvocacygroups 18%16% 41% 41% Directcontactwithnonlocalresearchers 16% Nationaladvocacygroups 12% 41% 33% Directcontactwithnonlocalresearchers 12% Socialmedia(e.g.,blogs) 7% 20% 33% Socialmedia(e.g.,blogs) 7% 20% 0% 100% 0% 100% We also asked researchers about their consulting and contract re- the media, health policy journals, health policy briefs and reports, search for state health agencies and whether they have testified be- webinars, out-of-state and in-state conferences, workshops with fore state legislatures. A large percentage (80 percent) reported that other state officials, state or local advocacy groups, national advoca- they regularly or occasionally consult for state government agencies cy groups, direct contact with researchers, and social media, among through either contracted research or technical assistance activities. others (Figure 4). The majority reported frequent use of traditional A much smaller percentage said that they regularly or occasionally sources such as health policy briefs and reports (68 percent), testify before a state legislature (35 percent) (data not shown). health policy journals (60 percent), and internal data and research departments (54 percent). Just 7 percent of state officials reported We asked state officials about their relationships with evidence that they frequently relied on social media to obtain policy-relevant producers and about the sources and methods they use for identi- research. Researchers and policymakers alike depend on traditional fying relevant research. The majority of state officials responding modes of dissemination, such as direct outreach as well as policy to the survey reported that they had established relationships with briefs and journals. researchers, including internal (state agency) policy and research departments (77 percent), university-based researchers in their state (63 percent), nonuniversity-based researchers in their state (56 n Evidence-based policymaking. Barriers and facilitators. percent), and researchers outside their state (68 percent). Next, we asked researchers about their perceptions of the policymaking process in their state of residence (or the state To assess how evidence users acquire information, we asked with which they are most familiar). We inquired about barriers respondents to describe the frequency with which they turned to that undermine evidence-based decision making and about various sources of or methods for receiving information, including factors that facilitate evidence-informed decision making. 4 Evidence-Based State Health Policymaking: Researcher and Policymaker Perspectives Figure 5. Researchers' Perceptions of Barriers to Evidence-Informed Policymaking Very Important Somewhat Important Not at All Important Don't know/Not sure No Response Politics/Ideology: Evidence users don't see the 3% 57% 23% 11% 6% need for the kind of evidence being produced 1% Evidence Base: Insufficient evidence base (in 48% 44% 6% terms of quality, quantity, timeliness of research) 3% Expertise: Inadequate agency staff-level expertise 57% 27% 8% 6% 1% Financial resources: Limited resources at the state 56% 32% 5% 6% level to pay for research Barriers. We asked researchers for their assessments of the Facilitators. We also asked researchers and policymakers to assess following barriers to evidence-informed decision making in their the importance of potential facilitators of evidence-informed state: the state’s capacity to fund program evaluations and other policymaking. In our survey of researchers, we asked about research; the expertise of state agency staff; the quality of the facilitators such as collaboration in the policymaking environment, evidence base; and political barriers. We described the political expertise among state agency staff, expertise in the state legislature, barriers as follows: “Decision makers don’t see the need for the kind and the presence of effective, third-party policy intermediaries. of evidence being produced; it does not fit within the ideological We also asked about consumer and provider associations, the role framework for program development.” Researchers rated all of the of the media, and the importance of staff turnover in executive above factors as very or somewhat important and, in particular, branch agencies and legislative staff. A majority of researchers cited acknowledged weaknesses in the evidence base for policy decision the following facilitators as very important: a collaborative policy making, with 92 percent deeming such weaknesses a very environment, high levels of expertise in the executive and legislative important or somewhat important barrier (Figure 5). branches, and effective policy intermediaries (Figure 7). Among evidence users, nearly three-quarters cited the expertise of state We asked evidence producers to assess the willingness and agency staff as a very important facilitator of evidence-informed capacity of decision makers in the executive branch and in the policymaking. Evidence users also pointed to the importance of legislature to (1) understand and use research; (2) to produce a collaborative policymaking environment and the role of policy research (in internal policy research offices, for example); and (3) intermediaries (Figure 8). to fund outside researchers at universities or consulting firms. With respect to using and understanding research, evidence n Strategies. Finally, we sought to collect evidence producers’ and producers gave the highest ratings to executive branch decision evidence users’ perceptions of the usefulness of various strategies to makers, with 72 percent rating their ability as high or medium. facilitate evidence-based policymaking. We identified nine strategy By comparison, researchers gave lower ratings to legislative options: (1) more state funding for internal research within state branch policymakers, with only 55 of respondents rating their agencies; (2) more state funding for external research; (3) improve- ability to use and understand research as high or medium. ments to data access for external researchers; (4) a “research priori- ties” network that would bring together external researchers with We also asked evidence users to rate the importance of certain state government policy and research staff to strategize and identify reasons that might help explain why research evidence is not research priorities; (5) policy analysis repositories that afford used — reasons related to the perceived quality and availability evidence users easy access to evidence on current policy topics; (6) of research findings. We identified factors that might account policy analysis training for state government researchers and deci- for the failure to use available evidence to inform policy sion makers; (7) media and translation training for researchers; (8) development, and we asked state policy officials to assess the training related to drafting and presenting legislative testimony for factors’ importance. The most frequently cited factors were the researchers; and (9) an outreach network that would bring together lack of timeliness and relevance of research findings (Figure 6). researchers with state officials, consumer advocates, providers, and others to share new research findings and discuss their implications for state policy decisions. 5 1% Financialresources:Limitedresourcesatthestate Evidence-Based State Health Policymaking: Researcher and Policymaker Perspectives 56% 32% 5% 6% leveltopayforresearch Figure6.ReasonsStateAgencyStaffReportedNotUsingAvailableEvidence Nottimely 48% Notrelevant 41% Tootheoretical/technical 31% Notobjective 28% Toomuchvolume/overwhelming 17% Toodense/detailed 14% Don'ttrustit 9% Don'tknowhowtofindevidence 9% Don'tknowhowtouseevidence 1% 0% Percentwhoreportedbarrier 100% Figure7.Researchers’AssessmentsofFactorswithinTheirStateThatFacilitate Evidence-InformedPolicyDecisionMaking VeryImportant SomewhatImportant Highlycollaborativepolicymakingenvironment 63% 24% Highlevelofexpertiseinstateagencies 57% 29% Veryeffectivepolicyintermediaries 53% 29% Highlevelofexpertiseinlegislative … 52% 32% Veryeffectivelocal(state-based)researchers 46% 38% Sophisticatedandeffectiveconsumerand … 37% 46% Lowturnoverinstateagencies 25% 61% Highlyattunedmedia 25% 51% Lowturnoveramonglegislativestaff 19% 59% 0% 100% Evidence producers reported that several of the above strate- and policy decision makers rated traditional policy briefs and Figure8.Users’AssessmentsofFactorswithinTheirStateThatFacilitate gies could be very important for improving a state’s policy de- systematic reviews very highly, with 55 percent agreeing that Evidence-InformedPolicyDecisionMaking be very important in improving evidence velopment process. A substantial proportion of researchers said these resources would that it would be very important to improve state funding for use and facilitating evidence-informed policymaking. Other research (63 percent) and data access (73 percent). A majority strategies and resources that received high rankings included a VeryImportant SomewhatImportant of the researchers responding to the survey (62 percent) also one-stop online clearinghouse for policy-relevant research and reported that it would be very important to convene a research reports on policies and performance of peer states (Figure 10). Highlevelofexpertiseinstateagencies 74% 17% priorities network (Figure 9). Highlycollaborativepolicymakingenvironment Study implications 51% 30% We asked state policymakers to assess the usefulness of a similar Veryeffectivepolicyintermediaries 48% 28% The survey findings point to strong demand among state poli- set of strategies (though the survey question was worded slightly cymakers for information and evidence in support of policy de- Veryeffectivelocal(state-based)researchers 40% 33% differently than in the survey sent to researchers). With regard velopment while researchers report that most state policymakers to the strategies thatHighlevelofexpertiseinlegislative … facilitate evidence use, state agency staff are38% 39% equipped to use research in policy development. At the same Lowturnoverinstateagencies 33% 45% 6 Sophisticatedandeffectiveconsumerandprovider … 28% 51% Evidence-Based State Health Policymaking: Researcher and Policymaker Perspectives Figure7.Researchers’AssessmentsofFactorswithinTheirStateThatFacilitate Evidence-InformedPolicyDecisionMaking VeryImportant SomewhatImportant Highlycollaborativepolicymakingenvironment 63% 24% Highlevelofexpertiseinstateagencies 57% 29% Veryeffectivepolicyintermediaries 53% 29% Highlevelofexpertiseinlegislative … 52% 32% Veryeffectivelocal(state-based)researchers 46% 38% Sophisticatedandeffectiveconsumerand … 37% 46% Lowturnoverinstateagencies 25% 61% Highlyattunedmedia 25% 51% Lowturnoveramonglegislativestaff 19% 59% 0% 100% Figure8.Users’AssessmentsofFactorswithinTheirStateThatFacilitate Evidence-InformedPolicyDecisionMaking VeryImportant SomewhatImportant Highlevelofexpertiseinstateagencies 74% 17% Highlycollaborativepolicymakingenvironment 51% 30% Veryeffectivepolicyintermediaries 48% 28% Veryeffectivelocal(state-based)researchers 40% 33% Highlevelofexpertiseinlegislative … 38% 39% Lowturnoverinstateagencies 33% 45% Sophisticatedandeffectiveconsumerandprovider … 28% 51% Highlyattunedmedia 17% 42% Lowturnoveramonglegislativestaff 12% 49% 0% 100% time, a variety of barriers, particularly gaps in data and data ac- policy analysis and program evaluation activities. Support for cess and in the quality of the communication between researchers a sophisticated data infrastructure for state agencies and state and policymakers, impedes evidence-based policy development. researchers could help address some concerns with the quality The findings suggest three strategies for improvement: and timeliness of evidence. n Strengthening state data and analytics. The survey responses n Maintaining a policy research clearinghouse. State policy- underscore the importance of wide-ranging and relevant state- makers and program officials suggested that improved access level data collection in parallel with executive branch agen- to information can expedite the use of research in policy de- cies’ enhanced analytic capability and expertise. Accordingly, velopment. Central data repositories that are carefully curated one strategy is to promote evidence-based policymaking by and maintained can help state decision makers obtain the evi- helping states strengthen their data infrastructure and analytic dence they need when they need it. These repositories need to capacity to conduct research while generously funding state include systematic evidence reviews that policymakers say they 7 Evidence-Based State Health Policymaking: Researcher and Policymaker Perspectives Figure9.Researchers'AssessmentsofOptionsforImprovingtheStatePolicy DevelopmentProcess VeryImportant SomewhatImportant Improvementstodataaccess 73% 19% Morestatefundingforexternalresearch 63% 28% Aresearchprioritiesnetwork 62% 25% Anoutreachnetworktosharenewresearchfindings 59% 24% Policyanalysisrepositories 52% 35% Policyanalysistrainingforstategovernment … 41% 48% Mediaandresearchtranslationtrainingfor … 41% 46% Morestatefundingforinternalresearch 34% 52% Legislativetestimonytraining 32% 47% 0% 50% 100% Figure10.Users'AssessmentsofStrategiestoImproveEvidenceUse VeryImportant SomewhatImportant SystematicReviews 55% 27% Policyanalysisbriefs 55% 22% One-stopshopping/onlineclearinghouse 51% 25% Reportsonpeerstates 46% 24% Onlinedirectoryofresearchers 46% 19% Reportsongeographicallyproximatestates 38% 30% Journalarticles 38% 28% Reportson"leadingstates" 38% 3% Buildingcapacityamongstateofficialstouse… 37% 33% Rapid-responseservice 33% 25% Peer-to-peermeetings 32% 29% Webinars 29% 33% Deliberativedialogues/collaborativemeetings 25% 36% 0% 100% need, as well as comparative assessments of state performance that is useful for policy development. Creating opportunities for across multiple domains. In particular, research repositories ongoing collaboration can help researchers understand emerging must maintain an inventory of briefs that provide context for issues and frame their research strategies. When decision makers research findings. Survey respondents were keenly aware that come together with evidence producers at the outset of a research individual studies are most relevant when considered in the project, they can help shape the questions to be asked. For their context of a body of evidence. To be most effective, however, part, researchers will be better prepared to translate their results repositories must devote attention to research translation and into findings comprehensible to policy audiences. to identifying the policy implications of research. Strategies such as these are not new, but our survey results sug- n Establishing mechanisms for ongoing communication. Both re- gest that they deserve renewed attention and investment. These searchers and policymakers pointed to the need for better commu- strategies may help increase the relevance of research and bring nication and collaboration as a prerequisite for generating evidence evidence to bear more effectively on state policy discussions. 8 Evidence-Based State Health Policymaking: Researcher and Policymaker Perspectives About the Authors 4. Using AcademyHealth mailing lists, we identified the names and email addresses of 2,921 state agency officials and 1,575 researchers. We sent the lists of potential Ellen O’Brien was a Senior Fellow at AcademyHealth from 2014- survey respondents an email message with a brief description of our project and the goals of the survey and included a link to an electronic survey. A large 2015. Enrique Martinez-Vidal is Vice President for State Policy and percentage of the email messages were delivered to in-boxes but went unopened. Technical Assistance at AcademyHealth. Of the email messages that were opened, roughly 20 percent of potential survey respondents completed the survey. Endnotes 5 Meyer, J.A., T.T. Alteras, and K.B. Adams. “Toward More Effective Use of Research in State Health Policymaking.” The Commonwealth Fund, December 1. AcademyHealth, unpublished case studies. See also Coburn, A. “The Role of 2006. Available at http://www.commonwealthfund.org/~/media/files/publica- Health Services Research in Developing State Health Policy,” Health Affairs, tions/fund-report/2006/dec/toward-more-effective-use-of-research-in-state- Vol. 17, No. 1, 1998, pp.139-51.Available at http://content.healthaffairs.org/ policymaking/meyer_towardmoreeffectiveusestatepolicymaking_980-pdf. content/17/1/139. 6 Sorian R, Baugh T. “Power of Information: Closing the Gap between Research 2. Soumerai, S.B., D. Ross-Degnan, E.E. Fortess, and B.L. Walser. “Determinants of and Policy.” Health Affairs. 2002;21(2):264-273. Change in Medicaid Pharmaceutical Cost Sharing: Does Evidence Affect Policy?” Milbank Quarterly, Vol. 75, No. 1, 1997, pp. 11-34. 7 Grande D, Gollust SE, Pany M, Seymour J, Goss A, Kilaru A, Meisel Z. “Trans- lating Research for Health Policy: Researchers’ Perceptions and Use of Social 3. Lavis, J.N., D. Robertson, J.M. Woodside, C.B. McLeod, J. Abelson, and the Media,” Health Affairs. 2014;33(7). Knowledge Transfer Study Group. “How Can Research Organizations More Ef- fectively Transfer Research Knowledge to Decision Makers?” Milbank Quarterly, Vol. 81, No. 2, 2003, pp. 221-48. 9