United States Government Accountability Office Report to Congressional Addressees ELECTRONIC May 2023 HEALTH RECORDS VA Needs to Address Management Challenges with New System GAO-23-106731 May 2023 ELECTRONIC HEALTH RECORDS VA Needs to Address Management Challenges with New System Highlights of GAO-23-106731, a report to congressional addressees Why GAO Did This Study What GAO Found VA uses the Veterans Health The Department of Veterans Affairs (VA) organizational change management Information Systems and Technology activities for the Electronic Health Record Modernization (EHRM) program were Architecture (VistA), which includes the partially consistent with seven leading practices and not consistent with one department's legacy EHR system, to leading practice (see table). manage health care for its patients. VistA is technically complex, costly to Extent to Which the Electronic Health Record Modernization (EHRM) Program's Activities maintain, and does not fully support Were Consistent with Organizational Change Management Leading Practices the need to exchange health data with Leading practice GAO assessment other organizations. In June 2017, VA Developing a vision for change Partially consistent initiated the EHRM program to replace Identifying and managing stakeholders Partially consistent VistA. Communicating effectively Partially consistent Congressional report language Assessing the readiness for change Partially consistent associated with the VA appropriations Increasing workforce skills and competencies Not consistent for fiscal years 2020 through 2022 Identifying and addressing potential barriers to change Partially consistent contained provisions for GAO to review Establishing targets and metrics for change Partially consistent VA's EHR deployment. GAO's Assessing the results of change Partially consistent objectives were to determine the extent Source: GAO analysis of Department of Veterans Affairs (VA) data. | GAO-23-106731 to which VA has (1) followed leading organizational change management Until the program fully implements the leading practices for change management, practices for the EHRM program, (2) future deployments risk continuing change management challenges that can assessed satisfaction with the new hinder effective use of the new electronic health record (EHR) system. system, and (3) identified and Most users have expressed dissatisfaction with the new system. VA's 2021 and addressed EHR system issues. GAO 2022 surveys showed that users were not satisfied with the system's identified leading change management practices and evaluated VA's activities performance or training. About 79 percent (1,640 of 2,066) of users disagreed or against these practices. It also strongly disagreed that the system enabled quality care. In addition, about 89 reviewed the results of surveys that VA percent (1,852 of 2,074) of users disagreed or strongly disagreed that the system conducted to determine users' made them as efficient as possible. Further, VA has not established targets (i.e., satisfaction with the new EHR, goals) to assess user satisfaction. Until it does so, VA lacks a basis for conducted interviews with selected determining when satisfaction has sufficiently improved for the system to be users, and interviewed officials on user deployed at additional sites. Such a basis helps ensure that the system is not satisfaction goals. Further, GAO deployed prematurely, which could risk patients' safety. analyzed VA's data on the contractor's performance meeting time frames for VA did not adequately identify and address system issues. Specifically, VA did addressing system trouble tickets. not ensure that trouble tickets for the new EHR system were resolved within timeliness goals. It subsequently worked with the contractor to reduce the What GAO Recommends number of tickets that were over 45 days old. Nevertheless, the overall number of open tickets has steadily increased since 2020. Accordingly, it is critical that GAO is making 10 recommendations system issues be resolved in a timely manner. Additionally, although VA has to VA to address change management, assessed the system's performance at two sites, as of January 2023, it had not user satisfaction, system trouble ticket, and independent operational conducted an independent operational assessment, as originally planned and assessment deficiencies. VA consistent with leading practices for software verification and validation. Without concurred with the recommendations such an independent assessment, VA will be limited in its ability to (1) validate and described actions the department that the system is operationally suitable and effective, and (2) identify, track, and plans to take to address them. resolve key operational issues. View GAO-23-106731. For more information, In April 2023, VA announced that it planned to halt future deployments of the new contact Carol C. Harris at (202) 512-4456 or EHR system to focus on making improvements at the five sites currently using harriscc@gao.gov. the system. United States Government Accountability Office Contents Letter 1 VA's Organizational Change Management Activities Were Partially Consistent with Leading Practices 3 Although Users Were Dissatisfied with the New System, VA Has Not Yet Established Goals for Improvement 6 VA Did Not Adequately Identify and Address System Issues 7 Conclusions 8 Recommendations for Executive Action 9 Agency Comments 10 Appendix I Briefing Presented to Staff Members of Congressional Committees and Requesters 13 Appendix II User Feedback on VA's New Electronic Health Record System from Structure Interviews 74 Appendix III Comments from the Department of Veterans Affairs 79 Appendix IV GAO Contact and Staff Acknowledgments 83 Table Table 1: Extent to Which the Electronic Health Record Modernization (EHRM) Program's Activities Were Consistent with Organizational Change Management Leading Practices 3 Page i GAO-23-106731 Electronic Health Record Modernization Abbreviations DOD Department of Defense EHR electronic health record EHRM Electronic Health Record Modernization VA Department of Veterans Affairs VHA Veterans Health Administration VISN Veterans Integrated Services Network VistA Veterans Health Information Systems and Technology Architecture This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page ii GAO-23-106731 Electronic Health Record Modernization Letter 441 G St. N.W. Washington, DC 20548 May 18, 2023 Congressional Addressees The use of IT is crucial to helping the Department of Veterans Affairs (VA) effectively serve the nation's veterans. Specifically, VA uses the Veterans Health Information Systems and Technology Architecture (VistA) to manage health care for its patients, which contains the department's electronic health record (EHR). However, VistA is technically complex, costly to maintain, and does not fully support the department's need to exchange EHRs with other organizations, such as the Department of Defense (DOD) and private health care providers. As such, in June 2017, VA initiated the Electronic Health Record Modernization (EHRM) program to replace VistA with the same Oracle Cerner EHR system DOD is acquiring. 1 VA has reported obligating about $9.42 billion on EHRM from fiscal year 2018 through the first quarter of fiscal year 2023. Congressional report language associated with the Military Construction, Veterans Affairs, and Related Agencies Appropriations Acts for Fiscal Years 2020 through 2022 contained provisions for us to review VA's EHR deployment to keep Congress apprised of VA's progress. 2 Our objectives were to determine the extent to which VA had (1) employed organizational change management strategies for the EHRM program consistent with leading practices, (2) assessed satisfaction with the new system, and (3) identified and addressed EHR system issues. On March 10, 2023, we provided a briefing to congressional staff on the results of our review. The purpose of this report is to deliver the published 1VA contracted with Cerner Government Services, Inc. (Cerner) for the department's new EHR system in May 2018. Subsequently, in June 2022, Cerner was acquired by Oracle Corporation and began formally identifying itself as Oracle Cerner, which is the name we use throughout this report. 2H.R. Rep. No. 116-63 at 5 (May 15, 2019) accompanying the Military Construction, Veterans Affairs, and Related Agencies Appropriations Act, 2020, Pub. L. No. 116-94, Div. F, 133 Stat. 2534, 2778 (2019); H.R. Rep. No. 116-445 at 5 (July 13, 2020) accompanying the Military Construction, Veterans Affairs, and Related Agencies Appropriations Act, 2021, Pub. L. No. 116-260, Div. J, 134 Stat. 1182, 1654 (2020); H.R. Rep. No. 117-81 at 79 (July 2, 2021) and S. Rep. No. 117-35 at 72-73 (Aug. 4, 2021) accompanying the Military Construction, Veterans Affairs, and Related Agencies Appropriations Act, 2022, Pub. L. No. 117-103, Div. J, 136 Stat. 49, 528 (2022). Page 1 GAO-23-106731 Electronic Health Record Modernization briefing slides to you and officially transmit our recommendations to VA. The briefing slides are reprinted in appendix I. In April 2023, VA announced that it planned to halt future deployments of the new EHR system to prioritize making improvements at the five sites currently using the system. VA is not planning to schedule additional deployments until it is confident that the new EHR system is effectively functioning at the five sites. To address the first objective, we identified leading change management practices and evaluated VA's activities against these practices. To assess whether the program's activities were aligned with leading practices, we reviewed EHRM change management documentation, such as wave deployment plans, stakeholder communication strategy and plan, change impact analyses, site deployment and work plans, and change readiness questionnaire reports. To address the second objective, we reviewed results of surveys that VA conducted to determine users' satisfaction with the new EHR and interviewed officials on user satisfaction goals. We also conducted structured interviews with selected users from the Mann-Grandstaff VA Medical Center, Jonathan M. Wainwright VA Medical Center (Walla Walla), and VA Central Ohio Health Care System (Columbus), the three locations where the new system was first deployed. Specifically, we conducted structured interviews with 63 users at these three locations between April and August 2022. See appendix II for the results of user feedback from our structured interviews. To address the third objective, we analyzed VA's data on the contractor's performance meeting time frames established in the service level agreement for the contractor to address system trouble tickets. We also obtained documentation of the EHRM program's testing activities, including test plans and results. We then analyzed the plans, as well as test activities that had already been completed, to determine whether they constituted an independent operational assessment, consistent with leading practices for software verification and validation. Additional detail on our objectives, scope, and methodology are included in appendix I. We conducted this performance audit from February 2021 through May 2023 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe Page 2 GAO-23-106731 Electronic Health Record Modernization that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. As we reported in our March 2023 briefing to congressional committees VA's Organizational and requesters, according to federal guidance and other leading Change Management practices, change management practices are intended to apply an organized and structured framework to the often chaotic and perplexing Activities Were world of organizational change. 3 Effective change management Partially Consistent techniques help managers to plan, organize, and negotiate successful changes in the organization. The objective of managing organizational with Leading change is to maximize the likelihood of successfully implementing change Practices quickly and with reduced risk. VA's organizational change management activities for the EHRM program were partially consistent with seven of the leading practices and not consistent with one leading practice (see table 1). Table 1: Extent to Which the Electronic Health Record Modernization (EHRM) Program's Activities Were Consistent with Organizational Change Management Leading Practices Leading practice GAO assessment Developing a vision for Partially consistent - The department developed a vision to have a comprehensive electronic health change record (EHR) accessible across the Department of Veterans Affairs (VA), the Department of Defense (DOD), and community care providers to enhance the quality of health care through a new EHR system and standardized clinical practice workflow processes. However, VA has not established a VA-driven strategy for change. A Veterans Health Administration commissioned report from April 2021 noted the need for a VA-driven change management strategy to formalize the structure and people capabilities to support the readiness of end users and drive adoption. As of January 2023, it had not provided documentation of a VA-driven change management strategy. 3ProjectManagement Institute, Inc., Managing Change in Organizations: A Practice Guide, (Newtown Square, PA: 2013); Office of Personnel Management (OPM), Migration Planning Guidance Information Documents, Change Management Best Practices, October 2011; GAO, Business Process Reengineering Assessment Guide-Version 3, (GAO/AIMD-10.1.15); Prosci, The Prosci ADKAR® Model, A Goal Oriented Change Management Model to Guide Individual and Organizational Change; ISACA, COBIT 2019 Framework, 2019. ADKAR® is a registered trademark of Prosci, Inc. Page 3 GAO-23-106731 Electronic Health Record Modernization Leading practice GAO assessment Identifying and managing Partially consistent - The program identified stakeholder groups, created a stakeholder communication stakeholders strategy and plan, and conducted numerous workshops at the national and local level for the purpose of engaging, identifying, and analyzing stakeholders. However, we previously reported that VA did not always effectively communicate information to stakeholders, including medical facility clinicians and staff, to ensure relevant representation at local workshop meetings and that the department did not have a stakeholder register to identify and engage key stakeholders for the EHRM program.a We recommended that VA develop such a tool. EHRM program officials said that in August 2022 they began conducting workshops with Directors from future implementation sites to focus on site stakeholder engagement. If VA continues to focus on site stakeholder engagement, this should better position the department to effectively identify and manage stakeholders, while addressing our open recommendation. Communicating Partially consistent - The program defined a stakeholder communication plan to engage with effectively stakeholders involved with the EHR system implementation and deployment. The program used various methods to communicate with program stakeholders, and documented over 5,000 completed communications between December 2018 and October 2022. However, users of the system indicated that information on system changes and the status of trouble tickets were not effectively communicated after initial system deployment. Further, in November 2020, the program identified a risk that a communication plan had not been established to inform VA end users of changes, major incident management, upgrades, and package releases and as of July 2022, this risk was still open and a communication plan for changes in sustainment had not been finalized. In October 2022, EHRM program officials said that rather than develop the sustainment communication plan they were communicating through weekly user updates. However, department documentation of feedback from sites continued to show the need to distribute more frequent updates on change requests and system downtimes. Assessing the readiness Partially consistent - The program assessed its readiness for change by conducting change readiness for change questionnaires to serve as a baseline assessment across the initial deployment sites and to allow a tailored change effort to address gaps. According to the program office, as of January 2023, VA had conducted 55 questionnaires at 28 deployment sites. However, VA received limited responses to questionnaires assessing readiness for change and results from the questionnaires indicated that users were not ready for the change. Further, the program did not have assurance that it had resolved potential problems in a timely fashion. Increasing workforce Not consistent – Numerous assessments and reports identified that training has been a weakness for the skills and competencies program. EHRM program officials acknowledged that training did not meet users' expectations and effectively support the transition because the contractor-provided training focused on using the system. However, users needed additional training and support for learning the new workflow processes simultaneously. They said the program took a number of actions to address training issues, including adding additional clinical experts to support onsite training and increased use of a hands-on practice environment (i.e., sandbox). In addition, in September 2022, the department's contractor, Oracle Cerner, announced that it would work with an outside entity to make the training more efficient, applicable, and useful for caregivers. To address the lack of familiarity with VA workflows and processes, the deparmtnet noted it worked with Oracle Cerner to define additional change management activities through development of role-focused adoption pathways. Nevertheless, with the halt in future deployments, VA has not yet been able to demonstrate whether these actions to increase workforce skills and competencies have been effective. Identifying and Partially consistent - The program identified activities to monitor resistance to change. For example, VA addressing potential conducted site visits and change readiness questionnaires, to gather feedback and propose actions or barriers to change recommendations to address feedback. In addition, the Secretary conducted a Strategic Review, which identified barriers that needed to be addressed. As of January 2023, VA had completed 45 of 69 actions identified in the review, and 24 were in progress. VA planned to complete these action items by October 2024. Page 4 GAO-23-106731 Electronic Health Record Modernization Leading practice GAO assessment Establishing targets and Partially consistent - The EHRM program proposed various metrics for change such as the amount of metrics for change time spent in the EHR system and the number of patients seen in an ambulatory setting. The program also identified metrics for performance of the new system such as measuring veteran experience, health care operations, workforce support, and quality and safety. However, VA had not fully established targets to measure the adoption of the change. In addition, the department did not have a plan that outlined the metrics, including agreed upon targets, to measure the results of the change. VA reported in November 2022 that it was continuing to refine functional and technical quality standards to define success, including metrics to define access to care and clinical operational efficiency, but did not provide a timeline for when it would be final. Assessing the results of Partially consistent - To measure adoption, the EHRM program collected data, such as amount of time change spent in the EHR system and the number of patients seen in an ambulatory setting. Further, the program has been tracking performance metrics such as veteran experience, health care operations, workforce support, and quality and safety since initial deployment in October 2020. However, VA had not fully identified specific targets and users shared examples of concerns about their productivity using the new system and veterans' access to care. In addition, the program had not demonstrated that it had taken action needed to ensure that the change has been reinforced and sustained. For example, a March 2021 survey aimed at measuring Mann-Grandstaff users' perspective on their ability to use the new system noted that 82 percent of users agreed or strongly agreed that the new EHR was cumbersome to use, and 84 percent agreed or strongly agreed that the new EHR was unnecessarily complex. Legend: Consistent – VA provided evidence that it conducted organizational change management activities mostly consistent with leading practices. Partially consistent – VA provided evidence that it conducted organizational change management activities consistent with some of the leading practice criteria, but some key parts were not followed. Not consistent – VA did not provide sufficient evidence that it followed leading practices. Source: GAO analysis of VA data. | GAO-23-106731 a GAO, Electronic Health Records: Ongoing Stakeholder Involvement Needed in the Department of Veterans Affairs' Modernization Effort, GAO-20-473 (Washington, D.C.: June 5, 2020) and Electronic Health Records: VA Needs to Address Data Management Challenges for New System, GAO-22-103718 (Washington, D.C.: Feb. 1, 2022). The program's organizational change management activities were not fully consistent with leading practices for several reasons. First, VA did not have a VA-driven strategy for how its efforts would supplement the contractor-led change management activities for deployment. According to EHRM program officials, the contractor's change management activities focused on the activities required to deploy the new system. However, these activities did not address user challenges with transitioning to new workflow processes. Further, EHRM officials noted that the program office had experienced transition in change management leadership and vacancies in their change management staffing. This limited the resources available for coordinating and implementing change management activities. Until the program implements all of the eight leading practices for change management, future deployments could be at risk of similar change management challenges. This could hinder users' ability to effectively use Page 5 GAO-23-106731 Electronic Health Record Modernization the system, impede their knowledge of new workflows, and limit the utility of system improvements. GAO and federal IT guidance recognize the importance of defining Although Users Were program goals and related performance targets and using such targets to Dissatisfied with the assess progress in achieving the goals. 4 Leading practices emphasize that periodic user satisfaction data be proactively used to improve New System, VA Has performance and demonstrate the level of satisfaction the project is Not Yet Established delivering. Measuring user satisfaction with the system is essential for monitoring progress towards pre-established goals or targets and allows Goals for programs to understand whether users' operational needs have been Improvement met. VA has taken steps to obtain feedback on the performance and implementation of EHRM. For example, in September 2022, VA worked with a contractor to conduct a survey of users from two regions, Veterans Integrated Services Network (VISN) 10 and VISN 20, where the new Oracle Cerner EHR system had been deployed, to determine user satisfaction. However, in December 2022, the contractor reported on VA's results in comparison to other health care systems, which indicated that users were not satisfied with the performance of the new system or the training for the new system. For example, about 79 percent (1,640 of 2,066) of users disagreed or strongly disagreed that the system enabled quality care. In addition, about 89 percent (1,852 of 2,074) of users disagreed or strongly disagreed that the system made them as efficient as possible. Additionally, VA had not established targets (i.e., goals) to assess user satisfaction. EHRM provided several reasons for why the program had not established specific goals for user satisfaction for the system. Specifically, 4GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, GAO-04-394G (Washington, D.C.: March 2004); Executive Office of the President, Office of Management and Budget, Evaluating Information Technology Investments, A Practical Guide (November 1995), Office of Management and Budget, Preparation, Submission, and Execution of the Budget, OMB Circular No. A-11 (August 2022), and General Services Administration, Modernization and Migration Management (M3) Playbook, accessed Oct. 20, 2022, https://www.ussm.gov/m3. Page 6 GAO-23-106731 Electronic Health Record Modernization • In February 2022, EHRM program officials stated there was an opportunity for additional metrics such as user satisfaction targets in the future. • In October 2022, EHRM program officials stated they were focused on addressing technical changes to improve the system usability before establishing targets or goals for user satisfaction. Nevertheless, until VA establishes goals for user satisfaction, the department will be limited in its ability to objectively measure progress toward improving EHRM users' satisfaction with the system. The department will also lack a basis for determining when satisfaction has improved. Such a basis would help ensure that the system is not prematurely deployed to additional sites, which could risk patients' safety. Efforts to identify and address system issues can be supported by VA Did Not activities such as resolving trouble tickets quickly and conducting an Adequately Identify independent operational assessment. VA did not adequately identify and address system issues. Specifically, VA did not ensure that trouble tickets and Address System for the new EHR system were resolved within timeliness goals, but Issues subsequently worked with the contractor to reduce the number of tickets that were over 45 days old. Oracle Cerner officials provided explanations for the difficulties with meeting timeliness goals, noting that VA's IT systems are complex, which contributes to a large number of trouble tickets and that Oracle Cerner needed to apply additional staffing resources to address the problem. Additionally, according to VA's strategic review status report, the department recognized that its capacity to resolve the volume of tickets at the five deployment sites was overwhelmed due to, among other things, an insufficient number of subject matter experts. Until the program resolves trouble tickets according to established time frames, users' system issues will not be resolved in a timely manner. In addition, there is a risk that VA will not be able to address users' system issues effectively going forward, particularly when larger sites go live. Additionally, although VA has assessed the system for user performance at two sites, as of January 2023, VA had not conducted an independent operational assessment to evaluate if the new EHR system satisfies the intended use and user needs in the operational environment. According to leading practices for software verification and validation, a product should be evaluated to determine whether it satisfies the intended use Page 7 GAO-23-106731 Electronic Health Record Modernization and user needs in the operational environment. 5 An operational assessment is an evaluation of operational suitability and effectiveness made by an independent operational test activity with user support as required. 6 The EHRM program's master test plan from May 2021 described plans to execute independent post-production validation and operational assessment to assess the degree to which the new EHR met the users' needs in their daily operational use in the production environment. According to the program's test plan, the purpose of the operational assessment was to evaluate the system's efficiency, effectiveness, usability, user satisfaction, and training. EHRM program officials said that they did not plan to execute an independent operational assessment because such an assessment would be duplicative to existing post-go-live evaluations, change assessment surveys, and be disruptive to site operations. However, until an independent operational assessment of the new EHR system is conducted, VA will be limited in its ability to validate that the system is operationally suitable and effective, and to identify, track, and resolve key operational issues. An operational assessment, particularly if conducted by an independent entity, would enable the department to systematically catalog, report on, and track resolution of assessment findings with greater rigor, transparency, and accountability. The program's organizational change management activities were not Conclusions fully consistent with eight leading practices. These practices are especially important given that VA's transition to the new EHR was challenging for users at the initial deployment sites. In addition, VA undertook several efforts to assess user satisfaction with the new system, but results indicated that users were dissatisfied with the system. Further, VA had not established targets or goals for user satisfaction. Consequently, it is not evident what basis the department will use to determine when satisfaction has sufficiently improved to support a decision to deploy the system at additional sites. Such a basis is critically important to ensuring that systems not be deployed prematurely and pose unnecessary risks to patient health and safety. 5IEEE Standards Association, IEEE Standard for System, Software, and Hardware Verification and Validation (IEEE Std. 1012-2016), (September 2017). 6Defense Acquisition University Glossary, accessed September 1, 2022, https://www.dau.edu/acquipedia/pages/articlecontent.aspx?itemid=46. Page 8 GAO-23-106731 Electronic Health Record Modernization Finally, VA did not ensure that system issues had been addressed within established timeliness goals nor has it conducted an independent operational assessment. This type of assessment is beneficial in validating that the system satisfies user needs in the operational environment. We are making the following 10 recommendations to VA: Recommendations for Executive Action • The Secretary of VA should ensure that VA documents a VA-specific change management strategy to formalize its approach to drive user adoption. (Recommendation 1) • The Secretary of VA should ensure that the department's planned improvements to communication of system changes meet users' needs for the frequency of the updates provided. (Recommendation 2) • The Secretary of VA should take steps to improve change readiness scores prior to future system deployments. (Recommendation 3) • The Secretary of VA should ensure steps taken by the EHRM program and Oracle Cerner to increase workforce skills and competencies through improved training and related change management activities have been effective. (Recommendation 4) • The Secretary of VA should address users' barriers to change, by ensuring planned completion of all actions identified in the Secretary's Strategic Review. (Recommendation 5) • The Secretary of VA should develop a plan, including a timeline, for establishing (1) targets for measuring the adoption of changes and (2) metrics and targets to measure the resulting outcomes of the change. (Recommendation 6) • The Secretary of VA should measure and report on outcomes of the change and take actions to support users' ability to use the system to reinforce and sustain the change. (Recommendation 7) • The Secretary of VA should establish user satisfaction targets (i.e., goals) and ensure that the program demonstrates improvement toward meeting those targets prior to future system deployments. (Recommendation 8) • The Secretary of VA should make certain that future system trouble tickets are resolved within established timeliness goals. (Recommendation 9) Page 9 GAO-23-106731 Electronic Health Record Modernization • The Secretary of VA should reinstitute plans to conduct an independent operational assessment to evaluate the suitability and effectiveness of the new EHR system for users in the operational environment. (Recommendation 10) We provided a draft of this report to VA for review and comment. In its Agency Comments comments, reproduced in appendix III, VA concurred with the recommendations. In addition, VA described actions the department plans to address them. For example, the department said that it planned to document a VA-specific change management strategy to formalize its approach to drive user adoption. The department also planned to establish user satisfaction targets to ensure that the program has demonstrated improvements toward meeting those targets before additional system deployments. Further, VA stated that it would reinstitute plans to conduct an independent operational assessment to ensure the new EHR system is efficient and effective for users in the operational environment. These actions, if implemented as described, should address our recommendations. VA also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to interested congressional committees and the Secretary of Veterans Affairs. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions on the matters discussed in this report, please contact me at (202) 512-4456 or at harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. Carol C. Harris Director, Information Technology Management Issues Page 10 GAO-23-106731 Electronic Health Record Modernization List of Congressional Addressees The Honorable Jon Tester Chair The Honorable Jerry Moran Ranking Member Committee on Veterans' Affairs United States Senate The Honorable Patty Murray Chair The Honorable John Boozman Ranking Member Subcommittee on Military Construction, Veterans Affairs, and Related Agencies Committee on Appropriations United States Senate The Honorable Mike Bost Chairman The Honorable Mark Takano Ranking Member Committee on Veterans' Affairs House of Representatives The Honorable John Carter Chairman The Honorable Debbie Wasserman Shultz Ranking Member Subcommittee on Military Construction, Veterans Affairs, and Related Agencies Committee on Appropriations House of Representatives The Honorable Matthew Rosendale, Sr. Chairman Subcommittee on Technology Modernization Committee on Veterans' Affairs House of Representatives Page 11 GAO-23-106731 Electronic Health Record Modernization The Honorable Jim Banks House of Representatives The Honorable Susie Lee House of Representatives Page 12 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Appendix I: Briefing Presented to Staff Members of Congressional Committees and Members of Congressional Committees and Requesters Requesters Page 13 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 14 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 15 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 16 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 17 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 18 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 19 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 20 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 21 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 22 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 23 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 24 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 25 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 26 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 27 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 28 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 29 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 30 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 31 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 32 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 33 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 34 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 35 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 36 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 37 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 38 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 39 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 40 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 41 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 42 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 43 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 44 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 45 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 46 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 47 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 48 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 49 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 50 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 51 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 52 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 53 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 54 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 55 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 56 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 57 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 58 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 59 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 60 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 61 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 62 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 63 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 64 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 65 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 66 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 67 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 68 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 69 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 70 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 71 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 72 GAO-23-106731 Electronic Health Record Modernization Appendix I: Briefing Presented to Staff Members of Congressional Committees and Requesters Page 73 GAO-23-106731 Electronic Health Record Modernization Appendix II: User Feedback on VA's New Appendix II: User Feedback on VA's New Electronic Health Record System from Electronic Health Record System from Structure Interviews Structure Interviews The Department of Veterans Affairs (VA) is deploying a new electronic health record (EHR) system as part of its Electronic Health Record Modernization (EHRM) program. To obtain user feedback on the new EHR system, we conducted structured interviews with 63 users from the Mann-Grandstaff VA Medical Center, Jonathan M. Wainwright VA Medical Center, and VA Central Ohio Healthcare System, the three locations where the new EHR was first deployed. Each structured interview comprised closed- and open-ended questions. In this appendix, we include aggregated results of responses to the closed-ended questions we asked in our interviews. 1. Compared to VistA/CPRS, how much time are you spending using the new EHR system to perform your job tasks? Total Much more time 26 More time 24 About the same amount of time 6 Less time 4 Much less time 1 No basis to judge/Don't know 2 2. To what extent have patient appointment durations increased or decreased as a result of using the new EHR system? Total Increased by more than 60 minutes 3 Increased by 31-60 minutes 12 Increased by 0-30 minutes 21 Neither increased nor decreased 13 Decreased by 0-30 minutes 2 Decreased by 31-60 minutes 0 Decreased by more than 60 minutes 0 No basis to judge 12 Page 74 GAO-23-106731 Electronic Health Record Modernization Appendix II: User Feedback on VA's New Electronic Health Record System from Structure Interviews 3. Do you agree or disagree with the following statements based on your current experience using the new financial systems? Neither No Agree basis Strongly nor Strongly to Statement Agree Agree Disagree Disagree Disagree judge 3a: The new EHR system 1 14 6 27 14 1 allows me to effectively perform the duties of my position. 3b. The new EHR system 2 9 3 23 25 1 meets the expectations I had prior to and during go-live. 3c. The new EHR system 1 18 11 29 4 0 is available when I need it. 3d. The new EHR 0 14 6 31 12 0 system's performance is timely. 3e. Health data was 0 11 6 22 21 3 migrated accurately from the old EHR system to the new EHR system. 3f. Information in the new 0 12 15 23 12 1 EHR system is presented in a logical manner. 3g. I do not rely on 1 6 2 21 30 3 workarounds to perform the duties of my position. 3h. Compared to 1 1 6 21 33 1 VistA/CPRS, the new EHR system requires fewer steps to accomplish what I need to do. 3i. I have been provided 0 10 6 24 23 0 effective training on how to use the new EHR system. 3j. The helpdesk provides 0 10 11 21 15 6 adequate support when I encounter problems with the new EHR system. 3k. The helpdesk 0 4 10 27 16 6 provides adequate resolution to problems I encounter with the new EHR system. Page 75 GAO-23-106731 Electronic Health Record Modernization Appendix II: User Feedback on VA's New Electronic Health Record System from Structure Interviews Neither No Agree basis Strongly nor Strongly to Statement Agree Agree Disagree Disagree Disagree judge 3l. Overall, I am satisfied 1 6 5 31 20 0 with the new EHR system. 4. For each of the following, to what extent did you find the topic challenging: Extremely Moderately Somewhat Slightly Not at all challenging challenging challenging challenging challenging Data accuracy 20 18 14 6 5 Support/Reporting 22 18 12 6 5 Issues Productivity 25 16 15 5 2 Accessibility/Ease 16 17 16 12 2 of use 5. Overall, in what way has the new EHR system affected your ability to perform the duties of your position? Total Very positive impact 1 Positive impact 3 Neither positive nor negative impact 8 Negative impact 37 Very negative impact 14 6. How has the new EHR system affected your productivity? Total Greatly increased productivity 1 Increased productivity 3 Neither increased nor decreased productivity 9 Decreased productivity 35 Greatly decreased productivity 15 Page 76 GAO-23-106731 Electronic Health Record Modernization Appendix II: User Feedback on VA's New Electronic Health Record System from Structure Interviews 7. How has the new EHR system affected veterans' quality of care at this facility? Total Greatly improved the quality of care 0 Improved the quality of care 2 Neither improved nor reduced the quality of care 16 Reduced the quality of care 28 Greatly reduced the quality of care 12 No basis to judge 5 8. How does your experience in using the new EHR system at this point in time compare to your experience in using the new system at the time of go-live? Total Greatly improved since go-live 10 Slightly improved since go-live 29 Did not improve or worsen since go-live 12 Slightly worsened since go-live 5 Greatly worsened since go-live 7 No basis to judge 0 9. To what extent did each of the following help in preparing you to use the new EHR system? Very Great Great Moderate Small Extent Extent Extent extent No Extent Communications from 6 20 13 13 10 management at my site Communications from 0 5 11 26 21 VA Central Office/EHRM IO Communications from 1 19 25 13 5 Cerner staff. Local Workshops 2 7 10 22 21 Page 77 GAO-23-106731 Electronic Health Record Modernization Appendix II: User Feedback on VA's New Electronic Health Record System from Structure Interviews 10. In general, how satisfied have you been in obtaining resolution to problems with the new EHR system when you have contacted the helpdesk? Total Very satisfied 0 Satisfied 9 Neither satisfied nor dissatisfied 10 Dissatisfied 21 Very dissatisfied 15 I have never contacted the helpdesk for assistance with the EHR system 8 11. To what extent has your use of the helpdesk increased or decreased since go-live? Total Increased by a large extent 8 Increased by a small extent 3 Neither increased nor decreased 19 Decreased by small extent 15 Decreased by large extent 13 I have never contacted the help desk for assistance with the EHR system 5 Page 78 GAO-23-106731 Electronic Health Record Modernization Appendix III: Comments from the Appendix III: Comments from the Department of Veterans Affairs Department of Veterans Affairs Page 79 GAO-23-106731 Electronic Health Record Modernization Appendix III: Comments from the Department of Veterans Affairs Page 80 GAO-23-106731 Electronic Health Record Modernization Appendix III: Comments from the Department of Veterans Affairs Page 81 GAO-23-106731 Electronic Health Record Modernization Appendix III: Comments from the Department of Veterans Affairs Page 82 GAO-23-106731 Electronic Health Record Modernization Appendix IV: GAO Contact and Staff Appendix IV: GAO Contact and Staff Acknowledgments Acknowledgments Carol C. Harris, (202) 512-4456 or harriscc@gao.gov GAO Contact In addition to the individual named above, the following staff made key Staff contributions to this report: Mark Bird (Assistant Director), Merry Woo Acknowledgments (Analyst-in-Charge), Tim Barry, Chris Businsky, Quintin Dorsey, Rebecca Eyler, Ash Harper, Igor Koshelev, Christy Ley, Monica Perez-Nelson, Rachael Scott, Eric Trout, Walter Vance, Adam Vodraska, and Charles Youman. Page 83 GAO-23-106731 Electronic Health Record Modernization The Government Accountability Office, the audit, evaluation, and investigative GAO's Mission arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is Obtaining Copies of through our website. Each weekday afternoon, GAO posts on its website newly GAO Reports and released reports, testimony, and correspondence. You can also subscribe to GAO's email updates to receive notification of newly posted products. Testimony Order by Phone The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. Contact FraudNet: To Report Fraud, Website: https://www.gao.gov/about/what-gao-does/fraudnet Waste, and Abuse in Automated answering system: (800) 424-5454 or (202) 512-7700 Federal Programs A. Nicole Clowers, Managing Director, ClowersA@gao.gov, (202) 512-4400, U.S. Congressional Government Accountability Office, 441 G Street NW, Room 7125, Washington, Relations DC 20548 Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Stephen J. Sanford, Managing Director, spel@gao.gov, (202) 512-4707 Strategic Planning and U.S. Government Accountability Office, 441 G Street NW, Room 7814, External Liaison Washington, DC 20548 Please Print on Recycled Paper.