United States Government Accountability Office Testimony Before the Subcommittee on Technology Modernization, Committee on Veterans' Affairs, House of Representatives ELECTRONIC HEALTH For Release on Delivery Expected at 3:00 p.m ET Tuesday, May 9, 2023 RECORDS Challenges with VA's New System Call for Management Improvements Statement of Carol C. Harris, Director, Information Technology and Cybersecurity GAO-23-106785 May 2023 ELECTRONIC HEALTH RECORDS Challenges with VA's New System Call for Management Improvements Highlights of GAO-23-106785, a testimony before the Subcommittee on Technology Modernization, Committee on Veterans' Affairs, House of Representatives. Why GAO Did This Study What GAO Found VA uses the Veterans Health The Department of Veterans Affairs (VA) organizational change management Information Systems and Technology activities for the Electronic Health Record Modernization (EHRM) program were Architecture (VistA), which includes the partially consistent with seven leading practices and not consistent with one department's legacy EHR system, to leading practice (see table). manage health care for its patients. VistA is technically complex, costly to Extent to Which the Electronic Health Record Modernization (EHRM) Program's Activities maintain, and does not fully support Were Consistent with Organizational Change Management Leading Practices the need to exchange health data with Leading practice GAO assessment other organizations. In June 2017, VA Developing a vision for change Partially consistent initiated the EHRM program to replace Identifying and managing stakeholders Partially consistent VistA. VA has reported obligating Communicating effectively Partially consistent about $9.42 billion on EHRM from Assessing the readiness for change Partially consistent fiscal year 2018 through the first quarter of fiscal year 2023. Increasing workforce skills and competencies Not consistent Identifying and addressing potential barriers to change Partially consistent GAO was asked to testify on its recent Establishing targets and metrics for change Partially consistent review to determine the extent to which Assessing the results of change Partially consistent VA has (1) followed leading organizational change management Source: GAO analysis of Department of Veterans Affairs (VA) data. | GAO-23-106785 practices for the EHRM program, (2) Until the program fully implements the leading practices for change management, assessed satisfaction with the new future deployments risk continuing change management challenges that can system, and (3) identified and hinder effective use of the new electronic health record (EHR) system. addressed EHR system issues. For its recent review GAO identified leading Most users expressed dissatisfaction with the new system. VA's surveys showed change management practices and that users were not satisfied with the system's performance and training. About evaluated VA's activities against these 79 percent (1,640 of 2,066) of responding users disagreed or strongly disagreed practices. It also reviewed results of that the system enabled quality care. In addition, users in GAO's structured surveys that VA conducted to interviews said the pharmacy-related system module presented challenges, such determine users' satisfaction with the as incorrect medication orders and increased prescription processing times, new EHR, conducted interviews with which contributed to patient safety risks. Further, VA has not established targets selected users, and interviewed (i.e., goals) to assess user satisfaction. Until it does so, VA lacks a basis for officials on user satisfaction goals. determining when satisfaction has sufficiently improved for the system to be Further, GAO analyzed system trouble deployed at additional sites. Such a basis helps ensure that the system is not ticket and related data regarding VA's deployed prematurely, which could risk patients' safety. service level agreement with its contractor. VA did not adequately identify and address system issues. Specifically, VA did not ensure that trouble tickets for the new EHR system were resolved within What GAO Recommends timeliness goals. It subsequently worked with the contractor to reduce the GAO made 10 recommendations to VA number of tickets that were over 45 days old. Nevertheless, the overall number of in its recent review to address change open tickets has steadily increased since 2020. Accordingly, it is critical that management, user satisfaction, system system issues be resolved in a timely manner. Additionally, although VA has trouble ticket, and independent assessed the system's performance at two sites, as of January 2023, it had not operational assessment deficiencies. conducted an independent operational assessment, as originally planned and VA concurred with the consistent with leading practices for software verification and validation. Without recommendations. such an independent assessment, VA will be limited in its ability to validate that the system is operationally suitable and effective. View GAO-23-106785. For more information, Subsequent to GAO's review, VA announced that it planned to halt future contact Carol C. Harris at (202) 512-4456 or harriscc@gao.gov. deployments of the new EHR system to focus on making improvements at the five sites currently using the system. United States Government Accountability Office Letter Letter Chairman Rosendale, Ranking Member Cherfilus-McCormick, and Members of the Subcommittee: I am pleased to participate in today's hearing on pharmacy-related functions of the Department of Veterans Affairs' (VA) new electronic health record (EHR) system. 1 As you know, the department is acquiring this system as part of its Electronic Health Record Modernization (EHRM) program. The use of IT is crucial to helping VA effectively serve the nation's veterans. Specifically, VA uses the Veterans Health Information Systems and Technology Architecture (VistA) to manage health care to its patients, which contains the department's EHR. In June 2017, VA initiated the EHRM program to replace VistA because it is technically complex, costly to maintain, and does not fully support the department's need to exchange EHRs with other organizations, such as the Department of Defense (DOD) and private health care providers. As a result, VA began to acquire the same Oracle Cerner EHR system DOD had selected. VA has reported obligating about $9.42 billion on EHRM from fiscal year 2018 through the first quarter of fiscal year 2023. My statement today is based on a recently completed review to determine the extent to which VA has (1) employed organizational change management strategies for the EHRM program consistent with leading practices, (2) assessed users' satisfaction with the new system, and (3) identified and addressed EHR system issues. 2 The results of our review are applicable to the EHRM program as a whole, including to the pharmacy-related concerns that are the focus of today's hearing. Appendix I provides information on our objectives, scope, and methodology. The work on which this statement is based was conducted in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions 1An EHR is a collection of information about the health of an individual and the care provided to that individual, such as patient demographics, medications, and past medical history. 2On March 10, 2023, we provided a final briefing on the results of our review to the House and Senate Committees on Appropriations, Subcommittees on Military Construction, Veterans' Affairs, and Related Agencies; House and Senate Committees on Veterans' Affairs; Rep. Jim Banks; and Rep. Susie Lee. We plan to publish the briefing in a May 2023 report. Page 1 GAO-23-106785 based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. VA's IT systems provide capabilities to establish and maintain EHRs that Background health care providers, including pharmacists, and other clinical staff use to view patient information in inpatient, outpatient, and long-term care settings. The department's health information system-VistA-serves an essential role in helping the department to fulfill its health care delivery mission. However, VistA is technically complex: it is comprised of about 170 clinical, financial, and administrative applications that support health care delivery at more than 1,600 medical facilities. In addition, VA has approximately 130 versions of the system department-wide. VA is in the process of replacing VistA because it has been in operation for more than 30 years, and as previously mentioned, is costly to maintain, and does not fully support VA's need to electronically exchange health records with other organizations, such as DOD. Toward this end, VA established the EHRM program and contracted with Oracle Cerner to acquire Millennium (the core EHR system) and HealtheIntent (a cloud- based software platform that aggregates health data from multiple data sources to create a longitudinal patient record). 3 VA's contract also includes requirements for Oracle Cerner to: • conduct reviews and assessments of medical facilities to determine facility needs prior to deployment (e.g., technology infrastructure); • provide services, including project management, change management, training, and testing; and • host and deploy EHRM across the VA enterprise. The EHRM Integration Office (EHRM IO) is the organization within VA that is responsible for planning and implementing the EHRM program. 4 3A cloud-based service can allow an agency to only pay for the IT services used, when executed effectively. 4Theoffice was previously referred to as the Office of Electronic Health Record Modernization. Page 2 GAO-23-106785 Deployment Schedule for Initially, VA planned to deploy the new system at sites in stages based on the New EHR System their geographical location over a 10-year period, through 2028. In October 2020, VA first deployed the new EHR at the Mann-Grandstaff VA Medical Center and planned to deploy it to other sites. 5 However, in March 2021, VA identified issues with the initial deployment, which led to a strategic review of the program. The strategic review identified eight challenge areas for EHRM, as well as plans and progress towards addressing those challenges. 6 After the review, VA deployed the new system to the following locations in 2022: • Jonathan M. Wainwright Memorial VA Medical Center (Walla Walla) in March 2022, • VA Central Ohio Health Care System (Columbus) in April 2022, and • Roseburg VA Health Care System and VA Southern Oregon Rehabilitation Center and Clinics (White City) in June 2022. In June 2022, VA announced that it would be pausing future deployments of the system until 2023 to allow time for improvements to the system. Subsequently, in October 2022, VA delayed deployments to address technical and other system performance issues. In April 2023, VA announced that it planned to halt future deployments of the new EHR system to prioritize making improvements at the five sites currently using the system. VA did not plan to schedule additional 5These sites are within the Veterans Health Administration's (VHA) Veterans Integrated Services Network 20 (VISN 20) and VISN 10. VHA is divided into areas called Veterans Integrated Services Networks (VISNs). There are currently 18 VISNs throughout VHA based on geographic location. VISNs provide oversight and guidance to the VA Medical Centers and VA Health Care Systems within their area and are sometimes called a "network." VISN 20 includes medical centers and community-based outpatient clinics in the states of Alaska, Washington, Oregon, most of Idaho, and one county each in Montana and California. VISN 10 serves veterans in the Ohio, Indiana, and Michigan areas. 6VA summarized the results of its strategic review in the Electronic Health Record Comprehensive Lessons Learned report. Department of Veterans Affairs, Electronic Health Record Comprehensive Lessons Learned (Washington, D.C.: July 2021). The eight challenge areas described in the report are improving the veteran experience, ensuring patient safety, providing extended training to the frontline employees, building confidence at VA sites, implementing organizational and program improvements, making governance effective, improving operational efficiencies, and centralizing data management for workers and veterans. Page 3 GAO-23-106785 deployments until it was confident that the new EHR system is effectively functioning at the sites that are currently using the system. EHRM Costs VA contracted with the Institute for Defense Analyses to provide an independent cost estimate for the program. In September 2022, the Institute reported that the EHRM life cycle cost estimate was $49.8 billion: • $32.7 billion for a 13-year implementation phase and • $17.1 billion in sustainment costs for the following 15 years. As previously mentioned, VA has reported obligating about $9.42 billion on EHRM from fiscal year 2018 through the first quarter of fiscal year 2023. This includes three areas: • the EHR contract ($4.49 billion), • IT infrastructure ($2.61 billion), and • program management ($882 million). In addition, VA reported obligating about $1.27 billion and $170 million on the program from the Veterans Health Administration (VHA) and the Office of Information and Technology (OIT), respectively. GAO Has Reported on VA In 2015, we designated VA health care as a high-risk area for the federal Health Care and EHRM government, in part due to its IT challenges. 7 In addition, we have previously reported on the EHRM program: • In June 2020, we reported on the process for configuring the department's new EHR system. 8 We noted that VA's decision-making procedures were generally effective, but the department did not always ensure key stakeholder involvement. We recommended (and VA concurred) that VA ensure the involvement of all relevant 7VA's IT issues were highlighted in our 2015 high-risk report and subsequent high-risk reports. See GAO, High-Risk Series: An Update, GAO-15-290 (Washington, D.C.: Feb. 11, 2015); High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others, GAO-17-317 (Washington, D.C.: Feb. 15, 2017); High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas, GAO-19-157SP (Washington, D.C.: Mar. 6, 2019); and High-Risk Series: Dedicated Leadership Needed to Address Limited Progress in Most High-Risk Areas, GAO-21-119SP (Washington, D.C.: Mar. 2, 2021). 8GAO, Electronic Health Records: Ongoing Stakeholder Involvement Needed in the Department of Veterans Affairs' Modernization Effort, GAO-20-473 (Washington, D.C.: June 5, 2020). Page 4 GAO-23-106785 deployment site stakeholders in the EHR system configuration decision process. The department has begun to improve subject matter expert identification and involvement, but this type of involvement needs to continue until different stages of modernization unfold. As such, our recommendation remains not fully implemented as of April 2023. • In February 2021, we reported that VA had made progress toward deploying the new EHR system by making configuration decisions, developing capabilities and interfaces, completing testing events, and deploying the system at the first site in October 2020. 9 However, we noted that the department was at risk of developing a system that may not perform as intended or could negatively impact the likelihood of successful adoption by users if critical and high severity test findings (that could result in system failure) were not resolved prior to future deployments. 10 We made two recommendations (and VA concurred) in February 2021, including that VA postpone deployment of the new EHR system at planned sites until any resulting critical and high severity test findings are appropriately addressed. VA stated that it planned to continue to test and appropriately adjudicate all critical and high severity test findings prior to future deployments. We will continue to monitor VA's actions to implement our recommendation as the department makes additional system deployments. • In February 2022, we reported that our work and VA's analyses indicated challenges with the quality of transferred data and with how the new EHR system worked for some users. 11 For example, VA identified errors in allergy, medication, and immunization data, which raise patient safety concerns. We recommended that VA establish and use performance measures and goals to ensure the quality of transferred data. The department concurred with our recommendation and began to take steps to address it. However, those steps had not yet been completed as of April 2023. 9GAO, Electronic Health Records: VA Has Made Progress in Preparing for New System, but Subsequent Test Findings Will Need to Be Addressed, GAO-21-224 (Washington, D.C.: Feb. 11, 2021). 10A critical test finding results in the failure of the complete software system. A high severity test finding results in the failure of the complete software system; however, there are acceptable workarounds. 11GAO, Electronic Health Records: VA Needs to Address Data Management Challenges for New System, GAO-22-103718 (Washington, D.C.: Feb. 1, 2022). Page 5 GAO-23-106785 According to federal guidance and other leading practices, change VA's Organizational management practices are intended to apply an organized and structured Change Management framework to the often chaotic and perplexing world of organizational change. 12 Effective change management techniques help managers to Activities Were plan, organize, and negotiate successful changes in the organization. The Partially Consistent objective of managing organizational change is to maximize the likelihood of successfully implementing change quickly and with reduced risk. with Leading Leading practices for change management activities include: (1) Practices developing a vision for change, (2) identifying and managing stakeholders, (3) effectively communicating, (4) assessing the readiness for change, (5) increasing workforce skills and competencies, (6) identifying and addressing potential barriers to change, (7) establishing targets and metrics for change, and (8) assessing the results of change. As shown in table 1, VA's organizational change management activities for the EHRM program were partially consistent with seven of the leading practices and not consistent with one leading practice. Table 1: Extent to Which the Electronic Health Record Modernization (EHRM) Program's Activities Were Consistent with Organizational Change Management Leading Practices Leading practice GAO assessment Developing a vision for Partially consistent - The department developed a vision to have a comprehensive electronic health change (EHR) accessible across Department of Veterans Affairs (VA), the Department of Defense (DOD), and community care providers to enhance the quality of health care through a new EHR system and standardized clinical practice workflow processes. However, VA has not established a VA-driven strategy for change. A Veterans Health Administration commissioned report from April 2021 noted the need for a VA-driven change management strategy to formalize the structure and people capabilities to support the readiness of end users and drive adoption. As of January 2023, it had not provided documentation of a VA-driven change management strategy. 12Project Management Institute, Inc., Managing Change in Organizations: A Practice Guide, (Newtown Square, PA: 2013); Office of Personnel Management (OPM), Migration Planning Guidance Information Documents, Change Management Best Practices, October 2011; GAO, Business Process Reengineering Assessment Guide-Version 3, (GAO/AIMD-10.1.15); Prosci, The Prosci ADKAR® Model, A Goal Oriented Change Management Model to Guide Individual and Organizational Change; ISACA, COBIT 2019 Framework, 2019. ADKAR® is a registered trademark of Prosci, Inc. Page 6 GAO-23-106785 Leading practice GAO assessment Identifying and managing Partially consistent - The program identified stakeholder groups, created a stakeholder communication stakeholders strategy and plan, and conducted numerous workshops at the national and local level for the purpose of engaging, identifying, and analyzing stakeholders. However, we previously reported that VA did not always effectively communicate information to stakeholders, including medical facility clinicians and staff, to ensure relevant representation at local workshop meetings and that the department did not have a stakeholder register to identify and engage key stakeholders for the EHRM program. 13 We recommended that VA develop such a tool. EHRM Integration Office officials said that in August 2022 they began conducting workshops with Directors from future implementation sites to focus on site stakeholder engagement. If VA continues to focus on site stakeholder engagement, this should better position the department to effectively identify and manage stakeholders, while addressing our open recommendation. Communicating Partially consistent - The program defined a stakeholder communication plan to engage with effectively stakeholders involved with the EHR system implementation and deployment. The program used various methods to communicate with program stakeholders, and documented over 5,000 completed communications between December 2018 and October 2022. However, users of the system indicated that information on system changes and the status of trouble tickets were not effectively communicated after initial system deployment. Further, in November 2020, the program identified a risk that a communication plan had not been established to inform VA end users of changes, major incident management, upgrades, and package releases and as of July 2022, this risk was still open and a communication plan for changes in sustainment had not been finalized. In October 2022, EHRM program officials said that rather than develop the sustainment communication plan they were communicating through weekly user updates. However, department documentation of feedback from sites continued to show the need to distribute more frequent updates on change requests and system downtimes. Assessing the readiness Partially consistent - The program assessed its readiness for change by conducting change readiness for change questionnaires (CRQ) to serve as a baseline assessment across the initial deployment sites and to allow a tailored change effort to address gaps. According to the program office, as of January 2023, VA had conducted 55 CRQs at 28 deployment sites. However, VA received limited responses to questionnaires assessing readiness for change and results from the CRQs indicated that users were not ready for the change. Further, the program did not have assurance that it had resolved potential problems in a timely fashion. Increasing workforce Not consistent – Numerous assessments and reports identified that training has been a weakness for the skills and competencies program. EHRM program officials acknowledged that training did not meet users' expectations and effectively support the transition because the contractor-provided training focused on using the system. However, users needed additional training and support for learning the new workflow processes simultaneously. They said the program took a number of actions to address training issues, including adding additional clinical experts to support onsite training and increased use of a hands-on practice environment (i.e., sandbox). In addition, in September 2022, the department's contractor, Oracle Cerner, announced that it would work with an outside entity to make the training more efficient, applicable, and useful for caregivers. Nevertheless, VA has not yet demonstrated if the actions to increase workforce skills and competencies have been effective. 13GAO, Electronic Health Records: Ongoing Stakeholder Involvement Needed in the Department of Veterans Affairs' Modernization Effort, GAO-20-473 (Washington, D.C.: June 5, 2020) and Electronic Health Records: VA Needs to Address Data Management Challenges for New System, GAO-22-103718 (Washington, D.C.: Feb. 1, 2022). Page 7 GAO-23-106785 Leading practice GAO assessment Identifying and Partially consistent - The program identified activities to monitor resistance to change. For example, VA addressing potential conducted site visits and change readiness questionnaires, to gather feedback and propose actions or barriers to change recommendations to address feedback. In addition, the Secretary conducted a Strategic Review, which identified barriers that needed to be addressed. As of January 2023, VA had completed 45 of 69 actions identified in the review, and 24 were in progress. VA planned to complete these action items by October 2024. Establishing targets and Partially consistent - The EHRM program proposed various metrics for change such as the amount of metrics for change time spent in the EHR system and the number of patients seen in an ambulatory setting. The program also identified metrics for performance of the new system such as measuring veteran experience, health care operations, workforce support, and quality and safety. However, VA had not fully established targets to measure the adoption of the change. In addition, the department did not have a plan that outlined the metrics, including agreed upon targets, to measure the results of the change. VA reported in November 2022 that it was continuing to refine functional and technical quality standards to define success, including metrics to define access to care and clinical operational efficiency but did not provide a timeline for when it would be final. Assessing the results of Partially consistent - To measure adoption, the EHRM program collected data, such as amount of time change spent in the EHR system and the number of patients seen in an ambulatory setting. Further, the program has been tracking performance metrics such as veteran experience, health care operations, workforce support, and quality and safety since initial deployment in October 2020. However, VA had not fully identified specific targets and users shared examples of concerns about their productivity using the new system and veterans' access to care. In addition, the program had not demonstrated that it had taken action needed to ensure that the change has been reinforced and sustained. For example, a March 2021 survey aimed at measuring Mann-Grandstaff users' perspective on their ability to use the new system noted that 82 percent of users agreed or strongly agreed that the new EHR was cumbersome to use, and 84 percent agreed or strongly agreed that the new EHR was unnecessarily complex. Legend: Consistent – VA provided evidence that it conducted organizational change management activities mostly consistent with leading practices. Partially consistent – VA provided evidence that it conducted organizational change management activities consistent with some of the leading practice criteria, but some key parts were not followed. Not consistent – VA did not provide sufficient evidence that it followed leading practices. Source: GAO analysis of VA data. | GAO-23-106785 VA's Organizational The program's organizational change management activities were not Change Management fully consistent with leading practices for a variety of reasons. First, VA did not have a VA-driven strategy for how its efforts would supplement the Activities Were Not Fully contractor-led change management activities for deployment. According Consistent with Leading to EHRM program officials, the contractor's change management Practices for Various activities focused on the activities required to deploy the new system. Reasons However, these activities did not address user challenges with transitioning to new workflow processes. Further, EHRM officials noted that the program office had experienced transition in change management leadership and vacancies in their change management staffing. This limited the resources available for coordinating and implementing change management activities. Until the program implements all eight of the leading practices for change management, future deployments could be at risk of similar change management challenges. This could hinder users' ability to effectively use Page 8 GAO-23-106785 the system, impede their knowledge of new workflows, and limit the utility of system improvements. Further, the results of post-deployment questionnaires demonstrate the need for improvements in organizational change management activities. Specifically, according to VA-reported data, users provided feedback that was below average regarding their abilities to use the new EHR system. 14 Based on the program's research, a score of 68 out of 100 was considered average and scores below 68 were below average (see table 2). Table 2: Department of Veterans Affairs Electronic Health Record (EHR) Modernization Program Post-deployment Feedback on New EHR System Average summed system usability Site Date scale score Mann-Grandstaff VA Medical Center February - March 2021 24.38 Jonathan M. Wainwright (Walla Walla) May - June 2022 32.33 Central Ohio (Columbus) July 2022 24.14 Roseburg July - August 2022 23.19 Southern Oregon (White City) August 2022 24.72 Source: GAO analysis of Department of Veterans Affairs reported data. | GAO-23-106785 In our interviews, users expressed concern about staff morale and burnout. One user reported working 60 hours a week and trying not to drown in carrying out duties because completing chart reviews, which used to take 15-30 minutes using the old system, was now taking hours or even days. Other users said that providers were volunteering their time, and one user said this was because tasks took 10-15 percent more time to complete. Finally, users noted that staff in their department had resigned, specifically due to problems with the new EHR system. Additional details about users' satisfaction with the new system are discussed in the next section of this statement. 14The response rates to these questionnaires ranged from 12 percent to 22 percent. Page 9 GAO-23-106785 Accordingly, we recently made seven recommendations to VA regarding change management activities. 15 Specifically, we recommended that the Secretary of VA: • document a VA-specific change management strategy to formalize its approach to drive user adoption; • ensure that the department's planned improvements to communication of system changes meets users' needs for the frequency of the updates provided; • take steps to improve change readiness scores prior to future deployments; • ensure steps taken by the EHRM program and Oracle Cerner to increase workforce skills and competencies through improved training and related change management activities have been effective; • address users' barriers to change, by ensuring planned completion of all actions identified in the Secretary's Strategic Review; • develop a plan, including a timeline, for establishing (1) targets for measuring the adoption of changes and (2) metrics and targets to measure the resulting outcomes of the change; and • measure and report on outcomes of the change and take actions to support users' ability to use the system to reinforce and sustain the change. VA concurred with our recommendations. 15These recommendations were conveyed in our March 10, 2023, briefing to congressional committees and members and will be published in a May 2023 report. Page 10 GAO-23-106785 GAO and federal IT guidance recognize the importance of defining Although Users Were program goals and related performance targets and using such targets to Dissatisfied with the assess progress in achieving the goals. 16 Also, leading practices identify continuous customer feedback as a crucial element of IT project success, New System, VA Has from project conception through sustainment. Particularly for IT programs Not Yet Established like EHRM, where development activities are ongoing, customer (i.e., end user) perspectives and insights can be solicited through various methods. Goals for Such methods include interviews and satisfaction surveys, to validate or Improvement raise questions about the project's implementation. Further, leading practices emphasize that periodic user satisfaction data be proactively used to improve performance and demonstrate the level of satisfaction the project is delivering. Measuring user satisfaction with the system is essential for monitoring progress towards pre-established goals or targets and allows programs to understand whether users' operational needs have been met. VA Has Taken Steps to VA has taken steps to obtain feedback on the performance and Obtain Feedback; Users implementation of EHRM. Specifically, in September 2022, VA conducted a survey of users from two regions, VISN 10 and VISN 20, where the new Generally Expressed EHR system had been deployed. In addition, VHA conducted another Dissatisfaction with the survey in September 2021 and September 2022 to assess Mann- New System Grandstaff employees' perceptions of the implementation of the new EHR. In September 2022, VA worked with a contractor to conduct a user satisfaction survey to determine user satisfaction with the Oracle Cerner system. In December 2022, the contractor reported on VA's results in comparison to other health care systems. The results of the survey indicated that users were not satisfied with the performance of the new system or the training for the new system. For example, about 79 percent (1,640 of 2,066) of users disagreed or strongly disagreed that the system 16GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, GAO-04-394G (Washington, D.C.: March 2004); Executive Office of the President, Office of Management and Budget, Evaluating Information Technology Investments, A Practical Guide (November 1995), Office of Management and Budget, Preparation, Submission, and Execution of the Budget, OMB Circular No. A-11 (August 2022); and General Services Administration, Modernization and Migration Management (M3) Playbook, accessed Oct. 20, 2022, https://www.ussm.gov/m3. Page 11 GAO-23-106785 enabled quality care. 17 In addition, about 89 percent (1,852 of 2,074) of users disagreed or strongly disagreed that the system made them as efficient as possible. 18 In addition, fewer VA users reported that they agreed that the system enabled them to deliver high-quality care when compared to DOD and other health care systems. For example, about 23 percent (1,000 of 4,432) of DOD users agreed that the system enabled quality care. In response to the survey of Mann-Grandstaff users' morale in September 2021, most users noted that as a result of the new EHR implementation, their morale, job satisfaction, and level of burnout had worsened (see table 3). 19 Table 3: Department of Veterans Affairs Veterans Health Administration Survey Feedback on New Electronic Health Record (EHR) System, as of September 2021 Improved Not changed Worsened Number Percent Number Percent Number Percent Total As a result of the EHR 6 0.7% 133 16.0% 691 83.3% 830 implementation, my morale has: As a result of the EHR 6 0.7% 173 20.8% 652 78.5% 831 implementation, my job satisfaction has: As a result of the EHR 4 0.5% 154 18.6% 670 80.9% 828 implementation, my level of burnout has: Source: GAO analysis of Department of Veterans Affairs' data. | GAO-23-106785 17In response to the statement, the EHR enables me to deliver high-quality care, 6 percent (120 of 2,066) users agreed, 15 percent (306 of 2,066) were indifferent, and 79 percent (1,640 of 2,066) disagreed or strongly disagreed. 18In response to the statement, the EHR makes me as efficient as possible, 4 percent (92 of 2,074) users agreed, 6 percent (130 of 2,074) were indifferent, and 89 percent (1,852 of 2,074) disagreed or strongly disagreed. 19About 56 percent (833 of approximately 1,500) recipients responded to this survey. Page 12 GAO-23-106785 In September 2022, most users still noted that as a result of the new EHR implementation, their morale, job satisfaction, and level of burnout had worsened (see table 4). 20 Table 4: Department of Veterans Affairs Veterans Health Administration Survey Feedback on New Electronic Health Record (EHR) System, as of September 2022 Improved Not changed Worsened Number Percent Number Percent Number Percent Total As a result of the EHR implementation, 91 12.3% 119 16% 532 71.7% 742 my morale has: As a result of the EHR implementation, 90 12.1% 142 19.2% 509 68.7% 741 my job satisfaction has: As a result of the EHR implementation, 87 11.8% 123 16.7% 528 71.5% 738 my level of burnout has: Source: GAO analysis of Department of Veterans Affairs' data. | GAO-23-106785 When asked to rate the improvement in the EHR since they began using the new medical record, of 741 respondents, 231 (31 percent) said no improvement, 372 (50 percent) said minimal improvement, 49 (7 percent) said moderate improvement, and 89 (12 percent) said great improvement. Similarly, our interviews from the first three deployment sites indicated that users were not satisfied with the new system. Specifically, 51 of 63 users said that they disagreed or strongly disagreed that overall they were satisfied with the new EHR system. In addition, 48 of 63 users said they disagreed or strongly disagreed that the new EHR system met the expectations they had prior to and during go-live. Table 5 provides the results from our interviews regarding user satisfaction of the new system. 20About 52 percent (742 of approximately 1,440) recipients responded to this survey. Page 13 GAO-23-106785 Table 5: Users Responses to Statements on New Electronic Health Record (EHR) System Strongly Agree Neither agree Disagree Strongly No basis Total agree nor disagree disagree to judge The new EHR system meets the 2 9 3 23 25 1 63 expectations I had prior to and during go- live. The new EHR system is available when I 1 18 11 29 4 0 63 need it. Health data was migrated accurately from 0 11 6 22 21 3 63 the old EHR system to the new EHR system. Compared to the old system the new EHR 1 1 6 21 33 1 63 system requires fewer steps to accomplish what I need to do. Overall, I am satisfied with the new EHR 1 6 5 31 20 0 63 system. Source: GAO analysis of interviews with Department of Veterans Affairs officials. | GAO-23-106785 Further, Mann-Grandstaff users' responses to structured interview questions only minimally indicated improved satisfaction or changes in the perceptions of the effect of the new EHR system on productivity or quality of care from our interviews in 2021 to our interviews in 2022. Specifically, as shown in table 6, in 2021, most users (23 of 26) said they disagreed or strongly disagreed that overall they were satisfied with the new EHR system. In 2022, most users (18 of 23) said they disagreed or strongly disagreed that they were satisfied with the new EHR system (see table 6). Table 6: Mann-Grandstaff User Satisfaction with New Electronic Health Record (EHR) System in 2021 and 2022 Do you agree or disagree with the following statement based on your current experience using the new EHR system? Mann-Grandstaff 2021 Mann-Grandstaff 2022 Overall, I am satisfied with the new EHR system. (Out of 26) (Out of 23) Strongly agree 0 0 Agree 0 1 Neither agree nor disagree 3 4 Disagree 6 9 Strongly disagree 17 9 Source: GAO analysis of interviews with Department of Veterans Affairs officials. | GAO-23-106785 Page 14 GAO-23-106785 In addition, participants in our structured interviews provided examples of challenges they experienced with the pharmacy system module that contributed to user dissatisfaction with the system. For example, system users told us • Processing prescriptions took much longer in the new EHR system, leading to increased backlogs and decreased efficiency, which led to patient safety concerns because the pharmacy could not complete prescriptions in a timely fashion. • The pharmacy department at one facility increased from 15 staff to 60 to manage increased workload associated with using the new system. • Multiple instances of double prescriptions, incorrect medication orders, and veterans receiving medications that were no longer prescribed had occurred. • Medications were mailed to the wrong address and veterans did not received medications in some cases. • Medications defaulted to being an inpatient medication rather than being available at the pharmacy for the patient to pick up. • Instructions for medications were sometimes based in Pacific standard time, when the medical facility was in the Eastern time zone. • VA users could mistakenly order medications that are only available to DOD. VA's own assessment also identified multiple pharmacy-related concerns such as inefficient processes requiring an increase in full-time employees, medication information not carrying over to the next appointment, and discontinued prescriptions without probable replacements. In July 2022, VA contracted with Oracle Cerner to make seven enhancements to address challenges with the pharmacy system module. Three of the seven enhancements were deployed in February 2023. VA plans to deploy the remaining four in August 2023. VA documentation noted that, as of March 2023, the initial three enhancements had provided small, incremental improvements but that many additional improvements were needed for the new EHR system to become safe, efficient, cost-effective, and compliant. However, as of March 2023, VA had not yet determined when the additional improvements to address several of the key pharmacy-related safety issues identified in its assessment will be completed. Page 15 GAO-23-106785 VA Has Not Established VA has not established targets (i.e., goals) to assess user satisfaction. Goals for User Satisfaction EHRM provided several reasons for why the program had not established specific goals for user satisfaction for the system: • In February 2022, EHRM program officials stated there was an opportunity for additional metrics such as user satisfaction targets in the future. • In October 2022, EHRM program officials stated they were focused on addressing technical changes to improve the system usability before establishing targets or goals for user satisfaction. Nevertheless, until VA establishes goals for user satisfaction, the department will be limited in its ability to objectively measure progress toward improving EHRM users' satisfaction with the system. The department will also lack a basis for determining when satisfaction has improved. Such a basis would help ensure that the system is not prematurely deployed to additional sites, which could risk patients' safety. Accordingly, we recently recommended that the Secretary of VA establish user satisfaction targets (i.e., goals) and demonstrate improvement toward meeting those targets prior to future system deployments. 21 VA agreed with this recommendation. Efforts to identify and address system issues can be supported by VA Did Not activities such as resolving trouble tickets quickly and conducting an Adequately Identify independent operational assessment. VA did not adequately identify and address system issues. Specifically, VA did not ensure that trouble tickets and Address System for the new EHR system were resolved within timeliness goals, but Issues subsequently worked with the contractor to reduce the number of tickets that were over 45 days old. Additionally, although VA has assessed the system for user performance at two sites, as of January 2023, VA had not conducted an operational assessment to evaluate if the new EHR system satisfies the intended use and user needs in the operational environment. VA and Its EHR System VA's contract with Oracle Cerner addressed the support and resolution of Contractor Have Worked trouble tickets during and after implementation of the new EHR system. to Improve Trouble Ticket Resolution 21These recommendations were conveyed in our March 10, 2023, briefing to congressional committees and members and will be published in a May 2023 report. Page 16 GAO-23-106785 Based on impact and urgency, each ticket is assigned a priority of critical, high, medium, or low. 22 According to a service level agreement (SLA) between VA and Oracle Cerner, resolution timeliness goals varied depending on the ticket priority levels as follows: • Critical tickets: 100 percent of trouble tickets resolved or mitigated through VA approved mitigation strategy within 5 hours and closed within 24 hours. 23 • High tickets: 90 percent of trouble tickets resolved within 16 hours and no single ticket exceeds 64 hours. • Medium tickets: 80 percent of trouble tickets resolved within 4 business days and no single ticket exceeds 60 calendar days. • Low tickets: 80 percent of trouble tickets resolved within 6 business days and no single ticket exceeds 60 calendar days. However, VA determined that during the 25 month period from December 2020 to December 2022, Oracle Cerner did not meet the SLA established for the resolution of system trouble tickets. Specifically, Oracle Cerner did not meet the SLA for: • critical severity trouble tickets for 4 of the 25 months 22Critical - A patient safety condition exists or greater than 25 percent of concurrent users across a medical center are unable to process transactions or access managed solutions critical to their ability to conduct daily business; and no bypass or alternative is available. High - When (15-25 percent) of concurrent users across a VAMC and associated facilities are unable to process transactions or access managed solutions required to conduct daily business or a component of managed software required to compete a crucial workflow is non-functional for more than one user and no bypass or alternative is available. Medium - A component, minor solution, or procedure is down, unusable, or difficult to use but, no immediate impact on service delivery, financial, or patient care. Critical and high problems that have an acceptable workaround, or bypass available will be assigned as a moderate incident. Low - A component, procedure or personal application (not critical to Client) is unusable. No impact to business, single Incident failure, and an acceptable workaround, alternative, or bypass is available. 23A ticket is considered 'resolved' when Cerner places the ticket in a 'Client Action' status for the client to approve / confirm the issue is addressed. A ticket is considered 'completely resolved' when VA has approved and confirmed that a trouble ticket placed in "Client Action" has been fully addressed. 'Completely resolved' and 'closed' are used interchangeably. In the trouble ticket data, 'closed' is a ticket which has been resolved and cannot be reopened. Page 17 GAO-23-106785 • high severity trouble tickets for 15 of the 25 months • medium severity trouble tickets for 21 of the 25 months, and • low severity trouble tickets for 24 of the 25 months. To address a higher-than-expected volume of tickets that were not addressed within 60 calendar days or less, in August 2022, Oracle Cerner developed a 120-day plan to reduce the number of open tickets that were 45 days or older. Oracle Cerner developed its plan in response to a VA letter of concern regarding the new EHR system's performance. As of January 2023, Oracle Cerner had reduced the number of tickets that were 45 days or older from 714 to 108. 24 Nevertheless, as of December 2022, VA had over 1,400 open tickets, which was more than the number of tickets at the end of 2020 and 2021. Figure 1 depicts the number of open trouble tickets per month from October 2020 to December 2022. Figure 1: Department of Veterans Affairs Electronic Health Record Modernization Open Trouble Tickets per Month Oracle Cerner officials provided explanations for the difficulties with meeting the SLA. 24According to Cerner's plan some incidents could be converted to change requests, if appropriate. Page 18 GAO-23-106785 • VA's IT systems are more complex than DOD's, which contributes to a larger number of trouble tickets. • Oracle Cerner relies on the local informatics staff to help triage the tickets, but some VA sites had little informatics support, which increases the burden on the Oracle Cerner help desk. 25 • Oracle Cerner needed to apply additional staffing resources to address the problem. Additionally, according to VA's strategic review status report, the department recognized that its capacity to resolve the volume of tickets at the five deployment sites was overwhelmed due to, among other things, an insufficient number of subject matter experts. Until the program resolves trouble tickets according to established time frames, users' system issues will not be resolved in a timely manner. Additionally, there is a risk that VA will not be able to address users' system issues effectively going forward, particularly when larger sites go live. Accordingly, we recently recommended that the Secretary of VA ensure that future system trouble tickets are resolved within established timeliness goals. 26 VA concurred with our recommendation. VA Has Not Conducted According to leading practices for software verification and validation, a Independent Operational product should be evaluated to determine whether it satisfies the intended use and user needs in the operational environment. 27 An operational Assessments to Validate assessment is an evaluation of operational effectiveness and operational That the System Meets suitability made by an independent operational test activity with user User Needs support as required. 28 The EHRM program's master test plan from May 2021 described plans to execute an independent post-production validation and operational 25Health Informatics is a multidisciplinary and integrative field that focuses on health information management and IT in support of health care. The field of health informatics draws from computer, cognitive, and social sciences for the development, change management, implementation, configuration, deployment and evaluation of systems that manage health information. 26These recommendations were conveyed in our March 10, 2023, briefing to congressional committees and members and will be published in a May 2023 report. 27IEEE Standards Association, IEEE Standard for System, Software, and Hardware Verification and Validation (IEEE Std. 1012-2016), (September 2017). 28Defense Acquisition University Glossary, accessed September 1, 2022, https://www.dau.edu/acquipedia/pages/articlecontent.aspx?itemid=46. Page 19 GAO-23-106785 assessment to assess the degree to which the new EHR met the users' needs in their daily operational use in the production environment. According to the program's test plan, the purpose of the operational assessment was to evaluate the system's efficiency, effectiveness, usability, user satisfaction, and training. However, VA has not conducted an operational assessment and, as of January 2023, did not plan to do so. EHRM program officials said that they did not plan to execute an independent operational assessment because such an assessment would be duplicative to existing post-go-live evaluations and change assessment surveys, and disruptive to site operations. Further, the EHRM Master Test Plan was updated to remove the requirement for an operational assessment. In July 2022, officials from VHA and the EHRM program office conducted a post-go-live study at the Columbus deployment site. These officials observed slow system response, system errors, user interface issues, and inefficient workflows that affected the end user experience. In addition to these observations, the study report stated that the scope of the assessment was limited and recommended further usability assessments. Following the July 2022 study, VA conducted a review focused on standardization, usability, and safety issues at the five deployment sites. 29 The study team reviewed 300 issues and prioritized 30 to address that were related to patient safety. Additionally, according to EHRM program officials, in September 2022 they visited the Columbus deployment site to obtain feedback from users on high-risk workflows. Program officials said they also conducted an assessment at two sites in an effort to improve system performance. However, because these assessments were not conducted independent of the program, they lack the objective evaluation and analysis characteristic of an independent operational assessment. Until an independent operational assessment of the new EHR system is conducted, VA will be limited in its ability to validate that the system is operationally suitable and effective, and to identify, track, and resolve key operational issues. An operational assessment, particularly if conducted by an independent entity, would enable the department to systematically catalog, report on, and track resolution of assessment findings with 29VA referred to this review as the EHRM Sprint Project. The Sprint Project work streams included VHA EHR governance processes, medical order issues, clinical episode review team review and assessment actions, and collaborative readiness. Page 20 GAO-23-106785 greater rigor, transparency, and accountability. Accordingly, we recently recommended that the Secretary of VA reinstitute plans to conduct an independent operational assessment to evaluate the suitability and effectiveness of the new EHR system for users in the operational environment. 30 VA agreed with this recommendation. In summary, the program's organizational change management activities were not fully consistent with eight leading practices. These practices are especially important given that VA's transition to the new EHR was challenging for users at the initial deployment sites. In addition, VA undertook several efforts to assess user satisfaction with the new system, but results indicated that users were dissatisfied with the system. Further, VA had not established targets or goals for user satisfaction. Consequently, it is not evident what basis the department will use to determine when satisfaction has sufficiently improved to support a decision to deploy the system at additional sites. Such a basis is critically important to ensuring that systems not be deployed prematurely and pose unnecessary risks to patient health and safety. Finally, VA did not ensure that system issues had been addressed within established timeliness goals nor has it conducted an independent operational assessment, which could be beneficial in validating that the system satisfies user needs in the operational environment. We will continue to assess these issues and to track VA's progress in response to our recommendations. Chairman Rosendale, Ranking Member Cherfilus-McCormick, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have. If you or your staff have any questions about this testimony, please GAO Contact contact Carol C. Harris, Director, Information Technology and and Staff Cybersecurity, at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found Acknowledgments on the last page of this statement. GAO staff who made key contributions to this testimony are Mark Bird (Assistant Director), Merry Woo (Analyst- in-Charge), Tim Barry, Chris Businsky, Quintin Dorsey, Rebecca Eyler, Ash Harper, Igor Koshelev, Christy Ley, Monica Perez-Nelson, Rachael Scott, Eric Trout, Walter Vance, Adam Vodraska, and Charles Youman. 30These recommendations were conveyed in our March 10, 2023, briefing to congressional committees and members and will be published in a May 2023 report. Page 21 GAO-23-106785 Appendix I: Objectives, Scope, and Appendix I: Objectives, Scope, and Methodology Methodology The review upon which this testimony was based culminated in a briefing to congressional committees and members. 1 The objectives of our review were to determine the extent to which (1) VA has employed organizational change management strategies for the Electronic Health Record Modernization (EHRM) program consistent with leading practices, (2) VA has assessed satisfaction with the new system, and (3) VA has identified and addressed electronic health record (EHR) system issues. To address the first objective, we conducted a literature search for organizational change management leading practices. 2 We identified leading change management practices that are applicable to organizational transitions, such as VA's EHR system modernization. We then evaluated VA's activities against these practices by examining program plans for organizational change management and discussing the program's approach with cognizant EHRM program officials. To assess whether the program's activities were aligned with its planned approach and leading practices, we reviewed EHRM change management documentation, such as wave deployment plans, stakeholder communication strategy and plan, change impact analyses, site deployment and work plans, and change readiness questionnaire reports. We supplemented our analysis with examples from interviews with users from the Mann-Grandstaff VA Medical Center, Jonathan M. Wainwright VA Medical Center (Walla Walla), and VA Central Ohio Health Care System (Columbus), the three locations where the new system was first deployed. To address the second objective, we obtained and reviewed results of surveys that VA conducted to determine users' satisfaction with the new EHR, including a survey conducted by VHA to assess Mann-Grandstaff 1On March 10, 2023, we provided a briefing on the results of our review to the House and Senate Committees on Appropriations, Subcommittees on Military Construction, Veterans' Affairs, and Related Agencies; House and Senate Committees on Veterans' Affairs; Rep. Jim Banks; and Rep. Susie Lee. We plan to publish the briefing in a May 2023 report. 2Project Management Institute, Inc., Managing Change in Organizations: A Practice Guide (Newtown Square, Pa.: 2013); Office of Personnel Management, Migration Planning Guidance Information Documents, Change Management Best Practices (Oct. 7, 2011); GAO, Business Process Reengineering Assessment Guide, version 3, GAO/AIMD-10.1.15 (Washington, D.C.: May 1997); ISACA, COBIT 2019 Framework (2019); and Prosci, The Prosci ADKAR® Model, A Goal Oriented Change Management Model to Guide Individual and Organizational Change, accessed Feb. 21, 2021, https://www.prosci.com/methodology/adkar. ADKAR® is a registered trademark of Prosci, Inc. Page 22 GAO-23-106785 Appendix I: Objectives, Scope, and Methodology employees' perceptions of the implementation of the Oracle Cerner EHR and post-deployment system usability surveys conducted by the EHRM program office. We obtained documentation regarding the department's administration of its user satisfaction surveys to determine that the data were sufficiently reliable for our purposes. We met with EHRM program officials and VHA officials to discuss whether the department had established any goals for user satisfaction. We conducted structured interviews with selected users from the Mann- Grandstaff VA Medical Center, Jonathan M. Wainwright VA Medical Center (Walla Walla), and VA Central Ohio Health Care System (Columbus), the three locations where the new system was first deployed. Specifically, we conducted structured interviews with 63 users at these three locations between April and August 2022. We reviewed the results of our structured interviews to identify examples of user dissatisfaction. The methodology for selecting interviewees was as follows: we received a list of Mann-Grandstaff VA Medical Center employees who have been involved with national EHR councils. First, we conducted two pre-test interviews with leadership staff and made minor revisions to our structured interview instrument. We then selected one user from each of the 16 departments represented among the councils. For departments that had multiple users involved in the national councils, a user was randomly selected. In addition, two users were selected based on recommendations from the Mann-Grandstaff Medical Center leadership. Finally, an additional six users were selected based on recommendations from interviewees for a total of 26 interviews between April 2021 and June 2021. Following these interviews, we conducted additional interviews with 23 of the same users between April 2022 and June 2022. While the users' responses cannot be generalized to the entire population of EHR users at the initial deployment site, they represent a broad range of user roles and clinical areas at the sites. Following interviews with Mann-Grandstaff VA Medical Center, we conducted structured interviews with selected EHR users from the Jonathan M. Wainwright Memorial VA Medical Center (Walla Walla) and VA Central Ohio Health Care System Center (Columbus). We conducted 40 interviews in total, 19 from Walla Walla and 21 from Columbus between June 2022 and August 2022. The methodology for selecting interviewees at these two locations was as follows: we requested and received a list of representatives from a variety of clinical areas from both sites. We then interviewed the chief of staff at Page 23 GAO-23-106785 Appendix I: Objectives, Scope, and Methodology each location. In addition, we selected 18 user representatives from Walla Walla and randomly selected 20 users from the list of user representatives from Columbus, excluding those who were not obvious users of the system. While these users' responses cannot be generalized to the entire population of EHR users at these deployment sites, they represent a broad range of user roles and clinical areas at the sites. To address the third objective, we obtained data on system trouble tickets from October 2020 to December 2022. We analyzed VA's data on the contractor's performance meeting time frames established in the service level agreement (SLA) for the contractor to address system trouble tickets. We also obtained a summary of monthly reports from Oracle Cerner to VA on trouble ticket resolution with respect to the SLA. We also analyzed the trouble ticket data for trends in the number of open tickets at the end of each month. We assessed the reliability of the trouble ticket data by reviewing it for obvious errors and missing data and interviewed responsible officials about any discrepancies in the data. We determined the data to be sufficiently reliable for the purposes of our briefing. We also obtained documentation of the EHRM program's testing activities, including test plans and results. We then analyzed the plans, as well as test activities that had already been completed, to determine whether they constituted an independent operational assessment. We supplemented our analyses for our objectives by interviewing relevant VA officials, including the EHRM Integration Office Executive Director, Functional Champion, and Deputy Chief Information Officer. We conducted our performance audit from February 2021 through May 2023 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Page 24 GAO-23-106785 This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. The Government Accountability Office, the audit, evaluation, and investigative GAO's Mission arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is Obtaining Copies of through our website. Each weekday afternoon, GAO posts on its website newly GAO Reports and released reports, testimony, and correspondence. You can also subscribe to GAO's email updates to receive notification of newly posted products. Testimony Order by Phone The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. Contact FraudNet: To Report Fraud, Website: https://www.gao.gov/about/what-gao-does/fraudnet Waste, and Abuse in Automated answering system: (800) 424-5454 or (202) 512-7700 Federal Programs A. Nicole Clowers, Managing Director, ClowersA@gao.gov, (202) 512-4400, U.S. Congressional Government Accountability Office, 441 G Street NW, Room 7125, Washington, Relations DC 20548 Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Stephen J. Sanford, Managing Director, spel@gao.gov, (202) 512-4707 Strategic Planning and U.S. Government Accountability Office, 441 G Street NW, Room 7814, External Liaison Washington, DC 20548 Please Print on Recycled Paper.