United States Government Accountability Office Report to the Chairwoman, Subcommittee on Health, Committee on Veterans' Affairs, House of Representatives February 2022 VA HEALTH CARE Clinical Contact Center Modernization Could Benefit from a Shared Schedule and Additional Guidance to Networks GAO-22-104620 GAO Highlights Highlights of GAO-22-104620, a report to the Chairwoman, Subcommittee on Health, Committee on Veterans' Affairs, House of Representatives Why GAO Did This Study Veterans reach out to VHA's contact centers via phone, video, and chat to access needed health care services. Historically, these contact centers provided services inconsistently to veterans and operated generally under local oversight. The COVID-19 pandemic saw veterans increasingly reach out to contact centers, which highlighted a need for VHA to examine them. GAO was asked to review several aspects of clinical contact center operations and oversight. This report examines (1) VHA's modernization planning and (2) VHA's guidance to networks related to assessing contact centers' performance, among other things. GAO reviewed Department of Veterans Affairs (VA) modernization documents and interviewed officials from OVAC; VA's Office of Information and Technology; VA's Veterans Experience Office; and six networks, selected for variation in geography and their reported progress in implementing the modernization effort. What GAO Recommends GAO is making four recommendations, including that VHA (1) update and share a schedule for the Clinical Contact Center Modernization effort beyond December 31, 2021; and provide networks with (2) guidance identifying finalized performance metrics and targets, and (3) guidance defining networks' performance assessment roles and responsibilities. VA concurred with GAO's recommendations and identified actions it is taking to address them. View GAO-22-104620. For more information, contact Sharon M. Silas at (202) 512-7114 or silass@gao.gov. VA HEALTH CARE Clinical Contact Center Modernization Could Benefit from a Shared Schedule and Additional Guidance to Networks What GAO Found The Veterans Health Administration (VHA) initiated the Clinical Contact Center Modernization effort in May 2020. Through the effort, VHA aims to improve veterans' access to virtual care and standardize operations of clinical contact centers (contact centers), such as services offered, staffing, training, and how performance is assessed. VHA's Office of Veterans Access to Care (OVAC)-which is leading the modernization effort-has taken several steps in planning for the modernization effort. One of these steps includes setting a milestone of December 31, 2021, for each of its Veterans Integrated Service Networks to establish a contact center that provides four core services (see table). Officials from six selected networks stated they anticipated meeting this date. OVAC officials stated that modernization is an ongoing effort, and they are working to possibly enhance the four core services with additional capabilities, such as mental health care, in the future. Moving forward, OVAC and networks could benefit from having a shared schedule that includes milestones, activities, and related timelines for networks implementing any future changes. For example, a schedule would provide a shared sequence of events that could help networks better prioritize modernization activities. Core Services That Veterans Integrated Service Networks Are Expected to Offer by December 31, 2021, as Part of VHA's Clinical Contact Center Modernization Effort Core service Veterans should be able to Primary care scheduling and administration Schedule, reschedule, and cancel appointments; and obtain information about VHA services. Clinical triage Discuss symptoms and concerns, and receive recommendations on the best course of action. Primary care virtual clinic visits | Meet virtually with providers to discuss health care needs. Pharmacy Request and track the status of medication refills and renewals; and ask medication-related questions. Source: GAO analysis of Veterans Health Administration (VHA) guidance | GAO 22-104620 Additionally, OVAC has not developed certain guidance, which may hinder its ability to assess contact centers' performance in providing services, including e Finalized performance metrics and targets that would clearly define performance expectations for centers and allow OVAC to assess center performance in a standardized way. e Finalized networks' roles and responsibilities for assessing performance. According to OVAC officials, upgrades to contact centers' telephone and quality management systems are important components of the modernization effort because they will provide data for assessing center performance. For example, the upgraded telephone system will provide data on call queues for helping to determine staff productivity. Officials told GAO that once the upgrades are fully implemented, which they anticipated would be completed in December 2021, they would have the data they need to finalize performance metrics and targets and define responsibilities for performance assessment. By providing additional guidance to networks, VHA may be able to improve its oversight of centers and determine if centers' performance is improving-two intended benefits of the modernization effort. United States Government Accountability Office Contents Letter Background 4 VHA Has Taken Several Steps in Its Planning for Contact Center Modernization; Moving Forward, It Could Benefit from a Shared Schedule 8 VHA Has Provided Staffing Guidance and Training Guidance to Networks 10 VHA Has Provided Limited Guidance to Networks for Assessing Contact Center Performance in a Standardized Way 12 Conclusions 15 Recommendations for Executive Action 15 Agency Comments 16 Appendix | Comments from the Department of Veterans Affairs 17 Appendix II GAO Contact and Staff Acknowledgments 20 Tables Table 1: Examples of Technology Upgrades That Are Part of Veterans Health Administration (VHA) Clinical Contact Center Modernization Effort 7 Table 2: Examples of Draft Performance Metrics for VHA Clinical Contact Center Modernization Effort 12 Figures Figure 1: Veterans Health Administration (VHA) Clinical Contact Center Modernization Core Services 5 Figure 2: Entities with Primary Clinical Contact Center Modernization Responsibilities 6 Page i GAO-22-104620 VHA Clinical Contact Centers Abbreviations COVID-19 Coronavirus Disease 2019 OIT Office of Information and Technology OVAC Office of Veterans Access to Care VA Department of Veterans Affairs VHA Veterans Health Administration This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page ii GAO-22-104620 VHA Clinical Contact Centers 1 U.S. GOVERNMENT ACCOUNTABILITY OFFICE 441 G St. N.W. Washington, DC 20548 February 1, 2022 The Honorable Julia Brownley Chairwoman Subcommittee on Health Committee on Veterans' Affairs House of Representatives Dear Madam Chairwoman: Within the Department of Veterans Affairs (VA), the Veterans Health Administration (VHA) operates one of the nation's largest health care systems, providing care to 9 million veterans at 1,293 facilities, including VA medical centers and community-based outpatient clinics. Many of these facilities and 18 Veterans Integrated Service Networks, referred to as networks, developed their own clinical contact centers, referred to as contact centers. Contact centers serve as entry points for veterans and their families, as they provide frequently used administrative and clinical services. According to VA, historically services differed among contact centers, but generally, veterans reached out to centers via phone, video, and chat to get assistance with some or all of the following services: appointment scheduling, clinical triage, and pharmacy services.? For example, some centers have staff who can schedule appointments for veterans at VA medical centers, some have staff who can refill prescriptions, and some have registered nurses who can provide over the phone clinical triage services-evaluation of symptoms and assessment of health care concerns. 1Networks are responsible for overseeing medical facilities and manage the day-to-day functions of facilities within their defined geographic area. Within VHA, there are other types of contact centers, some of which focus on specific veteran populations-for example, the Office of Women's Health operates a contact center for veterans to obtain information about women's health services. Some of VHA's contact centers focus on particular services-for example, according to VHA officials, the Health Eligibility Center has contact centers that allow veterans to enroll in health benefits over the phone. 2Department of Veterans Affairs, VA Health Connect Clinical Contact Center Modernization Guidebook, Release 3 (Washington, D.C.: Sept. 17, 2021). Page 1 GAO-22-104620 VHA Clinical Contact Centers Given the types of services contact centers provide and their position as entry points to care, they have played an important role in VHA's Coronavirus Disease 2019 (COVID-19) response, when VHA began providing less in-person care and more virtual care. According to VHA, this resulted in an increase in the number of calls to its contact centers, as veterans tried to reschedule appointments or convert in-person appointments to virtual ones. According to VHA's Veterans Health Administration Coronavirus Disease 2019 Response Report, the COVID- 19 response highlighted issues with the existing organization of its centers and the need to accelerate implementation of additional virtual care options.3 Specifically, there was no single entity within the agency that was responsible for monitoring all contact centers. According to the report, this resulted in VHA having limited visibility into many aspects of contact centers' operations and performance, including how many contact centers were operating across the nation. Moreover, VA medical center or VA network leadership locally managed many contact centers' operations and performance, which, according to VHA, resulted in lack of consistent methods of service delivery and staffing.4 According to VHA, the lack of consistency among centers made it challenging to ensure veterans had consistent experiences receiving services from centers. In May 2020, VHA initiated the Clinical Contact Center Modernization effort, subsequently referred to as the modernization effort, to address the lack of consistency in services provided by contact centers and veterans' experiences receiving services. Through the modernization effort, VHA intends to implement a standardized contact center model that will function as a "virtual front door" for veterans. These virtual care services consist of appointment scheduling and administrative services, clinical triage, virtual clinic visits, and pharmacy services.5 3Veterans Health Administration, Veterans Health Administration Coronavirus Disease 2019 Response Report (Washington, D.C.: Oct. 27, 2020). 4Department of Veterans Affairs, VA Health Connect Clinical Contact Center Modernization Guidebook, Release 3. 5Prior to the modernization effort, some facilities had centers that provided virtual care services, such as appointment scheduling and clinical triage. In June 2020, September 2020, and March 2021, VA released memos outlining modernization expectations for networks. For example, the September 2020 memo stated that networks are expected to provide scheduling services and virtual clinic visits for primary care beginning December 31, 2021. Page 2 GAO-22-104620 VHA Clinical Contact Centers You asked us to review VHA's contact center operations, including any initiatives VHA has undertaken to improve contact centers' operations. In our report, we examine VHA's contact center modernization effort: VHA's planning for the contact center modernization effort; 2. what guidance, if any, VHA has provided to its networks to standardize contact center staffing and training; and 3. what guidance, if any, VHA has provided to its networks to assess contact center performance. To examine VHA's planning for the modernization effort, we reviewed VA documents-the VA Health Connect Clinical Contact Center Modernization Guidebook, referred to as the guidebook, and VA memos issued in June 2020, September 2020, and March 2021-that outline the agency's expectations and milestones.® In addition, we interviewed officials from VHA's Office of Veterans Access to Care (OVAC), which is leading the modernization effort, and officials from a nongeneralizeable selection of six of the 18 networks about their roles in the modernization effort and steps they have taken to plan for the modernization effort. We selected these networks to ensure geographic diversity and variation in the extent of their progress as of February 2021 towards being able to provide the four core services.' We examined the extent to which VHA's planning steps aligned with best practices we identified in prior work for planning successful government programs.® These practices call for agencies to have a project schedule outlining milestones, the activities needed to meet those milestones, and timelines informing project stakeholders how activities should be prioritized and when each activity should be started and completed. 6Department of Veterans Affairs, VA Health Connect Clinical Contact Center Modernization Guidebook, Release 3. 7To ensure variation in network progress, we reviewed the number of centralized services provided by networks prior to the modernization effort and networks' self-assessments of their readiness to provide each of the four core services by December 31, 2021. Each network ranked the readiness for each of their four core services on a scale of 0 (not offered) to 5 (available and centralized). We summed the numerical readiness scores for each service by network to develop three categories: 0-5 (low readiness), 6-12 (medium readiness), and 13-20 (high readiness). We selected two networks from each of these categories to include in our review. 8GAO, Schedule Assessment Guide: Best Practices for Project Schedules, GAO-16-89G (Washington, D.C.: Dec. 22, 2015). Page 3 GAO-22-104620 VHA Clinical Contact Centers To examine what guidance, if any, VHA has provided to its networks to standardize contact center staffing and training and to assess contact center performance, we reviewed VA and VHA guidance. This included the guidebook and timelines VA established for implementing upgraded systems to collect and assess performance data. We interviewed officials from OVAC, VA's Veterans Experience Office, and VA's Office of Information and Technology (OIT) about their roles and responsibilities related to developing the guidance and the current status of the guidance. We also interviewed selected networks about what, if any, additional guidance from VHA may be helpful, and we obtained information from the selected networks about what guidance they received. We compared the current status of VHA's staffing, training, and performance assessment guidance against the guidance VHA is expected to provide as outlined in the guidebook. We also reviewed VHA's performance assessment guidance against performance measurement and assessment best practices identified by GAO, which call for agencies to identify performance measures and performance targets. In addition, we interviewed representatives from a veterans service organization, which previously reported on access issues, to obtain their perspectives of VHA contact center operations, veterans' experiences receiving services from contact centers, and the contact center modernization effort. We conducted this performance audit from November 2020 to February 2022 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background In October 2020, VHA published the VA Health Connect Clinical Contact Center Modernization Guidebook, which provides networks and other VA and VHA offices guidance on contact center staffing and training, among other things.'° The guidebook documents elements of the modernization ®8GAO, Managing for Results: Selected Agencies Need to Take Additional Efforts to Improve Customer Service, GAO-15-84 (Washington, D.C.: Oct. 24, 2014). 10Department of Veterans Affairs, VA Health Connect Clinical Contact Center Modernization Guidebook, Release 1 (Washington, D.C.: Oct. 2, 2020). As of October 2021, OVAC officials stated they anticipated a contact center directive outlining policy and responsibilities would be published in early fiscal year 2022. Page 4 GAO-22-104620 VHA Clinical Contact Centers effort, such as expectations and intended benefits, and the roles and responsibilities of networks, OVAC, and other offices in VHA. izati odernization effort expectations. The guidebook outlines the Modernization Mod tion effort tations. The guidebook outlines th Expectations and Intended agency's modernization expectations for contact center operations, Benefits including e centralized virtual access to four core services: primary care appointment scheduling and administration services, clinical triage, primary care virtual clinic visits, and pharmacy services at the network level (see fig. 1); and e use of standardized staffing, technology, data-assessment, and other processes in delivering the four core services. Figure 1: Veterans Health Administration (VHA) Clinical Contact Center Modernization Core Services Telephone . . Primary care scheduling and administration To make, reschedule, and cancel appointments and receive additional information about VHA services. Clinical triage To discuss symptoms and concerns and recieve recommendations on the best course of action. Primary care virtual clinic visits To talk to providers by phone, video, or chat to address needs in detail. Pharmacy services For refills, renewals, tracking, questions, and other medication- related concerns. oud ao \ Source: GAO analysis of Department of Veterans Affairs guidance. | GAO-22-104620 Page 5 GAO-22-104620 VHA Clinical Contact Centers Modernization effort intended benefits. These include e improve veterans' experiences receiving services from centers, e improve centers' performance on relevant metrics, e improve VHA's oversight of centers, and e upgrade technology systems to standardize data collection. Modernization Roles and OVAC, OIT, and networks have primary responsibility related to the Responsibilities modernization effort. (See fig. 2.) Figure 2: Entities with Primary Clinical Contact Center Modernization Responsibilities Cie lt Ma lr me eae Lice) a) Veterans Integrated Office of Veterans Access to Care Service Networks By ere Tatan med m Ke) Affairs Office of Information and Technology Overall responsibility for the Establish centralized contact centers modernization effort. that provide primary care appointment scheduling and administration services, clinical triage, primary care virtual clinic Implement hardware updates and configure new technology systems including Develop guidance, policy, and tools for modernization implementation. This visits, and pharmacy services across all facilities in each network. includes guidance related to staffing changes, training changes, and using upgraded technologies associated with the modernization effort. Source: GAO analysis of Department of Veterans Affairs guidance. | GAO-22-104620 telephone system, quality management system, and customer relationship management See Note: Additional offices within the Department of Veterans Affairs, such as the Veterans Experience Office, are working with the Office of Veterans Access to Care and the Veterans Integrated Service Networks to complete specific modernization activities, such as updating veterans' experience performance metrics and the technology necessary to capture the data. Page 6 GAO-22-104620 VHA Clinical Contact Centers In the Veterans Health Administration Coronavirus Disease 2019 Response Report, the Veterans Health Administration (VHA) reported data on two performance metrics-average speed to answer calls and the rate of incoming calls that are ended before staff can answer-across three categories of calls both prior to and during the pandemic. The three categories of calls were calls related to appointments, pharmacy services, and veterans' symptoms. The data were collected from 101 centers that used a centralized data reporting system that allowed VHA to review their performance data. Prior to the pandemic, VHA established targets: less than 30 seconds for average speed to answer, and 5 percent of incoming calls are ended before staff can answer. During the first week of February 2020, 68 of the 101 centers met the pre-pandemic targets across all three call categories. As part of the pandemic response, VHA modified the targets to less than 2 minutes and under 10 percent. In June 2020, 60 percent of centers met the performance targets across all three call categories. Because VHA does not know the total number of contact centers operating within its system, it was not able to determine if the 101 centers' performance was representative of all centers' performance. Source: VHA. | GAO-22-104620 OVAC has identified several upgraded systems that all centers will be expected to use to collect and review data on center performance, and, according to officials, OVAC is responsible for developing the procedures guiding their use in collaboration with OIT. (See table 1.) This will be a change from the disconnected, legacy systems used throughout VHA prior to modernization, which, according to VHA, prevented it from centrally tracking performance of all its contact centers.11 The upgraded systems are expected to allow for standardized collection and assessment of performance data, such as data on the number of veterans receiving services and staff productivity, by contact center, and, according to officials, they will allow OVAC to have a better understanding of performance issues across networks. Table 1: Examples of Technology Upgrades That Are Part of Veterans Health Administration (VHA) Clinical Contact Center Modernization Effort Technology upgrade Capabilities Telephone system Allow center staff to manage call volume and call queue Quality management system Allow for call recording, call transcription using artificial intelligence, and workforce management tools Customer relationship management system Integrate with veterans' electronic health records; and allow networks to share records of when veterans contacted centers and track veterans' experience across VHA facilities Source: GAO analysis of Department of Veterans Affairs documentation. | GAO-22-104620. Note: According to VHA, contact centers are expected to use the technology upgrades that will help ensure that all centers are collecting the same performance data and using the same systems to assess performance in a standardized way. 11Veterans Health Administration, Coronavirus Disease 2019 Response Report. According to VHA, prior to modernization, only 101 of the potentially hundreds of centers used a centralized contact center data reporting system that allowed it to review their performance data. Page 7 GAO-22-104620 VHA Clinical Contact Centers VHA Has Taken Several Steps in lts Planning for Contact Center Modernization; Moving Forward, It Could Benefit from a Shared Schedule Our review of agency documentation and interviews with OVAC officials shows that OVAC has taken several steps in planning for the VHA modernization effort, which OVAC officials say is ongoing. These steps include establishing initial milestones-points in time that mark key events and the accomplishment of a goal. For example, in a September 2020 memo to all network directors, OVAC identified December 31, 2021, as the date that each network is expected to provide primary care appointment scheduling and administration services, clinical triage, primary care virtual clinic services, and pharmacy services to veterans 24 hours a day, 7 days a week.12 In July 2021, OVAC officials stated they modified the December 31, 2021, milestone such that networks were no longer expected to provide virtual clinic services 24 hours a day, but rather only during regular business hours.'3 As of August 2021, officials from our six selected networks stated they anticipated meeting the December 31, 2021, milestone. In addition to identifying these milestones, OVAC also identified the activities necessary for networks to achieve them. These activities included establishing new employee and role-specific training plans; reassigning registered nurses, pharmacists, and medical support assistants from medical centers to the networks; and ensuring networks measure performance using various metrics, including metrics of staff workload and patient satisfaction.14 Besides expecting networks to have their contact centers offer the four core services by December 31, 2021, OVAC officials told us they are working with networks to possibly enhance the four core services with additional capabilities, such as enhancing the scheduling and administration service to include mental health capabilities, and to further centralize services at a consortium level.15 As of October 2021, OVAC officials stated they are early on in these discussions and did not have 12Department of Veterans Affairs, Veterans Integrated Service Network (VISN) Clinical Contact Center Expectations and Next Steps (VIEWS 03414352) (Washington, D.C.: Sept. 11, 2020). Other milestones OVAC identified include releasing an updated guidebook in June 2021 and publishing a directive in November 2021. 13QVAC defines regular business hours as the time when call volume rises in the morning until the call volume stabilizes in the evening. 14QVAC listed these activities in a self-assessment tool provided to each network. 1SQVAC defines consortiums as groups of networks that combine their resources to provide an increased scale of services. Specifically, a consortium-level contact center would provide the four core services to veterans across multiple networks. Page 8 GAO-22-104620 VHA Clinical Contact Centers any concrete plans to enhance the services. According to OVAC officials, they plan to update their current schedule, which was initially developed in May 2020 and lists activities that OVAC and OIT need to complete, as the modernization effort evolves; however, as of October 2021, they did not have a time frame for updating the schedule. As OVAC continues planning for the modernization effort and considering enhancing the services contact centers provide, it could benefit from updating and sharing its schedule for the modernization effort with networks prior to implementing any changes. Updates to the schedule would identify milestones, activities, and timelines-which list activities in the order in which they are to be conducted, with each activity starting or finishing before its successor activity, and establish how long each activity should take to complete for all entities with modernization effort responsibilities, including networks. We have previously identified that schedules should include the entire required scope of the effort, including what is necessary from all key parties for the effort's successful execution from start to finish; the development of a schedule prior to implementation as a best practice; and the benefits of a schedule.'* These benefits include allowing agencies to gauge progress, identify and resolve potential problems, provide all parties with information on the sequencing of activities and a shared understanding of milestones, and promote accountability at all levels of the program. Specifically, a shared schedule may help networks and OVAC better prioritize modernization effort activities moving forward. OVAC officials stated they did not share the schedule with networks because it included OVAC and OIT responsibilities only. In the absence of a shared schedule, officials from two of the six selected networks in our review noted that they prioritized activities differently from OVAC, which they said created challenges for them. For example, officials from one of these networks prioritized hiring a key scheduling position differently than OVAC, and subsequently had to wait 1 year for the position description to be developed. In addition, officials from a second network stated that it would have been helpful if OVAC had provided additional milestones in the lead up to December 31, 2021. Officials from the previous network and one additional network also stated that the timelines to hire and train staff to 16GAO-16-89G. GAO has additional guidance for agency reform efforts, which reflect aspects of good project planning, such as the need for implementation and communications plans. See GAO, Government Reorganization: Key Questions to Assess Agency Reform Efforts, GAO-18-427 (Washington, D.C.: June 13, 2018). Page 9 GAO-22-104620 VHA Clinical Contact Centers VHA Has Provided Staffing Guidance and Training Guidance to Networks meet modernization milestones has been challenging. Officials from one of those two networks explained that it was challenging to meet these timelines while managing competing priorities. OVAC officials acknowledged that in the lead up to the December 31, 2021, milestone, they could have sought more feedback from networks on which activities to prioritize. Moving forward, updating and sharing a schedule prior to networks beginning implementation of any enhancement of the four core services could help OVAC and networks better prioritize activities and reach modernization effort milestones. We found that OVAC has provided standardized staffing guidance and training guidance to networks in accordance with VHA's modernization guidebook.'" According to the modernization guidebook, OVAC's staffing guidance is designed to help ensure networks hire and organize staff in a standardized way. OVAC has provided networks with staffing guidance that includes the following: e Staffing calculators to assist them in determining the appropriate staffing models to establish a contact center and ongoing monitoring of staffing. e Organizational charts to assist networks in structuring their contact centers and making adjustments in response to center staffing changes. e Space and equipment guidelines to assist networks in determining physical space requirements and necessary equipment, among other items. e Position descriptions to outline for networks a standard set of qualifications, such as educational experience; license requirements; and job responsibilities for each staff position.'® In addition, OVAC has provided standardized training guidance to networks in accordance with the modernization guidebook. According to the modernization guidebook, OVAC's training guidance is intended to ensure that staff receive a standardized orientation process to familiarize themselves with the contact center and its organizational structure, have 17The guidebook contains OVAC-developed staffing and training guidance that outlines minimum expectations networks are expected to meet. 18QVAC officials told us that position descriptions are for GS-level employees, such as medical support assistants, and functional statements are for clinical staff, such as providers. For the purpose of this report, we are referring to them collectively as position descriptions. Page 10 GAO-22-104620 VHA Clinical Contact Centers consistent position-specific skills, and communicate with veterans in a standardized manner. OVAC has provided networks with training guidance that includes the following: e New employee orientation checklists that provide networks recommended orientation items for registered nurses, medical support assistants, pharmacy technicians, pharmacists, and providers. The checklists include standard processes for on-boarding staff, teaching new staff the technology systems, documenting veteran contacts, and supervising staff while handling their first calls with veterans. ¢ Employee competency checklists that provide networks a list of position-specific skills for medical support assistants, registered nurses, pharmacy technicians, and pharmacists.12 For each position, the supervisor is responsible for selecting a method, such as simulation, direct observation, or testing, to rate each employee on the skills listed on the checklist. The ratings for each skill determine if a staff member is competent with that skill or can perform it independently. For example, one competency on the registered nurse checklist is if nurses recognize the interaction of physical and mental health conditions, can identify treatment options, and can educate veterans and their caregivers on self-management and care strategies for conditions. e The Customer Experience Training program that will train staff in how to interact with veterans and their caregivers, including communication skills and strategies for how to handle challenging behaviors. This program was developed by VA's Veterans Experience Office. In addition, as of October 2021, OVAC officials stated they were developing one remaining training related to the VA Health ConnectCRM-the customer relationship management system contact centers are expected to use. This program will train staff in using and understanding the system. This system is intended to allow networks to share records of when veterans contacted networks and track veterans' experiences across VHA. OVAC officials stated they had not provided the training, as they are in the process of updating it after receiving feedback from pilot users of the system. In addition, OVAC stated they would provide each network with the training prior to the network receiving the 19When asked about ongoing employee competency checklists for providers, OVAC provided us with a document that provided initial checklists. Page 11 GAO-22-104620 VHA Clinical Contact Centers Health ConnectCRM software, which OIT officials stated they would begin implementing in December 2021. : Although VHA's OVAC has provided networks with some guidance VHA Has Provided related to assessing contact center performance, such as descriptions of Limited Guidance technology upgrades and their performance assessment functionalities, it has not provided other guidance that is important for helping to ensure to Networks for contact centers' performance is assessed in a standardized way. Assessing Contact Specifically, OVAC has not provided networks with guidance that identifies (1) finalized performance metrics and targets, (2) procedures for Center Performance using the upgraded technology systems to assess performance, and (3) in a Standardized networks' roles and responsibilities related to assessing performance. Wa Without additional guidance to networks, OVAC's ability to realize the y modernization intended benefits of improving center performance and oversight may be hindered. Metrics and targets. As of October 2021, OVAC had not provided networks with finalized metrics and related targets-that define performance expectations-which networks are expected to use to assess performance. OVAC officials stated that as of October 2021, they were revising draft performance metrics and are looking at private sector benchmarks to establish targets. (See table 2 for examples of the draft performance metrics.) eC Table 2: Examples of Draft Performance Metrics for VHA Clinical Contact Center Modernization Effort Metric Definition Average speed of answer Average time in seconds an incoming or transferred contact waits in the queue before staff answer? Abandonment rate Percentage of inbound contacts terminated by the person originating the contact before staff answer? First contact resolution Whether the veteran's clinical or administrative needs were resolved on the initial contact Scheduling success Percentage of time appointments are successfully booked within their recommended follow-up period Staff turnover Attrition rate during a designated time period Veteran experience Level of satisfaction veterans experience during a single contact Source: GAO analysis of Veterans Health Administration (VHA) documents. | GAO-22-104620. ®Contact center performance metric that VHA required contact centers to track prior to modernization. According to OVAC officials, the upgraded telephone system will provide them with performance data necessary to finalize performance metrics, and once those metrics have been finalized, they will be able to finalize Page 12 GAO-22-104620 VHA Clinical Contact Centers targets. As of October 2021, OVAC officials stated they did not have a time frame for when the metrics and targets would be finalized. As our previous work has shown, having performance metrics with related targets allows agencies to clearly define performance; in this case, it would allow VHA to establish expectations for networks and measure centers' progress in meeting those expectations-a best practice in improving customer service.2° In addition, VHA may be able to determine if one of the intended benefits of the contact centers-improved performance on relevant metrics-is being achieved. Procedures. As of October 2021, OVAC had not provided networks with guidance, such as standard procedures guiding the use of the upgraded technology systems-the telephone, quality management, and customer relationship management systems. Using these systems is an expectation outlined in the guidebook. According to OVAC officials, much of the success in achieving standardization is tied to the technology upgrades because the intended benefits of modernization, such as improved veterans' experience and oversight of centers, will not be realized if there continues to be large variation in how centers operate and assess performance. As such, OVAC officials have noted the importance of procedures being in place for staff to effectively use the upgraded systems to collect and assess center performance data in a standardized way. OIT officials anticipated that they would fully implement the telephone system at each network by December 31, 2021; the quality management system by January 31, 2022, and will fully implement the customer relationship management system by April 30, 2022. According to OVAC officials, the system configurations need to be finalized prior to developing procedures; however, as of October 2021, OVAC officials had not identified when they will provide networks with the relevant procedures. The technology upgrades have features that may assist OVAC in assessing center performance, if used uniformly across networks. Having standardized procedures will help OVAC ensure that networks are using the upgraded technologies uniformly to assess center performance, which in turn will strengthen OVAC's ability to determine if centers' performance is improving. For example, the new quality management system includes features such as call recording that will facilitate supervisors' ability to 20GAO-15-84, Page 13 GAO-22-104620 VHA Clinical Contact Centers review and assess staff performance by using artificial intelligence to score Calls.21 According to OIT officials, the new system will likely be the biggest change, in terms of technology, for center staff because previously there was not a standardized call recording system across centers. Without procedures guiding the use of the system, there may be a risk that center staff do not use the call recording functionality effectively to review performance. For example, supervisors may interpret the call scores provided by the system differently and provide differing levels of training or coaching in response. As such, OVAC may not be able to ensure all staff are receiving appropriate training necessary to improve veterans' experience and center performance. Roles and responsibilities. According to OVAC, networks will have a role in assessing contact center performance, along with OVAC; however, as of November 2021, OVAC had not provided networks with the guidance identified in the guidebook defining these roles and associated responsibilities. According to OVAC officials, they have convened a workgroup, including network officials, that is developing a quality, patient safety, and performance management program for centers. According to OVAC, this program will outline network responsibilities for improving outcomes and assessing performance, including using data to do so. OVAC developed a one-page overview of the program in May 2021, but as of November 2021, the program was under development. According to OVAC officials, they had not finalized the program because its development relies on the implementation of the upgraded telephone and quality management systems-which will allow them to collect the relevant data-that are underway. OVAC's plan to centralize oversight and performance assessment of the contact centers at the network and national level is expected to help VHA realize one of the intended benefits of modernization to improve the oversight and performance of centers. By defining networks' roles and responsibilities in guidance, OVAC can help ensure that networks assess center performance, which can provide OVAC with insights into performance and operations across all centers-something VHA previously did not have. 21According to OIT officials, the quality management software has the ability to record calls. Then it uses artificial intelligence to transcribe the calls and provide a score. The score is determined by whether or not center staff are providing the appropriate information during the call. Supervisors have the ability to review recorded data or view interaction live to identify areas for additional training. Page 14 GAO-22-104620 VHA Clinical Contact Centers Conclusions Contact centers play a central role for veterans to access VA health care services-particularly during the COVID-19 pandemic when VHA has seen an uptick in veterans who have used centers to coordinate and access virtual care. Through its modernization effort, VHA intends to improve veterans' experience using contact centers by implementing a standardized model of contact centers at the network level that provides four core services. Although VHA has identified milestones and associated activities, it could benefit from a schedule that includes networks' activities and timelines prior to implementing any future enhancements to the four core services or additional centralization of centers. Documented milestones, activities, and timelines for all entities can help ensure that OVAC and networks have a shared understanding of the time and pathway necessary to achieve future modernization milestones. Additionally, VHA has provided networks with standardized staffing and training guidance; however, VHA has not provided networks with several pieces of guidance needed to improve oversight and center performance. For example, VHA has not provided guidance identifying metrics and related targets or procedures for the appropriate use of upgraded technologies designed to enable consistent performance accountability. VHA has also not provided a clear understanding of roles and responsibilities in the assessment process that may help VHA improve its ability to assess centers' performance and provide greater insight into centers' operations. Without an updated and shared modernization schedule and additional guidance to networks, veterans may not be able to access the same needed services across contact centers and may have varying experiences obtaining services from centers. Recommendations for Executive Action We are making the following four recommendations to VHA: The Under Secretary for Health should update and share a schedule with networks for the Clinical Contact Center Modernization effort beyond December 31, 2021, which includes (1) milestones, (2) captures all activities, (3) sequences all activities, and (4) establishes the duration of all activities. (Recommendation 1) The Under Secretary for Health should provide networks with guidance identifying finalized performance metrics and performance targets. (Recommendation 2) Page 15 GAO-22-104620 VHA Clinical Contact Centers Agency Comments The Under Secretary for Health should provide networks with guidance for using the upgraded technology systems to collect contact centers' performance data. (Recommendation 3) The Under Secretary for Health should provide networks with guidance defining their performance assessment roles and responsibilities for contact centers. (Recommendation 4) We provided a draft of this report to VA for review and comment. VA provided written comments, which are reprinted in appendix |. In its comments, VA concurred with all four of our recommendations and identified actions it is taking to address them by March 2022. We are sending copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or silass@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff wno made key contributions to this report are listed in appendix II. Sincerely yours, Sh Su Sharon M. Silas Director, Health Care Page 16 GAO-22-104620 VHA Clinical Contact Centers Appendix |: Comments from the Department of Veterans Affairs DEPARTMENT OF VETERANS AFFAIRS WASHINGTON January 10, 2022 Ms. Sharon M. Silas Director Health Care U.S. Government Accountability Office 441 G Street, NW Washington, DC 20548 Dear Ms. Silas: The Department of Veterans Affairs (VA) has reviewed the enclosed Government Accountability Office (GAO) draft report: VA HEALTH CARE: Clinical Contact Center Modernization Could Benefit from a Shared Schedule and Additional Guidance to Networks (GAO-22-104620). The enclosure contains the actions to be taken to address the draft report recommendations. VA appreciates the opportunity to comment on your draft report. Sincerely, Va" Dad her. Tanya J. Bradsher Chief of Staff Enclosure Page 17 GAO-22-104620 VHA Clinical Contact Centers Appendix |: Comments from the Department of Veterans Affairs Enclosure Department of Veterans Affairs (VA) Comments to Government Accountability Office (GAO) Draft Report VA HEALTH CARE: Clinical Contact Center Modernization Could Benefit from a Shared Schedule and Additional Guidance to Networks (GAO-22-104620) Recommendation 1: The Under Secretary for Health should update and share a schedule for the Clinical Contact Center Modernization effort beyond December 31, 2021, which includes (1) milestones, (2) captures all activities, (3) sequences all activities, and (4) establishes the duration of all activities. VA Response: Concur. The Veterans Health Administration (VHA) manages the Integrated Master Schedule (IMS) for the Clinical Contact Center Modernization (CCCM) effort. The CCCM IMS was developed at the inception of the CCCM program and is a dynamic document containing program milestones, sequences, and duration of activities for the extent of the program. The IMS is a document used internally by program offices and is referenced to develop guidance that is shared with stakeholders. VHA will ensure Veterans Integrated Services Networks (VISNs) have milestone expectations by the end of Quarter 2, fiscal year (FY) 2022. Target Completion Date: March 2022 Recommendation 2: The Under Secretary for Health should provide networks with guidance identifying finalized performance metrics and performance targets. VA Response: Concur. VHA is currently working with the Enterprise Contact Center Council on the development of performance metrics and targets that are in alignment with previously established enterprise contact center standards. Dissemination to VISNs planned to occur by Quarter 2, FY 2022. Target Completion Date: March 2022 Recommendation 3: The Under Secretary for Health should provide networks with guidance for using the upgraded technology systems to collect contact centers' performance data. VA Response: Concur. As part of an Enterprise Contact Center Council initiative to standardize and modernize, all clinical contact centers will be implementing a standard quality tool for monitoring and evaluating performance. During implementation, new guidance will be provided on performance data, technology tools and utilization of reports and dashboards, which will guide their decision-making related to Clinical Contact Center operations. Target Completion Date: March 2022 Page 18 GAO-22-104620 VHA Clinical Contact Centers Appendix |: Comments from the Department of Veterans Affairs Enclosure Department of Veterans Affairs (VA) Comments to Government Accountability Office (GAO) Draft Report VA HEALTH CARE: Clinical Contact Center Modernization Could Benefit from a Shared Schedule and Additional Guidance to Networks (GAO-22-104620) Recommendation 4: The Under Secretary for Health should provide networks with guidance defining their performance assessment roles and responsibilities for contact centers. VA Response: Concur. VHA has developed the CCCM Directive 1006.04, which defines roles and responsibilities for this effort. The directive is currently going through the final stages of approval prior to dissemination to the field. Target Completion Date: March 2022 Page 19 GAO-22-104620 VHA Clinical Contact Centers Appendix II: GAO Contact and Staff Acknowledgments GAO Contact Sharon Silas at (202) 512-7114, silass@gao.gov Staff In addition to the contact named above, Janina Austin (Assistant Director), Q. Akbar Husain (Analyst-in-Charge), David Raymond, and Acknowledgments Caitlin Scoville made key contributions to this report. Also contributing were Jacquelyn Hamilton, Diona Martyn, and Vikki Porter. (104620) Page 20 GAO-22-104620 VHA Clinical Contact Centers GAO's Mission Obtaining Copies of GAO Reports and Testimony The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is through our website. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. You can also subscribe to GAO's email updates to receive notification of newly posted products. Order by Phone Connect with GAO To Report Fraud, Waste, and Abuse in Federal Programs Congressional Relations Public Affairs Strategic Planning and External Liaison The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. Contact FraudNet: Website: https://www.gao.gov/about/what-gao-does/fraudnet Automated answering system: (800) 424-5454 or (202) 512-7700 A. Nicole Clowers, Managing Director, ClowersA@gao.gov, (202) 512-4400, U.S. Government Accountability Office, 441 G Street NW, Room 7125, Washington, DC 20548 Chuck Young, Managing Director, youngc1 @gao.gov, (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Stephen J. Sanford, Managing Director, spel@gao.gov, (202) 512-4707 U.S. Government Accountability Office, 441 G Street NW, Room 7814, Washington, DC 20548 o¥ wae Please Print on Recycled Paper.