United States Government Accountability Office Statement for the Record for the Committee on Veterans' Affairs, House of Representatives VETERANS CRISIS LINE For Release on Delivery Expected at 10:00 a.m ET Tuesday, April 4, 2017 Further Efforts Needed to Improve Service Statement for the Record by Seto J. Bagdoyan, Director, Forensic Audits and Investigative Service and Randall B. Williamson, Director, Health Care GAO-17-545T April 4, 2017 VETERANS CRISIS LINE Further Efforts Needed to Improve Service Highlights of GAO-17-545T, a statement for the record for the Committee on Veterans' Affairs, House of Representatives Why GAO Did This Study What GAO Found VA established the VCL in July 2007 to In May 2016, GAO found that the Department of Veterans Affairs (VA) did not provide support to veterans in meet its call response-time goals for the Veterans Crisis Line (VCL), although emotional crisis. Between fiscal years extended call wait times were not common; issues with text messages were also 2008, its first full year of operation, and found. VA’s goal was to answer 90 percent of VCL calls at the VCL primary 2015, the number of calls received by center within 30 seconds in the period GAO studied. Calls not answered within the VCL increased almost 700 percent, 30 seconds were to route to VCL backup call centers; however, for 5 months of exceeding VA’s expectations. As VA fiscal year 2015, calls were routed to VCL backup call centers after 60 seconds. began to address increasing numbers According to VA officials about 65 to 75 percent of VCL calls were answered at of requests for assistance, reports of the VCL primary center in fiscal year 2015 within either 30 or 60 seconds. GAO’s dissatisfaction with VCL’s service covert testing in July and August 2015 confirmed this. Specifically, 119 covert periodically appeared in the media. test calls showed that an estimated 73 percent of calls made during this period This statement summarizes GAO’s were answered within 30 seconds. GAO also covertly tested the VCL’s text May 2016 report on VA’s oversight of messaging services and found that 4 of 14 GAO test text messages did not the VCL. Specifically, the statement receive responses. GAO recommended that VA regularly test the VCL’s text describes (1) the extent to which VA messaging system to identify issues and correct them promptly. In response, VA met response-time goals for VCL calls developed and implemented procedures to regularly test the VCL's text and text messages, (2) how VA messaging system, as well as its telephone and online chat systems. monitored VCL primary center call center operations, and (3) how VA In May 2016, GAO found that VA had taken a number of steps to improve its worked with VCL service partners to monitoring of the VCL primary center operations. For example, VA had help ensure veterans receive high- established a permanent VCL call center evaluation team and created a quality service. For the May 2016 mechanism for tracking complaints about the performance of the VCL primary report, GAO visited the VCL’s primary center from VCL callers or external parties. However, GAO found that VA had center and two backup call centers; not specified quantifiable or otherwise measurable targets and had not included tested VCL response time through a dates for when it would expect the VCL to complete actions covered by each key generalizable sample of covert performance indicator. This was inconsistent with guidance provided by the telephone calls and a nongeneralizable Office of Management and Budget. GAO recommended that VA document sample of text messages in July and clearly stated and measurable targets and time frames for key performance August 2015; reviewed internal reports indicators needed to assess VCL performance. Although VA agreed, as of March and policies and plans; and interviewed VA officials. GAO 2017, this recommendation remains open. contacted VA and SAMHSA for an update on actions taken in response to GAO also found that VA had established an interagency agreement with its the report’s recommendations. service partner, the Department of Health and Human Services’ (HHS) Substance Abuse and Mental Health Services Administration (SAMHSA), to manage the shared operations of the VCL and Lifeline (a public-private network What GAO Recommends that provides free and confidential support to people in suicidal crisis or emotional distress). Despite these efforts to coordinate, VA and SAMHSA did not In its May 2016 report, GAO made four collect information needed to assess how often and why callers intending to recommendations to VA and HHS to reach the VCL do not follow voice prompts and instead reach Lifeline local crisis improve the VCL. VA and HHS centers. As a result, VA and SAMHSA did not know the extent to which this concurred with the recommendations occurred and could not determine the underlying causes that may need to be and while action has been taken on addressed. GAO recommended that VA and SAMHSA (1) collect information on one, three remain open as of March how often and why callers intending to reach the VCL instead reach Lifeline local 2017. crisis centers and (2) review the information collected and, if necessary, develop View GAO-17-545T. For more information, plans to address the underlying causes. Although VA and SAMHSA agreed, as contact Seto J. Bagdoyan at (202) 512-6722 of March 2017, these recommendations remain open. or bagdoyans@gao.gov or Randall B. Williamson at (202) 512-7114 or williamsonr@gao.gov. United States Government Accountability Office Letter Letter Chairman Roe, Ranking Member Walz, and Members of the Committee: We are pleased to submit this statement on our May 2016 report regarding the Department of Veterans Affairs’ (VA) Veterans Crisis Line (VCL). 1 Upon returning home from deployments in Afghanistan, Iraq, Vietnam, and other locations, many servicemembers struggle with mental health issues, including post-traumatic stress disorder, depression, and substance abuse. Several of these mental health issues have been identified as risk factors for suicide among veterans. As part of the continuum of mental health services it provides, VA established the VCL in July 2007. 2 The VCL supports veterans in emotional crisis and helps implement VA’s goal of improving mental health outcomes for servicemembers, veterans, and their families through a number of actions—including reducing barriers to seeking mental health treatment and expanding access to VA services. During the time of our review for the May 2016 report, the VCL operated through a VA-operated primary center staffed with VA-employed responders and five backup call centers that provided additional responders and other services through a backup call coverage contract. 3 Veterans can access the VCL by calling a national toll-free number—1- 800-273-TALK (8255). The VCL and the National Suicide Prevention Lifeline (Lifeline) share this national number through an interagency agreement between the VA and the Substance Abuse and Mental Health 1 GAO, Veterans Crisis Line: Additional Testing, Monitoring, and Information Needed to Ensure Better Quality Service, GAO-16-373 (Washington, D.C.: May 26, 2016). 2 VA established its crisis line at the VA medical center located in Canandaigua, New York. The original name of VA’s crisis line was the National Veterans Suicide Prevention Hotline until it was rebranded as the VCL in 2011. 3 For the purposes of this statement, the term “VCL service partners” includes the Substance Abuse and Mental Health Services Administration (SAMHSA), the VCL backup call coverage contractor, and the backup call centers that this contractor used to provide coverage to the VCL at the time of our 2016 review. VA has since opened an additional call center in Atlanta. Page 1 GAO-17-545T Services Administration (SAMHSA). 4 In addition to responding to calls, the VCL can also be accessed via online chat and text message. Since it was established, demand for the VCL’s services has exceeded VA’s expectations. The VCL received about 534,000 calls in fiscal year 2015, an almost 700 percent increase from the about 67,000 calls it received in fiscal year 2008, its first full year of operation. In response, VA steadily increased the VCL’s spending from about $3 million to $30 million from fiscal year 2008 through fiscal year 2015, devoting additional staff and resources to the VCL over time. As VA endeavored to address increasing numbers of requests for assistance, reports of dissatisfaction with VCL service periodically appeared in the media, and the VA Office of Inspector General was asked to investigate complaints about the VCL’s lack of timely response to callers. 5 The Inspector General identified gaps in the VCL quality-assurance process, including challenges associated with supervisory review, tracking of issues, and collection and analysis of data from VCL backup call centers. In addition, the Inspector General found that in some cases callers did not receive immediate assistance from responders. Our statement discusses (1) the extent to which VA met response-time goals for calls, online chats, and text messages received through the VCL; (2) how VA monitored the performance of the VCL primary center responders and call center operations; and (3) how VA worked with VCL service partners—backup call centers and SAMHSA—to help ensure veterans receive high-quality service from responders. This statement is based on our May 2016 report on VA’s oversight of the VCL as well as updates from VA and SAMHSA about efforts to address the report’s recommendations. 4 The VCL is distinct from Lifeline, which operates through a network of private, nonprofit providers working independently of one another while maintaining agreed-upon clinical standards. SAMHSA is an agency within the U.S. Department of Health and Human Services that leads public health efforts to advance the behavioral health of the nation. SAMHSA funds a cooperative agreement grant to administer Lifeline with the same entity that VA contracts with to provide VCL backup call coverage. Through this interagency agreement, VA and SAMHSA set out to establish a seamless crisis management system through a collaborative and cooperative relationship between the agencies that provides consistent suicide prevention techniques to callers 5 Department of Veterans Affairs, Office of Inspector General, Veterans Crisis Line Caller Response and Quality Assurance Concerns (Washington, D.C.: 2016). Page 2 GAO-17-545T For the May 2016 report, we made covert test telephone calls, text messages, and online chats to assess the extent to which VA met its response-time goals through the VCL. The test calls included a generalizable sample of 119 calls that could be used to describe all callers’ wait times when calling the VCL during July and August of 2015. We also sent a nongeneralizable sample of 15 test online chats to the VCL and 14 test text messages during the same time period. In addition, we examined telephone call, online chat, and text message data and summary reports from January 2013 through December 2015 that VA maintained related to the timeliness of the VCL’s operations. 6 We reviewed VCL policies, procedures, and monitoring data and interviewed VA officials. We also compared VA’s use of key performance indicators to the Office of Management and Budget’s guidance on performance goals, which are consistent with the Government Performance and Results Modernization Act of 2010. 7 We observed call centers’ operations and interviewed officials and representatives of the VCL primary center and two of the five VCL backup call centers to examine the extent to which VA coordinates with the VCL’s service partners in ensuring that veterans receive high-quality service from responders. Further, we reviewed VA’s contract that provides backup call coverage and VA’s interagency agreement with SAMHSA. We also made 34 covert calls in which we mimicked the experience of veterans who did not follow the instructions of a voice prompt to press “1” to reach the VCL. Finally, to examine the extent to which VA had plans to improve VCL operations, we reviewed VA’s improvement plans and interviewed VA officials responsible for planning and implementing those improvements. More detailed information on our objectives, scope, and methodology for this work can be found in our 2016 report. We conducted the work on which this statement is based in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate 6 We reviewed telephone call data to determine how many calls were answered at the VCL primary center; we reviewed online chat data to determine how many online chat requests received by the VCL received a response within 1 minute; and we reviewed text message data to determine how many text messages sent to the VCL received a response within 2 minutes. 7 See Office of Management and Budget, Preparation, Submission, and Execution of the Budget—Strategic Plans, Annual Performance Plans, Performance Reviews, and Annual Program Performance Reports, Circular No. A-11, pt. 6 (Washington, D.C.: June 2015). Page 3 GAO-17-545T evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition, the related investigative work was performed in accordance with the standards prescribed by the Council of the Inspectors General on Integrity and Efficiency. In 2007, VA established the VCL, a 24-hour crisis line staffed by Background responders trained to assist veterans in emotional crisis. Through an interagency agreement, VA collaborated with SAMHSA to use a single, national toll-free number for crisis calls that serves both Lifeline and the VCL. 8 Through this interagency agreement, VA and SAMHSA set up a cooperative relationship between the agencies that would provide consistent suicide-prevention techniques to callers. The national toll-free number presents callers with choices. Callers are greeted by a recorded message that explains the function of the crisis line and prompts individuals to press “1” to reach the VCL. Callers who do not press “1” by the end of the message are routed to one of Lifeline’s 164 local crisis centers. 9 All callers who press “1” are routed first to the VCL primary center. Calls that are not answered at the VCL primary center within 30 seconds of the time that the caller presses “1” during the Lifeline greeting are automatically routed to one of five VCL backup call centers. If a call is not answered by the VCL backup call center that initially receives it, the call may be sent to another VCL backup call center. 10 VA entered into a contract with a firm to oversee the operations of the VCL backup call centers. At the time of our 2016 report, there were a total of 164 Lifeline local crisis centers, 5 of which also serve the VCL. 11 8 SAMHSA and the Mental Health Association of New York City launched Lifeline on January 1, 2005. Lifeline provides free and confidential emotional support to people in suicidal crisis or emotional distress, 24 hours a day, 7 days a week. 9 The automated greeting also prompts Spanish speakers to press “2” for assistance in Spanish. 10 Some VCL backup call centers do not allow calls to be rerouted to another VCL backup call center and instead hold the call in a queue awaiting response by that backup call center’s responders. 11 VA does not directly contract with any of the VCL backup call centers. Page 4 GAO-17-545T VA added online chat and text message capabilities to the VCL in fiscal years 2009 and 2012, respectively. The number of online chats and text messages handled by the VCL generally increased every year, though the number of online chats decreased in fiscal year 2015. Extended Call Wait Times Were Uncommon in July and August 2015, but VA Did Not Meet Its Call Response Time Goals and Some Text Messages Did Not Receive Responses VA Responded to Most In our covert testing of the VCL’s call response time in July and August Calls within 30 Seconds in 2015, we found that it was uncommon for VCL callers to wait an extended period before reaching a responder since all of our calls that reached the July and August 2015, but VCL were answered in less than 4 minutes. However, we also found VA Did Not Meet Its Goal to did not meet its goal of answering 90 percent of calls to the VCL within 30 Answer 90 Percent of seconds for test calls that we made. Our test calls included a Calls within 30 Seconds at generalizable sample of 119 test calls that could be used to describe all the VCL Primary Center callers’ wait times when calling the VCL during this period. 12 On the basis of our test calls, we estimated that during July and August 2015 about 73 percent of all VCL calls were answered at the VCL primary center within 30 seconds. 13 VA officials told us that, during fiscal year 2015, about 65 to 75 percent of VCL calls were answered at the VCL primary center and 12 For these test calls, callers’ wait times refer to the length of time that elapses between when callers press “1” and when responders at either the VCL primary center or backup call centers answer the calls. 13 In addition, we estimated that during July and August of 2015, 99 percent of all VCL calls were answered within 120 seconds and the median call response time was 17 seconds. These percentage estimates have a margin of error of within plus or minus 9 percentage points, and the median response times estimates have a relative margin of error that is less than 9 percent at the 95 percent confidence level. Page 5 GAO-17-545T about 25 to 35 percent of VCL calls were answered at the backup call centers. These VA-reported results indicate that about 65 to 75 percent of VCL calls were answered within either 30 or 60 seconds. 14 These results are consistent with our test results for July and August 2015. During our 2016 review, VA officials told us that VA attempts to maximize the percentage of calls answered at the VCL primary center because these responders have additional resources—including access to veterans’ VA electronic medical records—that are unavailable to VCL backup call center responders. All responders—whether at primary or backup centers—receive specialized training to assist callers in crisis. 15 To Help Achieve To improve its performance toward meeting the goal of answering 90 Response-Time Goals, VA percent of calls at the VCL primary center within 30 seconds, VA implemented two changes in fiscal year 2015—namely, staggered work Implemented Changes at shifts for responders and new call-handling procedures. the VCL Primary Center Staggered work shifts. VA implemented staggered shifts for responders at the VCL primary center on September 6, 2015. Staggered shifts are work schedules that allow employees to start and stop their shifts at different times as a way to ensure better coverage during peak calling periods. Specifically, it helps schedule more employees to work when call volume is highest and fewer employees to work when call volume is lowest. 16 Additionally, staggered shifts help limit disruptions in service as responders begin and end their shifts. By comparing VCL telephone call data from September through December of 2014 to that of September through December of 2015, we found that VA’s implementation of staggered shifts at the VCL primary 14 For approximately 5 months of fiscal year 2015, VA allowed calls to ring at the VCL primary center for 60 seconds before routing the calls to VCL backup call centers. VA then returned to the standard that calls not answered at the VCL primary center within 30 seconds are then routed to VCL backup call centers. 15 All VCL primary and backup call center responders are required to complete Applied Suicide Intervention Skills Training in which they learn to use a suicide intervention model to identify persons with thoughts of suicide, seek a shared understanding of reasons for dying and living, develop a safe plan based on a review of risk, be prepared to do follow- up, and become involved in suicide-safer community networks. 16 The International Customer Management Institute includes staggered shifts as a best practice among call centers. Page 6 GAO-17-545T center had mixed results. 17 For example, the average percentage of calls answered per hour at the VCL primary center from September through December 2015—after staggered shifts were implemented—was 75 percent, slightly less than the average of 79 percent answered during the corresponding period in 2014 before staggered shifts were implemented. However, the VCL received an average of about 1.3 more calls per hour during this period in 2015 than it received during the corresponding period in 2014 and, according to VA officials, the VCL primary center employed fewer responder staff in 2015 than 2014. New call-handling procedures. VA implemented new call handling procedures at the VCL primary center beginning in June 2015 that provided responders with specific guidance to more efficiently handle “noncore” callers—those callers who were not seeking crisis assistance but rather seeking help with other issues, such as help with veterans’ benefits questions. For example, if a caller reached the VCL with a question about VA disability benefits, the VCL primary center responder would verify that the caller was not in crisis and transfer the caller to the Veterans Benefits Administration to address the question. VCL telephone call data provided by VA suggest that the average time VCL primary center responders spent handling noncore calls decreased by approximately 30 percent over a 5-month period after responder training began on these new call-handling procedures. 18 We would expect that as the average time VCL primary center responders spent handling noncore calls decreased, these responders would have more time available to answer additional incoming calls. 17 Our analysis compared VCL call data from September 6, 2015, through December 31, 2015, to VCL call data from September 1, 2014, through December 31, 2014. The percentage of calls answered was likely affected by several factors, such as call volume, staffing levels, and complexity of calls. Our analysis controlled for day of the week, time of day, and holidays, but did not control for all factors that may affect the percentage of calls answered. 18 We did not test this aspect of VCL operations with covert test calls. Page 7 GAO-17-545T In July and August 2015, To determine the timeliness of the VCL’s responses to online chats and Most of Our Test Online text messages, we conducted a covert test in July and August 2015 using nongeneralizable samples of 15 online chats and 14 text messages. All Chats Were Answered 15 of our test online chats received responses within 60 seconds, 13 of Within 30 Seconds, but VA which were within 30 seconds. This result was consistent with VA data Did Not Ensure That that indicated VCL responders sent responses to over 99 percent of Veterans Received online chat requests within 1 minute in fiscal years 2014 and 2015. During Responses through Its our 2016 review, VA officials told us that all online chats are expected to be answered immediately. Although this was an expectation, we found in Text Messaging Service 2016 that VA did not have formal performance standards for how quickly responders should answer online chat requests and expected to develop them before the end of fiscal year 2016. However, our tests of text messages revealed a potential area of concern. Four of our 14 test text messages did not receive a response from the VCL. Of the remaining 10 test text messages, 8 received responses within 2 minutes, and 2 received responses within 5 minutes. As we reported in May 2016, VA officials stated that text messages are expected to be answered immediately, but, as with online chats, VA had not developed formal performance standards for how quickly responders should answer text messages. VA data indicated that VCL responders sent responses to 87 percent of text messages within 2 minutes of initiation of the conversation in both fiscal years 2014 and 2015. During our 2016 review, VA officials said that VA planned to establish performance standards for answering text messages before the end of fiscal year 2016. VA officials noted and we observed during a site visit that some incoming texts were abusive in nature or were not related to a crisis situation. 19 According to VA officials, in these situations, if this is the only text message waiting for a response, a VCL responder will send a response immediately. However, if other text messages are awaiting responses, VA will triage these text messages and reply to those with indications of crisis first. This triage process may have contributed to the number of our test texts that did not receive responses within 2 minutes. The VCL’s text messaging service provider offered several reasons for the possible nonresponses that we encountered in our test results. These included: (1) incompatibilities between some devices used to send text messages to the VCL and the software VA used to process the text 19 Our test text messages consisted of a simple greeting, such as “Hi” or “Hello.” Page 8 GAO-17-545T messages, (2) occasional software malfunctions that freeze the text messaging interface at the VCL primary center, (3) inaudible audio prompts used to alert VCL primary center responders of incoming text messages, (4) attempts by people with bad intentions to disrupt the VCL’s text messaging service by overloading the system with a large number of texts, and (5) incompatibilities between the web browsers used by the VCL primary center and the text messaging software. At the time of our 2016 review, VA officials told us that they did not monitor and test the timeliness and performance of the VCL text messaging system, but rather relied solely on the VCL’s text messaging service provider for such monitoring and testing. They said that the provider had not reported any issues with this system. According to the provider, no routine testing of the VCL’s text messaging system was conducted. Standards for internal control in the federal government state that ongoing monitoring should occur in the course of normal operations, be performed continually, and be ingrained in the agency’s operations. 20 We concluded that without routinely testing its text messaging system, or ensuring that its provider tests the system, VA cannot ensure that it is identifying limitations with its text messaging service and resolving them to provide consistent, reliable service to veterans. We recommended that VA regularly test the VCL’s text messaging system to identify issues and correct them promptly. In response, VA developed and implemented procedures to regularly test the VCL’s text messaging system, as well as its telephone and online chat systems. We believe this change will allow VA to more reliably and quickly identify and correct errors in the text messaging system and therefore help veterans reach VCL responders in a timelier manner. 20 See GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1 (Washington, D.C.: November 1999). Page 9 GAO-17-545T VA Had Taken Steps to Improve Its Monitoring of VCL Primary Center Performance but Had Not Established Targets and Time Frames for VCL Key Performance Indicators VA Established a Call As we reported in May 2016, VA had sought to enhance its capabilities Center Evaluation Team, for overseeing VCL primary center operations through a number of activities—including establishing a call center evaluation team, Implemented Revised implementing revised performance standards for VCL primary center Responder Performance responders, implementing silent monitoring of VCL primary center Standards, and Analyzed responders, and analyzing VCL caller complaints. 21 VCL Caller Complaints Establishment of a call center evaluation team. In October 2013, VA established a permanent VCL call center evaluation team that is responsible for monitoring the performance of the VCL primary center. 22 As we reported in May 2016, the call center evaluation team analyzes VCL data, including information on the number of calls received and the number of calls routed to backup call centers from the primary center. VA officials told us that they use these data to inform management decisions about VCL operations. Implementation of revised performance standards for VCL primary center responders. In October 2015, VA implemented new performance standards for all VCL primary center responders that will be used to assess their performance in fiscal year 2016. According to VA officials, 21 VCL also tracks and analyzes complaints about the services of VCL backup call centers as a part of this effort. 22 According to VA officials, this team was initially staffed with VA employees detailed from other areas of the department in December 2012. Permanent staff for call center evaluation were hired in October 2013. Page 10 GAO-17-545T these performance standards include several measures of responder performance—such as demonstrating crisis-intervention skills, identifying callers’ needs, and addressing those needs in an appropriate manner using VA approved resources. Silent monitoring of VCL primary center responders. In February 2016, VA officials reported that they were beginning silent monitoring of all VCL responders using recently developed standard operating procedures, standard data collection forms, and standard feedback protocols. Analysis of VCL caller complaints. In October 2014, VA created a mechanism for tracking complaints it receives from VCL callers and external parties, such as members of Congress and veterans, about the performance of the VCL primary and backup call centers. According to VA officials, each complaint is investigated to validate its legitimacy and determine the cause of any confirmed performance concerns. The results and disposition of each complaint are documented in VA’s complaint tracking database. VCL Key Performance In 2011, VA established key performance indicators to evaluate VCL Indicators Lacked primary center operations; however, in our May 2016 review, we found these indicators did not have established measureable targets or time Measureable Targets and frames for their completion. Time Frames VCL key performance indicators lacked measurable targets. We found that VA’s list of VCL key performance indicators did not include information on the targets the department had established to indicate their successful achievement. For example, VA included a key performance indicator for the percentage of calls answered by the VCL in this list but did not include information on what results would indicate success for (1) the VCL as a whole, (2) the VCL primary center, or (3) the VCL backup call centers. As another example, VA had not established targets for the percentage of calls abandoned by callers prior to speaking with VCL responders. Measureable targets should include a clearly stated minimum performance target and a clearly stated ideal performance target. 23 These targets should be quantifiable or otherwise measurable and indicate how 23 See GAO, VA Health Care: Additional Guidance, Training, and Oversight Needed to Improve Clinical Contract Monitoring, GAO-14-54 (Washington, D.C.: Oct. 2013). Page 11 GAO-17-545T well or at what level an agency or one of its components aspires to perform. 24 Such measurable targets are important for ensuring that the VCL call center evaluation team can effectively measure VCL performance. VCL key performance indicators lack time frames for their completion. We found that VA’s list of VCL key performance indicators did not include information on when the department expected the VCL to complete or meet the action covered by each key performance indicator. For example, for VA’s key performance indicator for the percentage of calls answered by the VCL, the department had not included a date by which it would expect the VCL to complete this action. As another example, VA had not established dates by which it would meet targets yet to be established for the percentage of calls abandoned by callers prior to speaking with VCL responders. Time frames for action are a required element of performance indicators and are important to ensure that agencies can track their progress and prioritize goals. 25 Guidance provided by the Office of Management and Budget states that performance goals—similar to VA’s key performance indicators for the VCL—should include three elements: (1) a performance indicator, which is how the agency will track progress; (2) a target; and (3) a period. 26 Without establishing targets and time frames for the successful completion of its key performance indicators for the VCL, we concluded that VA could not effectively track and publicly report progress or results for its key performance indicators for accountability purposes. 24 Consistent with the Government Performance and Results Modernization Act of 2010, the Office of Management and Budget states that a performance goal should include a tangible, measurable objective or a quantifiable standard, value, or rate. See Office of Management and Budget, Preparation, Submission, and Execution of the Budget— Strategic Plans, Annual Performance Plans, Performance Reviews, and Annual Program Performance Reports. 25 Consistent with the Government Performance and Results Modernization Act of 2010, the Office of Management and Budget defines a performance goal as a statement of the level of performance to be accomplished within a time frame. See Office of Management and Budget, Preparation, Submission, and Execution of the Budget—Strategic Plans, Annual Performance Plans, Performance Reviews, and Annual Program Performance Reports. 26 See Office of Management and Budget, Preparation, Submission, and Execution of the Budget—Strategic Plans, Annual Performance Plans, Performance Reviews, and Annual Program Performance Reports. Page 12 GAO-17-545T We recommended that VA document clearly stated and measurable targets and time frames for key performance indicators needed to assess VCL performance. While VA officials have informed us that they have created scorecards that track information related to calls answered, staffing, and average handle times, as of March 2017, clearly stated and measurable targets and time frames have not yet been developed. VA Was Strengthening Requirements for VCL Backup Call Centers, but VA and SAMHSA Did Not Collect Information to Assess How Often and Why Callers Were Not Reaching the VCL VA Was Enhancing As we reported in May 2016, VA’s backup call coverage contract, Performance awarded in October 2012 and in place at the time of our review, did not include detailed performance requirements in several key areas for the Requirements for Its VCL backup call centers. Clear performance requirements for VCL Backup Call Coverage backup call centers are important for defining VA’s expectations of these Contractor service partners. However, VA had taken steps to strengthen the performance requirements of this contract by modifying it in March 2015 and was beginning the process of replacing it with a new contract. According to VA officials, the new contract was awarded in April 2016. October 2012 backup call coverage contract. This contract provided a network of Lifeline local crisis centers that could serve as VCL backup call Page 13 GAO-17-545T centers managed by a contractor. 27 This contractor was responsible for overseeing and coordinating the services of VCL backup call centers that answer overflow calls from the VCL primary center. This contract as initially awarded included few details on the performance requirements for VCL backup call centers. For example, the contract did not include any information on the percentage of VCL calls routed to each VCL backup call center that should be answered. Detailed performance requirements on these key aspects of VCL backup call center performance are necessary for VA to effectively oversee the performance of the contractor and the VCL backup call centers. By not specifying performance requirements for the contractor on these key performance issues, we believe that VA missed the opportunity to validate contractor and VCL backup call center performance and mitigate weaknesses in VCL call response. As we reported in May 2016, VA officials told us about several concerns with the performance of the backup call centers operating under the October 2012 contract based on their own observations and complaints reported to the VCL. These concerns included the inconsistency and incompleteness of VCL backup call centers’ responses to VCL callers, limited or missing documentation from records of VCL calls answered by VCL backup call center responders, limited information provided to VA that could be used to track VCL backup call center performance, and the use of voice answering systems or extended queues for VCL callers reaching some VCL backup call centers. For example, VA officials reported that some veterans did not receive complete suicide assessments when their calls were answered at VCL backup call centers. In addition, VA officials noted that they had observed some VCL backup call centers failing to follow VCL procedures, such as not calling a veteran who may be in crisis when a third-party caller requested that the responder contact the veteran. According to VA officials, these issues led to additional work for the VCL primary center, including staffing one to two responders per shift to review the call records submitted to the VCL primary center by backup call centers and to determine whether these calls required additional follow-up from the VCL primary center. These 27 The backup call coverage contract in place at the time of our review was awarded in October 2012 with a 1-year base and two 1-year option periods (for a total of 3 years of coverage) and was set to expire in September 2015. However, according to VA officials, the contract was extended through May 2016 while the department was finalizing a new contract. VA officials reported that the new backup call coverage contract was awarded in April 2016. Page 14 GAO-17-545T officials estimated that 25 to 30 percent of backup call center call records warranted additional follow-up to the caller from a VCL primary center responder, including approximately 5 percent of backup call center call records that needed to be completely reworked by a VCL primary center responder. March 2015 backup call coverage contract modification. Given these concerns, in March 2015, VA modified the October 2012 backup call coverage contract to add more explicit performance requirements for its backup call coverage contractor, which likely took effect more quickly than if the department had waited for a new contract to be awarded. These modified requirements included (1) the establishment of a 24- hours-a-day, 7-days-a-week contractor-staffed emergency support line that VCL backup call centers could use to report problems, (2) a prohibition on VCL backup call centers’ use of voice answering systems, (3) a prohibition on VCL backup call centers placing VCL callers on hold before a responder conducted a risk assessment, (4) documentation of each VCL caller’s suicide risk assessment results, and (5) transmission of records for all VCL calls to the VCL primary center within 30 minutes of the call’s conclusion. Development of new backup call coverage contract. In July 2015, VA began the process of replacing its backup call coverage contract by publishing a notice to solicit information from prospective contractors on their capability to satisfy the draft contract terms for the new contract; this new backup call coverage contract was awarded in April 2016. 28 We found that these new proposed contract terms included the same performance requirement modifications that were made in March 2015, as well as additional performance requirements and better data reporting from the contractor that could be used to improve VA’s oversight of the VCL backup call centers. Specifically, the proposed contract terms added performance requirements to address VCL backup call center performance—including a requirement for 90 percent of VCL calls received by a VCL backup call center to be answered by a backup call center responder within 30 seconds and 100 percent to be answered by a backup call center responder within 2 minutes. In addition, the proposed contract terms included numerous data reporting requirements that could allow VA to routinely assess the performance of its VCL backup call 28 This notice—referred to as a sources sought notice—included a draft performance work statement. In April 2016, VA officials reported that this contract was awarded to the previous backup call coverage contractor. Page 15 GAO-17-545T centers and identify patterns of noncompliance with the contract’s performance requirements more efficiently and effectively than under the prior contract. The proposed terms for the new contract also state that VA will initially provide and approve all changes to training documentation and supporting materials provided to VCL backup call centers in order to promote the contractor’s ability to provide the same level of service that is being provided by the VCL primary center. VA and SAMHSA Did Not In May 2016, we found that when callers did not press “1” during the initial Collect Information Lifeline greeting, their calls may take longer to answer than if the caller had pressed “1” and been routed to either the VCL primary center or a Needed to Assess How VCL backup call center. 29 As previously discussed, VA and SAMHSA Often and Why Callers collaborated to link the toll-free numbers for both Lifeline and the VCL Were Not Reaching the through an interagency agreement. The greeting instructs callers to press VCL “1” to be connected to the VCL; if callers do not press “1,” they will be routed to one of SAMHSA’s 164 Lifeline local crisis centers. To mimic the experience of callers who did not press “1” to reach the VCL when prompted, we made 34 covert nongeneralizable test calls to the national toll-free number that connects callers to both Lifeline and the VCL during August 2015 and we did not press “1” to be directed to the VCL. 30 For 23 of these 34 calls, our call was answered in 30 seconds or less. For 11 of these calls, we waited more than 30 seconds for a responder to answer— including 3 calls with wait times of 8, 9, and 18 minutes. Additionally, one of our test calls did not go through, and during another test call we were asked if we were safe and able to hold. 31 VA’s policy prohibits VCL responders from placing callers on hold prior to completing a suicide assessment; Lifeline has its own policies and procedures. 32 29 At the time of our tests, the initial greeting was about 30 seconds long and prompted the caller to press “1” to be connected to the VCL at the end of the greeting. If callers did not press”1,” the call was routed to one of SAMHSA’s 164 Lifeline local crisis centers based on the area code of the callers’ telephone numbers. 30 These 34 calls were a random but nongeneralizable sample. 31 When asked if we were safe and could hold, we terminated this test call. 32 We did not review Lifeline’s policies and procedures as a part of our May 2016 report due to our focus on the VCL. We focused our review of Lifeline on those elements of their operations that interacted with the VCL or VA, such as the interagency agreement between VA and SAMHSA that governs the shared use of a single national toll-free number between the VCL and Lifeline. Page 16 GAO-17-545T According to officials and representatives from VA, SAMHSA, and the VCL backup call centers, as well as our experience making test calls where we did not press “1,” there are several reasons why a veteran may not press “1” to be routed to the VCL, including • an intentional desire to not connect with VA, • failure to recognize the prompt to press “1” to be directed to the VCL, • waiting too long to respond to the prompt to press “1” to be directed to the VCL, or • calling from a rotary telephone that does not allow the caller to press “1” when prompted. As we found in May 2016, VA officials had not estimated the extent to which veterans intending to reach the VCL did not press “1” during the Lifeline greeting. 33 These officials explained that their focus had been on ensuring that veterans who did reach the VCL received appropriate service from the VCL primary center and backup call centers. In addition, SAMHSA officials said that they also did not collect this information. 34 These officials reported that SAMHSA did not require the collection of demographic information, including veteran status, for a local crisis center to participate in the Lifeline network. However, they noted that SAMHSA could request through its grantee that administers the Lifeline network that local crisis centers conduct a one-time collection of information to help determine how often and why veterans reach Lifeline local crisis centers. SAMHSA officials explained that they could work with the Lifeline grantee to explore optimal ways of collecting this information that would be (1) clinically appropriate, (2) a minimal burden to callers and Lifeline’s local crisis centers, and (3) in compliance with the Office of Management and Budget’s paperwork reduction and information collection policies. The interagency agreement between VA and SAMHSA assigns SAMHSA responsibilities for monitoring the use of the national toll-free number that 33 According to SAMHSA officials, in 2014, about 383,000 callers abandoned their calls to Lifeline during the initial greeting used to direct callers to either Lifeline local crisis centers or the VCL. We did not assess the reasons these calls were abandoned. 34 According to SAMHSA officials, the SAMHSA grantee responsible for administering Lifeline conducted a survey in 2014 that captured veteran-related data. However, SAMHSA had no involvement with this survey or the data collection activities of the Lifeline local crisis centers that provided the information because it was outside the scope of SAMHSA’s grant to the organization. Further, HHS stated that the SAMHSA grantee did not share the results of the survey with SAMHSA. We did not evaluate the results of this survey. Page 17 GAO-17-545T is used to direct callers to both the VCL and Lifeline. These responsibilities include monitoring the use of the line, analyzing trends, and providing recommendations about projected needs and technical modifications needed to meet these projected needs. Using the information collected from the Lifeline local crisis centers on how often and why veterans reach Lifeline, as opposed to the VCL, VA and SAMHSA officials could then assess whether the extent to which this occurs merits further review and action. Although the results of our test were not generalizable, substantial wait times for a few of our covert calls suggested that some callers may experience longer wait times to speak with a responder in the Lifeline network than they would in the VCL’s network. We concluded that without collecting information to examine how often and why veterans do not press “1” when prompted to reach the VCL, VA and SAMHSA could not determine the extent veterans reach the Lifeline network when intending to reach the VCL and may experience longer wait times as a result. In addition, limitations in information on how often and why this occurs did not allow VA and SAMHSA to determine whether or not they should collaborate on plans to address the underlying causes of veterans not reaching the VCL. Standards for internal control in the federal government state that information should be communicated both internally and externally to enable the agency to carry out its responsibilities. 35 For external communications, management should ensure there are adequate means of communicating with, and obtaining information from, external stakeholders that may have a significant impact on the agency achieving its goals. We recommended VA and SAMHSA collaborate in taking the following two actions: (1) collect information on how often and why callers intending to reach the VCL instead reach Lifeline local crisis centers and (2) review the information collected and, if necessary, develop plans to address the underlying causes. We understand that VA and SAMHSA have been coordinating on these issues. However, as of March 2017, both of these recommendations remain open. 35 GAO/AIMD-00-21.3.1 Page 18 GAO-17-545T Chairman Roe, Ranking Member Walz, and Members of the Committee, this concludes our statement for the record. For questions about this statement, please contact Seto J. Bagdoyan at GAO Contact and (202) 512-6722 or bagdoyans@gao.gov or Randall B. Williamson at (202) Staff 512-7114 or williamsonr@gao.gov. Acknowledgments Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contacts listed, GAO staff who made key contributions to this statement are Jennie F. Apter, Christie Enders, Cathleen J. Hamann, Marcia A. Mann, Vikki Porter, Lisa Rogers, and Julie T. Stewart. (101959) Page 19 GAO-17-545T This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. The Government Accountability Office, the audit, evaluation, and investigative GAO’s Mission arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is Obtaining Copies of through GAO’s website (http://www.gao.gov). Each weekday afternoon, GAO GAO Reports and posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to http://www.gao.gov Testimony and select “E-mail Updates.” Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, http://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, LinkedIn, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at www.gao.gov and read The Watchblog. Contact: To Report Fraud, Website: http://www.gao.gov/fraudnet/fraudnet.htm Waste, and Abuse in E-mail: fraudnet@gao.gov Federal Programs Automated answering system: (800) 424-5454 or (202) 512-7470 Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400, Congressional U.S. Government Accountability Office, 441 G Street NW, Room 7125, Relations Washington, DC 20548 Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707 Strategic Planning and U.S. Government Accountability Office, 441 G Street NW, Room 7814, External Liaison Washington, DC 20548 Please Print on Recycled Paper.