United States Government Accountability Office Report to Congressional Committees VETERAN SUICIDE September 2022 VA Efforts to Identify Veterans at Risk through Analysis of Health Record Information GAO-22-105165 September 2022 VETERAN SUICIDE VA Efforts to Identify Veterans at Risk through Analysis of Health Record Information Highlights of GAO-22-105165, a report to congressional committees Why GAO Did This Study What GAO Found Veterans suffer a disproportionately The Veterans Health Administration (VHA) started using the Recovery higher rate of suicide compared to non- Engagement and Coordination for Health-Veterans Enhanced Treatment veterans. The Department of Veterans (REACH VET) program across VHA in 2017 to help identify veterans receiving Affairs established veteran suicide VHA care who are potentially at increased risk for suicide. REACH VET prevention as its highest clinical complements other methods of identifying at-risk veterans who may not have priority. It also established a goal of been already identified. According to VHA, the model identifies veterans who ensuring that at-risk and underserved may benefit from enhanced clinical care-such as specialty care, follow-ups for veterans receive the care and missed appointments, or additional mental health services. To do so, REACH resources they need to end veteran VET uses a predictive model that analyzes veterans' health record information suicide. each month. Upon identification, clinicians are expected to evaluate each The Commander John Scott Hannon identified veteran's risk for suicide, determine appropriate treatment approaches, Veterans Mental Health Care and contact the veteran to discuss options for care. Improvement Act of 2019 included a provision for GAO to review efforts to The REACH VET program model uses 61 variables included in each veteran's manage veterans at high risk for VHA electronic health record to identify veterans at the greatest statistical risk for suicide. This report describes 1) the suicide at each VHA facility. These variables include documentation of previous REACH VET program and 2) findings suicide attempts and mental health diagnoses. from VHA's analysis of the potential Examples of Variables Used in the Recovery Engagement and Coordination for Health- benefits of the REACH VET program. Veterans Enhanced Treatment (REACH VET) Model (as of March 2021) GAO reviewed VHA program guidance on REACH VET and interviewed officials at seven VHA medical facilities selected based on geography and complexity of services. GAO also compared and counted how frequently veterans were identified by REACH VET and whether those veterans already had a high risk for suicide flag from February 2017 through June 2021. GAO also reviewed VHA's published studies on the REACH VET program, and discussed future studies with VHA officials. GAO provided a draft of this report to VA for review and comment. VA provided technical comments, which VHA has studied the REACH VET program and determined that the REACH VET GAO incorporated as appropriate. predictive model identified veterans not already identified by other screening programs. Similarly, GAO analyzed VHA information and determined that the REACH VET model identified veterans who had not been identified through other methods. VHA has also studied the benefits to veterans of being involved in REACH VET and determined that there were associated benefits-such as increased completion of outpatient appointments. While VHA's studies provide evidence that the REACH VET program can identify veterans at high risk for suicide, VHA has acknowledged the need for, and told GAO they plan to View GAO-22-105165. For more information, conduct, further studies. For example, officials stated that future studies would contact Sharon M. Silas at (202) 512-7114 or evaluate how to increase the impact of REACH VET by targeting outreach to silass@gao.gov. veterans who show signs of disengagement from VHA health care. United States Government Accountability Office Contents Letter 1 Background 5 VHA Uses REACH VET to Help Identify and Provide Enhanced Care for Veterans Potentially at High Risk for Suicide 6 VHA's Analysis Shows Potential Benefits of REACH VET, and VHA Plans Further Study 10 Agency Comments 13 Appendix I Veterans at High Risk for Suicide: Identify, Intervene, and Monitor 15 Appendix II Veterans Health Administration Suicide Prevention Staffing 22 Appendix III VHA Mental Health Resources Offered to Family and Friends 25 Appendix IV REACH VET Program Predictive Model Variables 28 Appendix V VHA Published, Ongoing, and Future Studies of the REACH VET Model and Program Outcomes 32 Appendix VI Comments from the Department of Veterans Affairs 34 Appendix VII GAO Contact and Staff Acknowledgments 35 Tables Table 1: Recovery Engagement and Coordination for Health- Veterans Enhanced Treatment (REACH VET) Algorithm Variables by Category 29 Table 2: VHA Studies of REACH VET-Published, Ongoing, Planned and Proposed (April 2022) 32 Page i GAO-22-105165 Veteran Suicide Risk Figures Figure 1: Examples of Variables Used in VHA's REACH VET Model (as of March 2021) 7 Figure 2: Veterans Health Administration (VHA) Required Steps for Veterans Identified by REACH VET 9 Figure 3: Overlapping Veteran Populations Identified through REACH VET and with a High Risk for Suicide Patient Record Flag (February 2017 through June 2021) 11 Figure 4: Clinical Practice Guideline for Screening, Evaluating, and Managing Veterans at Acute Risk for Suicide 17 Abbreviations OMHSP Office of Mental Health and Suicide Prevention REACH VET Recovery Engagement and Coordination for Health- Veterans Enhanced Treatment SPC suicide prevention coordinator VA Department of Veterans Affairs VHA Veterans Health Administration This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page ii GAO-22-105165 Veteran Suicide Risk Letter 441 G St. N.W. Washington, DC 20548 September 14, 2022 The Honorable Jon Tester Chairman The Honorable Jerry Moran Ranking Member Committee on Veterans' Affairs United States Senate The Honorable Mark Takano Chairman The Honorable Mike Bost Ranking Member Committee on Veterans' Affairs House of Representatives Veterans suffer a disproportionately higher rate of suicide compared to non-veterans. According to Department of Veterans Affairs' (VA) data, in 2019, the suicide rate for veterans was 31.6 per 100,000 veterans, compared to 16.8 per 100,000 non-veteran U.S. adults. 1 Per the VA's Office of Mental Health and Suicide Prevention's (OMHSP) 2021 National Veteran Suicide Prevention Annual Report, approximately three in five veterans who died by suicide in 2019 did not receive Veterans Health Administration (VHA) health care in 2018 or 2019. 2 VA identified suicide prevention as its top clinical priority in its strategic plan for fiscal years 2018 through 2024, with a goal of ensuring that at-risk and underserved 1According to VA data, in 2019 (the most recent year for which cause of death information is available from the Centers for Disease Control and Prevention), the U.S. adult population was 235.4 million, and the veteran population was 19.8 million. The rate for veterans was 52 percent higher than for the general non-veteran adult population after accounting for differences in the demographic characteristics of veterans and non- veterans. Of the 45,861 U.S. adults who died by suicide in 2019, 6,261 were veterans. See Department of Veterans Affairs, Office of Mental Health and Suicide Prevention, 2021 National Veteran Suicide Prevention Annual Report (Washington, D.C.: September 2021). 2See Department of Veterans Affairs, Office of Mental Health and Suicide Prevention, 2021 National Veteran Suicide Prevention Annual Report (Washington, D.C.: September 2021). According to VHA officials, most veterans in this category either did not seek, or may not be eligible for VHA health care. VA data show that 6.1 million out of 20 million veterans used VA health care in fiscal year 2017. See Department of Veterans Affairs, VA Utilization Profile Fiscal Year 2017 (May 2020). Page 1 GAO-22-105165 Veteran Suicide Risk veterans receive the care and resources they need to end veteran suicide. 3 One effort by VA to identify at-risk veterans receiving VHA health care is the Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment (REACH VET) program. Fully implemented across VA in 2017, the REACH VET program includes a predictive model to analyze data from VHA health records to identify veterans at increased risk for suicide. 4 According to VHA, the REACH VET program was created to complement, and not replace, its other suicide prevention intervention strategies. These include regular clinical screening for suicide risk (Risk ID) and use of a high risk for suicide patient record flag (high- risk flag). The flag is a notification in the VHA electronic health record system that identifies for VHA staff veterans who may be at high risk for suicide and may need additional care. The Commander John Scott Hannon Veterans Mental Health Care Improvement Act of 2019 included a provision for us to review VA efforts to manage veterans at high risk for suicide. 5 In this report, we describe: 1. VHA's REACH VET program, and 2. findings from VHA's analysis of the potential benefits of the REACH VET program. In addition, appendixes I, II, and III provide information on (1) enhanced clinical care for and monitoring of veterans at high risk for suicide, (2) suicide prevention staffing levels at VHA medical facilities, and (3) the resources offered to family members and friends of veterans who have a mental health condition. To answer our objectives, we reviewed VHA program policies and guidance for the REACH VET program and for veterans who have high- risk flags and discussed these suicide prevention strategies with VHA and 3See Department of Veterans Affairs, Department of Veterans Affairs Fiscal Years 2018- 2024 Strategic Plan (Washington, D.C.: May 31, 2019). 4Department of Veterans Affairs Memorandum, REACH VET: Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment, August 10, 2016. 5Pub. L. No. 116-171, tit. II, § 205, 134 Stat. 778, 799-800 (2020). Page 2 GAO-22-105165 Veteran Suicide Risk facility officials. 6 We also reviewed VA Office of Inspector General evaluations of VHA medical facilities for fiscal year 2020, which contained VHA medical facility site reviews of VHA suicide prevention processes. 7 Additionally, we reviewed VHA studies of its REACH VET program, published from September 2015 through October 2021, and discussed with VHA officials their plans for further study. 8 To supplement findings from the VHA studies, we also analyzed VHA information to assess the extent of overlap between veterans receiving high-risk flags and being identified by the REACH VET model for the period of February 2017 through June 2021. Specifically, we compared two lists of veterans from VHA: • Veterans with high-risk flags at any of the 140 VHA medical facilities, for each month during the time period of February 2017 through June 2021. • Veterans who were identified by REACH VET at any facility for each month during the same February 2017 through June 2021 time period. We compared the two lists and identified how frequently veterans' names appeared on both lists at each facility. Additionally, we counted how many veterans were identified by REACH VET before receiving a high-risk flag, as well as the number of veterans who had high-risk flags before being identified by REACH VET. We also counted how many times a veteran 6See Department of Veterans Affairs Memorandum, Patients at High-Risk for Suicide, April 24, 2008; Department of Veterans Affairs, Inactivation Process for Category I High Risk for Suicide Patient Record Flags, VHA Notice 2021-10, (Washington, D.C.: May 28, 2021); and VA, REACH VET: Recovery Engagement and Coordination for Health– Veterans Enhanced Treatment. 7See Department of Veterans Affairs, Office of Inspector General, Comprehensive Healthcare Inspection Summary Report: Evaluation of Mental Health in Veterans Health Administration Facilities, Fiscal Year 2020, Report #21-01506-76 (Washington, D.C.: February 17, 2022). 8See John F. McCarthy et. al., "Predictive Modeling and Concentration of the Risk of Suicide: Implications for Preventive Interventions in the U.S. Department of Veterans Affairs," American Journal of Public Health,. Vol. 105, No. 9,September 2015; Kessler R.C., Hwang I, Hoffmire C.A, et al. "Developing a Practical Suicide Risk Prediction Model for Targeting High-Risk Patients in the Veterans Health Administration," International Journal of Methods in Psychiatric Research. 2017;26(3):epub 2017 Jul 4; and John F. McCarthy et. al., "Evaluation of the Recovery Engagement and Coordination for Health- Veterans Enhanced Treatment Suicide Risk Modeling Clinical Program in the Veterans Health Administration," JAMA 2021;4(10): Oct. 18, 2021. Page 3 GAO-22-105165 Veteran Suicide Risk was identified by REACH VET at each facility from February 2017 through June 2021. Additionally, we conducted virtual site visits at a selection of seven VHA medical facilities. 9 We selected facilities representing a range of geographic locations and complexity of the services provided. To further reflect a range of facility types, we selected facilities based on the number of unique veterans served, the number of facility veterans identified by the REACH VET model, and the number that already had a high-risk flag from February 2017 through June 2021, according to our analysis of the two VHA lists. During our site visits, we discussed facility officials' views of the REACH VET program. While not generalizable, these virtual site visits provide illustrative examples of how facilities are identifying and monitoring the care that veterans receive through identification by the REACH VET program. We also discussed the accuracy of the list of veterans that had received high-risk flags, as well as the list of veterans identified by REACH VET with facility officials. Additionally, we interviewed knowledgeable VHA officials and reviewed related agency documentation about how both lists were generated. We determined that the list of veterans with high-risk flags and those veterans identified through REACH VET were sufficiently reliable for our audit objectives. We conducted this performance audit from April 2021 to September 2022 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. 9VHA facilities selected were in Marion, Indiana; Shreveport, Louisiana; Poplar Bluff, Missouri; Albuquerque, New Mexico; Lebanon, Pennsylvania; Big Spring, Texas; and Spokane, Washington. Page 4 GAO-22-105165 Veteran Suicide Risk VHA operates the largest integrated health care system in the country, Background providing care to more than 9 million veterans at 1,298 locations, including medical centers and community-based outpatient clinics. 10 In May 2017, VHA established OMHSP to consolidate its mental health and suicide prevention programs under one office. 11 OMHSP is responsible for (1) monitoring and supporting the implementation of mental health and suicide prevention policies in VHA facilities, and (2) conducting ongoing evaluations of mental health services and policies. Additionally, according to VA policy, OMHSP has responsibility for overseeing and ensuring that data systems are used to identify and re- evaluate veterans at statistical risk for suicide. 12 In 2018, VA issued its National Strategy for Suicide Prevention (National Strategy) to provide a framework for identifying priorities, organizing efforts, and contributing to a national focus on veteran suicide prevention. 13 The National Strategy identifies four focus areas: (1) healthy and empowered veterans, families, and communities; (2) clinical and community preventative services; (3) treatment and support services; and (4) surveillance, research, and evaluation. Collectively, these four areas encompass VA's 14 goals for preventing suicide, which include objectives to deliver interventions to reduce suicidal thoughts and behaviors among veterans with suicide risk, and to refine and expand the use of predictive analytics for at-risk veterans. VA's National Strategy states that predictive analytical tools, such as the REACH VET model, have the potential to serve as a support tool to inform clinical decision-making to any system that has access to large sets of data. According to VA's National Strategy, the use of predictive analytics to identify veterans at the highest risk for suicide shows significant potential, but efforts to use these tools should also be 10Community-based outpatient clinics are stand-alone clinics that are geographically separate from VHA medical centers and provide outpatient primary care, mental health care, and, in some cases, specialty care services. These outpatient clinics are administratively assigned to a VHA medical center or a VHA health care system, which is made up of multiple VHA medical centers. 11Prior to May 2017, suicide prevention and mental health issues were organizationally located under different offices within VHA. 12See Department of Veterans Affairs, VHA Directive 1160.07 (Washington, D.C.: May 24, 2021). 13See Department of Veterans Affairs, National Strategy for Preventing Veteran Suicide 2018-2028 (Washington, D.C.: June 29, 2018). Page 5 GAO-22-105165 Veteran Suicide Risk continually refined and evaluated to improve efficiency and impact. The National Strategy also notes that a risk-based approach for targeting outreach to veterans is limited in use to individual-level impacts for relatively small numbers and cannot significantly reduce the overall veteran suicide rate. The REACH VET program is intended to complement other VA suicide VHA Uses REACH prevention strategies, such as Risk ID, by providing tools to help VET to Help Identify clinicians identify veterans at higher statistical risk for suicide, according to VHA officials. Clinicians determine whether the veterans identified and Provide through the REACH VET model are receiving appropriate care by Enhanced Care for reviewing veterans' VHA electronic health records. After reviewing the health records, clinicians then contact the veterans to discuss REACH Veterans Potentially VET identification and offer care enhancements that may help the at High Risk for veteran, including mental health or specialty care. Suicide REACH VET uses a predictive model to identify veterans at high risk for suicide. According to VHA officials, each month, VHA staff run the REACH VET model on an electronic file of veterans' VHA health records. The model produces a list containing the names of the top 0.1 percent of veterans at each VHA medical facility with the greatest statistical risk for suicide or other adverse events, which is stored on a dashboard on an internal VHA website. The number of veterans identified at each VHA facility varies based on the size of the veteran population that the facility serves. For example, for our seven selected facilities, the number of veterans identified by REACH VET in each medical facility during June 2021 ranged from 16 to 46 veterans. To identify veterans who are at the greatest statistical risk for suicide, VHA officials stated that the REACH VET program model analyzes six categories of variables-a total of 61 variables-in veterans' VHA electronic health records. These variables include documentation of previous suicide attempts and mental health diagnoses. 14 (See Figure 1 and for a full list of REACH VET model variables, see Appendix IV.) 14See Kessler R.C., Hwang I, Hoffmire C.A, et al. "Developing a Practical Suicide Risk Prediction Model for Targeting High-Risk Patients in the Veterans Health Administration," International Journal of Methods in Psychiatric Research. 2017;26(3):epub 2017 Jul 4. Page 6 GAO-22-105165 Veteran Suicide Risk Suicide Predictive Modeling According to the Veterans Health Administration (VHA), the Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment (REACH VET) program represents the first health system application of suicide predictive modeling to support clinical interventions. The work to develop REACH VET began in 2013, when scientists at Office of Mental Health and Suicide Prevention and the National Institute for Mental Health developed and validated a suicide predictive model among VHA patients. The original REACH VET model identified 381 variables from the VHA electronic health record. To improve the model, in 2017, VHA documents show that it applied machine- learning methods to generate an algorithm with comparable predictive power that required fewer variables, resulting in the current model that uses 61 variables. Source: GAO analysis of Veterans Health Administration information | GAO-22-105165 Figure 1: Examples of Variables Used in VHA's REACH VET Model (as of March 2021) Note: All of the veterans identified by Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment (REACH VET) received care from the Veterans Health Administration (VHA) within the previous 2 years. For veterans identified by the REACH VET model, VHA guidance identifies required intervention steps for addressing the potential risk for suicide. At each VHA medical facility, a REACH VET coordinator is responsible for accessing the list on the REACH VET dashboard Page 7 GAO-22-105165 Veteran Suicide Risk containing the names identified by the REACH VET model. 15 When a veteran is first identified by the REACH VET model at a facility, the facility's REACH VET coordinator is required to identify an appropriate VHA provider to support the identified veteran, and to document these actions in the veteran's electronic health record using a specific note (REACH VET coordinator note). 16 According to VHA guidance, the provider is then required to review the veteran's electronic health record to identify any potential gaps in care-such as additional specialty services that could help the veteran or follow-up for missed appointments. The provider is also required to notify the veteran of their increased statistical risk status, to discuss with them options for enhanced care, and to document the interaction with the veteran by placing a note in the veteran's electronic health record (REACH VET provider note). 17 (See Figure 2.) 15According to REACH VET guidance, facility directors are required to appoint a facility coordinator (REACH VET coordinator), who is responsible for: organizing the implementation and operations of the program; ensuring that mental health and primary care providers are aware of their responsibilities; and that notifications regarding veterans at increased statistical risk reach the appropriate providers. Additionally, mental health and primary care providers are responsible for acknowledging notification about which of their veteran patients are at high statistical risk; reviewing diagnoses and treatment plans; and enhancing access and care when appropriate. See Department of Veterans Affairs Memorandum, REACH VET: Recovery Engagement and Coordination for Health- Veterans Enhanced Treatment, August 10, 2016. 16According to VHA guidance, REACH VET coordinator tasks should be completed within one week of new names being populated on the dashboard and all provider tasks completed the week after. 17For veterans who are already receiving mental health services, VHA guidance states that the mental health provider will have responsibility for performing the REACH VET Provider tasks. For others, the primary care provider or mental health staff will have primary responsibility. Page 8 GAO-22-105165 Veteran Suicide Risk Figure 2: Veterans Health Administration (VHA) Required Steps for Veterans Identified by REACH VET Note: All of the veterans identified by Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment (REACH VET) received care from the Veterans Health Administration (VHA) within the previous 2 years. An official at one VHA site told us that REACH VET targets people who may benefit from mental health treatment due to medical issues. For example, the facility official stated that REACH VET has identified newly diagnosed cancer patients due to heavy hospitalization use. According to the official, although these patients are coming into the facility for medical treatment, they are not engaging with mental health. The official stated that REACH VET can help these veterans the most. When the REACH VET list of veterans is produced in the next month, facility staff and providers are not required to repeat outreach for veterans already identified by REACH VET. Instead, it is up to the veteran's provider to determine if outreach is necessary. When a veteran is identified by the REACH VET model in subsequent months, the REACH VET coordinator typically notifies the provider previously identified for the veteran through a REACH VET coordinator note in the veteran's electronic health record. According to VHA guidance, providers can determine whether additional outreach is necessary after reviewing the Page 9 GAO-22-105165 Veteran Suicide Risk veteran's electronic health record. 18 Unless a change in care is warranted, according to VHA officials, documentation in the VHA electronic health record is not necessary. VHA officials can track whether facility staff complete the required steps through VHA's REACH VET dashboard. According to VHA, the REACH VET dashboard is used to track whether the REACH VET coordinator, as required, identifies a provider and whether the provider re-evaluates the veteran's care, and conducts outreach to the veteran. 19 Additionally, as part of a 2021 study on REACH VET, VHA found that in December 2018, coordinators were identified for about 98 percent of veterans identified by the REACH VET model; care was evaluated for about 88 percent of veterans; and outreach was attempted for 86 percent of veterans. 20 VHA officials reported that, as of June 2022, coordinators were identified for 100 percent of veterans identified by the REACH VET model, care was evaluated for 99 percent of veterans, and outreach was attempted for 99 percent of veterans. VHA has conducted studies of the REACH VET program to determine VHA's Analysis whether the model identifies veterans at high risk for suicide and whether Shows Potential the program is providing a benefit to veterans. A 2015 VHA study of the REACH VET model noted that it largely identified veterans not already Benefits of REACH identified by other screening programs, supporting the idea that such a VET, and VHA Plans program added benefit. The 2015 study found there was some overlap between veterans identified by REACH VET and veterans with an active Further Study high-risk flag (21 percent). 21 These findings suggest that most veterans 18Departmentof Veterans Affairs REACH VET Program – Frequently Asked Questions, Updated November 8, 2021. 19According to VHA officials, the REACH VET dashboard captures information from REACH VET coordinator and REACH VET provider notes from a veteran's electronic health record. 20See John F. McCarthy et. al., "Evaluation of the Recover Engagement and Coordination for Health-Veterans Enhanced Treatment Suicide Risk Modeling Clinical Program in the Veterans Health Administration," JAMA Network Open. 2021;4(10):e2129900. doi:10.1001/jamanetworkopen.2021.29900, October 18, 2021. 21See John F. McCarthy et. al., "Predictive Modeling and Concentration of the Risk of Suicide: Implications for Preventive Interventions in the U.S. Department of Veterans Affairs," American Journal of Public Health. Vol. 105, No. 9, September 2015. According to the study, the goal of modeling was to provide information about the feasibility of selective, risk-stratified preventative interventions, so the study focused on the extent to which suicide risk was concentrated in specific strata of patients-the top 0.10 percent and 5.00 percent of calculated risk. Further, according to the study, 21 percent (1,253 of 5,969 veterans) in the top 0.10 percent of calculated risk had a high-risk flag. Page 10 GAO-22-105165 Veteran Suicide Risk identified by REACH VET did not already have an active high-risk flag. Further, the study concluded that the associations between patients with high-risk flags and those identified as potentially at high risk through predictive modeling support the validity of the REACH VET model, and confirmed that predictive modeling can provide new information about who is at risk. Similar to VHA's 2015 study, our analysis of VHA information shows that the REACH VET model identified veterans who are potentially at high risk for suicide, including those who had not been identified through other clinical methods. Specifically, we found that about 76 percent of the veterans identified by the REACH VET model (74,888 out of 98,814 veterans) from February 2017 through June 2021 did not have a high-risk flag prior to REACH VET identification. Additionally, 6 percent of veterans identified by the REACH VET model (5,438 out of 98,814 veterans) received a high-risk flag after REACH VET identification. In total, 29,364 veterans were identified by the REACH VET model and had a high-risk flag at least one month during the review period (see figure 3). Figure 3: Overlapping Veteran Populations Identified through REACH VET and with a High Risk for Suicide Patient Record Flag (February 2017 through June 2021) Notes: Overlap in the graphic represents the number of veterans receiving care from the Veterans Health Administration that were identified by Recovery Engagement and Coordination for Health- Veterans Enhanced Treatment (REACH VET) and received a high-risk flag at any time from February 2017 through June 2021. Of the 98,814 veterans identified by REACH VET, 6 percent (5,438) received a High Risk for Suicide Patient Record Flag following the REACH VET identification. Page 11 GAO-22-105165 Veteran Suicide Risk VHA also studied the benefits for veterans of being involved in the REACH VET program. Specifically, in October 2021, a VHA study found that for veterans identified through the REACH VET program, there were associated benefits-such as increased outpatient appointment completion, greater initiation of suicide safety plans, and reductions in documented suicide attempts. 22 Although the study acknowledged some limitations, including that it was not possible to test whether individual REACH VET program components were associated with these outcomes, the study's conclusions were positive overall. For example, the study concluded that clinical programs using risk modeling may be effective tools to support care enhancements and risk reduction, and that the REACH VET program represents a promising intervention that enhances care and that further study was needed. VHA medical facility officials have raised questions about the extent to which the REACH VET program is resulting in additional veterans receiving care who are not already receiving care through the VHA health care system. 23 Specifically, officials at all seven VHA facilities told us that most of the veterans identified through the REACH VET program were already known to them because the veterans were already engaged in care. Additionally, our analysis of the VHA lists found that a majority of REACH VET model identifications (72 percent) were for veterans who had already been identified by the model at least once. 24 However, according to VA officials, this observation is not unexpected, as the model 22See John F. McCarthy et. al., "Evaluation of the Recover Engagement and Coordination for Health-Veterans Enhanced Treatment Suicide Risk Modeling Clinical Program in the Veterans Health Administration," JAMA Network Open. 2021;4(10):e2129900, Oct. 18, 2021. The study addressed whether VHA's REACH VET program is associated with health care utilization, treatment engagement, suicide attempts, suicide safety plan documentation, and suicide mortality. The study compared changes in outcomes for veterans before and after REACH VET program entry to those of a similarly identified veteran cohort in the 0.1 risk suicide tier prior to the REACH VET program initiation. 23According to VHA officials, the REACH VET model only considers veterans who have used VHA services in the prior two years. As stated previously, VA data show that 6.1 million out of 20 million veterans used VA health care in fiscal year 2017. See Department of Veterans Affairs, VA Utilization Profile Fiscal Year 2017 (May 2020). 24Based on our review of VHA information, repeat identifications by REACH VET are common, and the majority of REACH VET identifications are for veterans who have previously been identified. Specifically, our analysis of VHA information found that 72 percent of REACH VET identifications from February 2017 through June 2021 (252,420 out of 351,234 identifications) were for veterans who had been identified by REACH VET more than once. Additionally, more than 1,400 veterans were identified by REACH VET more than 20 times from February 2017 through June 2021. Page 12 GAO-22-105165 Veteran Suicide Risk assesses VHA records in the prior 24 months and from month to month there may be substantial consistencies. VHA officials also told us that planned future studies would examine how to optimize the REACH VET program. Specifically, according to VHA officials, internal unpublished program evaluations of the REACH VET program's implementation found that the REACH VET model may predominately identify veterans who are already receiving VA's highest levels of mental health care. This situation can occur because the REACH VET model identifies some veterans repeatedly, as we found in our analysis. In addition, the model identifies veterans who had an inpatient mental health stay and already had a high-risk flag. VHA officials stated that these findings suggest that the REACH VET program may increase its impact by combining identification from the REACH VET model with information on current treatment engagement to more specifically target outreach to the highest risk patients who show signs of disengagement from VHA health care or their treatment plans. VHA has developed a proposal to pilot this approach and study the effects on both high-risk veterans who are disengaged from VHA health care and those that are already engaged in VHA health care. For a full list of VHA's proposed studies on the REACH VET program, see Appendix V. We provided a draft of this report to VA for review and comment. VA's Agency Comments response is reproduced in appendix VI. VA also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. Page 13 GAO-22-105165 Veteran Suicide Risk If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at silass@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs are on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VII. Sharon M. Silas Director, Health Care Page 14 GAO-22-105165 Veteran Suicide Risk Appendix I: Veterans at High Risk for Appendix I: Veterans at High Risk for Suicide: Identify, Intervene, and Monitor Suicide: Identify, Intervene, and Monitor Section 205 of the Commander John Scott Hannon Veterans Mental Health Care Improvement Act of 2019 included provisions for us to describe how the Department of Veterans Affairs (VA) identifies, intervenes, and monitors veterans considered as high risk for suicide. 1 Identify Veteran Suicide Veterans are screened, at least annually, for suicide risk when they Risk through Screening receive care from the Veterans Health Administration (VHA) or are identified by the Recovery Engagement and Coordination for Health- Veterans Enhanced Treatment (REACH VET) model. According to VA policy, VHA facilities are responsible for ensuring that all clinical processes incorporate screening for suicide risk. 2 In 2018, VHA adopted a standardized process for suicide risk screening and clinical evaluation (which VHA refers to as Risk ID) for all veterans receiving VHA health care. According to VHA officials, Risk ID is a two-stage process in which 1. veterans, when interacting with VHA health care providers, are asked screening questions about suicidal ideation, planning, and intent, as well as their history of suicidal ideation with intent to die, and 2. veterans who screen positive on the screening questions, are administered a more detailed suicide risk evaluation. 3 The policy further states that based on the veteran's detailed suicide risk evaluation, VHA clinicians are to determine the veteran's level of suicide 1Pub. L. No. 116-171, tit. II, § 205(b)(1)-(b) (3), 134 Stat. 778, 799-800 (2020). 2See Department of Veterans Affairs Memorandum, Eliminating Veteran Suicide: Suicide Risk Screening and Evaluation Requirements and Implementation (Risk ID Strategy), November 13, 2020. 3According to VA, Risk ID includes screening through use of the Columbia Suicide Severity Rating Scale, which includes specific questioning about suicidal ideation, planning, and intent and a history of suicidal ideation with intent to die. According to VA, a veteran is screened whenever engaged in care at a VHA facility. A positive Columbia Suicide Severity Rating Scale requires completion of the VA Comprehensive Suicide Risk Evaluation, which is a more detailed suicide risk assessment that informs clinical impressions about acute and chronic risk. Page 15 GAO-22-105165 Veteran Suicide Risk Appendix I: Veterans at High Risk for Suicide: Identify, Intervene, and Monitor acuity-that is, the level of immediate risk for suicide-and then administer the appropriate course of treatment based on the risk level. 4 The management of veterans at risk for suicide, according to the VA clinical practice guidelines, can range from mental health hospitalization for veterans with high acute risk to outpatient management for veterans with low acute risk. 5 (See figure 4.) 4See the Department of Veterans Affairs, Rocky Mountain Mental Illness Research, Education, and Clinical Center's Therapeutic Risk Management Risk Stratification Table, https://www.mirecc.va.gov/visn19/trm/ accessed June 8, 2022. According to the table, there are three levels of acute risk-high, intermediate, and low. High Acute Risk is defined as suicidal ideation with intent to die by suicide and an inability to maintain safety independent of external support or help, and often includes a plan for suicide or a recent attempt. Intermediate Acute Risk is suicidal ideation to die by suicide but with the ability to maintain safety, independent of external support. Intermediate risk generally includes the lack of intent based upon an identified reason for living. Low Acute Risk has no current suicidal intent, and no specific and current suicidal plan and no preparatory behaviors. 5Clinicalpractice guidelines are evidence-based recommendations intended to help providers improve the consistency and quality of care in determining the best treatment options for a particular condition. See Department of Veterans Affairs/Department of Defense, VA/DOD Clinical Practice Guideline for the Assessment and Management of Patients at Risk for Suicide. May 2019. Page 16 GAO-22-105165 Veteran Suicide Risk Appendix I: Veterans at High Risk for Suicide: Identify, Intervene, and Monitor Figure 4: Clinical Practice Guideline for Screening, Evaluating, and Managing Veterans at Acute Risk for Suicide Notes: When a veteran is identified by the Recovery Engagement and Coordination for Health- Veterans Enhanced Treatment (REACH VET) program model for the first time, a provider reviews the Page 17 GAO-22-105165 Veteran Suicide Risk Appendix I: Veterans at High Risk for Suicide: Identify, Intervene, and Monitor veteran's electronic health record and contacts the veteran. During the contact, the provider may screen or assess the veteran for suicide risk. According to VHA policy, when a clinician evaluates a veteran to be at high acute risk for suicide, VHA facility staff are to place a High Risk for Suicide Patient Record Flag (high-risk flag) in the veteran's electronic health record. According to VHA, the primary purpose of a high-risk flag is to alert VHA staff that a veteran is at high risk for suicide so clinical staff can take this information into account as they make treatment decisions and provide care. As such, high-risk flags are viewable to any VHA medical facility staff member with access to the veteran's electronic health record. Intervene with Enhanced According to VHA policy, facilities must offer enhanced clinical care to Clinical Care and Monitor veterans when they have a high-risk flag. 6 Specifically, VHA policy Veterans Identified as requires facilities to complete specific clinical actions for a veteran with a high-risk flag, including: High Risk for Suicide 1. offering at least four visits with a mental health professional within 30 days of the flag's activation; 7 2. completing a safety plan with the veteran within 7 days of the flag's activation; 8 and 3. reviewing the veteran's status to determine whether to continue or inactivate the flag between 80 to 100 days after the flag's initial activation. 6See Department of Veterans Affairs Memorandum, Patients at High-Risk for Suicide, April 24, 2008, (updated to High Risk for Suicide Patient Record Flags (HRS-PRF) Changes, October 5, 2021). 7According to VHA policy, patients who are admitted for hospitalization as a result of a high risk for suicide ideation, must be placed on the high-risk list, and kept on the list for a period of at least 3 months after discharge. They must be evaluated at least weekly during the first 30 days after discharge. Other patients identified as surviving a suicide attempt and those who are placed on the high-risk list for other reasons should be evaluated at least weekly for at least the next month. See Department of Veterans Affairs Memorandum, Patients at High-Risk for Suicide, April 24, 2008, (updated to High Risk for Suicide Patient Record Flags (HRS-PRF) Changes, October 5, 2021). 8According to VHA policy, safety plans are prioritized lists of concrete coping strategies and resources veterans create in collaboration with a care team provider to maintain safety and regain equilibrium. Safety plans should be used during times of distress and are meant to both de-escalate crises and provide effective strategies to help veterans avoid suicidal states. Page 18 GAO-22-105165 Veteran Suicide Risk Appendix I: Veterans at High Risk for Suicide: Identify, Intervene, and Monitor According to VHA officials, a veteran's voluntary engagement with VHA clinicians is required to complete these actions. Facility suicide prevention coordinators (SPC) are responsible for managing veterans with high-risk flags and ensuring they have been offered the required enhanced care, according to VHA's Suicide Prevention Program Guide. 9 Facility SPCs are also responsible for managing high-risk flags in veterans' records and determining whether to continue or deactivate a veteran's high-risk flag in consultation with the veteran's treatment providers. 10 VHA policy states that, for a veteran engaged in mental health care, the SPC and the veteran's providers must consult each other before a flag can be inactivated. The SPC must document the consultation in the veteran's electronic health record and show there has been a reduction in suicide risk. If a veteran chooses not to engage in the enhanced care offered by the provider, facilities are required to document their attempts to engage the veteran in care and consult with the veteran's treatment providers before inactivating the flag, according to VHA policy. Following inactivation of a high-risk flag, VHA requires facility SPCs to continue personal contact with the veteran through its U.S. Mail Program for Caring Communications, in which VHA sends monthly notes to all veterans who received a high-risk flag, for at least 1 year following flag inactivation. 11 9Case management is the practice of planning, facilitating, and coordinating care for health care patients. See Department of Veterans Affairs, Veterans Health Administration, Suicide Prevention Program Guide (revised November 2020, issued January 2021). 10Department of Veterans Affairs, Inactivation Process for Category I High Risk for Suicide Patient Record Flags, VHA Notice 2021-10, (Washington, D.C.: May 28, 2021). According to VHA policy, the decision to inactive a flag occurs between 80 to 100 days after the flag's initial activation. The inactivation process for veterans engaged in mental health care requires a clinical consultation between the veteran's clinicians and the SPC, as well as a review of the veteran's electronic health record. For veterans who have not engaged in care, VHA policy states that the flag may be inactivated after consideration of indicators of acute high risk of suicide. In addition to documenting all available indicators of risk, the rationale for inactivation in the veteran's electronic health record must demonstrate attempts to engage the veteran in care. 11According to VHA, the caring communication is an evidence-based intervention for suicide prevention. SPCs are to send regular, brief, non-demanding notes through the U.S. Postal Service to all veterans after a high-risk flag has been inactivated. SPCs are required to send caring notes to the veteran every month for a minimum of 1 year after inactivation of the high-risk flag. Page 19 GAO-22-105165 Veteran Suicide Risk Appendix I: Veterans at High Risk for Suicide: Identify, Intervene, and Monitor To assess the completion of required clinical actions for veterans with high-risk flags, VHA uses the High Risk Flag Patient Tracking Report Dashboard. 12 Both facility SPCs and VHA officials can access and review this dashboard, which provides a near real-time facility-level list of veterans with high-risk flags including veteran-specific data on 1. the date a safety plan was completed; 2. the date a high-risk flag is due for review; 3. the number of completed mental health visits within 30, 60, and 90 days of high-risk flag activation; 4. the most recent mental health encounters a veteran has completed; and 5. inpatient admissions and other possible adverse events. When facility staff enter a clinical action into a veteran's electronic health record-such as completion of a mental health visit or completion of a safety plan-the dashboard is updated. At each facility, the SPC is responsible for monitoring the dashboard. SPCs told us that the dashboard provided a snapshot to help determine whether the required high-risk flag actions were completed for each veteran and, if not completed, they could take steps to complete the action. For example, if a safety plan was not completed, the dashboard would show that one was required, and then the SPC would reach out to the responsible provider. Some facility officials told us that they also use locally-developed spreadsheets to help monitor the completion of required actions. Additionally, according to VHA officials in the Office of Mental Health and Suicide Prevention (OMHSP), the dashboard allows VHA to track compliance with high-risk flag actions because it aligns with VA's 12The High Risk Flag Patient Tracking Report Dashboard is a data repository that holds information about veterans with high-risk flags. Page 20 GAO-22-105165 Veteran Suicide Risk Appendix I: Veterans at High Risk for Suicide: Identify, Intervene, and Monitor Strategic Analytics for Improvement and Learning system. 13 VHA officials told us that local leadership-at the Veterans Integrated Service Network or the VHA facility-are responsible for ensuring that the actions are completed. If a facility is not achieving the benchmarks related to suicide prevention, OMHSP officials told us they would meet with the facility to resolve the issue. In addition to the monitoring performed through the dashboard, the VA Office of Inspector General has reviewed facility compliance with the high-risk flag program requirements. For example, from November 4, 2019 through September 21, 2020, the VA Office of Inspector General initiated unannounced inspections at 36 VHA medical facilities to assess whether facilities complied with selected requirements for SPC processes and provision of suicide prevention care, including compliance with high- risk flag requirements. The VA Office of Inspector General found general compliance across the 36 facilities with requirements for tracking of high- risk veterans and timely completion of suicide prevention safety plans with the required plan elements. 14 13VA's Strategic Analytics for Improvement and Learning system is a diagnostic tool that allows VHA to assess medical facilities' performance relative to their peers and determine year-to-year improvement based on relevant clinical data. For example, VA's system includes an online performance management tool that tracks performance measures related to medical center access, outcomes, and productivity, and includes an early warning system to notify the Veterans Integrated Service Network and facility officials of results that require action. See GAO, Veterans Health Administration: Regional Networks Need Improved Oversight and Clearly Defined Roles and Responsibilities, GAO-19-462 (Washington, D.C.: June 19, 2019). 14See Department of Veterans Affairs, Office of Inspector General, Comprehensive Healthcare Inspection Summary Report: Evaluation of Mental Health in Veterans Health Administration Facilities, Fiscal Year 2020, Report #21-01506-76 (Washington, D.C.: February 17, 2022). Page 21 GAO-22-105165 Veteran Suicide Risk Appendix II: Veterans Health Administration Appendix II: Veterans Health Administration Suicide Prevention Staffing Suicide Prevention Staffing Section 205 of the Commander John Scott Hannon Veterans Mental Health Care Improvement Act of 2019 included a provision for us to review staffing levels of suicide prevention coordinators (SPC) across the Veterans Health Administration (VHA). 1 According to VHA policy, SPC responsibilities include serving as the facility subject matter expert for matters related to suicide including prevention, intervention, and reporting, among other things. 2 This appendix provides information on VHA's SPC staffing levels as of April 15, 2022. VHA policy states that for each medical facility and very large community- based outpatient clinic, VHA medical facility directors are required to ensure that a minimum of one full time equivalent SPC is assigned to each location, with a full-time commitment to the Suicide Prevention Program. 3 VHA guidance also states that facilities are encouraged to hire additional staff to support their suicide prevention teams based on a staffing model of one full time equivalent for every 10,000 unique patients. 4 Based on this guidance, according to VHA officials, the minimum required number of SPCs is 266. According to VHA officials, as of April 15, 2022, the number of SPCs across VHA facilities is 345. 5 We conducted a review of VHA suicide prevention staffing in April 2021, and found that the staffing model benchmark may not accurately reflect 1Pub. L. No. 116-171, tit. II, § 205(b)(4), 134 Stat. 778, 800 (2020). 2See Department of Veterans Affairs, Veterans Health Administration, Suicide Prevention Program, VHA Directive 1160.07 (Washington, D.C.: May 24, 2021). 3See Department of Veterans Affairs, Veterans Health Administration, Suicide Prevention Program, VHA Directive 1160.07 (Washington, D.C.: May 24, 2021). VHA defines a very large community-based outpatient clinic as a clinic that serves more than 10,000 unique veterans each year. See Department of Veterans Affairs, Veterans Health Administration, Uniform Mental Health Services in VA Medical Centers and Clinics, VHA Handbook 1160.01, September 11, 2008 . 4See Department of Veterans Affairs Memorandum, Guidance for Reporting and Monitoring of Suicide Prevention Program Staffing, August 11, 2021. Additionally, the guidance states that facilities are required to report SPC staffing levels to the Office of Mental Health and Suicide Prevention on a monthly basis. 5We did not audit whether each facility had the required minimum number of SPCs, as this study was limited to recommendation follow-up as agreed with congressional committees. Page 22 GAO-22-105165 Veteran Suicide Risk Appendix II: Veterans Health Administration Suicide Prevention Staffing facilities' staffing needs because it was not developed according to the key practices for staffing model design we previously identified. 6 For example, in April 2021 we found that the benchmark did not account for the increasing workload of teams-such as the addition of activities related to suicide prevention initiatives over time-nor did it account for risk factors-such as suicide rates that vary among veterans served by various facilities. Additionally, we found discrepancies between the SPC staffing data that VHA's Office of Mental Health and Suicide Prevention (OMHSP) provided to us and the actual staffing levels at five VHA facilities included in our review, indicating that VHA's staffing model was not using accurate data. Because the staffing model was not developed following key practices, we found that VHA may not be able to help facilities to appropriately determine their staffing needs. As a result, we concluded that suicide prevention teams may be vulnerable to understaffing, which may leave facilities unable to meet veteran needs related to suicide prevention. In April 2021, we recommended that VHA should ensure that OMHSP incorporates key practices for staffing model design into its determination of facilities' suicide prevention staffing needs. VHA concurred with the recommendation and, in August 2021, reported that the agency had taken steps to address this recommendation-such as ensuring the accuracy of existing staffing data. Since November 2021, officials from OMHSP have provided us with documentation related to VHA's new staffing model, which provided evidence that the design of the model incorporated the key practices we previously identified. As a result of VHA's actions, we consider our recommendation related to suicide prevention staffing needs at VHA medical facilities implemented. In August 2021, VHA updated its staffing model in response to our recommendation to determine the number of SPCs needed to support workload demands at each VHA facility. Based on the updated staffing 6See GAO, VA Health Care: Efforts Needed to Ensure Effective Use and Appropriate Staffing of Suicide Prevention Teams, GAO-21-326 (Washington, D.C.: April 5, 2021). This report was conducted in response to the Support for Suicide Prevention Coordinators Act that included a provision for GAO to review the responsibilities and workload of suicide prevention coordinators. Pub. L. No 116-96 § 2, 133 Stat. 3250 (2019). Page 23 GAO-22-105165 Veteran Suicide Risk Appendix II: Veterans Health Administration Suicide Prevention Staffing model, according to VHA officials, 504 SPCs were recommended in order to satisfy workload demands. 7 7VHA provided us an update in November 2021 that showed the updates to the staffing model, so we consider the recommendation addressed. VHA officials told us that the positions identified by the staffing model are not required by VHA policy, and that some of these workload demands can be satisfied by other suicide prevention staff-such as suicide prevention case managers or program support assistants. Page 24 GAO-22-105165 Veteran Suicide Risk Appendix III: VHA Mental Health Resources Appendix III: VHA Mental Health Resources Offered to Family and Friends Offered to Family and Friends Section 205 of the Commander John Scott Hannon Veterans Mental Health Care Improvement Act of 2019 included a provision for us to review the resources and programming offered to family members and friends of veterans with mental health conditions to assist those veterans in treatment and recovery. 1 This appendix provides information on resources that the Veterans Health Administration (VHA) provides for family members and friends that we identified through our interviews with VHA officials at seven selected VHA medical facilities. 2 VHA provides resources at the national and local facility level to educate and assist family and friends of veterans with a mental health condition. According to VHA officials, at the national level VHA resources include: • Make The Connection – an online resource designed to connect veterans, family members, and friends with information and solutions to issues affecting their lives. 3 On the website, visitors can watch hundreds of veterans share their stories; read about a variety of life event and mental health topics, such as ways to start a conversation about mental health or signs a loved one may be experiencing mental health challenges; and locate nearby resources. • Coaching Into Care – a national telephone service that educates and supports family members or friends who are seeking care or services for a veteran. The goal is to help users find the appropriate services at their local VHA facilities. They also provide coaching from licensed psychologists or social workers, free of charge, to family or friends of veterans who see that a veteran may be having difficulty adjusting to civilian life after separating from the military. • S.A.V.E (Signs, Ask, Validate, and Encourage and Expedite) – VHA, in collaboration with the PsychArmor Institute, launched an online suicide prevention training video that is designed to help equip anyone who interacts with veterans to demonstrate care, support, and compassion when talking with a veteran who could be at risk for 1Pub. L. No. 116-171, tit. II, § 205(b)(5), 134 Stat. 778, 800 (2020). 2VHA facilities selected were in Marion, Indiana; Shreveport, Louisiana; Poplar Bluff, Missouri; Albuquerque, New Mexico; Lebanon, Pennsylvania; Big Spring, Texas; and Spokane, Washington. 3See https://www.maketheconnection.net/. Accessed June 22, 2022. Page 25 GAO-22-105165 Veteran Suicide Risk Appendix III: VHA Mental Health Resources Offered to Family and Friends suicide. According to VHA officials, the 25-minute training video is available for free. 4 • Veterans Crisis Line– provides 24-hours a day, seven days a week crisis services through phone, chat, and text services for all veterans and their family members and friends. Callers can access the crisis line to gain assistance in supporting their loved one. 5 According to VHA officials, when a family member or friend calls the crisis line expressing concerns about a veteran, crisis line responders follow a standard operating procedure for third-party customers that guides responders to identify the caller's concerns, offer outreach to the veteran of concern and provide approved resources to the third-party caller as appropriate. According to VHA officials, the resources may include providing the third-party with contact information for the local VHA medical facility suicide prevention coordinator (SPC) and additional contact information for Coaching into Care or the Caregivers Support Line. Based on our discussions with officials at selected VHA medical facilities, we confirmed that facility staff use the national level resources. 6. VHA facility officials also told us that they cannot confirm diagnoses without a veteran's consent, so they are not always able to provide diagnosis specific information to families. Officials we spoke with at three VHA medical facilities told us that the Veterans Crisis Line provides referrals to local SPCs for follow-up. For example, if a family member calls the crisis line expressing concern about a veteran, responders from the crisis line are to alert the suicide prevention team at the facility nearest to where the veteran lives. A member of the suicide prevention team is to contact the family member who reached out to provide information about services (without disclosing information about the veteran or confirming the veteran's diagnosis). For example, an official at one of our selected VHA medical facilities told us 4PsychArmor is a national nonprofit organization that provides education and training to improve the health and life outcomes of military-connected individuals. See https://psycharmor.org/courses/s-a-v-e/ (accessed June 22, 2022). 5The Veterans Crisis Line can be reached (1) by dialing 988 and pressing "1" to be connected with a responder; (2) online at www.VeteransCrisisLine.net (accessed June 28, 2022); or (3) by sending a text message to 838255. 6With the exception of family therapy, providing a count of services utilized by family and friends of veterans with a mental health condition at the local level is difficult because the services provided are not linked to specific veterans, and are therefore not documented in veterans' electronic health records. Page 26 GAO-22-105165 Veteran Suicide Risk Appendix III: VHA Mental Health Resources Offered to Family and Friends that a veteran's parent called the crisis line expressing concern that their child was experiencing issues related to substance misuse. Based on that call, the facility SPC reached out to the veteran to encourage treatment. With the consent of the veteran, families and friends can also be involved in specific care, according to VHA officials at some of our selected VHA medical facilities. For example: • Veterans are encouraged to include family members when developing safety plans (prioritized lists of coping strategies that veterans create in collaboration with a care team provider to maintain safety and regain equilibrium). For example, officials at three VHA medical facilities told us that they attempt to get the veteran's consent to involve the veteran's family during safety planning. • VHA can also provide couples counselling and family therapy services (if available) to directly support the veteran and the veteran's family. Officials at six of the seven VHA medical facilities told us that they do provide couples and family therapy services. For example, officials at one VHA medical facility told us that their facility performs marriage counselling for veterans diagnosed with post-traumatic stress disorder. Page 27 GAO-22-105165 Veteran Suicide Risk Appendix IV: REACH VET Program Appendix IV: REACH VET Program Predictive Model Variables Predictive Model Variables This appendix provides details on the evolution of the Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment (REACH VET) model, including variables that the model uses to identify veterans at risk for suicide. According to the Veterans Health Administration (VHA), the REACH VET program applies a suicide prediction model to support identification of patients with an elevated risk of suicide mortality. VHA and the National Institute of Mental Health scientists began work to develop and validate a suicide prediction model using VHA electronic health records in 2013. 1 VHA published an evaluation in 2015 describing its initial predictive model that used 381 variables measuring demographics, mental health, medical diagnoses, and service utilization. The evaluation found that observed suicide rates were 30 times greater in veterans in the top 0.1 percent calculated risk strata compared to an overall validation sample of patients. 2 These findings, according to the study authors, demonstrated that the model can identify strata of patients with substantial increases in their risk for suicide. 3 The size of the model created computational challenges, so VHA determined that it was necessary to create a model using fewer variables. To reduce the size of the model, VHA collaborated with scientists at Harvard University to use data-driven machine learning techniques to identify a model using fewer variables, but with comparable predictive 1The National Institute for Mental Health is the lead federal agency for research on mental disorders. It is one of the 27 institutes and centers that make up the National Institutes of Health, and is part of the U.S. Department of Health and Human Services. 2According to the study, VHA developed and validated an actuarial model of data from electronic medical records and its use for predicting risk of suicide over periods ranging from 1 month to 1 year. Both the development and validation samples contained data from all patients who died from suicide from October 1, 2008, to September 30, 2011, and had received VHA services in the previous 2 years. Because the goal of modeling was to provide information about the feasibility of selective, risk-stratified preventative interventions, the study focused on the extent to which suicide risk was concentrated in specific strata of patients-the top 0.10 percent and 5.00 percent of calculated risk. See John F. McCarthy et. al., "Predictive Modeling and Concentration of the Risk of Suicide: Implications for Preventive Interventions in the U.S. Department of Veterans Affairs," American Journal of Public Health. Vol. 105, No. 9, September 2015. 3The validation sample included VHA suicide decedents and a random sample of living patients in a given month who received VHA services in the previous 2 years. See John F. McCarthy et. al., "Predictive Modeling and Concentration of the Risk of Suicide: Implications for Preventive Interventions in the U.S. Department of Veterans Affairs," American Journal of Public Health. Vol. 105, No. 9, September 2015. Page 28 GAO-22-105165 Veteran Suicide Risk Appendix IV: REACH VET Program Predictive Model Variables power. According to a VHA study published in 2017, the use of 61 variables produced results with predictive power that were comparable to or slightly higher than the model that included 381 variables. 4 VA's Office of Mental Health and Suicide Prevention program evaluation staff at the Program Evaluation Resource Center and the Serious Mental Illness Treatment Resource and Evaluation Center developed coding and systems to apply the model with 61 variables, each month and to share with VHA medical facilities information regarding high-risk veterans. The 61 variables fall into six categories-demographics, diagnosis, medications, utilization, and interaction terms, such as the interaction between marital statuses with gender (see table 1). 5 Table 1: Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment (REACH VET) Algorithm Variables by Category Variable category Variable description Demographics Age Gender Marital Status Race Race/ Ethnicity Region Service Connected Group as Calculated by Veterans Benefits Administration: Any Service Connection Service Connected as Calculated: More than 30 percent Service Connected as Calculated: More than 70 percent Diagnosis Any Suicide attempt in the prior 18 months Any Suicide attempt in the prior 6 months Any Suicide attempt in the prior 1 month Arthritis diagnosis in prior 12 months Arthritis diagnosis in prior 24 months 4See Kessler RC, Hwang I, Hoffmire CA, et al. "Developing a Practical Suicide Risk Prediction Model for Targeting High-Risk Patients in the Veterans Health Administration". International Journal of Methods in Psychiatric Research. 2017;26(3):epub 2017 Jul 4. 5According to VHA, it should not be assumed that variables excluded from this model (such as bipolar II disorder) are not predictors of suicide risk. Results of the final model variable selection may be due to the large number of variables considered and the complex relationships among variables as suicide predictors. The goal of the machine learning methods for building the model is to select the best, smallest combination of variables to predict individuals at high risk for suicide, rather than to distinguish causal relationships or the significance of individual variables. See U.S. Department of Veterans Affairs, Report on REACH VET Program of Department of Veterans Affairs, March 2021. Page 29 GAO-22-105165 Veteran Suicide Risk Appendix IV: REACH VET Program Predictive Model Variables Variable category Variable description Bipolar disorder I diagnosis in the prior 24 months Chronic non-cancer pain diagnosis in the prior 24 months Depression diagnosis in prior 12 months Depression diagnosis in prior 24 months Diabetes mellitus diagnosis in prior 12 months Head and neck cancer diagnosis in prior 12 months Head and neck cancer diagnosis in the prior 24 months Substance use disorder diagnosis in the prior 24 months Systemic lupus diagnosis in the prior 24 months Diagnoses & Utilization Homelessness indicated in the prior 24 months Medications Alprazolam prescription in prior 24 months Antidepressant treatment in prior 24 months Antipsychotic treatment in prior 12 months Clonazepam prescription in the prior 12 months Clonazepam prescription in the prior 24 months Lorazepam prescription in the prior 12 months Mirtazepam prescription in the prior 12 months Mirtazepam prescription in the prior 24 months Mood stabilizer prescription in the prior 12 months Opioid prescription in the prior 12 months Sedative/anxiolytic prescription in the prior 12 months Sedative/anxiolytic prescription in the prior 24 months Statin prescription in the prior 12 months Zolpidem prescription in the prior 24 months Utilization Any Emergency Department visit in the past 2 months Any Emergency Department visit in the past month Any Inpatient Psychiatric Discharge in the prior 12 months Any Inpatient Psychiatric Discharge in the Prior 24 months Any Inpatient Psychiatric Discharge in the prior 6 months Any Inpatient Psychiatric Discharge in the prior month Any mental health treatment in the prior 12 months Any mental health treatment in the prior 24 months Days with inpatient mental health care Veterans Health Administration (VHA) use in the month that is 7 months prior to the selection date Days with inpatient mental health care VHA use in the month that is 7 months prior to the selection date, Squared Days with outpatient mental health care VHA use in the prior month Page 30 GAO-22-105165 Veteran Suicide Risk Appendix IV: REACH VET Program Predictive Model Variables Variable category Variable description Days with outpatient VHA use in the month that is 15 months prior to the selection date Days with outpatient VHA use in the month that is 23 months prior to the selection date Days with outpatient VHA use in the month that is 7 months prior to the selection date Days with outpatient VHA use in the month that is 8 months prior to the selection date Days with VHA use in the month that is 13 months prior to the selection date Days with VHA use in the month that is 7 months prior to the selection date First Use in Prior 5 Years Happened within the prior year Number of Emergency Department visits in the past 24 months Number of Emergency Department visits in the past month Interaction term Interaction between Other anxiety disorder in the prior 24 months (Y/N) and Personality disorder in the prior 24 months (Y/N) Interaction between Marital Status: Divorced (Y/N) and Gender: Male (Y/N) Interaction between Marital Status: Widowed (Y/N) and Gender: Male (Y/N) Source: U.S. Department of Veterans Affairs Report to Congress on the REACH VET Program | GAO-22-105165 Page 31 GAO-22-105165 Veteran Suicide Risk Appendix V: VHA Published, Ongoing, and Appendix V: VHA Published, Ongoing, and Future Studies of the REACH VET Model and Future Studies of the REACH VET Model Program Outcomes and Program Outcomes The Veterans Health Administration (VHA) has published three studies evaluating the Recovery Engagement and Coordination for Health- Veterans Enhanced Treatment (REACH VET) model and program outcomes. Additionally, according to VHA officials, seven studies are either ongoing or planned for the future (see Table 2). Table 2: VHA Studies of REACH VET-Published, Ongoing, Planned and Proposed (April 2022) Status Title Overview of the study Publication date or planned publication date Published Predictive Modeling and This study focused on evaluating the use of September 2015 Concentration of the Risk of Suicide: predictive modeling to identify patients at risk for Implications for Preventive suicide and to supplement ongoing care with Interventions in the US Department of risk-stratified interventions. Veterans Affairs Published Developing a practical suicide risk This study focused on the initial use of the May 2017 prediction model for targeting high‐ predictive model to identity veterans at high risk patients in the Veterans Health suicide risk to target care. Administration (VHA) Published Evaluation of the Recovery This study focused on VHA's implementation of October 2021 Engagement and Coordination for the national clinical program using a suicide risk Health–Veterans Enhanced prediction algorithm, Recovery Engagement Treatment Suicide Risk Modeling and Coordination for Health-Veterans Enhanced Clinical Program in the Veterans Treatment (REACH VET). This study included Health Administration supplemental content. Completed and in Veterans Health Administration's This study reports on the ability of the REACH To be determined press REACH VET Program: Suicide VET suicide prevention program to efficiently Predictive Modeling in Practice identify patients at risk and connect them with care. Ongoing Assessment of Variation in Suicide This study will report on analyses completed to To be determined Predictive Algorithm Performance assess differences in algorithm performance, measured in terms of suicide risk concentration (per McCarthy et al., 2015), by patient age, sex, and race groups. Ongoing Assessment of Variation in REACH This study will report on analyses completed to To be determined VET Program Effects extend VHA's REACH VET effectiveness evaluation (McCarthy et al., 2021) to assess differences in program effects by patient age, sex, and race groups. Planned Expansion of REACH VET Model This expansion study will elaborate on the Fall 2022 Related to Social Determinants REACH VET model to account for social and environmental determinants of health within the communities in which veterans live. Planned Expansion of REACH VET Model for This report will discuss the evaluation of the Summer 2022 Novel Risk Predictors additional benefits of including novel risk predictors in the REACH VET models. Page 32 GAO-22-105165 Veteran Suicide Risk Appendix V: VHA Published, Ongoing, and Future Studies of the REACH VET Model and Program Outcomes Status Title Overview of the study Publication date or planned publication date Proposed Addressing Potential Bias in The report will discuss the use of Stanford's To be determined Predictive Modeling Applied to Data Science for Good program's analytical REACH VET framework and data platform to enable the assessment of predictive models for bias by race, ethnicity, gender, and age, or other subpopulations characteristics. Proposed Expansion of REACH VET related to This report will discuss the implementation of a To be determined Improved Identification of Disengaged proposed revised intervention targeting plan, Veterans using a patient-level randomized program evaluation design. This randomized program evaluation would enable evaluation of the benefits of the REACH VET intervention for disengaged high-risk patients, and potential risks of not conducting case review and outreach with already engaged high-risk patients. Source: U.S. Department of Veterans Affairs | GAO-22-105165 Page 33 GAO-22-105165 Veteran Suicide Risk Appendix VI: Comments from the Appendix VI: Comments from the Department of Veterans Affairs Department of Veterans Affairs Page 34 GAO-22-105165 Veteran Suicide Risk Appendix VII: GAO Contact and Staff Appendix VII: GAO Contact and Staff Acknowledgments Acknowledgments Sharon M. Silas at (202) 512-7114 or silass@gao.gov GAO Contact: In addition to the contact named above, Marcia A. Mann (Assistant Staff Director), Jim Melton (Analyst-in-Charge), Kenisha Cantrell, Margaret Acknowledgments: Devlin, and Brittaini Maul made key contributions to this report. Also contributing were Todd Anderson, Jennie Apter, Jacquelyn Hamilton, Diona Martyn, Amber Sinclair, and Roxanna Sun. (105165) Page 35 GAO-22-105165 Veteran Suicide Risk The Government Accountability Office, the audit, evaluation, and investigative GAO's Mission arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is Obtaining Copies of through our website. Each weekday afternoon, GAO posts on its website newly GAO Reports and released reports, testimony, and correspondence. You can also subscribe to GAO's email updates to receive notification of newly posted products. Testimony Order by Phone The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. Contact FraudNet: To Report Fraud, Website: https://www.gao.gov/about/what-gao-does/fraudnet Waste, and Abuse in Automated answering system: (800) 424-5454 or (202) 512-7700 Federal Programs A. Nicole Clowers, Managing Director, ClowersA@gao.gov, (202) 512-4400, U.S. Congressional Government Accountability Office, 441 G Street NW, Room 7125, Washington, Relations DC 20548 Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Stephen J. Sanford, Managing Director, spel@gao.gov, (202) 512-4707 Strategic Planning and U.S. Government Accountability Office, 441 G Street NW, Room 7814, External Liaison Washington, DC 20548 Please Print on Recycled Paper.