United States Government Accountability Office Report to Congressional Committees ELECTRONIC June 2022 HEALTH RECORDS Additional DOD Actions Could Improve Cost and Schedule Estimating for New System GAO-22-104521 June 2022 ELECTRONIC HEALTH RECORDS Additional DOD Actions Could Improve Cost and Schedule Estimating for New System Highlights of GAO-22-104521, a report to congressional committees Why GAO Did This Study What GAO Found DOD operates one of the nation's The Department of Defense (DOD) began deploying Military Health System largest health care systems providing (MHS) GENESIS-a commercial electronic health record system intended to health care to about 9.6 million integrate inpatient and outpatient medical and dental information in 2017. MHS beneficiaries. It relies on multiple GENESIS's contract award totals $5.5 billion. DOD's plans call for implementing legacy electronic health record MHS GENESIS in 24 waves or phases. The first wave was completed in October systems to create, maintain, and 2017 with the last wave expected to be deployed by December 2023 and manage patient health information. additional activities planned through 2025. DOD determined that these systems, implemented over the past three DOD has not fully met the characteristics and associated best practices for decades, require modernization and developing MHS GENESIS cost and schedule estimates, as shown below. replacement with a comprehensive, real-time electronic health record, MHS GAO Assessment of DOD Cost and Schedule Estimates against Best Practice Characteristics GENESIS. Cost estimate Assessment of cost Schedule estimate Assessment of characteristic estimate characteristic schedule estimate Congress included a provision in the Comprehensive Substantially met Comprehensive Substantially met Joint Explanatory Statement Well-documented Partially met Well-constructed Partially met accompanying the Consolidated Accurate Substantially met Credible Partially met Appropriations Act, 2021 for GAO to Credible Minimally met Controlled Substantially met review DOD's deployment of MHS GENESIS. This report determines the Legend: substantially met = DOD provided evidence that satisfies a large portion of the criterion; extent to which DOD's MHS partially met = DOD provided evidence that satisfies about one-half of the criterion; minimally met = DOD provided evidence that satisfies a small portion of the criterion GENESIS's cost estimate and program Source: GAO assessment of the Department of Defense (DOD) Military Health System GENESIS program documentation. | GAO-22- schedule are consistent with best 104521 practices. GAO reviewed documentation supporting the MHS GENESIS's cost estimate was unreliable because it did not substantially program's October 2020 cost estimate meet all four characteristics of a reliable cost estimate, as described in GAO's against best practices. In addition, cost guide. Specifically, DOD minimally met the "credible" characteristic GAO reviewed the program's February associated with reliable cost estimates, in part, because it did not provide 2021 integrated master schedule, evidence that a sensitivity analysis, a risk and uncertainty analysis, or an specifically assessing three subproject independent cost estimate were conducted. Reliable cost estimates are critical schedules. Further, GAO interviewed for successfully delivering IT programs. DOD officials within the program office to understand their practices for In addition, the subproject schedules did not meet all four characteristics of a developing and maintaining the cost high quality, reliable schedule. Because an integrated master schedule estimate and program schedule. consolidates subproject schedules, errors and reliability issues in subproject schedules will be reflected in higher-level schedules. Therefore, the larger master What GAO Recommends schedule is unreliable. Specifically, the subproject schedules partially met the GAO is making two recommendations "well-constructed" characteristic associated with reliable schedule estimates, in to DOD that it develop reliable cost and part because their critical paths could not be validated or they exhibited total float schedule estimates for the MHS values that could allow activities and milestones to slip months or years before GENESIS program that are consistent delaying key program activities. A reliable schedule can assist with the with GAO-identified best practices. systematic execution of a program and the means by which to gauge progress, DOD concurred with GAO's identify and address potential problems, and promote accountability. recommendations. Because the MHS GENESIS program cost and schedule estimates were not reliable, DOD increases the risk that management will not have the information necessary for effective decision-making. Following cost and schedule best View GAO-22-104521. For more information, contact Carol C. Harris at (202) 512-4456 or practices can help minimize the risk of cost overruns and schedule delays, and harriscc@gao.gov would better position DOD for successful program implementation. United States Government Accountability Office Contents Letter 1 Background 3 Cost Estimate and Subproject Schedules Were Not Fully Consistent with Best Practices 9 Conclusions 16 Recommendations for Executive Action 16 Agency Comments 17 Appendix I Objectives, Scope and Methodology 18 Appendix II Comments from the Department of Defense 21 Appendix III GAO Contact and Staff Acknowledgments 23 Tables Table 1: Four Characteristics and Best Practices of a Reliable Cost Estimate, According to GAO's Cost Estimating and Assessment Guide 6 Table 2: Four Characteristics and Best Practices of a Reliable Schedule, According to GAO's Schedule Assessment Guide 8 Table 3: Assessment of the Department of Defense's (DOD) Military Health System (MHS) GENESIS's October 2020 Cost Estimate Compared to Cost Estimating Best Practices 9 Table 4: Assessment of the Extent to Which Selected Subproject Schedules for Military Health System GENESIS Collectively Met Best Practices 13 Page i GAO-22-104521 Electronic Health Records Abbreviations CARD cost analysis requirements description DOD Department of Defense ICE independent cost estimate IMS integrated master schedule MHS Military Health System OMB Office of Management and Budget SRA schedule risk analysis VA Department of Veterans Affairs WBS work breakdown structure This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page ii GAO-22-104521 Electronic Health Records Letter 441 G St. N.W. Washington, DC 20548 June 8, 2022 The Honorable Jon Tester Chair The Honorable Richard Shelby Ranking Member Subcommittee on Defense Committee on Appropriations United States Senate The Honorable Betty McCollum Chair The Honorable Ken Calvert Ranking Member Subcommittee on Defense Committee on Appropriations House of Representatives The Department of Defense (DOD) operates the Military Health System (MHS), one of the nation's largest health care systems providing health care to about 9.6 million beneficiaries, including service members, retirees, and their family members. MHS provides care to beneficiaries at more than 700 military hospitals and clinics (i.e., military treatment facilities) around the world. DOD relies on multiple legacy electronic health record systems to create, maintain, and manage patient health information. 1 The department has determined that these systems, implemented over the past three decades, require modernization and replacement with a comprehensive, real-time electronic health record system. In 2013, the Secretary of Defense chartered the Program Executive Office of the Defense Healthcare Management Systems (program office) to improve the health care of DOD's beneficiaries by modernizing the electronic health record systems and establishing medical data sharing among DOD, the Department of Veterans Affairs (VA), and the private sector. Toward this end, in 2017 DOD began deploying MHS GENESIS- 1An electronic health record is a collection of information about the health of an individual and the care provided to that individual, such as patient demographics, progress notes, problems, medications, vital signs, past medical history, immunizations, laboratory data, and radiology reports. Page 1 GAO-22-104521 Electronic Health Records a new electronic health record system intended to integrate inpatient and outpatient medical and dental information. The Joint Explanatory Statement accompanying the Consolidated Appropriations Act, 2021 included a provision for GAO to review DOD's deployment of MHS GENESIS. 2 Our objective for this review was to determine to what extent DOD's MHS GENESIS's cost estimate and program schedule are consistent with cost estimating and schedule assessment best practices. To address the objective, we evaluated documentation supporting the program's October 2020 cost estimate against the best practices for developing a comprehensive, accurate, well-documented, and credible cost estimate identified in GAO's Cost Estimating and Assessment Guide. 3 Additionally, we reviewed the MHS GENESIS integrated master schedule (IMS), 4 dated February 2021, and related supporting documentation against the best practices for developing a comprehensive, well-constructed, credible, and controlled schedule identified in GAO's Schedule Assessment Guide. 5 The IMS is detailed and highly complex due to the large number of integrated subprojects and activities. Because an IMS consolidates subproject schedules, errors and reliability issues in subproject schedules will be reflected in higher-level schedules. As such, we based our schedule evaluation on an assessment of three subprojects within the schedule. 6 The subprojects we assessed were selected based on whether they included a significant and varying number of discrete activities in the project schedule and were currently on-going. The three 2Joint Explanatory Statement, accompanying Consolidated Appropriations Act, 2021, Pub. L. No. 116-260, H Comm. Print, Book 1 of 2, Div. C at p. 702 (March 2021). 3GAO, Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Program Costs, GAO-20-195G (Washington, D.C.: March 2020). 4An integrated master schedule connects all the scheduled work of the government and the contractor in a network, or collection of logically linked sequences of activities. As a document that integrates the planned work, the resources necessary to accomplish that work, and the associated budget, it should be the focal point of program management. 5GAO, Schedule Assessment Guide: Best Practices for Project Schedules, GAO-16-89G (Washington, D.C.: December 2015). 6The reliability of an integrated master schedule depends in part on the reliability of its subordinate schedules-weaknesses in these schedules will be reflected in the overall schedule for the program effort. Page 2 GAO-22-104521 Electronic Health Records subprojects represent approximately 20 percent of the remaining activities in the IMS. The subprojects are Apple Product Improvement Engineering (subproject 1), Stage 6 (subproject 2), and Elderberry Product Improvement Engineering (subproject 3). Assessing three subprojects limits possible statements about the program's entire schedule. For example, if the program is not following best practices in creating and maintaining the three subproject schedules, we can conclude that the larger integrated schedule is unreliable. However, if the selected subprojects are deemed reliable, we cannot definitively determine the reliability of the integrated master schedule because the other subprojects that were not assessed may be unreliable. In addition, we interviewed DOD officials within the program office to understand their practices for developing and maintaining the cost estimate and program schedule. Further, we provided DOD with draft versions of our detailed analyses of the MHS GENESIS cost estimate and schedule and asked DOD officials to verify the information on which we based our findings. Appendix I provides additional details on our scope and methodology. We conducted this performance audit from September 2020 to June 2022 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. DOD operates the MHS to support medical readiness by ensuring the Background health and fitness of service members and to support morale by providing medical care to service members, retirees, and their families. In the MHS, health care services are often provided at military training facilities. A wide range of clinical services is available at these facilities, depending on their size, mission, and levels of capability. To support the delivery of health care services, DOD has, over time, developed, procured, and maintained a variety of legacy electronic health record systems. Each system has different functions and capabilities; for example, the department operates separate inpatient, outpatient, and dental systems. The department also operates several other individual systems that are used for managing referrals, tracking medical readiness, and sharing data with VA, among other things. Page 3 GAO-22-104521 Electronic Health Records Since 1998, DOD and VA have worked to exchange electronic health records. Further, in 2008, Congress mandated that the two departments achieve interoperability between their electronic health record systems. 7 DOD determined that its systems needed to be modernized and replaced. In order to modernize its systems and achieve interoperability with VA, DOD is acquiring a commercial product. In July 2015, DOD awarded a $4.3 billion contract to the Leidos Partnership for Defense Health to implement MHS GENESIS, based on commercially available products. In 2018, DOD modified the contract to support incorporating the United States Coast Guard into MHS GENESIS, among other activities, at an additional cost of $1.2 billion. The system is intended to integrate inpatient and outpatient medical and dental information, support the availability of medical records for all DOD beneficiaries worldwide, enable increased standardization, and integrate health care delivery. DOD revised its schedule for completing the deployment of MHS GENESIS in February 2021. This schedule calls for implementing MHS GENESIS in 24 waves (i.e., phases). The first wave was completed in October 2017 and the last wave is expected to deploy by December 2023 with additional activities planned through 2025. In February 2022, DOD estimated that its deployment of MHS GENESIS was 38 percent completed. In September 2021, we reported on the progress that DOD had made in implementing MHS GENESIS and the challenges that remain, including those associated with incidents identified during testing, training and communication, and frequent system changes. 8 We recommended that DOD develop an approach to retest incidents, improve training, and develop a plan to ensure MHS GENESIS users are aware of system 7Interoperabilityallows patients' electronic health information to move with them from provider to provider, regardless of where the information originated. If electronic health records conform to interoperability standards, they can be created, managed, and consulted by authorized clinicians and staff across more than one health care organization, thus providing patients and their caregivers the necessary information required for optimal care. 8GAO, Electronic Health Records: DOD Has Made Progress in Implementing a New System, but Challenges Persist, GAO-21-571 (Washington, D.C.: Sept. 20, 2021). Page 4 GAO-22-104521 Electronic Health Records changes. DOD concurred with and described plans to address the recommendations. Cost Estimating A high-quality, reliable cost estimate is a key tool for budgeting, planning, and managing federal programs. According to the Office of Management and Budget (OMB), programs must maintain current and well- documented estimates of program costs, and these estimates must encompass the full life cycle of the program. 9 Among other things, OMB policy states that generating reliable program cost estimates is a critical function necessary to support the capital programming process. Without this capability, agencies are at risk of experiencing program cost overruns, missed deadlines, and performance shortfalls. A cost estimate provides a structured accounting of all labor, material, and other efforts required to develop, produce, operate and maintain, and dispose of a program. The development of a cost estimate entails identifying and estimating all cost elements that pertain to the program from initial concept all the way through each phase in the program's duration. The program cost estimate encompasses all past (or sunk), present, and future costs for every aspect of the program, regardless of funding source. Our research has shown that in order to conduct oversight of the federal government, including agencies' stewardship of public funds, reliable cost information is required. GAO's Cost Estimating and Assessment Guide (referred to as the cost guide throughout) outlines best practices for developing reliable cost estimates that management can use to make informed decisions. 10 These practices are organized into four characteristics-comprehensive, well-documented, accurate, and credible. Table 1 summarizes the four characteristics and corresponding best practices of a reliable cost estimate identified in the cost guide. 9Office of Management and Budget, Preparation, Submission, and Execution of the Budget, Circular No. A-11 (Washington, D.C.: June 2006); Management of Federal Information Resources, Circular No. A-130 Revised (Washington, D.C.: Executive Office of the President, Nov. 28, 2000); and Capital Programming Guide: Preparation, Submission, and Execution of the Budget, Supplement to Circular A-11, Part 7 (Washington, D.C.: June 2006). 10GAO-20-195G. Page 5 GAO-22-104521 Electronic Health Records Table 1: Four Characteristics and Best Practices of a Reliable Cost Estimate, According to GAO's Cost Estimating and Assessment Guide Characteristic Corresponding best practices Comprehensive • The cost estimate includes all life cycle costs. • The technical baseline description completely defines the program, reflects the current schedule, and is technically reasonable. • The cost estimate is based on a work breakdown structure that is product-oriented, traceable to the statement of work, and at an appropriate level of detail to ensure that cost elements are neither omitted nor double-counted. • The cost estimate documents all cost-influencing ground rules and assumptions. Well-documented • The documentation should show the source data used, the reliability of the data, and the estimating methodology used to derive each element's cost. • The documentation describes how the estimate was developed so that a cost analyst unfamiliar with the program could understand what was done and replicate it. • The documentation discusses the technical baseline description and the data in the technical baseline are consistent with the cost estimate. • The documentation provides evidence that the cost estimate is reviewed and accepted by management. Accurate • The cost estimate is regularly updated to ensure it reflects program changes and actual costs. • The cost model was developed by estimating each work breakdown structure element using the best methodology from the data collected. • The estimate has been adjusted properly for inflation. • The cost estimate contains few, if any, minor mistakes. • Variances between planned and actual costs are documented, explained, and reviewed. • The estimate is based on a historical record of cost estimating and actual experiences from other comparable programs. Credible • The cost estimate included a sensitivity analysis that identifies a range of possible costs based on varying major assumptions, parameters, and data inputs. • A risk and uncertainty analysis was conducted that quantified the imperfectly understood risks and identified the effects of changing key cost driver assumptions and factors. • Major cost elements were cross-checked to see if results were similar. • An independent cost estimate was conducted by a group outside the acquiring organization to determine whether other estimating methods produce similar results. Source: GAO analysis of GAO-20-195G. | GAO-22-104521 The cost guide provides a framework for assessing the extent of an estimate's adherence to each characteristic based on the following scale: • Met: Evidence satisfies the entire criterion. • Substantially Met: Evidence satisfies a large portion of the criterion. • Partially Met: Evidence satisfies about half of the criterion. • Minimally Met: Evidence satisfies a small portion of the criterion. • Not Met: Evidence satisfies none of the criterion. Page 6 GAO-22-104521 Electronic Health Records GAO considers a reliable cost estimate as one that either met or substantially met each characteristic. According to GAO's cost guide, reliable cost estimates are critical for successfully delivering IT programs. Such estimates provide the basis for informed decision-making, realistic budget formulation, meaningful progress measurement, and accountability for results. Management minimizes the risk of cost overruns and unmet performance targets by ensuring cost estimates reflect those characteristics by meeting best practices. Program Scheduling The success of a project depends, in part, on having an integrated and reliable master schedule that defines when and how long work will occur, and how each activity is related to the others. Integrated master schedules are a consolidation of lower level project (i.e. subproject) schedules. A well-planned schedule is a fundamental management tool that can help government programs use public funds effectively by specifying when work will be performed in the future and measuring program performance against an approved plan. Moreover, as a model of time, an integrated and reliable schedule can show when major events are expected as well as the completion dates for all activities leading up to them, which can help determine if the program's parameters are realistic and achievable. We have previously reported in our Schedule Assessment Guide (referred to as the schedule guide throughout) that a reliable schedule can provide a road map for the systematic execution of a program and the means by which to gauge progress, identify and address potential problems, and promote accountability. 11 A schedule provides a time sequence for the duration of a program's activities and helps everyone understand both the dates for major milestones and the activities that drive the schedule. Moreover, it is an essential basis for managing tradeoffs between cost, schedule, and scope. Among other things, scheduling allows program management to decide between possible sequences of activities and determine the flexibility of the schedule according to available resources. Further, it can predict the consequences of managerial action or inaction on events, and allocate contingency plans to mitigate risks. Our research has identified 10 best practices associated with effective schedule 11GAO-16-89G Page 7 GAO-22-104521 Electronic Health Records estimating. These 10 best practices are grouped into four characteristics for sound schedule estimating: comprehensive, well-constructed, credible and controlled. Table 2 summarizes the four characteristics and corresponding best practices of a reliable schedule identified in the schedule guide. 12 Table 2: Four Characteristics and Best Practices of a Reliable Schedule, According to GAO's Schedule Assessment Guide Characteristic Corresponding best practices Comprehensive • Capture all activities, as identified in the work breakdown structure, which defines in detail the work for both the government and its contractors necessary to accomplish a program's objectives. • Establishes the duration of all activities in the same time unit (preferably days) and has specific start and end dates. • Reflects what resources (e.g., labor, materials, and overhead) are needed to do the work, whether all required resources will be available when needed, and whether any funding or time constraints exist. Well-constructed • Sequences all activities-that is, all activities are sequenced in the order that they are to be implemented with the most straightforward logic possible. • Establishes a valid critical path, which represents the chain of dependent activities with the longest total duration. A valid critical path is necessary to examine the effects of any activity slippage along this path. • Identifies the total float time-the amount of time by which an activity can slip before the delay affects the program's estimated finish date-so that a schedule's flexibility can be determined. Credible • Verifies that the schedule is (1) horizontally traceable, meaning that it reflects the order of events necessary to achieve aggregated products or outcomes; and (2) vertically traceable, meaning that activities in varying levels of the schedule align with one another and key dates presented to management in periodic briefings are consistent with the schedule. • Conducts a schedule risk analysis to predict a level of confidence in meeting the program's completion date and the level of necessary schedule contingency. Controlled • Updates schedule regularly using actual progress and logic to realistically forecast dates for program activities. • Maintains a baseline schedule to measure, monitor, and report the program's progress. Source: GAO analysis of GAO-16-89G. | GAO-22-104521 The schedule guide provides a framework for assessing the extent of an estimate's adherence to each characteristic based on the following scale: • Met: Evidence satisfies the entire criterion. • Substantially Met: Evidence satisfies a large portion of the criterion. • Partially Met: Evidence satisfies about half of the criterion. • Minimally Met: Evidence satisfies a small portion of the criterion. 12GAO-16-89G. Page 8 GAO-22-104521 Electronic Health Records • Not Met: Evidence satisfies none of the criterion. GAO considers a reliable schedule as one that either met or substantially met each characteristic. The MHS GENESIS cost estimate and subproject schedules collectively Cost Estimate and satisfied four of the eight characteristics, including the underlying best Subproject Schedules practices, associated with reliability. However, because neither the cost estimate nor the subproject schedules met or substantially met all of the Were Not Fully characteristics, they are unreliable. Consistent with Best Practices MHS GENESIS's Cost DOD's October 2020 MHS GENESIS's cost estimate was unreliable Estimate Substantially Met because it did not substantially meet all four characteristics of a reliable cost estimate described in our cost guide. Specifically, although the cost Two of Four estimate substantially met the comprehensive and accurate Characteristics for characteristics, it partially met the well-documented characteristic and Reliability minimally met the credible characteristic. Table 3 summarizes the results of our assessment of the characteristics and the best practices of MHS GENESIS's October 2020 cost estimate. Table 3: Assessment of the Department of Defense's (DOD) Military Health System (MHS) GENESIS's October 2020 Cost Estimate Compared to Cost Estimating Best Practices Characteristic and corresponding best Assessment Summary of assessment practices Comprehensive ◕ • The cost estimate included all known and anticipated federal • The cost estimate includes all life government and contractor costs from inception through full cycle costs. deployment plus ten years of sustainment. DOD officials reported that key out-of-scope elements were identified, as well as key • The technical baseline description critical dependencies that were outside of the program scope. completely defines the program, reflects the current schedule, and is • DOD officials reported that the baseline is documented with technically reasonable. sufficient detail to develop a robust cost estimate, as are its key changes over time. Specifically, the MHS GENESIS program office • The cost estimate is based on a work provided reports that indicate the cost estimate is based on a breakdown structure (WBS) that is programmatic and technical baseline in the cost analysis product-oriented, traceable to the requirements description (CARD). Officials reported that the CARD statement of work, and at an was developed and is updated in coordination with program appropriate level of detail to ensure management office staff and is reviewed for approval by the that cost elements are neither program manager. However, we found that the CARD was last omitted nor double-counted. updated in 2016, so we cannot confirm that the technical baseline • The cost estimate documents all description has been maintained and updated in preparation for cost-influencing ground rules and program reviews, milestone decisions, and major program assumptions. changes. Page 9 GAO-22-104521 Electronic Health Records Characteristic and corresponding best Assessment Summary of assessment practices • A WBS dictionary exists and is product-oriented and traceable because it defines what is included in each element and how it relates to others in the deliverable-oriented hierarchy. The cost estimate WBS matches the schedule WBS, as well as the earned value management WBS. In addition, the WBS contains an appropriate level of detail, outlining the end-product and major work of the program with hardware and software elements, program management and other common elements. • The cost estimate does not define and document the risks associated with the ground rules and assumptions and the rationale and historical data to support them. In addition, the ground rules and assumptions were not used for any sensitivity and uncertainty analyses. Well-documented ◑ • The electronic cost model listed methods used to identify costs. • The documentation should show the However, the cost estimate documentation does not list details source data used, the reliability of the regarding assumptions, methodology, and areas of uncertainty and data, and the estimating risk that drove the development of the point estimate by the WBS methodology used to derive each element for the cost estimate.a Further, there is no evidence of an element's cost. assessment of the accuracy of the data and reliability and circumstances affecting the data in the cost estimate • The documentation describes how documentation submitted. Finally, the electronic cost model does the estimate was developed so that a not describe how the data were normalized. cost analyst unfamiliar with the program could understand what was • The documentation does not fully describe how the estimate was done and replicate it. developed so that a cost analyst unfamiliar with the program could understand what was done and replicate it. Specifically, the • The documentation discusses the documentation does not contain a narrative with an executive technical baseline description and summary, introduction, and descriptions of methods, with data the data in the technical baseline are broken out by WBS cost elements, sensitivity analysis, risk and consistent with the cost estimate. uncertainty analysis, and updates that reflect actual costs and • The documentation provides changes for the cost estimate. evidence that the cost estimate is • The documentation discusses the technical baseline description. reviewed and accepted by However, the CARD was last updated in 2016 and therefore is management. outdated for purposes of our review because it would not contain current information regarding the current technical and program risks consistent with the cost estimate. • While DOD officials stated that costs are regularly reviewed by management, they did not provide evidence that management reviewed and accepted the cost estimate. Accurate ◕ • The electronic cost model and supporting documentation reflect • The cost estimate is regularly program changes and actual costs. However, the cost estimate did updated to ensure it reflects program not document the changes in technical or programmatic changes and actual costs. assumptions and how these changes affected it. • The cost model was developed by • DOD officials provided evidence that several methods were used. estimating each WBS element using However, they did not provide evidence in the cost model that the the best methodology from the data estimating method was always used appropriately. It was also not collected. clear if overhead and fee were added to the cost estimate when the build-up estimating methodology was used. • The estimate has been adjusted properly for inflation. • Officials reported that the cost data was adjusted for inflation so that it could be described in like terms and to ensure that • The cost estimate contains few, if comparisons and projections were valid. any, minor mistakes. Page 10 GAO-22-104521 Electronic Health Records Characteristic and corresponding best Assessment Summary of assessment practices • Variances between planned and • The cost estimate contained few, if any, minor mistakes. However, actual costs are documented, officials did not provide evidence that the program used a quality explained, and reviewed. control process to ensure the cost estimates contained few, if any • The estimate is based on a historical mistakes. record of cost estimating and actual • The cost estimate supporting documentation contained variances experiences from other comparable and indicated that the cost estimate was refined using lessons programs. learned. However, the variances and lessons learned within the cost estimate supporting documentation did not provide evidence of whether actual costs or schedules differed from the estimate and there was no evidence that sunk costs were compared to the original estimates for the cost estimate. • The cost estimate was based on historical records. However, the historical data did not completely describe the data sources used to determine if the data could be used to estimate accurate costs for the new program. In addition, officials did not provide evidence that steps had been taken to determine that the data were applicable to the program and that data were reliable for the cost estimate. Credible ◔ • DOD did not provide evidence that a sensitivity analysis was • The cost estimate included a performed for the cost estimate.b sensitivity analysis that identifies a • DOD did not provide evidence that a risk and uncertainty analysis range of possible costs based on was performed for the cost estimate. In addition, the time-phased varying major assumptions, risks in the risk tab of the electronic cost model were not allocated parameters, and data inputs. to WBS elements and it was not clear if and how the risk model • A risk and uncertainty analysis was accounted for correlation between risks, between cost elements, conducted that quantified the and how risk was distributed among cost elements when more than imperfectly understood risks and one was affected by a given risk. identified the effects of changing key • Neither the electronic cost model nor supporting documentation cost driver assumptions and factors. contained evidence that different cost estimation methods • Major cost elements were cross- produced similar results across major cost elements. checked to see if results were • Program officials stated that an ICE was performed in 2015 and similar. that they plan to conduct internal peer-reviewed assessments for a • An independent cost estimate (ICE) future update to the cost estimate. However, DOD officials stated was conducted by a group outside that an ICE was not performed for the current cost estimate. the acquiring organization to determine whether other estimating methods produce similar results. Legend: ● = Met: The program office provided complete evidence that satisfies the entire criterion. ◕ = Substantially Met: The program office provided evidence that satisfies a large portion of the criterion. ◑ = Partially Met: The program office provided evidence that satisfies about half the criterion. ◔ = Minimally Met: The program office provided evidence that satisfies a small portion of the criterion. ○ = Not Met: The program office provided no evidence that satisfied any of the criterion. Source: GAO analysis of Department of Defense data and GAO-20-195G. | GAO-22-104521 a A point estimate is the anticipated program costs without risk and uncertainty added. b Sensitivity analysis first identifies key elements that drive cost and their associated assumptions and then calculates the estimate's sensitivity to changes in the underlying assumptions. Page 11 GAO-22-104521 Electronic Health Records MHS GENESIS program officials provided information in response to our assessments of the well-documented and credible characteristics as partially met and minimally met, respectively. Specifically, the officials provided documentation of prior year cost estimates that they asserted included such items as ground rules and assumptions, methodology, a risk and uncertainty analysis, updates that reflect actual cost and changes, and programmatic and technical baseline descriptions. However, the information they referenced was from a previous estimate prepared in 2016 and is outdated for the purposes of our review. With respect to the credible characteristic, program officials also described actions they intend to address specific practices. For example, the officials stated that while they did not conduct an additional detailed sensitivity analysis, they do plan to conduct a formal sensitivity analysis as part of a future update to the cost estimate. Further, while a formal ICE was not conducted, the Program Executive Office for the Defense Healthcare Management System reviewed the cost estimate and planned to conduct internal peer-reviewed assessments as part of a subsequent update to the cost estimate. Without a cost estimate that includes complete documentation on how DOD developed the estimate or evidence that the department conducted an ICE, sensitivity, and risk and uncertainty analysis, questions about the approach and data used to create the estimate may not be answered and the scope of the estimate may not be defined, among other concerns. By implementing a cost estimate that does not meet all four characteristics and associated best practices, DOD is making budget decisions based on potentially inaccurate data. As such, the program risks being unable to effectively estimate future funding needs and using unreliable data to make budgetary decisions. MHS GENESIS's Although three of the subprojects in DOD's MHS GENESIS February Subproject Schedules 2021 schedule collectively substantially met the comprehensive and controlled characteristics, the subprojects collectively partially met the Substantially Met Two of well-constructed and credible characteristics of a reliable schedule. As a Four Characteristics for result, the MHS GENESIS schedule was unreliable. Table 4 summarizes Reliability our assessment of the characteristics and the best practices of the three subprojects in DOD's MHS GENESIS February 2021 schedule. Page 12 GAO-22-104521 Electronic Health Records Table 4: Assessment of the Extent to Which Selected Subproject Schedules for Military Health System GENESIS Collectively Met Best Practices Schedule characteristic Assessment Summary of assessment Comprehensive ◕ • The integrated master schedule (IMS) and subproject schedules • Captures all activities, as identified in contained activities for both the government and its contractors the work breakdown structure (WBS), necessary to accomplish a program's objectives through 2025, which defines in detail the work for although it was unclear whether the IMS captured the entire both the government and its duration of effort needed. The subproject schedules' activities contractors necessary to accomplish included a contractor WBS element and a contractor WBS a program's objectives. dictionary for tracking key deliverables. In addition, activity and milestones names were descriptive, however, there were minor • Establishes the duration of all instances of non-unique names. activities in the same time unit (preferably days) and has specific • The subproject schedules we evaluated included the resources start and end dates. needed to do the work, by name and labor category, and depicted when those resources were needed. However, we • Reflects what resources (e.g., labor, found instances of over allocated resources, which led us to materials, and overhead) are needed question whether all assignments were realistic. Officials did not to do the work, whether all required provide evidence of efforts made to resolve resource over resources will be available when allocation. Over allocated resources result in inefficiency (for needed, and whether any funding or example, staff are less productive because of extended time constraints exist. overtime) or program delay from unavailable resources. • The Military Health System GENESIS program office had a process in place to ensure that realistic activity duration estimates allow for discrete progress measurement. Activity durations in the subprojects we assessed were short enough to be consistent with the needs of effective planning and program execution. Well-constructed ◑ • There were no instances of dangling or start-to-finish logic, and • Sequences all activities-that is, all no summary links in the project schedules that we examined, activities are sequenced in the order and the subprojects include a relatively small number of missing that they are to be implemented with logic and unjustified and inactive date constraints. The the most straightforward logic subprojects contained a relatively small number of date possible. constraints, although the subproject 2 schedule had 43 start no earlier constraints-that is, constraints that prevent activities • Establishes a valid critical path, which from starting earlier than a specified date. We also identified 16 represents the chain of dependent activities in the subproject 2 schedule with lags, some as long as activities with the longest total 40 days.a duration. A valid critical path is necessary to examine the effects of any activity slippage along this path. • Identifies the total float time-the amount of time by which an activity can slip before the delay affects the program's estimated finish date-so that a schedule's flexibility can be determined. Page 13 GAO-22-104521 Electronic Health Records Schedule characteristic Assessment Summary of assessment • The subproject schedules varied in the quality of their critical paths. The subproject 3 schedule path was straightforward, and the critical activities demonstrated that they were driving the key milestones within the schedule. However, for subproject 1, while the path was continuous and free of constraints, we were unable to reconcile the activities marked as critical in the subproject's schedule with those on the driving path to the key completion milestone. That is, activities in the subproject 1 schedule marked as critical may not have been those that were truly driving the key milestone. In addition, we were unable to validate the subproject 2 critical path because its key completion milestone was being driven by 63 predecessors. In this case, it is unclear how management identified the sequence of activities that determine its finish date. Its critical path, as marked by the software, may have been hampered by date constraints. In addition, we were unable to fully trace the subproject 2 critical activities to those shown in the performance report's critical path. Finally, we were unable to fully trace the critical paths in the IMS to the custom critical path reported to stakeholders. • While subproject 1 substantially met the criteria for reasonable total float, subprojects 2 and 3 schedules exhibited positive total float values that probably did not represent the actual degree of flexibility in the schedule. For example, 38.4 percent of subproject 2 schedule activities have total float greater than 2 working months, and subproject 3 schedule has a maximum total float value of 1,122 days-that is, it appears that key activities and milestones could slip months or years before delaying key program activities. Credible ◑ • The IMS contained instances of valid vertical traceability. • Verifies that the schedule is (1) However, the IMS is not fully horizontally traceable because horizontally traceable, meaning that it delays to some activities did not appear to impact all finish dates reflects the order of events necessary of key milestones. That is, delays to some activities push the to achieve aggregated products or end date of some finish milestones, but not others. outcomes; and (2) vertically traceable, • Program officials provided the summary results of SRAs that meaning that activities in varying they conducted. However, there was no evidence of supporting levels of the schedule align with one detailed SRA assumptions or methodology documentation another and key dates presented to included. management in periodic briefings are consistent with the schedule. • Conducts a schedule risk analysis (SRA) to predict a level of confidence in meeting the program's completion date and the level of necessary schedule contingency. Page 14 GAO-22-104521 Electronic Health Records Schedule characteristic Assessment Summary of assessment Controlled ◕ • Program officials reported that their schedule narrative • Updates schedule regularly using information was input into a performance report by officials actual progress and logic to trained and experienced in critical path method scheduling and realistically forecast dates for program reported monthly to the Department of Defense. In addition, activities. there were only a small number of date anomalies in the IMS and that at least one critical activity was in progress in the • Maintains a baseline schedule to subprojects that we reviewed. However, there was no evidence measure, monitor, and report the of a discussion on changes in network logic, including lags, date program's progress. constraints, and relationship logic and their effect on the schedule time. • Program officials reported that both the Department of Defense and the contractor utilized baseline schedules as the basis for measuring performance. The subproject schedules had a baseline assigned. In addition, the February 2021 performance report listed baseline schedule variances for key milestones. However, we did not find a schedule basis document that detailed ground rules and assumptions used in developing the schedule, and justified constraints, lags, long activity durations, and any other unique features of the schedule. Legend: ● = Met: The program office provided complete evidence that satisfies the entire criterion. ◕ = Substantially Met: The program office provided evidence that satisfies a large portion of the criterion. ◑ = Partially Met: The program office provided evidence that satisfies about half the criterion. ◔ = Minimally Met: The program office provided evidence that satisfies a small portion of the criterion. ○ = Not Met: The program office provided no evidence that satisfied any of the criterion. Source: GAO analysis of Department of Defense data and GAO-16-89G. | GAO-22-104521 a A lag in a schedule denotes the passage of time between two activities. Lags simply delay a successor activity-no effort or resources are associated with this passage of time. MHS GENESIS program officials provided information in response to our assessments of the well-constructed and credible characteristics as partially met. With respect to the critical path, officials stated that while some of the tasks in subprojects 1 and 2 have zero float and thus are on the critical path, others are "near critical path" tasks that have 7 or fewer days of float and therefore are not critical path tasks in the strict sense. However, they stated that the tasks are deemed critical due to the fact that minor delays could negatively impact the critical path by pushing the completion date out. Unless the schedule can produce a true critical path, the program office will not be able to provide reliable timeline estimates or identify when problems or changes may occur and their effects on downstream work. Program officials did not provide an explanation for our findings related to total float. With respect to the credible characteristic, MHS GENESIS program officials did not agree with GAO's applicability of horizontal traceability for their program. Specifically, officials responded that horizontal traceability Page 15 GAO-22-104521 Electronic Health Records is coordinated among the team. However, we determined that the schedule did not demonstrate that delays to some activities affect key milestone dates. With respect to SRA, program officials disagreed with the extent to which the analysis should be documented. Specifically, the officials stated that the analyses prepared for program reviews included discussions that explored priorities, identified risks to project completion, and provided documentation of schedule risks. However, the documentation did not include key elements such as how schedule contingency was developed. Because the subproject schedules we assessed did not meet all four characteristics of a high quality, reliable estimate, DOD's MHS GENESIS schedule is also unreliable. As a result, the department faces an increased risk of experiencing uncertain completion dates, time extension requests, delays, and increased project costs. In addition, employing an unreliable schedule decreases management's ability to make informed decisions related to possible sequences of activities and the flexibility of the schedule according to available resources, among other things. The MHS GENESIS cost estimate and schedules for selected subprojects Conclusions substantially met half of the characteristics and best practices associated with reliable cost and schedule estimates. However, the MHS GENESIS cost and schedule estimates were not reliable because they did not fully or substantially meet all characteristics associated with reliable estimates. Without reliable cost and schedule estimates, DOD increases the risk that management will not have the information necessary for effective decision-making. Further, following cost and schedule estimating best practices helps minimize the risk of cost overruns and schedule delays and would better position MHS GENESIS for success which is important since the majority of deployments have yet to occur. We are making the following two recommendations to DOD: Recommendations for Executive Action The Secretary of Defense should direct the Program Executive Officer of Defense Health Management Systems to ensure that the program office develops a reliable cost estimate using best practices described in GAO's Cost Estimating and Assessment Guide, in particular, by addressing those cost practices that were partially or minimally met. (Recommendation 1) The Secretary of Defense should direct the Program Executive Officer of Defense Health Management Systems to ensure that the program office develops a reliable schedule using best practices described in GAO's Page 16 GAO-22-104521 Electronic Health Records Schedule Assessment Guide, in particular, by addressing those schedule practices that were partially met. (Recommendation 2) We provided a draft of this report to DOD for review and comment. In its Agency Comments comments, reproduced in appendix II, the department concurred with our recommendations. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the Secretary of Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions on the matters discussed in this report, please contact me at (202) 512-4456 or at harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. Carol C. Harris Director, Information Technology Management Issues Page 17 GAO-22-104521 Electronic Health Records Appendix I: Objectives, Scope and Appendix I: Objectives, Scope and Methodology Methodology The objective of this report was to determine to what extent Department of Defense's (DOD) Military Health System (MHS) GENESIS's cost estimate and program schedule are consistent with cost estimating and schedule assessment best practices. To address the objective, we evaluated documentation supporting the program's October 2020 cost estimate against the characteristics and associated best practices for developing an estimate that is comprehensive, accurate, well-documented, and credible, as identified in GAO's Cost Estimating and Assessment Guide. 1 To determine the reliability of the cost estimate data, we reviewed how the cost for each work breakdown structure (WBS) element was calculated with an emphasis on the basis for the estimates and the strength and quality of the supporting documentation. We verified that the parameters used to create the estimate were valid and applicable by comparing them to available cost estimating references. We verified that calculations were correct for each WBS element and that escalation was properly applied and that WBS elements were summed accurately to arrive at the overall program cost estimate. We determined that the data used were sufficiently reliable for the purpose of evaluating DOD's adoption of cost estimating best practices. Additionally, we reviewed the MHS GENESIS integrated master schedule (IMS), dated February 2021, and related supporting documentation against the characteristics and associated best practices for developing a schedule that is comprehensive, well-constructed, credible, and controlled, as identified in GAO's Schedule Assessment Guide. 2 As the IMS is detailed and highly complex due to the large number of integrated subprojects and activities, we conducted assessments of three subprojects contained within the IMS for analysis. Because an IMS consolidates subproject schedules, errors and reliability issues in subproject schedules will be reflected in higher-level schedules. The subprojects we assessed were selected based on whether they were on- 1GAO, Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Program Costs, GAO-20-195G (Washington, D.C.: March 2020). 2GAO, Schedule Assessment Guide: Best Practices for Project Schedules, GAO-16-89G (Washington, D.C.: December 2015). Page 18 GAO-22-104521 Electronic Health Records Appendix I: Objectives, Scope and Methodology going and whether they included a significant and varying number of discrete activities in the project schedule. The three subprojects represent approximately 20 percent of the remaining activities in the IMS. The subprojects are Apple Product Improvement Engineering (subproject 1), Stage 6 (subproject 2), and Elderberry Product Improvement Engineering (subproject 3). Assessing three subprojects limits possible statements about the program's entire schedule. For example, if the program is not following best practices in creating and maintaining the three subproject schedules, we can conclude that the larger integrated schedule is unreliable. However, if the selected subprojects are deemed reliable, we cannot definitively determine the reliability of the integrated master schedule because the other subprojects that were not assessed may be unreliable. To determine the reliability of the schedule estimate data, we employed GAO defined schedule filters to identify missing logic or constraints. We copied the resulting schedule data from each of the filters used into a spreadsheet to check for specific problems that could hinder the schedule's ability to dynamically respond to changes. Specifically, we searched for activities with missing dependencies and constraints, and identified any lags or leads. We determined if activities were resource loaded and examined whether resources were over allocated or not available when needed. We examined the length of activity durations and compared them to the program management review cycle and checked for horizontal and vertical integration within the schedule. Further, we examined the schedule's critical path, schedule float and whether the schedule was baselined. We determined that the data used were sufficiently reliable for the purpose of evaluating DOD's adoption of schedule estimating best practices. For both the cost estimate and subproject schedules, we assessed each characteristic relative to the following scale: • Met. DOD provided complete evidence that satisfies the entire criterion. • Substantially met. DOD provided evidence that satisfies a large portion of the criterion. • Partially met. DOD provided evidence that satisfies about one-half of the criterion. Page 19 GAO-22-104521 Electronic Health Records Appendix I: Objectives, Scope and Methodology • Minimally met. DOD provided evidence that satisfies a small portion of the criterion. • Not met. DOD provided no evidence that satisfies any of the criterion. In addition, we interviewed DOD officials in the MHS GENESIS program office to understand their practices for developing and maintaining the cost estimate and program schedule. Further, we provided DOD with draft versions of our detailed analyses of the MHS GENESIS cost estimate and schedule and asked the officials to verify the information on which we based our findings. When warranted, we updated our analyses based on the department's response and additional documentation provided to us. We conducted this performance audit from September 2020 to June 2022 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Page 20 GAO-22-104521 Electronic Health Records Appendix II: Comments from the Department Appendix II: Comments from the Department of Defense of Defense Page 21 GAO-22-104521 Electronic Health Records Appendix II: Comments from the Department of Defense Page 22 GAO-22-104521 Electronic Health Records Appendix III: GAO Contact and Staff Appendix III: GAO Contact and Staff Acknowledgments Acknowledgments Carol Harris, (202) 512-4456 or harriscc@gao.gov GAO Contact In addition to the contact named above, Mark Bird (Assistant Director), Staff Season Burris (Analyst in Charge), Juaná Collymore, Donna Epler, and Acknowledgments Umesh Thakkar made key contributions to this report. (104521) Page 23 GAO-22-104521 Electronic Health Records The Government Accountability Office, the audit, evaluation, and investigative GAO's Mission arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is Obtaining Copies of through our website. Each weekday afternoon, GAO posts on its website newly GAO Reports and released reports, testimony, and correspondence. You can also subscribe to GAO's email updates to receive notification of newly posted products. Testimony Order by Phone The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. Contact FraudNet: To Report Fraud, Website: https://www.gao.gov/about/what-gao-does/fraudnet Waste, and Abuse in Automated answering system: (800) 424-5454 or (202) 512-7700 Federal Programs A. Nicole Clowers, Managing Director, ClowersA@gao.gov, (202) 512-4400, U.S. Congressional Government Accountability Office, 441 G Street NW, Room 7125, Washington, Relations DC 20548 Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Stephen J. Sanford, Managing Director, spel@gao.gov, (202) 512-4707 Strategic Planning and U.S. Government Accountability Office, 441 G Street NW, Room 7814, External Liaison Washington, DC 20548 Please Print on Recycled Paper.