U.S. Department of Health and Human Services Office of Inspector General Selected NIH Institutes Met Requirements for Documenting Peer Review But Could Do More To Track and Explain Funding Decisions Suzanne Murrin Deputy Inspector General for Evaluation and Inspections June 2021, OEI-01-19-00140 U.S. Department of Health and Human Services Office of Inspector General Report in Brief June 2021, OEI-01-19-00140 Why OIG Did This Review Selected NIH Institutes Met Requirements Congress, the National Institutes of Health (NIH), and others have raised for Documenting Peer Review But Could Do concerns about the integrity of U.S. More To Track and Explain Funding medical research. In August 2018, the NIH Director stated that the risks Decisions to the integrity of peer review were Each year NIH uses a peer review increasing. Subsequently, Congress Key Takeaway process to evaluate thousands of provided the Office of Inspector applications for grants to support its General with $5 million for oversight NIH met its requirements for research mission. This process is of NIH. Peer review is how NIH uses documenting first-level peer intended to be fair, equitable, and scientific experts to evaluate grant review for extramural grant timely. ICs must document the peer applications for funding. This study applications in the six ICs we review process and associated assesses the extent to which select reviewed. It has limited decisions. In funding grants, NIH NIH institutes and centers (ICs) met insight, however, into the allows for-and benefits from-ICs' NIH's requirements for documenting extent and nature of the ICs' discretion to fund less-favorably first-level peer review when decisions to fund grants for ranked grants (a practice known as evaluating applications for grants for less favorably ranked funding out of rank order) on a case- extramural research, and the extent applications, which can raise by-case basis. The reasoning for to which ICs made funding decisions questions about those funding decisions must also be that were not strictly limited to the transparency, impartiality, documented. scores from the initial scientific and fairness. review (i.e., the extent of funding grant applications out of rank order). What OIG Found NIH's peer review process is central to its upholding its values of For the six ICs we reviewed, NIH met its basic requirements for transparency, impartiality, and documenting first-level peer review. It also followed up with applicants, fairness, among others. Therfore, it as required, to resolve concerns about protections for human subjects is important for NIH to ensure that and animals. However, ICs' documentation to justify funding grant the process works as intended. applications out of rank order often appeared to fall short of the How OIG Did This Review requirements in HHS's Grants Policy Administration Manual, and the We reviewed a representative documentation of reasoning for those funding decisions was missing in sample of extramural grants funded 37 out of 109 grants in our sample. As a result, NIH lacks insight into the by six ICs in fiscal year (FY) 2018. reasoning for the ICs' decisions that were not strictly limited to the scores For those grants, we assessed NIH's from the initial scientific review. A failure to document these compliance with selected aspects of justifications reduces transparency and can raise questions about undue its peer review process. We did so influence. by reviewing summary statements, documentation of NIH's followup to What OIG Recommends resolve peer reviewer concerns, and Without better insight into where and why funding out of rank order is justifications for funding grants out happening, the integrity of NIH's peer review process could come under of rank order. We also reviewed the question. We recommend that NIH centrally track and monitor data on Department of Health and Human funding out of rank order and update its policy and guidance to reflect Services' (HHS's) grant policy, NIH the latest HHS grants policy on justifying funding out of rank order. NIH policies, and NIH's written responses concurred with our recommendations. to our questions. TABLE OF CONTENTS BACKGROUND 1 Methodology 7 FINDINGS 10 NIH met requirements that ensure transparency of first-level peer review of grants funded by 10 select ICs in FY 2018 NIH followed up, as required, on all but one application to resolve-before funding grants- 12 concerns raised by first-level peer review about protections for human subjects and animals ICs varied widely in the extent to which they funded grant applications out of rank order for 12 the grants in our sample ICs did not appear to justify funding decisions in accord with HHS policy for a third of the 13 grants in our sample that were funded out of rank order NIH has limited insight into the extent and nature of funding out of rank order 16 RECOMMENDATIONS 17 NIH should centrally capture and monitor data on ICs' funding of grant applications out of 17 rank order NIH should update its policy and guidance to reflect the latest HHS grants policy on justifying 18 funding out of rank order AGENCY COMMENTS AND OIG RESPONSE 19 APPENDICES 20 A. Statistical Supplement 20 B. Agency Comments 22 ACKNOWLEDGMENTS AND CONTACT 25 ABOUT THE OFFICE OF INSPECTOR GENERAL 26 ENDNOTES 27 BACKGROUND Objectives 1. To determine the extent to which the National Institutes of Health's (NIH) Center for Scientific Review (CSR) and select Institutes and Centers (ICs) met NIH's requirements for documenting first-level peer review of applications for grants for extramural research funded in fiscal year (FY) 2018. 2. To determine the extent to which select ICs used their discretion by funding grant applications out of rank order for extramural research in FY 2018. For FY 2019, the U.S. Department of Health and Human Services (HHS) Office of Inspector General (OIG) received $5 million in congressional appropriations to conduct oversight of NIH grant programs and operations. 1 As indicated by the conference report that accompanied the legislation, "the conferees direct[ed] the OIG to examine NIH's oversight of its grantees' compliance with NIH policies." 2 This review is part of a larger body of HHS OIG work focused on oversight of NIH grant programs and operations. Our work will review (1) intellectual property and cybersecurity protections; (2) compliance with Federal requirements and NIH policies for grants and contracts; and (3) integrity of grant application and selection processes. As part of this oversight work, Congress directed that OIG examine (1) NIH's efforts to ensure the integrity of its grant evaluation and selection processes and (2) the effectiveness of NIH's and grantee institutions' efforts to protect intellectual property derived from NIH-supported research. In his August 2018 statement on protecting the integrity of U.S. biomedical research, Dr. Francis Collins, the Director of NIH, stated that the risks to the security of intellectual property and the integrity of peer review were increasing. Dr. Collins further stated that in response to the increasing risks, NIH would work to identify robust methods to protect the integrity of peer review, among other steps it would take. 3 This review examines NIH's peer review process, which is central to the integrity of evaluating and selecting grants. NIH intends for the peer review process to be fair, equitable, timely, and conducted in a manner that strives to eliminate bias. 4 Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Background | 1 NIH's Organization and Mission NIH consists of 27 ICs, most of which are focused on specialty areas (e.g., the National Cancer Institute and the National Heart, Lung, and Blood Institute). 5 NIH's mission is "to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability." 6 NIH's ICs accomplish this mission, in large part, by annually funding over $30 billion in extramural research through a grantmaking process. 7 CSR does not fund its own grants. Rather, it provides support to the other ICs through peer review and processing of incoming grant applications. NIH Peer Review Process In NIH's peer review process, experts evaluate grant applications for scientific and technical merit, among other requirements. NIH's core values for peer review drive it to seek the highest level of scientific and ethical standards. Those core values include expert assessment, transparency, impartiality, fairness, confidentiality, integrity, and efficiency. 8 Grant applications undergo two levels of review: an initial scientific review and a review by an advisory council. All NIH research applications undergo this two-level peer review process to ensure that applications are reviewed to the greatest extent possible in a fair, unbiased fashion. 9 An IC may choose to manage the first level of the peer review process itself or use CSR to manage it. 10 NIH annually receives about 80,000 grant applications for research funding across all ICs. 11 CSR conducts the first level of peer review on about 75 percent of these applications; the ICs conduct the first level of peer review on the remaining 25 percent. 12 NIH requires both CSR and ICs to follow NIH's peer review policies and procedures for evaluating applications for research grants. 13 Peer reviewers are mostly non-Federal scientists who have the necessary expertise to evaluate the scientific merit of the grant applications. NIH verifies peer reviewers' scientific expertise using multiple sources, such as their publication and grant histories. Reviewers may meet a few times over the course of several years or just one time on an ad-hoc panel to review grant applications on a highly specialized research topic. 14 First Level of Peer Review During the first level of peer review, or initial scientific review, peer reviewers at CSR or the funding IC evaluate a package of grant applications that NIH has received in response to a funding opportunity announcement. To conduct this stage of the review, peer reviewers engaged by CSR or the funding IC evaluate each application on the basis of the criteria included in the funding opportunity announcement. The funding opportunity announcement describes NIH's intent to fund discretionary grants or cooperative agreements that address a specified public health topic or Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Background | 2 scientific area. The funding opportunity announcement specifies the scored criteria and additional criteria that peer reviewers will use to assess grant applications. 15 A Scientific Review Officer, who is an NIH scientist, oversees the initial scientific review. 16 Scored Criteria. Each peer reviewer assigns a score to each scored criterion. Unless otherwise specified in the funding opportunity announcement, NIH's five scored criteria for research projects are significance, investigators, innovation, approach, and environment. Reviewers then consider their criteria scores to assign an overall impact score to the grant application that "reflects their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved." 17 (See Exhibit 1 below for a description of NIH's five scored criteria.) Exhibit 1: Peer reviewers generally assess five scored criteria to help determine the scientific merit of the research proposed in a grant application. Source: NIH notice number NOT-OD-09-025 Additional Review Criteria and Considerations. Reviewers assess, but do not score, other factors as well, depending on the requirements in the funding opportunity announcement and the type of research being proposed in an application. 18 Additional review criteria factor into the overall score but do not receive individual criterion scores, and additional review considerations do not receive criterion scores and do not factor into the overall score. (See Exhibit 2 for the additional criteria and considerations.) Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Background | 3 Exhibit 2. Peer reviewers assess two types of additional criteria, called Additional Review Criteria and Additional Review Considerations, as appropriate for the type of research proposed in a grant application. Additional Review Criteria • Protection for Human Subjects • Biohazards • Inclusion of Women, Minorities, • Vertebrate Animals and Children (Including Animal Welfare) • Resubmission Applications • Revision Applications • Renewal Applications • Study Timeline (for Clinical Trials) Additional Review Considerations • Budget and Period Support • Research Using Select Agents • Applications From Foreign • Resource-Sharing Plans Organizations • Authentication of Key Biological and/or Chemical Resources Sources: NIH, NIH Grants Policy Statement, Section 2.4, October 2017 revision. Bars to Funding. Of the additional criteria, those for the protection of human subjects; inclusion of women, minorities, and children; and vertebrate animals can prevent a grant application from being funded if the application does not satisfactorily address them. 19 NIH sometimes refers to these as being a "bar to funding" a grant. 20 When peer reviewers identify concerns with these criteria, NIH follows up with grant applicants. Grant applicants must then provide additional information or adjust their research plans before NIH will agree to fund their grants. 21 NIH may follow up on the other additional criteria when reviewers flag information addressing them as unacceptable. NIH policy instructs reviewers to consider Additional Review Criteria when assigning an overall impact score to grant applications. 22 For Additional Review Considerations, reviewers' comments serve as administrative information that ICs consider when they make funding decisions. Summary Statements. After first-level peer review is complete, the Scientific Review Officer produces a summary statement for each grant application in the package of applications from that review. The summary statement contains the overall impact score for the grant application. This score is derived from an average of the peer reviewers' impact scores (and the application's percentile ranking among the grant applications in the package, if applicable) and individual critiques from the peer reviewers that address the scored criteria and additional criteria, as appropriate. The summary statement also includes a resume, or summary, of the peer reviewers' discussion (when applicable), a summary of reviewers' comments by the Scientific Review Officer, and a roster of the reviewers who reviewed the application. 23, 24 After the review, NIH makes the summary statement available to the grant applicant. 25 Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Background | 4 IC Review and Funding Order On receiving a package of grant applications that have been through initial scientific review, officials at the funding IC review the package and make recommendations to the Director of the IC. Officials rank the applications by impact score or percentile and compare them to their IC's programmatic needs and funding plan. Generally, grant applications with rankings that fall below the funding IC's cutoff will not be funded but NIH's peer review process allows for discretion. A designated official at the IC may recommend to the IC's director that an application that is less favorably ranked for funding consideration move up, or that a more favorably ranked application move down. If the IC ultimately approves a less favorably ranked application for funding, this is referred to as "funding out of rank order." When an IC decides to fund a grant application out of rank order, HHS and NIH policy requires the IC to write a justification for doing so. HHS policy requires the justification to tie to factors documented in the funding opportunity announcement. 26, 27 (See Exhibit 3 for an illustration of funding out of rank order.) After IC review, the entire package of grant applications moves on the second level of peer review. Exhibit 3: ICs have the discretion to fund grant applications that are less favorably ranked than competing applications. Source: OIG analysis of NIH policy. Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Background | 5 Second Level of Peer Review The second level of peer review, or advisory council review, takes place at the funding IC. Each IC has an advisory council that conducts the review. 28 This council includes scientists from the extramural research community as well as public representatives. The advisory council examines applications and considers the overall impact scores that the applications received during the initial peer review process, including any recommendations to fund out of rank order, percentile rankings (if applicable), and the summary statements in light of the IC's priorities. The advisory council advises the IC Director on funding decisions. The IC Director makes the final funding decision on the basis of staff and council advice, and of the outcome of initial peer review. 29 Previous and Concurrent OIG Work In response to a congressional request, OIG examined the extent to which NIH's National Institute of Environmental Health Sciences (NIEHS) followed its peer review processes for funding research on bisphenol-A (BPA). In the resulting 2017 report, OIG found that although the NIEHS met the peer review process requirements for all grants, NIEHS used its discretion to fund applications with less favorable scores than competing applications for 14 percent of BPA grants, versus for 4 percent of other, non-BPA grants. OIG concluded that although NIEHS had discretion to fund grant applications with less favorable impact scores ahead of competing applications, applying its discretion frequently or disproportionately in one research area may create an appearance of impropriety. 30 OIG made no recommendations for NIH in this report. In addition to this review, OIG has a body of work completed or underway in response to the Departments of Defense and Labor, Health and Human Services, and Education Appropriations Act, 2019 and Continuing Appropriations Act 2019. Completed evaluations include reviews of NIH's vetting of prospective peer reviewers; NIH's monitoring of peer reviewers' handling of confidential information; and NIH's monitoring of extramural researchers' financial conflicts of interests. 31, 32, 33 Completed audits include reviews of NIH's controls to prevent duplication among NIH research grants and NIH's implementation of regulations regarding financial conflicts of interest, among others. 34, 35 This body of work resulted in recommendations calling for NIH to update its policies and training and to strengthen its approach to vetting and overseeing peer reviewers to identify foreign threats. It also called for NIH to do more to address financial conflicts of interest, including conducting periodic quality assurance to ensure the adequacy of its oversight. NIH concurred with all of these recommendations, and it has implemented three of them. NIH has been taking steps to implement the remaining recommendations, including working closely with grantee institutions, Federal law enforcement, and the White House's Office of Science and Technology Policy to help identify and mitigate foreign threats to research integrity. According to NIH, these steps have led to the removal of scientists as NIH grantees and peer reviewers. Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Background | 6 Methodology Scope We based this inspection on a review of documentation for a representative sample of grants funded by six ICs in FY 2018. For those grants, we assessed NIH's compliance with selected aspects of its peer review process. We did so by reviewing summary statements, documentation of NIH's followup to resolve bars to funding, and justifications for funding grants out of rank order. We also based this inspection on a review of NIH policies and HHS grants policy in effect in FY 2018 as well as NIH's written responses to questions we sent during the inspection. Our review of bars to funding did not include NIH's criterion on inclusion across the lifespan, as that took effect in January 2019. Sample Selection We used NIH's ExPORTER grants database to identify all new awards that NIH made in FY 2018 for extramural grants supporting basic and cooperative research. 36 This resulted in a list of 9,714 grants that were funded by 27 ICs. We divided the ICs into groups of small, medium, and large ICs based on the number of grants each funded. 37 To lessen the burden on NIH given numerous ongoing OIG reviews of ICs at the time, we purposively selected two ICs from each of these three groups. (See Exhibit 4 below for the ICs we selected.) Exhibit 4: The six ICs we selected for this review Large ICs National Institute of Neurological Disorders and Stroke (NINDS) National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) Medium ICs National Institute on Drug Abuse (NIDA) National Human Genomic Research Institute (NHGRI) Small ICs National Institute on Mental Health and Development (NIMHD) National Center for Advancing Translational Sciences (NCATS) We grouped grants into seven strata based on the entity that conducted the first level of peer review: either CSR or one of the six selected ICs. We selected a stratified random sample from the seven strata. The first stratum consisted of all grants for which CSR performed the first level of peer review. Strata 2 through 7 consisted of grants that were reviewed by each of the six selected ICs. This sampling approach allowed us to generalize and compare our findings regarding the first level of peer review and the overall extent to which ICs funded grants out of rank order. However, our sampling approach does not allow us to make generalizations about the subset of grants that had bars to funding or were funded out of rank order. For these subsets Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Background | 7 of findings, we have presented sample counts. (See Exhibit 5 below for details on the population and sample size for each of our strata.) Exhibit 5: Breakdown of population and sample size for seven strata Entity that Conducted First Grant Sample Stratum Level of Review Population Size 1 CSR 1184 150 2 NCATS 46 36 3 NIMHD 54 42 4 NHGRI 56 42 5 NIDA 176 82 6 NIDDK 161 82 7 NINDS 351 106 Total 2028 540 Document Review Summary Statements. For each of the grants in our sample, we requested the related summary statement that documents the first level of peer review. 38 We abstracted data from the summary statements into a database. These data include elements about the peer reviewers' assessment of scored criteria and additional criteria; a summary of reviewers' comments by the Scientific Review Officer who oversaw the peer review; and whether the summary statement included a roster of peer reviewers. Followup Documentation on Bars to Funding. We reviewed summary statements for instances documenting that peer reviewers raised concerns on criteria that were a bar to funding a grant. Those criteria were protection for human subjects; inclusion of women, minorities, and children; and vertebrate animals (including animal welfare). When we identified those instances of concern, we asked NIH for documentation showing that it resolved the concerns. For these grants, NIH sent us copies of email correspondence, worksheets, and other documents that we reviewed for evidence that NIH followed up with each grant applicant and approved the additional information or changes that the applicant made in response. Justifications. For any of our sampled grants that ICs funded out of rank order, we also requested the written justifications for doing so. We examined the text of these justifications for the reasoning for funding grants out of rank order. We grouped the justifications into three categories: (1) those that included a reason and an argument for funding out of rank order that spoke to the merits of the proposed research or applicant; (2) those that provided a reason, such as "IC priority area," but did not include an argument related to the specifics of the proposed research or applicant; and (3) those that provided no reason for funding the grant out of rank order. Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Background | 8 NIH Policy and Written Responses. We reviewed NIH policy on its peer review and funding processes that were in effect in FY 2018, as well as and written clarifications that NIH provided on its policy or specific grants in our sample. We also reviewed the relevant HHS policy for documenting decisions to fund grants out of rank order. Limitations We did not independently verify the information on summary statements. In our review of additional criteria, we did not determine whether peer reviewers assessed the appropriate criteria for each grant application. In our review of justifications for funding grants out of rank order, we did not assess the validity of the scientific explanations that ICs cited; rather, we checked for language that offered reasoning for the funding decision. Standards We conducted this study in accordance with the Quality Standards for Inspection and Evaluation issued by the Council of the Inspectors General on Integrity and Efficiency. Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Background | 9 FINDINGS NIH met requirements that ensure transparency of first-level peer review of grants funded by select ICs in FY 2018 We found that NIH met requirements for documenting the first level of peer review, also known as initial scientific review. These basic requirements are key to ensuring that the review reflects NIH's core values. For each grant application, NIH requires the Scientific Review Officer who oversaw the peer review to create a summary statement that documents the review. The summary statement must include complete sets of critiques from peer reviewers that address scored criteria and additional review criteria. It can also include a roster of the peer reviewers who evaluated the grant application. Overall, we found little variation in compliance with the requirements among the six ICs and CSR. NIH created the required summary statements for 100 percent of grants Summary statements are important because they serve as NIH's official record of the first level of peer review across all of the applications, ensuring an equivalent process. An equivalent process upholds NIH's core value of fairness. 39 All of the grant applications in our review had a summary statement. The critiques, scores, rosters, and other information in summary statements document the review process that each grant application underwent. Because NIH shares summary statements with grant applicants, summary statements support NIH's core value of transparency. Summary statements for 99 percent of grants had complete sets of scores from peer reviewers, as required Scored criteria reflect the peer reviewers' expert assessments of an application's scientific and technical merit. Expert assessments are another NIH value it strives to uphold. Those scored criteria are a consideration in an application's overall impact score, which determines its ranking and likelihood of being funded by NIH. NIH's five scored criteria for research projects are: significance, investigators, innovation, approach, and environment. Summary statements for 99 percent of grant applications included critiques from each peer reviewer with complete sets of scores for the five criteria. 40 In those few cases where the summary statement did not have Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Findings | 10 complete sets of scores, it was because a reviewer's critique failed to address a single scored criterion or a reviewer's critique omitted scores for all five criteria. Summary statements for 100 percent of grants had a roster of peer reviewers Rosters provide transparency with respect to the identities and institutional affiliations of the scientists who reviewed an application. This enables applicants and others to verify that the membership of a review group was not overly weighted toward a single type of scientific discipline or research institution. Summary statements for 99 percent of grants addressed additional criteria and considerations-most often the project budget, but sometimes potentially sensitive issues, such as the use of biological agents or involvement of foreign entities NIH's 14 additional criteria and considerations provide opportunity for peer reviewers to address factors relevant to the specific research proposed within a grant application. This ability for peer reviewers to tailor their reviews is important given the breadth of research that NIH sponsors across its 27 ICs. Indeed, our review of summary statements showed that peer reviewers assessed additional criteria and considerations for nearly all grant applications. For 22 percent of grants, peer reviewers or the Scientific Review Officer found an additional criterion or consideration to be unacceptable or in need of revision; the two found most often were protections for human subjects and budget and period of support. 41 See Appendix A for data on the percentage of grant applications for which reviewers assessed each additional criterion and consideration. In addressing the additional criteria and considerations, reviewers occasionally commented on potentially sensitive topics. For example, they commented on the proposed use of certain biological agents or toxins (also known as select agents) in 17 percent of grants. Such agents require special handling and can pose a threat to the public's health; one example is the Ebola virus. 42 For 6 percent of grants, the reviewers commented on the involvement of foreign entities. For applications submitted by foreign entities, NIH instructs reviewers to assess whether the use of foreign talent and other resources not readily available in the United States presents special opportunity to further research. 43, 44 NIH has elevated its scrutiny of grantees and its own processes since identifying risks associated with foreign influences on the integrity of U.S. biomedical research-in fact, in 2018 NIH added security as one of its core values. 45 Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Findings | 11 NIH followed up, as required, on all but one application to resolve-before funding grants-concerns raised by first-level peer review about protections for human subjects and animals NIH prohibits funding applications that peer reviewers identify as not adequately addressing human subjects protections; inclusion of women, minorities, and children in research; and animal welfare. When applicants did not adequately address these criteria in research, NIH flagged their applications for followup. Peer reviewers identified concerns with these particular criteria in the applications for 51 of the grants in our sample. NIH followed up on all but one application for additional information to address concerns raised during the initial scientific review. In response, the applicants responded via email with information that NIH staff reviewed and approved before allowing the grants to be funded. The one application on which NIH did not follow up was used by NIH as the basis to fund four of the grants in our Example of Followup on Concerns About Use of Vertebrate Animals Peer reviewers found that a grant applicant did not address all required criteria for the use of vertebrate animals. The applicant submitted additional information to NIH detailing the experimental procedure to be used; why the study needed to use animals; how the study team would minimize pain and discomfort for the animals; and how the animals would be euthanized. sample. According to NIH, after researching the application at our request, it determined that the flag was unnecessary. In this case, the flag concerned excluding children without scientific justification, but NIH ultimately determined that children were appropriately excluded from only certain aspects of the study. ICs varied widely in the extent to which they funded grant applications out of rank order for the grants in our sample NIH's grantmaking process provides ICs with discretion to fund a grant application that was ranked less favorably by the first level of peer review over an application that was ranked more favorably. Funding out of rank order allows the ICs to fund grants that align with their research priorities or that address emerging threats to public health, for example. Overall, ICs funded 109 of the 540 grants in our sample out of rank order. The second largest IC in our sample, NIDDK, funded 67 of its 130 sampled grants out of rank order. By contrast, the largest IC in our sample, NINDS, funded 8 of its 184 sampled grants out of rank order. (See Exhibit 6 for counts of sampled grants funded in and out of rank order by each IC.) Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Findings | 12 Exhibit 6: ICs varied in the extent to which they funded sampled grants out of rank order. 200 180 160 140 Sampled Grants 120 100 80 60 40 20 0 NINDS NIDDK NIDA NIMHD NHGRI NCATS Grants Funded in Rank Order Grants Funded Out of Rank Order ICs did not appear to justify funding decisions in accord with HHS policy for a third of the grants in our sample that were funded out of rank order Documentation that ICs created to justify funding grants out of rank order often appeared to fall short of the requirements in HHS's Grants Policy Administration Manual for documenting funding out of rank order. Those requirements call for approving officials to provide a statement of the specific reasons that influenced their judgment, including a justification for funding of the application that ties to factors documented in the funding opportunity announcement. 46 However, in documentation for 37 of the 109 grants funded out of rank order, ICs offered little or no insight into the reasoning for their decisions that were not strictly limited to the scores from the initial scientific review. For 22 of these grants, ICs offered a brief statement with little reasoning. For 15 grants, ICs offered no reasoning. Examples of Justifications That Provide Little Reasoning "High Program Priority. Opioid dependence." "Preclinical application that conducts stability tests and vaccine potency tests." "This grant is part of the Diversity R21 program." Justifications That Provide No Reasoning "Paid out of rank order by skipping other fundable applications." "Applications are in response to an RFA [request for applications]; funding plan established in accordance with RFA." Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Findings | 13 In contrast, documentation for 72 of the 109 grants funded out of rank order appears more in line with HHS's grants policy, referencing the strengths and promise of the research proposed in the grant applications. ICs noted the track record of the research team, the scientific contribution the research would make, or other merits. 47 This provided insight into ICs' reasoning behind their decision to fund grants out of rank order. Excerpts From Justifications That Provide Stronger Reasoning "This R21 project is innovative, potentially high impact, investigating . . . techniques to detect and predict response to therapy of . . . disease. Developing . . . techniques to track disease severity in these patients could be an essential adjunct to clinical practice . . . ." "Peer reviewers rated the proposal as innovative, feasible and highly consistent with the goals of the . . . initiative and noted that the research team is exceptionally strong, productive and well-chosen for the proposed research. Furthermore, the project offers geographic and investigator diversity that will enhance the overall community of researchers funded under this FOA [funding opportunity announcement]." "This proposal directly addresses the diagnosis and prognosis of concussion in . . . populations, which is a planning priority outlined [by NIH] and an area of recent Congressional inquiry. This proposal is from an outstanding multidisciplinary team of investigators at . . . with strong complementary expertise in clinical . . . . Review deemed this project highly rigorous and innovative . . . ." We note that ICs varied in how robustly they justified their decisions to fund less- favorably ranked grants in our sample. Three of the six ICs in our review provided reasoning for all of their decisions, while one IC-NIMHD-provided no justifications that explained its decisions for funding out of rank order. NIDDK, which funded the most grants in our sample out of rank order, provided stronger reasoning in three-quarters of the justifications it wrote. (See Exhibit 7 below for data on the breakdown of ICs' justifications by strength of reasoning.) Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Findings | 14 Exhibit 7: ICs varied in how well they documented their reasons for funding grants out of rank order. 80 Sampled Grants Funded Out of Rank Order 60 40 20 0 NIDDK NIDA NIMHD NHGRI NCATS Reasoning for Funding Out of Rank Order Stronger Little None Finally, the procedure for justifying funding grants out of rank order in NIH's Policy Manual does not incorporate current HHS policy on justifying decisions to fund grants out of rank order that requires agencies to explain the reasoning behind their decisions. NIH's policy manual does not cite the latest HHS grants policy, which created stronger requirements for documenting decisions to fund out of rank order NIH's Policy Manual 4204-204c cites HHS' retired Grants Policy Directive, which stated, "Should the official approve an application for funding out of rank order, the reason(s) for doing so must be documented in writing." In 2015, HHS's Grants Policy Administration Manual (GPAM) superseded the Grants Policy Directive. When agencies fund out of rank order, the GPAM requires "a statement of the specific reasons for the difference that influenced the judgment of the approving official. This must include a justification for funding of the particular application and tie to factors documented in the FOA [funding opportunity announcement]." Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Findings | 15 NIH has limited insight into the extent and nature of funding out of rank order Monitoring how often ICs exercise this discretion and requiring justification for doing so can serve as compensating controls to help NIH ensure that ICs' funding decisions uphold NIH's values and are sound and free from undue influence. However, our review raises concerns about these tools. NIH lacks centralized, readily available data on how often ICs fund grants that are less favorably ranked NIH cannot easily determine the extent to which ICs fund grant applications out of rank order. Data on funding out of rank order reside only with the ICs that made the funding decisions. To respond to our data request for this review, NIH relied on the individual ICs to identify grants that they funded out of rank order and provide their justifications. According to NIH, this task was a time-consuming, manual process that occurred at each of the six ICs we reviewed. It is understandable that the funding ICs have the source data on their decisions to fund less-favorably ranked grants. However, NIH's lack of insight into the extent of the practice across ICs raises questions about its ability to monitor and oversee ICs' funding decisions. Justifications that fail to provide reasoning for funding decisions limit NIH's ability to understand the nature of funding out of rank order ICs fund grants out of rank order for a number of reasons including, among others, to support their research priorities, respond to emerging threats to public health, and support NIH-wide initiatives. However, funding out of rank order could also signal potential problems with NIH's peer review process, such as concerns with the initial scientific review or indications of undue influence. We found that in 37 of 109 instances of funding out of order, no reasoning was documented. That means that NIH lacks insight into whether the ICs are funding out of rank order to further NIH's mission or because of problems that warrant NIH's attention. Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Findings | 16 CONCLUSION AND RECOMMENDATIONS The integrity of NIH's peer review process is critical to achieving NIH's research mission. In recent years, the integrity of the process has come under increased scrutiny. Prior Office of Inspector General reviews highlighted the importance of vetting peer reviewer nominees and overseeing reviewers' handling of confidential information. This review adds to that body of work by focusing on the peer review process itself. Our review found that NIH followed processes that are key to ensuring that peer review is in accordance with NIH's core values. For grants funded by the six ICs we reviewed, we found that NIH created summary statements that documented the requirements of first-level peer review. We found that NIH followed up, as required, on serious concerns that-if left unresolved-would bar it from funding grants. Taken together, these practices help ensure that applications submitted for funding are evaluated by scientific experts in a manner free from inappropriate influences, which is the intent of peer review. However, our review also revealed a need for improved transparency. In funding grants, NIH allows for-and benefits from-the discretion to make funding decisions that are not strictly limited to the scores provided during the first-level peer review. This, for example, enables ICs to respond to emerging threats to public health. Such discretion calls for controls to ensure that ICs exercise it appropriately. Yet we found NIH lacks ready access to information about which grants ICs funded out of rank order, and that ICs did not always provide clear justifications for doing so, as required by HHS grants policy. More transparency around funding out of rank order represents an important control to help NIH identify and track outliers, and to ensure that its funding processes are in fact upholding its core values. Therefore, we recommend that NIH: Centrally capture and monitor data on ICs' funding of grant applications out of rank order NIH needs better and more readily available information from the ICs to detect, analyze, and understand the reasons for funding out of rank order. NIH already has a tool, the eRA Pay Plan Module, that allows ICs to flag grant applications that they fund out of rank order; however, NIH does not require ICs to use it. NIH should require ICs to use the Pay Plan Module or another tool to centrally capture data on the extent and nature of funding out of rank order. NIH should use these data to monitor funding out of rank order across ICs and, where it finds outliers, ask followup questions to ensure that its peer review process is working as intended. Although ICs Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Conclusion and Recommendations | 17 have the discretion to fund less favorably ranked grant applications, doing so frequently could be a sign of systemic problems in first-level peer review or undue influence in funding decisions. Update its policy and guidance to reflect the latest HHS grants policy on justifying funding out of rank order Requiring justifications that explain the reasoning for funding less favorably ranked grants reinforces ICs' accountability for their funding decisions. Robust justifications for funding less-favorably ranked grant applications provide transparency into ICs' funding decisions and build trust that NIH's grantmaking is fair, equitable, and free from bias. However, NIH's policy does not provide guidance on what information to include in the written justification. NIH's policy also has not been revised since the publication of HHS's latest grant policy and omits HHS requirements with respect to the content of ICs' justifications. In particular, HHS's grants policy requires a statement of the specific reasons that influenced the judgment of the approving official, including a justification for funding the application that ties to factors documented in the FOA. NIH should update its grants policy to reflect HHS's latest grants policy and thereby define minimum expectations in policy to ensure that ICs provide a clear and compelling rationale for funding out of rank order. As such, NIH should also update its guidance to instruct ICs to identify the strengths of the grant application that warrant funding an application out of the rank order. It should also provide examples that illustrate acceptable justifications. Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Conclusion and Recommendations | 18 AGENCY COMMENTS AND OIG RESPONSE NIH concurred with our recommendations. Regarding the first recommendation, NIH stated that it will develop structured data for identifying, tracking, and providing compliance oversight on grants awarded out of rank order, with implementation planned for fiscal year 2021. NIH also stated it will conduct a comprehensive review in the same year, and implement further internal controls, guidance, and compliance testing as needed based on the results of the review. Regarding the second recommendation, NIH stated that it will update its internal policy and staff guidance to reflect the requirement of the HHS Grants Policy Administration Manual to tie funding justifications to factors documented in the Funding Opportunity Announcement. NIH stated it will take this step in fiscal year 2021. We thank NIH for the actions it plans to take to address our recommendations and for its commitment to increasing its oversight of funding decisions made out of rank order. For the full text of NIH's comments, see Appendix B. MMENTS AND OIG RESPONSE Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Agency Comments and OIG Response | 19 APPENDIX A – Statistical Supplement Point Estimates and Confidence Intervals Grants including summary statements Sample Point 95% confidence size estimate interval NIH provided summary statement 540 100% 98.9-100% for grant Summary statements with complete sets of scores Sample Point 95% confidence size estimate interval Summary statements with complete 540 99% 99.4-99.9% sets of scores from peer reviewers Summary statements with roster of peer reviewers Sample Point 95% confidence size estimate interval Summary statements with rosters of 540 100% 98.9-100% peer reviewers Additional criterion with comment by at least one reviewer Sample Point 95% confidence size estimate interval Budget and Period of Support 540 99% 99.3-99.9% Resource Sharing Plans 540 92% 89.8-94.6% Authentication of Key Biological 540 74% 70.5-77.9% and/or Chemical Resources Vertebrate Animals 540 58% 53.9-62.7% Biohazards 540 47% 41.9-51.3% Protection of Human Subjects 540 44% 39.3-48.3% Inclusion of Women 540 40% 35.5-44.3% Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Appendix A | 20 Additional criterion with comment by at least one reviewer Sample Point 95% confidence size estimate interval Inclusion of Minorities 540 40% 35.5-44.3% Inclusion of Children 540 39% 34.6-43.3% Resubmission Applications 540 38% 33.3-42.5% Select Agent Research 540 17% 13.0-20.1% Applications From Foreign Organizations 540 6% 3.8-8.0% Study Timeline 540 2% 1.0-2.7% Renewals 540 <1% <0.1-0.3% Peer reviewers identified concerns that could result in bars to funding (i.e., protection of human subjects; inclusion of women, minorities, and children; and vertebrate animals) Sample Point 95% confidence size estimate interval Peer reviewers identified concerns that 540 10% 7.1-13.5% could result in bars to funding Source: OIG analysis of NIH grants data from six ICs, 2020. Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Appendix A | 21 APPENDIX B Agency Comments Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Appendix B | 22 Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Appendix B | 23 Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Appendix B | 24 ACKNOWLEDGMENTS AND CONTACT Acknowledgments Ivan Troy served as the team leader for this study, and Matt Blackburn served as the lead analyst. Others in the Office of Evaluation and Inspections who conducted the study include Chris Galvin. Office of Evaluation and Inspections headquarters staff who provided support include Joe Chiarenzelli, Althea Hosein, and Christine Moritz. We would also like to acknowledge the contributions of other Office of Inspector General staff, including Jessica Swanstrom. This report was prepared under the direction of Joyce Greenleaf, Regional Inspector General for Evaluation and Inspections in the Boston regional office. Contact To obtain additional information concerning this report, contact the Office of Public Affairs at Public.Affairs@oig.hhs.gov. OIG reports and other information can be found on the OIG website at oig.hhs.gov. Office of Inspector General U.S. Department of Health and Human Services 330 Independence Avenue, SW Washington, DC 20201 Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Acknowledgments and Contact | 25 ABOUT THE OFFICE OF INSPECTOR GENERAL The mission of the Office of Inspector General (OIG), as mandated by Public Law 95- 452, as amended, is to protect the integrity of the Department of Health and Human Services (HHS) programs, as well as the health and welfare of beneficiaries served by those programs. This statutory mission is carried out through a nationwide network of audits, investigations, and inspections conducted by the following operating components: The Office of Audit Services (OAS) provides auditing services for HHS, either by conducting audits with its own audit resources or by overseeing audit work done by others. Audits examine the performance of HHS programs and/or its grantees and contractors in carrying out their respective responsibilities and are intended to provide independent assessments of HHS programs and operations. These audits help reduce waste, abuse, and mismanagement and promote economy and efficiency throughout HHS. The Office of Evaluation and Inspections (OEI) conducts national evaluations to provide HHS, Congress, and the public with timely, useful, and reliable information on significant issues. These evaluations focus on preventing fraud, waste, or abuse and promoting economy, efficiency, and effectiveness of departmental programs. To promote impact, OEI reports also present practical recommendations for improving program operations. The Office of Investigations (OI) conducts criminal, civil, and administrative investigations of fraud and misconduct related to HHS programs, operations, and beneficiaries. With investigators working in all 50 States and the District of Columbia, OI utilizes its resources by actively coordinating with the Department of Justice and other Federal, State, and local law enforcement authorities. The investigative efforts of OI often lead to criminal convictions, administrative sanctions, and/or civil monetary penalties. The Office of Counsel to the Inspector General (OCIG) provides general legal services to OIG, rendering advice and opinions on HHS programs and operations and providing all legal support for OIG's internal operations. OCIG represents OIG in all civil and administrative fraud and abuse cases involving HHS programs, including False Claims Act, program exclusion, and civil monetary penalty cases. In connection with these cases, OCIG also negotiates and monitors corporate integrity agreements. OCIG renders advisory opinions, issues compliance program guidance, publishes fraud alerts, and provides other guidance to the health care industry concerning the anti-kickback statute and other OIG enforcement authorities. Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 About OIG | 26 ENDNOTES 1 Department of Defense and Labor, Health and Human Services, and Education Appropriations Act, 2019, and Continuing Appropriations Act, 2019, P.L. No. 115-245 (Sept. 28, 2018). 2 Department of Defense and Labor, Health and Human Services, and Education Appropriations Act, 2019, and Continuing Appropriations Act, 2019, P.L. No. 115-245 (Sept. 28, 2018). 3 NIH, Statement on Protecting the Integrity of U.S. Biomedical Research. Accessed at https://www.nih.gov/about- nih/who-we-are/nih-director/statements/statement-protecting-integrity-us-biomedical-research on February 22, 2021. 4 NIH, Peer Review. Accessed at https://grants.nih.gov/grants/peer-review.htm on November 30, 2020. 5 NIH, NIH Mission. Accessed at https://www.nih.gov/about-nih/what-we-do/nih-almanac/nih-organization on November 30, 2020. 6 NIH, About the NIH. Accessed at https://www.nih.gov/about-nih/what-we-do/nih-almanac/about-nih on November 30, 2020. 7 NIH, Budget. Accessed at https://www.nih.gov/about-nih/what-we-do/budget on November 30, 2020. 8 NIH Grant Policy Manual, ch. 4204-204B. pt. III. 9 NIH Grant Policy Manual, ch. 4204-204B. pt. IV. 10 NIH, NIH Grants Policy Statement, Section 2.4.1.1. Accessed at https://archives.nih.gov/asites/grants/04-16- 2018/policy/nihgps/index.htm on November 30, 2020. 11 NIH, NIH Peer Review: Grants and Cooperative Agreements, p. 2. Accessed at https://grants.nih.gov/grants/peerreview22713webv2.pdf on November 30, 2020. 12 NIH, About CSR. Accessed at https://public.csr.nih.gov/AboutCSR on November 30, 2020. 13 42 U.S.C. § 284a and 42 CFR § 52h.7. 14 NIH, NOT-OD-10-089. Accessed at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-10-089.html on February 23, 2021. 15 NIH, NIH Grants Policy Statement, Section 2.4.1.2. Accessed at https://archives.nih.gov/asites/grants/04-16- 2018/policy/nihgps/index.htm on November 30, 2020. 16 NIH, NIH Grants Policy Statement, Section 2.1.1. Accessed at https://archives.nih.gov/asites/grants/04-16- 2018/policy/nihgps/index.htm on November 30, 2020. 17 NIH, NIH Grants Policy Statement, Section 2.4.1.2. Accessed at https://archives.nih.gov/asites/grants/04-16- 2018/policy/nihgps/index.htm on November 30, 2020. 18 NIH, NIH Grants Policy Statement, Section 2.4.1.2. Accessed at https://archives.nih.gov/asites/grants/04-16- 2018/policy/nihgps/index.htm on November 30, 2020. 19 NIH, Bars to Grant Awards-Research Animals SOP. Accessed online at https://www.niaid.nih.gov/research/grants-bar-awards-research-animals on April 6, 2021. NIH, Bars and Restrictions to Grant Awards-Human Subjects SOP. Accessed online at https://www.niaid.nih.gov/research/grants-bar-awards-human-subjects on April 6, 2021. NIH, Inclusion of Women and Minorities as Participants in Research Involving Human Subjects. Accessed online at https://grants.nih.gov/policy/inclusion/women-and-minorities.htm on April 6, 2021. The three bars to funding listed were those in place during the time of our review. For grant applications due beginning in January 2019, NIH added "inclusion across the lifespan" to address the consideration of age as an inclusion variable in research involving human subjects. 20 NIH, NIMH Summary Statement-Bars to Funding, retrieved at https://www.nimh.nih.gov/funding/grant-writing- and-application-process/summary-statement.shtml on February 22, 2021. Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Endnotes | 27 21 NIH, NIH Grants Policy Statement, Sections 4.1.1, 4.1.15, 4.1.15.8. Accessed at https://archives.nih.gov/asites/grants/04-16-2018/policy/nihgps/index.htm on November 30, 2020. 22 NIH, NIH Peer Review: Grants and Cooperative Achievements. 2019. Accessed online at https://grants.nih.gov/grants/PeerReview22713webv2.pdf on March 22, 2021. 23 NIH Grant Policy Manual, ch. 4204-204B, pts. V, Section D(2) and IX, Section C(2). 24 NIH also publishes all rosters of peer reviewers on its website. 25 NIH, NIH Grants Policy Statement, Section 2.4.4. Accessed at https://archives.nih.gov/asites/grants/04-16- 2018/policy/nihgps/index.htm on February 23, 2021. 26 HHS Grant Policy Administration Manual, pt. G, ch. 1, Section b(56)(ii). 27 NIH Grant Policy Manual, ch. 4204-204c. 28 42 U.S.C. § 284a. 29 NIH, NIH Grants Policy Statement, Section 2.4.3. Accessed at https://archives.nih.gov/asites/grants/04-16- 2018/policy/nihgps/index.htm on November 30, 2020. 30 OIG, As Funding for BPA Research Increased, NIEHS Followed Its Peer Review Process While Also Exercising Its Discretion, OEI-01-15-00150, August 2017. 31 OIG, Vetting Peer Reviewers at NIH's Center for Scientific Review: Strengths and Limitations, OEI-01-19-00160, September 2019. 32 OIG, NIH Has Acted to Protect Confidential Information Handled by Peer Reviewers, But It Could Do More, OEI-05- 19-00240, March 2020. 33 OIG, NIH Has Made Strides in Reviewing Financial Conflicts of Interest in Extramural Research, But Could Do More, OEI-03-19-00150, September 2019. 34 OIG, The National Institutes of Health Has Controls to Mitigate the Risk That Grantees Receive Duplicate Grant Funding, A-02-19-02002, March 2020. 35 OIG, The National Institutes of Health Has Limited Policies, Procedures, and Controls in Place for Helping To Ensure That Institutions Report All Sources of Research Support, Financial Interests, and Affiliations, A-03-19-03003, September 2019. 36 The NIH activity codes we used to identify grants supporting basic and cooperating research were R01, R03, R21, R24, R33, R35, R56, R61, RF1, U01, U19, U24, U2C, U34, U41, U54, UA5, UF1, UG1, UG3, UH2, UH3, UL1, and UM1. 37 We defined the groups based on the distribution of counts of grants that each IC funded in FY2018. Large ICs had grant counts that were in the top third of ICs, medium ICs in the middle third, and small ICs in the bottom third. 38 We received a total of 520 summary statements for the 540 grants in our sample. This is because certain types of grant applications can result in multiple grant awards; for example, multiproject grants. Twenty of the grants in our sample were from grant applications that NIH used as the basis for funding between two and five grants. 39 NIH, NIH Peer Review: Grants and Cooperative Agreements, p. 2. Accessed at https://grants.nih.gov/grants/peerreview22713webv2.pdf on November 30, 2020. 40 By complete sets of scores, we generally mean a set of scores from at least three peer reviewers, with each set including a numerical score for each of NIH's five scored criteria. However, for handful of grants, NIH's Funding Opportunity Announcements called for an alternative scoring approach, such as an overall single set of scores or no scores for the five scored criteria. In these cases, we reviewed scores against the specified alternative approach. 41 For these grants, at least three peer reviewers and/or the Scientific Review Officer deemed a criterion to be unacceptable. 42 For details on select agents, see https://www.selectagents.gov/sat/list.htm?CDC_AA_refVal=https%3A%2F%2Fwww.selectagents.gov%2FSelectAge ntsandToxinsList.html Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Endnotes | 28 43 NIH, NIH Grants Policy Statement, Section 16.3. Accessed at https://archives.nih.gov/asites/grants/04-16- 2018/policy/nihgps/index.htm on February 23, 2021. 44 We note that beyond peer review, NIH has additional processes for vetting and approving the use of select agents and involvement of foreign entities in research, e.g., vetting applicants through the U.S. Department of State and the Centers for Disease Control and Prevention. 45 NIH, Statement on Protecting the Integrity of U.S. Biomedical Research. Accessed online at https://www.nih.gov/about-nih/who-we-are/nih-director/statements/statement-protecting-integrity-us- biomedical-research on April 6, 2021; NIH, NIH Peer Review: Grants and Cooperative Achievements. 2019. Accessed online at https://grants.nih.gov/grants/PeerReview22713webv2.pdf on March 22, 2021. 46 HHS Grants Policy Administration Manual, pt. G. ch. 1. (b)(56(ii). 47 One grant was funded because the IC identified concerns with two more-favorably ranked grants ahead of it. In this case, the IC wrote justifications for not funding the more-favorably ranked grants. Those justifications acknowledged the strength of the proposed projects but cited administrative concerns with the principal investigators' ability to do the research. Examples include concerns that the investigator was overcommitted or uncertainty about whether the investigator's employment status at the grantee institution would soon end. Selected NIH Institutes Met Requirements to Document Peer Review But Could Do More To Track and Explain Funding Decisions, OEI-01-19-00140 Endnotes | 29