ther general comments: U.S. Department of Health and Human Services Office of Inspector General Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations OEI-01-19-00160 Suzanne Murrin September 2019 Deputy Inspector General for Evaluation and Inspections oig.hhs.gov Report in Brief U.S. Department of Health and Human Services September 2019 OEI-01-19-00160 Office of Inspector General Vetting Peer Reviewers at NIH’s Center for Why OIG Did This Review Scientific Review: Strengths and Limitations Congress, NIH, and Federal Each year the National Institutes of Health’s intelligence agencies have raised (NIH) Center for Scientific Review (CSR) Key Takeaway concerns about foreign threats to the relies on thousands of peer reviewers who NIH’s Center for Scientific integrity of U.S. medical research and review tens of thousands of extramural Review focuses its vetting of intellectual property. In August grant applications, helping NIH to peer reviewer nominees on 2018, Dr. Francis Collins, Director of determine the most promising research to scientific skills and NIH, raised concerns that peer fund. These peer reviewers are generally preventing undue influence reviewers were, in some cases, not Federal employees but are considered generally, but does not inappropriately sharing confidential professional service consultants. focus its vetting specifically information with foreign entities. on undue foreign influence. Subsequently, in 2018 Congress What OIG Found provided the Office of Inspector NIH’s CSR has strengths in its approach to vetting nominees’ ability to be General with $5 million for oversight effective peer reviewers. CSR verifies nominees’ scientific expertise using of NIH grant programs and multiple sources such as their publication and grant histories. It also assesses operations, including the nominees’ communication skills and their ability to leverage those skills in effectiveness of NIH’s efforts to a peer review setting. Peer reviewer nominees are typically well known to protect intellectual property derived CSR even before their nominations, having served as temporary reviewers or from NIH-supported research. This having been NIH grantees. This allows CSR to evaluate the nominees on the study assesses the strengths and basis of its past experiences with them. NIH also has controls to ensure that limitations of CSR’s vetting of peer it does not select nominees who have engaged in research misconduct or reviewer nominees before they begin breaches of peer review. However, CSR’s vetting of peer reviewer nominees reviewing applications for research gives little attention to foreign affiliation beyond requiring a justification for grants. These peer reviewers have reviewers who are not based in North America. Although nearly all of CSR’s a unique opportunity to access peer reviewers work at institutions within the United States, they include both confidential information in grant U.S. citizens and foreign nationals. CSR vets all of these reviewers in the applications. Because of this access, same way. CSR generally limits the sources it uses in vetting for other it is important for NIH to ensure that concerns (e.g., legal and moral controversies) to publicly reported peer reviewers do not information, which is unlikely to reveal such things as undisclosed foreign inappropriately disclose or divert support. confidential information, including intellectual property. What OIG Recommends NIH is taking steps to address concerns about foreign threats to research How OIG Did This Review integrity, but it could do more with respect to vetting peer reviewer We interviewed NIH staff at CSR, the nominees. We recommend that NIH update its guidance on vetting peer Office of Extramural Research, the reviewer nominees to identify potential foreign threats to research integrity, Office of Management Assessment, in consultation with national security experts, as needed, and that NIH work and the Office of Federal Advisory with the Department of Health and Human Services’ Office of National Committee Policy. We discussed Security to develop a risk-based approach for identifying those peer reviewer how CSR staff vet peer reviewer nominees who warrant additional vetting. NIH concurred with both nominees and how NIH policy recommendations. governs that vetting. We also reviewed NIH policy, guidance, and training materials related to vetting peer reviewers. Full report can be found at oig.hhs.gov/oei/reports/oei-01-19-00160.asp TABLE OF CONTENTS BACKGROUND 1 Methodology 5 FINDINGS NIH’s Center for Scientific Review has strengths in its approach to vetting nominees’ ability 7 to be effective peer reviewers Beyond requiring a justification for reviewers who are not based in North America, NIH’s 11 Center for Scientific Review gives little attention to foreign affiliation in its peer reviewer vetting NIH’s Center for Scientific Review generally limits the sources it uses in vetting for other 13 concerns to publicly available information CONCLUSION AND RECOMMENDATIONS NIH should update its guidance on vetting peer reviewer nominees to identify potential 15 foreign threats to research integrity, in consultation with national security experts, as needed NIH should work with the HHS Office of National Security to develop a risk-based approach 16 for identifying those peer reviewer nominees who warrant extra scrutiny AGENCY COMMENTS AND OIG RESPONSE 17 APPENDIX: AGENCY COMMENTS 18 ACKNOWLEDGMENTS 20 ENDNOTES 21 BACKGROUND Objective To describe and assess the National Institutes of Health’s (NIH’s) process for vetting peer reviewer nominees at its Center for Scientific Review (CSR). Concerns About For FY 2019, the U.S. Department of Health and Human Services (HHS) Office of Inspector General (OIG) received $5 million in congressional Foreign Threats to appropriations to conduct oversight of NIH grant programs and Research Integrity operations.1 As indicated by the conference report that accompanied the legislation, “the conferees direct[ed] the OIG to examine NIH’s oversight of its grantees’ compliance with NIH policies.”2 This review is part of a larger body of HHS OIG work focused on oversight of NIH grant programs and operations. Our work will review (1) intellectual property and cybersecurity protections; (2) compliance with Federal requirements and NIH policies for grants and contracts; and (3) integrity of grant application and selection processes. As part of this oversight work, Congress directed that OIG examine (1) NIH’s efforts to ensure the integrity of its grant evaluation and selection processes, and (2) the effectiveness of NIH’s and grantee institutions’ efforts to protect intellectual property derived from NIH-supported research.3 In his August 2018 statement on protecting the integrity of U.S. biomedical research, NIH Director Dr. Francis Collins stated that the risks to the security of intellectual property and the integrity of peer review were increasing. He identified specific areas of concern including, in some instances, the inappropriate sharing of confidential information by peer reviewers. Dr. Collins further stated that in response to the increasing risks, NIH would work with other government agencies, institutions, and organizations to identify robust methods to protect the integrity of peer review.4 At the same time, Dr. Collins wrote to NIH grantee institutions alerting them to these foreign threats, and stated that NIH would be working with the institutions to address those threats.5 Additionally, congressional committees have expressed concerns about potential threats to the integrity of taxpayer-funded research and intellectual property, including the theft of intellectual property and its diversion to foreign entities.6 Indeed, in a June 2019 Senate hearing, NIH Principal Deputy Director Dr. Lawrence A. Tabak testified that NIH was “aware that a few foreign governments have initiated systematic Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 1 OEI-01-19-00160 programs to capitalize on the collaborative nature of biomedical research and unduly influence U.S.-based researchers.”7 China’s Thousand Talents Plan China’s central government announced the Thousand Talents plan in 2008. One aspect of the plan provides financial incentives for Chinese scientists living abroad to return to China.8 According to NIH, access to foreign intellectual property is key to a scientist’s selection for the Thousand Talents program. In a 2018 meeting of NIH’s Advisory Committee to the Director, NIH identified several concerns related to the Thousand Talents plan:  undisclosed foreign financial conflicts;  undisclosed conflicts of commitment; and  peer review violations, including the inappropriate sharing of confidential information.9 The peer review process is central to the integrity of evaluating and selecting grants. According to NIH policy, the peer review process is intended “to ensure that applications for funding submitted to NIH are evaluated on the basis of a process that is fair, equitable, timely, and conducted in a manner that strives to eliminate bias.”10 However, as Dr. Collins’ statement suggested, peer reviewers also present a potential risk to intellectual property and other confidential information contained in grant applications if they inappropriately share that information.11 Because peer reviewers conduct the initial review of research grants submitted to NIH, they have a unique opportunity to access confidential information in grant applications. Accordingly, NIH policy seeks to ensure that peer reviewers do not inappropriately disclose or divert confidential information, including information related to intellectual property.12 Steps NIH Has In response to these concerns, NIH has taken steps to protect the integrity of the grant and peer review process. To further raise awareness of Taken To Address confidentiality in peer review, NIH has launched ongoing communications Concerns About with its staff, the research community, and grantee institutions, some of Foreign Threats to which have proactively raised concerns with NIH.13 NIH convened Research Integrity a working group of the Advisory Committee to the NIH Director to explore additional steps to protect the integrity of peer review.14 NIH is also working with Federal agencies both inside HHS (e.g., the HHS Office of National Security and the Office of Inspector General) and outside Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 2 OEI-01-19-00160 (such as the Federal Bureau of Investigation) to identify and follow up on scientists of possible concern among NIH grantees.15 According to NIH, it has identified about 250 scientists of concern to date. Furthermore, NIH has worked to improve the security of the electronic systems that researchers use to submit grant applications and that peer reviewers use to access these applications.16 NIH’s Peer Review Each year, NIH receives about 80,000 extramural grant applications. (Extramural research is research that NIH supports through funds to Process outside researchers and organizations.) It uses a two-level review process to evaluate the applications.17 For about 75 percent of those applications, CSR manages the first level of review. For the remaining 25 percent of applications, NIH’s other institutes and centers (ICs) manage the first level of review. At CSR, peer reviewers who serve on study sections (also known as scientific review groups) conduct the first level of review of grant applications.18 During this review, study sections evaluate whether proposed projects are likely to “have a major scientific impact” based on their scientific and technical merit.19 Study sections are organized by research area and include peer reviewers recognized as experts in that research area. After the study section conducts its review, an Advisory Council or Board conducts the second level of review.20 The Director of each IC or a designee makes the final funding decision.21 Over 27,000 peer reviewers review grant applications annually for NIH.22 First-Level Peer NIH seeks to ensure that peer reviewers review grant applications in Reviewers a manner free from inappropriate influences. It identifies the following core values for its peer reviewers: (1) expert assessment; (2) transparency; (3) impartiality; (4) fairness; (5) confidentiality; (6) security (added in 2018); (7) integrity, and (8) efficiency.23 NIH’s peer reviewers are generally not Federal employees or Special Government Employees.24 NIH considers them to be professional service consultants.25 However, up to 25 percent of the members of a first-level peer review group can be Federal employees.26 Nominating and Vetting Peer Reviewers at CSR Scientific Review Officers (SROs) at CSR are responsible for identifying potential peer reviewers and nominating them to CSR study sections for consideration. SROs recruit people to serve both as temporary and appointed peer reviewers. Most people are first recruited to be temporary reviewers before being nominated to be appointed peer reviewers. Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 3 OEI-01-19-00160 Temporary and Appointed Peer Reviewers Temporary peer reviewers provide specific expertise. As temporary reviewers, they offer a fresh perspective for individual peer review meetings. If temporary reviewers are successful, SROs may nominate them to become appointed peer reviewers. Appointed peer reviewers typically serve 4-year terms and are expected to attend all meetings of their respective standing study sections.27 NIH also offers the public several ways to apply for consideration as a peer reviewer, including contacting an SRO from CSR, applying to NIH’s Early Career Reviewer Program (a CSR program to help start research careers and enhance the pool of future reviewers), and e-mailing one’s curriculum vitae to an NIH peer reviewer mailbox.28 For appointed nominees, SROs prepare a nomination slate of peer reviewers based on the nominees’ expertise and the needs of each study section. CSR’s Nomination Slate Guidelines instruct SROs to nominate people with the necessary scientific expertise for that study section.29 CSR trains SROs on these guidelines. Other parts of NIH in addition to CSR also have roles overseeing and developing policies related to vetting peer reviewers. The Office of Extramural Research both oversees the integrity of the peer review process and writes policies governing the external grantmaking process.30 This office also maintains NIH’s Electronic Research Administration program, which provides information technology support to NIH’s grantmaking process. Part of this program is the Query/View/Report module, a system that contains profiles of all extramural grantees and peer reviewers.31 The Office of Management Assessment conducts internal risk assessments and advises NIH on program integrity matters.32 Finally, the Office of Federal Advisory Committee Policy develops policies related to the Federal Advisory Committee Act, which governs the establishment and operation of advisory committees across all Federal agencies.33 Related Work OIG conducted a previous evaluation related to NIH’s peer review process. In response to a congressional request, we examined the extent to which NIH’s National Institute of Environmental Health Sciences (NIEHS) followed its peer review processes for funding research related to bisphenol A (BPA). In our 2017 report, OIG found that although NIEHS met the requirements of the peer review process for all its grants, it used Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 4 OEI-01-19-00160 its discretion to fund applications with less favorable scores than competing applications for 14 percent of BPA grants, versus for 4 percent of other, non-BPA grants. OIG concluded that although NIEHS has discretion to fund grant applications with less favorable impact scores ahead of competing applications, NIEHS’s applying its discretion frequently or disproportionately in one research area may invite scrutiny to its funding decisions.34 OIG is currently evaluating how NIH oversees its peer reviewers once they begin reviewing grant applications. That work is forthcoming. Methodology Scope This report covers the vetting process that NIH used in 2019 for peer reviewers who serve on study sections organized by CSR. These study sections review about 75 percent of all NIH grant applications. This report defines vetting as the initial screening that NIH staff conduct for peer reviewer nominees before they begin reviewing grant applications. Future OIG work will focus on NIH’s oversight of peer reviewers once they begin reviewing grant applications.35 This report does not include the first level of review organized by other ICs (e.g., the National Cancer Institute), special emphasis panels, or the second level of grant application review. Scientific and disease experts who are appointed as Special Government Employees largely conduct the second level of review. Data Collection We requested from NIH its policies, procedures, and training materials governing peer reviewer vetting. Some of the procedures were specific to CSR, and some policies—like those from the Office of Extramural Research—applied to all of NIH. We interviewed staff at CSR about how they vet peer review nominees. We interviewed five SROs, four Integrated Review Group Chiefs, the CSR Research Integrity Officer, and staff from the Office of the Director of CSR. SROs conduct the initial vetting of peer reviewer nominees. They are followed by the Integrated Review Group Chiefs, who review the materials that the SROs have compiled. We also interviewed NIH staff at the Office of Extramural Research, the Office of Management Assessment, and the Office of Federal Advisory Committee Policy about their roles in setting policy regarding the vetting of peer reviewers. Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 5 OEI-01-19-00160 Data Analysis We analyzed the documents and interviews to identify strengths and limits of CSR’s approach to vetting peer reviewer nominees, and the extent to which CSR considered undue foreign influence. Limitations Although some of the policies that we reviewed apply to all of NIH, our findings and recommendations are focused on CSR. We did not verify staff’s compliance with NIH’s vetting policies. Standards We conducted this study in accordance with the Quality Standards for Inspection and Evaluation issued by the Council of the Inspectors General on Integrity and Efficiency. Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 6 OEI-01-19-00160 FINDINGS NIH’s Center for This report defines vetting as the initial screening that NIH staff conduct on peer reviewer nominees before they begin reviewing grant Scientific Review applications. Has Strengths in its CSR’s Nomination Slate Guidelines state that the expertise of peer Approach to Vetting reviewers should match the core scientific topics and methods that their Nominees’ Ability study sections review. SROs told us that their primary goal in vetting peer To Be Effective Peer reviewer nominees was to verify that the nominees did in fact possess the Reviewers required expertise, and that the nominees could effectively use that expertise within the context of peer review. CSR verifies nominees’ scientific expertise using multiple sources CSR’s Nomination Slate Guidelines describe indicators of expertise that SROs should use to assess nominees’ scientific expertise. These include obtaining their grants; presenting at conferences; being published in journals; holding patents; having professional appointments; and having editorial and review duties (e.g., at a journal). CSR staff told us that accordingly, they vet nominees’ scientific expertise and stature, and they described a thorough process for doing so. CSR staff research nominees’ publications in scientific journals; the websites of their research institutions or companies; their prior research grants from NIH and other Federal agencies; as well as the nominees’ curricula vitae. For example, one SRO we spoke with checks online for recorded speeches that nominees have given at scientific conferences and symposia. Furthermore, Integrated Review Group Chiefs told us they also perform their own check of these sources when SROs send them the nomination slates for review. According to NIH, it values the scientific community’s assessment of its nominees for peer review. CSR’s guidelines call for every nominee to have two recommendations from references who endorse their credentials and expertise, and to reflect independent, broad vetting of the nominee by leaders of the scientific community served by the study section. SROs told us that they sometimes talk in person with the reference to get a better sense of the nominee’s communication skills, objectivity, and understanding of the responsibilities of serving as a peer reviewer. With regard to stature and funding history, CSR’s guidelines say that a nominee should have academic rank as a full or associate professor and should be an active and successful researcher. The guidelines state that although NIH funding is preferred, other competitive sources can be used Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 7 OEI-01-19-00160 as indicators of a successful researcher. (CSR’s training materials for SROs offer as examples the Department of Defense and the Department of Veterans Affairs). NIH mentions that additional indicators of the nominee’s academic stature can be:  leadership roles in professional organizations;  faculty rank and any endowed chairs; and  editorial position. CSR also considers nominees’ ability to effectively leverage their scientific expertise in a peer review setting CSR values reviewers who can articulate their views succinctly, engage in productive exchanges, and demonstrate an ability to work collegially in a group setting.36 CSR guidelines state that nominees should serve as temporary reviewers before being nominated to serve 4-year terms as appointed peer reviewers. This enables multiple CSR staff—including SROs and Integrated Review Group Chiefs—to observe the temporary reviewers during the meetings and gauge their performance. CSR staff told us they look to see how well the temporary reviewers perform in their roles, including how well they interact with other reviewers. If this experience raises questions about a nominee’s effectiveness, the SRO may decline to nominate that individual. Finally, SROs told us they check nominees’ references to assess their communication skills, along with checking their scientific credentials. Peer reviewer nominees are typically well known to CSR, enabling CSR to vet them against its past experience In addition to having past experience with nominees as ad-hoc reviewers, NIH is likely to have had experience with them as grantees or appointed reviewers in other study sections before SROs nominate them. In fact, CSR staff told us that they have often known nominees for years, and that if a person applied to be an appointed peer reviewer without having had any of those roles, it would raise a red flag with the SROs. As a result, NIH typically has profiles of the nominees in its Query/View/Report database module, allowing SROs to vet the nominees against their past experiences with NIH. SROs also check the module to see whether it includes a Do Not Use alert for any of the nominees, which would prevent those nominees from serving as peer reviewers—in fact, the existence of such an alert prevents SROs from adding a person to a nomination slate or a meeting roster. NIH’s recent initiatives in working with grantee institutions and law enforcement partners have resulted in persons receiving Do Not Use alerts. Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 8 OEI-01-19-00160 Most nominees have been NIH grant recipients themselves. This is one of the criteria SROs look for when nominating an appointed peer reviewer. In fact, SROs must justify nominees who have repeatedly applied for and “Do Not Use” Alerts failed to obtain NIH grants. The NIH leadership can place a “Do Not nominees’ having been grantees Use” alert on an individual’s gives NIH more information about Query/View/Report profile. Persons their scientific expertise and record of with this flag cannot serve as peer accomplishment in their fields, which reviewers. are criteria in CSR’s nomination guidelines. For peer reviewer nominees who have already served as appointed reviewers on other study sections, CSR staff can rely on their own observations. CSR staff also share information with each other about these reviewers. In fact, CSR developed a database to share internal recommendations from other NIH staff to consider regarding potential peer reviewers. This enables SROs to nominate those with a record of effective review experience and to avoid those who did not perform well in the past. As one Integrated Review Group Chief explained, “It’s not like we’re suddenly hearing about this person out of the blue. We have plenty of time to really look at this person in quite some detail.” One SRO told us that if someone applied to be a peer reviewer without having any of these prior experiences, the SRO would encourage that person to apply for NIH’s Early Career Reviewer Program. NIH designed this program to help less experienced researchers gain skills to make them better grant applicants and enrich NIH’s pool of potential peer reviewers. NIH has controls to ensure that nominees with findings of research misconduct and peer review breaches are not selected to serve NIH relies not only on its own oversight but also that of others to prevent individuals who have engaged in research misconduct from being nominated. NIH works with HHS’s Office of Research Integrity regarding cases of research misconduct. When that office confirms a finding of research misconduct, it notifies NIH, which then places a Do Not Use alert in that person’s Query/View/Report profile, preventing SROs from nominating the person. Likewise, if NIH learns of misconduct allegations first, it will assess the allegations and then forward the case to the Office of Research Integrity for followup. Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 9 OEI-01-19-00160 The Office of Research Integrity On behalf of the Secretary of Health and Human Services, HHS’s Office of Research Integrity oversees and directs most research integrity activities—including oversight of inquiries and investigations related to research misconduct—in the Public Health Service divisions of HHS.37 Research misconduct is defined as "fabrication, falsification or plagiarism in proposing, performing or reviewing research, or in reporting research results” and “does not include honest error or differences of opinion.”38 Additionally, NIH prevents peer reviewers who have violated review policy from serving again. NIH may also take action against persons who violate peer review policy in times other than during a review panel—for example, by attempting to communicate with or manipulate other reviewers prior to a panel. NIH staff may become aware of such violations during its oversight of reviewers or learn about allegations from other reviewers. Being Vested in a Fair System NIH staff told us that they rely on peer reviewers to uphold a culture of fairness in the review process. They explained that because peer reviewers are also grantees, they are vested in ensuring that the system is fair. After all, for the majority of reviewers, their own grant applications must also go through the process. NIH relies on peer reviewers to report to SROs any potential breach of peer review integrity of which they are aware. SROs refer the allegations to Research Integrity Officers at CSR and the Office of Extramural Research, both of which evaluate such allegations. If the Research Integrity Officers confirm the allegations, NIH leadership bars the individuals from participating as peer reviewers in the future. Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 10 OEI-01-19-00160 Beyond Requiring a According to NIH, nearly all of CSR’s peer reviewers work at research institutions located within the United States. These reviewers include Justification for both U.S. citizens and foreign nationals. NIH does not require nominees Reviewers Who Are to disclose citizenship, but it typically knows the research institutions at Not Based in North which they are based. A very small number of peer reviewers— America, NIH’s 0.6 percent of reviewers, according to CSR—work at foreign research institutions. CSR staff told us that most of these foreign-based reviewers Center for Scientific are from Canadian institutions. Review Gives Little NIH focuses on preventing undue influence generally, but not Attention to Foreign specifically undue foreign influence Affiliation in its Peer One of the key ways in which NIH upholds the integrity of peer review is Reviewer Vetting by working to ensure that its peer reviewers cannot exert or be swayed by undue influence. Both CSR’s Nomination Slate Guidelines and the Federal Advisory Characterizing Undue Influence Committee Act39 specify NIH intends its peer review process criteria for preventing undue to be fair and conducted in a manner influence in peer review. that strives to eliminate bias. Bias These criteria assess a can include undue influence by and nominee individually and of peer reviewers. with respect to the peer Although NIH does not define undue reviewers with whom the influence, it considers several ways in nominee will serve. which it could occur. For example, if The CSR staff we spoke with a study section contained multiple described taking steps peer reviewers from a specific accordingly to prevent any research institution, or if it included individual research peer reviewers who had served on institution from being able that study section for a long time, to overly influence the those reviewers could hold too much review process, and any sway over the rest of the group, reviewer from being adversely affecting the objectivity of co-opted by others. For peer review. example, CSR staff said that Additionally, Congress and NIH have to avoid undue influence raised concerns about undue foreign from any single research influence—for example, if foreign institution, study sections entities systemically encouraged NIH rarely include more than one peer reviewers to disclose reviewer from the same confidential information or influence institution. CSR staff also grant funding decisions. said they try to ensure that study sections contain balanced and diverse representation in terms of demographics (e.g., race and gender) and region. Furthermore, CSR’s training materials for evaluating potential peer reviewers instruct SROs to consider diversity of Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 11 OEI-01-19-00160 scientific skills (e.g., methods and topics). Finally, NIH policy and guidelines also include provisions to ensure that particular nominees do not have undue influence during grant reviews. NIH policy prohibits any reviewer from serving on both the first and second levels of a given review, and CSR’s Nomination Slate Guidelines states that reviewers should generally not serve multiple terms on the same study section. However, neither CSR’s Nomination Slate Guidelines nor the Federal Advisory Committee Act address the circumstances in which foreign affiliation (e.g., persons employed by a foreign government or company, or who consult for a foreign government) could raise potential concerns of undue influence. Although nearly all of CSR’s peer reviewers work at research institutions within the United States, they include both U.S. citizens and foreign nationals. CSR vets all of these reviewers in the same way. According to NIH, identifying undue foreign influence is not something that SROs do or could do. NIH staff stated that many foreign affiliations are legitimate and are not, in and of themselves, suggestive of undue influence. In fact, NIH’s Review Integrity Training for SROs reminds them of the value that individuals with foreign affiliations and collaborations provide to NIH. This training also states that, unless a Do Not Use alert exists in a person’s Query/View/Report profile, the person is eligible to serve at the SRO’s discretion. Furthermore, it would be difficult for CSR staff to detect undue foreign influence even if peer reviewer nominees had previously disclosed on their curricula vitae—either as peer reviewer nominees or as NIH grant applicants—any foreign affiliations or foreign support. It would be even more challenging to identify undue foreign influence if nominees had failed to disclose such support. Because of this challenge, NIH risks bringing on board peer reviewers with conflicts of interest or commitment that could affect the outcome or integrity of peer review. NIH has efforts underway to address this challenge, led primarily by the Office of Extramural Research. NIH’s ongoing work with partners within HHS and elsewhere to identify and follow up on scientists of concern among NIH grantees shows that peer review is indeed at risk of undue foreign influence. In fact, according to NIH, of the 250 scientists it has identified to date as individuals of possible concern, roughly 30 percent had served as a peer reviewer over the past 2 years, and NIH’s follow up continues. CSR policy discourages—because of logistical considerations— the use of reviewers based outside of North America Because study section meetings generally take place in person in the United States, CSR policy favors the use of peer reviewers who work at Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 12 OEI-01-19-00160 North American research institutions. In its 2009 guidance on reviewers who are based abroad, CSR stated that in general, SROs should not nominate reviewers who reside and work abroad—regardless of their nationality—to serve as appointed peer reviewers. CSR may make exceptions in cases in which, for example, a nominee has unique knowledge or expertise. In such cases, SROs must first discuss the nominee’s candidacy with their Integrated Review Group Chiefs and submit a justification for approval to their Division Director. However, CSR instructs SROs to keep this practice to a minimum. This may explain in part why the overwhelming majority of peer reviewers work at research institutions based in the United States. The guidance focuses on logistical issues—such as the travel required, cost, and administrative burden—to explain why SROs should not nominate reviewers based abroad. The guidance exempts reviewers from Canada and Mexico from requiring special approval because travel costs for such reviewers tend to be comparable to those for reviewers based in the United States. In addition, because CSR has invited reviewers from Canada and Mexico to serve as temporary or appointed peer reviewers in the past, and because such individuals are often NIH grantees, they are likely to be familiar with the NIH peer review process. Furthermore, NIH views its close working relationships with grantee institutions to be a key part of its response to concerns over the security of intellectual property and the integrity of peer review. Using peer reviewers who are based at grantee institutions is important because the institutions—as the peer reviewers’ employers—have the authority to take actions against employees who violate the integrity of peer review.40 NIH’s Center for In vetting peer reviewers, verifying nominees’ scientific ability is NIH’s foremost priority. CSR’s Nomination Slate Guidelines address how to Scientific Review assess an individual’s expertise, stature, and funding history as generally limits the a researcher. The guidelines also state that SROs should conduct an sources it uses in ethics screening to identify legal or moral controversies surrounding a nominee (e.g., findings of sexual harassment). SROs have access to vetting for other candidates’ profiles in NIH’s Query/View/Report, and they check for any concerns to publicly Do Not Use alerts that were placed on any candidates’ profiles as a result reported of research misconduct or breaches of peer review, or as a result of any information nonpublic information that NIH becomes aware of. However, the guidelines do not advise vetting nominees against any type of law enforcement database. Instead, they suggest generally reviewing the first page of results from a Google search. NIH staff told us that in their screenings of nominees, they search to see whether any major news organizations have published anything regarding nominees and criminal activity, legal issues, or sexual misconduct. Relying on the Internet or Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 13 OEI-01-19-00160 news organizations for this type of information means that NIH is unlikely to discover concerns that are not reported in the public domain. NIH staff described the burdens that would be associated with conducting traditional background checks on all peer reviewers. They noted that SROs are scientists and not trained as investigators. In fact, NIH estimates that—in its experience in vetting about 250 scientists—conducting a preliminary investigation would require roughly 10 hours for each potential reviewer. NIH further estimates that vetting all 27,000 peer reviewers in this manner would require more than 100 additional full-time staff. Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 14 OEI-01-19-00160 CONCLUSION AND RECOMMENDATIONS NIH’s peer review process is a vast enterprise that is key to supporting its research mission. Each year, this process encompasses thousands of reviewers who review tens of thousands of grant applications. NIH’s peer review enterprise relies on experts in their field being willing to serve. NIH’s Center for Scientific Review places a high value on recruiting scientific experts to serve as peer reviewers. However, the world is changing, and threats have emerged. Peer reviewers who disclose confidential information to foreign entities could undermine confidence in the integrity of peer review, put intellectual property at risk, and diminish NIH’s reputation. NIH has been working with the HHS Office of National Security, Federal law enforcement (including the HHS Office of Inspector General and the Department of Justice), and research institutions to address concerns about foreign threats to research integrity in other contexts, such as grantees’ failing to disclose foreign financial interests as required. NIH has also worked to better secure the systems that peer reviewers use when accessing grant applications. However, in its procedures for vetting peer reviewer nominees, NIH has not addressed the concerns about foreign threats to the integrity of the peer review process. OIG recognizes that NIH has raised several concerns related to addressing foreign threats as part of the process of vetting peer reviewer nominees. These include concerns about added burden for the nominees and NIH staff; concerns about perceptions of racial/ethnic profiling and stigmatization; and concerns that identifying undue foreign influence is challenging and largely outside of NIH’s expertise. In addition, no system of vetting is foolproof. Nonetheless, NIH has an opportunity to enhance its vetting of peer reviewers and further protect its integrity and reputation. Therefore, we recommend that NIH: Update its guidance on vetting peer reviewer nominees to identify potential foreign threats to research integrity, in consultation with national security experts, as needed Much of CSR’s guidance to SROs about peer reviewer nominees with foreign affiliations is 10 years old and addresses only the logistical challenges of using reviewers who are based abroad. NIH could strengthen its guidance by expanding it to include foreign threats to research integrity such as foreign support that nominees have omitted from their curricula vitae. CSR is already including review integrity in its training for SROs. If CSR updated its guidance to include what is already in this training, it could highlight these concerns and help ensure consistency across SROs and other staff in acting on them. Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 15 OEI-01-19-00160 NIH could further update its guidance by drawing on its experience and consultation with national security experts and include examples of situations that warrant followup. The guidance could enhance awareness among SROs to report these situations to CSR’s Research Integrity Officer, who could then assess them as appropriate. We recognize that this could result in a higher volume of work for CSR’s Research Integrity Officer (and ultimately the Research Integrity Officer at the Office of Extramural Research), but SROs are not investigators trained in assessing threats to research integrity. Such guidance also provides NIH with the opportunity to elevate the prominence of its new core value—security—for peer reviewers. Work with the HHS Office of National Security to develop a risk-based approach for identifying those peer reviewer nominees who warrant additional vetting As the HHS agency focused on national security, the Office of National Security is a natural partner for NIH to strengthen the process of vetting peer reviewer nominees. In fact, the Office of National Security is already working with NIH to identify steps to mitigate threats to U.S. biomedical research from foreign entities. For example, the Office of National Security currently vets some employees and foreign visitors to NIH by using databases of its national security partners. Using a risk-based strategy would enable NIH to expand its vetting for a subset of peer reviewer nominees, while also limiting the increase in burden on its staff. For example, NIH could focus extra attention to vetting peer reviewer nominees who would be reviewing grant applications with particularly sensitive subject matter or that have lucrative commercial applications. Additionally, by working with the Office of National Security, NIH can leverage the Office of National Security’s expertise and expand the sources that NIH uses to vet peer reviewer candidates. Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 16 OEI-01-19-00160 AGENCY COMMENTS AND OIG RESPONSE NIH concurred with both of our recommendations. In response to the first recommendation, NIH noted that it is working closely with Federal partners both inside the Department (such as OIG and the HHS Office of National Security) and external agencies, including the Federal Bureau of Investigation, to update its guidelines for vetting peer reviewers. Regarding the second recommendation, NIH stated that is developing a systematic, risk-based, data-driven approach to identifying peer review nominees who warrant additional scrutiny. For the full text of NIH’s comments, see the Appendix. Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 17 OEI-01-19-00160 APPENDIX: AGENCY COMMENTS Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 18 OEI-01-19-00160 Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 19 OEI-01-19-00160 ACKNOWLEDGMENTS Ivan Troy served as the team leader for this study. Others in the Office of Evaluation and Inspections who conducted the study include Matt Blackburn and Shakye Jones. Office of Evaluation and Inspections staff who provided support include Clarence Arnold, Christine Moritz, and Melicia Seay. We would also like to acknowledge the contributions of other Office of Inspector General staff, including Lauren Anderson, Melissa Baker, Sarah Vogel, Kelly Waldhoff, and Abigail Wydra. This report was prepared under the direction of Joyce M. Greenleaf, Regional Inspector General for Evaluation and Inspections in the Boston regional office. To obtain additional information concerning this report or to obtain copies, contact the Office of Public Affairs at Public.Affairs@oig.hhs.gov. Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 20 OEI-01-19-00160 ENDNOTES 1 Department of Defense and Labor, Health and Human Services, and Education Appropriations Act, 2019, and Continuing Appropriations Act, 2019, P.L. No. 115-245 (Sept. 28, 2018). 2 Department of Defense for the Fiscal Year Ending September 30, 2019, and for Other Purposes. Conference Report to Accompany H.R. 6157, H.R. Rep. No. 115-952, 2018, pp. 531-532. 3 Ibid. 4 NIH, Statement on Protecting the Integrity of U.S. Biomedical Research. August 23, 2018. Accessed at https://www.nih.gov/about-nih/who-we-are/nih- director/statements/statement-protecting-integrity-us-biomedical-research on February 14, 2019. 5 NIH Advisory Committee to the Director, ACD Working Group for Foreign Influences on Research Integrity: December 2018 Report. Accessed at https://acd.od.nih.gov/documents/presentations/12132018ForeignInfluences_report. pdf on August 9, 2019. 6 Senator Charles Grassley, Chair, Senate Committee on Judiciary, letter to Dr. Francis Collins, October 23, 2018. Accessed at https://www.grassley.senate.gov/sites/default/files/constituents/2018-10- 23%20CEG%20to%20NIH%20(Research%20Threats).pdf on February 14, 2019. Association of American Medical Colleges, NIH Describes for Senate Panel Efforts to Protect Research Integrity, August 24, 2018. Accessed at https://www.aamc.org/advocacy/washhigh/highlights2018/490948/082418nihdescri besforsenatepaneleffortstoprotectresearchintegrit.html on February 14, 2019. 7 Testimony of Dr. Lawrence A. Tabak, Principal Deputy Director, NIH, before Senate Committee on Finance, June 5, 2019. Accessed at https://www.finance.senate.gov/imo/media/doc/05JUN2019TabakSMNT.pdf on June 20, 2019. 8 Jia, Hepeng, What is China’s Thousand Talents Plan? Accessed at https://www.nature.com/magazine-assets/d41586-018-00538-z/d41586-018-00538- z.pdf on June 20, 2019. 9 NIH Advisory Committee to the Director, ACD Working Group for Foreign Influences on Research Integrity: December 2018 Report. (See Endnote 5 for URL information.) 10 NIH, Grants Policy Statement, Part 1, October 2018, pp. 67-68. Accessed at https://grants.nih.gov/grants/policy/nihgps/nihgps.pdf on July 9, 2019. 11 NIH, Statement on Protecting the Integrity of U.S. Biomedical Research. August 23, 2018. (See Endnote 4 for URL information.) 12 NIH, Maintaining Confidentiality in NIH Peer Review. Accessed at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-14-073.html on January 17, 2019. NIH, Conflicts of Interest Rules: Information for Reviewers of NIH Applications and R&D Contract Proposals, July 2015. 13 Testimony of Dr. Lawrence A. Tabak, Principal Deputy Director, NIH, before Senate Committee on Finance, June 5, 2019. (See Endnote 7 for URL information.) 14 NIH Advisory Committee to the Director, ACD Working Group for Foreign Influences on Research Integrity: December 2018 Report. (See Endnote 5 for URL information.) Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 21 OEI-01-19-00160 15 Testimony of Captain Michael Schmoyer, Assistant Deputy Secretary for National Security, Director, Office of National Security, before Senate Committee on Finance, June 5, 2019. Accessed at https://www.finance.senate.gov/imo/media/doc/05JUN2019SchmoyerSMNT.pdf on June 24, 2019. Testimony of Dr. Lawrence A. Tabak, Principal Deputy Director, NIH, before Senate Committee on Finance, June 5, 2019. (See Endnote 7 for URL information.) 16 Testimony of Dr. Lawrence A. Tabak, Principal Deputy Director, NIH, before Senate Committee on Finance, June 5, 2019. (See Endnote 7 for URL information.) 17 NIH, Grants Policy Statement (October 2018), Page I-68. (See Endnote 10 for URL information.) 18 Ibid, pp. I-36 and I-68. 19 Ibid, p. I-69. 20 Ibid, pp. I-68 and I-72. 21 Ibid, p. I-73. 22 NIH, Data Book. Accessed at https://report.nih.gov/nihdatabook/category/12 on August 5, 2019. 23 NIH, Peer Review: Grants and Cooperative Agreements. Accessed at https://grants.nih.gov/grants/peerreview22713webv2.pdf on June 5, 2019. 24 As defined in 18 U.S.C. § 202, a Special Government Employee is an officer or employee who is retained, designated, appointed, or employed to perform temporary duties, with or without compensation, for not more than 130 days during any period of 365 consecutive days. 25 NIH, New Process for Reimbursement of NIH Peer Reviewers. August 5, 2005. Accessed at https://grants.nih.gov/grants/guide/notice-files/not-od-05-062.html on July 10, 2019. 26 42 CFR § 52h.4. 27 NIH, How Scientists are Selected to be Members of a Chartered Review Group. Accessed at https://public.csr.nih.gov/ForReviewers/BecomeAReviewer/CharteredReviewers on January 25, 2019. 28 NIH, How to Become A Reviewer. Accessed at https://public.csr.nih.gov/ForReviewers/BecomeAReviewer/HowToApply on January 17, 2019. 29 NIH, CSR Nomination Slate Guidelines for 2019 (document provided to OIG), and NIH Peer Review: Grants and Cooperative Agreements. Accessed at https://grants.nih.gov/grants/peerreview22713webv2.pdf on June 11, 2019. 30 NIH, About the Office of Extramural Research at NIH. Accessed at https://grants.nih.gov/aboutoer/intro2oer.htm on June 11, 2019. 31 NIH, Query, View, and Report. Accessed at https://archives.nih.gov/asites/era/04- 14-2016/nih_and_grantor_agencies/other/query_view_and_report.cfm on June 11, 2019. 32 NIH, Office of Management Assessment—About OMA Services. Accessed at https://oma.od.nih.gov/Pages/OMA-Services.aspx on June 11, 2019. 33 NIH, Office of Advisory Committee Policy—About Us. Accessed at https://ofacp.od.nih.gov/about_us/index.asp on June 11, 2019. General Services Administration, FACA Management Overview. Accessed at https://www.gsa.gov/policy-regulations/policy/federal-advisory-committee-act- faca-management-overview on June 11, 2109. Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 22 OEI-01-19-00160 34 OIG, As Funding for BPA Increased, NIEHS Followed its Peer Review Process While Also Exercising Its Discretion, OEI-01-15-00150, July 2017. 35 Ibid. 36 NIH, How Scientists are Selected to be Members of a Chartered Review Group. (See Endnote 27 for URL information.) 37 Office of Research Integrity, About ORI. Accessed at https://ori.hhs.gov/about-ori on June 20, 2019. 38 42 CFR pt. 93. 39 See 5 U.S.C. app. §5(c). 40 Testimony of Dr. Lawrence A. Tabak, Principal Deputy Director, NIH, before the Senate Committee on Finance, June 5, 2019. (See Endnote 7 for URL information.) Vetting Peer Reviewers at NIH’s Center for Scientific Review: Strengths and Limitations 23 OEI-01-19-00160 ABOUT THE OFFICE OF INSPECTOR GENERAL The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, as amended, is to protect the integrity of the Department of Health and Human Services (HHS) programs, as well as the health and welfare of beneficiaries served by those programs. This statutory mission is carried out through a nationwide network of audits, investigations, and inspections conducted by the following operating components: Office of Audit The Office of Audit Services (OAS) provides auditing services for HHS, either by conducting audits with its own audit resources or by overseeing Services audit work done by others. Audits examine the performance of HHS programs and/or its grantees and contractors in carrying out their respective responsibilities and are intended to provide independent assessments of HHS programs and operations. These assessments help reduce waste, abuse, and mismanagement and promote economy and efficiency throughout HHS. Office of Evaluation The Office of Evaluation and Inspections (OEI) conducts national evaluations to provide HHS, Congress, and the public with timely, useful, and Inspections and reliable information on significant issues. These evaluations focus on preventing fraud, waste, or abuse and promoting economy, efficiency, and effectiveness of departmental programs. To promote impact, OEI reports also present practical recommendations for improving program operations. Office of The Office of Investigations (OI) conducts criminal, civil, and administrative investigations of fraud and misconduct related to HHS Investigations programs, operations, and beneficiaries. With investigators working in all 50 States and the District of Columbia, OI utilizes its resources by actively coordinating with the Department of Justice and other Federal, State, and local law enforcement authorities. The investigative efforts of OI often lead to criminal convictions, administrative sanctions, and/or civil monetary penalties. Office of Counsel to The Office of Counsel to the Inspector General (OCIG) provides general legal services to OIG, rendering advice and opinions on HHS programs the Inspector and operations and providing all legal support for OIG’s internal General operations. OCIG represents OIG in all civil and administrative fraud and abuse cases involving HHS programs, including False Claims Act, program exclusion, and civil monetary penalty cases. In connection with these cases, OCIG also negotiates and monitors corporate integrity agreements. OCIG renders advisory opinions, issues compliance program guidance, publishes fraud alerts, and provides other guidance to the health care industry concerning the anti-kickback statute and other OIG enforcement authorities.