November, 2018 siepr.stanford.edu Policy Brief Encouraging Edge Science through NIH Funding Practices By Jay Bhattacharya and Mikko Packalen Science moves forward when were novel risked failure, but still The U.S. National Institutes of Health scientists take risks in their work pursued them. plays an important role in addressing and explore new, untested ideas. this failure in the market for scientific Research into novel concepts like For example, one of the hottest new research. With a $37 billion annual targeted cancer therapy might be ideas in cancer treatment involves budget, NIH is the world’s largest called edge science, in contrast using patients’ own immune cells funder of biomedical research. One with studies focused on refining to treat their cancer. New therapies of its explicit missions is to “foster well-established ideas. Without broad based on this idea, such as CAR fundamental creative discoveries and vigorous pursuit of edge science T-cell therapy, have rendered many (and) innovative research strategies” ideas, scientific progress stagnates. types of previously untreatable (National Institutes of Health, 2017). But there is an inherent problem in leukemia treatable. The scientific As a public institution, it can be encouraging such pioneering work: work that led to this breakthrough thought of as “patient capital,” a Research based on unproven ideas happened in the face of decades of funding source with a longtime is risky and prone to failure. That widespread skepticism that targeted horizon and an understanding that makes such research difficult to fund immune therapy for cancer would good ideas frequently lead down and to attract the critical mass of be possible. The researchers who blind alleys. By that standard, NIH scientists needed to develop an idea. worked on these ideas when they ought to be putting money into novel ideas that cannot get funding from private sources. That at least is the theory. But a About the Authors growing number of researchers have Jay Bhattacharya is a SIEPR Senior Fellow, a Professor of questioned whether NIH is doing all Primary Care and Outcomes Research. He is a core faculty it should to support groundbreaking member of Stanford Health Policy. His research focuses on the constraints that vulnerable populations face in making science (Alberts, Kirschner, decisions that affect their health status, as well as the effects Tilghman, and Varmus, 2011; Cook- of government policies and programs designed to benefit Deegan, 1996; Kolata, 2009). For vulnerable populations. reasons of science and politics, NIH Mikko Packalen is an Associate Professor of Economics at the may be subject to a conservative University of Waterloo. His current research examines how demographics, geography, and institutions influence the bias. Because it spends public money, adoption of new ideas by scientists and inventors. it is under pressure to produce visible results—and these can more readily be achieved with incremental Sam Zuckerman contributed editorial assistance to this Policy Brief. advances on proven ideas than with the idea type of each idea (e.g. gene, By contrast, from 1990 to 1999, scientific breakthroughs. Moreover, protein, drug, or diagnostic tool) NIH funded research based on new the senior scientists who serve on and the research area of the journal ideas at a higher rate than it funded NIH grant application review panels in which the article was published research drawing on well-established may favor tried and true approaches (e.g. cardiology, neoplasms, or ideas. This indicates that NIH has they are familiar with and know molecular biology). We categorized become less likely to support edge are publishable instead of more contributions that built on at least science over the past two decades. speculative avenues of investigation one relatively recent idea as novel, Second, NIH’s propensity to support (Joyner, Paneth, and Ioannidis, 2016). while we categorized contributions edge science varies across idea There is also evidence that NIH based on ideas with a longer history types. It is concentrated in a limited increasingly is funding scientists at in the literature as more traditional number of idea types in basic science later stages in their careers, when science. Using this new measure of where many ideas are relatively they are more likely to investigate novelty, we then compared papers novel, such as genomics, proteomics, well-established ideas. that acknowledged NIH funding and the general field of subcellular against those funded by other biology. When it comes to research sources. Measuring NIH Funding of on clinical ideas as opposed to basic Edge Science Our key finding was that NIH funded science, NIH has no preference for To address this question, we edge science at a higher rate than it novelty. Moreover, once we account performed a quantitative analysis funded less-innovative science during for NIH’s disproportionate funding to measure the extent to which the 67-year period we analyzed. of a few idea types that are hotbeds NIH funds novel ideas. Our study But this positive result is subject to of innovation, NIH’s funding rates made use of an innovative method several important qualifications that for edge science and traditional to determine the novelty of each raise questions about the extent of science are about the same. In other NIH’s commitment to edge science. words, NIH disproportionately directs article: a textual analysis of 24 funds to innovative areas of basic million biomedical research articles First, from 2010 to 2016, NIH science, and that explains its overall in the MEDLINE database published disproportionately funded biomedical preference for innovation. If NIH between 1950 and 2017 with an research based neither on the funded basic science research and American first author. An article was most recent ideas nor on the most clinical research at the same rate, considered novel if the newest idea longstanding ideas, but rather its preference for novelty would on which it built upon was relatively on those of intermediate vintage disappear. recent in the sense that the idea had introduced into the literature first appeared in any biomedical between 1990 and 2005. Specifically, These results are disheartening and research paper at most a few years NIH funded research based on 10- to consistent with a growing body of prior. In determining the novelty of 25-year-old ideas at a 55 percent rate scholarship that finds NIH review the ideas that each contribution built compared with a 45 percent funding panels becoming more conservative upon, our analysis also controlled for rate for more recent or older ideas. and risk-averse (Nicholson and 2 November, 2018 Policy Brief Ioannidis, 2012; Li and Agha, 2015; could help NIH carry out its mission when they apply for jobs and seek Li, 2017). This is consistent with of supporting groundbreaking promotion. And it should find ways evidence that suggests biomedical research: of evaluating grantee performance researchers as a group have become based on the novelty of the •NIH should reform the review more cautious in their research work. That implies downgrading process and rethink how review choices in recent years (Joyner, the importance of the number panel members are selected. In Paneth, and Ioannidis, 2016). One of scholarly citations a research particular, policies should be sign that NIH may be backing away project generates, since it may adopted that increase the number from edge science is that its grant take time for the importance of of accomplished younger scientists recipients are older than they used breakthroughs to be recognized. on these committees. to be. In the 1980s, the median age •NIH should change the way it of NIH grantees was in the thirties. Conclusion Today, the median age is in the mid- measures success to increase tolerance of failure. NIH should If the world’s foremost supporter forties. That is a problem because adopt the approach that is standard of biomedical research has indeed younger scientists are more likely in the world of venture capital become less open to edge science, to try out new ideas (Packalen and that many failures are needed in as our analysis indicates, it bodes Bhattacharya, 2017). order to innovate (Peifer, 2017; poorly for science. While our finding Zaringhalam, 2016). is somewhat discouraging, it is not Policy Reforms to Promote necessarily surprising. Some existing Edge Science •NIH should develop ways to studies have found evidence that directly measure the novelty NIH recognizes the danger of the scientific community as a whole of ideas. Currently, NIH grant underfunding high-risk ideas and has has been less disposed to accept proposals that reviewers deem taken a number of steps to counter the newest ideas in the 2000s than innovative are given preference, a creeping conservative bias and it was during 1990s. If this is true, it but assessment of innovation is boost support for novel biomedical is an ominous signal that progress subjective. Review panels should research. These policies include in science—and in medicine in use specific and concrete measures increasing the number of training particular—is in danger of stagnating. of innovation to adjudge proposal. awards, paying bonuses to young Also from this perspective, we Methods like the textual analysis researchers, and developing methods consider it essential that NIH and described in the policy brief may for identifying high-risk ideas. More other funding agencies recommit to be helpful. needs to be done. NIH should supporting work that tries out new embed promotion of groundbreaking •NIH should find additional ways ideas. While such work is risky, it is research more deeply in the grant- to reward scientists working on also essential for scientific progress making process and in monitoring novel ideas by taking steps to as it helps the most fruitful new ideas and evaluating the work it supports. advance their careers. NIH can develop from a germ of an idea to Here are some policy reforms that lend its prestige to these innovators transformative discoveries. 3 References Alberts, B., Kirschner, M.W., Li, D., and L. Agha. (2015) Big Names Packalen, M., and J. Bhattacharya. Tilghman, S., and H. Varmus. (2011) or Big Ideas? Do Peer-Review Panels (2018) Does the NIH Fund Edge Rescuing U.S. Biomedical Research Select the Best Science Proposals? Science? (NBER Working Paper Series from Its Systemic Flaws, Proceedings Science, 348(6233), 343-348. No. 24860) Cambridge, Mass. of the National Academy of Sciences, http://www.nber.org/papers/w24860. Li, D. (2017) Expertise versus Bias in 111(16), 5773-5777. Evaluation: Evidence from the NIH, Peifer, M. (2017) The Argument for Cook-Deegan, R.M. (1996) Does NIH American Economic Journal: Applied Diversifying NIH Grant Portfolio, need a DARPA? Issues in Science and Economics, 9(2), 60-92. Molecular Biology of the Cell, 28, Technology, 13(2), 25-28. 2935-2940. National Institutes of Health Joyner, M.J., Paneth, N., and J.P.A. (2017) Mission and Goals Zaringhalam, M. (2016) Failure in Ioannidis. (2016) What Happens (https://www.nih.gov/aboutnih/ Science Is Frequent and Inevitable— When Underperforming Big Ideas what-we-do/mission-goals). and We Should Talk More about It, Become Entrenched? JAMA. Scientific American Blog (https:// Nicholson, J.M., and J.P.A. Ioannidis. blogs.scientificamerican.com/ Kolata, G. (2009) Grant System Leads (2012) Conform and Be Funded, guest-blog/failure-in-science-is- Cancer Researchers to Play it Safe, Science 492, 34-36. frequent-and-inevitable-and-we- New York Times, June 28. Packalen, M., and J. Bhattacharya. should-talk-more-about-it/, accessed (2017) Age and the Trying Out of October 2016). New Ideas, Journal of Human Capital (forthcoming). About the Stanford Institute for Policy Briefs Location Economic Policy Research SIEPR Policy Briefs summarize research by our John A. and Cynthia Fry Gunn Building We support research that informs economic affiliated faculty and researchers. They reflect 366 Galvez Street policymaking while engaging future leaders the views and ideas of the author only. SIEPR Stanford, CA 94305-6015 and scholars. We share knowledge and build is a nonpartisan research institute. relationships among academics, government Online officials, the business community and the public. For Additional Copies siepr.stanford.edu Please visit SIEPR.stanford.edu @ siepr facebook.com/SIEPR/