February 2018 Sustaining Teen Pregnancy Prevention Programs The Office of Adolescent Health (OAH) in the U.S. Department of Health and Human Services supported a study to understand how programs are sustained after federal funding ends. This brief is the second in a series about the sustainability of OAH grantees and discusses key lessons learned from programs operated by the Teen Pregnancy Prevention (TPP) program whose grant period ended. Former TPP grantees highlighted five lessons related to program sustainability: (1) identify or develop a program that is responsive to the needs of the community, (2) plan ahead for implementation both during and after the grant period, (3) mobilize champions for the program in the community, (4) integrate the program into local institutions, and (5) build the capacity of implementation partners early in the grant period. The purpose of this brief is to share these lessons with current and future grantees so they can use them to build sustainable programs. Lessons Learned from Former OAH Grantees OFFICE OF ADOLESCENT HEALTH Authored by Julia Alamillo, Subuhi Asheer, Betsy Keating and Jean Knab for the Office of Adolescent Health, under contract number HHSP233201300416G Introduction after the TPP grant period, either on their own or with assistance from partner organizations. The others that Program practitioners and funders share a common sustained their TPP programs no longer played any role goal to develop programs that are sustainable beyond in the program, but reported that other agencies, usually a particular funding period. Both parties must devote former partners on the TPP grant, continued to deliver considerable time and effort to plan how to continue the program. Nearly all of the grantees that sustained their delivering services after the grant ends. Planning ahead for program experienced some changes to their programs after sustainability is critical for ensuring the long-term success the TPP grant ended. Among the 28 former grantees that of a program. The Office of Adolescent Health (OAH) sustained, 15 scaled-back the scope of their programs, in the U.S. Department of Health and Human Services eliminating some program components and/or serving is committed to supporting the success of the programs it fewer youth, to continue at a lower level of funding. In funds by working with former grantees to understand and contrast, 10 grantees scaled-up the scope of their programs, share best practices for sustainability (OAH’s efforts to adding new components and/or serving more youth, after support program sustainability, p. 6). their TPP grant ended. In addition, 10 grantees changed This brief is the second in a series about the sustainability of the target population that they served and 11 changed the former OAH grantees. The first described the experiences of setting where their programs were implemented. former Pregnancy Assistance Fund (PAF) grantees funded Former TPP grantees that sustained their programs after from 2010 to 2013 (Asheer et al. 2017; https://www.hhs. the federal grant period highlighted five key strategies gov/ash/oah/sites/default/files/paf-brief-sustainability-study. that enhanced sustainability: (1) identify or develop a pdf ). This brief offers lessons based on the experiences of program that is responsive to the needs of the community, former Teen Pregnancy Prevention (TPP) grantees funded (2) plan ahead for implementation both during and after from 2010 to 2015. TPP grantees were funded at one of two the grant period, (3) mobilize champions for the program tiers: Tier 1 grantees replicated evidence-based programs in the community, (4) integrate the program into local and Tier 2 grantees implemented new or innovative institutions, and (5) build the capacity of implementation programs. Of the 94 grantees that received TPP funding in partners early in the grant period. Grantees that were 2010, 64 were not awarded funding in the competitive cycle unable to sustain their programs reported several for 2015. Findings in this brief are based on interviews and a challenges implementing these strategies, such as issues review of administrative documents from 27 Tier 1 grantees with the fit and feasibility of their chosen program model, and 10 Tier 2 grantees that agreed to participate in OAH's the amount of support for their program in the community, sustainability study (Figure 1). the timing of sustainability planning, and the capacity Of the former grantees in the study, 70 percent of Tier of their partner organizations (Sustainability Do’s and 1 grantees and 90 percent of Tier 2 grantees sustained Don’ts, end of brief ). Grantees’ ability to overcome these all or part of their TPP programs after the federal grant challenges and ultimately sustain their programs depended period ended (Figure 2). Most of the former grantees that on a number of factors, including specific features of their sustained continued to operate the program themselves organizational and implementation contexts. About the Office of Adolescent Health’s Teen Pregnancy Prevention program OAH leads the Teen Pregnancy Prevention (TPP) program, established in 2010 to fund diverse programs working to prevent teen pregnancy across the United States. The OAH TPP program seeks to prevent pregnancy and associated sexual risk behaviors in youth ages 10 to 19. The program funds competitive grants totaling $100 million each year across two funding tiers: Tier 1 grants support the implementation and evaluation of evidence-based programs and Tier 2 grants support the development and evaluation of new and innovative programs. In 2010, OAH awarded the first cohort of 94 grantees (75 Tier 1 grantees and 19 Tier 2 grantees) for a five-year project period. Tier 1 grantees in 2010 could choose from a list of 28 program models identified by the U.S. Department of Health and Human Services Pregnancy Prevention Evidence Review. Grantees consisted of a mix of state and local agencies, including health departments, public school districts, universities, health clinics, and other community-based organizations. 2 Figure 1: Characteristics of former TPP grantees that participated in the study 17 8 5 5 2 community-based universities state health health clinics school districts organizations departments 27 37 Grantees 10 Tier 1 Tier 2 Grantees implemented their TPP programs in one or more settingsa 22 atorganizations community-based 19 in schools 4 in health clinics 4 online Grantees delivered programs themselves or with other implementing agencies 21 delivered all program components themselves 16 delivered program components using other implementing agencies Grantees served youth of different ages and with diverse characteristics 6 served middle- schoolers only 7 served high- schoolers only 22 served middle- and high-schoolers 1 served older youth only 1 served youth of all ages Some grantees also worked with vulnerable populations, including foster youth (5 grantees), youth in the juvenile justice system (1 grantee), or pregnant/parenting youth (1 grantee). a The number of grantees in this row is greater than 37 because several implemented their program in more than one setting. Figure 2: How 28 of these 37 former TPP grantees sustained their programs 28 grantees 19 that participated in the study 9 Tier 1 sustained all or part of their TPP program Tier 2 19 continued to operate the 15 scaled-back their program 10 changed their target 11 changed their program themselves by reducing program population implementation setting components or the number 9 reported that other of youth served 14 kept the same target 13 kept the same organizations had population implementation setting sustained the program 6 scaled-up their program by increasing program components or the number of youth served 3 kept the same scale 4 grantees did not know what their former TPP program looked like at the time of the interview and therefore are not counted in these columns 3 Lesson 1 Identify or develop a strong program model that is responsive to your community’s needs Programs that demonstrate direct relevance to the youth programs tailored to their unique needs. Service providers and families they serve are more likely to be sustained often embraced these programs because they resonated beyond a single funding cycle than programs that do not. It with youth in a way that other sexual health offerings did is important to ensure that the core content of the program not. This may have contributed to the higher sustainability addresses the needs of the target population. In addition, rates of Tier 2 programs at the end of the TPP grant grantees emphasized that a sustainable program model must period because these programs were developed with a also resonate with the backgrounds, values, and experiences of particular population, setting, and context in mind. youth and their families (see also Farb and Margolis 2016). Tier 2 grantees that worked with vulnerable populations, For example, several Tier 1 grantees decided to adapt such as foster youth or youth in the juvenile justice elements of their chosen curriculum to fit the particular system, reported that their communities often especially racial and ethnic or cultural backgrounds of their students. valued the programs they developed because these One grantee located in Hawaii produced new videos programs directly addressed the difficult circumstances for its program containing local actors, settings, and of their participants. storylines. The grantee worked with its partners to make these adaptations so the content would be consistent Despite the benefits of tailoring the program to fit across sites and engaging to all youth enrolled in the the target population, it was a challenge for grantees program. Several grantees also discussed the importance to balance the desire to make adaptations while also of sensitivity to local values and perceptions about sex maintaining fidelity to the program model. For example, education and teen pregnancy prevention programs. Many one grantee wanted to add elements to its chosen chose to implement programs that provided a flexible and curriculum to make it more relevant for tribal youth, value-neutral approach to teaching youth about sex and such as information about coming of age in the tribal healthy relationships. As one grantee explained about the community. But the grantee decided against doing so benefits of flexible programming, “I’ve watched young because it believed the modifications would reduce fidelity people go through programs that are less adaptable and and undermine the replication study. This grantee, which where people have to [explain] things a certain way. I’ve ultimately was unable to sustain its TPP program, felt watched [facilitators] stick rigidly to the way something that local schools and youth-serving organizations might was written, and I’ve watched them fail for some groups of have been more enthusiastic about adopting the program kids. There’s no such thing as one size fits all.” if adjustments had been made to ensure that the program resonated with participants. Some grantees overcame this In developing new, innovative programs, Tier 2 grantees challenge by working directly with program developers to also said they spent a substantial amount of time working make small adaptations to their program, such as replacing with local stakeholders, organizations, and youth to create outdated examples in the curriculum with new content. 4 Lesson Plan ahead so the program can be 2 implemented both during and after the grant period Former grantees that carefully considered their potential implementation contexts before choosing or developing Grantee Spotlight 1. One health clinic a program were more likely to continue operating after used its 2010 TPP grant to deliver the Carrera the federal grant period, than former grantees that only program to students in two school districts. focused on their immediate implementation needs. One Students attended the program 5 days a week strategy former grantees used was to select a flexible after school and over the summer, and also program model that could be delivered in several different participated in field trips and internships in the settings. For instance, some former grantees that had community. After the grant ended, the grantee initially planned to work only in schools ended up referred students to similar services in schools implementing their programs in several settings, including and the community, but was unable to sustain the during the regular school day, after school, and on the Carrera program. In retrospect, the grantee felt weekends; choosing a curriculum that could accommodate they began planning for sustainability too late in this level of flexibility was essential for recruiting sites the grant period, and without the federal grant, the to participate in the program during and after the TPP program was too costly to implement with fidelity. grant period. Similarly, former grantees that delivered Although they took a number of steps to identify their programs in schools found that schools were more additional funding after 2015, including developing willing to work with them if they used a curriculum with marketing materials, contacting state and local lessons that could be adapted to accommodate different representatives, and applying for new grants, school schedules. Several grantees suggested that future ultimately these activities occurred too late in the technical assistance efforts should address how to increase grant period to bring in the necessary revenue. a curriculum’s flexibility to market it more broadly. Based on this experience, they recommended that future grantees hire a staff member to focus Grantees found that building flexibility into their on sustainability from the beginning of the grant curriculum so it could be implemented across a variety period. They also recommended carefully selecting of settings made it more appealing to potential users. a program model that is feasible to implement even if funding is reduced. Although it may seem Former grantees were also more likely to sustain their appealing to deliver a highly intensive program after programs if they chose curricula they could adjust or scale- being awarded a large grant, the community may back in times of less funding. Most found it easier to actually benefit more from a less intensive program sustain programs that could be delivered within a limited that is easier to sustain after the grant period ends. timeframe, because they generally had more inexpensive licensing fees, were less costly to staff, and could more easily be continued after the TPP grant. However, a few community service retreats, after the end of its TPP grant. former grantees found ways to sustain more intensive The former grantee received another, smaller grant from programs while maintaining fidelity to the core program the state department of health that made it possible to model by scaling back non-essential components. For continue delivering TOP’s core components. Likewise, example, one former grantee that implemented the Teen a former grantee that implemented Carrera—a highly Outreach Program (TOP)—a nine-month program intensive, multiyear program that offers services six days consisting of weekly group lessons and 20 hours of per week—dropped the mental health and job-training community service—decided to serve fewer students and portions of the program, but used new funding from the discontinue optional components, such as field trips and state to continue the rest of the program components. 5 In contrast, some former grantees that implemented sustainability earlier in the grant period. As one grantee intensive program models found them to be too long or that did not sustain explained, “[Our] sustainability costly to sustain after the grant period without sacrificing planning and action planning should’ve taken place much fidelity to the core program model (Grantee Spotlight earlier.… When you get a five-year grant, year five seems 1). The length and cost of these programs also made like it’s far off, but it comes very fast.” Early planning it difficult to find partners to take over the program. was particularly essential for figuring out how to sustain Although schools and local organizations often valued more intensive programs. Although determining which these intensive programs and were grateful to receive components to retain and which to cut can be difficult, services from grantees, most lacked the time and financial grantees that planned ahead and either chose shorter resources to continue the program themselves. In general, programs or found ways to scale-back more intensive former grantees that could not sustain their programs ones were much more likely to sustain after the federal reported that they should have started planning for grant period. OAH’s efforts to support program sustainability OAH defines a sustained program as one in which organizations “effectively leverage partnerships and resources to continue programs, services, and/or strategic activities that result in improvements in the health and well-being of adolescents.” Based on grantees’ early experiences and input from experts, in 2011 OAH developed a framework and toolkit to support grantees in developing sustainable programs and creating sustainable impacts. In 2017, OAH updated the framework and toolkit, which includes a new planning template, a resource guide, and an e-learning module (https://www.hhs.gov/ash/oah/resources- and-training/tpp-and-paf-resources/community-mobilization-and-sustainability/index.html). In 2015, OAH launched a three-year study to better understand program sustainability after federal funding ends. The study focused on programs designed to prevent or delay teen pregnancy and examined whether and how grantees sustained programming. Two OAH initiatives in 2010 initially funded the programs or services in the study: the Teen Pregnancy Prevention (TPP) program and the Pregnancy Assistance Fund (PAF) program. Of the 111 TPP and PAF grantees that received funds in 2010, 71 (more than 60 percent) did not receive competitive funding in the second round, either because they did not reapply or their application was not funded. The sustainability study consisted of a review of grantees’ documents and up to two rounds of interviews with 43 former TPP and PAF grantees. In addition to the lessons discussed in this brief and an earlier brief (Asheer et al. 2017), three case studies will highlight successful strategies to help inform current and future efforts to sustain federally funded programs after funding ends. 6 Lesson 3 Mobilize local partners, beneficiaries, and champions for your program Former TPP grantees that sustained their programs the need for teen pregnancy prevention programming worked hard to gather buy-in from community members in their schools. Providing additional context and before and during the TPP grant period. Before applying highlighting the broader youth development aspects of the for the grant, one grantee held a series of town hall program garnered more support from potential partners. meetings with community members, school staff, parents, One grantee developed a script for staff to use when and teens to discuss their needs related to preventing discussing sexual education with school and community teen pregnancy and what their program should offer. groups to facilitate these conversations. For grantees that Several other grantees offered open houses for parents and faced resistance to providing teen pregnancy prevention community members during the grant period to answer services, meeting regularly with local school boards or questions and provide information about what participants other government officials helped them develop trust and would learn during the program. Former grantees also overcome community resistance. Grantees often found it discussed the importance of having program participants, helpful to use participants’ success stories to demonstrate parents, and staff raise awareness for their program in the the benefits of their programs. community. Many had community outreach specialists and participants attend community events, such as parades Turnover of key supporters in the community was also a or festivals, to advertise the program. Several organizations challenge for sustainability. One former grantee reported even developed logos and branding for their TPP that even though the school district in which it delivered programs so they would be visible in the community. services embraced its program at first, midway through the grant period the district’s superintendent and director Community members and organizations were more of health programming were replaced by individuals who likely to invest time and resources to sustain programs were not interested in sustaining the program. As a result, that had demonstrated local support. the former grantee was unable to continue operating its program in the school district after the TPP grant Former grantees that sustained also built community trust period ended. To mitigate the negative effects of turnover, by framing their programs in a way that would appeal grantees suggested encouraging key supporters to discuss to partners and stakeholders. Some grantees found that their endorsement of the program before they leave. school and community leaders were initially unsure about 7 Lesson Integrate the program into local 4 institutions such as schools and community-based organizations Former grantees that sustained their programs worked Grantees that aligned their programs with state standards closely with institutions and service providers in the often reported that it was the most influential step they community to strategize ways to continue all or part took to ensure their programs would be sustained. of their programs without former grantees’ direct involvement. Those that delivered programming in schools Grantees that delivered services outside of a school setting were especially well-positioned to integrate components also worked to integrate components of their programs of their programs into the regular school day. For example, into local institutions, including community-based one former grantee operated its program in a school organizations (CBOs) and businesses. Several grantees district that received funding from the Centers for Disease sustained all or part of their TPP programs through Boys Control and Prevention (CDC) to develop its own & Girls Clubs, YMCAs, or similar organizations. Former comprehensive sexual education curriculum for middle and grantees generally found it easier to approach CBOs high school students. Staff from the grantee organization about delivering elements of the TPP program if they worked with school personnel to develop the new were implementation partners on the grant, but in some curriculum, which ultimately included several elements cases former grantees worked with new sites to deliver from the TPP program evaluated under the grant. teen pregnancy prevention services after the grant period. Other former grantees worked with local businesses and Demonstrating alignment with state education standards nonprofits to continue offering youth development and was also critical for institutionalizing programs in schools. community service opportunities to the target population. During the grant period, several Tier 1 grantees modified For instance, one grantee that worked with youth in portions of their chosen program models to ensure they the foster care system established a community service met state standards for health and sexual education, network as part of its program. After the TPP grant thereby making it easier for schools and districts to ended, businesses and nonprofits in the network saw great adopt the programs. Many Tier 2 grantees also devoted value in continuing to offer service learning opportunities significant time to learning state education requirements to foster youth and became strong supporters of the and tailoring their curricula to meet these standards. grantee organization. Although this grantee did not sustain its pregnancy prevention programming, the TPP grant helped it build local support and integrate other important services into the community. 8 Lesson Build the capacity of implementing 5 agencies to deliver programming early in the grant period A final strategy grantees used to sustain their programs opportunities with these agencies during and after the beyond the TPP grant period was to directly train TPP grant period. As a result, three of the four agencies implementing agency staff to deliver the program that implemented the TPP program continued to offer themselves. Although the majority of grantees delivered services after the grant period ended. their programs on their own during the TPP grant period, others worked with staff at partner organizations, such as Grantees that worked with their implementation partners schools and CBOs, to implement the program. In some to develop sustainability plans were more likely to sustain cases, grantee staff implemented the program alongside after the federal grant period. partner staff, whereas in other cases, partner staff delivered the entire TPP program on their own. Some grantees also For some grantees, the implementation setting and staff offered training-of-trainer opportunities during the grant turnover posed significant challenges to building partners’ period so implementing agencies could continue to train capacity to sustain programming. Grantees that delivered new staff and keep the program going without grantees’ their programs in rural settings, for instance, sometimes continued presence. In general, this strategy of training struggled to offer adequate support to staff at their staff at partner sites to deliver the program worked best if implementation sites. High transportation costs and the grantees started these activities early in the grant period. time needed for travel posed challenges for training and Grantees that approached their implementation partners monitoring implementation staff in rural locations. One about training opportunities near the end of the TPP grant former grantee overcame these challenges by developing period were much less likely to find staff at these agencies to an online facilitator training to train staff to deliver the continue the program. program after the TPP grant period (Grantee Spotlight 2). High levels of turnover at partner organizations, particularly Grantees also strengthened the capacity of their at CBOs, was another common concern. Several grantees implementation partners by leading regular discussions devoted significant time and money to training their about sustainability throughout the grant period. For implementation partners, only to have key staff members example, one state health department reviewed OAH’s leave these organizations. Offering training-of-trainer sustainability toolkit with each of its four implementing opportunities for partners was one strategy grantees used to agencies to develop sustainability action plans tailored ensure the continued success of TPP programs in high- to each site. This grantee also shared relevant funding turnover settings. Grantee Spotlight 2. One grantee located in Hawaii that implemented its TPP program in schools, sustained the program by devising an innovative and cost-effective strategy to train teachers to deliver the curriculum. During the grant period, the grantee conducted in-person trainings for teachers delivering the program, as well as site monitoring visits to ensure the program was implemented with fidelity. However, because the schools delivering the program were geographically isolated on different islands, these trainings and visits were costly and ultimately not sustainable after the TPP grant ended. To overcome this challenge, the grantee developed an online facilitator training course and website containing resources for schools to use as they implement the program. The training is free and consists of a series of lessons and quizzes that teachers must pass to become certified to teach the curriculum. Ultimately, this strategy helped the grantee expand the reach of its program after the TPP grant period ended. 9 Conclusion Fourth, for a program to be sustainable in the long term, it is helpful to integrate it into local institutions, Sustaining a program with evolving implementation needs such as schools or CBOs. Former grantees reported and limited funding is a difficult task in any context. Federal that schools and districts were more likely to continue grants can bolster an organization’s capacity and provide programs that met their needs and were valuable for their support for program expansion for only a relatively short communities. Grantees found it especially helpful to align time. However, to continue serving their communities, their programs with school or district requirements for grantees must plan and take steps to sustain their programs, sexual health education. Finally, building the capacity in some cases before they even apply for the grant. of partner staff is critical so other agencies can continue implementing the program on their own. Successful Former TPP grantees who participated in this study grantees began building capacity early in the grant period highlighted five important strategies for sustainability, by training school and CBO staff, and by developing as well as some common pitfalls to avoid. First, the sustainability plans in collaboration with partner agencies. program must fit the target community’s needs. Whether they chose an existing program or developed a new one, former grantees found it vital to assess the local References context and tailor the program to ensure relevance for its Asheer, Subuhi, Julia Alamillo, Betsy Keating, and Jean beneficiaries. Second, grantees must think carefully about Knab. "Sustaining Programs for Expectant and Parenting what is feasible for them and their partners to implement, Teens: Lessons Learned from Former Pregnancy both during and after the federal grant period. Former Assistance Fund Grantees." Brief submitted to the U.S. grantees that sustained their programs considered all Department of Health and Human Services, Office of relevant logistics and chose a program they could adjust to Adolescent Health. Washington, DC: Mathematica Policy changing implementation and funding constraints. Third, Research, May 2017. grantees should cultivate and mobilize local champions, beneficiaries, and community leaders to build support for Farb, Amy Feldman and Amy L. Margolis. "The Teen program sustainability. Engaging with and inviting input Pregnancy Prevention Program (2010-2015): Synthesis of from youth and their families helped former TPP grantees Impact Findings." American Journal of Public Health, vol. develop trust and address community concerns directly. 106, no. S1, 2016, pp. S9-S15. 10 FEBRUARY 2018 SUSTAINABILITY DO’S AND DON’TS In 2015, the Office of Adolescent Health (OAH) in the U.S. Department of Health and Human Services launched a study designed to better understand whether and how programs supported by federal funding were sustained after their grant funding ends. Our second brief (Sustaining Teen Pregnancy Prevention Programs) highlights lessons and challenges informed by former OAH Teen Pregnancy Prevention (TPP) grantees’ sustainability efforts. The Do’s and Don’ts described here summarize the lessons in the Brief and could be useful for future grantees in planning for sustainability beyond the federal grant period. 1. DO identify or develop a strong program model 1. DON’T choose a program that is a poor match that is responsive to your community’s needs. Pro- for your target population and implementation grams that resonate with the backgrounds, values, setting. Programs that do not fit the needs of the and experiences of the target population are more target population or the implementation setting likely to be sustained. are less likely to be sustained. Before choosing a program, it is important to consider what your 2. DO plan ahead so the program can be population and setting may look like both during implemented during and after the grant period. and after the grant period. Before you choose a program, assess the flexibility of your chosen program model. It is easier to sustain 2. DON’T get discouraged by community resis- programs that can be implemented in a variety of tance to your program. Although community settings and scaled-back if funding is reduced. resistance can make sustainability challenging, it is important to work through this issue by culti- 3. DO mobilize local partners, beneficiaries, and vating program champions and collaborating with champions for your program. Outreach events participants and key stakeholders to speak-out in and regular communication with key stakeholders support of your program. in the community can build support for sustaining programs, especially when new or alternate funding 3. DON’T wait until it’s too late to begin integrating is required. your program into local institutions. Be proactive and work with implementing organizations 4. DO integrate the program into local institutions such throughout the grant period to figure out how to as schools and community-based organizations. sustain key program components. Work with implementation partners early in the grant period to devise a plan for continuing, at minimum, 4. DON’T assume implementing agencies have the core components of your program after the the capacity and resources to continue your grant ends. program. Limited resources and high levels of turnover at partner organizations are common 5. DO start building the capacity of implementing obstacles to sustainability. agencies early in the grant period. Train staff at part- ner organizations to deliver the program, offer training- of-trainer opportunities, and lead regular discussions about sustainability in order to build partners’ capacity to continue your program. OFFICE OF Office of Adolescent Health | Website: http://www.hhs.gov/ash/oah/ Email: oah.gov@hhs.gov Phone: (240) 453-2846 ADOLESCENT HEALTH