Contract Number: Growing Pains: Lessons from HHSP23320082911YC WAIT Training in Florida Mathematica Reference Number: 06549.097 Submitted to: Office of Adolescent Health Department of Health & Human Services 1101 Wootton Parkway, Suite 700 Implementation Report Rockville, MD 20852 Project Officer: Amy Farb September 4, 2014 Submitted by: Mathematica Policy Research P.O. Box 2393 Betsy Keating Princeton, NJ 08543-2393 Rachel Shapiro Telephone: (609) 799-3535 Facsimile: (609) 799-0005 Project Director: Brian Goesling This page has been left blank for double-sided copying. Growing Pains: Lessons from WAIT Training in Florida CONTENTS INTRODUCTION ............................................................................................................. 1 THE CONTEXT AND BACKGROUND OF WAIT TRAINING .......................................... 3 NEW OPPORTUNITIES AND CHALLENGES: LIVE THE LIFE’S PARTICIPATION IN THE FEDERAL TEEN PREGNANCY PREVENTION GRANT PROGRAM ........................................................................................................ 5 LtL Developed Plans to Deliver a Broad-Based, Replicable Program ............. 5 LtL’s Participation as a Site in Federal Evaluation Further Shaped Program Delivery ....................................................................................... 6 After Developing the Program Delivery Plan, LtL Recruited Willing Schools ...................................................................................................... 8 IMPLEMENTATION OF WAIT TRAINING: SCHOOL RECRUITMENT, TEACHER TRAINING, AND PROGRAM DELIVERY ..................................................... 9 LtL Encountered Challenges in Forging Bonds in New Territory and Maintaining Interest at Overburdened Schools .......................................... 9 These Challenges Led to Prolonged School Recruitment and Outreach .................................................................................................. 10 LtL Staff’s Roles in Implementing WAIT Training Focused on Teacher Training ...................................................................................... 10 Teachers and External Facilitators Adapted the Program Model to Fit Their Schedules .................................................................................. 13 LtL Was Unable to Achieve the Planned Monitoring Levels ........................... 13 Lack of Consistent Program Monitoring and Support Might Have Contributed to Teachers’ and Students’ Discomfort with the Material .................................................................................................... 14 iii Growing Pains: Lessons from WAIT Training in Florida LESSONS FOR THE FIELD AND FUNDERS ............................................................... 15 Extending Program Activity to New Sites Brings Role Changes and Adjustments ............................................................................................. 15 Broadening Program Delivery Requires Devoting Time to Building Relationships ........................................................................................... 15 Schools Might View Outside Programming as a Low Priority and a Distraction ................................................................................................ 16 Funders and Program Operators Should Anticipate the Need for Repeated Teacher Trainings to Ensure Successful Program Implementation ........................................................................................ 16 REFERENCES .............................................................................................................. 19 APPENDIX A: IMPLEMENTATION STUDY DATA SOURCES AND METHODOLOGY ............................................................................. A.1 APPENDIX B: SUMMARY OF WAIT TRAINING LESSONS........................................ B.1 APPENDIX C: SAMPLE CHARACTERISTICS ............................................................ C.1 iv Growing Pains: Lessons from WAIT Training in Florida TABLES 1 WAIT Training Units and Objectives................................................................. 3 2 Number of Teachers Trained and Reasons for Turnover ............................... 11 A.1 Data Sources ................................................................................................ A.4 B.1 Summary of WAIT Training Lessons ............................................................. B.3 C.1 Sample Characteristics ................................................................................. C.3 FIGURES 1 Levels of Risky Behavior Among Program Participants ................................... 8 v This page has been left blank for double-sided copying. Growing Pains: Lessons from WAIT Training in Florida INTRODUCTION Nonprofit social service organizations sometimes face a dilemma when they consider whether to embrace opportunities to extend programming to new locations. Organizations often rely on the support of outside funders to broaden or alter their programs. Grants that enable programs to implement and test programs on a wider scale can be attractive for several reasons. First, they can help organizations to scale up their mission and reach a broader population. Second, they can strengthen organizations by giving them the resources to recruit and retain new staff. However, increasing the breadth of a program brings challenges. It is often hard for organizations to gauge whether there is real demand that warrants an extension of program services. They can find it challenging to manage growth, whether that means adding staff, expanding geographic reach, or both. As organizations add staff, they encounter more complicated personnel issues regarding administration and supervision. Expanding the geographic scope of a program might entail changing program logistics to serve new populations. Furthermore, it can be difficult for organizations to identify new partners and build the broader stakeholder network needed to support broader program implementation. This report examines these common challenges through the case of Live the Life Ministries (LtL). LtL is a faith-based organization that provides social services to youth and families in the Florida Panhandle. In 2010, LtL received a grant from the Office of Adolescent Health (OAH) in the U.S. Department of Health and Human Services (HHS) to implement and test an adapted version of the WAIT (Why Am I Tempted) Training abstinence education program in Florida middle and high schools. The grant also provided resources to evaluate the program’s implementation and impacts, through a partnership with an ongoing federal evaluation. Because of factors ranging from staffing constraints to organizational inexperience, LtL’s project did not go as planned. This report examines these issues in the context of the WAIT Training program to highlight implications for funders and organizations looking to deliver similar programs in other contexts. WAIT Training Implementation and Evaluation—A Snapshot • Part of the national multiyear Evaluation of Adolescent Pregnancy Prevention Approaches (PPA): - Funded by the Office of Adolescent Health, U.S. Department of Health and Human Services - Conducted by Mathematica Policy Research, with Child Trends and Twin Peaks Partners, LLC - Assessing effectiveness of seven programs, including WAIT Training and six others • 13 schools recruited and randomly assigned—7 to a program group that received WAIT Training and 6 to a control group that received a diet and exercise program • Eight one- hour sessions presented on consecutive days in April and May 2012 and again in 2013 by trained teachers to classes of youth, segregated by gender, in contrast to the original program, which offers over 100 activities that can be implemented daily for nine weeks • Topics covered: characteristics of a healthy relationship, cultural influences, differences between males and females, consequences of premarital sex, sexual refusal skills and conflict resolution, and the value of marriage • Impact study suspended before the completion of survey data collection 1 Growing Pains: Lessons from WAIT Training in Florida This analysis of LtL’s implementation of WAIT Training is part of the larger federal Evaluation of Adolescent Pregnancy Prevention Approaches (PPA). PPA is a national OAH-funded evaluation to study the effectiveness of various teen pregnancy prevention approaches in seven sites. The evaluation aims to provide rigorous evidence about impacts, document program implementation, and generate insights about the successes and challenges of program delivery. In each site, Mathematica Policy Research planned for both an implementation study and a random assignment program impact study. This report presents findings from the WAIT Training implementation study and draws on various data collected by Mathematica: observations of program delivery (in spring 2012 and 2013), interviews with LtL staff and teachers delivering the program (in spring 2012 and 2013), and focus groups with teachers and students participating in the program (in spring 2012). The observations, interviews, and focus groups were with a nonrepresentative sample of participating schools. (Appendix A provides details on data sources and methodology for the implementation study.) Although the accompanying impact study was ultimately suspended before the completion of survey data collection, Mathematica’s collaboration with LtL yielded important lessons for program practitioners and funders. This implementation study thus serves as the main documentation of the WAIT Training site in the PPA study. 2 Growing Pains: Lessons from WAIT Training in Florida THE CONTEXT AND BACKGROUND OF WAIT TRAINING LtL, a nonprofit organization based in Tallahassee, Florida, works to strengthen families and marriages. LtL provides coaching and counseling programs for students and married and premarital couples. The organization offers a variety of youth-oriented programs to foster healthy habits and youth development, from retreats and alternative spring break trips to one-on-one coaching sessions and facilitator-led classes. The WAIT Training program aligns closely with LtL’s organizational mission. WAIT Training is an abstinence program for middle and high school students created by the Center for Relationship Education (CRE). CRE developed the curriculum in 1996 and trains and certifies individuals to implement the program across the country. The curriculum provides students with information on behaviors and skills the developer deems important for healthy relationships, stressing abstinence until marriage as the most important way to avoid potential life pitfalls, such as sexually transmitted infections (STIs) and pregnancy. The curriculum emphasizes that youth should define goals and focus on achieving them. The WAIT Training curriculum, as originally developed, has more than 100 exercises organized around seven units designed to enable students to learn about (1) themselves and their peers; (2) characteristics of a healthy relationship; (3) cultural influences; (4) differences between males and females; (5) consequences of premarital sex; (6) sexual refusal skills and conflict resolution; and (7) the value of marriage (Table 1). The WAIT Training curriculum can be taught in versions of varying length and depth. The entire curriculum, with all activities included, would be taught every day for nine weeks. CRE also approves abbreviated versions (for example, 5-, 8-, or 10- day lesson plans) of the curriculum created by other organizations or individuals. Table 1. WAIT Training Units and Objectives Curriculum Unit Objective To help students learn about themselves and their peers; discover their Learning About Yourself and own skills, talents, hopes, and dreams; and understand and respect Others individualism and diversity To teach students the characteristics of a healthy relationship and how to Friendship, Dating, and Love develop healthy, fulfilling, satisfying, and committed relationships To educate students about various cultural influences (for example, Cultural Influences media and alcohol consumption) To educate students about the physical, emotional, and behavioral Differences Between Men differences between men and women, and improve communication and Women between the genders To help students understand the physical, emotional, and financial Consequences of Teen Sex ramifications of premarital sexual activity Dealing Effectively with To help students learn and practice sexual refusal skills and conflict Conflict resolution To increase students’ value of the institution of marriage and help them understand the benefits of being and staying married, as well as the Commitment and Marriage many negative health and social outcomes associated with fractured or never-married families Source: The Center for Relationship Education. “WAIT Training Community Curriculum.” Denver, CO: CRE, November 2010. Prior to receiving the OAH grant in 2010, LtL developed some initial experience providing WAIT Training in local schools. From 2006 to 2010, LtL worked at different times to provide WAIT Training in 14 schools in eight Florida counties, with funding available in those years from the federal Community-Based Abstinence Education program. During these initial efforts, LtL worked with 3 Growing Pains: Lessons from WAIT Training in Florida schools to deliver an abbreviated version of the curriculum over 5 to 10 school days. Schools taught the program in one grade level—typically 7th, 8th, or 9th grade. WAIT Training was implemented at a different time of year in each school, so LtL could focus its resources on one school at a time. LtL staff worked closely with each school to identify and train interested teachers, develop a customized lesson plan, and monitor program implementation. Schools implemented different lesson plans across varying schedules, so the number and intensity of activities they implemented varied. During this period, LtL supplemented the school-based WAIT Training classes with several enrichment activities outside the classroom for students, parents, and the community. LtL sponsored an essay contest, in which students wrote about the importance of their father or a father figure (for example, a stepfather, uncle, or coach) in their lives, to help students understand that fathers should play a key role in their lives. LtL also held quarterly seminars for parents in which staff discussed improving parent-child relationships. Through these activities, LtL hoped to maintain contact with participating students and to involve students at other schools and the larger community in the program. In addition, LtL hoped to strengthen familial bonds across the community. 4 Growing Pains: Lessons from WAIT Training in Florida NEW OPPORTUNITIES AND CHALLENGES: LIVE THE LIFE’S PARTICIPATION IN THE FEDERAL TEEN PREGNANCY PREVENTION GRANT PROGRAM Building on its initial experience, in 2010, LtL submitted a proposal for and received an OAH Tier 2 Teen Pregnancy Prevention (TPP) grant to support the implementation and evaluation of WAIT Training. OAH funded the Tier 2 TPP grants as research and demonstration awards to implement and test new approaches to teen pregnancy prevention (HHS 2010). The funding went specifically to support “untested” programs that did not have existing research evidence of effectiveness. As a part of the funding requirements, grantees were required to assess the impact of their programs through rigorous evaluations and to document lessons from program implementation. OAH’s ultimate goal was to identify new programs that were effective in achieving the desired impact on the target population, and could be considered for broader dissemination in other communities. LtL was to use the TPP grant to test an adapted version of WAIT Training. LtL Developed Plans to Deliver a Broad-Based, Replicable Program Consistent with the overarching goals and purposes of the OAH grants, LtL generated plans to deliver a fully developed, replicable program that, if shown to be successful in improving youth outcomes, could later be packaged and implemented in other locations. LtL planned broad-based implementation across a large number of schools, development of a standardized curriculum to be used sequentially across three grade levels (7th, 8th, and 9th grades), and a formalized and systematic documentation of program delivery. Implementation on a broad scale. Grant funds enabled LtL to provide WAIT Training simultaneously across multiple schools. All schools delivering programming under the grant would do so during the same period. This delivery period was determined by the recruitment targets LtL set for itself in its implementation plan and school schedules, which had limited time available for program delivery. The schools would deliver eight one-hour sessions in the spring of 7th, 8th, and 9th grades. The administrators and selected teachers at each school would determine the exact timing of the program delivery, after assessing when the program would best fit into the school schedule during those months. Use of a standardized curriculum over three years. LtL proposed delivering the WAIT Training program over three years, from 7th through 9th grade. All schools would follow the same lesson plan for delivery over eight days. LtL developed, and CRE approved, lesson plans for delivery in 7th and 8th grades. (LtL also intended to develop a 9th grade curriculum; however, it was never fully developed because the program ended before students reached 9th grade.) With CRE’s guidance, LtL selected activities for each lesson plan from the larger WAIT Training curriculum; schools would not select the activities. CRE reviewed the activities that LtL selected to ensure that they adequately addressed each of the seven units of the curriculum. Each lesson was composed of written assignments, group work, and interactive activities such as role play. (Appendix B, Table B.1 provides a detailed summary of the lessons in the 7th- and 8th-grade lesson plans.) The lesson plans at the 7th-, 8th-, and 9th-grade levels were to address the same core units of WAIT Training, but be tailored by LtL to the expected student maturity level in each grade. The adapted 7th-grade curriculum broadly focused on setting future goals, having healthy relationships, and understanding the consequences of having sex as teenagers. The 8th-grade curriculum presented similar concepts, but at a more sophisticated level. For example, in 7th grade, the spread and prevention of HIV/AIDS was discussed in lecture format. In 8th grade, students took a more active 5 Growing Pains: Lessons from WAIT Training in Florida role; in a group, they identified risk levels for contracting HIV associated with different activities (such as kissing, monogamy with an uninfected partner, or intercourse without a condom). The 9th- grade curriculum was expected to make further adjustments for the continuing maturation of the students. Examples of Activities in 7th- and 8th- Grade Lesson Plans Hopes and Dreams How Do You Spell Love Steps of Physical Intimacy Unit: Learning about Yourself Unit: Friendship, Dating, and Unit: Friendship, Dating, and and Others Love Love Day 1 of 7th- and 8th-grade Day 3 of 7th- and 8th-grade Day 5 of 7th-grade plan and plans plans Day 8 of 8th-grade plan 1. Teacher draws hallway on a 1. Groups of students write 1. Students arrange cards with blackboard, with doors for words/phrases to define steps of intimacy (such as negative influences (such as love and post them on the holding hands, kissing, and premarital sex, drugs, and wall. having sexual intercourse) alcohol) that can lead in order from the least to 2. Teacher discusses how the students off the paths to most physically intimate their goals. word love is used and asks behavior. students to differentiate 2. Teacher and students between love and 2. Teacher explains each step discuss short- and long- infatuation for each of intimacy and discusses term goals and how to reach word/phrase on the wall. which to reserve for them. marriage. Teacher reminds 3. Teacher reads and students students how to give and discuss a story about a receive affection without couple’s attraction. being at risk for STIs, 4. Students discuss what pregnancy, or negative characteristics are important emotional or social effects for strong relationships. of premarital sexual activity. 3. Teacher discusses how sex is misused (for example, to get payback, as proof of loyalty, or as punishment). Formalized and systematic documentation of program delivery. To help ensure the quality of program implementation, LtL planned to systematically assess how closely the teachers followed the prescribed lesson plan. LtL planned to collect data on program fidelity and quality of delivery, as documented by staff in observation forms after viewing selected sessions. LtL would collect this information using a common set of standardized measures developed for use across OAH-funded grantees. These measures focused on specific aspects of program delivery, such as clarity of the teacher’s explanations, the pacing of the session, and the extent to which students appeared to understand and engage with the material. LtL’s Participation as a Site in Federal Evaluation Further Shaped Program Delivery To determine the effectiveness of the funded programs in improving youth outcomes, all OAH Tier 2 research and demonstration grantees were required to conduct evaluations of their programs. Some grantees received technical assistance on implementation of their own local evaluation. Other sites, such as LtL, were selected for participation in the federal PPA evaluation. Participation in the 6 Growing Pains: Lessons from WAIT Training in Florida federal evaluation further shaped LtL’s planned implementation of WAIT Training in Florida schools. Random assignment of schools. The evaluation design for WAIT Training involved random assignment of schools. Half of the recruited schools would be randomly assigned to the treatment group and receive WAIT Training. The other half would be assigned to the control group and receive a diet and exercise program called LEAN (developed by nutritionists, in consultation with LtL). To ensure a strong test of the program, the design called for recruitment of 16 to 20 schools overall. Recruitment of new schools. To reach the school recruitment target, LtL had to branch out to schools that had not recently received WAIT Training. Including recently served schools would create a risk that older students who had received the program might affect the behavior of students in the evaluation sample, and thus possibly distort estimates of the program’s impact. However, a few previously served schools could be included because the program had not been implemented in the schools for several years, and this risk was therefore minimal. Recruitment focused on schools in high-risk areas, with the aim of reaching students in particular need of receiving education on sexual health. To ensure that the evaluation would be rigorous, LtL excluded schools if they were implementing any other abstinence or sex education program. Overall, these criteria disqualified all except two of the schools in which LtL had previously worked. To encourage participation, LtL created incentives for schools. LtL gave treatment and control schools financial compensation based on the number of students who completed the WAIT Training or LEAN programming. LtL originally planned to provide schools with $32 for each student who attended all eight sessions ($4 for each class attended). However, because LtL found that most students were not attending all eight lessons in the first year of implementation, LtL later determined that schools would receive the incentive for every student who attended 75 percent of programming (six of eight classes). Removal of community-based activities from the program model. To eliminate any chance that control students might be affected by LtL-sponsored programming, the evaluation design called for dropping community-based enrichment activities such as parent seminars and the student essay contest (which could have reached students from both treatment and control schools in the same districts). Under this proposed design, all WAIT Training activities therefore would be confined to classrooms in the treatment schools. (LtL, however, continued to deliver some of these activities during the evaluation period, in contrast to the planned design.) Increased emphasis on data collection. Participation in the program and the evaluation required parental consent and data collection. In the absence of an evaluation, LtL would have collected parental consent for students to participate in a single year of the program. Under the evaluation, by contrast, students had to receive signed parental permission to participate in both the program and the evaluation, covering all three years of program participation (in 7th, 8th, and 9th grades). Teachers at treatment schools would pull consented students from their regularly scheduled classes to attend WAIT Training in 7th grade (in spring 2012), 8th grade (in spring 2013), and 9th grade (in spring 2014). To measure program impacts for the evaluation, a survey would be distributed to consented students at baseline (before 7th-grade program delivery in spring 2012) and after program delivery (in 9th and 10th grades). To minimize the burden on LtL and participating schools, Mathematica completed the consent process and administered the baseline surveys. Because the program ended before 9th grade and the accompanying impact study was ultimately suspended 7 Growing Pains: Lessons from WAIT Training in Florida before the completion of survey data collection, only baseline surveys were administered to students; follow-up surveys were not administered. After Developing the Program Delivery Plan, LtL Recruited Willing Schools Thirteen geographically dispersed schools were recruited, based on agreements reached with school principals and their district administrators. (Although LtL did not meet its recruitment target of 16 to 20 schools, Mathematica determined that the recruited sample of 13 schools was sufficient for the purposes of the evaluation.) Seven schools were assigned to the treatment group to implement WAIT Training and 6 were assigned to the control group to implement LEAN. The schools covered a large area and were located in predominately rural, lower-income counties. Nine schools were located in the Florida Panhandle, 2 in northern Florida, one in central Florida, and one in southern Georgia (bordering Florida). According to the National Center for Education Statistics, most students at the schools (69 percent of students at treatment schools and 68 percent at control schools) qualified to receive free or reduced-price lunch during the 2010–2011 school year (the most recent year for which data are available). Eleven of the schools had no prior experience with WAIT Training; 2 schools had previously implemented WAIT Training with an older cohort of students. At the seven treatment schools, program participants were racially and ethnically diverse, and generally had low to moderate levels of risky behavior. On average, participants were age 13 in winter 2012, before receiving the program. Almost half (45 percent) of the participants were white, 29 percent were black, 15 percent were Hispanic, and 11 percent were another race or ethnicity. Program participants reported moderate levels of sexual intercourse, expectations for having sexual intercourse, and levels of alcohol and drug use (Figure 1). Additional details on the program participants are in Appendix C. Figure 1. Levels of Risky Behavior Among Program Participants Levels of Alchohol Use Had alcoholic beverage in past 30 days 11% Ever had an alcoholic beverage 31% Levels of Drug Use Smoked marijuana in past 30 days 5% Ever smoked marijuana 9% Expectations for Sexual Activity Expects to have sexual intercourse before marriage 39% Expects to have sexual intercourse next year 15% Levels of Sexual Activity Ever had sexual intercourse 10% 0% 20% 40% 60% Source: Youth surveys administered by the PPA evaluation team in January through May 2012. 8 Growing Pains: Lessons from WAIT Training in Florida IMPLEMENTATION OF WAIT TRAINING: SCHOOL RECRUITMENT, TEACHER TRAINING, AND PROGRAM DELIVERY Florida requires school instruction to include comprehensive health education, covering not only general health issues but also “family life, including the awareness of the benefits of sexual abstinence as the expected standard and the consequences of teenage pregnancy;” “prevention and control of disease;” and “the characteristics of healthy relationships” (Florida Legislature 2013). The eight-day lesson plans developed by LtL for the adapted version of WAIT Training cover all of these items. However, recruiting schools to implement an external program during the school day is not an easy feat. It requires connections with school administrators, organizational capacity, and acceptance by school staff. Each step of program implementation was affected by LtL’s staffing constraints and the organizational challenges of implementing a program on such a large scale and in new communities and schools for the first time. Three LtL staff members covered roughly all of the tasks required for implementing WAIT Training (for example, conducting classroom observations, and delivering program content when teachers were unavailable or unable to handle the subject matter); they received support for monitoring from two additional staff members in 2011 and four staff members in 2012. Over the course of program delivery, LtL did not expand the number of staff to accommodate these needs. In addition, school schedules and other priorities of school leaders complicated program delivery. LtL Encountered Challenges in Forging Bonds in New Territory and Maintaining Interest at Overburdened Schools To fulfill the evaluation’s school recruitment goals, LtL went beyond its sphere of established relationships. Informing school administrators about the LtL organization, the WAIT Training program, and the evaluation were essential to recruiting schools to participate in the program and evaluation. However, meetings and other contacts with school administrators took considerable time. LtL devoted more time than expected maintaining previously established relationships and forging new connections with school administrators. Many schools were stressed and faced practical challenges that made them wary of implementing the program. Schools faced pressure from the state to improve student achievement. All schools in Florida are graded by the state’s accountability system based on students’ results on standardized tests (the Florida Comprehensive Assessment Test, end-of-course assessments, and Florida Alternate Assessments) in math, reading, writing, and science. Four of the 12 schools ultimately recruited in Florida—compared with roughly 23 percent of schools across the state—were graded as failing or close to failing in 2012. These four schools (three treatment schools and one control school) faced particular pressure to improve scores in tested subjects. Accordingly, some leaders at these schools viewed sex education and WAIT Training as a distraction rather than a way to help improve achievement. In addition, at many of the participating schools, leadership was in flux. In 2012, in midstream of implementation of WAIT Training, school districts representing 4 of the 7 treatment schools (and 4 of the 6 control schools) elected new superintendents; at least one of these new superintendents was not as interested in the program as the previous superintendent. Consequently, LtL had to deal with shifts in the local leadership it was trying to engage to receive approval for implementation of the outside curriculum. Posing a further challenge for recruitment, LtL encountered unforeseen competition from other, more well-known pregnancy prevention 9 Growing Pains: Lessons from WAIT Training in Florida programs, such as the Teen Outreach Program (TOP). In 2010, the Florida Department of Health received a TPP-Tier 1 grant to implement TOP in 26 county high schools. The reach of this program further limited the pool of schools from which LtL could recruit. In this environment, LtL faced an uphill battle in generating support for WAIT Training. School and district leadership often placed low priority on outside programs, particularly those, like WAIT Training, that did not cover subjects assessed as part of the state accountability measures. In particular, leadership deemed outside programs that did not cover assessed subjects nonessential for academic progress. In addition, the constant shift in school leadership hindered LtL’s ability to sustain district and school support, both prior to the first year of implementation (in 2012) and through the second year of implementation (in 2013). These Challenges Led to Prolonged School Recruitment and Outreach As a consequence of these challenges, school recruitment took longer than expected and stretched over a full year, from spring 2011 to spring 2012. When school recruitment began in spring 2011, LtL initially anticipated completing recruitment by summer 2011. By June 2011, LtL staff had reached out to school- and district-level staff in 11 districts in the Florida Panhandle. However, most of these contacted districts were not able to commit to participate in the evaluation (to implement either WAIT Training or LEAN); only 2 districts committed to participate. Consequently, LtL cast a wider net across Florida and southern Georgia to try to reach its recruitment targets. LtL continued recruitment during the 2011–2012 school year; recruitment ended in April 2012. By the time recruitment concluded, LtL had contacted district or school staff in approximately 35 different school districts and secured approval at 13 schools (in 8 districts). LtL had to both initiate and sustain contact with schools to keep them involved. This lasted the better part of two years, from summer 2011 through spring 2013. LtL held multiple in-person meetings with district superintendents, principals, and teachers to obtain buy-in from key stakeholders, some of whom were newly hired and not involved in the schools’ initial decisions to implement the program. LtL’s continued responsibility to recruit schools and maintain relationships with them was time consuming and took time away from working out implementation details (such as scheduling and training) with school administrators. LtL Staff’s Roles in Implementing WAIT Training Focused on Teacher Training Under LtL’s plan, principals at participating schools—not the LtL staff—would select WAIT Training teachers. All except one principal selected the teachers to deliver the program at their schools without input from LtL. (One principal did not want to supplant teachers from their routinely taught classes to deliver WAIT Training and asked LtL to select external facilitators; the individuals LtL selected had previous relationships with LtL staff, but did not have formal training as teachers.) In selecting teachers, principals did not pay close attention to the mix of skills and interest that would be ideal for the program; many were selected because they had time in their schedules to facilitate the class. As a result, not all teachers who delivered the program had the appropriate skills. 10 Growing Pains: Lessons from WAIT Training in Florida Characteristics of WAIT Training Teachers • Teachers taught a variety of subjects (science, language arts, history, health, and physical education), and most did not have knowledge of sexual health issues. One teacher reported that he was “thrown” into the class; he felt a health teacher would have been more prepared to lead the class. Another teacher reported that, as a physical education teacher, he was not prepared to lead discussions on sensitive topics. • Some teaches lacked strong interest in or commitment to the program. Several teachers were unaware that they had been selected to teach about abstinence and sexual health until they arrived at the training. Although several teachers expressed their approval of the overall messages of WAIT Training, teachers at the treatment schools had varying levels of investment and interest in the program. • Several teachers were new to the classroom and lacked strong classroom management skills. A Mathematica researcher found that one teacher, who was in her first year of teaching in 2012, was visibly uncomfortable managing a group of adolescent girls. Another teacher reported that she learned how to effectively manage the classroom only through trial and error. With most principals selecting the teachers, LtL’s role in shaping program delivery was thus limited to training teachers, and training did not occur close to the start of program. The demands of training 13 teachers across five school districts led to an imperfect training schedule. (Although there was teacher turnover across the years, as noted in Table 2, a total of 13 teachers were trained and delivered the program each year.) Ideally, training would occur as close to implementation as possible, so that the material would still be fresh in teachers’ minds when they delivered the program content in the classroom. Program delivery was planned for late April to mid-May of 2012 and 2013 because participating schools wanted to implement the program after students had completed state standardized tests. (The program ended before the planned spring 2014 delivery.) However, the schedule for training was limited by existing teacher commitments. Teachers could not attend training in the weeks before program delivery because they were in charge of administering standardized tests to students during that time. Working around this schedule, LtL fit in a two-day training for the 7th-grade curriculum in late March 2012 for three treatment schools and in late April 2012 for four schools. For the 8th-grade implementation in 2013, teachers at all seven treatment schools attended a one-day training in early March before statewide testing began. (Two of the teachers were unable to attend LtL’s group training; they attended a separate training organized by LtL.) As a result, training took place one to two months before program delivery in 2012 and two months before program delivery in 2013. This left a substantial interval after training before teachers could practice what they had learned. Some teachers reported misplacing curriculum materials during this interval. Table 2. Number of Teachers Trained and Reasons for Turnover Number of Number of Teachers Teachers Supplemental School Trained in 2012 Trained in 2013 LtL Staff Teacher Turnover/Addition A 2 2 n.a. None In 2012, the program was delivered by two B 2 2 1 (2013) teachers. In 2013, an LtL volunteer also delivered the program. In 2012, the program was delivered by two LtL staff members and one teacher. In C 1 1 2 (2012) 2013, a different teacher delivered the program; there were enough students for only one class. D 2 2 n.a. None. 11 Growing Pains: Lessons from WAIT Training in Florida Number of Number of Teachers Teachers Supplemental School Trained in 2012 Trained in 2013 LtL Staff Teacher Turnover/Addition One of the external facilitators who delivered the program in 2012 was unable E 2 2 n.a. to teach in 2013, and was replaced by another external facilitator. The teacher who delivered the program in 2012 did not return to the school the next F 2 2 n.a. year (for reasons unrelated to WAIT Training). In 2013, a different teacher delivered the program. G 2 2 n.a. None. Total 13 13 3 Source: 2012 and 2013 interviews with LtL staff. Note: All LtL staff were trained and certified to deliver WAIT Training. n.a. = not applicable. First-year training (in spring 2012) spanned the full WAIT Training curriculum rather than being adjusted to focus on the eight lessons selected by LtL. The two-day training, led by CRE (with the help of LtL staff), focused on introducing teachers to abstinence training and certifying teachers in the broader WAIT Training curriculum (see text box). CRE staff followed their standard training and certification processes for WAIT Training, rather than tailoring the training to the activities included in the Components of 2-Day Teacher Training adapted curriculum created by LtL. Although CRE • Overview and history of sex education demonstrated some activities, these activities were not necessarily included in the 7th-grade lesson plan. As a • Introduction to the complete WAIT Training curriculum result, there was little room for providing specific tools and training (for example, on sexual health) to • Demonstration of select activities implement the 7th-grade lesson plan. LtL staff provided • Review quiz and certification teachers with the 7th-grade lesson plan, but did not discuss it with teachers during the two days of training. The two-day training was not revised for new teachers in spring 2013; however, a third day of training was added (for new and returning teachers) to focus specifically on the 8th-grade lesson plan. Outside of the demonstrations at the trainings, there was little opportunity for LtL staff to provide in-class modeling of lesson delivery. After the trainings were completed, teachers were expected to be able to deliver lessons on their own, without the active support of LtL staff. This model of program delivery was consistent with the goal of developing and testing a replicable, packaged program with the potential to be delivered on a broader scale. However, it departed from LtL’s previous practice, which involved a more active role for program staff, especially in the early stages of program delivery. For example, in years before the grant funding, LtL staff would typically lead the first class while the teachers observed and sometimes remained in the classroom for the rest of program delivery. Perhaps as a result of switching to a more hands-off approach, LtL staff found that teachers no longer adhered to the lesson plan; when they came to schools to conduct scheduled observations, LtL staff found that teachers were either not teaching from the curriculum or teaching only some of the prescribed lessons. 12 Growing Pains: Lessons from WAIT Training in Florida Because of these factors, LtL’s monitoring visits occasionally led staff to step in and teach classes. For example, at one school in 2012, an LtL staff member consistently taught the 7th-grade boys because the teacher was visibly uncomfortable with the material and required assistance. In 2013, an LtL staff member reported that, when she found that a female teacher at another school was not implementing lessons at the scheduled time, she intervened and delivered the lesson; this occurred four times. Teachers and External Facilitators Adapted the Program Model to Fit Their Schedules In both years of implementation, teachers and external facilitators were often unable to complete the curriculum as planned in eight days. Disruptions to the regular school schedule caused by required statewide tests and end-of-course assessments, field trips, assemblies, and hurricane drills eroded their time. In addition, class periods were often interrupted (for example, by schoolwide announcements), further reducing time available for program delivery. Forty-minute periods, the predominant class length, were often inadequate for full lesson delivery, even without such interruptions. The limited time for program delivery led both teachers and external facilitators to modify the lesson plan. Some teachers and external facilitators would shorten all of the activities in the lesson plan—their intent was to try to cover all of the material, but in an abbreviated way. Other teachers and external facilitators would select just a few activities that could be completed in the shortened class period. Teachers and external facilitators who took the latter approach reported that they selected activities based on what they felt was most important or would be most relevant and engaging for their students. Mathematica staff witnessed both of these approaches: teachers and facilitators in the observed classes either limited the class to one or two activities or appeared to abbreviate the activities to fit them into 40 minutes. LtL Was Unable to Achieve the Planned Monitoring Levels LtL initially planned to conduct classroom monitoring and technical assistance to help the teachers—who were not all well selected and trained—to do the best job possible. LtL planned to monitor both the seven treatment and six control schools (for delivery of WAIT Training and LEAN, respectively). To accomplish this, LtL developed a first-year schedule for monitoring each teacher twice, on the first and last of the eight days of program delivery. LtL thought this schedule would be sufficient to provide targeted guidance and feedback to teachers at both treatment and control schools. However, monitoring was a challenge. LtL could not meet its planned levels of monitoring and technical assistance because staff spent their time on multiple tasks and often were overcommitted. Rather than concentrating on one component of program implementation, staff were conducting recruitment, outreach, monitoring, and technical assistance at the same time across a large number of schools. In 2012, LtL did not have enough staff to observe the concurrent implementations (in late April to mid-May) of WAIT Training and LEAN, which were implemented over a broad geographic area. Although LtL had multiple offices throughout Florida, the staff involved in implementing WAIT Training primarily operated out of one office in Tallahassee. As a result, the staff had to travel long distances—at times driving more than two hours—to reach the schools for monitoring or implementation. Even during the limited monitoring sessions in 2012, LtL observers noticed teachers having trouble with program delivery. Consequently, LtL increased the monitoring 13 Growing Pains: Lessons from WAIT Training in Florida schedule in 2013 to three observations per teacher and increased the number of staff conducting the monitoring (from five to seven staff members). Increased monitoring in 2013 showed that several treatment teachers were not adhering to the implementation schedule and were not delivering WAIT Training at all on some scheduled days. Lack of Consistent Program Monitoring and Support Might Have Contributed to Teachers’ and Students’ Discomfort with the Material As with any new program, many teachers were initially uncomfortable delivering WAIT Training and students were apprehensive about the class. In 2012, Mathematica researchers observed several teachers who struggled to deliver the curriculum confidently and comfortably. These teachers either did not teach from the selected activities in the 7th-grade lesson plan or chose to cover only topics with which they were more familiar or comfortable. For example, Mathematica observed one teacher implementing an activity from the broader curriculum (not in the 7th-grade lesson plan); she later reported that she commonly selected lessons from this curriculum and did not limit herself to following the prescribed lesson plan. In 2012 focus groups, teachers reported that the training provided by LtL and CRE left them feeling unprepared to teach the material and insecure about teaching abstinence. This insecurity might have rubbed off on students, whom teachers reported were unprepared to discuss sexual health and were uncomfortable with the interactive activities (for example, role plays in which students act out verbal and nonverbal refusal skills). Several students in 2012 focus groups said that, at first, it was “awkward” and “weird” to discuss healthy relationships and other sexual health topics, such as how STIs are spread, with their teachers, as opposed to with their parents. Teachers reported that students became more comfortable over the course of the curriculum, but as the program was delivered over only eight days, students did not have much time to grow accustomed to the discussion of sensitive topics. LtL’s inability to take an active role in program delivery might have exacerbated instances of teachers’ and students’ discomfort. Previously, LtL staff taught the first class, thereby modeling the activities and messages for teachers. LtL staff also were present in the classroom to provide ad hoc technical assistance and further modeling, as needed. According to LtL staff, this assistance helped teachers quickly become confident in their ability to teach the curriculum, minimizing the period of unfamiliarity and discomfort teachers would have with WAIT Training. Likewise, the extra assistance enabled students to become more receptive to the material. However, with their limited classroom presence in 2012 and 2013, LtL staff were largely unavailable to help teachers implement the curriculum. Without targeted assistance from LtL, teachers took longer to work through their initial discomfort and unease, and some never gained confidence in their ability to teach the material. 14 Growing Pains: Lessons from WAIT Training in Florida LESSONS FOR THE FIELD AND FUNDERS LtL’s experience implementing WAIT Training in Florida presents considerations for organizations seeking to extend a program’s reach to new service delivery contexts and for grantors seeking to fund organizations to implement school-based programming. LtL faced challenges in simultaneously (1) conducting outreach to and monitoring a greater number of schools across a more geographically dispersed region than originally planned, (2) developing consistent lesson plans, and (3) providing training and technical assistance to teachers. In addition, LtL was hindered by the lack of buy-in and prioritization from district- and school-level administrators and teachers, particularly at the many recruited schools that were struggling academically. Leadership turnover at the district- and school-level also required LtL to devote increased time to maintaining relationships with schools. These issues are not unique to LtL; implementation of outside programming during the school day is challenging, and outside organizations face many hurdles in successfully delivering school- based programs (Ott et al. 2010; Saunders et al. 2011; Sutherland et al. 2011). Successful implementation requires completing a defined sequence of stages, from assessing site readiness and fit through staff training, program piloting, and finally full-scale implementation (Fixsen et al. 2009, 2013). Especially for outside organizations implementing programming in schools, it is critical to complete each stage successfully before moving to the next. Failing to do so risks undermining the quality of the ultimate program implementation, as happened in the case of LtL. Next, we discuss additional lessons and findings that practitioners and funders can use to mitigate these risks and help promote the successful delivery of school-based programs by external organizations. Extending Program Activity to New Sites Brings Role Changes and Adjustments Small organizations typically lack the reserve staff capacity to take on the extra burdens of expanding to new schools. In preparation for delivering a program on a larger scale, organizations must typically dedicate staff to the development of detailed implementation plans and school and teacher recruitment. During program delivery, organizations also must anticipate the additional staff resources needed to conduct expanded program oversight and technical assistance. Such oversight is needed to monitor program delivery to assess the quality of implementation and to provide feedback to teachers to help them improve their delivery of the program. As was the case with LtL, small organizations might be understaffed and perhaps lack the necessary skills to carry out school recruitment and systematic monitoring of program delivery. Funders and program directors should anticipate the new challenges and staff roles associated with large-scale program delivery and plan for adjustments. When funding broad program implementation, grantors should ensure that organizations have a plan in place to augment staff, as necessary, to conduct all necessary tasks for program outreach and delivery. In this plan, leaders of the organization should define how this enlarged staff will be allocated across roles that encompass recruitment and outreach of schools, and how staff will be trained and supported to oversee program implementation across participating schools, monitor program delivery, and provide facilitators with feedback. Broadening Program Delivery Requires Devoting Time to Building Relationships Organizations trying to implement school-based sex education programs are likely to have a better chance of success if they can draw on wide support from community- and school-level 15 Growing Pains: Lessons from WAIT Training in Florida stakeholders. Small organizations such as LtL often have strong personal connections that can be very helpful in implementation at a small number of schools. These relationships are generally with well-connected members of the community, such as school board members, district-level superintendents, parents and community members, and leaders at other community-based organizations. However, organizations can struggle when expanding into target areas in which they lack such relationships. It takes time to develop new relationships, especially as part of expansion to a greater number of implementation locations. Based on LtL’s experiences in reaching out to and forging relationships with stakeholders at Florida schools, funders should expect that it will take at least a year for organizations to develop fruitful relationships with community-level stakeholders, such as school- and district-level administrators, presidents of parent-teacher associations and organizations, and leaders of organizations with similar interests. When the relationships are formed, these stakeholders can help maintain stability in program planning and implementation. They can accomplish this by voicing their support of the planned programming and facilitating meetings between their constituents and the implementing organization. In addition, because school district administrators and school staff are often in flux (as LtL found in implementing WAIT Training), organizations have to anticipate leadership turnover and dedicate staff time to reengage district and school leaders in later years of implementation. Schools Might View Outside Programming as a Low Priority and a Distraction Schools often view teaching about sex education as a distraction from the goal of improving academics. Many schools devote resources to improving students’ scores on state and national standardized tests, thereby improving the schools’ reputations. This can be particularly true in states (such as Florida) that have strict accountability systems in place. Even if outside organizations are able to help a school meet some state requirements (for example, health education requirements), they must understand that they are imposing further demands on the school’s scheduling capabilities. School administrators might be wary about complicating their already busy schedules with further programming demands, regardless of when the program is implemented in the year. To combat these concerns, outside organizations operating in schools must recognize the tight limitations on physical space and time for implementing nonacademic programming within the school day. They should identify and devise a plan to work within these constraints, and design their programming to be as unobtrusive as possible during the school day. They should also be flexible in working with school staff to determine where and when to hold classes, and to identify teachers available to lead the class. In addition, outside organizations must have a clear message to present to school administrators about the program’s value. This message must capture not only the program’s potential short-term value in improving students’ knowledge of how to act responsibly and prevent risky behavior, but also potential broader impacts in enhancing students’ likelihood of success both inside and outside of the classroom. Funders and Program Operators Should Anticipate the Need for Repeated Teacher Trainings to Ensure Successful Program Implementation An essential first step in preparing for a pregnancy prevention program is selecting facilitators who are open to the material and comfortable teaching students about sensitive topics. Regardless of the program, teachers will want detailed guidance on how to deliver the material effectively, even when they have general experience with the subject matter. Organizations should offer teachers 16 Growing Pains: Lessons from WAIT Training in Florida initial and continued training on classroom management (especially for those new to teaching), program content, and delivery. During training, organizations should provide detailed run-throughs of all planned program activities to ensure that teachers are familiar and comfortable with the content. After this initial training and during program delivery, organizations should provide supplemental training and technical assistance to ensure that teachers maintain content knowledge. If trained this way, teachers are more likely to be both interested in the material and willing and able to adhere to the lesson plan. However, outside program operators might always be swimming upstream. Although organizations and funders often view schools as optimal settings for program implementation because of the opportunity they provide to reach a large number of youth at one time—as opposed to offering programming after school or in a community-based setting, where youth must be recruited and often small numbers of youth are served—delivering outside programming in school also comes with its challenges. Principals might prioritize other programs and classes they view as more central to their schools’ performance. They are likely to be more concerned with improving the academic success of their students than with ensuring the successful implementation of an outside program. Consequently, principals might not be concerned with selecting the most appropriate teachers, training them adequately to implement the program, and ensuring that these teachers then implement the program. As a result, program operators should anticipate these potential barriers to the selection and thorough training of teachers, and address these barriers in advance as they plan for program implementation. For example, program operators could hold meetings with school administrators that focus on the ideal characteristics of teachers implementing the program. In addition to these meetings, program operators could observe the teachers in the classroom to see if their style, classroom management, and temperament meet the needs of the program; they could then recommend to school administrators those teachers who they deem have the ideal skills for program implementation. Funders of these programs should take care to guide those they fund to include these considerations in their plans. 17 This page has been left blank for double-sided copying. Growing Pains: Lessons from WAIT Training in Florida REFERENCES Fixsen, Dean L., Karen B. Blase, Sandra F. Naoom, and Frances Wallace. “Core Implementation Components.” Research on Social Work Practice, vol. 19, 2009, pp. 531–540. Fixsen, Dean, Karen Blase, Allison Metz, and Melissa van Dyke. “Statewide Implementation of Evidence-Based Programs.” Exceptional Children, vol. 79, no.2, 2013, pp. 213–230. Florida Legislature. “K–20 Education Code, Required Instruction.” 2013 Florida Statutes. Title XLVIII, Chapter 1003, Section 42. Tallahassee, FL” Florida Legislature, 2013. Available at http://www.leg.state.fl.us/Statutes/index.cfm?App_mode=Display_Statute&Search_String=& URL=1000-1099/1003/Sections/1003.42.html. Ott, Mary A., Maura Rouse, Jamie Resseguie, Hannah Smith, and Stephanie Woodcox. “Community-Level Successes and Challenges to Implementing Adolescent Sex Education Programs.” Maternal and Child Health Journal, vol. 15, 2011, pp. 169–177. Patton, Michael Quinn. Qualitative Research and Evaluation Methods: Third Edition. Thousand Oaks, CA: Sage Publications, 2002. Ritchie, Jane, and Liz Spencer. “Qualitative Data Analysis for Applied Policy Research.” In The Qualitative Researcher’s Companion, edited by A. Michael Huberman and Matthew B. Miles. Thousand Oaks, CA: Sage Publications, 2002. Saunders, Edward, Bushra Sabri, Barbara Huberman, T.W. Klaus, and Laura Davis. “Assessing Capacity to Promote Science-Based Programs: A Key Informant Study of State Teen Pregnancy Prevention Organizations.” American Journal of Sexuality Education, vol. 6, 2011, pp. 227–246. Scientific Software Development. “Atlas.ti: Visual Qualitative Data Analysis Management Model Building in Education Research and Business.” Berlin, Germany: SSD, 1997. Sutherland, M., B. Araia, and M. Finkelstein. “Translating Abstinence Education Theory into Practice: A Case Study of the Implementation Challenges of Operationalizing Logic Models in Federal Demonstration Projects.” Journal of Children and Poverty, vol. 17, no. 1, 2011, pp. 25–43. U.S. Department of Health and Human Services. “Teen Pregnancy Prevention (TPP): Research and Demonstration Programs and Personal Responsibility Education Program (PREP). Funding Opportunity Announcement and Application Instructions.” Washington, DC: HHS, 2010. 19 This page has been left blank for double-sided copying. APPENDIX A IMPLEMENTATION STUDY DATA SOURCES AND METHODOLOGY This page has been left blank for double-sided copying. Growing Pains: Lessons from WAIT Training in Florida Three data sources provided the information for this report: (1) site visits, (2) observation logs, and (3) survey data (a baseline survey administered by Mathematica Policy Research). Site Visits We conducted site visits to collect in-depth data on (1) the planned intervention, (2) adherence to the planned intervention, (3) delivery of the teacher training (by Live the Life [LtL]) and curriculum (by trained teachers), (4) participants’ responsiveness to the curriculum, and (5) successes and challenges encountered during program implementation. In May 2012, two researchers conducted in-person interviews with staff from LtL and observed six facilitated sessions of the WAIT Training curriculum. In May 2013, an analyst observed three facilitated sessions of the WAIT Training curriculum. We also conducted focus groups and interviews with teachers who implemented the curriculum in 2012 and 2013, as well as some of the students who received WAIT Training in spring 2012. In May 2012, a survey researcher and a research assistant/programmer conducted four in-person focus groups with seven teachers (five teachers from three treatment schools and two teachers from a control school), and two focus groups with 18 students from two treatment schools. In June 2012, the survey researcher and research assistant/programmer conducted four telephone interviews with five teachers (four teachers from three treatment schools and one teacher from a control school). In 2013, an analyst conducted telephone interviews with three teachers from three treatment schools. Table B.1 details the sources for the data collected, the time period during which these data were collected, and the topics covered. Analysis approach. Qualitative analysis of the site visit data involved an iterative process using thematic analysis and triangulation of data sources (Patton 2002; Ritchie and Spencer 2002). Because of the number of interviews conducted, we used a qualitative analysis software package, Atlas.ti (Scientific Software Development 1997), to facilitate organizing and synthesizing the qualitative data. First, we developed a coding scheme for the study, organized according to key research questions. Within each question, we defined codes for key themes and subtopics we expected to cover in the interviews. Then, we applied the codes to passages in the interview and focus group notes. To ensure accurate and consistent coding, two research assistant/programmers independently coded site visit and focus group data from 2012, and one analyst coded site visit and interview data from 2013. A researcher (a member of the site visit team in 2012) reviewed the coded documents and reconciled any differences in coding. To address the research questions, we used the software to retrieve relevant passages, and then examined the patterns of responses across respondents and identified themes emerging from the responses. Observation Forms LtL staff observed 40 WAIT Training classes in spring 2012 and spring 2013 (19 classes in 2012 and 21 in 2013). During these observations, staff completed a form to note adherence to the planned intervention, quality of the observed session, facilitators’ comfort with the material, facilitator–youth interactions, and the engagement and receptiveness of youth to the material. However, the extent to which LtL staff completed the forms varied. Hence, we used the forms solely to tabulate the extent to which LtL was able to monitor classes. A.3 Growing Pains: Lessons from WAIT Training in Florida Table A.1. Data Sources Topic Areas Adherence to Challenges Planned Training Planned Participants’ and Data Source Number Date(s) Context Intervention and TA Intervention Responsiveness Successes In-Person Interviews Live the Life Staff 3 May 2012 X X X X X X Telephone Interviews June 2012, Treatment Group Teachers 7 2013 X X X X Control Group Teachers 1 June 2012 X X X X Live the Life Staff 2 July 2013 X X X X X X Focus Group Discussions Treatment Group Teachers 5 (2 groups) May 2012 X X X X A.4 Treatment Group Participants 18 (2 groups) May 2012 X X Control Group Teachers 2 (1 group) May 2012 X X X X Classroom Observations May 2012, Classroom Observations 9 May 2013 X X Observation Forms May 2012, Program Observation Forms 40 May 2013 X X TA = technical assistance. Growing Pains: Lessons from WAIT Training in Florida Survey Data Population served. We gathered data on the population served by WAIT Training from several sources. The baseline instrument collected data on demographic and background characteristics, risk-taking behavior, previous receipt of sex education, and knowledge and attitudes toward sexual activity and contraceptive use of consented youth. We administered the instrument to consented youth from January to May 2012; the data in this report are from the 334 youth in treatment schools who completed the baseline survey. Data on school year 2011–2012 National School Lunch Program qualification came from the National Center for Education Statistics. 1 1 Available at http://nces.ed.gov/ccd/elsi/. A.5 This page has been left blank for double-sided copying. APPENDIX B SUMMARY OF WAIT TRAINING LESSONS This page has been left blank for double-sided copying. Growing Pains: Lessons from WAIT Training in Florida Table B.1. Summary of WAIT Training Lessons Activity Methods Activity Objectives 7th-Grade Lesson Plan Day 1 Introduction • Teacher introduces program and defines norms and expectations • Students’ discomfort allayed through introductions for the class to the class and one another • Students play an adapted version of Bingo that has squares for student characteristics; students check off a square when they meet students with the characteristic in that square Asset Assessment • Teacher reads through list of assets (for example, support, • Students identify their assets and understand that empowerment, boundaries and expectations, constructive use of students with more assets are less likely to be time, commitment to learning, positive values, social involved in high-risk behaviors competencies, and positive identity) • Students circle their assets on worksheets Personal Inventory • Students list traits they admire about themselves, • Students identify and articulate their strengths, accomplishments of which they are proud, goals they hope to accomplishments, and goals reach by age 28, and facts about themselves no one knows • Teacher collects lists and reads select items; students guess who is being described Hopes and Dreams • Teacher draws hallway on a blackboard, with doors for negative • Students conceptualize and understand the influences (such as premarital sex, drugs, and alcohol) that can importance of planning for the future B.3 lead students off the paths to their goals • Teacher and students discuss their short- and long-term goals and how they can reach them Toilet Paper Roll Activity • Teacher unrolls roll of toilet paper, equating individual squares • Students understand the value of thinking about the with key times in life (for example, infancy through marriage) future • Teacher points out that students have the rest of their lives (after marriage) to have sex, and that there are rewards to waiting to have sex Day 2 Future Orientation • Student volunteer attempts to balance a broomstick on his or her • Students understand the value of thinking about the palm while focusing on the stick end (representing the immediate future future) • Student volunteer then attempts to balance the broomstick while focusing on the broom end (representing the distant future) • Teacher and students discuss why it is easier to balance the broomstick while focusing on the broom end • Teacher and students then discuss how to focus on the present versus the future to achieve a balanced life (that is, think about the end goal first) • Teacher emphasizes that, in relation to sex, “abstinence doesn’t mean never” Growing Pains: Lessons from WAIT Training in Florida Activity Methods Activity Objectives Sequence of Success • Teacher reviews three steps to help students achieve goals of • Students learn steps to take to help achieve education, marriage, and children: (1) get an education, (2) wait success until your 20s to get married, and (3) don’t have sex until you get married Personality Styles • Students circle their character traits on worksheet • Students discover their personality styles and • Teacher then defines and discusses with students four groups of appreciate the differences in others these traits and how each group handles relationships How Do You Feel Loved • Teacher defines how people give and receive love • Students understand how to give and receive love • On worksheet, students circle how they feel most loved Choices and Effects • Teacher draws bull’s-eye diagram with words (physical, • Students learn choices have implications for intellectual, emotional, social, spiritual, and financial) down the themselves and others side • Teacher asks students to whom they belong, and whether they or their partner would be most affected by a positive pregnancy test • Teacher writes students’ responses and asks how a person would be affected physically, intellectually, emotionally, socially, spiritually, and financially (by a positive pregnancy test) • Teacher then asks who else would be affected, and how • Teacher discusses how students’ choices affect others Differences Between Men and • Teacher explains how men and women act differently • Students learn about the behavioral differences Women • Students discuss how to appreciate these differences without between men and women stereotyping • Students learn not to stereotype men and women B.4 • Teacher discusses physical differences between men and women • Students have increased appreciation for the physical differences between men and women Day 3 How Do You Spell Love • Groups of students write words or phrases to define love and post • Students differentiate between love and lust them on the wall • Teacher discusses diverse ways the word love is used and asks students to differentiate between love and infatuation for each word or phrase on the wall • Teacher reads and students discuss a story about a couple’s attraction • Students discuss what characteristics are important for strong relationships Obstacles to Success: Fact or Fiction • Teacher reads statements about the effects of drugs, alcohol, and • Students learn myths, rumors, and facts about the tobacco, and student volunteers respond whether they think the effects of drugs, alcohol, and tobacco statements are fact or fiction • Students have information to make healthy • Teacher reviews correct answers with students decisions Pornography: Fact or Fiction • Teacher reads statements about the effects of pornography, and • Students learn myths, rumors, and facts about the student volunteers respond whether they think the statements are effects of pornography fact or fiction • Students have information to make healthy • Teacher reviews correct answers with students decisions A Distorted View • 4 student volunteers sit in chairs, with pantyhose over their faces • Students learn about the dangers of pornography • 4 student volunteers slowly pull the pantyhose off of the other and recognize its harmful effects Growing Pains: Lessons from WAIT Training in Florida Activity Methods Activity Objectives students’ faces, and make observations about what happened to the students’ faces as a result of the pantyhose • Teacher tells students that pantyhose distorted students faces and did not reflect reality, and that this is an analogy for the effects of pornography, which distorts views on sex • Teacher encourages students to be wary of exposing themselves to pornography for the sake of future relationships Day 4 Boundaries Are Your Friend • Teacher explains how clear boundaries can provide children with • Students have positive feelings about boundaries confidence and safety, and freedom within those boundaries • Two student volunteers demonstrate personal boundaries; one student walks toward a blindfolded student, who says when he or she feels the other student has invaded his or her personal space • Teacher and students discuss why people feel uncomfortable when personal boundaries are not respected • Teacher discusses safety provided by personal boundaries, and the value of developing, maintaining, and respecting them Setting Boundaries Early • Teacher discusses violations of boundaries and how to recognize • Students discover the value of boundaries and them refusal skills • Teacher encourages students to consider ways to ensure boundaries are respected by others Refusal Skills • Teacher models nonverbal refusal skills (for example, stiff body • Students learn sexual refusal skills B.5 posture, facial expressions, hand gestures) and verbal refusal skills (for example, saying no or suggesting an alternative) • On worksheet, students write their responses (verbal/nonverbal refusals) to various scenarios • Paired students role-play their responses • Teacher discusses tips for avoiding unwanted sexual advances, reasons to wait to have sex Day 5 Steps of Physical Intimacy • Teacher creates flashcards with steps of intimacy (ranging from • Students identify healthy ways to give and receive “eye to body” to “French kiss”) affection that do not put them or their partners at • Students select flashcards and arrange them in order from the risk for STIs, pregnancy, or negative least to most physically intimate behavior emotional/social effects of premarital sexual activity • Teacher explains each step of intimacy and discusses which • Students learn how sex is misused should be reserved for marriage • Teacher reminds students of the healthy ways to give and receive affection without putting them at risk for STIs, pregnancy, or negative emotional/social effects of premarital sexual activity • Teacher discusses how sex is misused (for example, to get payback, as proof of loyalty, as punishment) Growing Pains: Lessons from WAIT Training in Florida Activity Methods Activity Objectives Day 6 Teen Pregnancy • Teacher reads stories about teen pregnancy from teen male and • Students understand the consequences of teen female perspectives pregnancy from teen male and female perspectives • Students brainstorm and discuss how the pregnancy (in the story) affected the teen male and female Love Bugs • In groups, students review a fact sheet on STIs and write up and • Students understand how STIs are transmitted, present findings on 2 or 3 STIs prevented, and treated • Teacher discusses why teens are at higher risk than adults for STIs Spit in a Cup • 7 student volunteers line up at the front of the class, with 7 • Students understand consequences of having placards (human papillomavirus, Chlamydia, Herpes, HIV/AIDS, multiple partners Gonorrhea, Syphilis, Have not had sex) • Teacher asks each student to spit into a cup, and asks the person who has the “Have not had sex” placard to “drink” from the cup • Teacher discusses value of thinking before one acts and emphasizes that, if students are sexually active, they should get tested for STIs/HIV Pulling Down • Teacher has students demonstrate how it is easier to pull behavior • Students understand the power and influence of down than to pull behavior up by asking one student (standing on belonging and acceptance with a group of peers a chair) to try to pull another (seated on the floor) into a chair, and then asking the students to reverse their roles The Influence of Drugs and Alcohol • Students share thoughts about underage drinking or drug use • Students learn how drugs and alcohol compromise B.6 • Teacher and students discuss implications of underage drinking or abilities to complete a task and increases their drug use vulnerability to sexual advances • Teacher uses relay race (with students dressing/undressing) • Students have increased awareness of research others in various clothing as analogy for how alcohol and drug use associated with alcohol and drug use affects adulthood • Teacher and students discuss research on alcohol and drug use • Teacher discusses how substance use affects sexual activity The Media’s Influence • Students complete a worksheet on how the media influence them • Students discover how the media influence their • Teacher discusses how advertisements are intended to influence lives thoughts and actions What Else Influences Me • Teacher discusses additional influences (for example, parents, • Students recognize other influences on their culture, and religion) with students thoughts and behaviors Day 7 Let’s Get Cookin’ • Student groups read through recipes, then cut the recipes in strips • Students understand how to follow steps to reach and attempt to glue the steps in the correct order desired goal • Teacher discusses the importance of following the correct order of steps to get the desired outcome • Students develop a “life” recipe and brainstorm a series of goals (ingredients) for each category of their future lives (for example, living debt-free, going to college, or getting married); students then organize the goals in the best order for success Growing Pains: Lessons from WAIT Training in Florida Activity Methods Activity Objectives • Students share life recipes with class and discuss what would happen if they prepared the ingredients in different orders (for example, the impact of getting married before graduating high school) • Teacher stresses importance of defining and pursuing goals in sequential order to increase chances of success Characteristics of a Healthy • Teacher discusses how to avoid sexual exploitation and have a • Students understand components of a healthy Relationship healthy relationship, and how teens misuse sex relationship Why I Will Wait • Teacher reviews reasons to wait to have sex • Students learn it is okay to wait to have sex • On a worksheet, students write down their reasons to wait to have sex Day 8 Marriage: It’s Good Stuff • Student groups brainstorm and write down benefits of marriage • Students understand the advantages and value of • Teacher and students discuss benefits of marriage being and staying married Web of Support Closing Activity • Teacher reviews why students should wait, how to wait, and • Students share benefits of WAIT Training classes support for waiting to have sex • As they comment on their experience with the class, students toss a ball of yarn among them to create a “web of support” 8th-Grade Lesson Plan Day 1 B.7 Introduction • Teacher reminds students about the program and defines norms • Students’ discomfort allayed through reminder and expectations for the class about the class Hopes and Dreams • Teacher draws hallway on a blackboard, with doors for negative • Students conceptualize and understand the influences (for example, premarital sex, drugs, and alcohol) that importance of planning for the future can lead students off paths to their goals • Teacher and students discuss their short- and long-term goals and how they can reach them Day 2 Personal Inventory • Students list traits they admire about themselves, • Students identify and articulate their strengths, accomplishments of which they are proud, goals they hope to accomplishments, and goals reach by age 28, and facts about themselves no one knows • Teacher collects lists and reads select items; students guess who is being described Asset Assessment • Teacher reads through list of assets (for example, support, • Students identify their assets and understand that empowerment, boundaries and expectations, constructive use of students with more assets are less likely to be time, commitment to learning, positive values, social involved in high-risk behaviors competencies, and positive identity) • Students circle their assets on a worksheet Future Orientation • Student volunteer attempts to balance a broomstick on his or her • Students understand the value of thinking about the palm while focusing on the stick end (representing the immediate future future) Growing Pains: Lessons from WAIT Training in Florida Activity Methods Activity Objectives • Student volunteer then attempts to balance the broomstick while focusing on the broom end (representing the distant future) • Teacher and students discuss why it is easier to balance the broomstick while focusing on the broom end • Teacher and students then discuss how to focus on the present versus the future to achieve a balanced life (that is, think about the end goal first) • Teacher emphasizes that, in relation to sex, “abstinence doesn’t mean never” Day 3 Differences Between Men and • Teacher explains how men and women act differently • Students learn about the behavioral differences Women • Students discuss how to appreciate these differences without between men and women stereotyping • Students learn not to stereotype men and women • Teacher discusses physical differences between men and women • Students have an increased appreciation for the physical differences between men and women How Do You Spell Love • Groups of students write words or phrases to define love and post • Students differentiate between love and lust them on the wall; teacher discusses diverse ways the word love is used, and asks students to differentiate between love and infatuation for each word or phrase on the wall • Teacher reads and students discuss a story about a couple’s attraction • Students discuss what characteristics are important for strong B.8 relationships Day 4 How Do You Feel Loved • Teacher defines how people give and receive love • Students understand how to give and receive love • On a worksheet, students circle how they feel most loved Friendship and Dating • Students brainstorm words associated with the term dating • Students identify purposes of dating • Teacher and students discuss why people date and how teens • Students learn to adopt healthy dating guidelines date • Teacher explains purposes of dating and dating standards Day 5 The Media’s Influence • Students complete a worksheet on how the media influence them • Students discover how the media influence their • Teacher discusses how advertisements are intended to influence lives thoughts and actions Decision Making • Teacher and students discuss four situations: peer pressure, • Students learn about refusal skills and healthy sexting, going through puberty, and depression relationships • Students discuss how they would react in each situation The Road Map to Success • Teacher discusses what students can do to achieve success (for • Students learn how to achieve success example, discover what motivates them, network with others, get an education, listen to others, stay focused, and stay out of debt) Growing Pains: Lessons from WAIT Training in Florida Activity Methods Activity Objectives Day 6 When You Know Better, You Do • Students match STIs with their descriptions • Students learn about STIs Better • Teacher reviews correct answers Love Bugs • In groups, students review a fact sheet on STIs and write up and • Students understand how STIs are transmitted, present findings on 2 or 3 STIs prevented, and treated • Teacher discusses why teens are at higher risk than adults for STIs Day 7 The Crowded Bed • Teacher creates a bed on the floor and distributes name cards to • Students understand the long-term impact of students premarital sexual activity • Teacher reads a story about having multiple partners and asks students to sit on the bed when the name on the card is read; at the end of the story, the bed is crowded • Teacher and students discuss how sexual behaviors in the story put the characters at risk of STIs and AIDS, and how to start a conversation about past sexual history • Teacher tells students to get tested if they are sexually active and emphasizes the value of starting over Overcoming Potential Obstacles • Student groups brainstorm ways to make healthy choices to • Students learn healthy ways to overcome obstacles overcome obstacles (for example, drug and alcohol use or abuse, negative peer pressure, pregnancy, and diseases) B.9 • Teacher and students discuss responses HIV/AIDS: Fact or Fiction • Teacher prepares flashcards that list activities (for example, insect • Students learn the truth and myths about HIV/AIDS bites, masturbation, hugging, monogamy, donating blood) • Teacher distributes flashcards to students and asks them to say whether the activities on the card have high, some, or no risk for contracting HIV • Teacher and students discuss why these activities are risky or not HIV/AIDS Quiz • Students take a quiz on HIV/AIDS (for example, definition, • Students show their knowledge of HIV/AIDS methods of transmission, and prevention) and discuss responses with teacher Something Fishy • Student groups brainstorm on the consequences of teen sex • Students understand the consequences of teen sex pertaining to the following words: physical, intellectual, emotional, social, spiritual, and financial • Teacher holds up a fish bowl and takes the fish out of the bowl to give it more “freedom,” and then puts the fish back in after showing students how it struggles to survive without water • Teacher uses this as an analogy: fish bowl is to fish as marriage is to sex • Teacher tells students that boundaries are healthy and provide freedom, safety, and comfort Growing Pains: Lessons from WAIT Training in Florida Activity Methods Activity Objectives Day 8 Under the Influence • Teachers discuss statistics on teen alcohol use • Students have increased awareness of statistics on teen alcohol use Pulling Down • Teacher has students demonstrate how it is easier to pull behavior • Students understand the power and influence of down than to pull behavior up asking one student (standing on a belonging and acceptance with a group of peers chair) to try to pull another (seated on the floor) into a chair, and then asking the students to reverse their roles Steps of Physical Intimacy • Teacher creates flashcards with steps of intimacy (ranging from • Students identify healthy ways to give and receive “eye to body” to “French kiss”) affection that do not put them or their partners at • Students select flashcards and arrange them in order from the risk for STIs, pregnancy, or negative emotional or least to most physically intimate behavior social effects of premarital sexual activity • Teacher explains each step of intimacy and discusses which should be reserved for marriage • Teacher reminds students of the healthy ways to give and receive affection without putting them at risk for STIs, pregnancy, or negative emotional or social effects of premarital sexual activity Bonding • Teacher puts tape on volunteer’s arm, saying that the tape • Students learn about how having multiple partners represents his or her partner, and this is the first time they are both affects a person’s future having sex • Students learn that they should not rush intimacy • Student removes the tape as a sign of a break up, leaving DNA on the tape B.10 • Teacher repeats exercise with other volunteers and discusses with students how repeating this process frequently might affect the person’s marriage • Teacher emphasizes that virginity should not be given away freely • Teacher discusses with students the power of hormones • Students brainstorm how to set boundaries when they are dealing with hormones, and not to rush intimacy Note: The 7th-grade lesson plan includes time at the end of each day for questions and answers, and to emphasize the value of starting over (using a mechanical drawing toy or tool analogy). The 8th-grade lesson plan includes time at the beginning of days 2 through 8 to review of concepts learned the previous day. APPENDIX C SAMPLE CHARACTERISTICS This page has been left blank for double-sided copying. Growing Pains: Lessons from WAIT Training in Florida Table C.1 Sample Characteristics Percentage of Treatment Group Students (unless noted) Demographic and Background Characteristics Age in Years (mean) 12.9 Female 57.1 Language Spoken at Home English only 94.5 Spanish only 4.3 Other 1.2 Race/Ethnicity White Non-Hispanic 45.2 Black Non-Hispanic 29.3 Hispanic 14.6 Other (including multiple) 10.9 Lives with Both Biological Parents 50.5 Considers Religion Very Important in His or Her Life 55.5 Attends Religious Services/Activities at Least Once a Week 47.5 Levels of Risky Behavior Alcohol and Drug Use Ever had an alcoholic beverage 31.1 Had alcoholic beverage in past 30 days 11.3 Ever smoked marijuana 8.6 Smoked marijuana in past 30 days 5.0 Ever Had Sexual Intercourse 9.5 Behavioral Expectations Expects to have sexual intercourse next year 14.9 Expects to have sexual intercourse before marriage 39.0 Understanding of Contraceptive Effectiveness and Risk of Pregnancy and HIV/STIs Condoms Decrease the Risk of Pregnancy … Not at all 3.6 A little 25.8 A lot 35.3 Don’t know 32.3 Missing 3.0 Condoms Decrease the Risk of HIV/AIDS … Not at all 8.1 A little 17.1 A lot 33.5 Don’t know 38.6 Missing 2.7 C.3 Growing Pains: Lessons from WAIT Training in Florida Percentage of Treatment Group Students (unless noted) Birth Control Pills Decrease the Risk of Pregnancy … Not at all 2.1 A little 23.7 A lot 31.4 Don’t know 36.2 Missing 6.6 Birth Control Pills Decrease the Risk of HIV/AIDS … Not at all 27.5 A little 8.7 A lot 9.9 Don’t know 44.6 Missing 9.3 a Sample Size 334 Source: Youth surveys administered by the PPA evaluation team from January through May 2012. a Indicates number of students who completed the baseline survey. The sample sizes for each variable differ due to item nonresponse and logical skips, and ranged from 274 to 334. C.4