Findings from an Innovative Teen Pregnancy Prevention Program Evaluation of Healthy Futures in Public Middle Schools in Three Northeastern Massachusetts Cities Final Impact Report for The Black Ministerial Alliance of Greater Boston, Inc. December 30, 2015 Prepared by JSI Research & Training Institute, Inc. Tamara Vehige Calise, DrPH, MEd Wendy Chow, MPH Katelyn Flaherty Doré, MPH Calise TV, Chow W, and Doré KF. JSI Research & Training Institute, Inc. Evaluation of Healthy Futures in Three Northeastern Massachusetts Cities: Findings from an Innovative Teen Pregnancy Prevention Program. Final Impact Report for The Black Ministerial Alliance of Greater Boston, Inc. Prepared for the Office of Adolescent Health, U.S. Department of Health and Human Services. 2015. Acknowledgements: The authors would like to express their thanks to The Black Ministerial Alliance of Greater Boston, Inc., Healthy Futures staff, and the administration, staff, students, and parents of the participating schools for their involvement in this study. Without their support and participation this research would not have been possible. This publication was prepared under Grant Number TP2AH000006 from the Office of Adolescent Health, U.S. Department of Health & Human Services (HHS). The views expressed in this report are those of the authors and do not necessarily represent the policies of HHS or the Office of Adolescent Health. Table of Contents I. Introduction……………………………………………………………………………………………….4 A. Introduction and study overview………………………………………………………………..4 B. Primary research question(s)……………………………………………………………………5 C. Secondary research question(s)…………………………………………………………………5 II. Program and comparison programming…………………………………………………………………6 A. Description of program as intended…………………………………………………………….6 B. Description of counterfactual condition………………………………………………………...8 III. Research design…………………………………………………………………………………………9 A. Sample recruitment……………………………………………………………………………..9 B. Study design…………………………………………………………………………………...10 C. Data collection…………………………………………………………………………………11 D. Outcomes for impact analysis…………………………………………………………………12 E. Study sample…………………………………………………………………………………..13 F. Baseline equivalence…………………………………………………………………………..14 G. Methods………………………………………………………………………………………..15 IV. Study findings…………………………………………………………………………………………17 A. Implementation study findings………………………………………………………………..17 B. Impact study findings………………………………………………………………………….21 V. Conclusion……………………………………………………………………………………………...22 VI. References……………………………………………………………………………………………..24 Appendices A. Massachusetts teen birth rates, 2007 .........................................................................................27 B. Healthy Futures logic model .....................................................................................................28 C. Data used to address implementation research questions......................................................... 29 D. Cluster and youth sample sizes by treatment status ..................................................................35 E. Baseline equivalence of analytic samples used to evaluate secondary research questions .......37 F. Impact model specification ....................................................................................................... .39 G. Data cleaning protocol ...............................................................................................................41 H. Sensitivity analyses ................................................................................................................ ...44 I. Methods used to operationalize each implementation element ...................................................47 J. Implementation fidelity for Healthy Futures program ................................................................50 K. Quality of Nu-CULTURE classroom-based program................................................................55 L. Health educator training, knowledge, confidence, and satisfaction ..........................................56 M. Context ......................................................................................................................................61 EVALUATION OF HEALTHY FUTURES IN THREE NORTHEASTERN MASSACHUSETTS CITIES: FINDINGS FROM AN INNOVATIVE TEEN PREGNANCY PREVENTION PROGRAM I. Introduction Adolescents who engage in sexual activity are at risk of sexually transmitted infections (STIs) and pregnancy. Despite declines in teen pregnancy and birth rates in the United States over the past four decades, the national teen pregnancy rate remains higher than in other developed countries and rates of unintended pregnancies and STIs (e.g., HIV, chlamydia) remain disproportionally higher among minority youth (Hamilton, 2014; Ventura, 2014). Teenage pregnancy is associated with negative consequences for the young parents, their children, and society (Hoffman & Maynard, 2008). Compared to children born to older parents, children of teen parents are more likely to have poorer educational, behavioral, and health outcomes throughout their lives. A number of public costs have been associated with teen pregnancy including $2.1 billion annually in public sector health care costs, $3.1 billion annually in child welfare costs, and $2 billion annually in costs of incarceration (The National Campaign to Prevent Teen and Unplanned Pregnancy, 2013). Addressing teen pregnancy prevention requires the development and implementation of comprehensive evidence-based programs. In 2010, the U.S. Department of Health and Human Services (HHS) launched the Teen Pregnancy Prevention (TPP) initiative with the goal of reducing teen pregnancy by replicating evidence-based models (TPP-Tier 1) and developing, refining, and testing innovative strategies (TPP-Tier 2) (Koh, 2014; HHS, 2010). This report describes the implementation and impact evaluation findings of the Healthy Futures (HF) program (TPP-Tier 2) in public middle schools in three Massachusetts cities. A. Introduction and study overview There is a great need for TPP programs in Massachusetts. The Centers for Disease Control and Prevention Youth Risk Behavior Survey data from 2009 indicate that 46% of all Massachusetts high school students reported ever having had sexual intercourse, 5% reported having had sex before age 13, and 13% reported having had sex with four or more persons (during their life). Nearly 35% of those students who reported ever having had sexual intercourse were currently sexually active. Although Massachusetts had one of the lower teen birth rates in the U.S. in 2009, racial and ethnic disparities persist (Ventura, 2014). In Massachusetts, compared to non-Hispanic White youth, Black and Hispanic youth were more likely to report ever having had sex—44.1% compared to 54.1% and 55.9%, respectively. Identifying evidence-based approaches that not only address contraception and STIs but also increase protective factors (e.g., self-efficacy, communication with parents) is particularly important in Massachusetts. Despite the adverse outcomes associated with teen sexual activity, only half (49%) of Massachusetts high school students reported speaking with their parents or another adult in the family about sexual health topics (e.g., sexuality, HIV, pregnancy prevention) during the past year, according to the Massachusetts Department of Elementary and Secondary Education and the Massachusetts Department of Public Health (2009). For many 4 Massachusetts teens, school-based health education may be their only opportunity to learn medically accurate information about sexual and reproductive health. Even with the great need for health programs, according to the Massachusetts Budget and Policy Center’s Children’s Budget (2015), state funding for school-based health programs was cut by $5,000,000 between 2009 and 2010. HF is a multi-level program framed on the social ecological model, which posits that lasting changes in health behaviors require not only increased knowledge and skills at the individual level but also supportive physical and social systems (McLeroy, 1988). The main HF component is Nu-CULTURE, a classroom-based relationship education curriculum, offered by health educators for eight 50-minute sessions each in 6th, 7th, and 8th grade. Nu-CULTURE content aims to increase student knowledge and build skills to enhance individual protective factors. In addition, HF aims to enhance supportive social systems through the Nu-CULTURE daily parent connection forms that encourage parent and youth communication as well as through the virtual on-line classrooms, after-school and summer peer programs, and parent workshop components. The model is described in more detail in Section IIA: Program and comparison programming. In 2010, The Black Ministerial Alliance of Greater Boston, Inc.— HF’s fiscal agent—was awarded one of nineteen TPP-Tier 2 grants to develop, refine, and test HF. JSI Research & Training Institute, Inc. (JSI) was subcontracted to design and implement a school-cluster randomized controlled trial of HF in the middle school setting. HF has been delivered to thousands of youth throughout Massachusetts since 2002 and previous research on the program indicated Healthy Futures was more effective in increasing knowledge and changing intentions in middle school students as compared to high school students (Calise, 2012). It was hypothesized that by offering HF before youth have sex, the increased knowledge and skills would help them to delay sexual activity. Therefore, middle schools in three racially and ethnically diverse cities—Haverhill, Lowell, and Lynn—were chosen to participate in this study. As illustrated in Appendix A, prior to the implementation of the study, these cities were ranked among the 20 communities in Massachusetts with the highest teen birth rates (O’Keefe, 2007). B. Primary research question(s) The primary research question for the impact evaluation was: What is the impact of the 3- year Healthy Futures relationship education program on the prevalence of students who have ever had vaginal sex by the end of the 8th-grade Nu-CULTURE curriculum, compared to the control group? C. Secondary research question(s) Secondary aims of this study examined the impact of Healthy Futures on the primary outcome for different subgroups, other behavioral outcomes, and outcomes at a different focal time point (Table I.1). 5 Table I.1. Secondary research questions What is the impact of Healthy Futures on the prevalence of ever having had vaginal sex by the end of the 8th-grade Nu-CULTURE curriculum compared to the control group, by ethnicity? What is the impact of Healthy Futures on the prevalence of ever having had vaginal sex by the end of the 8th-grade Nu-CULTURE curriculum compared to the control group, by gender? What is the impact of Healthy Futures on the prevalence of sexual activity within the past 3 months by the end of the 8th-grade Nu-CULTURE curriculum, compared to the control group? What is the impact of Healthy Futures on the prevalence of engagement in unprotected sex without an effective type of birth control within the past 3 months by the end of the 8th-grade Nu-CULTURE curriculum, compared to the control group? What is the impact of Healthy Futures on the prevalence of ever having had vaginal sex at one year follow-up (9th grade), compared to the control group? II. Program and comparison programming A. Description of program as intended HF is a multi-level program that includes a number of interacting components at different social ecological levels – individual, interpersonal, and organizational. The intended program components, content, implementation, and theory of change are described below. HF components include: 1) Nu-CULTURE, a classroom-based relationship education curriculum that includes daily parent connection forms, 2) unlimited access to virtual classroom content, 3) Rhymin’ it Write, an after-school program, 4) Code A, a summer program, and 5) a parent website and workshops (True Connections). The target population, setting, duration, and dosage for each component are outlined in Table II.1. Table II.1. Healthy Futures components Component Target Population Setting Duration Dosage 6th grade Classroom, 50-minute 8 sessions/year 7th grade Take-Home Nu-CULTURE sessions 24 sessions total 8th grade Forms Parents/Guardians of Nu-CULTURE students Unlimited Virtual Classroom Self-selected subset of Nu-CULTURE Online Unlimited access access students After-School 90-minute 10 sessions/year Rhymin’ it Write Self-selected subset of Nu-CULTURE Program sessions 30 sessions total students Summer 240-minute Code A Self-selected subset of Nu-CULTURE 18 sessions total Program sessions students Parent Website & Self-selected parents/guardians of Online, Unlimited Unlimited access, True Connections Nu-CULTURE students Workshops access, 4 sessions 120-minute sessions The main component is Nu-CULTURE, the relationship education curriculum offered in 6th- th 8 grade classrooms during the school day. A pair of health educators facilitate eight consecutive 50-minute sessions each year for a total of 24 sessions. Nu-CULTURE is implemented during regularly scheduled classes (e.g., health, science, physical education, social studies) and all 6 students are required to participate unless opted out by their parent, school, or themselves. Health educators are supposed to be racially, ethnically, and linguistically diverse; recruited from the communities being served; independent of the school system; and trained and supervised by HF. Health educators are expected to attend a week-long training on the curriculum, classroom management, and sexual health topics each year and booster trainings as necessary. All sessions are designed to be interactive and engage youth through multiple teaching strategies including lectures, discussions, role play, skits, and multimedia. Multimedia methods include an audio vignette illustrating a teen pregnancy scenario and a number of videos depicting teens faced with tough decisions (e.g., alcohol use, sexual activity). Nu-CULTURE addresses three key areas: 1) human anatomy and physiology, 2) life skills, and 3) relationships. The eight 6th-grade sessions aim to build a foundation for a healthy teen. Lessons include puberty and reproduction; gender reflection; identifying emotional needs; assertiveness and refusal skills; conflict resolution; qualities of a good friend; sexual abuse; and cyber assault. The eight 7th- grade sessions further build on the 6th-grade foundation in the three areas. Health educators discuss: puberty and pregnancy; identifying and achieving dreams; basic human needs; consequences of choices; handling stress in healthy ways; overcoming peer pressure by enhancing refusal skills; and exploring levels of friendship, positive character traits, and sexual harassment. The eight 8th-grade sessions expand on the information discussed in 6th and 7th grade and enhance students’ skills to identify goals, dreams, emotional needs, and set personal limits in relationships. Sessions emphasize the connection between teens’ choices and their ability to achieve goals and dreams. At the end of each of the 24 sessions, students receive parent connection forms to complete with their parent at home. Each form reviews the topics covered that day and provides additional resources for parents (e.g., reminders to visit ontheirlevel.org). Students are encouraged to return the bottom portion of 21 of the 24 forms (completed and signed by a parent) during the next session. The second component of the HF program is virtual classroom education. All participating Nu-CULTURE students are encouraged to visit one of two age-appropriate websites (onmylevel.org for 6th- and 7th-grade students and doinitright.org for 8th-grade students) for additional information and to ask sensitive questions that they may feel embarrassed to ask in- person and/or may not have a trusted adult to ask. All questions are answered by trained HF staff. The third and fourth components, Rhymin’ it Write and Code A, are out-of-school time programs that are optional to participants and serve a subset of Nu-CULTURE students. Three parent connections forms each year promote the Rhymin’ it Write component. Students who return the forms marked “yes” as “interested in attending” the after-school program are recruited to participate in Rhymin’ it Write. Code A participants are recruited from the Rhymin’ it Write after-school program. These programs are delivered by the Nu-CULTURE health educators and are designed to reinforce Nu-CULTURE sessions and encourage youth leadership through the arts. Rhymin’ it Write is offered in 6th-8th grades and Code A is offered the summer after 8th grade. The fifth component of the HF program is the website (ontheirlevel.org) and workshops (True Connections) designed for parents. These parent activities provide skills for building parent-child connections and discussing relationship and sexual and reproductive health topics. Three parent connections forms promote True Connections each year. Parents of students who return the forms marked “yes” as “interested in coming to a parent program” are recruited to participate in True Connections. The HF parent coordinator delivers school-based True 7 Connections workshops during the school year on a day and time identified by the school and interested parents. HF is informed by the social ecological model which suggests that lasting changes in individual health behaviors require social systems (e.g., universal relationship education) and interpersonal relationships (e.g., parent-teen communication, health educators, and peer leaders) that support positive lifestyle habits (McLeroy, 1988). HF is also informed by social learning theory, which suggests that human behaviors are the result of three interacting factors, mediated by contingencies of reinforcement and observational learning: pre-existing behaviors; cognitive, affective, and biologic internal events; and external events within the environment (Bandura, 1994; Carter-Jessop, 2000). Accordingly, HF includes four components directed at changing or strengthening these interacting factors: 1) informational, to increase youth’s awareness and knowledge of health risks and the benefits of prevention; 2) social and self-regulatory skills, to teach youth how to translate knowledge into effective action through observations of role models; 3) skill enhancement and resilience self-efficacy, opportunities to practice healthy skills and bolster belief in one’s capability to effect change, such as role-playing; and 4) social supports for personal change, to provide a broad network of social supports, such as health educators, teachers, parents, and peers. Founded on these principles, HF aims to: 1) influence attitudes, behavioral and normative beliefs, and self-efficacy regarding healthy relationship choices and risk-reduction behaviors by creating an environment where healthy choices are supported by health educators, peers, and family; 2) strengthen beliefs about the benefits of delaying sexual activity; 3) increase intentions to delay sexual activity; and 4) develop leaders to be positive role models. See Appendix B for the HF logic model. B. Description of counterfactual condition The control program offers 6th- through 8th-grade students two 50-minute classroom-based sessions each year on general health topics (in accordance with the Massachusetts Health Education Standards). The control program is implemented during regularly scheduled classes (e.g., health, physical education, social studies) and all students are required to participate unless opted out by their parent, school, or themselves. The target population, setting, duration, and dosage for the control condition are outlined in Table II.2. Table II.2. Healthy Futures Control Components Target Population Setting Duration Dosage 6th grade 50 minute 2 sessions/year, 7th grade Classroom sessions 6 sessions total 8th grade The 6th-grade curriculum includes one class on puberty and reproduction and one class on bullying prevention; the two 7th-grade classes cover dating-violence prevention; and the two 8th- grade classes are on mental health promotion and suicide prevention. The control curriculum is unique from the Nu-CULTURE curriculum delivered to the treatment students. Each control session was co-led by a pair of health educators who are separate from the Nu-CULTURE educators. Similar to the Nu-CULTURE educators, the control educators were external to the school system and trained and supervised by HF. 8 III. Research design A. Sample recruitment Public school districts in three communities were targeted because of their racial/ethnic diversity and their rank among the 20 communities in Massachusetts with the highest teen birth rates prior to the grant. Table III.1 provides state and community demographics and Appendix A lists 2007 teen birth rates from the targeted communities. Table III.1. Targeted community demographics (2010) Total % % % % % City Median Household Income Population White Hispanic Black Asian Other MA $62,072 6,547,629 80% 10% 7% 5% 8% Haverhill $60,611 60,879 86% 15% 3% 2% 3% Lowell $51,471 106,519 60% 17% 7% 20% 4% Lynn $44,367 90,329 58% 32% 13% 7% 5% Source: U.S. Census Bureau, 2010 Census. The public middle schools from these targeted communities were eligible to participate. The HF Executive Director and JSI Evaluation Director discussed the program and evaluation with superintendents, principals, and teachers from the targeted school districts. 1 Study conditions included agreeing to participate regardless of the school assignment into either the treatment or control group. Memoranda of Understanding (MOUs) were signed prior to random assignment by 15 middle schools in three communities: Haverhill (4 schools), Lowell (8 schools), and Lynn (3 schools). Specifically, at the start of the 2011-2012 school year, MOUs were signed by 10 schools from Lowell and Lynn. In March 2012, the four Haverhill schools agreed to participate in the evaluation. These fourteen schools comprise Cohort 1. One additional school in Lynn agreed to participate starting in the 2012-2013 school year and served as the single Cohort 2 school. These schools represented all of the public middle schools serving 6th-8th grade in the target communities. 2 HF was implemented and evaluated in classroom (in-person and virtual), out-of-school (afterschool and summer), and non-classroom school (parent workshops) settings in the 15 public middle-schools. The control setting included only in-person classroom setting. Students were eligible to participate in the study if they: 1) attended one of the 15 participating public middle schools; 2) were enrolled in the 6th grade in the 2011-2012 school year (Cohort 1) or 2012-2013 school year (Cohort 2); and 3) were not withdrawn from HF program participation by a parent or school administrator(s) prior to the consent/assent process for the study. Students were not eligible to participate in the study if they were not in the 6th 1 Two additional districts (Chelsea and Lawrence) were approached. One declined to participate. At the start of the grant and prior to the launch of the study, including randomization, the second school district went under receivership. Given the uncertainty of the future of this district, the HF Executive Director and JSI Evaluation Director decided to recruit Haverhill instead. 2 Each district also had vocational and alternative schools serving small populations of 6th-8th grade students (fewer than 20 6th-8th grade students per school). These schools were not included because implementing a 3-year program as designed would not have been feasible. 9 grade at the time of consent or if they enrolled in the school after the consent/assent process occurred. Prior to program implementation, each school provided the evaluation team with rosters for each of the 6th-grade classrooms (referred to as “sections”) in which the Nu- CULTURE and control programs were to be implemented. The evaluation team worked with HF staff and each school coordinator to distribute program information and study consent/assent forms to eligible 6th-grade students. Students were eligible to participate in HF and the control curriculum regardless of their study consent status. B. Study design The study was a school-cluster randomized controlled trial. Schools within each city were rank-ordered by the percentage of students eligible for free/reduced price lunch and were matched in pairs. Schools from each pair were randomized with a 50% probability of being assigned to either the treatment (n=7) or the control group (n=8) by the evaluation team. Randomization occurred after school principals signed MOUs but prior to parent consent and student assent. Cohort 1 was followed for four school years (2011-2012 to 2014-2015) and Cohort 2 was followed for three school years (2012-2013 to 2014-2015). In Cohort 1, 14 schools were randomly assigned to condition. At the start of the 2011-2012 school year, 8 schools in Lowell and 2 in Lynn were matched and randomized to condition. Later that year, the four Haverhill schools were matched and randomized. The single Cohort 2 school was randomized to the control group in August 2012 (without a pair, but is considered part of the Lynn “pair” for the purposes of analyses, described in Section III.G.1.). Program staff and schools were made aware of the randomization outcome for logistical reasons (i.e., to inform scheduling, planning, and coordination of either the 8-day Nu-CULTURE curriculum and supplemental activities or the 2-day control curriculum). Sixth-grade students enrolled in the targeted schools during the first year of the study were eligible for inclusion. No sub-sampling was conducted after randomization and both students and parents were unaware of the school’s group assignment during recruitment because consent forms did not indicate treatment status and both interventions were referred to as HF during program delivery. Study protocols and tools were reviewed and approved first by the Essex Institutional Review Board (2010–2014) and then by the JSI Institutional Review Board (2014–2015). C. Data collection 1. Impact evaluation Five self-administered student surveys were collected over four years: 1) at baseline before students received the HF program or the control; 2) immediately after the Nu-CULTURE curriculum in 6th grade (the 6th-grade follow-up); 3) immediately after the Nu-CULTURE curriculum in 7th grade (the 7th-grade follow-up); 4) immediately after the Nu-CULTURE curriculum in 8th grade (the 8th-grade follow-up); and 5) for Cohort 1 only, approximately one year after the completion of the 8th-grade curriculum (the 9th-grade follow-up). Treatment group surveys were scheduled on day 1 (baseline) of Nu-CULTURE before the 8-day program began and on day 8 (6th-, 7th-, 8th-grade survey) after program completion. Control group surveys were scheduled to coincide with the treatment group data collection on day 1 (baseline) before the 2- 10 day control program and approximately one week after the control program (6th-, 7th-, 8th-grade survey). Trained data collectors from the evaluation team administered classroom-based paper- and-pencil surveys to participating students with study consent. Students who did not provide study consent/assent and those who did not return consent forms received a similar length “faux” survey on unrelated topics (e.g., physical activity, television viewing) to ensure all students were treated equally. A retention protocol was developed and implemented to: 1) obtain accurate participant registration or transfer status in 6th-9th grades; 2) collect updated contact information at 7th- and 8th-grade follow-up surveys; 3) schedule and administer school-based make-up surveys for students who were absent or transferred to a non-study school within a participating district (e.g., alternative or vocational schools); 4) re-contact participants who were absent, truant, or moved out of study schools; 5) schedule and administer phone surveys; and 6) mail a thank you letter and incentive upon phone survey completion. 3 For participants who returned Spanish consent forms, re-contact letters and calls were in Spanish. Consistent with the paper survey administration, phone surveys were implemented only in English. Surveys measured intermediate outcomes such as knowledge, attitudes, beliefs, intentions, resistance skills, and relationships and communication with parents and family members. These data will be used for additional analyses to be published elsewhere. The 8th- and 9th-grade surveys asked about sexual activity, condom and birth control use, pregnancy, and vaginal and oral sex intentions (these questions were not asked at either baseline or follow-up surveys in 6th- or 7th-grade due to students’ young age). 2. Implementation evaluation A detailed list of implementation evaluation data collection is included in Appendix C, to supplement the overview provided below. Adherence and fidelity to the Healthy Futures model. Adherence and fidelity data were collected on the number of HF and control activities offered, number of HF and control activities received by participants, how the HF and control content were delivered, and who delivered the HF and control content. These data were obtained from confirmed schedules, attendance sheets, online daily fidelity checklists (HF only), direct program observations conducted by the evaluation team, and lists of health educators hired and trained. Data were reported to the evaluation team daily (fidelity checklists), weekly (schedules), at the completion of each section (attendance, program observations), or biannually (staff lists, trainings). The evaluation team also monitored unplanned adaptations to implementation through weekly reviews of the HF fidelity checklists completed by the HF health educators and direct classroom observations. Unplanned adaptations were reported to the HF Executive Director to inform continuous quality improvement efforts. The evaluation team also collected data on students’ experiences, including mobility between control and treatment schools and self-report of students’ exposure to other TPP programming. HF submitted website analytics for each of the virtual components. 3 Students who completed the phone surveys received a Target gift card (ranging from $25-$40) as a thank you for their time. 11 Quality of Healthy Futures implementation and youth engagement. The evaluation team assessed quality of implementation and youth engagement through direct program observations of the HF and control sessions. Two evaluation team members observed nine percent of the implemented HF sessions and five percent of the implemented control sessions. Observations were scheduled on a rolling basis to capture a representative sample of classes and educators at all HF schools and sections. The HHS’ program observation form was used to collect quality and engagement data (HF and Control) and a HF-specific form was used to collect implementation and engagement data (HF only). The HHS form assessed quality of program delivery using seven measures (Table III.2). Each measure was rated on a 1-5 scale with higher scores indicating higher quality. Once observation data were collected, the evaluation team compared observations to assess inter-rater reliability and ensure agreement. Table III.2. Quality measures Measure # Measure description 1 Clarity of the program implementer’s explanations of activities 2 Extent to which implementer keeps track of time during the session and activities 3 Extent to which the presentation of materials seem rushed or hurried 4 Extent to which the participants appear to understand the material 5 Level of active participation by students Implementer’s qualities (e.g., knowledge of program, level of enthusiasm, poise and confidence, 6 rapport and communication with participants, and ability to effectively address questions and concerns) 7 Overall quality of the program session (a summary measure of measures 1-6) The HF-specific form mirrored the online daily fidelity checklists completed by health educators. It assessed whether content was not presented, content was presented but modified, content was presented but the delivery was modified, or content was taught using the designed materials and activities. The form also assessed if health educators used positive reinforcement of student participation (yes/no question) and the level of student engagement (0-100% in 10% increments). Context. To better understand the environment within which HF was delivered, the evaluation team collected school, community, state, and federal data. This included monitoring school-, state-, and federally-funded TPP programs, sex education, and HIV/STI prevention services available in study schools and communities. These data were collected via web searches, informal discussions with district and school-level staff, and students’ self-reported exposure to other TPP-related programs in the classroom-based pencil-and-paper surveys. D. Outcomes for impact analyses The primary research question was assessed using the 8th-grade follow-up response to the yes/no question “Have you ever had sex?” as shown in Table III.3. The secondary research 12 questions examined the impact of HF on the primary outcome for different subgroups, other behavioral outcomes, and outcomes at a different focal time point as shown in Table III.3. Table III.3. Behavioral outcomes used for primary and secondary research questions Outcome Measures Description of Outcome Timing of Measure Primary Outcome . . Response to “Have you ever had sex? Immediate follow-up Ever had vaginal sex [Yes coded as 1, No coded as 0] in the 8th grade Secondary Outcomes . . Response to “Have you ever had sex? Ever had vaginal sex Immediate follow-up analyzed by gender [Yes coded as 1, No coded as 0] in the 8th grade Response to “Have you ever had sex? Ever had vaginal sex Immediate follow-up analyzed by ethnicity [Yes coded as 1, No coded as 0] in the 8th grade Response to “In the past 3 months, have you had sex, even Immediate follow-up Sex in the past 3 months once?” [Yes coded as 1, No coded as 0] in the 8th grade Response to “In the past 3 months, how many times have you Unprotected sex in the past had sex without using an effective method of birth control? 3 months without using an effective method of birth [Response greater than 0 coded as yes/1; a zero response Immediate follow-up control coded as no/0] in the 8th grade Response to “Have you ever had sex? 1-year follow-up in Ever had vaginal sex [Yes coded as 1, No coded as 0] the 9th grade Notes: A definition of sex was provided on the survey: “By sex, we mean a male putting his penis into a female’s vagina. Other ways of saying sex are “doing it” or “going all the way.” A list of effective methods for birth control was provided on the survey: condoms, birth control pills, the shot, the patch, the ring (Nuva Ring), IUD (Mirena or ParaGard), implant. Students who never had sex were coded as “No/0” on all sub-questions. E. Study sample A total of 2,346 students across the 15 randomized schools were eligible for the evaluation (HF: 1,055; Control: 1,291) (see Appendix D for details across both cohorts). Of the eligible students, 434 were opted out by a parent or personally declined to participate, and 568 students did not return consent forms. The remaining 1,344 students (57%) had both parent consent and assent to participate (HF: 597; Control: 747) and were enrolled. Consent form return rates were 73% and 78% for the treatment and control schools respectively. Of students who returned forms, similar percentages in the treatment (78%) and control group (74%) consented. A total of 1,127 students (84% of 1,344 from Cohorts 1 and 2) were surveyed at the immediate 8th-grade follow-up (HF: 488/597; Control: 639/747). The 217 students who were not surveyed in the 8th grade (but had baseline data) were similar to the 1,127 surveyed except those who were not retained were more likely to be other non-Hispanic (chi-square, p = 0.03) or from a single-parent/non-two parent household (p < 0.01). Of the 1,127 surveyed, 94.1% (1,060) contributed data to the 8th grade primary outcome of interest (HF: 464; Control: 596). A total of 996 students (80% out of possible 1,247 from Cohort 1) were surveyed at the 9th- grade 1-year follow-up (HF: 476/597; Control: 520/650). No statistically significant differences were found between students surveyed and those not surveyed in the 9th grade, except those not 13 retained were less likely to be Asian non-Hispanic (10.1% versus 17.4% among those retained; p < 0.01). Of those surveyed, 895 contributed data to the ever had sex question (HF: 437; Control: 458). F. Baseline equivalence Baseline equivalence tests were conducted on the analytic samples to assess if there were any differences between the treatment and control groups. Mixed effects models were regressed on each baseline demographic variable of interest, with covariates including the group assignment and match-pair indicator and adjusted for school clustering. Table III.4.a shows the baseline equivalence for the immediate post 8th-grade sample used for the primary research outcome. 4 Students averaged 12 years old, about 47.8% were female, and 40.0% were Hispanic, with no statistically significant differences between the treatment and control groups. Table III.4.a. Summary statistics of key baseline measures for youth completing the immediate 8th-grade follow-up survey (n=1,060) Treatment Treatment versus versus Treatment Control control control Mean or % (standard Mean or % (standard Mean p-value of Baseline Demographics deviation) deviation) difference difference Age (years) 12.2 (.59) 12.3 (.55) -0.1 0.12 Gender (female) 45.9% 49.3% -3.4 0.24 Race/ethnicity Hispanic 40.1% 39.8% 0.3 0.81 White, non-Hispanic 32.8% 25.2% 7.6 0.69 Asian, non-Hispanic 11.0% 19.0% -8.0 0.40 Black, non-Hispanic 4.5% 5.4% -0.9 0.95 Other, non-Hispanic 11.6% 10.7% 0.9 0.58 Have a boy/girlfriend 19.6% 17.6% 2.0 0.80 Sample size 464 596 Notes: Baseline measures of behavioral outcomes were not measured due to students’ young age in 6th grade. P- values were derived from models that regressed the demographic variable of interest on covariates that included group assignment and the free-reduced lunch match pair indicator used to stratify random assignment, and that adjusted for school clustering. 4 See Appendix E tables E.1a-E.1b and tables E.2a-E2c for baseline equivalence by gender and by ethnicity subgroups respectively for evaluating differences in the ever had sex outcome between the treatment and control groups. For secondary research outcomes related to past 3 months sex, condom and birth control use which were also based on responses to the immediate 8th-grade follow-up survey, a smaller analytic sample of 1,029 was used instead of 1,060; 31 students with a response to the ever had sex question did not respond to the past 3 months sex questions. Appendix E table E.3 shows the baseline equivalence for the 1,029 sample. No statistically significant differences were found. 14 Table III.4b shows the baseline equivalence for the 1-year follow-up 9th-grade survey sample used for analysis of the secondary outcome of ever had vaginal sex. Students averaged 12 years old, about 47.6% were female, and 39.7% were Hispanic, with no statistically significant differences between the treatment and control groups. Table III.4.b. Summary statistics of key baseline measures for youth completing the immediate 9th-grade follow-up survey (n=895) Treatment Treatment versus versus . Treatment Control control control Mean or % (standard Mean or % (standard Mean p-value of Baseline Demographics deviation) deviation) difference difference Age (years) 12.2 (.60) 12.2 (.53) -0.1 0.06 Gender (female) 45.5% 49.6% -4.0 0.27 Race/ethnicity . . . . Hispanic 38.7% 40.6% -1.9 0.24 White, non-Hispanic 31.1% 22.9% 8.2 0.28 Asian, non-Hispanic 13.3% 21.2% -7.9 0.65 Black, non-Hispanic 4.6% 4.8% -0.2 0.75 Other, non-Hispanic 12.4% 10.5% 1.9 0.35 Have a boy/girlfriend 19.2% 17.5% 1.7 0.79 Sample size 437 458 . . Notes: Baseline measures of behavioral outcomes were not measured due to students’ young age in 6th-grade. P- values were derived from models that regressed the demographic variable of interest on covariates that included group assignment and the free-reduced lunch match pair indicator used to stratify random assignment, and that adjusted for school clustering. G. Methods 1. Impact evaluation The analytic samples included all students who completed the 8th- or 9th-grade surveys depending on the primary or secondary research questions (see Section III.D. Outcomes for impact analyses). Data from all participants were pooled across schools and analyzed based on their initially assigned status (HF vs. Control) regardless of program participation, in accordance with the intent-to-treat framework. Dependent variables were binary (yes/no). Hierarchical generalized linear mixed effects regression models were used to account for school-level clustering (the non-independence among students within the same schools) using Proc GLIMMIX in SAS 9.4 (SAS Institute Inc., Cary, NC). Type III tests of fixed effects were used to determine significance. For all outcomes, the model specifications included the group 15 assignment (HF vs. Control) indicator, the randomization matched-pair indicator, 5 and participant demographics of gender (male/female), age, race/ethnicity (Hispanic, White, non- Hispanic, other non-Hispanic), and boy/girlfriend (yes/no) status in the 6th grade as fixed effects and school as a random effect (see Appendix F for detailed model specifications). Students’ young age precluded asking about sex experiences at baseline. Regression-adjusted mean probabilities are reported for all outcomes, with results considered statistically significant at p < 0.05 using two-tailed tests. No adjustments for multiple hypothesis testing were made. For examining program impact on the “ever had sex” outcome by gender and ethnicity, analyses were conducted separately for each student subgroup. For the benchmark analysis, missing 8th-grade outcome data were derived from the 9th-grade survey where appropriate (e.g., students who were missing a response to the “ever had sex” question in the 8th grade, but answered “no” in 9th grade, were coded as “no” to ever had sex in the 8th grade; if answered “yes”, age of first sex was used to determine if they have had sex by 8th grade) (see Appendix G for data cleaning protocol). Of the 1,127 students surveyed in 8th grade, 94.1% had data on the primary outcome of “ever had sex.” Sensitivity analyses were conducted to evaluate whether results differed due to data cleaning methods and covariate adjustments (see Appendix H for description of sensitivity analyses). 2. Implementation evaluation Evaluations of adherence and fidelity, quality of implementation and youth engagement, control experience, and community context were conducted using both quantitative and qualitative approaches (Appendix I). Adherence and Fidelity. For both the HF and control programs, the evaluation team analyzed how many and how often sessions were offered, what and how much programming was received, what content was delivered, who delivered the material on an annual (6th-8th grade) and cumulative (all three years) basis, and how many visits each of the virtual classrooms received from the target communities. Methods included: 1) totaling the number of sessions for each component (Nu-CULTURE, Rhymin’ it Write, Code A, Parent Workshops, control), 2) calculating average session duration in minutes for each component, 3) calculating dosage (i.e., average number of sessions attended) each year and cumulatively over three years, 4) comparing the number of activities expected to the number of activities completed for each component, 5) totaling the number of staff delivering the HF and control program, 6) reporting on HF staff age, race/ethnicity, languages spoken, community of residence, and training, and 7) totaling the number of visits to each virtual classroom from the target communities during the implementation period (September 2011-August 2014). Fidelity checklists were monitored for unplanned changes to HF and control delivery and program improvements were discussed in weekly meetings with the HF Executive Director. The evaluation team also documented the percent of control students who were exposed to the HF program (due to student transfer between control and HF schools). 5 The single cohort 2 school in Lynn was randomly assigned to condition outside of a pair, but was considered part of the Lynn “pair” for analyses. 16 Quality of implementation and youth engagement. HF health educator-participant interaction quality was calculated as the percentage of observed interactions scored as “high quality” (score 4 out of 5) or “excellent” (5 out of 5). Youth engagement was calculated as a percentage of observations scored as “moderate” (score 4 out of 5) or “active” (5 out of 5). Context. All TPP programming available to both treatment and control students at the school, district, and community levels were documented. IV. Study findings A. Implementation study findings Nu-CULTURE, the classroom-based relationship education component of HF, was implemented in all seven treatment schools enrolled in the study. Over the three year program 89% of the activities were delivered as prescribed by the model (Appendix J). However, the other three in-person HF components—Rhymin’ it Write, Code A, and True Connections—were not implemented as planned (Table IV.1). Rhymin’ it Write was offered to all treatment schools and students. Only schools in Lowell and Lynn opted to participate and students were recruited from the subset of students who returned parent connection forms indicating interest in the Rhymin’ it Write after-school program. Code A was offered to all treatments schools. Only Lowell opted to participate and students were recruited from the subset of students participating in the Rhymin’ it Write after-school program. True Connections was offered to all treatment schools. Only Lowell opted to participate. Table IV.1. Adherence to the Healthy Futures model in-person components % Activities % HF Study # Treatment Implemented as Students/Parents Median Number Schools Prescribed by Participating of Days Component Participating Model (n=597) Attended Nu-CULTURE (6th grade 7 87% 99.5% 8 Nu-CULTURE (7th grade) a 7 88% 83.8% 8 Nu-CULTURE (8th grade) b 7 90% 75.5% 8 Rhymin’ it Write (6th grade) c 4 59% 6.5% 9 Rhymin’ it Write (7th grade) d 3 66% 1.5% 6 Rhymin’ it Write (8th grade) e 3 71% 0.7% 7 Code A f 2 98% 0.2% 9 True Connections g 2 97% 0.5% 4 a 57 activities prescribed per section in the 6th- and 7th-grade Nu-CULTURE curriculum, delivered over 8 days each year b 51 activities prescribed per section in the 8th-grade Nu-CULTURE curriculum, delivered over 8 days c 77 activities prescribed per section in the 6th-grade Rhymin’ it Write curriculum, delivered over 10 weeks (1 day per week) d 41 activities prescribed per section in the 7th-grade Rhymin’ it Write curriculum, delivered over 9 weeks (1 day per week) e 46 activities prescribed per section in the 8th-grade Rhymin’ it Write curriculum delivered over 10 weeks (1 day per week) f 54 activities prescribed in the Code A curriculum delivered over 18 sessions (3 sessions per week for 6 weeks) g 54 activities prescribed in the True Connections curriculum delivered over 4 weeks (1 day per week) 17 Furthermore, despite promoting the virtual classroom components during Nu-CULTURE sessions and on the parent connection forms, these components were infrequently visited by users from the target communities. During the implementation period of September 1, 2011, to August 31, 2014, the 6th- and 7th-grade virtual classroom (onmylevel.org) had 494 visits (25 from Haverhill, 184 from Lowell, 28 from Lynn, and 257 visits from non-study communities) and the 8th-grade virtual classroom (doinitright.org) had 281 visits (6 from Haverhill, 54 from Lowell, and 9 from Lynn, and 212 visits from non-study communities). In addition, the parent website (ontheirlevel.org) received 323 visits (1 from Haverhill, 65 from Lowell, 5 from Lynn, and 252 visits from non-study communities). 6 Furthermore, these may be over-estimates as the websites may have been accessed by students and parents not in the evaluation and may reflect HF staff visits to the websites as HF had two offices in the target communities (Lowell and Lynn) during this time period. Adherence and fidelity to the Nu-CULTURE model. While Nu-CULTURE, the classroom-based relationship education component of HF, was implemented in all seven treatment schools enrolled in the study with high adherence and fidelity, variations occurred at the school level (Appendix J). The average student attendance rate over the intended 24 sessions was 76%. 7 The average student attendance rate over the intended sessions was highest in 6th grade (86%) but declined in 7th and 8th grades (73% and 68%, respectively). According to school schedules, the average program duration (intended to be 50 minutes) was 51 minutes in 6th grade, 53 minutes in 7th grade, and 49 minutes in 8th grade. Of the 24 daily connection forms sent home with students over the course of the 3-year Nu-CULTURE program, health educators requested 21 (sessions 1-7 of 8 each year) to be signed and returned to HF. Fewer than 15% of students attending at least one HF session in the respective year returned the seven completed forms (14% in 6th grade, 6% in 7th grade, and 11% in 8th grade), suggesting HF may not have been successful in fostering student-parent interactions through the daily parent connection forms. However, it is possible students completed the forms with their parent but failed to return them to HF. According to the daily fidelity checklists completed by HF health educators, HF delivered the Nu-CULTURE program with high fidelity (more than 80% activities were implemented as prescribed) in the majority of treatment schools across all three years of implementation. 8 In 6th grade, HF implemented Nu-CULTURE in six of the seven treatment schools with high fidelity. HF adapted the program to be delivered over two days (instead of 8) in the seventh school due to school scheduling conflicts. This school only received 49% of the activities prescribed by the model. In 7th grade, HF implemented Nu-CULTURE in six of the seven treatment schools with high fidelity. For the seventh school, HF adapted the program to be delivered over four days due to school scheduling conflicts. 9 This school only received 41% of the activities prescribed by the model. In 8th grade, HF again implemented Nu-CULTURE in six of the seven treatment schools with high fidelity. For the seventh school, HF adapted the program to be delivered over 5 days 6 Google Analytic Reports for September 1, 2011-August 31, 2014. 7Average treatment student attendance was 18 sessions and median student attendance was 21 sessions (of the 24-session program). 8A random sample of these fidelity checklists were verified by comparing the health educator’s self-reported activities against evaluation team observations of the sessions implemented. 9 This was the same school that also received the adapted program in 6th grade. 18 for one of the seven classroom sections due to school scheduling conflicts and inclement weather resulting in school closure. 10 This school received 79% of the activities prescribed by the model. A detailed description of the Nu-CULTURE implementation fidelity by year and school is included in Appendix J. Adherence and fidelity to the Rhymin’ it Write, Code A, and True Connections models. HF did not implement the Rhymin’ it Write, Code A, or True Connections components as intended. First, the 3-year, 30-session Rhymin’ it Write after-school component was implemented in only two of the three study school districts (Lowell and Lynn) and had limited participation by students enrolled in the study (39 in 6th grade, 9 in 7th grade, and 4 in 8th grade). In addition, Rhymin’ it Write was implemented for only 9 of the intended 10 days in 7th grade. According to the weekly fidelity checklists completed by HF health educators, the Rhymin’ it Write program was also not implemented with high fidelity. 11 Second, the 18-session Code A summer component was implemented in only one of the three study school districts (Lowell) and only one student enrolled in the study participated. According to the weekly fidelity checklists completed by the HF health educators, the Code A program was delivered with high fidelity, implementing 98% of the activities prescribed by the model. Third, the 4-session True Connections program that was designed to be available to parents each year was implemented in only one of the three study school districts (Lowell) during the first year of the study (6th grade). Furthermore, parent participation was limited and only three parents of students enrolled in the study participated. According to the weekly fidelity checklists completed by the HF parent coordinator, the True Connections program was delivered with high fidelity, implementing 97% of the activities as prescribed by the model. A detailed description of the Rhymin’ it Write, Code A, and True Connections implementation fidelity by year and school is included in Appendix J. Quality of Nu-CULTURE implementation and youth engagement. 12 The evaluation team rated 100% of 6th- and 7th-grade sessions observed and 87% of 8th grade sessions as excellent. Youth were actively engaged in 90% of the observed sessions in 6th grade, 100% of the observed sessions in 7th grade, and 95% of the observed sessions in the 8th grade. Quality ratings for Nu-CULTURE during each grade level are presented in Appendix K. Experiences of the control group. HF implemented the 3-year, 6-session control program during the school day each year (September 2011 – October 2014). The average student attendance rate of the 6 intended sessions was 76%. 13 According to school schedules, the 10 This was a different school than the school that received the adapted program in 6th and 7th grade. 11Only 59% of Rhymin’ it Write activities were implemented as prescribed by the model in 6th grade, 66% in 7th grade, and 71% 8th grade. 12 The evaluation team observed 17of the 323 classroom sessions implemented in 6th grade, 37 of the 313 sessions implemented in 7th grade, and 37 of the 338 sessions implemented in 8th grade (37 observations). 13Average control student attendance was 5 sessions and median student attendance was 6 sessions (of the 6- session program) 19 average program duration (intended to be 50 minutes) was 52 minutes in 6th grade, 56 minutes in 7th grade, and 53 minutes in 8th grade. Healthy Futures health educator training and characteristics. Over the course of the project, 17 health educators were hired and trained to facilitate the HF program (Table IV.2) with an average of 9 health educators implementing the program each year. Educator characteristics matched the intended cultural diversity and community representation with the exception of one educator who was from Boston (a non-study community). Table IV.2. Healthy Futures Educators # Educators Age Component Trained Range Race/Ethnicity Languages Spoken City Black Non-Hispanic (3), Nu-CULTURE, Black Hispanic (1), English (16), Khmer Boston (1) Rhymin’ It Write, 21-37 Cambodian (3), Hispanic (3), Spanish (6), Thai Lowell (9) Code A a 16 years (3), White (6) (1) Lynn (6) True Connections b 1 49 years Hispanic English, Spanish Lynn (1) a Age range from the final year of implementation (2013-2014) b Age range from the only year of implementation (2011-2012) HF health educators attended 14 days of training for the 6th- and 7th-grade curricula each and 17 days of training for the 8th-grade curriculum (greater than the minimum 5 days per year expected) conducted by the HF Executive Director, Senior Project Manager, and Lead Educator (Appendix L, Table L.1). Trainings included: 1) observation of an experienced educator teaching each lesson (outside of class), 2) reviewing curriculum, 3) “teaching back” the curriculum to identify strengths and areas for improvement, 4) developing an individualized presentation style, 5) classroom and behavior management, 6) participant engagement, 7) developing personal motivation to repeatedly teach the same content, 8) facilitating a discussion, 9) how to answer student questions age-appropriately and medically accurately, 10) session logistics (e.g., computer equipment, materials, accountability), 11) mandatory reporting issues (e.g., reported sexual abuse, personal or interpersonal violence), and 12) participant evaluation forms. Context. The evaluation team monitored treatment and control student mobility between schools for 6th-8th grade: 13 control students (2%) were exposed to an average of 7 days of the 7th grade Nu-CULTURE program and 18 control students (2%) were exposed to an average of 7 days of the 8th grade Nu-CULTURE program. On the 8th-grade follow-up survey, 8% (39 of 488) of treatment and 10% (64 of 639) of control students reported exposure to programs that address sexual activity or pregnancy other than HF (Appendix M, Table M.1). On the 9th-grade follow-up survey, 16% of treatment and control students (77 of 476 and 81 of 520, respectively) reported exposure to programs that address sexual activity or pregnancy other than HF in 9th grade (Appendix M, Table M.2). Programs available in the study communities were identified by the evaluation team during regular web searches (Appendix M, Table M.3) and were similar to those reported by students. 20 B. Impact study findings An impact evaluation was conducted to assess the effectiveness of HF on reducing the prevalence of ever having vaginal sex overall and by student gender and race/ethnicity, as well as other secondary outcomes of sex in the past three months and unprotected sex in the past 3 months by the 8th-grade follow-up. The impact on prevalence of ever having vaginal sex by the 9th-grade follow-up was also examined. 1. Impact study findings: primary research question HF did not significantly reduce the prevalence of students who have ever had vaginal sex by th 8 -grade follow-up (Table IV.3). Overall, 6.4% of treatment students reported ever having vaginal sex, compared to 9.4% of control students. The estimated impact of 3% point difference was not statistically significant (p = 0.07). To assess the robustness and sensitivity of findings, additional analyses were conducted: 1) using the raw data without any data cleaning, 2) using data that were cleaned for within-survey logic, but not for across-survey logic, and 3) without controlling for baseline student demographic covariates. In two of the three sensitivity analyses conducted using the raw data and the model without baseline covariate adjustments, there was a statistically significant overall program impact in reducing prevalence of ever vaginal sex (both p = 0.04) and the third analysis showed a similar finding to the benchmark approach (p = 0.07), which suggest some evidence of program effectiveness (see Appendix H for results of the sensitivity analyses). Table IV.3. Post-intervention estimated effects using data from the immediate 8th-grade follow-up survey to address the primary research question Treatment Compared to . Treatment Control Control % % Outcome measure (Standard error) (Standard error) Mean difference (p-value) Ever had vaginal sex 6.4% (1.9) 9.4% (2.1) -3.0% (0.07) Source: Primary outcome measured at immediate 8th-grade follow-up survey was collected during the 2013-2014 school year from the 14 Cohort 1 schools and during the 2014-2015 school year for the single Cohort 2 school. n=1,060 (HF: 464, Control: 596). Notes: See Table III.3 for a detailed description of measure and section III for a description of the impact estimation methods. 2. Impact study findings: secondary research questions The analyses revealed impacts on some measures (Table IV.4). The HF program had a statistically significant impact in reducing the prevalence of ever having vaginal sex at the 8th- grade follow-up for females and Hispanic students. Specifically, 2.9% of females in the treatment group reported ever having sex versus 6.4% of those in the control group (p = 0.04). Among Hispanic students, 6.3% in the treatment group versus 15.7% in the control group reported ever having sex (p = 0.002). No statistically significant differences were observed for males or other non-Hispanic students. White non-Hispanic students were the only group with higher rates of ever having sex by 8th grade in the treatment (13.4%) than in the control group (7.8%), although this difference was not statistically significant. At the 8th-grade follow-up, no statistically significant differences were found in the prevalence of sex in the past 3 months or 21 birth control use in the past 3 months between the treatment and control groups. At the one-year 9th grade follow-up, equal proportions of students (23%) in both the treatment and control groups reported ever having had sex. Sensitivity analyses were consistent with the benchmark approach (see Appendix H). Table IV.4. Post-intervention estimated effects using data from the immediate 8th-grade follow-up and one-year follow-up 9th-grade survey to address secondary research questions Treatment Compared to .. Treatment Control Control Mean or % Mean or % Mean difference Outcome measures (Standard error) (Standard error) (p-value) Ever had vaginal sex by 8th grade, by gender . . . and race/ethnicity Males 9.9% (3.4) 12.2% (3.3) -2.3% (0.43) Females 2.9% (1.7) 6.4% (2.9) -3.4% (0.04)* Hispanic 6.3% (3.0) 15.7% (5.8) -9.4% (0.00)* White non-Hispanic 13.4% (7.6) 7.8% (3.7) +5.6% (0.29) Other, non-Hispanic 2.8% (3.9) 3.1% (4.2) -0.3% (0.83) Had sex in the past 3 months by 8th grade 1.2% (5.1) 1.3% (5.7) -0.1% (0.72) Had unprotected sex in past 3 months without an effective method of birth control by 8th grade 0.01% (0.0) 0.01% (0.0) 0.0% (0.47) Ever had vaginal sex by 9th grade 22.8% (2.6) 22.5% (2.5) -0.3% (0.92) Source: Secondary outcomes measured at immediate 8th-grade follow-up survey were collected during the 2013- 2014 school year from the 14 Cohort 1 schools and during the 2014-2015 school year from the single Cohort 2 school. For the ever vaginal sex outcome by subgroup: n=1,060 (HF: 464, Control: 596). For outcomes related to past 3 months sex and contraception use: n=1,029 (HF: 453, Control: 576). Secondary outcomes measured at one year 9th-grade follow-up survey were collected during the 2014-2015 school year from the 14 schools (Cohort 1). For the ever had vaginal sex outcome: n=895 (HF: 437, Control: 458). Notes: See Table III.3 for a detailed description of each measure and section III for a description of the impact estimation methods. *Denotes statistically significant difference at p < 0.05. V. Conclusion This is one of the first rigorous evaluations of a longitudinal, school-based multi-level TPP program that is designed to increase knowledge and skills at the individual level and enhance the physical and social systems that influence teens’ sexual decision-making. At the end of the 8th grade, after students in the treatment group were offered the Nu-CULTURE curriculum for three consecutive school years, this study did not find an overall impact in reducing the prevalence of ever having had vaginal sex, with results just above the significance threshold. However, two of the three sensitivity analyses did demonstrate evidence of effectiveness and the third also showed a similar significance level as in the benchmark approach. Additionally, the program did reduce 22 the prevalence of ever vaginal sex among females and Hispanic youth. There was no impact on the prevalence of sex or unprotected sex without effective birth control within the past three months. Furthermore, there was no impact in reducing the prevalence of ever having had sex one year later at the 9th grade, suggesting the program is more effective at delaying sex in the early years. Given the negative consequences of teen pregnancies, including the associated costs, reallocating resources to ensure continued health education into high school as well as to create systems and environments that support healthy behaviors is important to adolescent health. Additional research should be conducted to better understand these associations. Given Nu-CULTURE’s lack of an impact in reducing the prevalence of male students who reported having had vaginal sex, additional research is needed to determine if the gender differences in program impact exist if the study were replicated. The evidence base in changing teen males’ sexual behavior is growing and existing studies indicate that programs that 1) engage males in thinking positively about their future and developing skills to achieve their goals (Clark et al., 2005), 2) include out-of-school components (Coyle et al., 2006), and (3) discuss gender equity (Ricardo et al., 2010) have resulted in positive outcomes among males. While Nu- CULTURE includes many theory-driven activities designed to build students’ positive vision for their future, the limited participation in the HF out-of-school components may be one explanation for the lack of HF impact on males in this study. In addition, while the Nu- CULTURE curriculum included a high level discussion about stereotypical gender differences, it did not specifically address gender roles related to sexual and reproductive decision-making (e.g., power dynamics, negotiating contraception use). With respect to Nu-CULTURE’s lack of an impact in reducing the prevalence of non- Hispanic students who reported having had vaginal sex, additional research is needed to determine if the ethnicity differences in program impact exist if the program was replicated or adapted. For example, the Nu-CULTURE program may be implemented by health educators who are not representative of the student population or the content may be delivered using a different method (e.g., role-play as opposed to a video). The statistically significant lower rates of vaginal sex among Hispanic students in the treatment versus control group suggest that the program was especially effective for a population that experiences disproportionately high teen pregnancy rates (Centers for Disease Control and Prevention, 2015). Public health models (e.g., community health workers) have demonstrated the important role that personal relationships and knowledge of a community plays in health behavior change (Katigbak, 2015). In addition, family influence and abstinence are highly valued among Hispanic females (Villarruel, 1998) and programs emphasizing the importance of delaying parenthood to pursue goals (specifically educational goals) are highly effective among Hispanics (Scott, 1998). Nu-CULTURE emphasizes the importance of positive relationships with family and peers and strongly recommends abstinence for teens. Furthermore, research shows that television and radio are proven, effective channels in targeting Hispanics (Sonderup, 2010). HF included diverse teaching methods in the Nu-CULTURE curriculum including an audio vignette and a number of videos. These teaching methods may also have contributed to the greater program impact on Hispanic youth. While it is possible that HF’s multi-level approach may contribute to the impacts found, significant findings can only be attributed to the Nu-CULTURE classroom-based component since the after-school, summer, parent, and virtual components were not received by the majority 23 of study participants. The limited uptake of these other components was most likely a result of competing priorities for schools, students, and parents during out-of-school time and the self- selection process HF used to recruit students and parents. Additionally, while HF was a multi- level program, it did not address the community-level risk and protective factors (e.g., socioeconomic determinants) that affect teen sexual decision making and behavior (Penman-Aguilar, 2013). This study is not without its limitations. First, while the sample includes youth from three diverse Massachusetts communities, it is not representative of the U.S. youth population, Hispanic youth or female teens, and hence the results may not be generalizable to all adolescents. Second, program impact estimates may either be under or overestimated depending on the accuracy and honesty of students’ self-reported sex behavior which can be influenced by underlying biases and motivations such as social desirability to respond a certain way. Data collectors emphasized survey anonymity and faux surveys were administered to non-study participants in the classroom to ensure not only equal treatment but student confidentiality and comfort. This study has several strengths. First, minimal student attrition increases confidence in our findings. Second, this study is among the first to look at the impact of a longitudinal, school- based, multi-level program on sexual activity. HF’s ability to successfully implement the three year Nu-CULTURE classroom-based program is credited to HF’s efforts to maintain and enhance relationships with school administrators and teachers and to implement the program during regularly scheduled classes (e.g., health, science, physical education, social studies, etc.). Future research is needed to inform TPP interventions that effectively target teen pregnancy at the individual, interpersonal, organizational, and community levels. VI. References Bandura A. Social cognitive theory and exercise of control over HIV infection. In: DiClemente RJ, Peterson JL, eds. Preventing AIDS: Theories and Methods of Behavioral Interventions. New York: Plenum Press; 1994:25-59. Calise TV, Howland J, Bliss CA, Flaherty KL, & Gordon R. Differential Effects of the Healthy Futures Program on Sexual Risk Knowledge, Beliefs, Behavioral Intentions and Behaviors by Middle School vs. High School Grades. Journal of Health Behavior & Public Health. 2012; 2(3):8–14. Carter-Jessop L, Franklin LN, Heath JW, Jr., Jimenez-Irizarry G, & Peace MD. Abstinence education for urban youth. J Community Health. Aug 2000; 25(4):293–304. Centers for Disease Control and Prevention. 1995-2013 High School Youth Risk Behavior Survey Data. Retrieved from nccd.cdc.gov/YouthOnline. Accessed May 2015. Clark, L. F., Miller, K. S., Nagy, S. S., Avery, J., Roth, D. L., Liddon, N., & Mukherjee, S. (2005). Adult identity mentoring: Reducing sexual risk for African-American seventh grade students. Journal of Adolescent Health, 37, 337.e1-3337.e10. 24 Coyle, K., Kirby, D., Robin, L., Banspach, S., Baumler, E., & Glassman, J. (2006). All4You! A randomized trial of an HIV, other STD, and pregnancy prevention intervention for alternative school students. AIDS Education & Prevention, 18(3), 187–203. Hamilton BE, Martin JA, Osterman MJK, & Curtin SC. (2014). Births: Preliminary data for 2013. National vital statistics reports 63(2). Hoffman SD & Maynard RA (Eds). (2008). Kids having kids: Economic costs and social consequences of teen pregnancy (2nd ed.). Washington, DC: Urban Institute Press. Katigbak C, Van Devanter N, Islam N, & Trinh-Shevrin C. Partners in health: a conceptual framework for the role of community health workers in facilitating patients' adoption of healthy behaviors. Am J Public Health. 2015 May;105(5):872–80. Koh H. The Teen Pregnancy Prevention Program: An Evidence-Based Public Health Program Model. J Adolescent Health. 2014; 54:S1-S2. Massachusetts Budget and Policy Center’s Children’s Budget. School-Based Health Programs. Last Updated January 2015. Retrieved from children.massbudget.org/school-based-health- programs. Accessed May 2015. Massachusetts Department of Elementary and Secondary Education and the Massachusetts Department of Public Health. Health and Risk Behaviors of Massachusetts Youth, 2009. McLeroy KR, Bibeau D, Steckler A, Glanz K. An Ecological Perspective on Health Promotion Programs. Health Education Quarterly 1988; 15:351–77. The National Campaign to Prevent Teen and Unplanned Pregnancy. (2013). Counting it up: The public costs of teen childbearing: key data. Washington, DC: The National Campaign to Prevent Teen and Unplanned Pregnancy. Retrieved from thenationalcampaign.org/sites/default/files/resource-primary-download/counting-it-up-key-data- 2013-update.pdf. O’Keefe G, Cohen B, & Nyberg S. Massachusetts Births 2007. Massachusetts Department of Public Health. Retrieved from mass.gov/eohhs/docs/dph/research-epi/birth-report-2007.pdf. Accessed May 2015. Penman-Aguilar, Carter M, Snead MC, & Kourtis AP. Socioeconomic Disadvantage as a social determinant of teen childbearing. U.S. Public Health Rep. 2013 Mar-Apr; 128 Suppl 1:5–22. Ricardo C, Nascimento M, Fonseca V, & Segundo M. (2010). Program H and Program M: Engaging young men and empowering young women to promote gender equality and health. Washington, DC: Pan American Health Organization. Scott ME, Berger A, Selma C, Hickman S, & Moore K. 2014. Research Brief: Preventing Teen Pregnancy among Latinos: Recommendations from research, evaluation, and practitioner experience. Child Trends. 25 Sonderup, L. (2010). Hispanic marketing: A critical market segment. Advertising & Marketing Review. U.S. Department of Health and Human Services. Offices of Adolescent Health and Public Health and Science and Administration on Children, Youth, and Families. Teenage Pregnancy Prevention (TPP): Research and Demonstration Programs and Personal Responsibility Education Program (PREP) Funding Opportunity Announcement and Application Instructions. 2010. Ventura SJ, Hamilton BE, & Mathews TJ. (2014) National and state patterns of teen births in the United States, 1940-2013. National vital statistics reports; vol 63 no 4. Hyattsville, MD: National Center for Health Statistics. Villarruel, A. M. (1998). Cultural influences on the sexual attitudes, beliefs, and norms of young Latina adolescents. Journal for Specialists in Pediatric Nursing, 3, 69–78. 26 Appendix A. Massachusetts teen birth rates, 2007 Table A.1. Massachusetts Cities and Towns with the 20 Highest Teen Birth Ratesǂ, 2007 Teen Birth Rate Municipality Rank Teen Births (per 1,000) State Total - 4,944 22.0 Holyoke 1 144 95.4 Springfield 2 512 84.3 Chelsea 3 82 82.0 Southbridge 4 43 77.1 Lawrence 5 230 76.0 New Bedford 6 197 66.7 Fall River 7 167 59.0 Lynn* 8 175 56.7 Lowell* 9 215 54.2 Pittsfield 10 67 52.7 Brockton 11 171 47.0 Revere 12 48 45.6 Fitchburg 13 71 45.4 Chicopee 14 76 43.7 Everett 15 43 40.6 Leominster 16 46 36.7 Worcester 17 251 35.7 Haverhill* 18 67 35.1 Taunton 19 55 33.7 Attleboro 20 35 30.9 Source: O’Keefe G, Cohen B, and Nyberg S. Massachusetts Births 2007. Massachusetts Department of Public Health. February 2009. ǂRates are per 1,000 females ages 15-19 per city/town. *Targeted communities enrolled in the evaluation study. 27 Appendix B. Healthy Futures logic model The goals of Healthy Futures are to: (1) Influence attitudes, behavioral and normative beliefs, and self-efficacy regarding healthy relationship choices and risk-reduction behaviors by creating an environment where healthy choices are supported by peers and family. (2) Strengthen beliefs about the benefits of delaying sexual activity. (3) Increase intentions to delay sexual activity. (4) Develop leaders to be positive role models. 28 Appendix C. Data used to address implementation research questions Types of data used to assess whether the Healthy Futures components and control program Frequency/sampling of data Implementation Element were implemented as intended collection Party responsible for data collection Adherence Adherence Adherence Adherence How many and how often . . . were Healthy Futures sessions offered? Number of 1) classroom-based Number of Healthy Futures sessions Number of Healthy Futures sessions Healthy Futures staff (schedules, fidelity program, 2) Rhymin’ it Write delivered for each component delivered captured for each component checklist) and evaluation staff (direct after-school program, 3) True captured by confirmed schedules and and documented in the Tracking observations) collected data on number Connections parent program, cross checked with online educator Database. Number of sessions delivered of sessions for each component. The and 4) Code A summer peer fidelity checklists and direct program sampled at completion of each section evaluation team documented total leadership program sessions observations. and component. number of sessions for each component delivered in the Tracking Database. Average duration of 1) Length (number of minutes) of Duration of all Healthy Futures sessions Healthy Futures staff (schedules, fidelity classroom-based program, 2) Healthy Futures sessions delivered delivered captured for each component checklist) and evaluation staff (direct Rhymin’ it Write after-school for each component captured by and documented in the Tracking observations) collected data on duration program, 3) True Connections confirmed schedules and cross Database. Duration of session sampled of session for each component. The parent program, and 4) Code A checked with direct program at completion of each component evaluation team documented duration of summer peer leadership observations. section. sessions for each component in the program sessions delivered Tracking Database. What and how much Healthy . . . Futures programming was received? Average attendance for 1) Session attendance for each Healthy Attendance for each Healthy Futures Healthy Futures staff (attendance forms, classroom-based program, 2) Futures component captured by component captured using detailed fidelity checklists) and evaluation staff Rhymin’ it Write after-school attendance forms and cross checked attendance forms. Attendance data (direct observations) collected program, 3) True Connections with online daily educator fidelity captured in the Tracking Database. attendance data for each component parent program, and 4) Code A checklists and direct program Attendance sampled at the completion of session. The evaluation team summer peer leadership observations. each component section. documented duration of sessions for program sessions delivered each component in the Tracking Database. 29 Types of data used to assess whether the Healthy Futures components and control program Frequency/sampling of data Implementation Element were implemented as intended collection Party responsible for data collection What Healthy Futures . . . content was delivered? Healthy Futures content was Number of activities prescribed and All implemented content was reported by Healthy Futures staff (fidelity checklists) delivered to youth through 1) implemented captured by fidelity Healthy Futures staff to the evaluation and evaluation staff (direct observations) classroom-based program, 2) checklists and direct program team staff using online daily fidelity collected content data for each Rhymin’ it Write after-school observations. checklists. Representative sample of 9% component session. The evaluation program, 3) Code A summer of classroom-based sessions were team documented session content for peer leadership program selected for observation. Additional each component in the Tracking sessions components were observed but not Database. reported on due to limited program uptake. Fidelity checklists were monitored weekly. Healthy Futures content was Number of activities prescribed and All implemented content was reported by Healthy Futures staff (fidelity checklists) delivered to parents through implemented captured by fidelity Healthy Futures staff to the evaluation and evaluation staff (direct observations) True Connections parent checklists and direct program team staff using online fidelity checklists. collected content data for each program sessions observations. No parent program sessions were component session. The evaluation selected for observation due to limited team documented session content for program uptake. each component in the Tracking Database. Healthy Futures content was Number of visitors overall and from Total number of visits was reported by Healthy Futures staff (Google Analytics delivered to students and each of the target communities Healthy Futures staff to the evaluation reports) collected website traffic data for parents through virtual captured by Google Analytics. team staff using Google Analytic reports the implementation period. The classrooms by website (onmylevel.org, evaluation team documented traffic in doinitright.org, ontheirlevel.org). the Tracking Database. Who delivered the Healthy . . . Futures material? List of Healthy Futures staff List of Healthy Futures staff hired and Data on all Healthy Futures staff Healthy Futures Executive Director delivering the program trained to implement program members reported by Healthy Futures maintained staff lists and reported to the components to youth and components (i.e., health educators, Executive Director to JSI evaluation evaluation team as needed. The parents parent coordinator). team. Reported as needed or biannually. evaluation team documented staff in the Tracking Database. List of Healthy Futures List of Healthy Futures staff trainings Data on staff trainings and staff Healthy Futures Executive Director trainings and staff attendance managed by the Healthy Futures attendance at trainings reported by maintained a list of Healthy Futures at trainings Executive Director. Healthy Futures Executive Director to trainings and staff attendance at the evaluation team. Reported trainings and reported to the evaluation biannually. team every six months. The evaluation team documented trainings and attendance in the Tracking Database. 30 Types of data used to assess whether the Healthy Futures components and control program Frequency/sampling of data Implementation Element were implemented as intended collection Party responsible for data collection Quality Quality Quality Quality Quality of Healthy Futures Evaluation team observations of Quality of Healthy Futures staff- Evaluation staff collected observation staff-participant interactions for interaction quality using tool participant interactions was reported by data on Healthy Futures staff-participant 1) classroom-based program, developed by HHS and Healthy the evaluation team using HHS Program interactions for 9% of implemented 2) Rhymin’ it Write after-school Futures fidelity observation tool Observation Form and the Fidelity classroom-based sessions. The program, 3) True Connections developed by JSI. Observation Form. Representative evaluation team documented parent program, and 4) Code A sample of 9% of the classroom-based observation data in the Tracking summer peer leadership sessions were selected for observation. Database. program sessions A sample of the after-school/summer programs were observed but not reported on due to limited uptake. Quality of youth engagement Evaluation team observations of Quality of Healthy Futures youth Evaluation staff collected observation with Healthy Futures program youth engagement with Healthy engagement was reported by the data on Healthy Futures youth components Futures program using tool evaluation team using HHS Program engagement for 9% of implemented developed by HHS and Healthy Observation Form and the Fidelity classroom-based sessions. The Futures fidelity observation tool Observation Form. Representative evaluation team documented developed by JSI. sample of 9% of the classroom-based observation data in the Tracking sessions were selected for observation. Database. A sample of the after-school/summer programs were observed but not reported on due to limited uptake. Control Control Control Control How many and how often . . . were control sessions offered? Number of control program Number of control program sessions Number of control program sessions Healthy Futures staff (schedules) and sessions delivered delivered by confirmed schedules delivered documented in the Tracking the evaluation staff (direct observations) and cross checked with direct Database. Number of sessions delivered collected data on number of control program observations. sampled at completion of each section. sessions. The evaluation team documented total number of control sessions in the Tracking Database. Average duration of control Length (number of minutes) of control Duration of all control sessions delivered Healthy Futures staff (schedules) and program sessions delivered sessions delivered for each captured documented in the Tracking the evaluation staff (direct observations) component captured by confirmed Database. Duration of session sampled collected data on duration of control schedules and cross checked with at completion of each component sessions. The evaluation team direct program observations. section. documented duration of control sessions for each component in the Tracking Database. 31 Types of data used to assess whether the Healthy Futures components and control program Frequency/sampling of data Implementation Element were implemented as intended collection Party responsible for data collection Average attendance for control Session attendance for each control Attendance for each control component Healthy Futures staff (attendance forms) program sessions delivered section captured by attendance forms captured using detailed attendance and the evaluation staff (direct and cross checked with direct forms. Attendance data captured in the observations) collected attendance data program observations. Tracking Database. Attendance sampled for each control session. The evaluation at the completion of each component team documented duration of control section. sessions for in the Tracking Database. What control content was . . . delivered? Control content was delivered Number of control activities covered All implemented control content was Healthy Futures staff (control program to youth through a classroom- captured by detailed control program reported by Healthy Futures staff to the outline) and the evaluation staff (direct based program outline and direct program evaluation team staff using a detailed observations) collected content data for observations. control program outline. The evaluation control sessions. The evaluation team team collected data on control content documented control session content in through observations of control the Tracking Database. sessions. Representative sample of 5% of control classroom-based sessions were selected for observation. Who delivered the control . . . material? List of Healthy Futures staff List of health educators hired and Data on all control staff members Healthy Futures Executive Director delivering the control program trained to implement control program. reported by Healthy Futures Executive maintained control staff list and reported components to youth and Director to the evaluation team as to the evaluation team as needed. The ongoing monitoring of staff needed (i.e., any loss or addition of evaluation team monitored control delivering control program to staff). program staff to avoid contamination avoid contamination and documents staff in the Tracking Database. List of control curriculum List of control staff trainings managed Data on staff trainings and control staff Healthy Futures Executive Director trainings and control staff by the Healthy Futures Executive attendance at trainings reported by maintained a list of Healthy Futures attendance at trainings Director. Healthy Futures Executive Director to trainings and control staff attendance at the evaluation team biannually. trainings and reported to the evaluation team every six months. The evaluation team documented trainings and attendance in the Tracking Database. 32 Types of data used to assess whether the Healthy Futures components and control program Frequency/sampling of data Implementation Element were implemented as intended collection Party responsible for data collection Experiences of control . . . group Monitor student mobility List of students who have transferred Data on student transfers reported by Healthy Futures Scheduling Coordinator between schools and out of or between schools obtained each district/school to Healthy Futures provided the evaluation team with a list conditions from district and school and the evaluation team at the beginning of students and their current school administrators to monitor student of each school year and at the time each assignment at the beginning of each mobility between schools and school is scheduled. school year and as each school was conditions. scheduled. The evaluation team documented student status and noted mobility between schools or conditions in the Tracking Database. Context Context Context Context Self-reported information on Self-reported data from classroom- Data on treatment and control The evaluation team collected data at other teen pregnancy based pencil-and-paper survey on participant exposure to teen pregnancy 6th-grade baseline and immediate follow- prevention experiences exposure to and information about prevention programming. Data collected up, 7th-grade follow-up, 8th-grade follow- other teen pregnancy prevention twice in 6th grade and once per year in up, and 9th-grade follow-up surveys. The programming. 7th, 8th, and 9th grades. evaluation team documented control participant exposure to teen pregnancy prevention programming in the Tracking Database. Bi-monthly web searches of School- and community-level data Data on school and community JSI evaluation team collected data on programming offered in study from strategic web searches pregnancy prevention, sex education, school-based and community teen school districts and identifying other teen pregnancy and HIV/STI prevention programming. pregnancy prevention, sex education, communities prevention, sex education, or HIV/STI Data collected every other month. and HIV/STI prevention programming prevention programming in offered in study school districts and participating school districts and communities. The evaluation team communities. followed up with programs identified to learn more about their reach and participants (e.g. age) and documents results in the Tracking Database. Interviews with key school School-level data from in-person Data on school pregnancy prevention, Healthy Futures Executive Director district staff interviews identifying other teen sex education, and HIV/STD prevention maintained close communications with pregnancy prevention, sex education, programming. Interviews conducted at a district-level and school-level staff in or HIV/STD prevention programming minimum yearly and more frequently if participating districts to monitor other in participating school districts. necessary. teen pregnancy prevention programming. Healthy Futures Executive Director reported findings to the evaluation team to document in the Tracking Database. 33 Types of data used to assess whether the Healthy Futures components and control program Frequency/sampling of data Implementation Element were implemented as intended collection Party responsible for data collection Other teen pregnancy District- and school-level data from Data on district and school staff. Healthy Futures Executive Director prevention programming in-person interviews identifying Interviews conducted at a minimum maintained close communications with available or offered to study turnover in key school staff. yearly and more frequently if necessary. district-level and school-level staff in participants (both Healthy participating districts to monitor staff Futures and control students) turnover. Healthy Futures Executive Director reported findings to the evaluation team to document in the Tracking Database. External events affecting Community-, district-, and school- Data on communities, districts, and Healthy Futures Executive Director and implementation (both Healthy level data from news, web searches, schools. External events are the evaluation team maintained close Futures and control students) and in-person interviews identifying documented as they occur, interviews communications with community-, external events affecting conducted at a minimum yearly and district-, and school-level staff to monitor implementation. more frequently if necessary. events that may affect implementation Healthy Futures Executive Director reported findings to the evaluation team to document in the Tracking Database. Substantial unplanned Confirmed school schedules, outlines Data on school requests for program Healthy Futures Scheduling Coordinator adaptation(s) developed by Healthy Futures adaptations (e.g., the number of days (school schedule), Healthy Futures curriculum development team to school will allow Healthy Futures Executive Director (adaptation guide), inform Health Educator adaptations, program to implement). Significant Healthy Futures Health Educators online daily fidelity checklists, and adaptations occur infrequently and are (fidelity checklists), and the evaluation direct observations by the evaluation documented immediately by Healthy team (direct observations). Healthy team. Futures staff and the evaluation team. Futures reported data to the evaluation team to document in the Tracking Database. 34 Appendix D. Cluster and youth sample sizes by treatment status Treatmen Control Total Treatment Total Control . Time period t sample sample response response sample size response rate size size rate rate Number of Clusters . . . . . . . 1. At beginning of study (for analysis of 15 7 8 . . . 8th grade results) . 1a. At beginning of study (for analysis of . . . . 14 7 7 9th grade results) 2. Contributed at least one youth at 15 7 8 100 100 100 baseline Baseline 3. Contributed at least one youth at Immediate 6th- 100 100 100 15 7 8 follow-up grade follow-up 4. Contributed at least one youth at Immediate 7th- 100 100 100 15 7 8 follow-up grade follow-up 5. Contributed at least one youth at Immediate 8th- 100 100 100 15 7 8 follow-up grade follow-up 5a. Contributed at least one youth at 1-year follow-up 9th 100 100 100* 14* 7 7* follow-up (from high schools) grade Number of Youth for 8th-Grade Sample for Analysis of Primary . . . . . . . Research Outcome: Ever had Sex 6. In non-attriting clusters/sites at time of assignment . 2,346* 1,055 1,291* . . . 7. Who consented and assented . 1,344 597 747 57.3 56.6 57.9 8. Contributed a baseline survey . 1,184 518 666 88.1 (50.5) 86.8 (49.1) 89.2 (51.6) Immediate 6th- 9. Contributed a follow-up survey grade follow-up 1,294 587 707 96.3 (55.2) 98.3 (55.6) 94.6 (54.8) Immediate 7th- 10. Contributed a follow-up survey grade follow-up 1,232 533 699 91.7 (52.5) 89.3 (50.5) 93.6 (54.1) 11. Contributed data to 8th grade ever Immediate 8th- had sex primary outcome grade follow-up 1,060 464 596 78.9 (45.2) 77.7 (44.0) 79.8 (46.2) 35 Treatmen Control Total Treatment Total Control . Time period t sample sample response response sample size response rate size size rate rate Number of Youth for 9th-Grade Sample for Analysis of Secondary Research . . . . . . . Outcome: Ever had Sex 6a. In non-attriting clusters/sites at time of assignment . 2,138* 1,055 1,083* . . . 7a. Who consented and assented . 1,247 597 650 58.3 56.6 60.0 8a. Contributed a baseline survey . 1,094 518 576 87.7 (51.2) 86.8 (49.1) 88.6 (53.2) Immediate 6th- 9a. Contributed a follow-up survey grade follow-up 1,220 587 633 97.8 (57.1) 98.3 (55.6) 97.4 (58.4) Immediate 7th- 10a. Contributed a follow-up survey grade follow-up 1,146 533 613 91.9 (53.6) 89.3 (50.5) 94.3 (56.6) Immediate 8th- 11a. Contributed a follow-up survey grade follow-up 1,046 488 558 83.9 (48.9) 81.7 (46.3) 85.8 (51.5) th 12a. Contributed data to 9th-grade ever 1-year follow-up 9 had sex secondary outcome grade 895 437 458 71.8 (41.9) 73.2 (41.4) 70.5 (42.3) *One control school enrolled during the 2012-2013 school year, one year after the initial start of the study; students from this school (Cohort 2) were followed for 3 years only and were not eligible to have a 1-year follow-up in the 9th grade. There were 208 eligible youth in the school at the time of random assignment; 97 students consented/enrolled and 111 students did not consent/did not return consent form. The effective total number of schools eligible for the 9th-grade follow-up was 14 overall and 7 for the control group. The effective total number of youth eligible for the 9th-grade follow-up was 2,138 (2,346-208) overall and 1,083 (1,291- 208) for the control group. The percentages in parentheses in rows 8-11 and 8a-12a represent the response rates that were calculated using the denominator of all youth in non-attriting clusters at time of assignment (including those students who did not consent). 36 Appendix E. Baseline equivalence of analytic samples used to evaluate secondary research questions Table E.1a. Summary statistics of key baseline measures for girls completing the immediate 8th-grade follow-up survey – for evaluating differences in the ever had sex outcome by gender (n=507) Treatment Treatment Treatment Control versus versus control control Baseline Demographics Mean or % (standard Mean or % (standard Mean p-value of deviation) deviation) difference difference Age (years) 12.2 (.58) 12.2 (.52) 0.0 0.33 Race/ethnicity . . . . Hispanic 39.9 42.5 -2.6 0.83 White, non-Hispanic 34.3 23.8 10.5 0.42 Asian/Black/Other, non- Hispanic 25.8 33.7 -7.9 0.63 Have a boy/girlfriend 18.8 18.0 -0.8 0.46 Sample size 213 294 . . Table E.1b. Summary statistics of key baseline measures for boys completing the immediate 8th-grade follow-up survey – for evaluating differences in the ever had sex outcome by gender (n=553) Treatment Treatment . Treatment Control versus versus control control Mean or % (standard Mean or % (standard Mean p-value of Baseline Demographics deviation) deviation) difference difference Age (years) 12.2 (.60) 12.3 (.57) -0.1 0.09 Race/ethnicity . . . . Hispanic 40.2 37.1 3.1 0.44 White, non-Hispanic 31.5 26.5 5.0 0.70 Asian/Black/Other, non- Hispanic 28.3 36.4 -8.1 0.64 Have a boy/girlfriend 20.3 17.2 3.1 0.67 Sample size 251 302 . . 37 Table E.2a. Summary statistics of key baseline measures for Hispanic youth completing the immediate 8th-grade follow-up survey – for evaluating differences in the ever had sex outcome by ethnicity (n=423) Treatment Treatment Treatment Control versus versus control control Mean or % (standard Mean or % (standard Mean p-value of Baseline Demographics deviation) deviation) difference difference Age (years) 12.3 (.62) 12.4 (.60) -0.1 0.54 Gender (female) 45.7 52.7 -7.0 0.21 Have a boy/girlfriend 21.0 18.1 2.9 0.93 Sample size 186 237 . . Table E.2b. Summary statistics of key baseline measures for White non-Hispanic youth completing the immediate 8th-grade follow-up survey – for evaluating differences in the ever had sex outcome by ethnicity (n=302) Treatment Treatment Treatment Control versus versus control control Mean or % (standard Mean or % (standard Mean p-value of Baseline Demographics deviation) deviation) difference difference Age (years) 12.2 (.52) 12.2 (.51) 0.0 0.33 Gender (female) 48.0 46.7 1.3 0.88 Have a boy/girlfriend 17.8 16.7 1.1 0.85 Sample size 152 150 . . Table E.2c. Summary statistics of key baseline measures for Other non-Hispanic youth completing the immediate 8th-grade follow-up survey – for evaluating differences in the ever had sex outcome by ethnicity (n=335) Treatment Treatment . Treatment Control versus versus control control Mean or % (standard Mean or % (standard Mean p-value of Baseline Demographics deviation) deviation) difference difference Age (years) 12.0 (.59) 12.2 (.49) -0.2 0.01 Gender (female) 43.7 47.4 -3.7 0.44 Have a boy/girlfriend 19.8 17.7 2.1 0.86 Sample size 126 209 . . 38 Table E.3. Summary statistics of key baseline measures for youth completing the immediate 8th-grade follow-up survey – for evaluating differences in past 3 months sex and contraception use between treatment and control groups (n=1,029) Treatment Treatment . Treatment Control versus versus control control Mean or % (standard Mean or % (standard Mean p-value of Baseline Demographics deviation) deviation) difference difference Age (years) 12.2 (.58) 12.3 (.55) -0.1 0.15 Gender (female) 46.4 50.0 -3.6 0.21 Race/ethnicity . . . . Hispanic 40.4 38.9 1.5 0.63 White, non-Hispanic 32.9 25.5 7.4 0.77 Asian, non-Hispanic 10.8 19.3 -8.5 0.42 Black, non-Hispanic 4.4 5.4 -1.0 0.99 Other, non-Hispanic 11.5 10.9 0.6 0.65 Have a boy/girlfriend 19.2 17.5 1.7 0.76 Sample size 453 576 . . *Baseline measures of behavioral outcomes were not measured due to students’ young age in 6th grade. P-values derived from models that regressed the demographic variable of interest on covariates that included group assignment and free-reduced lunch match pair indicator used to stratify random assignment, and that adjusted for school clustering. Appendix F. Impact Model Specification Analytic Sample: Data on all participants were pooled across the schools/clusters regardless of participation in the program under the intent-to-treat framework to meet the U.S. Department of Health & Human Services evidence review standards. Schools served as the cluster and students were the primary unit of analysis. Individual level analyses were conducted comparing mean outcomes between the treatment and control groups using logistic regression methods for categorical variables and linear regression methods for continuous variables that account for correlated data at the cluster/school level. Model Specifications & Covariates: To assess the impact of HF on reducing the prevalence of students who have ever had sex by the 8th grade, for example, the binary outcome of ever had sex (yes/no) was the dependent variable. Covariates included a dichotomous group assignment variable, the free/reduced lunch matched pair indicator, and baseline student demographics of gender, age (male/female), race/ethnicity (Hispanic, White, non-Hispanic, other non-Hispanic), and boy/girlfriend status (yes/no). The cluster effect adjusted for non-independence or intra-class correlation among students within the same school. The basic model structure used was: where is the outcome of interest for a particular student in a given school, is a vector for baseline characteristics, is the indicator variable for students in the intervention or control group (1 = HF, 0 = Control), is the random error term, is the cluster random effect, and 39 is the estimated program impact. The same model specifications were used for evaluating impact for all outcomes. Mixed effects regression analyses accounting for site-level clustering were conducted in SAS using PROC GLIMMIX for dichotomous outcomes. Regression-adjusted mean prevalence rates across conditions were reported to improve interpretation. Results were considered statistically significant at the p < 0.05, using two-tailed tests. SAS 9.4 (SAS Institute Inc., Cary, NC) was used for all analyses. 40 Appendix G. Data cleaning protocol Surveys were tracked in a Microsoft Access database that documented student eligibility, enrollment status, school assignment, program attendance, and survey IDs assigned at each data collection time point. “Evaluation” surveys completed by study students were scanned and verified using TeleForm. Stray marks were corrected, text field responses were entered, and Likert scale questions with more than one response selected were set to missing. Fake or “faux” surveys filled out by students without consent forms were destroyed. Data quality checks included evaluating missing, invalid, or inconsistent data. For a few demographics questions, answers in open text fields that corresponded to an existing response category were recoded where appropriate (e.g., if Hispanic/Latino was written in text field, it was recoded Hispanic = yes for ethnicity question; if Chinese was written in text field, it was recoded Asian = yes for race). For behavioral outcomes most questions required a Yes/No response; some questions asked for a number (e.g., age at first sex, number of times). Skip instructions were used to minimize invalid data for questions asked in the 8th and 9th grades. For example, students who responded “yes” to the “ever had sex” question were instructed to continue onto sub-questions about number of partners, age at first sex, pregnancy, and contraception use; students who responded “no” skipped these sub-questions. The following describes the data cleaning decision rules that were implemented. Since students were asked behavioral outcomes questions at two point in time (8th- and 9th-grade surveys), two stages of cleaning were employed. Stage 1 cleaning examined and cleaned data within a survey. Stage 2 cleaning examined and cleaned data within and across surveys. Each cleaning stage produced one cleaned dataset; these two analytic data files and the raw, uncleaned data set were used in sensitivity analyses (Appendix H). Stage 1: Within Survey Data Cleaning 1) Inconsistent responses. Inconsistent responses within a survey were evaluated and the following data editing decision rules were implemented to reinforce all skip logic (i.e., the first response was assumed to be correct): a. If response to “ever had sex” is “no”, then any responses to sub-questions including age at first sex, number of sex partners, ever pregnant, number of times pregnant, and sex in the past 3 months related questions were set to missing to reinforce skip logic as students should have skipped the sub-questions. b. If response to “ever had sex in the past 3 months” is “no”, then any responses to sub- questions on frequency of sex in the past 3 months, sex in the past 3 months without a condom, without effective method of birth control, and number of partners in the past 3 months were set to missing. 2) Outliers. Responses to continuous variables, such as age at first sex, number of sex partners, number of times had sex in the past 3 months were examined. Outliers and invalid responses (e.g., age at first sex older than current age at survey; implausibly high number of sex partners reported: 68, 97, 99) were set to missing. 41 3) Missing data or item non-response within survey. Missing behavioral outcomes data within each survey were evaluated and filled in based on responses to other questions within the survey where available and as appropriate based on the following data editing decision rules, such as: Sex without Sex without Ever Ever condom effective birth pregnant/gotten had Sex past past 3 control past 3 someone sex 3 months months months pregnant Data Decision Rule Yes Missing Missing Missing Missing No change. Cannot determine. Yes Yes Missing Missing -------- No change. Cannot determine. Change sex past 3 months to YES if either unprotected sex/sex without effective birth Yes Missing Yes Yes -------- control is YES. Yes Missing No No -------- No change. Cannot determine. Change ever had sex to YES if any of the 4 other outcomes is Missing Yes Yes Yes Yes YES. No change. Cannot determine Missing Missing Missing Missing Missing (see Stage 2 Cleaning). Skip pattern. If NO to ever had sex, students instructed to skip sub-questions. Sub-questions No Missing Missing Missing Missing were coded as NO for analysis. Stage 2: Within and Across Survey Data Cleaning 1) Inconsistent/Invalid Response Patterns. For students that completed both 8th- and 9th- grade surveys, inconsistent/invalid responses across surveys were evaluated with the data cleaning rule set to accept the first response as the correct response. These rules only applied to “ever” type questions. For example, if a student reported “yes” to ever had sex in 8th grade, then their response to the “ever had sex” question in the 9th grade should also be “yes”. . 8th grade survey 9th grade survey Data Decision Rule First response was considered the correct Ever had sex Yes No/Missing response. Set 9th grade to “Yes”. Ever pregnant or gotten someone First response was considered the correct pregnant Yes No/Missing response. Set 9th grade to “Yes”. 42 2) Missing data or item non-response - across survey fill in. Missing behavioral outcomes data that cannot be filled in using other data provided within a survey were evaluated and filled in as appropriate using data from another survey (8th or 9th grade) where available, based on the following data editing decision rules: a. For students who completed the 9th-grade survey only or have missing response to 8th-grade survey on the following: i. If response to “ever had sex” is “no” in the 9th grade, then missing “ever had sex” data in the 8th grade was set to “no”. All other sub-questions were also set to “no” or null. ii. If response to “ever had sex” is “yes” in the 9th grade, then we determined whether the student had sex by 8th grade by comparing student’s current age at 9th-grade survey and reported age at first sex (if provided). 1. If [age at 9th-grade survey – reported age at first sex] >=1 year (i.e., had 1st sex more than 1 year ago, in 8th grade or earlier), then we filled in the following for 8th-grade data: Ever had sex in the 8th grade = yes and age at first sex in the 8th grade = reported age of first sex at 9th grade. NOTE: No other possible corrections were made to the sex in past 3 months questions in the 8th grade. Pregnancy also cannot be back filled as we do not have date of pregnancy. 2. If the difference between current student age in 9th grade and reported age at first sex was less than 1 year ago (which indicates had first sex within the current year, and not in 8th grade/earlier), then we filled in the following for the 8th-grade data: Ever sex in the 8th grade = no. All other sub-questions were also set to “no” or null. b. For students who completed the 8th-grade survey only or missing response to 9th-grade survey on the following: i. If response to “ever had sex” is “yes” in the 8th grade, then missing “ever had sex” data in the 9th grade was set to “yes”. Age at first sex reported in the 8th grade was used to fill in for missing in the 9th grade. Number of partners reported in the 8th grade was also used to fill in for missing in the 9th grade, although actual count may be higher. ii. If response to “ever pregnant” is “yes” in the 8th grade, then missing “ever pregnant” data in the 9th grade was set to “yes”. Number of pregnancies reported in the 8th grade was also used to fill in for missing in the 9th grade. We examined a listing of the frequency of missing, inconsistent, or invalid data for each outcome, overall and by control and treatment groups for each survey administration period. All data editing decision rules were programmed into SAS. Raw data were inputted and cleaned datasets were created. The survey datasets were merged with the participant tracking database, which contained assignment group, cluster/school ID, program attendance or dosage, and study participation status details. Additional analytic variables were created. For sensitivity analyses, three data files were used: the raw data without any cleaning, cleaned analytic dataset that incorporated all within survey cleaning decision rules, and a second cleaned analytic dataset that incorporated both within and across surveys data cleaning rules as described above. 43 Appendix H. Sensitivity analyses Sensitivity analyses were conducted to evaluate the robustness of findings and to compare if different data cleaning methods and models in terms of covariate adjustments influenced estimated impacts. The Benchmark Approach for both the primary and secondary research questions used the cleaned analytic dataset that includes corrections for missing data by looking at other responses within the survey, corrects and reinforces logic, and fills in missing outcome data using responses students provided on a follow-up or prior survey where available. The benchmark model and all sensitivity approaches, adjust for school clustering and include covariates for baseline student demographics (age, gender, race/ethnicity, boy/girlfriend status in the 6th grade) and the school group assignment and free/reduced lunch match pair indicators. Sensitivity Approach 1 used the raw, un-cleaned analytic dataset that does not correct for missing data such as by looking at other responses within the survey that may be used to fill in missing information, does not correct for incorrect logic, and does not fill in missing outcomes from another survey round where applicable. Model specifications were the same as those described for the benchmark approach. Sensitivity Approach 2 used the cleaned analytic dataset that includes data cleaning within the survey only and does not account for cleaning across surveys where applicable. For example, missing outcomes data on the 8th-grade survey were left as missing and data provided on the 9th- grade survey where available were not used to fill in the missing. Model specifications were the same as those described for the benchmark approach. Sensitivity Approach 3 used the cleaned analytic dataset as in the benchmark approach, but the model excluded baseline student demographic covariates. However, the model included the school group assignment and free/reduced lunch match pair indicator, and adjusted for school clustering, as in the benchmark approach. Tables H.1 and H.2 show the findings from the sensitivity analyses. For the primary research outcome, the sensitivity approach using raw data and the benchmark model without adjustment for baseline covariates found a statistically significant HF impact on reducing ever vaginal sex by 8th grade (both p = 0.04). For all other outcomes, findings were consistent with the benchmark approach. 44 Table H.1. Sensitivity of impact analyses using data from immediate post-8th-grade follow-up survey to address the primary research question Sensitivity Sensitivity Sensitivity Sensitivity Sensitivity Sensitivity Benchmark Benchmark 1: 1: 2: 2: 3: 3: 8th-grade 8th-grade . 8th-grade 8th-grade 8th-grade 8th-grade sample - sample - Bench-mark Bench-mark sample - sample - sample - sample - cleaning cleaning without without raw dataset raw dataset cleaning cleaning within and within and demo- demo- without any without any within within across across graphics graphics cleaning cleaning survey only survey only surveys surveys . Diff. p-value Diff. p-value Diff. p-value Diff p-value Intervention compared with . . . . . . comparison . . Ever had vaginal sex -3.0% 0.07 -3.4% 0.04* -3.1% 0.07 -2.5% 0.04* Source: Primary outcome measured at immediate 8th-grade follow-up survey was collected during the 2013-2014 school year for 14 Cohort 1 schools and during the 2014-2015 school year for the single Cohort 2 school. Notes: See Table III.3 for a detailed description of the measure and section III for a description of the impact estimation methods. *Denotes statistically significant difference at p < 0.05. 45 Table H.2. Sensitivity of impact analyses using data from 8th- or 9th-grade follow-up survey to address secondary research questions . Bench- Bench- Sensitivity Sensitivity Sensitivity Sensitivity Sensitivity Sensitivity mark mark 1: 1: 2: 2: 3: 3: 8th- or 9th- 8th- or 9th- 8th- or 9th- 8th- or 9th- 8th- or 9th- 8th- or 9th- grade grade grade grade grade grade sample - sample - sample - sample - sample - sample - Bench- Bench- cleaning cleaning raw dataset raw dataset cleaning cleaning mark mark within and within and without without within within without without across across any any survey survey demo- demo- surveys surveys cleaning cleaning only only graphics graphics . Diff. p-value Diff. p-value Diff. p-value Diff. p-value Intervention compared with comparison . . . . . . . . Ever had vaginal sex by 8th grade, by . . . . . . . . gender and race/ethnicity Males -2.3% 0.43 -1.4% 0.60 -0.7% 0.81 -1.5% 0.45 Females -3.4% 0.05^* -4.5% 0.01* -4.5% 0.01* -2.9% 0.01* Hispanic -9.4% 0.00* -9.5% 0.00* -9.0% 0.01* -6.8% 0.00* White, non-Hispanic 5.6% 0.29 2.3% 0.23 3.0% 0.14 2.2% 0.49 Other, non-Hispanic -0.3% 0.83 -0.3% 0.34 -1.4% 0.26 -0.1% 0.71 Sex in the past 3 months by 8th grade -0.1% 0.72 -0.3% 0.42 -0.2% 0.61 -0.5% 0.47 Unprotected sex in the past 3 months without an effective method of birth 0.0% 0.47 0.0% 0.33 0.1% 0.53 0.0% 0.63 control by 8th grade Ever had vaginal sex by 9th grade -0.3% 0.92 3.1% 0.31 3.7% 0.23 -1.2% 0.63 Source: Secondary outcomes measured at immediate 8th-grade follow-up survey were collected during the 2013-2014 school year from the 14 Cohort 1 schools and during the 2014-2015 school year from the single Cohort 2 school. Secondary outcomes measured at the one year follow-up 9th-grade survey including ever had vaginal sex were collected during the 2014-2015 from the 14 Cohort 1 schools only. Notes: For estimates of past 3 month outcomes, students who never had sex were coded ‘no’ to sex in the past 3 months and all sub-questions. Students who did not have sex in the past 3 months were coded ‘no’ to unprotected sex in the past 3 months without an effective method of birth control. ^Rounded up to 0.05 from an actual p-value of 0.046. *Denotes statistically significant difference at p < 0.05. See Table III.3 for a detailed description of each measure and section III for a description of the impact estimation methods. 46 Appendix I. Methods used to operationalize each implementation element Implementation Element Methods used to operationalize each implementation element Adherence Adherence Number of (1) classroom-based program, (2) Rhymin’ it Write The total number of sessions for each component is a sum of the sessions captured for each after-school program, (3) True Connections parent program, and grade and/or component in the Tracking Database. (4) Code A summer peer leadership program sessions delivered Average duration of (1) classroom-based program, (2) Rhymin’ it Average session duration for each component was calculated as the average of the Write after-school program, (3) True Connections parent implemented session lengths each grade/component, measured in minutes. program, and (4) Code A summer peer leadership program sessions delivered Average attendance for (1) classroom-based program, (2) Average number of sessions attended for each component was calculated as the average of Rhymin’ it Write after-school program, (3) True Connections the number of sessions that each participant attended. parent program, and (4) Code A summer peer leadership program sessions delivered Proportion of (1) classroom-based program, (2) Rhymin’ it Write Proportion of sessions implemented is the total number of sessions implemented for each after-school program, (3) True Connections parent program, and component compared to the total number of sessions expected for each component. (4) Code A summer peer leadership program sessions implemented as compared to sessions expected based on Healthy Futures model Number of visits to the virtual components by target community. The total number of visits for each website according to Google analytics. Healthy Futures content was delivered to youth through (1) Total number of activities completed for each component based on data reported in the online classroom-based program, (2) Rhymin’ it Write after-school daily fidelity checklist and cross-referenced with representative program observations. program, (3) Code A summer peer leadership program sessions Healthy Futures content was delivered to parents through True Total number of activities completed for each component based on data reported in the online Connections parent program sessions daily fidelity checklist and cross-referenced with representative program observations. (1) Total number of staff delivering the Healthy Futures program is a count of staff members List of Healthy Futures staff delivering the program components implementing the program. (2) Average number of staff members implementing the program at to youth and parents any one point in time during the 3 year Healthy Futures program. List of position requirements or qualifications for Healthy Futures Report of Healthy Futures staff requirements or qualifications based on job descriptions staff delivering program components to youth and parents provided by Healthy Futures Executive Director. List of Healthy Futures trainings and staff attendance at trainings Number of staff trained calculated as simple count of staff members who were trained. Quality Quality Quality of Healthy Futures staff-participant interactions for (1) An indicator of Healthy Futures staff-participant interactions was be calculated as the classroom-based program, (2) Rhymin’ it Write after-school percentage of observed interactions where the evaluation team scored the interaction as “high program, (3) True Connections parent program, and (4) Code A quality/excellent” on the HHS observation form. The full range of quality indicators was also summer peer leadership program sessions analyzed and reported. While the representative sample of observations was used to capture Healthy Futures staff-participant interaction quality and every effort was made to make observations at all schools and of all educators, since only 9% of classroom-based sessions were observed this measure might not be capture all possible interactions. Due to limited uptake of the after-school, parent, and summer programs, observation data is not reported on those components. 47 Implementation Element Methods used to operationalize each implementation element Quality of youth engagement with Healthy Futures classroom- An indicator of Healthy Futures youth engagement was calculated as the percentage of based program component observations where the evaluation team scored youth engagement as “moderate” (score of 4) or “active” (score of 5) on the HHS observation form. Control Control Number of control program sessions delivered The total number of control sessions is a sum of the sessions captured for each grade and/or component in the Tracking Database. Average duration of control program sessions delivered Average duration for control sessions was calculated as the average of the implemented session lengths each measured in minutes. Average attendance for control program sessions delivered Average number of control sessions attended was calculated as the average of the number of sessions that each participant attended. Proportion of control program sessions implemented as Proportion of control sessions implemented is the total number of sessions implemented for compared to sessions expected based on placebo education each compared to the total number of sessions expected for each component. outline Control content was delivered to youth through a classroom- Total number of control activities completed based on data reported by Healthy Futures based program Executive Director cross-checked with observations. List of Healthy Futures staff delivering the control program (1) Total number of control staff delivering the control program is a count of staff members components to youth and ongoing monitoring of staff delivering implementing the control program. (2) Average number of staff members implementing the control program to avoid contamination control program at any one point in time during the 3 year control program. List of position requirements or qualifications for Healthy Futures Report of control staff requirements or qualifications based on job descriptions provided by staff delivering control program to youth Healthy Futures Executive Director. List of Healthy Futures control trainings and control staff Number of control staff trained was calculated as simple count of the number of control staff attendance at trainings members who were trained to deliver the program. Monitor student mobility between schools and conditions Percent of control participants who were exposed to the treatment program was calculated as the number of control students with at least 1 day of exposure to the Healthy Futures program divided by the total number of control students. Self- reported information on other teen pregnancy prevention The data on the survey question on other teen pregnancy prevention experiences presented experiences as frequency counts and percentages. 48 Implementation Element Methods used to operationalize each implementation element Context Context Self- reported information on other teen pregnancy prevention All the teen pregnancy prevention programming available to both Healthy Futures and control experiences groups. Bi-monthly web searches of programming offered in study school All the teen pregnancy prevention programming available to both Healthy Futures and control districts and communities groups. External events affecting implementation (for instance school The number of schools that were closed or under receivership as a result of district turmoil turnover, budget cuts, etc.) (unrelated to the teen pregnancy prevention programming in this project). Substantial unplanned adaptation(s) The resulting change in time allocated for facilitation of sessions. 49 Appendix J. Implementation fidelity for Healthy Futures program Table J.1. 6th-grade Nu-CULTURE implementation fidelity # Activities % Activities % Activities # Activities Prescribed # Activities Implemented Implemented Prescribed by by Program Delivered as Prescribed as Prescribed Adaptation c School Code a Model b by Model by Adaptation 210 (2 Classroom Sections) 114 113 99% 213 (2 Classroom Sections) 114 111 97% 215 (5 Classroom Sections) 285 272 95% 216 (9 Classroom Sections) 513 498 97% 319 (15 Classroom Sections) d 855 840 98% 422 (4 Classroom Sections) e 228 112 111 49% 99% 423 (7 Classroom Sections) 399 367 92% Median % Activities Implemented in 6th Grade 90% 99% Source: Online daily fidelity checklists completed by HF health educators Note: Students had an average attendance of 7 days and median attendance of 8 days for the 6th-grade Nu-CULTURE component a School codes assigned by evaluation team b 57 activities prescribed per section in the Healthy Futures 6th grade Nu-CULTURE curriculum, delivered over 8 days c Adaptations = changes made to content, frequency, sequencing, and/or duration prior to implementation d Condensed to 7 days for 5 of the 15 sections due to last minute school scheduling conflicts e Adapted to be delivered over 2 days due to school scheduling conflicts, received an average of 133 minutes of program vs. 400 minutes prescribed 50 Table J.2. 7th-grade Nu-CULTURE implementation fidelity # Activities % Activities % Activities Prescribed # Activities Implemented Implemented by Program Prescribed by # Activities as Prescribed as Prescribed School Code a Model b Adaptation c Delivered by Model by Adaptation 210 (2 Classroom Sections) 114 . 109 96% . 213 (2 Classroom Sections) 114 . 114 100% . 215 (6 Classroom Sections) 342 . 329 96% . 216 (6 Classroom Sections) d 342 . 335 98% . 319 (15 Classroom Sections) e 855 . 820 96% . 422 (5 Classroom Sections) f 285 140 118 41% 84% 423 (7 Classroom Sections) g 399 393 342 86% 87% Median % Activities Implemented . . . in 7th Grade 88% 86% Source: Online daily fidelity checklists completed by HF health educators a School codes assigned by evaluation team b 57 activities prescribed per section in the Healthy Futures 7th-grade Nu-CULTURE curriculum, delivered over 8 days c Adaptations = changes made to content, frequency, sequencing, and/or duration prior to implementation d Condensed to 7 days for 1 of the 6 sections due to last minute school scheduling conflicts e Condensed to 7 days for 5 of the 15 sections due to inclement weather f Adapted to be delivered over 4 days due to school scheduling conflicts g Adapted to be delivered over 7 days for 3 of the 7 sections due to school scheduling conflicts 51 Table J.3. 8th-grade Nu-CULTURE implementation fidelity % Activities % Activities # Activities # Activities Implemented as Implemented as Prescribed by Prescribed by # Activities Prescribed by Prescribed by School Code a Program Model b Adaptation c Delivered Model Adaptation 210(2 Classroom Sections) 102 . 97 95% . 213 (3 Classroom Sections) 153 . 148 97% . 215 (5 Classroom Sections) d 255 231 214 84% 93% 216 (6 Classroom Sections) 306 . 297 97% . 319 (16 Classroom Sections) e 816 779 700 86% 90% 422 (5 Classroom Sections) 255 . 235 92% . 423 (7 Classroom Sections) f 357 334 283 79% 85% Median % Activities Implemented in . . . 8th Grade 90% 89% Source: Online daily fidelity checklists completed by HF health educators a School codes assigned by evaluation team b 51 activities prescribed per section in the Healthy Futures 8th-grade Nu-CULTURE curriculum, delivered over 8 days c Adaptations = changes made to content, frequency, sequencing, and/or duration prior to implementation d Adapted to be delivered over 6 days for 3 of the 5 sections due to school scheduling conflicts e Adapted to be delivered over 7 days for 7 of the 16 sections due to school scheduling conflicts f Adapted to be delivered over 5 days for 1 of the 7 sections due to school scheduling conflicts and inclement weather 52 Table J.4. 6th-grade Rhymin’ it Write implementation fidelity # Activities % Activities % Activities Prescribed by # Activities Implemented Implemented as Program Model Prescribed by # Activities as Prescribed Prescribed by School a Adaptation b Delivered by Model Adaptation 210 77 . 51 66% . 215 c 77 72 48 62% 67% 216 77 . 53 69% . 319 77 . 29 38% . Median % Activities Implemented . . . in 6th Grade 59% 67% Source: Online weekly fidelity checklists completed by HF health educators a 77 activities prescribed in the Healthy Futures 6th-grade Rhymin’ it Write curriculum, delivered over 10 weeks (1 day per week) b Adaptations = changes made to content, frequency, sequencing, and/or duration prior to implementation c Adapted to be delivered over 9 days for 1 of the schools due to scheduling conflicts Table J.5. 7th-grade Rhymin’ it Write implementation fidelity # Activities Prescribed by % Activities Implemented as School Program Model a # Activities Delivered Prescribed by Model 216 41 26 63% 319 41 28 68% Median % Activities . . Implemented in 7th Grade 66% Source: Online weekly fidelity checklists completed by HF health educators Notes: Program was implemented in three treatment schools (2 in Lowell, 1 in Lynn) but only two of the schools had participants from the study cohort. a 41 activities prescribed in the Healthy Futures 7th-grade Rhymin’ it Write curriculum, delivered over 9 weeks (1 day per week). Originally designed to be a 10- week program but due to school schedules the 7th-grade program was implemented as a 9-week program and adapted prior to implementation. 53 Table J.6. 8th-grade Rhymin’ it Write implementation fidelity # Activities Prescribed by % Activities Implemented as School Program Model a # Activities Delivered Prescribed by Model 215 46 28 61% 216 46 37 80% Median % Activities . . Implemented in 8th Grade 71% Source: Online weekly fidelity checklists completed by HF health educators Note: Program was implemented in three treatment schools (2 in Lowell, 1 in Lynn) but only two of the schools had participants from the study cohort. a 46 activities prescribed in the Healthy Futures 8th-grade Rhymin’ it Write curriculum, delivered over 10 weeks (1 day per week). Table J.7. Code A Implementation Fidelity # Activities Prescribed by % Activities Implemented School Program Model a # Activities Delivered as Prescribed by Model 215 b 54 53 98% Source: Online daily fidelity checklists completed by HF health educators a 54 activities prescribed in the Healthy Futures Code A curriculum, delivered over 18 sessions (3 sessions per week for 6 weeks) b Delivery was modified to allow more time for parents to express their concerns, provide extra support and encouragement, and better meet the current needs of the parents 54 Appendix K. Quality of Nu-CULTURE classroom-based program Table K.1. Quality of Nu-CULTURE classroom-based program implementation 6th Grade a 7th Grade b 8th Grade c . Average Average Average Clarity of health educator's explanations of activities 5.0 4.8 4.6 1 = Not Clear, 3 =Somewhat Clear, 5 = Very Clear Extent health educator kept track of time during the session and activities 4.3 4.5 4.1 1= Not on Time, 3 = Some Loss of Time, 5 = Well on Time Extent presentation of materials seem rushed or hurried 4.5 4.3 4.1 1=Very Rushed, 3 = Somewhat Rushed, 5 = Not Rushed Participants appeared to understand the material 4.9 4.7 4.4 1= Little Understanding, 3 = Some Understanding, 5 = Good Understanding Youth participation in discussions and activities 4.6 4.7 4.5 1 = Little Participation, 3 = Some Participation, 5 = Active Participation Health educator's knowledge of the program 4.9 4.8 4.7 1 = Poor, 3 = Average, 5 = Excellent Health educator's level of enthusiasm 4.8 4.9 4.9 1 = Poor, 3 = Average, 5 = Excellent Health educator's poise and confidence 4.9 4.8 4.9 1 = Poor, 3 = Average, 5 = Excellent Health educator's rapport and communication with participants 4.7 4.8 4.7 1 = Poor, 3 = Average, 5 = Excellent Health educator's effectiveness in addressing questions/concerns d 4.5 4.3 3.9 1 = Poor, 3 = Average, 5 = Excellent Overall quality of the program session 4.5 4.5 4.2 1 = Poor, 3 = Average, 5 = Excellent Source: Evaluation team program observations a JSI conducted 17 observations of the 6th grade Nu-CULTURE program b JSI conducted 37 observations of the 7th grade Nu-CULTURE program c JSI conducted 37 observations of the 8th grade Nu-CULTURE program d Participants did not ask questions or raise concerns in all sessions observed (7 of 17 in 6th, 10 of 37 in 7th, and 11 of 37 in 8th) 55 Appendix L. Health educator training, knowledge, confidence, and satisfaction Table L.1. List of Healthy Futures health educator trainings Attendance Date Training Training Content (# Educators) Year 1: 2011-2012 . . . 9/20/2011–Sessions 6.1 & 6.2: friendship, decision making, puberty 9/21/2011–Sessions 6.3 & 6.4: gender reflections, emotional needs, 9 inappropriate/appropriate ways to show affection in middle school, types of love 9/20/2011– 6th-Grade Nu-CULTURE 9/22/2011–Sessions 6.5 & 6.6: healthy relationships, abstinence, assertiveness, refusal skills, 9/22/2011 Curriculum sexual activity 9/27/2011–Sessions 6.7 & 6.8: sexual abuse/assault, online safety, conflict resolution, review game 9 9/27/2011– 6th-Grade Nu-CULTURE 9/28/2011–Transitioning between Session 6.1 topics 9/29/2011 Curriculum 9/29/2011–Transitioning between Sessions 6.2-6.6 topics 9 10/5/2011 6th-Grade Nu-CULTURE Curriculum Time management, allowable adaptations 2/23/2012 Webinar “When Cyberbullying Spills into School” 8 2/22/2012– 6th-grade curriculum booster, staff relationship building, cultural differences, fidelity monitoring, 9 2/24/2012 All Staff Training Week performance measures 4/17/2012- 4/17/2012–Sessions 6.1 & 6.2, time management, allowable adaptations 9 4/19/1012 6th-Grade Nu-CULTURE 4/18/2012–Sessions 6.3-6.6, time management, allowable adaptations Curriculum 4/19/2012–Sessions 6.7 & 6.8, time management, allowable adaptations 8 6/25/2012 Meeting with HF administrators, educators, and JSI evaluation team: debriefed 6th grade 6th-Grade Debrief implementation and evaluation findings, reviewed fidelity monitoring and performance measures 7th-Grade Nu-CULTURE 8 6/26/2012 Curriculum Introduction to 7th-grade curriculum, Session 7.2: basic human needs, levels of friendship 7th-Grade Nu-CULTURE 8 6/28/2012 Curriculum Sessions 7.3 & 7.4: puberty, pregnancy, teen stress, positive character traits, peer pressure 56 Attendance Date Training Training Content (# Educators) 7/10/2012–Sessions 7.1 & 7.2: keys to success, basic human needs, levels of friendship 7/11/2012–Sessions 7.3 & 7.4: teen stress, positive character traits, peer pressure, puberty, pregnancy 7/12/2012–Sessions 7.5-7.8: consequences of sexual activity (pregnancy, STIs, emotional 7 heartache), sexual offenses, review game 7/10/2012– 7th-Grade Nu-CULTURE 7/16/2012–Transitioning between Session 7.1-7.4 topics 7/17/2012 Curriculum 7/17/2012-Transitioning between Session 7.5-7.8 topics 6 8/13/2012 Rhymin’ it Write Program Rhymin’ it Write after-school program content and activities 7th-Grade Nu-CULTURE 6 8/15/2012 Curriculum 7th-grade curriculum booster Year 2: 2012-2013 . . . 9/4/2012–Sessions 7.1-7.3, time management, allowable adaptations 6 9/4/2012– 7th-Grade Nu-CULTURE 9/5/2012–Sessions 7.4-7.7, time management, allowable adaptations 9/6/2012 Curriculum 9/6/2012–Sessions 7.8, time management, allowable adaptations 1 9/18/2012 Online Training Rutgers University STD Basics Course 9/18/2012–Sessions 7.1-7.3, classroom management 6 9/18/2012– 7th-Grade Nu-CULTURE 9/19/2012–Sessions 7.4-7.6, classroom management 9/20/2012 Curriculum 9/20/2012–Sessions 7.7 & 7.8, classroom management, parent connection forms 6 10/21/2012 Health Educator Meeting Discuss educator questions and concerns with administrative staff 3/13/2013 Online Training Fred Pryor: Switching Between Presenters 1 3/13/2013 Online Training Fred Pryor: 60 Minutes of PowerPoint Secrets 1 1 3/13/2013 Leadership Seminar Fred Pryor: Team-Building, Mentoring and Coaching Skills for Managers & Supervisors Lead Educator 3/19/2013 Online Training Fred Pryor: Improve Your Memory, Improve Your Productivity 1 1 3/29/2013 Leadership Seminar Fred Pryor: Basic Supervision Lead Educator 57 Attendance Date Training Training Content (# Educators) 4/2/2013 Online Training Fred Pryor: Power Speaking 1 6/25/2013 Health Educator Training Fred Pryor: Presentation Skills 6 7/15/2013–Introduction 8th-Grade Nu-CULTURE curriculum 6 7/15/2013– 8th-Grade Nu-CULTURE 7/16/2013–Session 8.1: (re-)introduction to Healthy Futures; goals and dreams; emotional needs 7/17/2013 Curriculum 7/17/2013–Session 8.2: Puberty and Pregnancy 8th-Grade Nu-CULTURE 6 7/22/2013 Curriculum 7/22/2013–Sessions 8.3 & 8.4: Contraception and STIs/STDs 7/24/2013–Sessions 8.5-8.8: healthy relationships, emotional needs and bonding, self-control, sex and the law, media talk 6 7/24/2013- 8th-Grade Nu-CULTURE 7/25/2013–Sessions 8.1-8.3 7/26/2013 Curriculum 7/26/2013–Sessions 8.4-8.8 8th-Grade Nu-CULTURE 6 7/31/2013 Curriculum Presentation Skills 1 7/31/2013 Leadership Seminar Fred Pryor: Creative Leadership for Managers, Supervisors and Team Leaders Lead Educator 8th-Grade Nu-CULTURE 4 8/2/2013 Curriculum Presentation Skills Meeting with HF administrators, educators, and JSI evaluation team: debriefed 6th-grade 7 8/30/2013 7th-Grade Debrief implementation and evaluation findings, reviewed fidelity monitoring and performance measures Year 3: . . . 2013-2014 8th-Grade Nu-CULTURE 9/9/2013 Curriculum 9/9/2013–Session 8.1 7 9/10/2013–Sessions 8.2 & 8.3 9/11/2013-Sessions 8.4-8.6 9/10/2013– 8th-Grade Nu-CULTURE 9/12/2013-Session 8.7 9/13/2013 Curriculum 9/13/2013-Session 8.8 7 9/17/2013–Session 8.1 9/17/2013– 8th-Grade Nu-CULTURE 9/18/2013–Sessions 8.2-8.6 9/19/2013 Curriculum 9/19/2013-Sessions 8.7 & 8.8 7 58 Attendance Date Training Training Content (# Educators) 9/26/2013– 9/27/2013 Team Building Workshop Belbin team roles training 7 10/10/2013 Team Building Workshop Conflict resolution training 7 8th-Grade Nu-CULTURE 2/17/2014 Curriculum Sessions 8.1-8.8, classroom management 6 2/18/2014– 8th-Grade Nu-CULTURE 2/18/2014–Sessions 8.1-8.8, classroom management 2/19/2014 Curriculum 2/19/2014–Sessions 8.1-8.8, time management, allowable adaptations 8 8th-Grade Nu-CULTURE 2/24/2014 Curriculum 2/24/2014–New educators observe Session 8.6 implementation 7 2/25/2014– 2/25/2014–New educators observe Session 8.7 implementation 2/26/2014 Health Educator Training 2/26/2014–New educators observe Session 8.8 implementation 8 8th-Grade Nu-CULTURE 2/27/2014 Curriculum 2/27/2014-New educators observe Session 8.1 implementation 9 8th-Grade Nu-CULTURE 2/28/2014 Curriculum 2/28/2014-New educators observe Session 8.2 implementation New 7 Team Building 6/10/2014–Sessions 6.1-6.8, team building 6/10/2014– Workshop; 6th-Grade 6/11/2014–Sessions 6.1-6.8, professional development 6/12/2014 Booster 6/12/2014–Sessions 6.1-6.8. professional development 7 7/24/2014 Team Building Workshop Personal story Training 7 Meeting with HF administrators, educators, and JSI evaluation team: debriefed 6th-grade 11/4/2014 8th Grade Debrief implementation and evaluation findings, reviewed fidelity monitoring and performance measures 6 59 Table L.2. Health Educator feedback 6th Grade a 7th Grade b 8th Grade d . Average Average Average Adequacy of Training Received 1 = Poor, 3 = Average, 5 = Adequate 3.9 4.2 3.8 Knowledge of Curriculum 1 = Poor, 3 = Average, 5 = Excellent 4.0 3.8 3.5 Confidence Facilitating Nu-CULTURE Curriculum 1 = Not Confident, 3 = Average, 5 = Very Confident 4.1 4.7 3.3 Confidence Facilitating Lessons on Healthy Relationships 1 = Not Confident, 3 = Average, 5 = Very Confident 4.4 4.7 4.5 Confidence Facilitating Lessons on Pregnancy and Reproduction 1 = Not Confident, 3 = Average, 5 = Very Confident 4.4 5.0 c 4.3 Confidence Facilitating Lessons on Contraception (E.g., condoms, pill, shot, patch, ring, IUDs) 1 = Not Confident, 3 = Average, 5 = Very Confident Not Asked 4.0 c 3.7 Confidence Facilitating Lessons on STIs and STDs 1 = Not Confident, 3 = Average, 5 = Very Confident Not Asked 3.5 c 4.2 Rapport and Communication with Participants 1 = Not Confident, 3 = Average, 5 = Very Confident 4.6 4.3 4.5 Ability to Effectively Address Participants’ Questions and Concerns 1 = Not Confident, 3 = Average, 5 = Very Confident 4.0 3.8 3.8 Notes: Questions on contraception and STIs/STDs were not asked in 6th grade because the Nu-CULTURE curriculum does not have sessions on these topics. a Completed by 8 Health Educators between 6/15/2012 and 6/21/2012 b Completed by 6 Health Educators between 7/11/2013 and 7/31/2013 c Only Answered by 2 of the 6 Health Educators d Completed by 6 Health Educators between 7/8/2014 and 8/27/2014 60 Appendix M. Context Table M.1. Number and percent of 8th-grade students reporting exposure to programs that address sexual activity or pregnancy (n=1,127) Treatment Treatment Control Control Program Site (n=488) (n=488) (n=639) (n=639) . Percent Number Percent Number Church a 1.0% 5 1.7% 11 Community Organization b 1.8% 9 1.4% 9 School c 4.9% 24 6.6% 42 Other <1.0% 1 <1.0% 2 Total 8.0% 39 10.0% 64 Note: Student could only choose one response to this question. a Includes Vale Esperar b Includes Gregg House c Includes: After-School Programs, Athletics, Clubs, Girls Inc., Health Class, Project YES, ROTC Table M.2. Number and percent of 9th-grade students reporting exposure to programs that address sexual activity or pregnancy (n=996) Treatment Treatment Control Control Program Site (n=476) (n=476) (n=520) (n=520) . Percent Number Percent Number Church a 1.9% 9 1.5% 8 Community Organization 2.5% 12 1.5% 10 School b 11.3% 54 11.7% 61 Other <1.0% 2 <1.0% 2 Total 16.2% 77 15.6% 81 Note: Student could only choose one response this question. a Includes: Vale Esperar, CCD b Includes: Athletics, Clubs, Girls Inc., Health Class, Lowell Community Health Center Teen Block, Peer Health Exchange, ROTC, Sister 2 Sister 61 Table M.3. Web search results of programs available in study communities Organization Program/Event Topic Sister 2 Sister, Will Power Won’t General Health Topics, Media Literacy, Power, Preventing Adolescent Relationships, Abstinence, Refusal Skills, Girls Inc. Pregnancy HIV/STI/STD Prevention, Sexual Health Outreach/Education as Requested, General Health Topics, Sexual and Services Focus on High Reproductive Health Topics Health Quarters School/Recently Graduated Youth Youth Violence Prevention, Teen Lowell Community Health Teen Coalition, School Health Pregnancy Prevention, HIV/STI/STD Center Centers Prevention One-on-One Mentoring, Promoting Academic Growth and Healthy Youth Mentoring and Advocacy Project Development, Gang Membership (MAP) Prevention, Teen Pregnancy Prevention. Light of Cambodian Children Health Education for Students and School Health Centers, Families, KAYA After-School Program that Lynn Community Health Khmer American Youth in Action Includes Health Education and AIDS Center (KAYA) Awareness After-School and Summer Program Tutoring, Recreation, and Life Skills Project YES Focused on Gang Prevention Including Sexual Risk Reduction Teen Scene Drop in Center Life Skills, Health Education Abstinence Education, Mentoring Vale Esperar Workshops for Hispanic Teens Services, Healthy Relationships 62