Findings from an Innovative Teen Pregnancy Prevention Program Evaluation of Youth all Engaged (YAE) in Denver, CO Final Impact Report for Denver Public Health August 31, 2015 Prepared by Sheana Salyers Bull, PhD, MPH Sarah Schmiege, PhD Sharon Devine, PhD, JD Recommended Citation: Bull, S., Schmiege, S., Devine, S. 2015. Evaluation of Youth all Engaged (YAE) in Denver, CO. Final Impact Report for the Denver Public Health Department. Written Report. Acknowledgements: We gratefully acknowledge the support and hard work of staff at The Boys & Girls Clubs of Metro Denver (Boys & Girls Club), including the Facilitator of the TOP® program across all participating clubs over four years. We also acknowledge the partnership of Colorado Youth Matter in their role as trainers and fidelity monitors for the TOP® program delivery. We acknowledge the leadership of the management team at the Denver Public Health Department, including the team that programmed and implemented the text messages associated with the program. Finally, we wish to acknowledge the multiple students and community members who assisted with data collection related to this evaluation. This publication was prepared under Grant Number TP2AH000016 from the Office of Adolescent Health, U.S. Department of Health & Human Services (HHS). The views expressed in this report are those of the authors and do not necessarily represent the policies of HHS or the Office of Adolescent Health. Table of Contents I. Introduction............................................................................................................................ 4 A. Introduction and study overview ......................................................................................... 4 B. Primary research question(s) .............................................................................................. 5 C. Secondary research question(s) ......................................................................................... 5 II. Program and comparison programming ............................................................................ 5 A. Description of program as intended .................................................................................... 5 B. Description of counterfactual condition ............................................................................... 6 III. Study design .......................................................................................................................... 7 A. Sample recruitment ............................................................................................................. 7 B. Study design ....................................................................................................................... 7 C. Data collection..................................................................................................................... 8 D. Outcomes for impact analyses ............................................................................................ 9 E. Study sample .................................................................................................................... 11 F. Baseline equivalence ........................................................................................................ 12 G. Methods ............................................................................................................................ 18 IV. Study findings...................................................................................................................... 20 A. Implementation study findings .......................................................................................... 20 B. Impact study findings ........................................................................................................ 22 V. Conclusion ........................................................................................................................... 25 VI. References ........................................................................................................................... 26 VII. APPENDICES: TABLES AND FIGURES ............................................................................ 27 Appendix A: Program materials and data collection efforts ....................................................... 27 Table A.1. Examples of text messages sent to YAE Participants .............................................. 27 Table A.2. Data collection efforts used in the impact analysis of YAE and timing ..................... 28 Table A.3. Data used to address implementation research questions ....................................... 28 Appendix B: Study consort diagram .......................................................................................... 31 Figure 1: Consort diagram .......................................................................................................... 31 Appendix C: Study sample .......................................................................................................... 32 Table C.1a. Cluster and youth sample sizes by intervention status ........................................... 32 Appendix D: Implementation evaluation methods and selected outcomes ........................... 33 Table D.1. Methods used to address implementation research questions ................................ 33 Appendix E: Implementation Evaluation .................................................................................... 35 Table E.1 Fidelity and Quality for delivery of TOP®.................................................................... 35 Table E.2 TOP® Quality Monitoring re Session Delivery 2011-2015 .......................................... 37 Appendix F: Sensitivity analyses ................................................................................................ 50 Table F.1. Sensitivity of impact analyses using data from assessment at TOP® Program completion to address the primary research questions ...................................................................... 51 3 EVALUATION OF TEEN OUTREACH PROGRAM (TOP®) PLUS YOUTH ALL ENGAGED IN DENVER, CO FINDINGS FROM AN INNOVATIVE TEEN PREGNANCY PREVENTION PROGRAM I. Introduction A. Introduction and study overview Teen mothers are more likely to drop out of high school and face reduced economic opportunities compared to their childless peers, and their children face a higher risk of poverty, child abuse and neglect. Preventing teen births reduces taxpayer funded health care costs. i Between 2004 and 2013, Colorado teen birth rates declined by 47% to 24.3 births per 1,000 teens ages 15-19, a historic low. Nevertheless, 3,807 females ages 15-19 gave birth in Colorado in 2013, with disparities between non-Hispanic White teens (14 births per 1,000), Black teens (26 births per 1,000) and Hispanic teens (42 births per 1,000). ii In Denver, the largest city in Colorado, three quarters of all babies born to teens are to Hispanic teens. iii Poverty and lack of access to health care and quality primary education are factors associated with subsequent teen pregnancy and parenting. iv, v In Denver in 2012, 29% of children lived in poverty, 72% of school aged children qualified for a free or reduced-price lunch, and 47% of children under 18 were enrolled in Medicaid. Denver’s high school graduation rate for 2012-2013 was 61%, with rates of 64%, 55% and 71% for African American, Hispanic and White youth respectively. vi Using a rigorous review process, the U.S. Department of Health and Human Services identified 36 programs with evidence of efficacy for prevention of teen births. vii One program is the Teen Outreach Program (TOP®), focused on youth development, including human development and sexuality education targeting youth aged 12-18 with a minimum of 25 one- hour classroom sessions delivered by an adult, and 20 hours of community service learning (CSL). viii Although there is evidence of TOP® efficacy, this and other programs of similar length and intensity face challenges in effectively serving diverse youth. ix When program content does not resonate, youth are more likely to drop out of interventions. x, xi, xii In a dynamically changing educational and social environment, programs delivered in school classrooms and those relying exclusively on face-to-face delivery are becoming obsolete because they fail to capitalize on youth communication preferences including cell phones and social media. xiii In this study, we first sought to explore opportunities to reach youth at risk for teen pregnancy outside a traditional classroom setting by delivering TOP® in the Denver Metro Boys & Girls Club, who offer after-school programs primarily for racial and ethnic minority youth and those living in poverty. xiv Because youth utilize cell phones in substantial numbers and are the largest consumers of cell phones, cell phone minutes and text messaging, xv, xvi, xvii we also endeavored to couple TOP® with a text message program called Youth All Engaged (YAE), previously known as TOP411, to better assist diverse youth in engaging fully with and reinforcing the TOP® curriculum. This report describes the implementation and impact of YAE added to TOP® relative to TOP® alone. 4 B. Primary research question(s) In this study we explored two primary research questions: 1) What is the impact of ‘YAE, added to TOP®, relative to TOP® alone on the proportion of sex acts protected by condoms over the past 3 months assessed at TOP® program completion? 2) What is the impact of YAE added to TOP®, relative to TOP® alone on the proportion of sex acts protected by contraception over the past 3 months assessed at TOP® program completion? C. Secondary research question(s) We explored two secondary research questions: 1) What is the impact of YAE, added to TOP®, relative to TOP® alone on ever being pregnant or causing a pregnancy assessed at TOP® program completion? 2) How did the impacts of YAE, added to TOP, relative to TOP alone differ between Hispanic and non-Hispanic participants on the three outcomes – the proportion of sex acts protected by condoms, the proportion of sex acts protected by contraception, and ever being pregnant or causing a pregnancy? II. Program and comparison programming A. Description of program as intended Those in the intervention condition received Wyman’s TOP® curriculum and individualized text messaging, a program we call ‘YAE’. According to the Wyman website, (wymancenter.org) TOP® “is an evidence-based best practice program specifically designed for teens in 6th to 12th grade. Guided by an engaging and relevant nine-month curriculum, TOP® reduces the risk of problem behavior while promoting healthy choices and empowering teens to lead successful lives and build strong communities.” The TOP® curriculum consists of three interconnected components: adult program facilitation, classroom-based activities related to youth development, and classroom- and community-based service learning. The TOP® curriculum includes a required 25 educational peer sessions of approximately 40 minutes in length and a minimum of 20 service-learning hours completed by each participant. Trained program staff delivers the sessions about once per week for 26 weeks over a school year (with breaks for holidays). TOP® includes sessions on values clarification, relationships, communications/assertiveness, influence, goal setting, decision-making, human development and sexuality and community service learning. Youth select their community service activities under the supervision of trained program staff. In addition to TOP®, the intervention group receive the YAE text-messaging component that consists of an average of five to seven text messages per week. Text messages reinforced specific topics covered each week in the TOP® sessions and offered additional information and resources related to the topics covered in TOP® each week. Once the TOP® program was completed, text messages continued at the average rate of three per week for an additional 12 weeks. 5 Following the iterative message development with youth from Boys & Girls Clubs described in detail in Devine et al., xviii we finalized text messages and automated them for delivery, using such features as quizzes, polls, and links to websites with video and/or additional content on particular topics. Additionally, we tailored some message content for selected messages based on gender, age, and race/ethnicity; all youth would receive messages in the same format (via quiz or poll or link) but tailored messages varied content. Because some messages rely on popular culture—e.g., lyrics from a popular song—we reviewed messages annually to ensure continued relevance and resonance with the target audience. Examples of our text messages are shown in Table A.1. in appendix A. There are three different categories of messages: Program content messages: These messages support and enhance the TOP® curriculum lessons and include a variety of bi-directional message types. These include: Questions, where youth can respond to queries about relevant TOP® topics; quizzes, where youth can send a response to a multiple choice question and then receive a reply with the correct answer; “myth/fact” where youth can guess if something is true or false; follow-ups, where youth were asked about their experience in TOP® activities; and polls, where participants vote on their preference for a response and later receive a message with the results. Unidirectional messages from the system that do not request a response are “fun facts,” videos, quotes, and other messages. These messages offer information relevant to sexual and reproductive health, links to resources, and inspirational quotes from popular figures. Selected unidirectional and bi- directional messages are tailored for gender, race, and ethnicity. Event-based messages: Reminders for TOP® club meetings, community service events, and other event-specific messages are tailored to specific clubs and can be unidirectional messages providing TOP® club information or bi-directional weekly messages reminding youth that a TOP® session is scheduled and inviting them to respond if they are planning to come. Responses to unsolicited incoming messages: These include messages used to answer questions and direct participants to appropriate resources when participants respond unexpectedly to an outbound message. These message protocols are put in place to ensure that participants receive timely and accurate support and referrals for care as needed. Participants who wish to withdraw from the text messaging can do so by sending a ‘STOP’ request in response to any text. Participants sending such a request are reminded they are no longer eligible for research study incentives associated with participation. The TOP® curriculum was delivered at Boys & Girls Clubs throughout the Metro Denver area by facilitators trained in delivery of the curriculum. Much of the text messaging was automated, i.e., messages that are relevant for each week and sent out on a specific schedule; messages to remind youth to complete TOP® homework assignments, and messages with links for youth to seek additional information. However, staff members at Denver Public Health were available to answer individualized questions from text messages. B. Description of counterfactual condition Those in the counterfactual condition received the TOP® curriculum only. Content and implementation of the counterfactual condition was identical to the intervention condition, minus the YAE component. 6 III. Study design A. Sample recruitment This study was implemented in eight of the nine Boys & Girls Clubs of Metro Denver (Boys & Girls Clubs) each year for four school years beginning in September 2011 and concluding in May 2015. The ninth club had no teen program and was not eligible. Youth were eligible to participate if they were between the ages of 14 and 18 and had attended one of the study clubs in the three-month period prior to the inauguration of TOP® program delivery (N=8890). Youth have to sign into the club every time they attend, with their name and age. Whenever a youth aged 13-17 signed in during a weekday, a club staff would note their attendance and approach them. They approached the youth who were aged 13 in anticipation of their eligibility at the time of enrollment and those under 18 with the knowledge that upon turning 18 youth would no longer be eligible for the study. Not all of the 8890 who had attended a club previously came again during the recruitment period. A total of 3,643 youth were approached by club staff when they came to one of the participating Boys & Girls Clubs and invited to participate in TOP® and potentially YAE, and 864 agreed to enroll. During the informed consent and enrollment process for the randomized trial at the start of each implementation year, participants were asked to provide their personal mobile phone numbers, which were then verified by study staff to be mobile numbers. To reach participants who did not have personal mobile phones or who temporarily lost access to their phones (e.g., lost or broken phone), the free web-based VoIP service, Pinger Textfree for Web, was used to send program text messages via an online site. TOP® program facilitators worked with participants to set up individual Textfree accounts as needed. Text messages were sent to these Textfree numbers and participants are able to check Textfree accounts on any web- enabled device, including computers at their program implementation sites. B. Study design This is a cluster-randomized controlled trial, where Boys & Girls Clubs were the unit of randomization. At the start of the study, clubs were randomly assigned to condition in a manner that ensured that each club would serve as a treatment site for two years and a control site for two years, and ensured that within each year, four clubs were assigned to the treatment condition. Effectively, this was a random assignment study of 32 club-year combinations. In all inferential analyses (baseline equivalence assessments and impact analyses), we incorporate 10 dummy variables for these strata (7 club dummies [with one club as the omitted category] and 3 year dummies [with one year as the omitted category]), to appropriately account for the random assignment procedure. A member of the evaluation staff conducted the random assignment for all four years at the outset of the study. TOP® facilitators and youth were blinded to their study assignment until the time of study enrollment. In Year 1, 4 clubs were randomly assigned to intervention and 4 clubs were randomly assigned to the counterfactual within Year 1 only. In Year 2, again 8 clubs were randomly assigned, 4 to the intervention and 4 to the counterfactual. Based on the results of random assignment within Years 1 and 2, some of the assignment within Years 3 and 4 were constrained to ensure the proper distribution across clubs (e.g., if Club 1 was assigned to the intervention in Years 1 and 2, it had to be assigned to the counterfactual in Years 3 and 4). The remaining clubs where Year 3 and 4 assignment was not yet constrained were randomly assigned within Year 3; Year 4 assignments were then determined based on assignments from Years 1-3. Contamination was not expected to be a problem within a club between years (e.g., if 7 one year the club is in the intervention and the next year it is in the control) because different adolescents are enrolled in the program in each year and the main distinguishing characteristic between the intervention and control conditions is the text messaging component, which occurs separately from the club setting. Randomization procedures also ensured that four clubs would be assigned to the intervention condition and four to the control in each year to stabilize necessary staff resources for delivering the intervention from year to year. C. Data collection 1. Impact evaluation Survey data were collected from youth at three time points: baseline, post-intervention (at TOP® program completion) and 12 months following TOP® program completion, for each year for the first three years. In the fourth year of the study, data were collected from youth at baseline and at post-intervention, but not at the one-year follow-up. This report presents only data from the first follow-up point (at TOP® program completion), which was available for all four cohorts. Specific time frames for data collection are presented in Appendix A, Table A.2. University of Colorado staff collected data in Boys & Girls Clubs where TOP® sessions were held in the beginning and end of each program year. Participants self-administered the evaluation instrument using a computer and the Internet available in the club. Data collection procedures were identical between the two study conditions. In the first two years of data collection, youth were offered $5 to complete the baseline and $10 to complete each of the follow-up surveys. To generate greater program commitment, we offered $5 for baseline and $15 for each follow-up in years three and four of the program. 2. Implementation evaluation Colorado Youth Matter (CYM) is a local training and advocacy agency that facilitates access to comprehensive youth sexuality education statewide. They served as trainers for the TOP® program. CYM trained all TOP® facilitators in program delivery. Each year during program implementation, trained facilitators visited all clubs on three occasions throughout the year where TOP® was delivered to observe fidelity and quality of TOP® program delivery. Fidelity of TOP® implementation Facilitators completed TOP® fidelity logs after every session. This log tracked qualitative elements of the lesson: what went well during the lesson, what they might change and what common questions were asked. The logs also tracked quantitative elements: were the number of activities planned for the lesson implemented, were modifications or adaptions made to the lesson, were lesson objectives met, were ice breakers conducted and was the TOP® facilitation style used, among other questions. These logs were compiled into a single log for each lesson and as well as a yearly report. Logs were reviewed to determine overall attendance and compliance with delivery of intended elements for each session. Quality of implementation To assess quality during the observations of TOP®, CYM staff documented when facilitators offered clear instructions, were timely, delivered content in a clear and comprehensive manner, encouraged participation, demonstrated confidence in facilitation, and established a rapport with youth. CYM added notes during the observation to clarify and standardize rating criteria to improve inter-rater reliability. CYM debriefed with facilitators after each session to give feedback 8 on all the items in the program observation form. A copy of the program observation form was then sent to the facilitators and their supervisor. Text message data collection A periodic contact information update form was completed by all youth, allowing study staff to verify that participants’ mobile phone numbers were accurate in the automated text message delivery system using a patient relationship management (PRM) platform. Updates to mobile phone numbers were made and documented throughout the implementation period. The PRM short message service (SMS) or text message system also stored data about text message activity. In addition to the manually uploaded demographic and contact information provided by participants, data fields collected included outbound text message content, inbound responses, message delivery status, message category, and other relevant information. We documented any errors in message delivery or in system performance. More detail on items used to assess implementation is available in Appendix A Table A.3. D. Outcomes for impact analyses Assessments included documentation of demographics, including age, gender, ethnicity (do you consider yourself to be Hispanic?) followed by self-reported race, where participants indicated identification with one or more racial groups including White, Black, Asian, American Indian, Native Hawaiian or Other. We considered anyone who indicated Hispanic ethnicity as Hispanic in sub-analyses related to ethnicity regardless of their self-reported race. To answer the primary research questions, participants were asked if they ever had sex and the number of times they had sex in the past three months. Those who gave a response greater than zero were subsequently asked how many of those encounters were protected by condoms and how many of them were protected by contraception. The proportion of protected sex acts was calculated as the number of times sexual encounters were protected by condoms (for the first outcome measure) and contraception (for the second outcome measure) in the prior three months divided by the number of times a person had sex in the prior 3 months. Scores were distributed continuously and could range from 0 (never protected) to 1.0 (fully protected). Participants reporting 0 sexual encounters in the last 3 months were considered abstainers from sex. Abstainers were coded as 1.0 (fully-protected). To answer the secondary research question, we asked participants to indicate if they had ever been pregnant or caused a pregnancy. The specific items used to measure each of our outcomes are detailed in Tables III.1 and III.2. 9 Table III.1. Behavioral outcomes used for primary impact analyses research questions Timing of measure Outcome name Description of outcome relative to program Proportion of sex The variable is a measure of the proportion of sex acts At TOP® program acts protected by over three months protected by condoms. The measure completion condoms is taken directly from the following items on the survey: • In the past 3 months, how many times have you had sexual intercourse? • In the past 3 months, how many times have you had sexual intercourse without using a condom? The variable is constructed as continuous variable, as the proportion of sex acts in the past three months protected by condoms Participants reporting 0 sexual encounters in the last 3 months were considered abstainers from sex. Abstainers were coded as 1.0 (fully-protected) Proportion of sex The variable is a measure of the proportion of sex acts At TOP® program acts protected by over three months protected by contraception. The completion contraception measure is taken directly from the following items on the survey: • In the past 3 months, how many times have you had sexual intercourse? • In the past 3 months, how many times have you had sexual intercourse without using at least one of the following forms of contraception? (Condoms, Birth control pills The shot (Depo Provera) The patch The ring (NuvaRing) IUD (Mirena or Paragard) Implant (Nexplanon/Implanon) The variable is constructed as continuous variable, as the proportion of sex acts in the past three months protected by contraception Participants reporting 0 sexual encounters in the last 3 months were considered abstainers from sex. Abstainers were coded as 1.0 (fully-protected) 10 Table III.2. Behavioral outcomes used for secondary impact analyses research questions Timing of measure Outcome name Description of outcome relative to program Ever pregnant or The variable is a measure among all participants on At TOP® program ever caused a whether they had ever been pregnant or caused a completion pregnancy pregnancy. The measure is taken directly from the following item on the survey: • To the best of your knowledge, have you been pregnant or gotten somebody pregnant?” The variable is constructed as a dummy code where those who say they have been or have caused a pregnancy (yes) are coded as 1 and those who have never been pregnant or caused a pregnancy (no) are coded as 0. E. Study sample As described above, this is a cluster-randomized controlled trial, where Boys & Girls Clubs were the unit of randomization. At the start of the study, clubs were randomly assigned to condition in a manner that ensured that each club would serve as a treatment site for two years and a control site for two years, and ensured that within each year, four clubs were assigned to the treatment condition. Effectively, this was a random assignment study of 32 club-year combinations. In 2012, one club terminated its teen program, but a new program was opened in a school nearby, and we used this as a replacement for the club terminating their teen program. This new club assumed and continued to follow the originally assigned study conditions attributed to the club it replaced for the remainder of the study (comparison for cohort 2, intervention for cohort 3, and comparison for cohort 4). Similarly in 2013, one club experienced a substantial drop in its teen census, and we replaced that club with a club that was newly built in the same geographic area serving teens with a similar profile. This new club assumed and continued to follow the originally assigned study conditions attributed to the club it replaced for the remainder of the study (intervention for cohort 3 and comparison for cohort 4). All randomization occurred at the beginning of the study and no re-randomization occurred. There were 854 participants enrolled, but two participants never completed a baseline survey and were subsequently withdrawn from the study. Of the 852 participants enrolled and completing a baseline assessment, 436 were in clubs assigned to the intervention arm and 416 were in clubs assigned to the control arm. There were 317 participants completing the follow-up survey in the intervention and 315 in the control arm. See Figure 1 in Appendix B for the study consort diagram. For the primary study question of effects of the intervention on proportion of sex acts protected by condoms, our analytic sample included 240 intervention and 222 control participants. For the primary study question of effects of the intervention on proportion of sex acts protected by contraception, our analytic sample included 276 intervention and 278 control participants. For the study question of effects of the intervention on ever being pregnant or ever causing a pregnancy, included 286 intervention and 288 control participants. Variability in the sample sizes is the result of difference in reporting either condom use or contraceptive use. Because youth self-administered the surveys, we have no specific information as to why 11 responses on study measures varied and therefore, do not present any reasons for missing data. The mean age for participants was close to 15 years of age, with most participants identifying as Hispanic (over 40%) or mixed race/ethnicity (over 20%) and smaller proportions identifying as Black (around 20%) or white (around 10%) depending on each outcome assessed. We observed ceiling effects of over 90% for sex acts protected by condoms and contraception, and up to 5% of the sample had experienced or caused a pregnancy. See table C.1a in Appendix C for final sample sizes and response rates. F. Baseline equivalence Baseline equivalence between conditions was examined for demographic characteristics such as participant gender, age, and ethnicity, and for all primary outcome variables. Because individuals are nested within one of 32 groups, statistical tests of baseline equivalence were conducted in a multilevel regression framework. In addition to clustering by the club/year combination, dummy coded Club and Year variables were examined as covariates in the analyses (these were treated as classification variables in SAS, which created 7 dummy codes for Club and 3 for Year). Baseline equivalence was examined among all study eligible participants who completed the baseline survey to test the effectiveness of random assignment to conditions. Tests of baseline equivalence were repeated for the analytic sample for each of the primary and secondary outcomes among the sample of participants who completed the immediate post-program survey to ensure equivalence within the analytic samples. The baseline equivalence results for each of the research questions are presented in Table III.3.1-Table III 3.6. Tables III.4-III.6 show results stratified by Hispanic ethnicity. The two conditions were equivalent for the vast majority of baseline measures. Only pregnancy history was statistically significant (p = 0.03). Pregnancy history at baseline (0=no, 1=yes) was therefore included as a covariate in all impact evaluation analyses. 12 Table III.3.1 Summary statistics of key baseline measures for youth who completed the post- intervention follow-up survey and had complete baseline and posttest data on proportion of protected acts for condom use Intervention Intervention Intervention Comparison versus versus mean or % mean or % comparison comparison (standard (standard mean p-value of Baseline measure deviation) deviation) difference difference Age 14.92 (1.04) 14.98 (1.13) 0.06 .38 Gender (female) 54.39% 50.91% 3.48 .97 Race/ethnicity: White 10.00% 9.91% 0.09 .68 Race/ethnicity: Black 20.00% 18.02% 1.98 .79 Race/ethnicity: Hispanic 46.25% 46.40% 0.15 .75 Race/ethnicity: Multiple/Other 23.75% 25.68% 1.93 .92 Proportion of sex acts protected by 0.95 (0.20) 0.95 (0.21) 0.00 .67 condoms Sample size 240 222 . . Source: TOP® and YAE, 2012-2015 Baseline surveys administered prior to program enrollment among those who also completed a post-intervention follow-up survey and had complete data for proportion of protected acts for condom use (i.e., analytic sample for this outcome). Notes: All analyses conducted in multilevel modeling framework to account for participants nested within clubs/years. Presented p-values are adjusted for dummy coded club and year variables, with exception of analyses for Black race/ethnicity where model did not converge when adjusted for club. 13 Table III.3.2 Summary statistics of key baseline measures for youth who completed the post- intervention follow-up survey and had complete baseline and posttest data on proportion of protected acts for contraceptive use Intervention Intervention Intervention Comparison versus versus mean or % mean or % comparison comparison (standard (standard mean p-value of Baseline measure deviation) deviation) difference difference Age 14.89 (1.02) 14.98 (1.15) 0.09 .60 Gender (female) 53.09% 52.54% 0.55 .75 Race/ethnicity: White 7.99% 8.63% 0.64 .67 Race/ethnicity: Black 17.71% 17.99% 0.28 .43 Race/ethnicity: Hispanic 44.20% 44.60% 0.40 .82 Race/ethnicity: Multiple/Other 26.45% 28.78% 2.33 .69 Proportion of sex acts protected by 0.98 (0.13) 0.97 (0.16) 0.01 .15 contraception Sample size 276 278 . . Source: TOP® and YAE, 2012-2015 Baseline surveys administered prior to program enrollment among those who also completed a post-intervention follow-up survey and had complete data for proportion of protected acts for contraceptive use (i.e., analytic sample for this outcome). Notes: All analyses conducted in multilevel modeling framework to account for participants nested within clubs/years. Presented p-values are adjusted for dummy coded club and year variables, with exception of analyses for Black and Other race/ethnicity where model did not converge when adjusted for club. 14 Table III.3.3 Summary statistics of key baseline measures for youth who completed the post intervention follow-up survey and had complete baseline and posttest data on ever being pregnant or ever causing a pregnancy Intervention Intervention Intervention Comparison versus versus mean or % mean or % comparison comparison (standard (standard mean p-value of Baseline measure deviation) deviation) difference difference Age 14.89 (1.03) 15.00 (1.16) 0.11 .51 Gender (female) 51.58% 51.75% 0.17 .66 Race/ethnicity: White 9.79% 7.99% 1.80 .33 Race/ethnicity: Black 19.23% 17.71% 1.52 .65 Race/ethnicity: Hispanic 43.71% 45.14% 1.43 .70 Race/ethnicity: Multiple/Other 27.27% 29.17% 1.90 .73 Ever pregnant/cause pregnancy 0.35% 4.17% 3.82 .03 Sample size 286 288 . . Source: TOP® and YAE, 2012-2015 Baseline surveys administered prior to program enrollment among those who also completed a post-intervention follow-up survey and had complete data for pregnancy variable (i.e., analytic sample for this outcome). Notes: All analyses conducted in multilevel modeling framework to account for participants nested within clubs/years. Presented p-values are adjusted for dummy coded club and year variables, with exception of analyses for Black race/ethnicity and pregnancy variables where model did not converge when adjusted for club. Note that the statistically significant difference in pregnancy history is based on the relatively rare event for this age group to have experienced a pregnancy. We do not have data to support hypotheses about why one club or clubs would have more pregnancies in intervention years compared to other clubs. Given that each club was assigned to be in the intervention group two years and in the control group two years, we would not expect differences within clubs to affect pregnancy experience. 15 Table III.3.4 Summary statistics of key baseline measures, stratified by Hispanic versus non- Hispanic ethnicity, for youth who completed the post-intervention follow-up survey and had complete baseline and post-intervention data on proportion of protected acts for condom use Intervention Intervention Intervention Comparison versus versus mean or % mean or % comparison comparison (standard (standard mean p-value of Baseline measure . deviation) deviation) difference difference Hispanic 14.96 (1.02) 15.13 (1.18) 0.17 .16 Age Non- 14.88 (1.06) 14.86 (1.08) 0.02 .74 Hispanic Hispanic 60.36% 54.46% 5.90 .89 Gender (female) Non- 49.22% 47.90% 1.32 .84 Hispanic Hispanic 0.93 (0.25) 0.94 (0.24) 0.01 .23 Proportion of sex acts protected by condoms Non- 0.01 0.97 (0.14) 0.96 (0.19) .22 Hispanic Hispanic 111 103 . . Sample size Non- 129 119 . . Hispanic Source: TOP® and YAE, 2012-2015 Baseline surveys administered prior to program enrollment among those who also completed a post-intervention follow-up survey and had complete data for proportion of protected acts for condom use (i.e., analytic sample for this outcome). Notes: All analyses conducted in multilevel modeling framework to account for participants nested within clubs/years. Presented p-values are adjusted for dummy coded club and year variables. 16 Table III.3.5 Summary statistics of key baseline measures, stratified by Hispanic versus non- Hispanic ethnicity, for youth who completed the post-intervention follow-up survey and had complete baseline and posttest data on proportion of protected acts for contraceptive use Intervention Intervention Intervention Comparison versus versus mean or % mean or % comparison comparison (standard (standard mean p-value of Baseline measure . deviation) deviation) difference difference Hispanic 14.92 (1.03) 15.15 (1.19) 0.23 .21 Age Non- 14.87 (1.03) 14.84 (1.10) 0.03 .40 Hispanic Hispanic 56.56% 54.92% 1.64 .70 Gender (female) Non- 50.33% 50.65% 0.32 .88 Hispanic Hispanic 0.98 (0.13) 0.97 (0.16) 0.01 .75 Proportion of sex acts protected by Non- contraception 0.98 (0.12) 0.97 (0.16) 0.01 .37 Hispanic Hispanic 122 124 . . Sample size Non- 154 154 . . Hispanic Source: TOP® and YAE, 2012-2015 Baseline surveys administered prior to program enrollment among those who also completed a post-intervention follow-up survey and had complete data for proportion of protected acts for contraceptive use (i.e., analytic sample for this outcome). Notes: All analyses conducted in multilevel modeling framework to account for participants nested within clubs/years. Presented p-values are adjusted for dummy coded club and year variables. 17 Table III.3.6 Summary statistics of key baseline measures, stratified by Hispanic versus non- Hispanic ethnicity, for youth who completed the post-intervention follow-up survey and had complete baseline and posttest data on proportion of protected acts for ever being pregnant or ever causing a pregnancy Intervention Intervention Intervention Comparison versus versus mean or % mean or % comparison comparison (standard (standard mean p-value of Baseline measure . deviation) deviation) difference difference Hispanic 14.90 (1.03) 15.18 (1.20) 0.28 .13 Age Non- 14.88 (1.03) 14.85 (1.12) 0.03 .43 Hispanic Hispanic 55.20% 55.47% 0.27 .54 Gender (female) Non- 48.75% 48.73% 0.02 .89 Hispanic Undefined Hispanic 0.00% 3.85% 3.85 due to zero Ever pregnant/cause value pregnancy Non- 3.81 0.62% 4.43% .08 Hispanic Hispanic 125 130 . . Sample size Non- 161 158 . . Hispanic Source: TOP® and YAE, 2012-2015 Baseline surveys administered prior to program enrollment among those who also completed a post-intervention follow-up survey and had complete data for pregnancy variable (i.e., analytic sample for this outcome). Notes: All analyses conducted in multilevel modeling framework to account for participants nested within clubs/years. Presented p-values are adjusted for dummy coded club and year variables. G. Methods 1. Impact evaluation Analyses utilized a multilevel linear regression framework, using SAS Proc Mixed for continuous outcomes and a multilevel logistic regression framework SAS Proc Glimmix for categorical outcomes, accounting for nesting within 32 club/year combinations. Analysis of missing data examined whether those lost to follow-up were different at baseline than those retained at TOP® program completion and whether this differed by condition. There were main effects of retention status. Those retained reported higher likelihoods of (a) sexual acts protected by contraception (p = .006), and (b) condom use at last intercourse (p = .019), and lower likelihoods of (a) using substances during intercourse (p = .015), (b) cutting class (p = .004), and (c) being suspended (p = .032). The effect of retention status depended on condition for age at baseline (p = .04), where intervention participants lost to follow-up were older than those retained (p = .006). Baseline values of each respective outcome, club, year, and baseline 18 pregnancy history were included as covariates when predicting outcomes in impact analyses. Each impact analysis included dummy variables for club and year, and was also adjusted for any baseline variables found to be significantly associated with attrition. Primary analyses were conducted as ANCOVA models predicting each outcome from intervention condition and the identified covariates. After applying a Bonferroni correction to adjust for multiple comparisons, the critical p-value for the outcome analyses was p < .017, correcting for three outcomes. Sensitivity analyses were conducted as part of examining the impact of intervention on follow-up outcomes, including: (1) an unadjusted model of intervention effects on outcomes without covariates, (2) multiple imputation to create a complete dataset for all covariates/baseline values to address missing data on relevant covariates (follow-up values based on available sample size), (3) examination of additional covariates compared to the benchmark approach based on differences observed in attrition in six variables mentioned above, i.e. age, sexual acts protected by contraception, condom use at last intercourse, lower likelihood of using substances during intercourse, cutting class and being suspended, and (4) examining changes from pretest to posttest rather than the ANCOVA approach where pretest scores are included as covariates. Secondary subgroup analyses included ethnicity as a factor in the ANCOVA models to examine if the effects of condition depended on ethnicity. This was done by testing the interaction term between ethnicity and condition and including this term in the model along with the main effects of ethnicity and condition and the covariates. Where possible, logical imputation was used for missing values (e.g., those reporting never having intercourse were coded as “no” on the pregnancy variable); however, when participants chose “do not want to answer” they were coded as missing for that question. 2. Implementation Evaluation a. Evaluation of TOP® delivery All of the items evaluated for the implementation evaluation are noted in Table D.1 in the appendix. We conducted a simple descriptive evaluation to document how many sessions were attended and how many hours of service learning completed, and the quality of staff-participant interaction. b. Analysis of Text message engagement We reviewed the total number of text messages sent, and documented the number of responses to bi-directional text messages per participant. These data were analyzed to document overall and mean participant response frequency and we employed chi square, z test, and ANOVA analyses to identify any statistical differences in participant response frequency by demographic characteristics including age, gender and race/ethnicity. We also examined message response in the aggregate, by looking at the total number of messages sent by participants in response to all bi-directional messages. We then reviewed these response frequencies by message type, i.e., question, quiz, club reminder, myth/fact, polls and club reminders and by content as it related to the eleven curriculum elements of TOP®, such as volunteering, gender roles, influence, decision making, as reported in Devine et al.xviii Finally, we explored the frequency of response over time. We report on text message engagement for the first two years of the program. 19 IV. Study findings A. Implementation study findings 1. TOP® program implementation The Boys & Girls Club management team assured appropriate ratios of no more than 25 participants per facilitator by staffing two trained TOP® facilitators per club. CYM delivered between one to two trainings in every club each year to ensure new and ongoing facilitators received TOP® required training. Over the course of the program, CYM provided five facilitator trainings to 26 Boys & Girls Club facilitators. The three yearly observations in clubs were typically scheduled, in October-November, January-February and March-April. Typically in the first observations of each year, TOP® facilitators would score between 3-4 on a 5-point scale on fidelity and quality assessments. By the year’s final observation they were scoring between 4-5, showing steady improvement and positive changes throughout the year. All facilitators delivered all intended sessions (100% of those planned). These results are shown in Table E.1 in the Appendix. The suggestions for how facilitators may improve specific content delivery based on observations of quality of program delivery are shown in Table E.2 in the Appendix. YAE participants attended a mean of 10.9±8.8 sessions (42% of those offered) with a mean of 10.8±13.9 community service learning hours and TOP® participants a mean of 13.0±8.9 sessions (50% of those offered) and 12.5±12.9 community service learning hours. This represented low adherence in TOP® implementation. Anecdotally, facilitators indicated that youth tended to drop out of the program early, after attending only a few sessions, or after the first semester, given competing priorities for other spring activities such as basketball. 2. YAE Implementation We have completed assessment of YAE Implementation for the first two years of the project. In total, 40,006 text messages were sent to 221 program participants during the first two years. Of these, 16,501 messages (41%) were bi-directional messages that requested a response (e.g., quizzes, polls, etc.). The remaining 23,505 messages were unidirectional and did not request a response (e.g., facts, resources, auto-reply messages). Each participant was sent an average of 74.6 messages. We received 2,764 responses to the 16,501 bi-directional messages (16.8%). Among the 221 intervention participants in YAE in years one and two, 41 (18.5%) did not respond to any of the text messages they received. Just over one-fifth (21.6%) responded to between 1-3 messages, and just under one fifth (19.8%) responded to between four and nine messages. The remaining 40.1% responded to ten or more messages. Participants responded an average of 12.5 times to bi-directional messages overall. The number of responses among participants varied based on participant characteristics and message category. There were statistically significant differences by age, ethnicity, and gender (p < .001). Participants ages 16 and 18 responded more frequently (an average of 18.6 and 19.3 responses received per participant over the program year) as did Hispanic (14.2 messages compared to 11.0 for non- Hispanic), and female (15.2 messages compared to 9.7 for male) participants. 20 Messages categorized as questions received the most response (18.8% of outbound messages received a response), followed closely by quizzes (17.6%), club reminders (17.4%), polls (16.0%), and myth/facts (16.4%). Responses were also analyzed by “content” area corresponding to the curriculum elements of TOP®. Texts about gender roles (19.7%), values (19.5%), goal setting (19.4%), volunteering (17.2%), sexuality (16.7%), and relationships (16.1%) received the highest numbers of responses (average responses by topic shown in parentheses). Response averages for the other five curriculum elements ranged in frequency from 15.1% (decision-making and influence each) to health and hygiene (9.6%). Other topic areas covered in the curriculum include development (14.7%) and communication (13.6%). Differences in response rates between the content areas and the other topics were significant (p= .001). Responses to bi-directional text messages decreased over the course of the program year. Each program year started in the fall (late September or October) and ended in May. Response frequency decreased throughout the year with a slight increase in January when participants returned to programming after winter break. We experienced several barriers to participants consistently receiving text message programming. Some of these barriers were technical (e.g., phone issues, text system issues) and others were individualized issues (e.g., stop requests received). For participants using mobile phones (78.3% of all participants), there were frequent issues with mobile phones being broken, lost, or stolen. These issues were reported to program facilitators. Participants using the web-based Textfree service to receive messages (19.0%) were able to access their text messages from any web-enabled device, but these Textfree accounts expired after four weeks of inactivity, leading to some participants needing to be periodically re-enrolled with new accounts. Participants using Textfree were also less likely to respond to messages (average 1.8 responses per participant) compared to participants using a mobile phone (average of 15.1 responses), or other web-based texting capability (average of 13.2 responses). Some participants also opted out of programming; in total, 79 “stop” request messages were received from 35 unique participants across all four years of programming. The most “stop” requests were received in November (11 participants) and January (10 participants) and were more likely to come from younger participants (ages 14 and 15 year of olds, 13 participants each). Some of these stop requests were due to incorrect numbers (e.g., participant gave out a family member’s mobile number, Textfree account number expired and was re- assigned, etc.) and others were from participants who wished to opt-out from the text message programming. All but two of the participants who put in “stop” requests ultimately chose to remain in the program after speaking with a facilitator. Facilitators reported that these were participants who had stopped attending the TOP® sessions. Facilitators indicated that after a brief review of the purpose of the program, participants generally agreed to continue receiving messages. Delays in facilitator-participant communication sometimes meant that text messages would be turned off for participants for a period of time. Text message suspension in these cases ranged from one day to the remainder of the program year. In several cases, facilitators could not reach those making a “stop” request. These were participants who had both stopped attending the TOP® sessions and coming to the Boys & Girls Clubs altogether. We were not able to reach them to document their reasons for their “stop” request. 21 The PRM SMS system experienced occasional technical issues. Across the first two program years, there were 28 PRM system-related problems with the majority of the issues experienced in the fall at the beginning of the program year (20 from October to December compared to 8 from January to May). Of these 28 identified problems, 71% related to message delivery (multiple duplicate messages sent to all participants), 14% were associated with message scheduling (messages sent at the wrong time of day), 7% involved message content (truncated messages), and 7% involved message status reporting (no data recorded on whether message was sent). All problems were corrected promptly, but may have affected data quality and accuracy, program fidelity, and participant experience. After the second year of programming we had no additional technical issues that affected message delivery. B. Impact study findings 1. Primary analyses: Intervention impact Table IV.1 shows follow-up proportions for the primary outcomes. Post intervention estimated effects show no significant difference between TOP® alone and YAE. Specifically, there was no difference in the extent to which the students in the two groups avoided risky sexual behavior by avoiding sex altogether or using protection when they did engage in sex acts, in the three months preceding the point of measurement. We calculated effect size estimates for each outcome to demonstrate the potential effects possible when implementing the intervention above and beyond TOP® alone, though effects were small at Cohen’s d = .04 and d = .06. Small effect sizes may be attributable, in large part, to the high proportion of abstainers in the dataset, resulting in ceiling effects on the two primary outcomes. Age was found to be significantly associated with attrition, and thus all impact models were adjusted for age. Table IV.1. Post-intervention estimated effects using data from TOP® and YAE to address the primary research questions Intervention compared to comparison Intervention Comparison mean proportion proportion difference (p- (standard (standard value of Outcome measure deviation) deviation) difference) Proportion of sex acts protected by condoms 0.93 (0.21) 0.92 (0.24) 0.01 (.63) Sample Size 240 222 . Proportion of sex acts protected by contraception 0.99 (0.14) 0.98 (0.19) 0.01 (.46) Sample Size 276 278 . Source: TOP® and YAE 2012-2015 Follow-up surveys administered at post-intervention follow-up Notes: Adjusted means and standard deviations are reported. Presented p-values are adjusted for baseline scores, baseline pregnancy history, participant age, and dummy coded club and year variables. See Table III.3 for a more detailed description of each measure and Section III for a description of the impact estimation methods. 22 2. Secondary analyses: Intervention impact, by ethnicity Our secondary research questions explored the effects of YAE versus TOP® on ever being pregnant or causing a pregnancy and explored whether YAE was differentially effective for Hispanic versus non-Hispanic youth. These were considered secondary analyses as the study was not specifically powered to conduct them. As shown in Table IV.2, there was no effect of YAE versus TOP® on ever being pregnant or causing a pregnancy. However, there was a significant interaction between condition and Hispanic ethnicity on pregnancy status at follow-up (p = .02), with condition differences observed among Hispanics but not among non-Hispanics (see Table IV.3). The percentage of pregnancies reported among Hispanics in the control condition was 6.15%, compared to 1.60% among Hispanics in the intervention condition. This finding was the only significant finding of a large number of effect modifier analyses, though it provides initial evidence that the intervention may be particularly beneficial for Hispanic individuals in preventing pregnancy. Table IV.2. Post-intervention estimated effects using data from TOP® and YAE to assess effects on ever being pregnant or causing a pregnancy Intervention compared to Intervention Comparison comparison mean or mean or mean proportion proportion difference (p- (standard (standard value of Outcome measure deviation) deviation) difference) Ever pregnant/cause pregnancy 0.19 0.13 0.69 (.55) Sample Size 286 288 . Source: TOP® and YAE, 2012-2015 Follow-up surveys administered 9 months after program enrollment. Notes: Presented p-values are adjusted for baseline pregnancy history, participant age, and dummy coded club and year variables. See Table III.3 for a more detailed description of each measure and Chapter III for a description of the impact estimation methods. Table IV.3. Post-intervention estimated effects by Hispanic versus non-Hispanic Comparison mean or Intervention mean or proportion proportion (standard (standard deviation) deviation) Ethnicity by Condition non- Interaction Outcome measure Hispanic non-Hispanic Hispanic Hispanic p-value Proportion of sex acts protected by condoms 0.95 (0.21) 0.95 (0.21) 0.92 (0.26) 0.94 (0.22) .66 Proportion of sex acts protected by contraception 0.99 (0.10) 0.97 (0.16) 0.96 (0.20) 0.97 (0.18) .35 Ever pregnant/cause pregnancy 0.16 0.37 0.62 0.13 .02 23 3. Sensitivity analyses The results of several sensitivity analyses are shown in Appendix F, Table F.1. The finding of no difference between TOP® alone and YAE for the primary outcomes was maintained across the sensitivity analyses. 24 V. Conclusion We found no statistically significant differences between participants in YAE compared to TOP® with regard to proportion of sex acts protected by condoms or contraception or pregnancy. Overall, the treatment and the control groups were about equally likely to avoid risky sexual activity, by avoiding sex altogether (most students abstained) or by using condoms or contraception when they did engage in sex acts. We did, however, find that a significantly lower percentage of Hispanic teens exposed to YAE had pregnancies compared to the control group. We consider this a robust and methodologically rigorous study that has generated multiple important findings. We are cautious about recommendations that YAE be replicated using the current TOP® programming model of 25 sessions over a nine month period. Our implementation data suggest it is challenging for youth to attend the required 25 sessions of TOP®. With a mean attendance of 11.9 sessions of 25 required, it is difficult to assume the TOP® program can be replicated with fidelity in Boys & Girls Clubs. Based on observations of TOP® facilitators and Boys & Girls Club staff, the expectations for youth to remain engaged over nine months and attend 25 weekly sessions is unrealistic. Our recommendations would be to repackage the TOP® program as a shorter program designed for delivery over three months to improve program engagement. This may increase benefit of the text message enhancement, given other data showing declines in engagement with text over time.xviii The significant positive difference in ever being pregnant or causing a pregnancy among Hispanic participants in YAE compared to TOP® is notable, but should be interpreted with caution as it is the only significant finding of a large number of modifier analyses. Future analyses will consider whether there is a relationship between greater engagement with the YAE text messages and pregnancy outcomes for the sample and the subgroup of Hispanic youth. However, it provides initial evidence that YAE may be particularly beneficial for Hispanic individuals in preventing pregnancy. Given this population nationally experiences the highest rates of teen pregnancy compared to White or African American teens, this is an encouraging result. The primary limitation of this research is related to small numbers of sexually active youth. Our initial sample size estimates were based on populations with similar demographic characteristics, but the Boys & Girls Club population exhibited lower sexual activity in comparison. The low percentage of sexually active participants led to ceiling effects in the primary outcomes, making it difficult to detect group differences. Because these data were from a trial where all 14- to 18-year-old members of eight Boys & Girls Clubs were eligible for participation, we are confident our findings can generalize to the population of youth in this age group who are Boys & Girls Club members, although generalizing beyond this group would require being able to replicate these findings with other audiences in the same age range. 25 VI. References i The National Campaign to Prevent Teen and Unplanned Pregnancy. Making the Case: For Wanted and Welcomed Pregnancy 2015. 2015. http://thenationalcampaign.org/why-it-matters. Accessed March 15, 2015. ii Colorado Department of Public Health and Environment HSS. National Vital Statistics Denver, CO 2004-2013. iii Colorado Department of Public Health and Environment HSS. National Vital Statistics Denver, CO 2013. iv Colorado Department of Public Health and Environment. STI Reporting 2014. 2014. https://www.colorado.gov/pacific/cdphe/sti-and-hiv-data-and-trends. Accessed March 15, 2015. v Singh S, Darroch JE. Adolescent pregnancy and childbearing: levels and trends in developed countries. Family planning perspectives. 2000;32(1):14-23. vi Ricketts S, Klingler G, Schwalberg R. Game change in Colorado: widespread use of long-acting reversible contraceptives and rapid decline in births among young, low-income women. Perspectives on sexual and reproductive health. 2014;46(3):125-132. vii Colorado Children's Campaign. Colorado Kids Count. 2015. http://www.coloradokids.org/data/kidscount/. Accessed March 15, 2015. viii The National Campaign to Prevent Teen and Unplanned Pregnancy. What Works: Curriculum- Based Programs That Help Prevent Teen Pregnancy. 2012. http://thenationalcampaign.org/resource/what-works. Accessed March 15, 2015. ix WYMAN. Wyman's Teen Outreach Program (TOP). Wyman. 2013. http://wymancenter.org/nationalnetwork/top/. Accessed 1/18/2013. x Allen J, Philliber S, Herrling S, Kuperminc G. Preventing teen pregnancy and academic failure: experimental evaluation of a developmentally based approach. Child Dev. 1997;64:13. xi Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health education research. 2006;21(5):688-694. xii Botvin GJ, Griffin KW, Diaz T, Ifill-Williams M. Preventing binge drinking during early adolescence: one- and two-year follow-up of a school-based preventive intervention. Psychology of addictive behaviors: Journal of the Society of Psychologists in Addictive Behaviors. 2001;15(4):360-365. xiii Shrier LA, Ancheta R, Goodman E, Chiou VM, Lyden MR, Emans SJ. Randomized controlled trial of a safer sex intervention for high-risk adolescent girls. Archives of pediatrics & adolescent medicine. 2001;155(1):73-79. xiv Villarruel AM, Jemmott LS, Jemmott JB, Eakin BL. Recruitment and retention of Latino adolescents to a research study: lessons learned from a randomized clinical trial. Journal for specialists in pediatric nursing: JSPN. 2006;11(4):244-250. xv Lenhart A MM, Rankin Macgill A, Smith A. Teens and social media: The use of social media gains a greater foothold in teen life as they embrace the conversational nature of interactive online media. 2007. xvi Lenhart A. 9 things you need to know about teens, technology and privacy. PEW Internet and American Life. 2013. http://www.pewinternet.org/2013/11/07/9-things-you-need-to-know-about-teens- technology-online-privacy/. Accessed 2/1/2014. xvii Horrigan J. Mobile Access to data and information. PEW Internet and American Life. 2008. xviii Devine S, Leeds C, Leytem A, Beum R, Shlay JC, Bull S. Methods to assess youth engagement in a text messaging supplement to an effective teen pregnancy prevention program. J Biomed Inform. 2015. 26 VII. APPENDICES: TABLES AND FIGURES Appendix A: Program materials and data collection efforts Table A.1. Examples of text messages sent to YAE Participants Desired theoretical Type Content outcome Question TOP&theClub: If ur friend got pregnant, what would u tell Social Support her to do? Quiz TOP&theClub: Where do people in Colorado volunteer Civic Engagement most? 1=hospitals 2=education/schools 3=religious places 4=other/unsure Club TOP&theClub: Ur TOP club will meet at <club name> on Cue to Action Reminder <date> at <time>. Will u go? Reply 1=yes, 2=no, 3=not sure Myth-Fact Female: Myth/Fact: If a guy wants to have sex, it's his Norms (tailored for responsibility to get the condoms. 1=myth 2=fact gender) 3=unsure. Male: Myth/Fact: If a girl wants to have sex, it's her responsibility to get birth control. 1=myth 2=fact 3=unsure. Myth-Fact- TOP&theClub: Myth or Fact: "Less than 50% U.S teens Norms are having sex." Reply 1=myth 2=fact 3=unsure Myth-Fact- TOP&theClub: It’s a myth! The Centers for Disease Norms Control reported that only 43% of U.S teens are having sex. Not every1 is doing it. Poll Will an unplanned pregnancy prevent you from reaching Improved Future your goals? Text 1=yes; 2=no; 3=unsure Orientation (social capital theory) Fun Fact TOP&theClub: 1 out of 3 teens say that it's hard to talk Positive Norms re: about condoms. Think safe sex is important? Learn to Healthy how to talk about it! Communication Quote TOP&theClub: "Communication is key." “Be polite...you Role Modeling for don't start taking each other for granted ever, you know.” - Healthy Ice Loves Coco Communication 27 Table A.2. Data collection efforts used in the impact analysis of YAE and timing Data collection effort Cohort 1 Cohort 2 Cohort 3 Cohort 4 Start date of 9/19/11 9/24/12 9/23/2013 9/22/14 programming Baseline survey* 9/15-9/29/11 9/24-10/4/12 9/16-10/3/13 9/22-10/6/14 Immediate post 4/23-5/7/12 4/29-5/9/13 4/21-5/8/2014 4/20-5/5/15 survey 12-month post 5/13-5/23/13 4/21-5/5/14 4/20-5/7/15 not applicable program survey *Youth were not permitted to attend any programming until after they had completed the baseline survey. Table A.3. Data used to address implementation research questions Types of data used to assess whether the element of the intervention was Implementation implemented as Frequency/sampling Party responsible element intended of data collection for data collection Adherence: How often All sessions offered Monthly Program staff were sessions are captured by offered? How many documenting session were offered? attendance in a reporting system to the funder a Adherence: What and Session attendance Weekly Program staff how much was records captured by received? documenting session attendance in a reporting system to the funder a Adherence: What Topics covered Classroom Colorado Youth content was delivered captured by observations occurred Matter (official trainer, to youth? documenting session three times a year fidelity monitor) attendance in a Discussion of content Leadership Team reporting system to every two months by the funder a Leadership Team Observation Logs a 28 Types of data used to assess whether the element of the intervention was Implementation implemented as Frequency/sampling Party responsible element intended of data collection for data collection Adherence: Who List of staff members Discussion of content Program staff delivered material to hired and trained to every two months by youth? implement program Leadership Team Background qualifications of staff members from staff applications Quality: Quality of Observations of Convenience sample Colorado Youth staff-participant interaction quality of 3 sessions were Matter (official trainer, interactions using protocol selected for fidelity monitor) developed by Wyman observation per Club per year Quality: Quality of Observations of Three sessions per Colorado Youth youth engagement engagement captured club per year were Matter staff with program by documenting selected for session attendance in observation a reporting system to the fundera Implementation of System delivery logs Continuous Denver Health Text Messages from the PRM system throughout program to document time/date implementation of text message delivery for all in bound and outbound messages Counterfactual: Survey items on Post-intervention Evaluation staff Experiences of baseline and follow- comparison condition up assessments to document awareness of and satisfaction with TOP Context: Other TPP a Interview with lead Once per year Evaluation staff programming program staff to available or offered to determine if they are study participants using or offering other (both intervention and TPP programs comparison) 29 Types of data used to assess whether the element of the intervention was Implementation implemented as Frequency/sampling Party responsible element intended of data collection for data collection Context: External Not measured N/A N/A events affecting implementation Context: Substantial Not measured N/A N/A unplanned adaptation(s) a OAH-approved instruments used for this data collection. TPP = Teen Pregnancy Prevention. 30 Appendix B: Study consort diagram FIGURE 1: Consort diagram 31 Appendix C: Study sample Table C.1a. Cluster and youth sample sizes by intervention status Total Total sample Intervention Comparison response Intervention Comparison Number of: Time period size sample size sample size rate response rate response rate Clusters: At beginning of study . 32 16 16 N/A NA N/A Clusters: Contributed at least one youth at baseline Baseline 32 16 16 100% 100% 100% Immediately Clusters: Contributed at least one post- youth at follow-up* programming 32 16 16 100% 100% 100% Youth: In non-attriting clusters/sites at time of assignment ** . 3,643 1,764 1,879 N/A NA N/A Youth: Who consented . 854 438 416 23% 25% 22% Youth: Contributed a baseline survey . 852 436 416 23% 25% 22% Immediately post- Youth: Contributed a follow-up survey programming 632 317 315 74% 73% 76% Youth: Contributed follow-up survey Immediately and had complete data on proportion post- of protected acts for condom use programming 462 240 222 73%. 75% 70% Youth: Contributed follow-up survey and had complete data on proportion Immediately of protected acts for contraceptive post- use programming 554 276 278 88% 87% 88% Youth: Contributed follow-up survey and had complete data on ever being Immediately pregnant or ever causing a post- pregnancy programming 574 286 288 91% 90% 91% *One club in 2012 and one club in 2013 was replaced because their teen census dropped substantially and teen programs were eliminated in these clubs. The replacement clubs assumed the randomization status of the original club and no re-randomization occurred. **Numbers represent youth personally invited to enroll 32 Appendix D: Implementation evaluation methods and selected outcomes Table D.1. Methods used to address implementation research questions Implementation element Methods used to address each implementation element Adherence: How The total number of sessions is the sum of the sessions captured in the often were sessions program. Average session duration is calculated as the average of the offered? How many observed session lengths, measured in minutes. Average weekly were offered? frequency is calculated as the total number of sessions divided by the total number of weeks when programming was offered. Average number of sessions attended is calculated as the average of the Adherence: What number of sessions that each student attended. Percentage of sessions and how much was attended is calculated as the total number of sessions attended divided received? by the total number of sessions offered. Adherence: What Topics are documented as they are covered during the observation. content was (Note: a limitation to this measure is that three observation points may delivered to youth? not be a reliable way to see whether all of the content was covered.) Adherence: What text message We documented how many text messages were sent and responded to, content was as described in the text. We also documented “Stop” requests, i.e. delivered to youth? requests from participants to stop receiving text messages Total number of staff delivering the program is a simple count of staff Adherence: Who members implementing the program. Percentage of staff trained is delivered material to calculated as the number of staff members who were trained divided by youth? the total number of staff who delivered the program. (Note: a limitation to the staff background information is that it is self-reported, and some staff may have indicated they had experiences that are not accurate.) An indicator of staff-participant interactions is calculated as the Quality: Quality of percentage of observed interactions in which the independent evaluator staff-participant scored the interaction as “high quality” on a scale of 1-3 using an OAH interactions approved measurement. (Note: because a convenience sample of observations was used to capture staff-participant interaction quality, this measure may not be representative of all possible interactions.) Quality: Quality of A benchmark of the quality of youth engagement is calculated as the youth engagement percentage of sessions in which the independent evaluator scored youth with program engagement as “moderately engaged” using a scale of 1-3 where 1=no engagement and 3= completely engaged. Counterfactual: Experiences of counterfactual The data on the survey question on experiences of the counterfactual at condition follow-up are presented as frequency counts and percentages. 33 Implementation element Methods used to address each implementation element Context: Other TPP programming available or offered to study participants (both intervention Interview with lead program staff to determine if they are using or offering and counterfactual) other TPP programs Context: External events affecting implementation None (not applicable) Context: Substantial unplanned adaptation(s) None (not applicable) TPP = Teen Pregnancy Prevention. 34 Appendix E: Implementation Evaluation Table E.1 Fidelity and Quality for delivery of TOP® Findings related to changes in knowledge about TOP® and confidence delivering TOP® content following training for facilitators (2011-2014) How knowledgeable/ How confident were knowledgeable/ you before this confident are As I think about implementing session? you now? TOP®, I feel I can... N=26 N=26 % change Effectively set ground rules with 3.4 3.9 15% increase youth Facilitate lessons format the 2.5 3.8 52% increase TOP® changing scenes curriculum using experiential learning cycle Provide accurate factual 3.3 3.5 6% increase information about content related to sexuality Facilitate conversations around 3.4 3.7 9% increase sensitive subjects in values neutral perspective Ensure my facilitation style 2.9 3.7 28% increase addresses multiple intelligences and varied learning styles of youth participants Use various reflections 2.8 3.7 32% increase techniques to help youth learn from their experiences Support youth in planning and 2.9 3.7 28% increase executing a service learning project Plan and sequence a school year 2.4 3.7 54% increase of TOP®, including changing scenes curriculum and community service learning Tell others about the impact 2.5 3.8 52% increase TOP® can have for youth 35 Findings related to changes in knowledge about TOP® and confidence delivering TOP® content following training for facilitators (2011-2014) How knowledgeable/ How confident were knowledgeable/ you before this confident are As I think about implementing session? you now? TOP®, I feel I can... N=26 N=26 % change Explain how to implement TOP® 2.3 3.7 61% increase so that our organization is consistently following the TOP® approach 36 Table E.2 TOP® Quality Monitoring re Session Delivery 2011-2015 2011-2015 Implementation Planning Document Lesson Comments, suggestions, changes provided by session observers from #/name Colorado Youth Matter, trainers for delivering the TOP® curriculum 3.10/2.10 Intro • Be prepared for personal questions about your own values and how to Values you will respond to them. • Do a good ‘get to know you’ ice breaker before this lesson-: Would You Rather… • Have the teens draw their own interpretation of their values. • Add love to the values the teens had to vote on. • After clustering the word ‘Values’ as a large group, have a concrete and concise definition written down to reveal to bring the activity all together. • Give working definitions for some of the values (in a value neutral way) in advance to get a shared understanding of some of the values. • It’s hard to stay values neutral. • Handout needs to be updated. 3.11 Value • Pre read all value statements so you are prepared to discuss them in a Voting values neutral way. • Be prepared to facilitate this in different ways depending on the kind of group you have and how much they like to participate (body voting, holding up signs, note cards etc.) • Remind of group rules and that the goal is not to have everyone agree but to be heard. • Incorporate movement. • Plan to change the wording around in the situations for greater spectrum of voices. • With a bigger group, it is more difficult to give youth equal speaking time and invite the quiet youth to share. • Be aware of the time especially if teens start really opening up and sharing. • The situations presented seemed are really relevant to their experiences. • The writing prompt is very successful in bringing the lesson all together. • Values neutral is really important and hard for this lesson. • Change/eliminate the rape values statement. 37 2011-2015 Implementation Planning Document Lesson Comments, suggestions, changes provided by session observers from #/name Colorado Youth Matter, trainers for delivering the TOP® curriculum 4.12 Value • Make sure they know what an auction is and looks like (YouTube Auction video prior to lesson.) • There is lots of flexibility in this lesson to run the session how you want-be creative. • Can do in a silent auction format or half live auction, half silent- depending on group. • Do an Icebreaker to jump start the conversation. (i.e., someone like me) • Youth get the value to hold on to after they bid on it. Certificates could be an extra step. • Can use more money/higher values for a bigger group. • Ties- if more than 1 person bids the all their money on the same value? Or if youth give others their money? Make your own rules before the lesson that feel most comfortable. • Smaller groups for discussion/debrief. • Remind of group rules and respecting different opinions. • Brainstorm values before the auction. Add in values the youth have identified as important to them to the auction. • Arrange chairs in a circle for maximum participation. • Facilitators need very high energy, fast pace and to make the values auction as fun as possible-teens will be very engaged. • It’s important to have teens process through and discuss what values were most important to them and why and understanding that their values will differ from others in the room. • Get silly (wear hats, having your faces on the dollar bills) this gives space for the youth to become engaged and let loose. • Need better framing about the purpose and meaning of this activity and what it symbolizes. 38 2011-2015 Implementation Planning Document Lesson Comments, suggestions, changes provided by session observers from #/name Colorado Youth Matter, trainers for delivering the TOP® curriculum 2.8 Using • A computer lab is great place to do this lesson. Community • Know the resources available in the community you are working in. Resources • Be prepared to help search- keywords or topics, maybe a cheat sheet. • Worksheet created might be helpful for all groups. • Brainstorm the of need community resources instead of using given scenarios. • Create a simple “Resource Directory” worksheet that is easier to follow and contains more relevant information. Allow adequate time for the research directory portion of the lesson. • Split into groups - each group reads a specific scenario, talks about it amongst themselves and then presents on it to the group. – Better for time management. • Rather than giving participants a list of scenarios, brainstorm as a group what type of problems people in the community might face that are too big to handle on their own. • Difficult to facilitate with a large group as youth each need a lot of individual attention. • The scenarios created engaged the youth and put a personal perspective to the discussion. 3.32 Intro to • Ice breaker: hot potato with decisions. Decision • Write out steps like a recipe. Making & • Ways people make decisions matching activity. "Would you • Impulse decision-making is hard to explain. Rather?" • Using group scenarios as examples to discuss. • Both group and individual work well for this lesson • Teens wrote down examples of difficult choices they’ve had to make recently, balled it up and threw it in the middle of the group; this was how we chose our example of the decision making model. • Recreate worksheet to make it easier for the teens to follow. • Opening discussion about the ways people make decisions. • Make this lesson more active • Some decisions are personal and the teens may want to keep them private. Provide an alternative option when providing instructions on this portion of the lesson. • Create new scenarios for the group wide modeling that are relevant to teens. • Some teens need help coming up with relevant examples and struggled with this decision making model. • This lesson needs better debrief questions. 39 2011-2015 Implementation Planning Document Lesson Comments, suggestions, changes provided by session observers from #/name Colorado Youth Matter, trainers for delivering the TOP® curriculum 4.25 The Art of • Prep the participants who are performing the experiment with the skits. Communication • Make sure you understand the directions to all the experiments and they are clearly explained • During experiment 2, ensure groups are completely separated and cannot hear each other. • Use a flip chart paper to have a list of good and bad communication techniques and add to it throughout the lesson. • Eliminating the communication evils handout helps get through the lesson with more time for thoughtful discussion. • It is important to go over the difference between aggressive and assertive. • Be prepared to be actively involved in the reflection of this lesson. 2.27 Speaking • Clarify instructions - the way they are written is a bit confusing. Up for Yourself • Use the scenarios teens write. • Present definitions and have the youth brainstorm what the definitions mean. • Find ways to make assertiveness more relevant to them. • Prepare group for the role plays so they know what will be expected of them. • Reviewing I messages- have a poster of I messages in the room. • Time is a challenge. The discussion may take longer with a talkative group. • Topic resonated with the youth-especially with the role plays and being assertive. • Lesson is set up very well-good flow. • Role plays are very important in illustrating how difficult assertive techniques can be to use. • Youth saw the connection between the “speaking up for yourself” techniques and the list. given to them in regards to healthy relationship behavior. • A high energy ice breaker keeps the energy up throughout the lesson. • Humor is a huge tool during this lesson. 40 2011-2015 Implementation Planning Document Lesson Comments, suggestions, changes provided by session observers from #/name Colorado Youth Matter, trainers for delivering the TOP® curriculum 3.19 What is • Sex vs. sexuality. – Important to really clarify and make sure they all Sexuality understand. Teens have a hard time getting past the “SEX” or sexy part of sexuality. • This lesson may take longer as there are lots of questions. (Have question box out.) • Walk around while they are working on presentations to answers questions. • Play music while they work on presentations. • Give examples for presentations and provide resources (magazines, speakers, etc.) so they can use different kinds of media for the presentations. • This lesson uses and encourages lots of creativity and an opportunity for the youth to lead and facilitate. • Some youth focused on sexuality to mean sexual orientation and the LGBTQ Community. • Start with a stereotype icebreaker to get into the topic of sexuality. • Do an ice breaker that mixes them up so they are not sitting with friends. • Let youth know its ok to step out if they are uncomfortable with the topic. • Lesson well laid out and takes the full hour. LGBTQ FLASH • Time is a very big issue in this lesson. A lot of information for 1 hour. • Do 2 LGBTQ Lessons—One basic level: definitions. One more in depth: gender socialization (which is the root of why it is difficult for people to understand sexual identity and gender identity). • Setting guidelines is very important. • Need more activity-lesson feels lecture heavy. • Genderbread person is a great tool. Handout clearly lays out spectrums of sexuality and the difference between sex and gender. • Have the LGBTQ terminology written out on poster board prior to the lesson. • Discussion on how one can be an ally or combat bullying. • Divide into 2 groups for the terms and definitions. • Video- “Love is all you need” to replace video listed in FLASH lesson. • Added “Biological SEX” and “A-Sexual” to list of definitions. • Ice Breaker: Stand UP/ Sit down- Target for discrimination activity. • Coming out stars is a great activity and good fit for this lesson but takes too long as an icebreaker. May need to find another time to do it meaningfully. 41 2011-2015 Implementation Planning Document Lesson Comments, suggestions, changes provided by session observers from #/name Colorado Youth Matter, trainers for delivering the TOP® curriculum FLASH healthy • Time is an issue with this lesson- all content is relevant but hard to fit relationships in an hour. • Not enough time to dedicate to role plays. • “Evaluating a Relationship” worksheet served as a great wrap-up / self-reflection. • LoveIsRespect.org has lots of great resources for information on Relationships for any future lessons.(Relationship spectrum handout) • Rewrite scenarios to be more relevant to your youth. • The youth were engaged in the “How I want to be treated” activity and were very willing to share their personal opinions. 4.26 • Do an Ice breaker that helps to define these terms. Assertiveness • Give the definitions and help the teens form their own understanding Techniques of the definitions. • Write own scenarios or make the role plays more personal to the teens by changing names, locations, and using situations teens have mentioned. • Separate groups. • Have youth do role-plays for the whole group rather than in pairs. • Review Assertiveness and exploring reasons why it’s a better approach to a situation over being passive or aggressive is important. • It’s repetitive to the “Speaking UP for Yourself” lesson but good review of the themes they’ve been looking at and how they have/will apply them to life. Maybe review for 30 minutes then add another lesson to introduce something new. FLASH STI • Ice Breaker: Hand shake/STI contraction demonstration. • Utilize the question box. Have note cards for questions. • Cover windows so others cannot see in-due to video. • Pass out ‘In case you are curious’ handouts at end (or other resources.) • Prepare for this lesson in advance and address any baggage from past STI lessons. 42 2011-2015 Implementation Planning Document Lesson Comments, suggestions, changes provided by session observers from #/name Colorado Youth Matter, trainers for delivering the TOP® curriculum 3.16 Intro to • Change guided imagery or don’t do it. Not conducive to large groups. Romantic • Set time limits for brainstorming or do it in small groups. Difficult Relationships + discussions for large groups. Healthy • Do a relevant ice breaker. Music trivia ice breaker- all songs had to do Relationships with love. Icebreaker • Structure of the lesson was very successful in allowing time for open discussion. • As a “While you wait” activity we asked all teens to write/draw what loves means to them. At the end of the lesson we went back to their original answers and reflected on any additional insight that they would like to add. • Add online situations to meeting people. • The level of challenge was not very high. • Each group wrote their responses on sticky notes then brought the groups together and compiled the lists into the Venn-diagram. This was a successful approach to the activity. • Groups were able to identify rather quickly that an ideal romantic partner would have the same qualities as a best friend. 3.17 Influences • Rather than using the participant handout, we hung poster board on Love & 2.18 around the room, each labeled with a different source heading. As a Handouts group, we allowed individuals 7 minutes to wonder, explore, and add to each heading accordingly. After one minute, each team rotated clockwise until each team had the opportunity to add their thoughts to each flipchart. Group discussion to follow. Or use post-its so they can work individually first. • Created an “Agree/Disagree” movement game for the “How do you know its love” handout rather than doing the handout itself. • Great debrief questions. Helps lead to good discussions. • Ice breaker: name that love song, healthy relationship ice breaker. • Create more movement out of this lesson. • Open the room up for more peer to peer discussion and teen facilitation. • Review group rules-respectful place for different views of love. • Use the handout as the icebreaker to get the teens thinking about their ideas on love. • This was overall a really great lesson that brought up some great discussions. • The teens wanted explanations on what the worksheet statements meant. • Maybe a tough lesson for younger youth- very conceptual. • Find ways to go deeper with this lesson. 43 2011-2015 Implementation Planning Document Lesson Comments, suggestions, changes provided by session observers from #/name Colorado Youth Matter, trainers for delivering the TOP® curriculum 1.15 Anatomy • Review groups rules. Review • Prepare for this lesson in advance and let youth know its coming. • Prepare the space for privacy and safety. • Have the anonymous question box present and encourage questions. • More clarity needed on what to cover- Terms or processes. • Relay race is great way to end lesson. • Did not do the line graph activity, rather facilitated a discussion around puberty. • Change some of the wording around the documents and poster boards to make them more gender inclusive. Stress gender inclusivity in this lesson. • Address vernacular and slang language at beginning. • Fact or fiction as an ice breaker. 2.23 Into to • Competition really helps motivate, keeps youth engaged, and helps Sexuality: them learn this topic. Myths and • Various ways of doing this lesson: in pairs, in a large group, teams. Facts • Ice breaker- movie / music about sexuality. • Have youth write out what they heard about sexuality as an intro to this lesson. • Have question box out for this lesson. Give out note cards for questions. • Review definition of sexuality and use knowledge of what teens already know about sexuality. • Change some of the statements and add some statements that needed to be revisited from previous lessons. • Set the myth/fact portion up as a game/competition to keep the entire group engaged for whole session. • Be flexible with the lesson plan and allow teens to ask their own questions. • Be prepared for lots of questions. • Reword statements to be more relevant- pick out a few and spend time discussing them rather than doing all of them. 44 2011-2015 Implementation Planning Document Lesson Comments, suggestions, changes provided by session observers from #/name Colorado Youth Matter, trainers for delivering the TOP® curriculum 3.20 When Sex • Ice breaker: hot potato with decisions. Enters the • Write out steps like a recipe. Equation & • Update/ add to ‘bringing it full circle’ questions. 1.29 • Ways people make decisions matching activity. • Impulse decision making is hard to explain. • Use group scenarios as examples to discuss. • Both groups and individual work well for this lesson. • Great Handout for Session Wrap-up: 100 Ways to Show Love (without having sex) and let the teens read out their favorite ones from the list. • Rating activity- may cause a particular youth to feel shamed or judged. Might be done better as an individual worksheet activity. • Added in a “While you wait” activity-asked youth to write down what are the ways that they can show a partner that they care. • The teens were very engaged. • This lesson was very biased towards abstinence and not inclusive or values neutral. • Took out rating system and had discussions about values and decision making. • Developmental stages and ages may have big influence in the success of this lesson. 45 2011-2015 Implementation Planning Document Lesson Comments, suggestions, changes provided by session observers from #/name Colorado Youth Matter, trainers for delivering the TOP® curriculum 4.21 Using • Depending on group size and age variations, this lesson works in Condoms various ways: breaking into smaller groups to do the activities or pairs Correctly or one large group. • Condom lineup is a great intro. • Condom relay race is a great way to end the lesson. Can give prizes to the winning team. • Need to clarify step “relax.” • Be prepared for lots of questions, especially ones that may be off topic. • Have the questions box out. • Have a clear plan of how to run the condom demos: size of group, how you will split groups and how you will monitor and supervise the groups. • Go in depth about how to put on and use female condoms and dental dams. • Be more inclusive by sharing how condoms can be used on sex toys and dildos, so that they recognize STIs can still be transmitted that way as well. • Change line up activity by adding steps- “receive and give consent’ “orgasm/ejaculation (if it occurs)” • Use a video called “More on Condoms” by the Sexplanations site on YouTube. • Pass out a document called “The Big Five” detailing the 5 main types of condoms. • Pass out wallet size instructions on how to put on a condom, dental dam, and in condom. • Offer options for participation as this may be uncomfortable or embarrassing for some teens. (Group, partner demo, for example) as well as the freedom to step out if they need to. • Ask about latex allergies before the lesson begins. • Revisit ROPES beforehand. • This lesson does not fill the full hour. • Plenty of time for practice, questions and condom races at the end. • Ages and stages of development may impact success of this lesson. 46 2011-2015 Implementation Planning Document Lesson Comments, suggestions, changes provided by session observers from #/name Colorado Youth Matter, trainers for delivering the TOP® curriculum 4.20 Basics of • Vagina model needed. Contraception • Going in order of contraceptive effectiveness is a good model for this session. • Ice breaker before to engage teens as this is content heavy lesson. • Have good food after or add a fun wrap up activity. • Notice the group; a break halfway through might be helpful. • A handout or worksheet with all the information to cover might be helpful for engaging the visual learners and managing all the content. • Difference between contraception and barrier methods in transmission of STI’s. • Include universal condoms in the conversation- discuss their vaginal and anal usage. • Bring anatomy charts for this lesson as it is helpful when describing how to properly use certain forms of contraception. • Include discussion of how all partners need to be involved in the process, and the ways that they can be. • The attention and engagement was very high. • Time is a big challenge in this lesson. There is so much to cover and so many questions. Maybe two sessions? 4.18 When • Need a lead in, engaging ice breaker- Simon says, I will/I won’t from Relationships Draw the Line. Lead to • This lesson is shorter so can plan to use the extra time for CSL or Pressure something else. • Gallery style works well. • Discussions questions as a group rather than free write. • Add an activity, dig deeper, add some application. • Role plays or scenarios would help the application of this lesson as it is conceptual. • Review Relationship Spectrum; characteristics of a healthy, unhealthy, and abusive relationship. • Starting with the free write helps this lesson to flow. • Loveisrepect.org has a ton of great resources for this lesson- Relationship Spectrum. • Incorporate skits into this lesson to model the behavior addressed as well as discussing it. • Sequence should be closer to other lessons about relationships. 47 2011-2015 Implementation Planning Document Lesson Comments, suggestions, changes provided by session observers from #/name Colorado Youth Matter, trainers for delivering the TOP® curriculum 4.19 Date • True /false quiz- the statements were restructured to be values neutral Rape but the answer were not. They need to be redone. • Activities to look at all sides and bring in application discussions about FLASH – personal boundaries. Sexual • Handout: “The Relationship Spectrum” from loveisrepect.org . Violence • Resources from Colorado Coalition Against Sexual Assault. Prevention • Connect more to current events and cases they can relate to. • Show a 5 minute video about teen dating violence from Be Smart Be Well. • The Draw the Line Activity served as a great ice breaker and wrap up activity. • Focus on consent and bystander intervention since many teens have not discussed it before • There is so much information to go through- the teens had a lot of questions. • Turn some of the activities into ice breakers in order to conserve time for discussions. • There are lots of misconceptions and misinformation about this topic. • Too much to cover in an hour. Very information heavy. • Add application to make it relevant. 2.24 STD • Good to have the flexibility between jeopardy and basketball as they Jeopardy work for different groups. • Having a real jeopardy game and structuring questions like a real game is more engaging. • It’s important they leave feeling successful and that they know the right answers. • Find ways that engage both teams at the same time. • Have questions box out for this lesson. • Make sure this is a very interactive and high energy game-add some silly rules to make it more fun and group oriented like “Phone a friend” and “crowd poll.” • The teens respond very well to competition so lessons like these are very successful. • Let teens to pick their own teams • Offer bonus points for in depth answers. • Additional questions can be added to the game. • Offer prizes for additional incentives. 48 2011-2015 Implementation Planning Document Lesson Comments, suggestions, changes provided by session observers from #/name Colorado Youth Matter, trainers for delivering the TOP® curriculum 4.22 Teaching • Challenging lesson for small group; Bingo style is an alternative for Peers and small groups. Condom • Besider.org is a good resource to prepare for presentations. Review • Provide the youth with resources. • Give and set a short amount of time for the presentations. • Partner people that are very talkative and quiet, so there is a mix in the group. • Give each person a specific role to play. • Presentations served as great way to review of material. • Gives teens a chance to creatively teach to their peers in way that they felt most comfortable and effective. Creative presentations are always a fun way to bring out each individual style and personality in the group. • This lesson is challenging if teens have missed the first lesson on contraception. Pair them with teens that were present for both. • Good data and handouts from this website helps teen prepare. (http://www.reproductiveaccess.org/) • Sequence- this lesson should follow the contraception lesson 2.35 Someone • Gallery walk. Who & 1.40 • Various ways to do this activity and make it more engaging for Compliment different types of groups. Circle • Having complements they can use instead of creating their own. • TOP® Reflection Question: How have you changed or grown in the last 9 months? • Really important to have closure. • Compliments on papers plates that are passed around, possibly read aloud and something they can take with them to remember TOP® experience. • What we learned in TOP® tree. • Satisfaction surveys to learn more about what they liked/didn’t like, etc. about TOP®. • Ending in a symbolic way is important. 49 Appendix F: Sensitivity analyses This appendix shows findings from sensitivity analyses including: (1) an unadjusted model of intervention effects on outcomes without covariates, (2) multiple imputation to create a complete dataset for all covariates/baseline values to address missing data on relevant covariates (follow-up values based on available sample size), (3) examination of covariates (i.e., baseline characteristics with observed differences between those retained and those lost to attrition, including age, sexual acts protected by contraception, condom use at last intercourse, lower likelihood of using substances during intercourse, cutting class and being suspended) compared to the benchmark approach, and (4) a repeated measures analysis with a treatment- by-time interaction. For these analyses, study conclusions did not change and no further analyses are reported. 50 Table F.1. Sensitivity of impact analyses using data from assessment at TOP® Program completion to address the primary research questions Additional Intervention Multiple Multiple Additional Covariates compared Benchmark Benchmark No No Imputation Imputation Covariates Based on Pre/Post with approach approach Covariates Covariates (covariates) (covariates) Based on Retention Pre/Post Analyses comparison difference p-value difference p-value difference p-value Retention p-value Analyses p-value Proportion of sex acts protected by condoms 0.01 .63 .02 .55 .01 .68 .005 .95 .03 .23 Proportion of sex acts protected by contraception 0.01 .46 .02 .30 .01 .45 .07 .25 .01 .37 Ever pregnant/cause pregnancy 0.69 .55 0.45 .78 0.68 .55 .87 .88 1.67 .13 Source: TOP® and Youth All Engaged, 2012-2015 Follow-up surveys administered at program completion. Notes: All analyses conducted in multilevel modeling framework to account for participants nested within clubs/years. The first sensitivity approach presents mean differences/p-values between intervention and control groups without any covariates in the model. The second approach used multiple imputation to create a complete dataset for all covariates/baseline values (follow-up values were still based on available sample size). The third approach examined additional covariates to the benchmark approach based on differences observed by attrition in six variables. The fourth approach examines changes from pretest to posttest, rather than the traditional ANCOVA approach where pretest scores are included as a covariate; the reported p-value is for the time by treatment interaction. See Table III.3 for a more detailed description of each measure and Section III for a description of the impact estimation methods. 51