POSITIVE PREVENTION PLUS Findings from an 1 Innovative Teen Evaluation of the Positive Prevention PLUS Teen Pregnancy Prevention Program Pregnancy Evaluation of the Prevention Program Positive Prevention PLUS Teen Pregnancy REVISED 2/22/16 Prevention Program Final Impact Report for San Bernardino County Superintendent of Schools Prepared by Robert G. LaChausse, PhD California State University, San Bernardino This publication was made possible by Grant Number TP2AH000007 from the Office of Adolescent Health, U.S. Department of Health & Human Services. The views expressed in this report are those of the author and do not necessarily represent the policies of HHS or the Office of Adolescent Health. POSITIVE PREVENTION PLUS 2 Acknowledgements This study was a collaborative effort between the San Bernardino County Superintendent of Schools (SBCSS) and California State University, San Bernardino (CSUSB). Dr. Kim R. Clark served as the Project Director. The author is grateful to Jessica Folmer, Research Associate, for her assistance in data collection, data entry, and assessing implementation fidelity, Carolyn Tillman at SBCSS for her assistance in project implementation, and to the students, teachers, and district staff who participated in the study. Finally, the author would like to thank Jaqueline Berman, PhD and Russell Cole, PhD at Mathematica Policy Research for their guidance, on-going support, technical assistance, and collegiality throughout the study. Recommended Citation: LaChausse, R. (2015). Evaluation of the Positive Prevention PLUS Teen Pregnancy Prevention Program. USDHHS, Office of Adolescent Health. TP2AH000007. POSITIVE PREVENTION PLUS 3 Table of Contents I. Introduction ............................................................................................................................. 7 A. Introduction & Study Overview ....................................................................................... 7 B. Primary Research Questions ............................................................................................ 9 II. Program and Comparison Programming ................................................................................. 9 A. Description of Program as Intended ................................................................................. 9 B. Description of Counterfactual Condition ....................................................................... 12 III. Study Design ...................................................................................................................... 13 A. Sample Recruitment ....................................................................................................... 13 B. Study Design .................................................................................................................. 14 C. Data Collection............................................................................................................... 14 1. Impact Evaluation .......................................................................................................... 14 2. Implementation Evaluation ............................................................................................ 15 D. Outcomes for Impact Analyses ...................................................................................... 17 E. Study Sample.................................................................................................................. 18 F. Baseline Equivalence ..................................................................................................... 18 G. Methods .......................................................................................................................... 19 1. Impact Evaluation .......................................................................................................... 19 2. Implementation Evaluation ............................................................................................ 20 IV. Study Findings ................................................................................................................... 22 A. Implementation Study Findings ..................................................................................... 22 B. Impact Study Findings ................................................................................................... 24 V. Conclusion ......................................................................................................................... 26 POSITIVE PREVENTION PLUS 4 VI. References .......................................................................................................................... 28 Appendix A. Program Description and Logic Model ................................................................... 29 Appendix B: Data Collection Periods ........................................................................................... 31 Appendix C: Primary Research Questions ................................................................................... 32 Appendix D: Intervention Status ................................................................................................. 33 Appendix E: Implementation Fidelity Methods ........................................................................... 34 Appendix F: Implementation Fidelity Data .................................................................................. 36 Appendix G. Baseline Equivalence and Program Impacts ........................................................... 41 Appendix H: Sensitivity Analyses ................................................................................................ 44 POSITIVE PREVENTION PLUS 5 Tables Table A.1: Application of Social Cognitive Theory (SCT) Constructs in the Positive Prevention PLUS Program ............................................................................................................ 29 Table B.1. Data Collection Efforts Used in the Impact Analysis of Positive Prevention PLUS and Timing. ........................................................................................................................ 31 Table C.1: Behavioral Outcomes Used for Primary Impact Analyses Research Questions ......... 32 Table D.1: Sample Size by Intervention Status ............................................................................ 33 Table E.1: Methods Used to Address Implementation Research Questions ................................ 34 Table F.1: Findings from the Implementation Fidelity Data ........................................................ 36 Table F.1a: Details of program attendance ................................................................................... 36 Table F.2: Characteristics of program staff and interactions ........................................................ 37 Table F.2a: Demographic characateristics of teachers in treatment condition ............................. 38 Table F.3: Experiences of the treatment and comparison conditions ........................................... 39 Table F.3a: Health topic coverage, as described by students ....................................................... 39 Table F.3b: Health topic coverage, ad described by teachers ....................................................... 40 Table F.4: Implementation context ............................................................................................... 40 Table G.1: Summary Statistics of Key Baseline Measures for the Analytical Sample ................ 41 Table G.2 Post-intervention Estimated Effects using Data from PP+ Student Survey to Address the Primary Research Questions ..................................................................................... 42 Table G.3 Benjamini- Hochberg Correction for Multiple Comparisons ...................................... 43 Table H.1: Logical Imputation Rules............................................................................................ 44 Table H.2: Sensitivity of Impact Analyses using Data from PP+ Student Survey to Address the Primary Research Questions ................................................................................................... 45 POSITIVE PREVENTION PLUS 6 Figures Figure A.1: Program Logic Model................................................................................................ 30 POSITIVE PREVENTION PLUS 7 I. Introduction A. Introduction & Study Overview The teen birth rate in the U.S. continues to decline and has dropped below 26.5 births for every 1,000 adolescent females ages 15-19. 1 While this reflects overall progress at achieving lower rates of teen pregnancy, progress in uneven. For example, Hispanic teens continue to have the highest birth rate at 42.7 per 1,000. 2 National data reveals that over 48 percent of all students in grades 9-12 have had sexual intercourse by the time they graduate and only 41 percent of teens had used a condom the last time they had sexual intercourse with Hispanic teens reporting slightly lower rates of condom use. 3 Since a significant number of adolescents engage in sexual risk behaviors, the need for effective teen pregnancy programs cannot be overstated. Although many teen pregnancy prevention programs can increase student knowledge about the consequences of becoming pregnant, only a few have demonstrated an impact on students’ behaviors. In 2009, the U.S. Department of Health and Human Services (HHS) sponsored a systematic review examining the effectiveness of programs to reduce teen pregnancies and associated sexual risk behavior among teens. Of the approximately 2,000 potentially relevant studies published between 1989 and January 2011, 200 met screening criteria. Of these, 88 studies received a high or moderate rating and only 31 provided credible evidence demonstrating a statistically significant positive program impact on at least one sexual behavior or reproductive health outcome of interest (for example, sexual activity, contraceptive use, or pregnancy). 4 The review concluded that there is a need for improved research quality and reporting to inform policy initiatives and programming decisions. As a result, the Office of Adolescent Health (OAH) under HHS introduced a grant program to support the replication of evidence-based programs and to study the impact of new program POSITIVE PREVENTION PLUS 8 models and strategies. With funding from OAH, the Positive Prevention PLUS, an 11-lesson school- based teen pregnancy prevention (TPP) program, was developed based on the existing literature surrounding school-based prevention programs that has supported programs that use experiential, interactive activities to emphasize abstinence and risk reduction techniques. 5 Positive Prevention PLUS applied the findings from an earlier study of the Positive Prevention HIV/STD Curriculum for Students Grades 9-12 that employed clustered randomized controlled trial and met the criteria for a high rating under the HSS TPP evidence review. Findings from the study showed a positive, statistically significant impact on sexual initiation at the 6 month follow-up. 6 However, the HHS TPP evidence review concluded that the impact estimates could have been biased because the analyses controlled for post-implementation variables that could have been affected by the program, so there was no evidence of effectiveness. 7 Positive Prevention PLUS utilizes Social Cognitive Theory (SCT) as its theoretical basis. SCT posits that the likelihood of taking preventive action is determined by an understanding of what must be done to avoid pregnancy, a belief that one is able to use this method, and the belief that this method will successfully decrease the chance of getting pregnant. 8 Past research has suggested that the use of social learning theory in pregnancy and STD prevention, particularly with adolescent youth, is far superior to other theoretical approaches. 9 This report provides the results of an external evaluation of the Positive Prevention PLUS teen pregnancy prevention program for 9th grade students in Southern California. A clustered randomized controlled trial (CRCT) was employed in which participating high school sites were randomly assigned either to a treatment group that implemented the Positive Prevention PLUS program or a control group. Students completed a self-administered survey at baseline (prior to program implementation) and a 6 month follow-up survey (post program implementation). The POSITIVE PREVENTION PLUS 9 purpose of this study is to provide scientific evidence concerning the effectiveness of a school- based, theory-driven teen pregnancy prevention program. B. Primary Research Questions 1. What is the impact of Positive Prevention PLUS relative to a control group on the initiation of sexual activity 6 months after the end of the intervention? 2. What is the impact of Positive Prevention PLUS relative to a control group on ever been pregnant 6 months after the end of the intervention? 3. What is the impact of Positive Prevention PLUS relative to a control group on having sexual intercourse without using birth control in the prior three months 6 months after the end of the intervention? It was hypothesized that students receiving Positive Prevention PLUS would be less likely to initiate sexual intercourse six months after the end of the intervention than those in the control group. Further, it was expected that the program would decrease the likelihood that students would become pregnant (or get someone pregnant), and decrease the likelihood that adolescents would not use birth control. II. Program and Comparison Programming A. Description of Program as Intended Positive Prevention PLUS was developed after a review of the literature of effective sexuality education programs that found that programs based on SCT can increase an adolescent’s ability to use risk –reduction skills (e.g. assertive communication), and either abstain from sexual intercourse or use birth control when engaging in sexual activity. SCT posits that behavior change occurs through several constructs including observational learning, behavioral capability, and self-efficacy.6, 9 SCT is used throughout the Positive Prevention POSITIVE PREVENTION PLUS 10 PLUS lessons. For example, lessons 1 and 2 explore reasons for teens to be sexually abstinent and make responsible decisions (observational learning), lesson 7 teaches students how to recognize and avoid risky situations and use condoms (behavioral capability), and lesson 8 uses interactive role-plays to increase confidence in using refusal skills in everyday life (self- efficacy). For a program description and logic model see Appendix A, Figure A.1. Positive Prevention PLUS consists of eleven 45-minute lessons aimed at students in grade 9 delivered during the normal school day in science, health, or physical education courses. The program was implemented in public schools in Southern California. The lessons were taught by trained classroom teachers to students in group form during a regular class period. Teachers were given a 3 week period in the fall of 2013 to complete the 11 lessons in consecutive order. The curriculum lessons are as follows: • Getting Started: Discussion on group agreements, the sexual health of teens, overview of the sexual health concerns of young men and women, and a review of ways to maintain reproductive health. • Lesson 1- Life Planning: Review of the importance of personal goals and life plans. • Lesson 2- Healthy Relationships: A discussion on intimacy and the characteristics of a quality relationship. • Lesson 3- Relationship Violence: Signs of an abusive relationship, showing the relationship violence video, and the setting boundaries. • Lesson 4- Family Planning and Contraceptives: Review of contraception, displaying and describing FDA-approved contraceptives, including their effectiveness, the California Safe Surrender law, and the benefits of abstinence. POSITIVE PREVENTION PLUS 11 • Lesson 5- Myths and Stereotypes: Video of persons infected by HIV and an activity on how HIV affects various people without regard to sexual orientation or family structure. • Lesson 6- HIV and AIDS: A review of the HIV/AIDS epidemic, how a healthy immune system functions, and information on managing risk situations. • Lesson 7- Recognizing and Reducing Risk: Steps in condom use, a condom demonstration, condom success and failure rates, and practicing negotiating condom use. • Lesson 8- Peer and Media Pressures: Peer and media pressures, media analysis skills, and communication (assertiveness) and refusal skills practice. • Lesson 9- HIV/STD Testing and Community Resources: An activity which focuses on demonstrating STD transmission and community resources. • Lesson 10- Steps to Success: Review of personal goals and life plan and making a commitment to either abstaining from sexual intercourse or using birth control. Teachers in participating public schools implemented the program. Each teacher must have at least a bachelor’s degree and a single subject teaching credential in life science, physical science, physical education, or health education. Teachers were provided training about one month prior to program implementation from the program developer and his staff. Teachers participated in an initial two-day training consisting of an overview of teen pregnancy in the U.S. and local area, characteristics of effective teen pregnancy prevention programs (theory, pedagogy, fidelity, etc.), the purpose of the PP+, the program’s logic model, and the need to implement the program with fidelity including the specified sequence. Program trainers demonstrated each of the eleven lessons. On day two, teachers were asked to teach back one of POSITIVE PREVENTION PLUS 12 the eleven lessons (randomly selected) and received feedback from the trainers. After the initial training, teachers were asked to complete an online training program consisting of various modules to review the key components of lessons and to observe each lesson being taught by a veteran teacher. Approximately three weeks after the initial training (1week before implementation), teachers were brought back as a group for a one day refresher to emphasize key lessons, lesson activities, program quality and adherence, as well as to address any concerns teachers may have regarding program implementation. During program implementation, the project director periodically communicated with teachers to provide encouragement and technical assistance. B. Description of Counterfactual Condition Students in control group classrooms received the standard health, science or physical education curriculum. Control groups schools and teachers were asked to refrain from providing any sexuality- related classroom instruction or school wide teen pregnancy or STD prevention focused activities, however control group teachers were allowed to discuss human reproduction if relevant to their curriculum (for example, in a biology course). POSITIVE PREVENTION PLUS 13 III. Study Design A. Sample Recruitment The Project Director approached 6 school districts to assess each district’s ability to participate in the study and to obtain a memorandum of understanding (MOU). School districts were approached based on previous data that indicated that there were a high number of teen births and/or STD rates and the lack of an existing evidence-based comprehensive sexuality education or teen pregnancy prevention program in the school district. For each school district, the feasibility of participating in a randomized controlled trial was assessed including the district’s ability to implement a teen pregnancy prevention program like PP+, understand and maintain the treatment/control contrast, provide access to students for data collection, and ability to provide attendance data. Once MOUs were finalized, teacher training was provided and parent/guardian consent was requested. All six school districts agreed to participate in the study. Twenty-two high schools in Southern California were approached to participate in this study. Eligibility to participate included interest in the program, a signed MOU, and having required science, health, or physical education courses for 9th grade students. Ultimately, twenty- two public high schools (with thirty-six teachers) across six school districts within Southern California met initial eligibility requirements for participation. All male and female 9th grade students in mandatory 9th grade health, science, or physical education (PE) classes at each school site (n= 22) were eligible for inclusion in the study. Prior to randomization, parental consent forms were distributed to teachers by the Principal Investigator (PI). Teachers distributed the consent forms to students. The consent form described the general purpose and nature of the study, issues of confidentiality of responses, and contact information for the PI. Additionally, the consent form included a statement that the survey questions are general in nature, including POSITIVE PREVENTION PLUS 14 questions on sexual behavior, but this does not imply that their child is sexually active. The consent form was provided in both English and Spanish. A small incentive (two movie tickets) was offered to the teacher with the highest percentage of parent consent forms returned at each school site regardless of whether parents agreed to have their child participate. The study was approved by the Institutional Review Board (IRB) at California State University, San Bernardino. B. Study Design A clustered randomized control trial (CRCT) was employed to determine program impacts. No stratification was employed. After parent/guardian consent was obtained, twenty one eligible school sites were randomly assigned into either the treatment or control condition using the select cases (RANDOM) function in SPSS 22. 1 School sites, students and their parent/guardian were not aware of their respective condition prior to consent. Treatment sites (n = 11) agreed to implement the Positive Prevention PLUS program in their 9th grade health (n = 7), PE (n = 1), or science (n = 4) classes. Ten control sites agreed to not provide any pregnancy prevention or STD prevention education within the study period. C. Data Collection 1. Impact Evaluation Baseline data was collected for both the treatment and control group in October 2013. Six month follow-up data collection occurred in May 2014 (See Table B.1). Data collection occurred for each school site within each district on the same day. Participants completed a self- administered, paper and pencil survey during their regular class period. All survey data 1 Twenty two schools were part of the sample for random assignment, but one school was determined not to have any eligible 9th grade youth in health classes, and thus, was excluded from the study as ineligible. POSITIVE PREVENTION PLUS 15 collection was conducted by the Principal Investigator (PI). Project staff, teachers, site administrators, and other school staff were not allowed to provide any instructions, provide guidance, or answer student questions during data collection. Baseline data collection occurred approximately 8 weeks after parental consent forms were distributed and 1 week prior to the beginning of the intervention. For the 6 month follow-up, participating students were pulled into a central location (e.g. library) by their original class period at baseline. Data was collected on the same date for all study schools (both treatment and control) in each district. Each baseline survey was pre-printed with a random ID number to track individuals’ survey responses across data collection periods. The same ID number was used for each follow-up survey to student responses over time. Data collectors were trained to respond to student questions with “Try your best to answer the question or leave it blank.” Data collectors were not allowed to read aloud any survey items or response formats, or provide any other guidance. The data collection procedure was the same for both the treatment and control groups. The student survey included a brief demographics section (gender, age, ethnicity) and outcome measures: 1) whether or not participants have ever had sexual intercourse, 2) whether or not participants have been pregnant (or gotten someone pregnant), and 3) whether or not participants had sex without using birth control in the prior 3 months (See Table C.1). The average time to complete the survey was 14 minutes and the reading level was 5.7 (Flesch- Kincaid grade level equivalent). Participants received one raffle entry for a chance to win two (2) movie tickets each time she/he completed a survey. 2. Implementation Evaluation Adherence Adherence to the program model was measured using the teacher curriculum log. This log was developed by the program evaluator based on other fidelity logs used in previous impact POSITIVE PREVENTION PLUS 16 evaluation studies. 10 Program teachers completed this log after each class period and logs were collected by the PI after program implementation was completed. Intervention group teachers were asked to self-report the extent to which each topic, worksheet, or activity was covered completely, partially or not at all. Teachers were asked if any adaptations were made to any of the lesson activities. Program reach was measured using school attendance data. This data was collected using each school district’s student information system (SIS). The evaluator requested and received this data from each school district for the program implementation dates. The study considers a student “attended” if the teacher marked the student as “attended” or “tardy” for that class period per the district’s SIS. If the student had an absence for that class period (for any reason), the student was marked as “not attended” for the study. Quality of Implementation The PI or a research assistant conducted a direct observation of each teacher using the classroom lesson observation form to measure lesson quality and student engagement. This observation assessed, the overall quality of the delivery of lesson activities including the teacher’s explanation of lesson activities, teacher’s ability to pace the lesson (i.e. keep on task), knowledge of the lesson, poise and confidence, level of enthusiasm, and student participation. One lesson observation was conducted during one class period for each teacher during the implementation period. The observation was conducted at random so the teacher would not know the day, period, or lesson the observation would be conducted. Each observer was trained in the curriculum during the teacher trainings and received additional training in conducting observational research. POSITIVE PREVENTION PLUS 17 Experiences of the counterfactual Data regarding the difference between treatment and control experiences was collected on the student survey for both the treatment and control group participants at 6 month follow-up using the item “Which of the following health topics have you learned about this school year? - pregnancy prevention, AIDS or HIV infection, human sexuality, taking care of a baby, using condoms, or abstinence.” Respondents were asked to select each of the topics that they learned about (See Table E.1). Context Information regarding the context of program implementation was gained from surveys from teachers. Teachers in both the treatment and control group were asked “Which of the following health topics have you taught about in the last 6 months?- pregnancy prevention, AIDS or HIV infection, human sexuality, taking care of a baby, using condoms, or abstinence.” on the teacher survey. Additionally, teachers were asked an open-ended question about whether there were any issues that occurred at the school site that may have affected program implementation. D. Outcomes for Impact Analyses The student survey assessed three outcome measures: 1) ever had sexual intercourse, 2) ever been pregnant (or gotten someone pregnant), and 3) ever had sexual intercourse without using birth control. Skip patterns are used so that if a participant reported that they had never had sex, she/he would skip the items that pertain to sexually active respondents and complete the remaining items in the survey. Sexual initiation was constructed from the survey question “Have you ever had sexual intercourse?” A dummy variable was created in which respondents who respond yes were coded as 1 and no were coded as 0. Missing data was coded as 999. The item “To the best of your knowledge, have you been pregnant or gotten somebody pregnant?” The POSITIVE PREVENTION PLUS 18 variable is constructed as a dummy variable with values where respondents who respond yes have been pregnant were coded as 1 and those who responded no are coded as 0. Missing data resulting from the skip pattern of the survey was logically imputed to 0 since one can infer that someone who had never had sex has never been pregnant. The variable “In the past 3 months, have you had sexual intercourse without you or your partner using any of these methods of birth control?”- Condoms, birth control pills, the patch, the ring (NuvaRing), IUD, Implant (Impanon). The variable was constructed as a dummy variable where respondents who respond yes have had sex in last 3 months without birth control are coded as 1 and those who responded no are coded as 0. Missing data due to the skip pattern was coded as 0 since one can infer that someone who had never had sex had also never had sex without birth control (See Table C.1). E. Study Sample Treatment sites (n = 11) agreed to implement the Positive Prevention PLUS program in their 9th grade health (n = 7), P.E. (n = 1), or science (n = 3) classes. Control sites (n = 10) agreed to not provide any teen pregnancy prevention education within the investigation period. A total of 7,042 students were eligible to participate in the study. Of the 4,969 students who returned the parent consent form, 4,267 had positive parental consent to participate. Of those students, 3,554 students participated in the baseline survey and 3,490 students participated in the 6th month follow-up survey. There were 2,113 participants in the intervention group and 1,377 in the control group that made up the final analytical sample (i.e. provided both baseline and follow-up data). See Table D.1. F. Baseline Equivalence The analytical sample is all students in the school sites that have parental consent and provide data at baseline and 6 month follow-up. Data is pooled across the school sites. Baseline POSITIVE PREVENTION PLUS 19 equivalence for the analytical sample is provided in Table G.1. This table displays the equivalence between the treatment and control groups on the following measures: age (in years), percent female, race, and percent Hispanic. Baseline equivalence was also examined on pretest data of the outcome measures including ever had sexual intercourse, ever been pregnant, and having had sex without using birth control in the prior 3 months. Equivalence was calculated using a linear regression model predicting the variable of interest from a grouping variable (dummy coded) for each measure adjusting for the clustering effect in the CRCT using Huber- White adjusted standard errors. Once adjusting for the clustered nature of the data, no statistically significant differences at baseline were observed. G. Methods 1. Impact Evaluation Analyses were conducted on student-level data and used an intent-to-treat framework. To answer the primary research questions and estimate possible program impacts, hierarchical linear regression models using a random effects approach were used. A two-level model was expressed in a multilevel framework using a random effects ANOVA to determine what portion of the variance in the outcome variable of interest is due to site level differences (i.e. school sites) as compared to individual differences. This approach accounted for clustering using maximum likelihood to estimate parameters that specified the structure of the covariance between individuals in clusters. 11 The model was written as: β 1_1ij β 0 j + eij = β 0 j y00 + δ 0 j = where β 1_1 is the outcome of interest (e.g. ever had sex at follow-up) for an individual in a site, β 0 j was the mean score for a site and y00 was the grand mean (i.e. the mean across all POSITIVE PREVENTION PLUS 20 individuals and sites). The level 1 error term indicates how an individual’s score deviates from the mean in the site in which the individual resides. The level 2 error term indicated how the mean score in a particular site deviates from the grand mean. This was calculated using the XTMIXED function in Stata 8.0 by first estimating the percent variance due to differences across sites by examining the interclass correlation (ICC) and the amount of variance attributable to individual differences. Using the XTREG function in Stata 8.0, a random effects model was calculated using the Group variable to determine the amount of variance that is explained by the Group variable (treatment vs. control). Finally, the XTREG command was used to predict the outcome variable of interest from Group with the baseline score, gender, age, and ethnicity (Hispanic vs. non-Hispanic) used as covariates. Findings are considered statistically significant if the p-value is less than 0.05, using a two-sided test. Because of the number of a priori hypotheses (3), the Benjamini-Hochberg adjustment was made to control for the false discovery rate. 12 2. Implementation Evaluation There were four sources of data used to assess adherence, quality of implementation, counterfactual experiences, and context: teacher curriculum logs, classroom lesson observation forms, teacher surveys, and participant attendance data. This data is limited as it relies on the self-report of teachers. Adherence Adherence to the program components was calculated by summing the number of completed lesson activities divided by the number of activities assigned. Lesson completion is defined as completing all lesson components (e.g. activities). The number of scheduled activities was calculated by multiplying the number of activities within each lesson by the number of class sections. The number of completed activities were identified from the self-reported teacher POSITIVE PREVENTION PLUS 21 curriculum logs and summed for each lesson. An adaptation is defined as any changes to the lesson activity or pedagogical strategy. The percent of activities completed was calculated by dividing the number of completed activities by the number of scheduled activities. The number of lessons taught with adaptations were identified from the self-reported teacher curriculum logs and summed for each lesson. The percent of activities completed with an adaptations was calculated by dividing the number of lessons taught with at least one adaptation to the total number of class sections. Attendance data was also collected using each school district’s student information system (SIS). The study considers a student “attended” if the teacher marked the student as “attended” or “tardy” for that class period per the district’s SIS. If the student had an absence for that class period (for any reason), the student was marked as “not attended” for the purposes of this study. Quality Lesson quality was calculated by dividing the number of high quality lesson observations by the number of total observations. An average score of four or greater on a scale ranging from 1= poor to 5=excellent across individual session elements was considered a high quality observation. The quality of student participation was captured through classroom observations (e.g., “How exactly did the group members participate in the discussions and activities?” 1= Little participation, 3= Some participation, 5= Active participation). High student engagement was measured by summing the participation observation scores of 4 or better and dividing it by the total number of observations. Experiences of the Counterfactual To measure the counterfactual, students were asked which TPP health related topics they had learned about in the past 6 months including pregnancy prevention, AIDS or HIV infection, POSITIVE PREVENTION PLUS 22 taking care of a baby, using condoms, human sexuality, and abstinence. This was measured on the 6 month follow-up survey. Likewise, a teacher survey was conducted at the 6 month follow- up assessing the extent to which teachers had taught various health related topics outside of the PP+ program in the past 6 months and the presence of any school-wide teen pregnancy, STD, or HIV/AIDS prevention activities. The percent of TPP health topics received by students in each group and from the teacher’s self-report was calculated by summing the responses from each item (1 = yes, 0 = no) and dividing them by the total number of possible responses. Table E.1 outlines the methods used to collect information about implementation fidelity. Context Teachers were asked if their school site had conducted any school-wide activities like assemblies, club events, or guest speakers regarding pregnancy prevention, sexuality, HIV/AIDS, or reproductive health in the past 6 months and asked to explain the type and duration of that activity. The percent of TPP health topics taught by teachers in each group (not in the PP+ program) was calculated by summing the responses from each teacher (1 = yes, 0 = no) and dividing them by the total number of possible responses. This information was captured through the curriculum fidelity log and informal conversations with treatment group teachers. IV. Study Findings A. Implementation Study Findings Adherence Eleven school sites within six school districts were assigned to the treatment group. Thirty-four teachers with a total of 123 class sections (periods) implemented the program. Teachers had 18 days between the dates of October 28th, 2013 to November 14th, 2013 to implement the 11 lessons in consecutive order. Ninety-five percent of the scheduled 1,353 POSITIVE PREVENTION PLUS 23 lessons were completed. A total of 6,124 lesson activities were completed of the 6,396 lesson activities scheduled to be delivered (95%). Some lessons were completed with adaptions. For example, 36% of the completed lesson 4 (Family Planning and Contraceptives) had adaptions. Table F.1 shows the findings from the implementation fidelity data. The number of students who participated in the program was 2,139. Overall, average program attendance across all sessions was 91 percent. The average number of lessons attended by each student was 10.01 of 11 lessons (SD = 1.64). Lesson Quality Twenty-six classroom lesson observations were conducted. An average score of four or greater was considered a high quality observation. Seventy-three percent (73%) of the observed lessons received a high quality observation. Most teachers received high marks for her/his explanation of lesson activities while the lowest scores were associated with teacher’s ability to pace the lesson (i.e. keep on task) or low levels of student participation in the lesson. Only 65% of the observed lessons were rated with high student participation. Observers noted that in most cases, teachers’ inability to correctly pace the lesson led to adaptations in a way that made the lesson activity less interactive. For example, a teacher would model refusal skills in front of the class rather than give students an opportunity to practice refusal skills. Counterfactual experiences The treatment group students reported learning 60 percent of the TPP health topics, whereas the control group reported learning approximately 17 percent of the TPP health topics. Additionally, teachers in the treatment group and control group were surveyed at the 6 month follow-up and teachers in the treatment group reported teaching 58 percent of the TPP health topics, whereas control teachers taught 10 percent of the TPP health topics. None of the teachers POSITIVE PREVENTION PLUS 24 in either group reported any additional school-wide activities related to pregnancy prevention, sexuality, HIV/AIDS or reproductive health. Context Six treatment teachers within one school were not allowed to perform condom demonstrations per their district’s school board policy. Teachers could explain but not demonstrate the steps in condom use. No other issues were documented. B. Impact Study Findings The results of the impact of the Positive Prevention PLUS program on participants' likelihood to engage in sexual intercourse, become pregnant, or have sex without birth control are presented in Table G.2. There was a significant effect of the Positive Prevention PLUS program on delaying sexual activity. Relative to the control group, participants in the treatment group were approximately 4 percentage points less likely to have had sex at 6 month follow-up (b = -.04, t = -2.38, p = .01). There was no impact of the Positive Prevention PLUS program on getting pregnant at 6 month follow-up (b = -.01, t = -1.87, p = .07). There was a significant effect of the program on ever having sex without birth control in the prior 3 months measured at the 6 month follow-up (b = -.02, t = -2.61, p = .01). Relative to the control group, participants in the treatment group were approximately 2 percentage points less likely to have had sex without birth control at 6 month follow-up. Table G.3 shows the results of the analyses using the Benjamini- Hochberg adjustment. To explore the sensitivity of results from the benchmark sample, two different sensitivity analyses were conducted. First, because data on one or more of the outcome variables of interest were missing for some participants, this could lead to biased impact estimates including causality bias. To address missing data from partially completed surveys, multiple imputation (MI) was used. Five multiple imputed datasets were created and the regression models used in the impact POSITIVE PREVENTION PLUS 25 analysis were computed on the each of the 5 imputed data sets and on the pooled estimates. Variables used in the imputation included the same study outcomes of interest and covariates (gender, age, and Hispanic (binary)) used in the benchmark analysis. Imputations were conducted separately for treatment and control groups. The impact regression model used in the benchmark analysis was tested in the MI sample providing pooled estimates of the coefficients from the 5 imputed data sets. The second sensitivity analysis used logical imputation. That is, imputing a missing score based on how each respondent should have answered. For example, if a participant had reported having sex in lifetime at pretest but reported had not having sex at posttest, their score on posttest was changed to having had sex in lifetime (i.e. carried through to follow-up). The impact regression model used in the benchmark analysis was tested in the logical imputation sample. Further explanation of the logical imputation approach is presented in Table H.1. Table H.2 shows the results of the sensitivity analyses compared to the benchmark analysis. When impact estimation used the logical imputation sample, estimated impacts were similar to those in the benchmark analysis with the impact of the program on delaying sexual activity remaining robust. When compared to the impact estimation using the multiple imputation (MI) sample, impact of the program on delaying sexual activity only approaches statistical significance. This may be due to the way the impact estimates are calculated on the MI sample. That is, the impact regression model tested in the MI sample uses pooled estimates of the coefficients from the 5 imputed data sets. The standard error tends to be much larger than those in each individual imputed dataset because there are two sources of variation in the calculation: between iterations and within each iteration. It should be noted that the impact regression models POSITIVE PREVENTION PLUS 26 computed on each of the 5 imputed datasets (not presented) in the MI sample individually yielded similar results to one another as well as to the benchmark analysis (See Table G.2). V. Conclusion This report focused on the impact of the Positive Prevention PLUS (PP+) teen pregnancy prevention program on delaying sexual intercourse, pregnancy, and using birth control among those students who were offered the program. This study indicates that the program has a small but statistically significant impact on delaying sexual intercourse and birth control use at the six- month follow-up. However, the program did not affect whether or not participants would become pregnant (or get someone pregnant). These findings are consistent with a previous study on an earlier version of Positive Prevention.6 The use of a clustered randomized control trial with an analytic sample equivalent at baseline provides rigorous evidence concerning the effectiveness of the program. While the results of this are promising, several methodological limitations should be noted. First, data were collected using self-reported surveys. Although it is impossible to be completely confident of the validity of self-report responses, there is some evidence that supports the general validity of adolescents’ self-report of health behaviors.10 Second, the survey items asking student about which TPP-related topics they had learned about in the past 6 months is only reflective of student experiences after the intervention occurred. As a result, this may not reflect the experience of the control group during the same time period that the treatment group was receiving the intervention even though the intervention is relatively short. Third, variation in program implementation may have affected the strength of the treatment received. Prevention programs are seldom implemented perfectly and several studies have revealed the extent to which program fidelity occurs and how it affects program outcomes.12 POSITIVE PREVENTION PLUS 27 Since a significant number of a youth engage in sexual risk behaviors, the need for effective school-based teen pregnancy curriculums cannot be overstated. Although many sexuality education programs can increase student knowledge, only a few have demonstrated an impact on students’ behaviors. Findings from this study suggest that the Positive Prevention PLUS program is effective in the short term on reducing sexual initiation and unprotected sex. These results have implications for both health educators and researchers. Future studies should investigate the mechanisms by which teen pregnancy prevention programs affect behavior. This could include an examination of the possible relationships between program activities, determinates of behaviors (e.g. self-efficacy, attitudes, behavioral capability) and behavioral outcomes (e.g. birth control use). Additionally, the literature would be enhanced by exploring the long term impacts of the Positive Prevention PLUS program on adolescent sexual risk-taking behaviors and teen pregnancy. POSITIVE PREVENTION PLUS 28 VI. References 1 Martin J, Hamilton B, Martin J, Osterman M, et al. Births: Final Data for 2013. National Center for Health Statistics Website. http://www.cdc.gov. Accessed January 23, 2015. 2 Kost K. Unintended Pregnancy Rates at the State Level: Estimates for 2010 and Trends Since 2002. New York: Guttmacher Institute Website. http://www.guttmacher.org/pubs. Accessed January 23, 2015. 3 Takahashi ER, Florez CJ, Biggs MA, Ahmad S, Brindis CD. Teen Births in California: A Resource for Planning and Policy. Sacramento, CA: California Department of Public Health, Maternal, Child and Adolescent Health Division and Office of Family Planning, and the University of California, San Francisco., 2008. 4 Goesling B, Colman S, Trenholm C, Terzian M, Moore K. Programs to reduce teen pregnancy sexually transmitted infections, and associated sexual risk behaviors: A systematic review. J Adolescent Health. 2014;54(5):499-507. 5 Pedlow C, Teal C, Michael P. Developmentally Appropriate Sexual Risk Reduction Interventions for Adolescents: Rationale, Review Interventions, and Recommendations for Research and Practice. Ann Behav Med. 2004;(27):172- 184. 6 LaChausse, R. Evaluation of the Positive Prevention HIV/STD Curriculum. Am J Health Educ. 2006;37:203–209. 7 Goesling B, Colman S, Trenholm C, Terzian M, Moore K. Programs to reduce teen pregnancy, sexually transmitted infections, and associated sexual risk behaviors: A systematic review. J Adolesc Health. 2014;54:499- 507. 8 Rolleri LA, Fuller TR, Firpo-Triplett R, Lesesne CA, Moore C, Leeks KD. Adaptation Guidance for Evidence- Based Teen Pregnancy and STI/HIV Prevention Curricula: From Development to Practice. Am. J. Sex. Educ. 2014;9(2):135-154. 9 Lopez LM, Tolley EE, Grimes DA, Chen-Mok M. Theory-based interventions for contraception. Cochrane Database Syst. Rev. (Online):CD007249, 2009. 10 Griffin K, Botvin G, Nichols T. Effects of a school-based drug abuse prevention program for adolescents on HIV risk behaviors in young adulthood. Prev Sci. 2006;(7):103-112. 11 Raudenbush S, Bryk A. Hierarchical Linear Models (Second Edition). Thousand Oaks, CA: Sage Publications; 2002. 12 Schochet P. Technical Methods Report: Guidelines for Multiple Testing in Impact Evaluations. (NCEE 2008#4018). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, 2008. POSITIVE PREVENTION PLUS 29 Appendix A. Program Description and Logic Model Table A.1: Application of Social Cognitive Theory (SCT) Constructs in the Positive Prevention PLUS Program Applications in Positive Prevention PLUS SCT Construct Definition (including lesson #) Environment Seeing data on teen sexual abstinence (Getting Started) Exploring myths and stereotypes re: HIV-infected persons (4) Identifying teen-friendly reproductive health services Factors physically external to in the community (2,6,9) the person (e.g. social norm). Analyzing media pressures (8) Situation Perception of the Decision-making re: an unplanned pregnancy (3) environment; correct Recognizing warning signs in relationships (1) misperceptions and promote Recognizing forms of intimacy (1) healthful forms Risk-recognition and risk-reduction strategies (7) Expectations Understanding family planning, using contraception, being abstinent (2) Anticipatory outcomes of a Safely surrendering a newborn baby (3) behavior Using assertiveness and refusal skills (8) Behavioral Knowledge to perform a Practicing assertiveness and refusal skills (8) Capability given behavior (e.g. refusal Recognizing and avoiding risk situations, incl. skills, contraceptive use) universal precautions, condom use (7) Self-Efficacy Practicing assertiveness and refusal skills (8) The person’s confidence in Recognizing and avoiding risk situations (5,7), using performing a particular condoms (7) behavior Knowing how & where to access STD testing (9) Reciprocal Utilizing teen-friendly reproductive health services Determinism (2, 6, 9) Recognizing the existence of Safe Surrender The dynamic interaction of Sites in the community (3) the person, the behavior, and Youth discuss sexual attitudes and behaviors with the environment parents (1-10) Reinforcement Youth receive praise for proper practice of the refusal Responses to a person’s model (8), for identifying ways to help HIV-infected behavior that increase or persons (4), for identifying reasons to be sexually decrease the likelihood of abstinent (2), for making responsible decisions (3), reoccurrence. for recognizing pressure situations (8). Observational Youth observe others (receive praise for) proper Learning practice of the refusal model (8), for identifying Behavior change occurs by reasons to be sexually abstinent (2), for making watching the actions and responsible decisions (3), recognizing pressure outcomes of others’ behavior situations (8). POSITIVE PREVENTION PLUS 30 Figure A.1: Program Logic Model POSITIVE PREVENTION PLUS 31 Appendix B: Data Collection Periods Table B.1. Data Collection Efforts Used in the Impact Analysis of Positive Prevention PLUS and Timing. Data collection effort Timing Parental Consent 8/23/13-8/30/13 Random Assignment 9/3/13 Initial Teacher Training (2 Days) 9/10-9/11/13 Follow-up Training (1 day) 10/15/13 Baseline Survey 10/21-10/25/13 Program Implementation 10/28-11/15/13 6 Month Follow-Up 5/08-5/20/14 Notes: Data collection occurred during one time period and was pooled across sites. POSITIVE PREVENTION PLUS 32 Appendix C: Primary Research Questions Table C.1: Behavioral Outcomes Used for Primary Impact Analyses Research Questions Outcome Timing of measure name Description of outcome relative to program Ever had The variable is a yes/no measure of whether a person has ever 6 months after sexual had sexual intercourse. The measure is taken directly from the program ends intercourse following item on the survey: • “Have you ever had sexual intercourse?” The variable is constructed as a dummy variable where respondents who respond yes they have had sex are coded as 1 and no are coded as 0. Missing data was coded as 999. Ever been The variable is a yes/no measure of whether a person has ever 6 months after pregnant been pregnant or gotten someone pregnant. The measure is program ends taken directly from the following item on the survey: • “To the best of your knowledge, have you been pregnant or gotten somebody pregnant?” The variable is constructed as a dummy variable where respondents who respond yes have been pregnant were coded as 1 and those who responded no are coded as 0. Missing data resulting from the skip pattern of the survey was logically imputed to 0 since one can infer that someone who had never had sex has never been pregnant. Had sex in The variable is a yes/no measure of whether a person has sex in 6 months after last 3 months the past 3 months without using birth control. The measure is program ends WITHOUT taken directly from the following item on the survey: birth control • “In the past 3 months, have you had sexual intercourse without you or your partner using any of these methods of birth control?” -Condoms -Birth control pills -The shot (Depo Provera) -The patch -The ring (NuvaRing) -IUD (Mirena or Paragard)-Implant (Impanon) The variable was constructed as a dummy variable where respondents who respond yes have had sex in last 3 months without birth control are coded as 1 and those who responded no are coded as 0. Missing data due to the skip pattern was coded as 0 since one can infer that someone who had never had sex had also never had sex without birth control. POSITIVE PREVENTION PLUS 33 Appendix D: Intervention Status Table D.1: Sample Size by Intervention Status Total Intervention Comparison Total Intervention Comparison response response response Number of: Time period sample size sample size sample size rate rate rate Clusters: At beginning of study . 21a 11 10 N/A. . . Clusters: Contributed at least one October 21st- youth at baseline 25th, 2013 21 11 10 =1.00 =1.00 =1.00 Clusters: Contributed at least one May 8th- youth at follow-up 20th, 2014 21 11 10 =1.00 =1.00 =1.00 Youth: In non-attriting clusters/sites at time of assignment . 4,267 2,483 1,784 . . . Youth: Contributed a baseline survey . 3,554 2,149 1,405 =.83 =.87 =.79 Youth: Contributed a follow-up May 8th- survey 20th, 2014 3,490 2,113 1,377 =.82 =.85 =.77 POSITIVE PREVENTION PLUS 34 Appendix E: Implementation Fidelity Methods Table E.1: Methods Used to Address Implementation Research Questions Implementation element Methods used to address each implementation element Adherence: How often were The teacher curriculum log measured the date each lesson was taught, adherence to each curriculum sessions offered? How many component or lesson activity, and adaptations made to lessons. Teachers in the treatment group were asked were offered? to self-report the extent to which each topic, worksheet, or activity was covered completely or if any changes were made and why after teaching each lesson. Adherence: What and how much was received? This was measured using attendance data from each school district’s student information system (SIS). Adherence: What content was The teacher curriculum log measures the date each lesson was taught, adherence to each curriculum delivered to youth? component or lesson activity, and if any adaptations were made to a lesson activity. Teachers in the treatment group were asked to self-report the extent to which each topic, worksheet, or activity was covered completely or if any changes were made. Adherence: Who delivered material to youth? Demographic information on program implementers (teachers) was captured on the teacher survey. Quality: Quality of staff- Classroom observations were completed on one lesson per teacher at random using the classroom participant interactions observation form. Scores from each of the observations ranged from one to five (1 = Poor Understanding, 3 = Some understanding, 5 = Good understanding). To calculate each teacher’s average quality score, the sum of each teacher’s observation scores were divided by the number of observation questions (n=10). An average high score of 4 or better indicates a high-quality rating. Quality: Quality of youth The quality of youth engagement was captured through classroom observations (e.g., “How exactly did the engagement with program group members participate in the discussions and activities?” 1 = Little participation, 3 = Some participation, 5 = Active participation). Student engagement was measured by dividing the student participation observation scores of 4 or better and dividing it by the total number of observations. POSITIVE PREVENTION PLUS 35 Implementation element Methods used to address each implementation element Counterfactual: Experiences of Teachers in both the treatment and control groups were asked questions (teacher survey) about their counterfactual condition background, comfort teaching health related topics and if they had taught any health related topics in the past six months (i.e., Which of the following health topics have you taught about in the last six months? Drugs/tobacco/alcohol, abstinence, nutrition, decision making etc. Students were given a similar surveys asking “Which of the following health topics have you learned about this school year? -pregnancy prevention, AIDS or HIV infection, human sexuality, taking care of a baby, using condoms, or abstinence.” The percent of TPP health topics received by students in each group was calculated by summing the responses from each student (1 = yes, 0 = no) and dividing them by the total number of possible responses. The percent of TPP health topics taught by teachers in each group was calculated by summing the responses from each teacher (1 = yes, 0 = no) and dividing them by the total number of possible responses. Context: Other TPP Teachers were asked if their school site had conducted any school-wide activities like assemblies, club programming available or events, or guest speakers regarding pregnancy prevention, sexuality, HIV/AIDS, or reproductive health in offered to study participants the past 6 months and asked to explain the type and duration of that activity. The percent of TPP health (both intervention and topics taught by teachers in each group (not in the PP+ program) was calculated by summing the responses counterfactual) from each teacher (1 = yes, 0 = no) and dividing them by the total number of possible responses. Context: External events This information was captured through the curriculum fidelity log and informal conversations with affecting implementation treatment group teachers. Context: Substantial This information was captured through the curriculum fidelity log and informal conversations with unplanned adaptation(s) treatment group teachers. TPP = Teen Pregnancy Prevention POSITIVE PREVENTION PLUS 36 Appendix F: Implementation Fidelity Data Table F.1: Implementation adherence summary Implementation element Summary of Findings Adherence: How often There were 1,353 lessons offered in the program (123 sections of the 11 lesson program). Each lesson was were sessions offered? approximately 40 minutes in length. The average weekly frequency was 455 lessons within the 18 school day How many were offered? implementation period. Adherence: What and The average number of lessons attended by each student was 10.01 (SD = 1.63). Ninety-one percent of the entire how much was received? program was attended (21,418 attended lessons of the 23,529 possible lessons (2,139 students with 11 lessons each)). Table F.1a: Details of program attendance (Adherence) Student attendance by lesson n = 2,139 % PP+ Getting Started 1,712 80.03 Lesson 1: Life Planning 1,912 89.39 Lesson 2: Healthy Relationships 2,117 98.97 Lesson 3: Relationship Violence 2,128 99.49 Lesson 4: Family Planning and Contraceptives 2,130 99.58 Lesson 5: Myths and Stereotypes 2,129 99.53 Lesson 6: HIV Disease and AIDS 2,129 99.53 Lesson 7: Recognizing and Reducing Risk 2,129 99.53 Lesson 8: Peer and Media Pressures 2,129 99.53 Lesson 9: HIV/STD Testing and Community Resources 2,128 99.49 Lesson 10: Steps to Success 2,114 98.83 POSITIVE PREVENTION PLUS 37 Table F.2: Characteristics of program staff and interactions Implementation element Summary of Findings Adherence: Who delivered material to youth? Thirty-six teachers were trained in the program by project staff. Ninety-four percent (thirty- four teachers) of the trained teachers implemented the program. Thirty-three teachers participated in a teacher survey before their first training session. Quality: Quality of staff-participant interactions Seventy-three percent of the 26 observed sessions received a high quality observation. Quality: Quality of youth engagement with Sixty-five percent of the observed sessions were rated with high student engagement (17 of program the 26 observed sessions). POSITIVE PREVENTION PLUS 38 Table F.2a: Demographic characteristics of teachers in treatment condition Treatment Teacher Demographics n = 33 % Gender . . Female 27 81.80 Male 6 18.20 Ethnicity . . White 19 63.30 African American 2 6.70 Asian 1 3.30 Hispanic 5 16.70 Other race 3 10.00 Teaching Credential . . Health Education 17 51.50 Other credential 16 48.50 Degree . . Health Education 8 27.60 Physical Education 3 10.30 Science 13 44.80 Other degree 5 17.20 Years Teaching (M/SD) 10.58 7.33 POSITIVE PREVENTION PLUS 39 Table F.3: Experiences of the treatment and comparison conditions Implementation element Summary of Findings Counterfactual: Experiences of The percent of TPP health topics received by students in each group was calculated by summing the comparison condition responses from each student (1 = yes, 0 = no) and dividing them by the total number of possible responses. The percent of TPP health topics taught by teachers in each group was calculated by summing the responses from each teacher (1 = yes, 0 = no) and dividing them by the total number of possible responses. Table F.3a: Health topic coverage, as described by students “Which of the following health topics have you Treatment Control learned about this school group youth group youth year?” N % N % Pregnancy Prevention 1,620 86.30 354 28.30 AIDS or HIV Infection 1,878 92.90 475 38.00 Human Sexuality 1,257 66.90 300 24.00 Taking Care of a Baby 277 14.70 92 7.40 Using Condoms 1,643 87.50 97 7.80 Abstinence 1,234 65.70 128 10.20 POSITIVE PREVENTION PLUS 40 Table F.3b: Health topic coverage, as described by teachers “Which of the following health Treatment Control topics have you taught about in group group the last 6 months?” teachers N % teachers N % Pregnancy Prevention 21 67.70 1 4.80 AIDS or HIV Infection 23 74.20 6 28.60 Human Sexuality 17 54.80 2 9.50 Taking Care of a Baby 6 19.40 0 0.00 Using Condoms 20 64.50 1 4.80 Abstinence 20 64.50 3 14.30 Table F.4: Implementation context Implementation element Summary of Findings Context: Other TPP programming available or offered to None of the teachers in either group reported any additional school-wide activities study participants (both intervention and comparison) related to pregnancy prevention, sexuality, HIV/AIDs or reproductive health. Context: External events affecting implementation None Context: Substantial unplanned adaptation(s) Six teachers at one school site were not allowed to conduct the condom demonstration per their school district policy. POSITIVE PREVENTION PLUS 41 Appendix G. Baseline Equivalence and Program Impacts Table G.1: Summary Statistics of Key Baseline Measures for the Analytical Sample Intervention Intervention Intervention Comparison versus versus mean or % mean or % comparison comparison (standard (standard mean p-value of Baseline measure deviation) deviation) difference difference Age or grade level 14.63 (.50) 14.63 (.48) 0 .99 Gender (female) .52 .56 . .78 Race: Asian/Pacific Islander .04 .05 -.01 .41 Race:Black .07 .07 0 1.00 Race: White .14 .16 -.02 .29 Race: Native American .03 .03 0 1.00 Race: Multiple Races .60 .59 .01 .55 Race: Unkown .12 .10 .02 .29 Ethnicity: Hispanic .74 .73 . .48 Ever Had Sex .12 .12 0 .76 Ever Been Pregnant .06 .06 0 .73 Had Sex WITHOUT BC in Prior 3 Months .02 .03 .01 .39 Sample sizeb 1,902-1,886 1,238-1,197 . . Notes: a Estimate with Huber-White Robust Standard Errors; b Due to item non-response, N’s vary slightly by variable. POSITIVE PREVENTION PLUS 42 Table G.2 Post-intervention Estimated Effects using Data from PP+ Student Survey to Address the Primary Research Questions Intervention compared to comparison mean difference Intervention Comparison (p-value of Outcome measure mean or % (N) mean or % (N) difference) Ever had sexual intercourse .14 (1,900) .18 (1,238) -.04 (.01) Ever been pregnant (or gotten someone pregnant) .02 (1,902) .03 (1,243) -.01 (.07) Ever had sexual intercourse without using birth control .04 (1,886) .06 (1,197) -.02 (.01) Source: Follow-up surveys administered 6 months after the program. POSITIVE PREVENTION PLUS 43 Table G.3 Benjamini- Hochberg Correction for Multiple Comparisons Reported p value from Benchmark Statistical Analysis of Significance Analytical After BH Outcome measure Sample p value rank Correction Ever had sexual intercourse .01 1.5 0.025 Yes Significant Ever been pregnant (or gotten someone Not pregnant) .07 -- 0.050 No Significant Ever had sexual intercourse without using birth control .01 1.5 0.025 Yes Significant Source: Follow-up surveys administered 6 months after the program. POSITIVE PREVENTION PLUS 44 Appendix H: Sensitivity Analyses This appendix presents impacts of the Positive Prevention PLUS program estimated using the benchmark analytic sample and two alternative samples concerning the treatment of missing data: (1) impact models involving data where logical imputation was made and (2) impact models involving data where multiple imputation was made. The primary impacts are predominantly robust to alternative specifications. When impact estimation used the logical imputation sample, estimated impacts were similar to the primary impact estimates in magnitude and identical in statistical significance. The data decision rules for logical imputation are shown in Table G.1. Table H.1: Logical Imputation Rules IF THEN Had sex in past 3 months= Yes and Ever had sex= No. Impute both to missing value. Ever been pregnant=Yes and Ever The following data Had Sex= No Impute both to missing value. decision rules were Having had sex without birth control applied consistently in prior 3 months = Yes and Ever had across both groups Impute both to missing value. sex= No and for baseline and follow-up data: Between survey inconsistencies (e.g. if Ever had Sex at baseline=Yes and Ever had Sex at 6 month follow-up= Impute Ever had Sex at 6 No or missing.) month follow-up=Yes. POSITIVE PREVENTION PLUS 45 Table H.2: Sensitivity of Impact Analyses using Data from PP+ Student Survey to Address the Primary Research Questions Data Data Data Data Applying Applying Applying Applying Missing Missing Logical Logical Data Data Imputation Imputation Imputation Imputation Intervention Using Data Using Data (Using (Using compared Benchmark Benchmark Decision Decision Multiple Multiple with approach approach p- Rules Rules p- Imputation) Imputation) comparison differencea value difference value differenceb p-value Ever had sex -.04 .01 -.03 .01 .-.04 .10 Ever been pregnant -.01 .07 -.03 .19 -.05 .53 Having had sex without birth control in prior 3 months -.02 .01 -.06 .22 -.11 .28 Source: Follow-up surveys administered 6 months after the program. Notes: a Using estimated adjusted means; b Using pooled estimates of coefficients from MI regression; imputations made separately for the treatment and control groups. PP+= Positive Prevention PLUS.