OAH Evaluation Report Impact Report from the Evaluation of Adolescent Pregnancy Prevention Approaches Final Impacts of the POWER Through Choices Program September 2016 This page has been left blank for double-sided copying. Purpose statement: This study reports the final impact findings from a large-scale demonstration project and evaluation of POWER Through Choices, a comprehensive sexual health education curriculum designed specifically for youth in foster care and other out-of-home care settings. The study reports the long-term impacts of the program on measures of teen pregnancy and associated sexual risk behaviors. The findings build on an earlier report that examined the program’s interim impacts on measures of youth knowledge, attitudes, and intentions. September 2016 Recommended citation: U.S. Departmet of Health and Human Services, Office of Adolescent Health, Final Impacts of the POWER Through Choices Program. Washington, D.C., September 2016. This document is available at http://www.hhs.gov/ash/oah/ OAH is grateful to the authors Reginald D. Covington, Brian Goesling, Christina Clark Tuttle, Molly Crofton, Jennifer Manlove, Roy F. Oman, and Sara Vesely This page has been left blank for double-sided copying. FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM CONTENTS ACKNOWLEDGEMENTS ............................................................................................................................ XI I. INTRODUCTION.............................................................................................................................. 1 A. The POWER Through Choices program ................................................................................... 2 B. Summary of interim impact findings .......................................................................................... 3 C. Research questions ................................................................................................................... 5 II. STUDY DESIGN .............................................................................................................................. 9 A. Evaluation setting ...................................................................................................................... 9 B. Random assignment procedures............................................................................................. 11 C. Enrollment and retention of individual youth ........................................................................... 12 D. Baseline sample characteristics .............................................................................................. 14 E. Treatment and control conditions ............................................................................................ 16 III. DATA, MEASURES, AND ANALYSIS ........................................................................................... 19 A. Outcome measures ................................................................................................................. 19 1. Unprotected sex ................................................................................................................ 21 2. Sexual activity ................................................................................................................... 22 3. Pregnancy risk .................................................................................................................. 22 4. STI risk .............................................................................................................................. 22 5. Knowledge ........................................................................................................................ 23 6. Awareness of available health resources ......................................................................... 24 7. Attitudes toward safe sex and methods of protection ....................................................... 24 8. Perceived self-efficacy to avoid unprotected sex .............................................................. 25 B. Analytic approach .................................................................................................................... 25 IV. RESULTS ....................................................................................................................................... 29 A. Unprotected sexual activity...................................................................................................... 29 B. Other sexual risk behaviors and reproductive health outcomes ............................................. 30 C. Intermediate outcomes ............................................................................................................ 31 D. Subgroup analysis ................................................................................................................... 33 V. DISCUSSION ................................................................................................................................. 36 REFERENCES ............................................................................................................................................ 39 APPENDIX A NONRESPONSE ANALYSIS .............................................................................................A-1 APPENDIX B DATA AND MEASURES ....................................................................................................B-1 APPENDIX C SENSITIVITY ANALYSIS.................................................................................................. C-1 APPENDIX D IMPACT AT EACH FOLLOW-UP...................................................................................... D-1 v This page has been left blank for double-sided copying. FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM TABLES I.1. POWER Through Choices program sessions ................................................................................. 3 I.2. Immediate post-test impacts of the PTC program ........................................................................... 4 II.1. Group home characteristics, by state ............................................................................................ 10 II.2. Baseline demographic and personal characteristics ..................................................................... 15 II.3. Baseline sexual risk behaviors ....................................................................................................... 16 II.4. Percentage of youth living in a group home at follow-up ............................................................... 18 III.1. Outcome measures at 12 months .................................................................................................. 20 IV.1. Impacts on unprotected sexual activity .......................................................................................... 30 IV.2. Impacts on sexual activity, pregnancy risk, and STI risk ............................................................... 31 IV.3. Impacts on intermediate outcomes at 12 months .......................................................................... 32 IV.4. Subgroup impacts, by age group ................................................................................................... 34 IV.5. Subgroup impacts, by gender ........................................................................................................ 35 A.1. Baseline demographic and personal characteristics ....................................................................A-4 A.2. Baseline sexual risk behaviors ......................................................................................................A-5 B.1. Coding rules and sensitivity tests for behavioral outcomes ..........................................................B-4 B.2. Measures of baseline sample characteristics ...............................................................................B-8 C.1. Sensitivity of impacts on primary outcomes ................................................................................. C-4 C.2. Sensitivity of impacts on sexual activity, pregnancy risk, and STI risk ........................................ C-6 This page has been left blank for double-sided copying. FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM FIGURE II.1. Flow of group homes and individual youth..................................................................................... 13 II.2. Attendance in PTC sessions .......................................................................................................... 17 D.1. Differences in rates of sex without any effective method of protection ........................................ D-4 D.2. Differences in rates of sex without using a condom .................................................................... D-4 D.3. Differences in rates of sexual activity in the past three months ................................................... D-5 D.4. Differences in rates of pregnancy in the past three months ........................................................ D-5 D.5. Differences in rates of lifetime pregnancy .................................................................................... D-6 D.6. Differences in rates of having multiple sexual partners in the past three months ....................... D-7 D.7. Differences in rates of being tested for a STI in the past 12 months ........................................... D-7 D.8. Differences in rates of being diagnosed with an STI in the past 12 months ................................ D-8 ix This page has been left blank for double-sided copying. FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM ACKNOWLEDGEMENTS This evaluation was made possible only through the hard work and support of many people and organizations. First, we wish to acknowledge the support of staff from the Office of Adolescent Health (OAH) in the U.S. Department of Health and Human Services (HHS). We especially thank Amy Farb, the OAH project officer for the Evaluation of Adolescent Pregnancy Prevention Approaches (PPA) study, and Alexandra Warner, who have provided outstanding support and guidance throughout the project. We also thank staff from the Family and Youth Services Bureau within the Administration for Children and Families at HHS, especially Itege Bailey and Marc Clark. At the Oklahoma Institute for Child Advocacy (OICA), a special thanks goes to Janene Fluhr, whose tireless commitment to the POWER Through Choices program served as the driving force behind the evaluation and was instrumental to the study’s success. We also thank Sheila Cavallo from OICA, as well as a large team of talented and dedicated data collectors, program facilitators, and administrators from OICA, Kern County Superintendent of Schools, and Planned Parenthood of Maryland. At the University of Oklahoma Health Sciences Center, we thank Jeannie Williams and Jennifer Green for their expert oversight of a highly successful data collection effort. At Mathematica Policy Research, we thank Christopher Trenholm and Alan Hershey for their guidance and support during the initial study design phase. We also thank Melissa Thomas for supporting the data collection effort, and Melissa Clark of Mathematica and Elizabeth Karberg of Child Trends for useful comments on an earlier draft of the report. Finally, we extend our greatest thanks and gratitude to the youth, program staff, and administrators of the many group homes that agreed to participate in the study. We hope that our report does justice to the time and effort they devoted to making the study possible. Reginald D. Covington Brian Goesling Christina Clark Tuttle Molly Crofton Jennifer Manlove Roy F. Oman Sara Vesely xi This page has been left blank for double-sided copying. FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM I. INTRODUCTION This report presents the final impact findings from a large-scale demonstration project and evaluation of POWER Through Choices (PTC), a comprehensive sexual health education curriculum designed specifically for youth in foster care, the juvenile justice system, and other out-of-home care settings. Prior research indicates that youth in out-of-home care are at particularly high risk for teen pregnancy, sexually transmitted infections (STIs), and associated sexual risk behaviors (Dworsky and Courtney 2010). However, many of these youth report difficulty accessing reproductive health information and services (Freundlich 2003; Crottogini et al. 2008), as well as relatively low levels of knowledge about reproductive health and methods of protection (Hudson 2012). PTC is one of the only comprehensive sexual health education curricula designed to address the needs and risks specific to this population. In an earlier report, we found several promising short-term effects of the PTC program on youth outcomes (Goesling et al. 2015). Drawing on data from a large-scale, multisite random assignment evaluation involving more than 1,000 youth from residential group homes in three states (California, Maryland, and Oklahoma), we found that youth in group homes offering the PTC program reported increased exposure to information on reproductive health, methods of protection, and pregnancy and STI prevention. They also reported increased knowledge of reproductive health, STIs, and methods of protection, as well as increased awareness of available health resources and where to get methods of protection. The program also led to increased support for protected sex, a greater sense of self-empowerment or self-efficacy to avoid unprotected sex, and increased intentions to avoid unprotected sex by using condoms. We measured these outcomes just after youth had attended the 10 program sessions. We did not examine program impacts on behavioral outcomes such as the prevalence of unprotected sex because we did not expect these impacts to emerge until later follow-up periods. In the present report, we examine the program’s longer-term impacts measured 12 months after the PTC program sessions had ended for youth in the treatment group. As our primary research question, we examine whether the PTC program succeeded in reducing rates of unprotected sex among study youth. To supplement our analysis of this primary research question, we also examine several secondary questions of interest, including (1) the 12-month impacts of the program on other sexual risk behaviors and reproductive health outcomes of interest, such as the prevalence of sexual intercourse and the incidence of pregnancy; (2) whether the program’s shorter-term effects on youth knowledge, attitudes, and intentions persisted at the time of the 12-month follow-up survey; and (3) whether the impacts of the program on youth sexual risk behaviors and reproductive health outcomes vary by demographic characteristics such as gender and age. The evaluation has involved a unique collaboration and partnership among several organizations. The Oklahoma Institute for Child Advocacy (OICA) originally designed the evaluation in collaboration with researchers from the University of Oklahoma Health Sciences Center (OUHSC). In fall 2010, OICA received competitive federal grant funding for the evaluation through the Family and Youth Services Bureau within the Administration for Children and Families of the U.S. Department of Health and Human Services (HHS). In winter 2011, the PTC was then selected as one of seven program to participate in the Evaluation of Adolescent Pregnancy Prevention Approaches (PPA) study, a major federal effort to expand 1 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM available evidence on effective ways to prevent and reduce pregnancy and related sexual risk behaviors among teens in the United States. Mathematica Policy Research and its partners, Child Trends and Twin Peaks Partners, LLC, conduct the PPA study under contract with the Office of Adolescent Health within HHS. Participating in PPA provided the PTC evaluation additional resources to support data collection and analysis. In addition, researchers from the PPA evaluation team have collaborated with OICA and OUHSC to refine the evaluation design, support data collection, and plan the analysis. Two other regional organizations—the Kern County Superintendent of Schools and Planned Parenthood of Maryland—have also played key roles in supporting program implementation and study data collection, as explained later in the report. The report is organized into five chapters. In the rest of this chapter, we provide a more detailed description of the PTC program and summarize key findings from our earlier interim report. Chapters II and III provide a detailed description of the study design, data, and analytic methods. Chapter IV presents findings from the final impact analysis, and Chapter V summarizes and discusses the implications of the results. A. The POWER Through Choices program PTC is a comprehensive teen pregnancy, HIV, and STI prevention program designed specifically to address the needs of youth living in foster care, juvenile justice facilities, and other out-of-home care settings. The program comprises 10 90-minute sessions (Table I.1) delivered once or twice a week for 5 to 10 weeks. The sessions feature interactive skill-building activities delivered by trained facilitators to groups of 8 to 20 youth. The program targets male and female youth ages 13 to 18 years. The program developers designed the PTC program for use with youth living in any number of out-of-home care settings, including family foster homes, kinship foster care, residential group homes, transitional living centers, and juvenile justice facilities. We provide a more detailed description of the program in our earlier interim impact report (Goesling et al. 2015) and an accompanying implementation report (Meckstroth et al. 2014). The program sessions emphasize two main themes: self-empowerment and the impact of choices. Prior research suggests that youth in out-of-home care often lack the positive skills and resources necessary to avoid risky sexual behaviors and teen pregnancy, such as dependable family supports and social networks, a strong sense of personal autonomy or control, and the skills necessary to identify and successfully attain personal goals (Becker and Barth 2000). The PTC program seeks to address these barriers by empowering youth to make informed decisions about their sexual risk behaviors and to help them recognize the potential consequences of these decisions for their future goals. The program challenges youth to envision a positive future for themselves and to make choices about relationships and health behaviors that help promote their future success. The present study focuses on the second edition of the PTC program. Researchers in the Family Welfare Research Group at the University of California, Berkeley developed the first edition in the mid-1990s (Becker and Barth 2000). Beginning in 2005, OICA launched a five- year effort to update and expand the original curriculum. This second edition of the PTC program maintains the format, goals, and interactive nature of the original curriculum but 2 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM features two new sessions on reproductive health and STIs, revised role-playing scenarios, and updated data and resource information. The second edition also promotes inclusivity of sexual orientation—for example, through role-playing scenarios on gay and lesbian relationships. Table I.1. POWER Through Choices program sessions Session Title Purpose 1 Introduction to PTC Introduce curriculum, assess participants’ knowledge regarding pregnancy prevention and sex education, and demonstrate role playing 2 Adolescent Reproductive Increase knowledge of male and female reproductive anatomy, the Health Basics process of fertilization and conception, and the menstrual cycle 3 Creating the Future You Want Identify planning involved in practicing positive sexual behaviors, outline individual choices involved in sexual decision making, and discuss abstinence as a viable choice 4 Making Choices Clear Help participants to build assertiveness and communication skills related to sexual activity 5 Understanding STIs and HIV Increase knowledge and understanding of STI/HIV transmission and and How to Reduce Your Risk prevention 6 Increasing Contraceptive Increase knowledge about contraceptive methods Knowledge 7 Practice Makes Perfect Discuss the level of risk associated with various sexual behaviors, use role playing to demonstrate the importance of dual methods, and learn condom use skills 8 Using Resources to Support Discuss ways to improve communication about contraception with foster Your Choices parents, guardians, and group home staff members; learn how to access local sexual and reproductive health resources 9 Making Choices That Fit Your Develop a plan for avoiding unwanted pregnancies and STIs, set short- Lifestyle and long-term goals, and identify choices needed to attain goals 10 Plan + Prepare + Practice = Reinforce themes and messages of the curriculum POWER B. Summary of interim impact findings To assess the impacts of the PTC program on youth outcomes, we designed and conducted a large-scale, multisite random assignment evaluation involving more than 1,000 youth living in residential group homes. As discussed in greater detail in Chapter II, OICA and its partners recruited youth from a volunteer sample of 44 residential group homes across three states: California, Maryland, and Oklahoma. Within each state, about half the group homes were randomly assigned to a treatment group that offered the PTC program. The other homes were assigned to a control group that did not offer the program. In both research groups, we administered four rounds of surveys to study youth: (1) a baseline survey administered before random assignment, (2) an immediate post-test survey administered just after the program had ended in the treatment group, (3) a longer-term follow-up survey administered about 6 months after the PTC program ended, and (4) a final follow-up survey administered about 12 months after the PTC program ended. In an earlier report, we used data from the baseline and immediate post-test surveys to assess the short-term impacts of the program measured just after the end of the 10 program sessions (Goesling et al. 2015). This focus on very short-term program impacts, measured while most study participants still lived in the participating group homes, naturally limited our focus to such 3 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM outcomes as youth knowledge, attitudes, and intentions. We did not examine impacts on any measures of sexual risk behaviors or pregnancy. Table I.2 summarizes the key findings from this report; the rest of this section discusses the findings in greater detail. Table I.2. Immediate post-test impacts of the PTC program Treatment Control Measure group group Difference p-value . . . . Percentage of youth who reported receiving information on the following topics:a Where to obtain birth control 94.1 55.1 39.0** <.01 Talking to partner about sex or birth control 93.4 62.8 30.6** <.01 How to say no to sex 91.6 59.6 32.0** <.01 Relationships, dating, marriage, and family life 91.4 76.9 14.5** <.01 Methods of birth control 90.3 57.2 33.0** <.01 Abstinence from sex 89.1 60.1 29.0** <.01 STIs 88.8 64.9 23.9** <.01 Knowledge of reproductive anatomy and fertilityb 2.75 2.39 0.35** <.01 Knowledge of HIV and STIsc 5.33 4.51 0.82** <.01 Knowledge of methods of protectiond 7.90 6.19 1.71** <.01 Percentage of youth reporting they are very sure where to get methods of protection 66.0 51.1 14.9** <.01 . . . . Percentage of youth reporting they strongly agree that Condoms are pretty easy to get 64.1 58.2 5.9 0.31 Birth control is easy to get 31.7 27.8 3.9 0.57 General support for methods of protectione 3.48 3.32 0.16** <.01 Perceived barriers to methods of protectionf 2.50 2.44 0.06 0.33 Perceived ability to communicate with partner g 3.54 3.28 0.26** <.01 Perceived ability to plan for and avoid unprotected sexh 3.35 3.08 0.26** <.01 . . . . Percentage of youth reporting intentions to engage in the following behaviors in the next 12 months: Have sexual intercourse 69.9 68.5 1.4 >0.99 Use condoms if having sex 57.6 43.9 13.7** <.01 Use other protection method if having sex 45.0 39.7 5.3 0.67 Source: Goesling et al. (2015) a Questions refer to information received in the 12 months before survey administration. b Index ranging from 0 to 4, with higher values indicating a greater number of correct responses. c Index ranging from 0 to 7, with higher values indicating a greater number of correct responses. d Index ranging from 0 to 10, with higher values indicating a greater number of correct responses. e Scale ranging from 1 to 4, with higher values indicating greater support. f Scale ranging from 1 to 4, with higher values indicating fewer perceived barriers. g Scale ranging from 1 to 4, with higher values indicating greater perceived ability. h Scale ranging from 1 to 4, with high values indicating greater perceived ability. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. 4 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM As shown in Table I.2, we found that the PTC program had large and statistically significant impacts on youth exposure to information on reproductive health and sexual education topics. Among youth assigned to the treatment group, more than 90 percent reported receiving information on topics such as where to obtain birth control (94.1 percent); talking to a partner about sex or birth control (93.4 percent); how to say no to sex (91.6 percent); relationships, dating, marriage, and family life (91.4 percent); and methods of birth control (90.3 percent). Fewer youth in the control group had received such information. For example, less than 60 percent of youth in the control group reported receiving information on how to say no to sex (59.6 percent), methods of birth control (57.2 percent), or where to obtain birth control (55.1 percent). We also found statistically significant impacts on key measures of knowledge and attitudes. For example, on a four-item index of knowledge of reproductive anatomy and fertility, we found an average score of 2.75 for youth in the treatment group and 2.39 for youth in the control group. This difference represents a roughly 15 percent increase in knowledge for youth in the treatment group relative to the control group (0.35/2.39 = 15 percent). Youth in the treatment group also reported higher scores on an index of knowledge of HIV and STIs, an index of knowledge of methods of protection, and attitude scales measuring perceived ability to communicate with a partner and to plan for and avoid unprotected sex. Youth in the treatment group were more likely than those in the control group to report feeling very sure where to get methods of protection (66.0 percent versus 51.1 percent). They were also more likely to agree with the statements “condoms are easy to get” and “birth control is easy to get,” though the reported differences for these two outcomes were not statistically significant at the 5 percent level. The findings for the intentions measures were also generally consistent with the program messages. Slightly more than two-thirds of both the treatment and control groups said they planned to have sex in the next 12 months (69.9 percent for the treatment group and 68.5 percent for the control group). This finding aligned with expectations, first, because most sample members had already had sex before enrolling in the study (87.4 percent for the treatment group and 87.0 percent for the control group) and, second, because although the program teaches abstinence as the only way to fully avoid the risk of pregnancy and STIs, it does not promote or value abstinence over other methods of protection. The program strongly emphasizes condom use—both as a method to reduce pregnancy risk and especially for protection against STIs. Consistent with this emphasis, we found that youth in the treatment group were more likely than those in the control group to say they planned to use a condom when having sex (57.6 versus 43.9 percent). Youth in the treatment group were also more likely to say they planned to use other measures of protection, such as birth control pills or intrauterine devices (IUDs), though the difference from the control group is not statistically significant (45.0 versus 39.7 percent). C. Research questions The present report adds to these findings by examining the PTC program’s longer-term impacts measured 12 months after the program sessions had ended. The study was ultimately designed to assess the program’s success in reducing rates of unprotected sex among study youth at the time of the 12-month follow-up survey. The earlier post-test and 6-month follow-up surveys served secondary purposes, such as examining shorter-term impacts on intermediate outcomes (knowledge, attitudes, and so on) and analyzing trends or changes in youth behaviors 5 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM (see Appendix E). For the purpose of this report, we thus focus our analysis on data from the 12- month follow-up and the following primary research question: • Is the PTC program effective in reducing rates of unprotected sex? To provide a comprehensive assessment of the program, we also supplement our analysis of this primary research question with several secondary research questions of interest. First, we examine the 12-month impacts of the PTC program on other sexual risk behaviors and reproductive health outcomes of interest. As noted earlier, although the program provides information on condoms and other methods of protection, it teaches abstinence as the only way to fully avoid the risk of pregnancy and STIs. For this reason, we collected data on overall rates of sexual activity among study youth in addition to data on rates of unprotected sex. We also collected data on pregnancy history and STI risk factors, such as number of sexual partners and STI diagnoses and testing. These outcomes are central to the long-term goals of the PTC program, but we did not know when designing the study whether the reported prevalence rates of outcomes such as pregnancy or STI diagnoses would be high enough among study youth to permit an analysis of program impacts on these outcomes. As a result, we designated these measures as secondary rather than primary outcomes. To examine the program’s impacts on these additional sexual risk behaviors and reproductive health outcomes of interest, we use data from the 12-month follow-up survey to answer the following secondary research questions: • Does the PTC program reduce overall rates of sexual activity? • Is the PTC program effective in reducing the risk of pregnancy? • Does the PTC program affect STI risk factors, such as number of sexual partners and STI diagnoses and testing? As additional secondary analyses, we also examine the 12-month impacts of the program on measures of youth knowledge, attitudes, and awareness. As described earlier, although our interim report found promising short-term effects on these outcomes, the analysis was based on data from an immediate post-test survey administered just after the end of the 10 program sessions. To assess whether these short-term impacts persisted in the months after the program ended, we use longer-term data from the 12-month follow-up survey to answer the following research questions: • Does the PTC program have sustained, longer-term impacts on youth knowledge of reproductive health, STIs, and methods of protection? • Does the PTC program have sustained, longer-term impacts on youth awareness of available health resources and where to get methods of protection? • Does the PTC program have sustained, longer-term impacts on levels of support for protected sex and methods protection? • Does the PTC program have sustained, longer-term impacts on feelings of self- empowerment or self-efficacy to avoid unprotected sexual activity? 6 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Finally, for the measures of sexual risk behavior and reproductive health outcomes, we also test for possible variation in program impacts for subgroups of youth defined by gender and age group. As described in greater detail in Chapter III, we designed the study with the goal of estimating impacts for the full sample of youth enrolled in the study. As such, we answer our primary research question using full-sample data. However, we also recognize the possibility that the effects of sexual health education programs like PTC could vary on the basis of personal or demographic characteristics. To account for this possibility, we estimate program impacts separately by gender and age group. 7 This page has been left blank for double-sided copying. FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM II. STUDY DESIGN We designed the study as a cluster randomized trial involving youth recruited from 44 residential group homes across three states: California, Maryland, and Oklahoma. Within each state, we randomly assigned about half the group homes to a treatment group that offered the PTC program and half to a control group that did not offer the program. This design resulted in about half the study youth being offered the PTC program and half not being offered the program. We calculate program impacts by comparing youth outcomes between the two groups roughly 12 months after the end of the PTC program. In this chapter, we begin by describing the evaluation setting and recruitment of group homes. We then describe the random assignment procedures. Next, we describe the enrollment and retention of individual youth within the participating group homes and the baseline characteristics of the study sample. We end by providing a summary description of the treatment and control conditions. The next chapter describes the data, measures, and analytic methods that we used to estimate impacts of the PTC program. A. Evaluation setting Identifying an appropriate setting posed a key early challenge for the evaluation. The program developers designed the PTC program for use among youth living in any number of out-of-home care settings, including family foster homes, kinship foster care, residential group homes, transitional living centers, and juvenile justice facilities. However, the evaluation required implementing the program on a very large scale to hundreds of participating youth. The need to implement the program on such a large scale ruled out common settings such as family foster homes or kinship foster care that could not generate the necessary sample size but were otherwise appropriate for the program. Recognizing this challenge, OICA and OUHSC made the key early decision to conduct the evaluation with youth living in residential group homes. These homes are licensed or approved facilities that provide 24-hour care in a small (fewer than 12 youth) or large (12 or more youth) group setting operated or contracted by a state child welfare agency, a state juvenile justice agency, or a private care provider. They typically serve a very high-risk population of youth referred for full-time residential housing and care through the state child welfare (foster care) or juvenile justice systems. The youth placed in these homes typically live on site for several months, often as a result of a court order. The combination of a residential setting and high-risk population made these homes a feasible and important venue for evaluating the PTC program. To achieve the necessary sample size for the evaluation, OICA also recognized a need to recruit residential group homes across multiple states. OICA led the recruitment of group homes within its home state of Oklahoma and partnered with two organizations to help recruit homes and implement the PTC program in other states. The Kern County Superintendent of Schools (California) recruited homes from Kern and San Lois Obispo counties, California. Planned Parenthood of Maryland recruited homes from seven counties around Baltimore, Maryland. OICA selected these organizations because they had existing relationships with their state foster care systems as well as history and experience delivering sexual health education (Meckstroth et al. 2014). 9 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM OICA and its partners recruited residential group homes across Oklahoma, California, and Maryland on a rolling basis from summer 2011 through fall 2013. In each state, members of the study team worked with the state and local officials in the child welfare and juvenile justice systems to identify prospective homes. The selected homes were not intended to be a random or representative sample of all group homes in the targeted geographic areas. Rather, the team sought a nonprobability or purposive sample of homes with the capacity and commitment to support the study. The PTC implementation report (Meckstroth et al. 2014) provides a more detailed description of the recruitment process. Table II.1. Group home characteristics, by state . California Maryland Oklahoma Total Total number 19 homes 10 homes 15 homes 44 homes (465 youth) (196 youth) (376 youth) (1,037 youth) Contracted to serve youth from . . . . Child welfare system 0 0 8 8 Juvenile justice system 2 0 7 9 Both child welfare and juvenile justice systems 17 10 0 27 . . . . Gender serveda Male 11 5 10 26 Female 7 3 4 14 Both 1 2 1 4 . . . . Size (number of beds) 8 or fewer 16 5 0 21 9 to 16 1 0 14 15 17 to 31 0 4 0 4 32 or more 2 1 1 4 . . . . Primary setting or type Cottage or home setting 17 5 0 24 Campus or dormitory setting 1 4 15 18 Academy 1 1 0 2 . . . . Schooling On site 3 0 12 15 Off site (public schooling) 16 7 0 23 Both 0 3 3 6 Unsupervised leave allowedb . . . . Yes (contingent on approval or behavioral status) 17 10 0 27 No (not at all) 2 0 15 17 Source: Meckstroth et al. (2014). a The evaluation excluded homes that serve pregnant and parenting teens. b Excluding a pass to visit home. Unsupervised leave refers to leave within the community where the group home is located. These recruitment efforts led to a final sample of 44 group homes spread across three participating states (Table II.1). The homes served a mix of youth from the child welfare and juvenile justice systems. Some of the homes contracted with only one of the two systems. However, even in these homes, it is possible that some of the resident youth were dually adjudicated, meaning they had ongoing court involvement in both the juvenile justice and child 10 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM welfare systems. The homes ranged in size from small cottage-style homes with eight or fewer residents to larger campus or dormitory settings with dozens of residents. Reflective of the fact most youth placed in group homes are male, nearly 60 percent of the homes served only male residents. The other homes served only female residents (30 percent) or were coed (10 percent). Most homes allowed residents to travel off site to attend public schools. A smaller number offered schooling on site as part of their core program services. The homes also had different policies concerning unsupervised leave for reasons other than school attendance. B. Random assignment procedures Three main factors shaped the study’s approach to random assignment: 1. Need to assign youth in clusters. It was not feasible to randomly assign youth to treatment and control groups within the same home because the residential setting allowed for potential spillover or contamination effects if a youth assigned to the treatment group interacted with youth in the control group. Randomly assigning individual youth also presented a logistical challenge of transporting youth assigned to the treatment group to a central location to receive the intervention. To address these complications, we designed the evaluation so that all youth living in the same group home were assigned to the same research status, either treatment or control. This approach yielded a cluster random assignment design in which youth were assigned to the treatment or control conditions in intact clusters or groups. 2. Homes with multiple dormitories or living facilities. Some of the group homes recruited for the study had multiple dormitories or living facilities located on a single campus. Some of these dormitories or facilities were organized by gender; others were organized by different types of care or treatment services. In some cases, these dormitories or facilities were large and separate enough to randomly assign on their own as independent clusters. In part for this reason, some of the 44 group homes recruited for the study yielded more than one cluster of youth for the purpose of random assignment. In other cases, the individual dormitories or facilities were too small to randomly assign on their own and were instead combined for random assignment. 3. Rolling recruitment of group homes. Because we recruited group homes on a rolling basis, we did not wait until the end of the recruitment period to conduct random assignment. Rather, we randomly assigned homes in small pairs or groups as we recruited new homes into the study. This rolling approach to random assignment also allowed for the possibility of randomly assigning the same group home more than once, after the population of youth in the home had turned over fully. The opportunity to generate multiple clusters of youth from the same group home further boosted the number of randomized clusters. These features of the design ultimately yielded a sample of 80 clusters of youth for the purpose of random assignment across the 44 group homes (Figure II.1). To help ensure an even balance between the treatment and control groups, and to improve the precision of the impact estimates, we divided the clusters into 39 separate strata on the basis of location (California, Maryland, or Oklahoma); recruitment date; size; and gender of youth served. We randomly assigned the first four homes recruited in California as a stratum of four clusters. All other clusters were grouped into matched pairs of two clusters for random assignment. Within each 11 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM matched pair or stratum, we randomly assigned an equal number of clusters to the treatment and control groups. None of the 80 clusters was lost to follow-up during the study period. C. Enrollment and retention of individual youth Within each home, all resident youth ages 13 to 18 were eligible to participate in the study, contingent on the study’s consent and assent requirements. The study required consent from a legally authorized representative for each youth. The identity of this representative varied across states, depending on state law and regulations. In California, the consent process required a signature from the individual lawyer or probation officer assigned to each youth or, in some cases, a biological parent. In Maryland, officials at the state level had authority to complete much of the consent process without involvement from individual social workers, biological parents, or other representatives. In Oklahoma, for youth in the child welfare system, the consent process required a signature from a caseworker. For youth in the juvenile justice system, consent required signature from a state official or, in some cases, a biological parent. In addition to these consent requirements, the study also required individual youth to assent to participate in the study. In each participating home, the study team began the consent and assent process before random assignment to avoid any bias that might result from knowledge of the random assignment results. The resulting consent and assent rates for the study were high (Figure II.1). The study team obtained consent for almost 98 percent of all youth eligible for enrollment. Only one youth declined to provide assent for the study. The sites achieved these high rates by taking an individualized approach to gathering consent and assent. They began by working with the relevant state child welfare and juvenile justice agencies to gain buy-in for the study and identify the appropriate process of obtaining consent. For youth who required consent from an individual case worker, the sites had their data collectors visit the case workers in person to explain the study and request consent. For youth who required consent from a biological parent or legal guardian, the sites worked with the administrator in each participating group home to explore the possibility of incorporating the study’s consent form into the home’s existing intake or enrollment materials. The data collectors also relied on staff in the participating group homes to help facilitate the process of gathering assent from study youth. The data collectors sought to connect personally with eligible participants by visiting the group homes, meeting with individual youth, and clearly describing the purpose of the study and what participation would involve. All youth enrolled at the beginning of the study were eligible to complete the later follow-up surveys. In the present report, we focus primarily on data from the final follow-up survey administered about 12 months after PTC program sessions ended for youth in the treatment group. The data collectors attempted to administer the surveys as close as possible to each youth’s scheduled survey completion date. However, to accommodate scheduling issues and other practical constraints, the data collectors had a window of two weeks before and after the scheduled completion date to administer the surveys. Prior surveys of youth have successfully used this approach of providing a short window for data collection (Oman et al. 2009). According to the data collection protocol established at the start of the study, we considered any surveys completed outside the defined window invalid and excluded them from our subsequent analyses. Chapter III provides a more detailed description of the data collection process. 12 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Figure II.1. Flow of group homes and individual youth Response rates for the 12-month follow-up survey were similar for youth in the treatment and control groups. Among the 517 youth assigned to the treatment group, 436 completed the 12-month survey, for a response rate of 84 percent. Among 519 youth assigned to the control group, 449 completed the 12-month survey, for a response rate of 87 percent. As described later in this chapter, not all study participants still lived in the group homes at the time of the 12- month follow-up survey. However, all participants were eligible for the follow-up survey regardless of their living arrangements. See Appendix A for a nonresponse analysis examining the characteristics of the youth who did not complete the 12-month survey. 13 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM D. Baseline sample characteristics The demographic characteristics of the study sample (Table II.2) reflect the characteristics of the group homes recruited for the study. At the time of the baseline survey, before the start of the program, the participants ranged in age from 13 to 18 years. The large majority were male (78.7 percent of the treatment group and 77.5 percent of the control group). The study participants had diverse racial and ethnic backgrounds. Hispanics represented the largest group (36.6 percent of the treatment group and 38.2 percent of the control group), but the sample also had sizeable numbers of non-Hispanic blacks and whites. Most participants reported having lived in their current group home for fewer than six months. More than 4 in 10 participants were behind in school relative to their age, and more than a quarter of the study sample did not expect to graduate from high school. The study participants reported high rates of sexual activity and associated risk behaviors at baseline (Table II.3). In both study groups, nearly 9 in 10 participants reported some lifetime experience with sexual intercourse with a partner of the opposite sex. About 7 in 10 participants reported having early sexual intercourse—defined as first having sexual intercourse by the age of 14. A large majority reported having more than three lifetime sexual partners (64.9 percent in the treatment group and 64.1 percent in the control group). One-third of the sample reported having had sex without a condom in the past three months. As a result of these risk behaviors, more than one-third of the sample also reported having been pregnant or gotten a partner pregnant at some point in their lives. All of these rates are substantially higher than national averages. 14 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table II.2. Baseline demographic and personal characteristics Treatment Control Measure group group Difference p-valuea . . . . Age in years (%) Younger than 15 years old 11.9 13.4 -1.5 0.16 15 years old 16.5 16.7 -0.2 . 16 years old 27.8 28.3 -0.5 . 17 years old 31.2 34.1 -2.9 . 18 years old or older 12.6 7.6 5.0 . Male (%) 78.7 77.5 1.2 0.68 . . . . Race/ethnicity (%) Hispanic 36.6 38.2 -1.6 0.97 Non-Hispanic black 19.8 18.8 1.0 . Non-Hispanic white 19.1 20.5 -1.5 . Non-Hispanic Native American 4.1 3.6 0.6 . Non-Hispanic Asian or Pacific Islander 1.6 1.6 0.1 . Non-Hispanic other or multiple racesb 18.9 17.4 1.4 . . . . . Months in current group home (%) Fewer than 3 months 42.8 43.3 -0.5 0.98 3–6 months 36.8 36.1 0.7 . More than 6 months 20.4 20.7 -0.3 . Behind grade level (%) 41.2 43.5 -2.3 0.48 . . . . Highest level of education likely to complete (%) Less than high school 26.4 28.1 -1.8 0.55 Graduate from high school 17.2 20.8 -3.6 . Some college or technical training 15.8 14.5 1.3 . Graduate from a 2-year college 9.2 7.6 1.6 . Graduate from a 4-year college 31.4 29.0 2.4 . . . Sample sizec 436 449 Source: Baseline surveys administered to study participants before the start of the program. a Reported p-values are based on two-tailed t-tests for dichotomous measures and chi-squared tests for categorical measures. b This category includes respondents who selected multiple races. c Reported sample size is the number of participants who completed the 12-month survey and are included in the analysis; it does not account for item nonresponse for any measures included in the table. 15 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table II.3. Baseline sexual risk behaviors Treatment Control Measure group group Difference p-valuea Ever had sex (%) 87.4 87.0 0.4 0.88 . . . . Age at first sexual intercourse (%) Younger than 13 years old 32.6 33.0 -0.4 0.99 13 or 14 years old 37.8 37.3 0.5 . 15 years old or older 16.8 16.6 0.2 . Never had sexb 12.8 13.0 -0.2 . . . . . Lifetime number of sexual partners (%) 1–3 21.3 21.8 -0.5 0.41 4–8 22.7 22.3 0.4 . 9–14 23.4 18.8 4.6 . 15 or more 18.8 23.0 -4.2 . Never had sexb 13.8 14.2 -0.4 . . . . . In past three months (%) Had sex 37.7 37.4 0.4 0.91 Had sex without a condom 30.2 29.3 0.9 0.77 Had sex without any method of protection 24.4 24.0 0.4 0.90 Been pregnant or gotten partner pregnant 9.5 7.2 2.3 0.23 Ever been pregnant or gotten partner pregnant (%) 35.0 35.8 -0.8 0.79 . . . . In the past 12 months (%) Tested by doctor or nurse for an STI 56.1 61.2 -5.0 0.13 Told by doctor or nurse had an STI 9.1 8.2 0.9 0.64 . . Sample sizec 436 449 Source: Baseline surveys administered to study participants before the start of the program. a Reported p-values are based on two-tailed t-tests for dichotomous measures and chi-squared tests for categorical measures. b Reported differences in rates of never had sex reflect differences in item nonresponse across measures. c Reported sample size is the number of participants who completed the 12-month survey and are included in the analysis; it does not account for item nonresponse for any measures included in the table. E. Treatment and control conditions Youth assigned to the treatment group were offered the 10-session PTC program in their group homes. Teams of two trained facilitators from OICA, Kern County Superintendent of Schools, or Planned Parenthood of Maryland delivered the program. The facilitators traveled to each group home to deliver the PTC program as a supplement to any other educational programs and services offered in the home. The facilitators worked with program staff in each home to determine the best schedule for delivering the program. Although some group homes chose to have the PTC program delivered once a week for 10 weeks, most completed the program in 5 weeks (two sessions a week). The shorter 5-week schedule helped reduce the chances of youth leaving the home before the end of the program. Offering the program twice a week also helped to build rapport quickly between youth and the program facilitators. All program facilitators received training in the PTC program. OICA provided an initial four-day in-person training for facilitators at the start of the study in spring 2011. Additional in- 16 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM person training sessions held in fall 2012 and fall 2013 reviewed the program materials and discussed facilitators’ real-world experiences in delivering the program. Between these in-person training sessions, OICA convened periodic group discussions and supplemental training sessions via conference call. Facilitators had the opportunity to practice delivering the curriculum during the training sessions and during an initial pilot of the evaluation. They received feedback and technical assistance on their performance from OICA and a designated site coordinator in each state. Our accompanying implementation study of the PTC program (Meckstroth et al. 2014) and program attendance data collected by PTC program staff suggest that the program was well implemented. Most youth in the treatment group (65 percent) attended all 10 program sessions, and 82 percent attended at least 8 sessions (Figure II.2). In addition, a large majority of youth rated the program facilitators positively and reported that the program had been useful. Youth were particularly engaged during the time at the end of each session devoted to answering youth questions. The session length sometimes made it difficult for youth to remain engaged throughout the 90-minute sessions. The facilitators began adding a snack break in the middle of sessions to offset the fatigue. In one site, facilitators noted that some youth, particularly boys, felt uncomfortable engaging in some of the role-playing activities. Such discomfort might have stemmed in part from delivering the program in single-gender settings, which required participants to conduct role plays and other program activities exclusively in same-sex pairs. However, such reports of discomfort were isolated and did not represent the majority experience. The PTC implementation report (Meckstroth et al. 2014) provides a more detailed description of the implementation successes and challenges). Figure II.2. Attendance in PTC sessions Source: Attendance data collected by PTC program staff. Youth assigned to the control group were not offered the PTC program but retained access to any other existing community and group home services available to them. Interviews with program and group home staff in the control group found that youth generally received limited or no sexual and reproductive health education or services. In the few group homes that did provide 17 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM other sources of sex education, these programs and services were typically limited to a single- session class or one-on-one counseling provided to youth on an individual, as-needed basis. The other types of educational programs and services offered to youth varied from home to home but typically included a mix of case management, individual or family therapy, substance abuse counseling and treatment, and various types of education and training programs. In the California and Maryland sites, program and group home staff reported the availability of some outside, community-based services for youth in foster care and other out-of-home care settings. For example, Bakersfield, California, and Baltimore, Maryland, have dedicated resource centers where youth can access family planning and related support services. In all three participating states, public schools are required to provide education on HIV/AIDS but do not have broader sexual health education requirements mandated for all students. See Meckstroth et al. (2014) for a more detailed description of the programs and services available to youth in the control group. At the time of the 12-month follow-up survey, the percentage of study participants still living in a group home varied by age and state (Table II.4). Among younger youth who enrolled in the study before they had turned 17, slightly less than half (48 percent) still lived in a group home at the time of the 12-month follow-up. This percentage varied across states from a low of 38 percent for Maryland to a high of 51 percent for California. Among older youth, the percentages were lower. For the overall sample, 33 percent of the older youth still lived in a group home at the time of the 12-month follow-up. This percentage varied across states from a low of 30 percent for both Maryland and Oklahoma to a high of 36 percent for California. Table II.4. Percentage of youth living in a group home at follow-up Age at baseline California Maryland Oklahoma Total Younger than 17 51 38 49 48 17 or older 36 30 30 33 All ages 44 34 42 41 Source: 12-month follow-up survey administered by the study team. Note: Numbers in the table reflect the percentage of youth who responded yes on the 12-month follow-up survey to the question “Do you currently live in a group home?” These differences in living arrangements at the time of the 12-month follow-up likely reflect the specific characteristics and governing policies of the 44 group homes recruited for the study. For example, states have different policies governing the required length of stay in juvenile justice facilities. These policy differences might partly explain the differences in living arrangements of study participants across states. Similarly, the differences in living arrangements by age group could reflect older youth aging out of the child welfare system as they reach their 18th birthdays. In the child welfare system, recent policy efforts have aimed to extend sources of foster care support beyond age 18 into young adulthood (Napolitano et al. 2015). In practice, however, large numbers of youth leave the child welfare system after reaching age 18. We did not collect information on the specific reasons for youth staying in or leaving the group homes during the study period. We discuss the implications of these differences in living arrangements for our study findings in Chapter IV. 18 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM III. DATA, MEASURES, AND ANALYSIS The impact analysis presented here is based primarily on data from two rounds of surveys completed by youth in the treatment and control groups. An initial baseline survey was administered about one week before homes in the treatment group began the program. A long- term follow-up survey was administered about 12 months later, after completion of the 10- session curriculum. Trained data collectors administered the baseline surveys on site in the group homes as paper-and-pencil questionnaires. The follow-up surveys were administered in groups or individually depending on whether the study participants still resided in participating group homes. The data collectors read questions and possible responses aloud to minimize any problems with reading comprehension or skipped questions. Respondents received incentives of $10 for completing the baseline survey and $50 for completing the 12-month follow-up survey. The rest of this chapter describes the outcome measures constructed from the 12-month follow- up survey data. We then discuss the analytic methods used to assess the impacts of the PTC program on participants’ outcomes. For more detailed information on the measures, see Appendix B. A. Outcome measures Drawing on data from the 12-month follow-up survey, we constructed eight groups of outcome measures, each corresponding to one of the study’s eight research questions: (1) unprotected sex; (2) sexual activity; (3) pregnancy risk; (4) STI risk; (5) youth knowledge of reproductive health, STIs, and methods of protection; (6) youth awareness of available health resources; (7) youth attitudes toward safe sex and methods of protection; and (8) youth self- empowerment or self-efficacy. Table III.1 summarizes these measures and the rest of this section describes them in greater detail. 19 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table III.1. Outcome measures at 12 months Measure Definition Unprotected sex . Had sex without any effective Binary variable: equals 1 if youth reported having had sex in the prior 3 method of protection months without using any effective method of protection; equals 0 if youth reported not having had sex or always using an effective method of protection Had sex without a condom Binary variable: equals 1 if youth reported having had sex in the prior 3 months without using a condom; equals 0 if youth reported not having had sex or always using a condom Sexual activity . Recent sexual activity Binary variable: equals 1 if youth reported having had sex in the prior 3 months; equals 0 if youth reported not having had sex Pregnancy risk . Lifetime pregnancy Binary variable: equals 1 if youth reported having ever been pregnant (females) or gotten someone pregnant (males); equals 0 if youth reported having never been pregnant (females) or gotten someone pregnant (males) Recent pregnancy Binary variable: equals 1 if youth reported having been pregnant (females) or gotten someone pregnant (males) in the prior 3 months; equals 0 if youth reported having not been pregnant (females) or gotten someone pregnant (males) in the prior 3 months STI risk . Multiple sexual partners Binary variable: equals 1 if youth reported having had 2 or more sexual partners in the prior 3 months; equals 0 if youth reported having 0 or 1 partners in the prior three months Tested for STI Binary variable: equals 1 if youth reported being tested for an STI in the prior 12 months; equals 0 if youth reported not being tested Diagnosed with STI Binary variable: equals 1 if youth reported being diagnosed with an STI in the prior 12 months; equals 0 if youth reported not being diagnosed. Knowledge . Knowledge of reproductive anatomy Continuous index variable: sum of correct responses to 4 survey and fertility questions; variable ranges from 0 to 4, with higher values indicating greater knowledge Knowledge of HIV and STIs Continuous index variable: sum of correct responses to 7 survey questions; variable ranges from 0 to 7, with higher values indicating greater knowledge Knowledge of methods of protection Continuous index variable: sum of correct responses to 10 survey questions; variable ranges from 0 to 10, with higher values indicating greater knowledge Awareness of available health . resources Ability to find methods of protection Binary variable: equals 1 if youth reported feeling very sure he or she could find place to obtain methods of protection; equals 0 if youth did not feel very sure Perceived access to condoms Binary variable: equals 1 if youth reported he or she strongly agrees that condoms are pretty easy to get; equals 0 if youth did not strongly agree Perceived access to birth control Binary variable: equals 1 if youth reported he or she strongly agrees that birth control is pretty easy to get; equals 0 if youth did not strongly agree 20 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Measure Definition Attitudes toward safe sex and the . use of protection General support for methods of Continuous scale variable: average of responses to 6 survey questions; protection variable ranges from 1 to 4, with higher values indicating stronger support Perceived barriers to methods of Continuous scale variable: average of responses to 5 survey questions; protection variable ranges from 1 to 4, with higher values indicating fewer perceived barriers Perceived self-efficacy to avoid . unprotected sex Perceived ability to communicate Continuous scale variable: average of responses to 3 survey questions; with partner variable ranges from 1 to 4, with higher values indicating greater perceived ability Perceived ability to plan for and Continuous scale variable: average of responses to 4 survey questions; avoid unprotected sex variable ranges from 1 to 4, with higher values indicating greater perceived ability 1. Unprotected sex As discussed in Chapter I, we designed the study with the primary goal of assessing program impacts on rates of unprotected sexual activity measured 12-months after study enrollment. To achieve this goal, we used data from the 12-month follow-up to construct the following two outcomes: • Had sex without any effective method of protection. To assess the program’s impacts on a general measure of unprotected sexual activity, the 12-month survey asked participants to report the number of times in the past three months they had sex without using any effective method of protection. We used responses to this question to create a binary (yes/no) indicator for whether the participant reported having unprotected sex. The survey defined effective methods of protection as condoms, birth control pills, the shot (Depo Provera), the patch, the ring (NuvaRing), an IUD (Mirena or Paragard), or a hormonal implant (Implanon). Participants who reported abstaining from sexual intercourse in the past three months were retained in the analysis by coding them as protected and combining them with respondents who reported always using an effective contraceptive method. The question was limited to voluntary vaginal intercourse, not involuntary sexual activity or voluntary oral or anal sex. • Had sex without a condom. To assess the program’s impacts on a more narrowly defined measure of unprotected sex, the 12-month survey included a separate question asking participants to report the number of times in the past three months they had sex without a condom. Condom use is a relevant outcome for this study, first, because the program emphasizes condoms as a way to reduce the risk of pregnancy and especially STIs and, second, because the study sample is predominately male, and condoms are the only effective method of protection designed for men (excluding vasectomies). We used responses to this survey question to create a binary (yes/no) indicator for whether the participant reported having sex without a condom. Participants who reported abstaining from sexual intercourse in the past three months were retained in the analysis by coding them as protected and combining them with respondents who reported always using a condom. The question was 21 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM limited to voluntary vaginal intercourse, not involuntary sexual activity or voluntary oral or anal sex. Appendix B provides additional detail on how we constructed these outcomes and the cleaning rules we used to account for item nonresponse and any inconsistencies in responses across survey questions. 2. Sexual activity To supplement our primary analysis of program impacts on rates of unprotected sex, we constructed several other outcomes of interest. To assess the program’s impacts on overall rates of sexual activity, the 12-month survey asked youth to report the total number of times they had sex in the past three months. We used responses to this question to create a binary (yes/no) indicator for whether a participant reported having sexual intercourse in the past three months. The question was limited to voluntary vaginal intercourse, not involuntary sexual activity or voluntary oral or anal sex. 3. Pregnancy risk We used responses from the 12-month survey to construct two separate pregnancy-related outcomes: • Lifetime pregnancy. The survey asked participants if they had ever been pregnant (females) or gotten anyone pregnant (males). We used responses to create a binary (yes/no) indicator for whether the participant reported a lifetime pregnancy. Participants who reported a lifetime pregnancy at the time of the baseline survey, before the start of the program, were retained in the analysis by coding them as having experienced a lifetime pregnancy and combining them with participants who had a first pregnancy during the study period. • Recent pregnancy. As a separate question, the 12-month survey asked participants if they had been pregnant (females) or gotten someone pregnant (males) in the past three months. We used responses to create a binary (yes/no) indicator for whether the participant reported a recent pregnancy. We included this measure of recent pregnancy in addition to the measure of lifetime pregnancy because the measure of recent pregnancy captures both primary and repeat pregnancies. By definition, the measure of lifetime pregnancy captures only primary (not repeat) pregnancies. The main limitation of the recent pregnancy measure is that the effective reference period varies for males and females. For females, the reference period potentially captures (1) any new pregnancies occurring in the past three months; and (2) any earlier pregnancies occurring in the past nine months, for participants still pregnant at the time of the follow-up survey. For males, the reference period captures only new pregnancies caused in the past three months, not any earlier pregnancies. Appendix B provides additional details on how we constructed these outcomes and the cleaning rules we used to account for item nonresponse and any inconsistencies in responses across survey questions. 4. STI risk We constructed three outcomes related to the transmission and incidence of STIs: 22 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM • Multiple sexual partners. The follow-up survey asked youth how many different people they had sex with in the past three months. The possible response categories ranged from “I have not had sex in the past 3 months” to “6 or more people.” We used responses to this question to create a binary (yes/no) indicator for youth who reported having two or more sexual partners in the past three months. The question was limited to voluntary vaginal intercourse, not involuntary sexual activity or voluntary oral or anal sex. • Tested for an STI. The survey asked youth if they had been tested by a doctor or nurse for an STI in the past 12 months. We used responses to create a binary (yes/no) indicator for whether the participant reported having been tested. We hypothesized that program impacts on this outcome could run in either direction. On the one hand, the program might cause more youth to seek testing, resulting in a higher testing rate among youth in the treatment group. On the other hand, the program also aims to reduce the incidence of STIs, which, other things being equal, would result in a lower testing rate. • Diagnosed with an STI. The survey asked youth if they had been told by a doctor or a nurse in the past 12 months that they had an STI. We used responses to create a binary (yes/no) indicator of STI diagnoses. We interpret this outcome as a measure of STI diagnoses rather than a measure of the true incidence of STIs, because some youth may have had STIs that were not diagnosed. The study was not designed to conduct biological STI testing. 5. Knowledge As discussed in Chapter I, we found in our earlier interim report that PTC had favorable short-term effects on measures of youth knowledge, attitudes, and awareness (Goesling et al. 2015). These findings were based on data from the immediate post-test survey administered just after the end of the 10 program sessions. To examine whether these effects persisted over a longer period, we used data from the 12-month follow-up survey to construct the same measures we examined in our earlier report. To measure youth knowledge of reproductive health, STIs, and methods of protection, we constructed three different outcomes: • Knowledge of reproductive anatomy and fertility. The survey asked youth a series of four knowledge questions about reproductive anatomy and fertility. For example, it asked youth to identify the part of the female body where the baby grows during pregnancy, with possible response categories of cervix, uterus, vagina, ovary, and don’t know. We summed the number of correct responses to these four knowledge questions to create an index of knowledge of reproductive anatomy and fertility. The index ranged from 0 to 4, with higher values indicating greater knowledge. • Knowledge of HIV and STIs. In a separate series of questions, the survey asked youth to respond to a series of seven true-or-false questions concerning general knowledge of HIV and STIs. For example, one question read “HIV destroys the immune system’s ability to fight off infections and diseases.” The survey asked youth to respond in one of three categories: true, false, or don’t know. We summed the number of correct responses to these seven questions to create an index of knowledge of HIV and STIs. The index ranged from 0 to 7, with higher values indicating greater knowledge. 23 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM • Knowledge of methods of protection. The survey asked youth to respond to a series of 10 questions concerning their general knowledge of condoms and other methods of protection. For example, one true-or-false question read, “A condom can be used more than once.” We summed the number of correct responses to these 10 questions to create an index of knowledge of methods of protection. The index ranged from 0 to 10, with higher values indicating greater knowledge. 6. Awareness of available health resources To assess the program’s longer-term impacts on youth awareness of available health resources, we used three separate questions from the 12-month follow-up survey. First, the survey asked youth how certain they felt about their ability to perform the following activity: “Find a place in your community to obtain methods of protection from pregnancy and STIs.” The four possible response categories ranged from very sure to very unsure. We used responses to this question to construct a binary measure comparing youth who felt very sure of their ability with youth who felt less certain. Second, the survey asked youth whether they agreed or disagreed with the following statement: “Condoms are pretty easy to get.” The four possible response categories ranged from strongly agree to strongly disagree. We constructed a binary measure comparing youth who said they strongly agreed with this statement to youth who did not strongly agree. Third, the survey asked youth whether they agreed or disagreed with the statement “Birth control is pretty easy to get.” We constructed a binary measure comparing youth who said they strongly agreed with this statement to youth who did not strongly agree. 7. Attitudes toward safe sex and methods of protection To measure youth attitudes toward safe sex and methods of protection, we used data from the 12-month follow-up survey to construct the same two measures of youth attitudes featured in our earlier interim report: • General support for methods of protection. The survey asked youth whether they agreed or disagreed with six statements concerning support for methods of protection, such as “Two people having vaginal intercourse should use some method of protection if they are not ready for a child” and “Condoms should always be used if a person your age has sexual intercourse.” The four possible response categories ranged from strongly agree to strongly disagree. We averaged responses across the six items to create a composite scale of general support for methods of protection. Higher values on the scale indicate stronger levels of support. • Perceived barriers to methods of protection. The survey asked youth whether they agreed or disagreed with five statements concerning possible barriers to using methods of protection, such as “Condoms are a hassle to use” and “Birth control has too many negative side effects.” The four possible response categories ranged from strongly agree to strongly disagree. We averaged responses across the five items to create a composite scale of youth perceived barriers to methods of protection. Higher values on the scale indicate fewer perceived barriers. 24 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM 8. Perceived self-efficacy to avoid unprotected sex We used data from the 12-month follow-up survey to construct two outcomes of perceived self-empowerment or self-efficacy to avoid unprotected sex: • Perceived ability to communicate with partner about sex. The survey asked youth how certain they felt about their ability to perform each of the following three activities: (1) “Tell your partner your feelings about what you do and do not want to do sexually,” (2) “Say no if your partner puts pressure on you to be involved sexually,” and (3) “Talk with your partner about methods of protection if you have sex with him or her.” The four possible response categories ranged from very sure to very unsure. We averaged responses across the three items to create a composite scale of perceived ability to communicate with a partner. Higher values on the scale indicate a greater perceived ability. • Ability to plan for and avoid unprotected sex. The survey asked youth a similar series of four questions about their perceived ability to avoid unprotected sex. In particular, youth were asked how certain they felt about performing activities such as “plan ahead to have some method of protection available.” The four possible response categories ranged from very sure to very unsure. We averaged responses across the four items to create a composite scale of perceived ability to plan for and avoid unprotected sex. Higher values indicate greater perceived ability. B. Analytic approach Three key features of the study design shaped our approach to estimating the impacts of the PTC program on youth outcomes. First, as discussed in Chapter II, the design involved randomly assigning clusters of youth, not individual youth, to the treatment and control groups. This method of random assignment introduces a design effect that must be captured when estimating standard errors and conducting statistical significance tests. Second, in randomly assigning clusters of youth, we grouped them into matched pairs or strata and randomized an equal number of clusters to the treatment and control groups. The analysis must also capture this stratification to appropriately adjust for the method of random assignment. Third, the number of clusters randomly assigned is relatively large (N = 80), allowing for a range of possible analytic approaches. In particular, our sample size exceeds the cutoff of roughly 20 clusters per research arm needed to support a regression-based analysis of program impacts (Hayes and Moulton 2009; Donner and Klar 2000). To incorporate these design features, we used a multivariate regression framework to estimate the impact of the PTC program on each outcome measure. For our main analyses presented in Chapter IV, we estimated an ordinary least squares (OLS) regression model for each outcome. Each regression included a binary variable for treatment status and a series of indicator variables for the matched pairs or strata created for random assignment. To account for the design effect introduced by randomly assigning youth to the treatment and control conditions in groups, we calculated cluster-robust standard errors based on the 80 clusters of youth that we randomly assigned (White 1984; Liang and Zeger 1986). To test the sensitivity of our results to these modeling decisions, we also estimated impacts using the following alternative specifications of the regression model: (1) using a logistic regression model instead of an OLS models for binary outcomes, (2) accounting for the clustered designed using a linear mixed 25 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM effects or multilevel regression model instead of cluster-robust standard errors, and (3) accounting for the cluster designed by conducting a cluster-level analysis instead of an individual-level regression model. We report findings from these sensitivity tests in Appendix C. Each regression model included additional covariates designed to improve the precision of the impact estimates and adjust for any marginal differences in the characteristics of the treatment and control groups. These additional covariates included age, race, gender, and a baseline measure of the outcome variable (when available). We also selected covariates empirically through a data-driven forward selection procedure (described below) designed to identify covariates with strong predictive power and the potential to improve the precision of the impact estimates. For all these covariates, we accounted for missing data using dummy variable adjustment, which involves replacing missing values with an arbitrary constant value and then adding an additional indicator variable to the regression model to statistically adjust for these replaced values (Puma et al. 2009). For the forward selection procedure, we used a data-driven stepwise procedure developed previously in the literature (Social and Character Development Research Consortium 2010). For this procedure, we considered as candidate covariates (1) any baseline variable for which the observed difference between the treatment and control groups had a p-value of 0.20 or less based on a two-sided t-test and (2) other baseline variables that other studies have shown to have a strong link with risky sexual behavior. Appendix B provides a complete list of the covariates considered. From this list of candidate covariates, the forward selection procedure involves gradually adding covariates to the model in order from most to least predictive of the outcome (as defined by the t-statistic on each covariate’s regression coefficient). We conducted the selection procedure separately for each of the sexual risk behavior outcomes. We then compared the selection results across outcomes and identified those covariates meeting either one of two conditions: (1) the covariate was selected by the stepwise procedure for at least 60 percent of the outcomes or (2) the covariate was selected for only one outcome but the observed baseline difference between treatment and control groups in that covariate had a p-value of 0.20 or less. From among the full list of candidate covariates listed in Appendix B, we selected only the covariates meeting these conditions to include in the impact analysis. The forward selection procedure selected the following list of covariates: (1) number of different lifetime sexual partners, (2) ever had sex, and (3) aspire to graduate from college. Appendix C explores the robustness of our results to the use of models that exclude this covariate selection procedure. We adjusted the statistical significance tests (p-values) from our regression models to account for multiple hypothesis testing. As discussed earlier, we constructed multiple outcomes to answer each of our research questions. For example, we constructed two separate outcomes to assess the impacts of the PTC program on rates of unprotected sex. Unless taken into account, this multiplicity can increase the chances of making a false discovery and lead to spurious claims about the program’s effectiveness. We adjusted for multiple hypothesis testing using a procedure outlined by Hothorn et al. (2008) and Schochet (2009). In brief, this procedure involves adjusting the reported p-value for each outcome to account for other tests conducted within the same family of related measures. We began by estimating a separate multivariate regression model for each outcome as planned. To calculate the p-values for the impact estimates, we then compared the estimated t-statistic from each regression model against critical values from a multivariate t distribution determined by the combination of regression models estimated within the same 26 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM family of outcomes. Similar to other methods of adjusting for multiple hypothesis testing, this procedure yields a 5 percent false positive rate across all outcomes within the same family. However, the procedure is less restrictive than other common adjustment methods (for example, the Bonferroni correction) because it also accounts for any correlation in test statistics across outcomes within the same family. We made this adjustment separately for each of the eight groups of outcome measures described earlier in this chapter (and presented in Table III.1). That is, we adjusted the p-values accounting for multiple outcomes within each of the eight groups of measures, but not for multiple outcomes measured across the different groups. We followed this approach because each group of outcomes aligns with a different research question. We base our substantive conclusions for each question only on the corresponding group of outcome measures. The number of outcomes measured in other groups has no bearing on our substantive conclusions for each question and therefore does not warrant an additional adjustment for multiple hypothesis testing. To examine the robustness of our results to this adjustment procedure, we present p- values without an adjustment for multiple hypothesis testing in Appendix C. We designed the study with the goal of estimating impacts for the full sample of youth enrolled in the study. As such, we conducted our analysis using data for all study participants who responded to the 12-month survey. However, as described in Chapter I, to provide a comprehensive assessment of the program, we also estimated program impacts separately for subgroups of youth defined by gender and age group. We conducted these analyses by estimating a separate regression model for each subgroup. For gender, we divided the sample into groups of males (N = 691) and females (N = 194). Because we had stratified the group homes by the gender of youth served for the purpose of random assignment (see Chapter II), dividing the sample by gender involved dropping 57 male only clusters from the female subgroup analysis and 19 female only clusters from the male subgroup analysis. For age group, we organized the sample into groups for (1) youth younger than 17 at the time of the baseline survey (N = 507) and (2) youth ages 17 and older at baseline (N = 378). Although we did not stratify group homes by age group for the purpose of random assignment, we found in the data that a relatively small number of clusters did not include any younger (or older) youth. For the subgroup analysis of youth younger than 17, we dropped two clusters from the analysis that did not have any youth in that age group. For the subgroup analysis of youth ages 17 and older, we dropped seven clusters from the analysis that did not have any youth in the age group. To focus on the outcomes most central to the program, and to limit the possibility of falsely detecting a spurious impact of the program, we report the findings of our subgroup analyses only for the measures of sexual risk behavior and reproductive health outcomes, not for the outcomes related to youth knowledge, attitudes, or awareness. For each of the four subgroups examined (males, females, younger youth, and older youth), we found that the treatment and control groups were comparable on baseline characteristics. 27 This page has been left blank for double-sided copying. FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM IV. RESULTS Our findings for the overall study sample varied by outcome. We found a statistically significant difference between the treatment and control groups in the percentage of youth who reported ever having been pregnant or gotten someone pregnant. In particular, the percentage of youth who reported a lifetime pregnancy was 5.1 percentage points lower for the treatment group than for the control group (50.8 versus 55.9 percent). We also found favorable 12-month impacts on the intermediate outcomes such as knowledge, awareness of available health resources, attitudes, and perceived self-efficacy. In contrast, we found little difference between the treatment and control groups for our primary measures of unprotected sex. For the overall sample, youth in the treatment group were just as likely as those in the control group to report both having had sex without any effective method of protection and having had sex without a condom. The two groups were also similar in the percentage of youth who reported having had sex in the past three months, having been pregnant or gotten someone pregnant in the past three months, having had multiple sexual partners, and being tested for or diagnosed with an STI in the past 12 months. Our exploratory subgroup analyses help explain these differences in results across outcomes. For the subgroup analysis by age group, we found that the favorable program impact on pregnancy rates was driven largely by the older subgroup of youth who were ages 17 or older at baseline. Among this older subgroup of youth, we found that the percentage of youth who reported a lifetime pregnancy was 8.6 percentage points lower in the treatment group than the control group. For this older subgroup, we also found favorable impacts of the program on rates of sexual activity and unprotected sex. In particular, among youth who were ages 17 or older at baseline, we found that youth in the treatment group were less likely than those in the control group to report having had sex in the past three months and to report having had sex without any effective method of protection. For the younger subgroup of youth who were younger than 17 at baseline, we found no evidence of an impact of the program for any behavioral outcome. These results thus show a more consistent pattern across outcomes, but only when looking separately by age group. We detail these findings in the rest of this chapter. A. Unprotected sexual activity For the overall sample, we found no difference between the treatment and control groups for our primary measures of unprotected sex (Table IV.1). In both groups, slightly more than a third of the study participants reported having had sex in the past three months without using any effective method of protection (35.1 percent for the treatment group versus 35.6 percent for the control group). Nearly half of the participants in both groups reported having had sex in the past three months without using a condom (47.0 percent for the treatment group and 45.5 percent for the control group). For both outcomes, the estimated difference between groups was small and not statistically significant (-0.5 percentage points for sex without any effective method of protection and 1.5 percentage points for sex without a condom). 29 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table IV.1. Impacts on unprotected sexual activity Treatment Control Measure group group Difference p-value Percentage of youth who reported the following in the . . . . past 3 months: Had sex without using any effective method of 35.1 35.6 -0.5 >0.99 protection Had sex without using a condom 47.0 45.5 1.5 >0.99 Source: Youth surveys administered by the study team. Note: For each outcome, the numbers in the columns labeled “Treatment group” and “Control group” are regression-adjusted predicted values of outcomes at the 12-month follow-up survey. Each regression model included the following covariates: age, race, gender, treatment status, indicator variables for the matched pairs or strata created for random assignment, number of sexual partners, ever had sex, aspires to graduate from college, and a baseline measure of the outcome (when available). The sample size accounting for item nonresponse is 826 for both measures. Reported p-values are adjusted for multiple outcomes measured within a single domain. See Chapter III for a more detailed description of the analytic methods. B. Other sexual risk behaviors and reproductive health outcomes For the overall sample, we found a statistically significant difference in the percentage of youth who reported having ever been pregnant or gotten someone pregnant (Table IV.2). In particular, we found that the percentage of youth reporting a lifetime pregnancy was 5.1 percentage points lower for youth in the treatment group than for those in the control group (50.8 percent for the treatment group versus 55.9 percent for the control group). We found no difference between the treatment and control groups for the other sexual risk behaviors and reproductive health outcomes examined (Table IV.2). Youth in the treatment group were just as likely as those in the control group to report having had sex in the 3 months before completing the 12-month follow-up (59.9 percent for the treatment group and 60.4 percent for the control group). The two groups were also similar in the percentage of youth who reported having been pregnant or gotten someone pregnant in the past 3 months (15.4 percent for the treatment group and 16.0 percent for the control group), the percentage of youth who reported having had multiple sexual partners (34.1 percent for the treatment group and 34.2 percent for the control group), and the percentage of youth who reported being tested for or diagnosed with an STI in the past 12 months (for testing, percentages of 60.7 for the treatment group and 58.4 for the control group; for diagnoses; percentages of 11.0 for the treatment group and 9.7 for the control group). 30 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table IV.2. Impacts on sexual activity, pregnancy risk, and STI risk Treatment Control Measure group group Difference p-value Percentage of youth who reported having had sex in the past 59.9 60.4 -0.5 0.83 3 months Percentage of youth who reported: . . . . Ever been pregnant or gotten someone pregnant 50.8 55.9 -5.1** <0.01 Been pregnant or gotten someone pregnant in the past 3 15.4 16.0 -0.6 >0.99 months Percentage of youth who reported: . . . . Had multiple sexual partners in the past 3 months 34.1 34.2 -0.1 >0.99 Tested for an STI in the past 12 months 60.7 58.4 2.3 0.97 Diagnosed with an STI in the past 12 months 11.0 9.7 1.3 >0.99 Source: Youth surveys administered by the study team. Note: For each outcome, the numbers in the columns labeled “Treatment group” and “Control group” are regression-adjusted predicted values of outcomes at the 12-month follow-up survey. Each regression model included the following covariates: age, race, gender, treatment status, indicator variables for the matched pairs or strata created for random assignment, number of sexual partners, ever had sex, aspires to graduate from college, and a baseline measure of the outcome (when available). Sample sizes accounting for item nonresponse range from 821 to 881 depending on the measure. Reported p-values are adjusted for multiple outcomes measured within a single domain. See Chapter III for a more detailed description of the analytic methods. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. C. Intermediate outcomes For the overall sample, our results indicate that the PTC program had favorable 12-month effects on most of the intermediate outcomes examined (Table IV.3). For all three knowledge measures, youth in the treatment group had a higher average score than youth in the control group. Youth in the treatment group were also more likely than those in the control group to report feeling very sure of where to get methods of protection (67.3 percent for the treatment group versus 59.2 percent for the control group). For the measure of general support for methods of protection, we found that youth in the treatment group reported a higher average level of support than youth in the control group (an average score of 3.40 for the treatment group versus 3.32 for the control group). In addition, youth in the treatment group reported higher averages scores on both the scale of perceived ability to communicate with a partner (an average score of 3.58 for the treatment group versus 3.42 for the control group) and the scale of self-efficacy or self-empowerment to avoid unprotected sex (an average score of 3.41 for the treatment group and 3.28 for the control group). Our impact estimates for the three other intermediate outcomes examined were not statistically significant. In particular, we found no statistically significant 12-month impacts on the percentage of youth who strongly agreed that condoms are easy to get (56.5 percent for the treatment group versus 53.4 percent for the control group) or the percentage of youth who strongly agreed that birth control is easy to get (30.9 percent for the treatment group versus 30.2 percent for the control group). Our impact estimate for the scale of perceived barriers to methods of protection was also small and not statistically significant (an average score of 2.53 for the treatment group and 2.49 for the control group). 31 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table IV.3. Impacts on intermediate outcomes at 12 months Treatment Control Measure group group Difference p-value Knowledge of reproductive anatomy and fertility (index score, 2.59 2.33 0.26** <0.01 ranges from 0 to 4)a Knowledge of HIV and STIs (index score, ranges from 0 to 7)b 4.86 4.43 0.43** <0.01 Knowledge of methods of protection (index score, ranges from 7.25 6.04 1.21** <0.01 0 to 10)c Percentage of youth reporting they are very sure where to get 67.3 59.2 8.1** <0.01 methods of protection Percentage of youth reporting they strongly agree that condoms 56.5 53.4 3.1 0.63 are “easy to get” Percentage of youth reporting they strongly agree that birth 30.9 30.2 0.7 >0.99 control is “easy to get” General support for methods of protection (index score, ranges 3.40 3.32 0.09** <0.01 from 1 to 4)d Perceived barriers to methods of protection (index score, ranges 2.53 2.49 0.03 0.28 from 1 to 4)e Perceived ability to communicate with partner (scale score, 3.58 3.42 0.16** <0.01 ranges from 1 to 4)f Perceived ability to plan for and avoid unprotected sex (scale 3.41 3.28 0.13** <0.01 score, ranges from 1 to 4)g Source: Youth surveys administered by the study team. Note: For each outcome, the numbers in the columns labeled “Treatment group” and “Control group” are regression-adjusted predicted values of outcomes at the 12-month follow-up survey. Each regression model included the following covariates: age, race, gender, treatment status, indicator variables for the matched pairs or strata created for random assignment, number of sexual partners, ever had sex, aspires to graduate from college, and a baseline measure of the outcome (when available). Sample sizes accounting for item nonresponse range from 833 to 885 depending on the measure. Reported p-values are adjusted for multiple outcomes measured within a single domain. See Chapter III for a more detailed description of the analytic methods. a This index counts the number of correct responses to a series of four knowledge questions. Possible values range from 0 to 4 with higher values indicating a greater number of correct responses. b This index counts the number of correct responses to a series of seven knowledge questions. Possible values range from 0 to 7 with higher values indicating a greater number of correct responses. c This index counts the number of correct responses to a series of 10 knowledge questions. Possible values range from 0 to 10 with higher values indicating a greater number of correct responses. d This scale averages responses to six questions on support for methods of protection. Possible values range from 1 to 4 with higher values indicating greater support. e This scale averages responses to five questions on perceived barriers to methods of protection. Possible values range from 1 to 4 with higher values indicating fewer perceived barriers. f This scale averages responses to three questions on perceived ability to communicate with a partner. Possible values range from 1 to 4 with higher values indicating greater perceived ability. g This scale averages responses to four questions on perceived ability to plan for and avoid unprotected sex. Possible values range from 1 to 4 with high values indicating greater perceived ability. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. For all of the intermediate outcomes examined, we found that the estimated impacts were smaller at the time of the 12-month follow-up than those found in our earlier interim report based on the immediate post-test survey. For example, our earlier interim report found an impact of 14.9 percentage points for the measure of the percentage of youth who reported feeling very sure 32 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM of where to get methods of protection (Goesling et al. 2015). For the 12-month follow-up, our findings show that the estimated impact had declined to 8.1 percentage points (Table IV.3). We found similar declines in the estimated impacts for all of the intermediate outcomes examined. D. Subgroup analysis Our subgroup analysis by age group found different patterns of results for older versus younger youth (Table IV.4). For the older subgroup of youth who were ages 17 or older at the time of the baseline survey, we found a consistent pattern of favorable program impacts for most of the behavioral outcomes examined. For this older subgroup, youth in the treatment group were less likely than those in the control group to report having had sex without using any effective method of protection (31.4 versus 45.3 percent). They also reported lower rates of sexual activity (57.9 versus 66.5 percent), were less likely to report having ever been pregnant or gotten someone pregnant (53.4 versus 62.0 percent), and were less likely to report having been pregnant or gotten someone pregnant in the past three months (15.2 versus 21.0 percent). By contrast, for the subgroup of youth who were younger than 17 at baseline, we found only small differences between the treatment and control groups for all of the measures of unprotected sex, sexual activity, and pregnancy. For this younger subgroup, youth in the treatment group were just as likely than those in the control group to report having had sex, having had unprotected sex, or having experienced a recent or lifetime pregnancy. We found no program impacts on the measures of STI testing or diagnoses for either subgroup. For our subgroup analysis by gender (Table IV.5), we found that the pattern of results for both males and females resembled the pattern for the overall study sample. For our primary outcome measures of unprotected sex, males in the treatment group were just as likely as those in the control group to report having had sex without any effective method of protection (38.3 percent for males in the treatment group and 37.4 percent for males in the control group) and having had sex without a condom (48.9 percent of males in the treatment group and 47.2 percent of males in the control group). For females, youth in the treatment group were somewhat less likely than those in the control group to report having had sex without any method of protection (24.2 percent of females in the treatment group versus 28.6 percent of females in the control group) and to report having had sex in the past three months (46.6 percent of females in the treatment group and 52.0 of females in the control group). However, these differences were not statistically significant at the 5-percent level. Both males and females in the treatment group were less likely than those in the control group to report ever having been pregnant (females) or gotten someone pregnant (males), though only the impact for females (48.8 percent of the treatment group versus 59.2 of the control group) was statistically significant at the 5 percent level. 33 Table IV.4. Subgroup impacts, by age group . Youth younger than 17 at baseline Youth ages 17 or older at baseline Treatment Control Treatment Control Measure group group Difference p-value group group Difference p-value Percentage of youth who reported the . . . . . . . . following in the past 3 months: Had sex without using any effective 34.2 31.9 2.3 >0.99 31.4 45.3 -13.9** <0.01 method of protection Had sex without using a condom 45.4 42.4 3.0 0.88 44.9 53.9 -9.0 0.06 Percentage of youth who reported having 59.4 57.8 1.6 0.71 57.9 66.5 -8.6** 0.01 had sex in the past 3 months Percentage of youth who reported: . . . . . . . . Ever been pregnant or gotten someone 49.2 51.0 -1.8 0.96 53.4 62.0 -8.6** <0.01 pregnant Been pregnant or gotten someone 14.1 13.7 -0.4 >0.99 15.2 21.0 -5.9* 0.04 pregnant in the past 3 months Percentage of youth who reported: . . . . . . . . 34 Had multiple sexual partners in the past 31.9 33.1 -1.2 >0.99 34.2 38.8 -4.6 0.63 3 months Tested for an STI in the past 12 months 57.9 57.9 0.0 >0.99 63.1 60.4 2.7 >0.99 Diagnosed with an STI in the past 12 10.6 11.8 -1.2 >0.99 10.8 7.6 3.2 0.86 months Source: Youth surveys administered by the study team. Note: For each outcome, the numbers in the columns labeled “Treatment group” and “Control group” are regression-adjusted predicted values of outcomes at the 12-month follow-up survey. Each regression model includes the following covariates: race, gender, treatment status, indicator variables for the matched pairs or strata created for random assignment, number of sexual partners, ever had sex, aspires to graduate from college, and a baseline measure of the outcome (when available). Sample sizes accounting for item nonresponse range from 471 to 503 for youth younger than 17 at baseline and from 350 to 378 for youth 17 or older at baseline, depending on the measure. Reported p-values are adjusted for multiple outcomes measured within a single domain. See Chapter III for a more detailed description of the analytic methods. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. Table IV.5. Subgroup impacts, by gender . Males Females Treatment Control Treatment Control Measure group group Difference p-value group group Difference p-value Percentage of youth who reported the . . . . . . . . following in the past 3 months: Had sex without using any effective method 38.3 37.4 0.9 >0.99 24.2 28.6 -4.4 0.51 of protection Had sex without using a condom 48.9 47.2 1.7 >0.99 40.5 39.2 1.3 >0.99 Percentage of youth who reported having had 63.2 63.2 0.0 >0.99 46.6 52.0 -5.4 0.26 sex in the past 3 months Percentage of youth who reported: . . . . . . . . Ever been pregnant or gotten someone 51.3 54.9 -3.5 0.13 48.8 59.2 -10.4** <0.01 pregnant Been pregnant or gotten someone pregnant 15.5 17.3 -1.9 0.34 15.9 10.6 5.3 0.41 in the past 3 months Percentage of youth who reported: . . . . . . . . Had multiple sexual partners in the past 3 38.5 38.0 0.5 >0.99 17.8 21.8 -4.0 >0.99 35 months Tested for an STI in the past 12 months 59.0 54.4 4.6 0.21 66.4 72.9 -6.5 0.58 Diagnosed with an STI in the past 12 months 9.6 8.9 0.7 >0.99 17.0 11.9 5.1 0.30 Source: Youth surveys administered by the study team. Note: For each outcome, the numbers in the columns labeled “Treatment group” and “Control group” are regression-adjusted predicted values of outcomes at the 12-month follow-up survey. Each regression model includes the following covariates: age, race, treatment status, indicator variables for the matched pairs or strata created for random assignment, number of sexual partners, ever had sex, aspires to graduate from college, and a baseline measure of the outcome (when available). Sample sizes accounting for item nonresponse range from 181 to 192 for females and from 639 to 689 for males, depending on the measure. Reported p-values are adjusted for multiple outcomes measured within a single domain. See Chapter III for a more detailed description of the analytic methods. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM V. DISCUSSION This report presents the final 12-month impact findings from a large-scale demonstration project and evaluation of PTC, a comprehensive sexual health education curriculum designed specifically for youth in foster care and other out-of-home care settings. In an earlier report based on data collected soon after youth had received the 10 program sessions, we found several favorable short-term effects of the program on measures of youth knowledge, attitudes, perceived self-efficacy, and intentions (Goesling et al. 2015). For the present report, we examined whether and how these short-term effects on intermediate outcomes led to longer-term effects on behavioral outcomes measured 12 months after the PTC program ended for youth in the treatment group. For our primary research question, we examined whether the PTC program succeeded in reducing rates of unprotected sex among study youth. For additional secondary analyses, we also examined program impacts on other sexual risk behaviors and reproductive health outcomes of interest, as well as potential differences in program impacts by gender and age group. Our findings suggest that the 12-month impacts of the PTC program varied by age group. For the older youth ages 17 or older at the time of the baseline study survey, we found a consistent pattern of favorable program impacts across most of the behavioral outcomes examined. For these older youth, we found that youth in the treatment group reported lower rates of sexual activity and unprotected sex relative to the control group. Youth in the treatment group were also less likely than those in the control group to report having ever been pregnant (for females) or gotten someone pregnant (for males). For the younger subgroup of youth younger than 17 at the time of the baseline study survey, we found a different pattern of results. For this younger subgroup, differences in most of the behavioral outcomes examined were small and not statistically significant. We found no measurable program impacts on our primary measures of unprotected sex or any of the other sexual risk behaviors or reproductive health outcomes examined. In part because of these differences by age group, our impact findings for the overall study varied across outcomes. For one of our two measures of pregnancy risk, we found a large, statistically significant difference in the percentage of youth who reported having ever been pregnant or gotten someone pregnant. In particular, we found that the percentage of youth who reported a lifetime pregnancy was 5.1 percentage points lower in the treatment group than in the control group. We also found favorable 12-month impacts of the program on measures of youth knowledge, awareness of available health resources, attitudes, and perceived self-efficacy. However, for our primary measures of unprotected sexual activity, we found no differences in rates between the treatment and control groups. Youth in the treatment group were just as likely as those in the control group to report having had sex without any effective method of protection and having had sex without a condom. In addition, we found that overall rates of sexual activity were similar for youth in the treatment and control groups. On the basis of these findings, we considered the possibility that the PTC program is more effective for older than younger youth, but we found little corroborating evidence to support this claim. As described in Chapter I, the PTC curriculum materials emphasize the dual themes of self-empowerment and the impact of choices. It is possible that these themes resonate more strongly with older youth—for example, because their age gives them a greater sense of 36 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM autonomy or because they are closer to adulthood and the point of aging out of the foster care or juvenile systems. To investigate this possible explanation, we conducted an additional post hoc exploratory analysis examining the 12-month impacts of the PTC program on our two scales of perceived self-efficacy (described in Chapter III) separately by age group. From this analysis, we found that the magnitude of the program impact on these measures of perceived self-efficacy was larger (not smaller) for the younger study participants. For example, on our 4-point scale of perceived ability to communicate with a partner about sex, we found a favorable impact of 0.20 scale points for the subgroup of youth younger than 17 at baseline compared with a smaller impact of 0.07 scale points for the subgroup of older youth. Similarly, on our 4-point scale of perceived ability to plan for and avoid unprotected sex, we found a favorable impact of 0.21 scale points for the younger subgroup compared with an impact of -0.01 scale points for the older subgroup. We thus found no evidence to support the hypothesis that the program resonated more strongly with older youth. Instead, we believe a more likely explanation for our study findings involves differences in the living arrangements of study participants. As described in greater detailed in Chapter II, slightly more than 40 percent of the study participants still lived in a group home at the time of the 12-month follow-up survey. However, due to the characteristics and governing policies of the group homes recruited for the study, the living arrangements of study participants varied by age group. Among the subgroup of youth younger than 17 at the time of the baseline survey, a relatively higher percentage of youth (48 percent) still lived in a group home at the time of the 12-month follow-up survey. Among the subgroup of youth ages 17 or older, the percentage was lower (33 percent). As a result, it is possible that the younger study participants had relatively less opportunity to engage in the types of sexual risk behaviors the PTC program aims to discourage and the protective behaviors it aims to promote. If so, differences in living arrangements between older and younger youth could partly or fully explain the lack of impacts on behavioral outcomes for the younger subgroup. To investigate this possible explanation, we conducted an additional post hoc exploratory analysis of differences in rates of sexual risk behaviors by living arrangements. We conducted this additional post hoc analysis using 12-month survey data only for participants in the control group, to account for any possible correlation between living arrangements and random assignment status (treatment or control). On the basis of this analysis, we found that rates of sexual activity and unprotected sex were indeed lower among youth who still lived in a group home at the time of the 12-month survey (regardless of age group). For example, among youth who still lived in a group home at the time of the 12-month survey, 45 percent reported having had sex in the past 3 months, compared with 67 percent of youth who did not live in a group home at the time of the survey. Similarly, youth still living in a group home were less likely to report having had sex without a condom in the past 3 months compared with youth not living in a group home (30 versus 52 percent). These findings offer some support for the hypothesis that differences in living arrangements might partly or fully explain the different pattern of program impacts we report for younger versus older youth. Finally, as first noted in our interim report, we caution that the findings from this study might not generalize to all parts of the country or to all youth living in foster care or other out-of- home care settings. The study focused on a predominately male, high-risk sample of teens ages 13 to 18 living in residential group homes. Upon enrolling in the study, nearly 9 in 10 37 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM participants reported some lifetime experience with sexual intercourse, and more than one-third of the sample reported having been pregnant or gotten a partner pregnant at some point in their lives. In addition, we recruited the participating group homes from a select group of three states (California, Maryland, and Oklahoma). By design, the homes were not drawn as a representative sample but instead on the basis of their capacity and commitment to support the program and study activities. For all of these reasons, it is possible that other organizations might achieve different results when implementing the program on a different scale, with a different target population, or in a different setting. Nonetheless, our findings establish the PTC program as one of the first and only curricula for youth in out-of-home care with demonstrated evidence of effectiveness in reducing teen pregnancy and associated sexual risk behaviors. 38 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM REFERENCES Becker, M.G., and R.P. Barth. “POWER Through Choices: The Development of a Sexuality Education Curriculum for Youths in Out-of-Home Care.” Child Welfare, vol. 79, no. 3, 2000, pp. 269–282. Cavazos-Rehg, P.A., M.J. Krauss, E.L. Spitznagel, M. Schootman, K.K. Bucholz, J.F. Peipert, V. Sanders-Thompson, L.B. Cottler, and L.J. Bierut. “Age of Sexual Debut Among US Adolescents.” Contraception, vol. 80, no. 2, 2009, pp. 158–162. Crottogini, J., J. Fajardo, E. Villasenor, and K. Ward. “Understanding the Sexual Health Needs of Bay Area Foster and Former Foster Care Youth: A Collaborative Project Between Health Initiatives for Youth (HIFY) and San Francisco State University.” San Francisco, CA: San Francisco State University, 2008. Donner, A., and N. Klar. Design and Analysis of Cluster Randomization Trials in Health Research. London: Arnold, 2000. Dworsky, A., and M. E. Courtney. “The Risk of Teenage Pregnancy Among Transitioning Foster Youth: Implications for Extending State Care Beyond Age 18.” Children and Youth Services Review, vol. 32, no. 10, 2010, pp. 1351–1356. Freundlich, M. Time Running Out: Teens in Foster Care. New York: Children’s Rights, 2003. Goesling, B., R.D., Covington, J. Manlove, M. Barry, R.F. Oman, and S. Vesely. “Interim Impacts of the POWER Through Choices Program.” Princeton, NJ: Mathematica Policy Research, August 2015. Hayes, R.J., and L.H. Moulton. Cluster Randomized Trials. Chapman & Hall/CRC Interdisciplinary Statistics Series. Boca Raton, FL: CRC Press, 2009. Hothorn, T., F. Bretz, and P. Westfall. “Simultaneous Inference in General Parametric Models.” Biometrical Journal, vol. 50, no. 3, 2008, pp. 346–363. Hudson, A.L. “Where Do Youth in Foster Care Receive Information About Preventing Unplanned Pregnancy and Sexually Transmitted Infections?” Journal of Pediatric Nursing, vol. 27, no. 5, 2012, pp. 443–450. Liang, Kung-Yee, and Scott L. Zeger. “Longitudinal Data Analysis Using Generalized Linear Models.” Biometrika, vol. 73, no. 1, 1986, pp. 13–22. Meckstroth, A., M. Barry, B. Keating, E. Kisker, and K. Andrews. “Addressing Teen Pregnancy Risks for Youth Living in Out-of-Home Care: Implementing POWER Through Choices 2010.” Washington, DC: Office of Adolescent Health, U.S. Department of Health and Human Services, 2014. Napolitano, L., Y. Sulimani-Aidan, and M.E. Courtney. “Extended Foster Care in California: Youth and Caseworker Perspectives.” Chicago: Chapin Hall at the University of Chicago, 2015. 39 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Oman, R.F., S.K. Vesely, C.B. Aspy, J. Fluhr, L. Marshall, G. Calhoun, and S. Rodine. “Methodological Considerations in a Community-Based Longitudinal Study.” American Journal of Health Behavior, vol. 33, no. 1, 2009, pp. 58–68. Puma, M.J., R.B. Olsen, S.H. Bell, and C. Price. “What to Do When Data Are Missing in Group Randomized Controlled Trials.” NCEE 2009-0049. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education, 2009. Schochet, P.Z. “An Approach for Addressing the Multiple Testing Problem in Social Policy Impact Evaluations.” Evaluation Review, vol. 33, no. 6, 2009, pp. 539–567. Social and Character Development Research Consortium. “Efficacy of Schoolwide Programs to Promote Social and Character Development and Reduce Problem Behavior in Elementary School Children.” NCER 2011–2001. Washington, DC: National Center for Education Research, Institute of Education Sciences, U.S. Department of Education, 2010. White, H. Asymptotic Theory for Econometricians. Orlando, FL: Academic Press, 1984. 40 APPENDIX A NONRESPONSE ANALYSIS This page has been left blank for double-sided copying. FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM This appendix examines the characteristics of the study participants lost to follow-up at the time of the 12-month follow-up survey. As reported in Chapter II of this evaluation of POWER Through Choices (PTC), among the 1,036 youth who enrolled in the study and were randomly assigned to the treatment and control groups, 885 completed the 12-month follow-up survey, for an overall response rate of 85 percent. The remaining 151 participants did not complete the 12- month follow-up survey and the study team therefore excluded them from the impact analyses presented in this report. To better understand the characteristics of the study participants lost to follow-up, we used data from the baseline survey to compare the samples of follow-up survey respondents (N = 885) and nonrespondents (N = 151). The characteristics of the survey nonrespondents were generally similar to those of the participants who responded to the survey. The two groups had similar average ages, had similar racial and ethnic backgrounds, were equally likely to be behind a grade level, and had similar educational aspirations (Table A.1). The two groups were also similar on baseline measures of sexual risk behaviors (Table A.3), with one exception: survey nonrespondents had higher lifetime rates of sexual activity at the time of the baseline survey (94.7 for the survey nonrespondents versus 87.2 for those who responded to the 12-month survey). This difference in lifetime rates of sexual activity also accounts for the statistically significant difference we found when examining age at first sexual intercourse and number of lifetime sexual partners. For both measures, nonrespondents were more likely than youth who responded to the survey to report ever having had sex. This observed difference in lifetime rates of sexual activity suggests that the nonrespondents might have engaged in more risky sexual behavior before enrolling in the study relative to the overall study sample. This difference could affect the external validity or generalizability of our study findings—meaning that our results might not necessarily generalize to youth with the highest reported lifetime rates of sexual activity. However, none of the observed differences present a threat to the internal validity of our impact estimates. As shown in Chapter II, all measures of sexual risk behaviors were similar for survey respondents in the treatment and control groups. A-3 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table A.1. Baseline demographic and personal characteristics Measure Respondents Nonrespondents Difference p-valuea . . . . Age in years (%) Younger than 15 years old 12.7 7.3 5.4 0.18 15 years old 16.6 13.9 2.7 16 years old 28.0 27.2 -0.8 17 years old 32.7 38.4 -5.7 18 years old or older 10.1 13.3 -3.2 Male (%) 78.1 82.1 -4.0 0.26 . . . . Race/ethnicity (%) Hispanic 37.4 33.8 1.4 0.17 Non-Hispanic black 19.3 24.5 -5.2 Non-Hispanic white 19.8 25.2 -5.4 Non-Hispanic Native American 3.9 3.3 0.6 Non-Hispanic Asian or Pacific Islander 1.6 2.0 -0.4 Non-Hispanic otherb 18.1 11.3 6.8 . . . . Months in current group home (%) Fewer than 3 months 43.0 47.6 -4.6 0.59 3 to 6 months 36.4 34.3 2.1 More than 6 months 20.6 18.2 2.4 Behind grade level (%) 42.3 43.0 -0.7 0.87 . . . . Highest level of education likely to complete (%) Less than high school 27.3 29.8 -2.5 0.87 Graduate from high school 19.0 18.5 0.5 Some college or technical training 15.2 15.2 0.0 Graduate from a 2-year college 8.4 6.0 2.4 Graduate from a 4-year college 30.2 30.5 -0.3 Sample sizec 885 151 . . Source: Baseline surveys administered to study participants before the start of the program. a Reported p-values are based on two-tailed t-tests for dichotomous measures and chi-squared tests for categorical measures. b This category includes respondents who selected multiple races. c Reported sample size does not account for item nonresponse for any measures included in the table. A-4 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table A.2. Baseline sexual risk behaviors Measure Respondents Nonrespondents Difference p-valuea Ever had sex (%) 87.2 94.7 -7.5 0.01 . . . . Age at first sexual intercourse (%) Younger than13 years old 32.8 33.3 -0.5 0.05 13 or 14 years old 37.5 40.0 -2.5 . 15 years old or older 16.7 21.3 -4.6 . Never had sexb 12.9 5.3 7.6 . . . . . Lifetime number of sexual partners (%) 1 to 3 21.5 17.6 3.9 0.03 4 to 8 22.5 32.4 -9.9 . 9 to 14 21.1 22.5 -1.4 . 15 or more 21.0 20.4 0.6 . Never had sexb 14.0 7.0 7.0 . . . . . In past three months: Had sex without condom (%) 29.7 36.0 -6.2 0.14 Had sex without any method of protection (%) 24.2 26.1 -1.9 0.63 Ever been pregnant or gotten partner pregnant -0.4 (%) 35.4 35.8 0.92 Been pregnant or gotten partner pregnant in the past 3 months (%) 8.3 8.7 -0.4 0.87 . . . . In the past 12 months (%) Tested by doctor or nurse for an STI 58.4 57.7 0.7 0.87 Told by doctor or nurse had an STI 9.2 6.8 2.4 0.35 Sample sizec 885 151 . . Source: Baseline surveys administered to study participants before the start of the program. a Reported p-values are based on two-tailed t-tests for dichotomous measures and chi-squared tests for categorical measures. b Reported differences in rates of never had sex reflect differences in item nonresponse across measures. b Reported sample size does not account for item nonresponse for any measures included in the table. STI = sexually transmitted infection. A-5 This page has been left blank for double-sided copying. APPENDIX B DATA AND MEASURES This page has been left blank for double-sided copying. FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM This appendix provides more detailed information on the measures used to assess the 12- month impacts of the POWER Through Choices (PTC) program. We begin by providing a more detailed description of how we constructed the outcome measures. We then list the baseline measures considered as candidate covariates for the regression models. A. Outcome measures As discussed in Chapter III, our impact analysis focused on eight different groups of outcomes, each corresponding to one of the study’s research questions: (1) unprotected sex, (2) sexual activity, (3) pregnancy risk, (4) sexually transmitted infection (STI) risk, (5) knowledge, (6) awareness of available resources, (7) attitudes toward safe sex and the use of protection, and (8) perceived self-efficacy to avoid unprotected sex. Our interim report (Goesling et al. 2015) provides a detailed description of the measures of knowledge, awareness, attitudes, and self- efficacy. In this section, we describe how we constructed the behavioral outcomes of interest: unprotected sex, sexual activity, pregnancy risk, and STI risk. 1. Unprotected sex To assess the program’s impacts on rates of unprotected sexual activity measured 12 months after study enrollment, we constructed two separate measures of unprotected sexual activity: (1) a binary indicator (yes/no) of whether the respondent reported having had sex in the past three months without using any effective method of protection, and (2) a binary indicator (yes/no) of whether the respondent reported having had sex in the past three months without using a condom. We constructed these variables in a stepwise fashion from the following series of questions included on the survey: • Now please think about the past three months. In the past three months, how many times have you had sexual intercourse? • In the past three months, how many times have you had sex without you or your partner using a condom? • The next question is about your use of the following methods of birth control: condoms, birth control pills, the shot, the patch, the ring, intrauterine device (IUD), and implant. In the past three months, how many times have you had sexual intercourse without using any of these methods of birth control? For each question, the survey asked respondents to either check a box indicating none or provide their best guess of the number of times they participated in the referenced activity. Using responses to these questions, we first constructed a binary indicator (yes/no) of whether the respondent reported having had sexual intercourse in the past three months. We then constructed the variables for sex without a condom and sex without any effective method of protection. If participants reported being abstinent in the past three months, we retained them in the analysis and assigned them a value of zero on all three outcomes. In constructing these outcomes, we accounted for both missing data (item nonresponse) and inconsistent responses across questions (Table B.1). For example, to account for missing data, for respondents who skipped the question on unprotected sex, we logically imputed their B-3 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table B.1. Coding rules and sensitivity tests for behavioral outcomes Coding rules Sensitivity tests Had sex without any effective method of protection 1. For the question on frequency of sex without any effective method of protection, if Alternative 1: Ignore the respondent checked the box for none but also provided a (positive) numerical coding rule 3 response, we accepted the numerical response and coded the respondent as having engaged in the activity. 2. For respondents who skipped the question on frequency of sex without any effective Alternative 2: Ignore method of protection, we logically imputed their outcome as having not engaged in coding rule 2 this activity if they reported 0 or none in response to the questions on frequency of sex or frequency of sex without a condom. 3. If the coded value for this outcome conflicted with the coded values for sexual Alternative 3: Ignore activity or had sex without a condom, we recoded the outcome as missing. coding rules 2 and 3 Had sex without a condom 1. For the question on frequency of sex without a condom, if the respondent checked Alternative 1: Ignore the box for none but also provided a (positive) numerical response, we accepted the coding rule 3 numerical response and coded the respondent as having engaged in the activity. 2. For respondents who skipped the question on frequency of sex without a condom, Alternative 2: Ignore we logically imputed their outcome as having engaged in this activity if they reported coding rule 2 a positive numerical value to the other question on frequency of sex without any effective method of protection. 3. If the coded value for this outcome conflicted with the coded values for sexual Alternative 3: Ignore activity or had sex without any effective method of protection, we recoded the coding rules 2 and 3 outcome as missing. Recent sexual activity 1. For the question on frequency of sexual activity, if the respondent checked the box Alternative 1: Ignore for none but also provided a (positive) numerical response, we accepted the coding rule 3 numerical response and coded the respondent as having engaged in the activity. 2. For respondents who skipped the question on frequency of sexual activity, we Alternative 2: Ignore logically imputed their outcome as having engaged in this activity if they reported a coding rule 2 positive numerical value to the other questions on frequency of sex without a condom or frequency of sex without any effective method of protection. 3. If the coded value for this outcome conflicted with the coded values for had sex Alternative 3: Ignore without a condom or had sex without any effective method of protection, we recoded coding rules 2 and 3 the outcome as missing. Lifetime pregnancy 1. For respondents who reported a positive number in response to the question Alternative 1: Ignore on total number of lifetime pregnancies and either (1) reported a pregnancy in coding rule 2 the past three months or (2) reported having had a baby in the past three months, we coded them as having had a lifetime pregnancy, regardless of how they responded to the separate yes or no question on lifetime pregnancy. 2. For respondents who reported a lifetime pregnancy on either the immediate post- Alternative 2: Ignore test survey or 6-month survey, we coded them as having had a lifetime pregnancy, coding rule 1 regardless of how they responded to the lifetime pregnancy question on the 12- month survey. 3. If the coded value for this outcome conflicted with the coded value for recent Alternative 3: Ignore all pregnancy, we recoded the outcome as missing. coding rules Recent pregnancy 1. For respondents who skipped the question on recent pregnancy, we logically Alternative 1: Ignore imputed their outcome to having not experienced a recent pregnancy if they coding rule 2 responded no to the question on lifetime pregnancy. B-4 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Coding rules Sensitivity tests 2. If the coded value for this outcome conflicted with the coded value for lifetime Alternative 2: Ignore pregnancy, we recoded the outcome as missing. coding rule 1 . Alternative 3: Ignore all coding rules Multiple sexual partners 1. For respondents who skipped the question on multiple sexual partners, we logically Alternative 1: Ignore imputed their outcome as having not had multiple sexual partners if their coded coding rule 2 value on the sexual activity outcome indicated that they had not had sex in the past three months. 2. If the coded value for this outcome conflicted with the coded value for the measure Alternative 2: Ignore of sexual activity, we recoded the outcome as missing. coding rule 1 . Alternative 3: Ignore all coding rules Tested for STI 1. If the coded value for this outcome conflicted with the coded value for the measure Alternative 1: Ignore of STI diagnosis, we recoded the outcome as missing. coding rule 1 Diagnosed with STI 1. If the coded value for this outcome conflicted with the coded value for the measure Alternative 1: Ignore of STI testing, we recoded the outcome as missing. coding rule 1 STI = sexually transmitted infection. outcomes as not having engaged in the activity if they reported a value of 0 or none for the separate question on frequency of sexual activity. Similarly, to account for inconsistent responses across questions, we compared values for each respondent across the three measures of sexual activity and unprotected sex. If the values conflicted, we recoded the outcomes as missing. Table B.1 describes the specific coding rules used for each outcome. Because of the importance of these outcomes for our analysis, we also tested the sensitivity of our results to these coding decisions. In particular, we created up to three alternative codings of each outcome by ignoring one or more of the coding rules used for our main or primary outcome. We describe these alternative codings in Table B.1 and present the results of our sensitivity tests in Appendix C. 2. Sexual activity To assess the program’s impacts on overall rates of sexual activity, the 12-month survey asked youth to report the total number of times they had sex in the past three months. We used responses to this question to create a binary (yes/no) indicator for whether youth reported having sexual intercourse in the past three months. The question was limited to voluntary vaginal intercourse, not involuntary sexual activity or voluntary oral or anal sex. To account for missing data (item nonresponse) and any inconsistencies in responses across survey items, we coded this outcome in combination with the measures of unprotected sex following the coding rules specified in Table B.1. We tested the sensitivity of our results to these coding decisions by constructing three alternative codings of the outcome as listed in Table B.1. B-5 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM 3. Pregnancy risk We used responses from the 12-month survey to construct two separate pregnancy-related outcomes: (1) lifetime pregnancy and (2) recent pregnancy. For the measure of lifetime pregnancy, the survey asked respondents the following question: “To the best of your knowledge, have you ever been pregnant or gotten anyone pregnant, even if no child was born? Be sure to answer ‘yes’ if you are currently pregnant or had any pregnancy that ended in birth, an abortion, stillbirth, miscarriage, or live birth after which the baby died.” For the measure of recent pregnancy, the survey asked: “Have you been pregnant or gotten someone pregnant during the past 3 months? Be sure to answer ‘yes’ if you are currently pregnant or had any pregnancy that ended in birth, an abortion, stillbirth, miscarriage, or live birth after which the baby died.” To account for missing data (item nonresponse) and any inconsistencies in responses across survey items, we compared each respondent’s answers across the two questions. For example, for respondents who skipped the question on recent pregnancy, we logically imputed their outcome to having not experienced a recent pregnancy if they responded no to the question on lifetime pregnancy. We also compared respondents’ answers to these questions with their answers to two other pregnancy-related questions included on the 12-month survey: 1. To the best of your knowledge, how many times have you been pregnant or gotten someone pregnant? 2. Have you ever had a baby or has anyone you got pregnant had the baby? For example, for respondents who reported a positive number in response to the question on total number of lifetime pregnancies and either (1) reported a pregnancy in the past three months or (2) reported having had a baby in the past three months, we coded them as having had a lifetime pregnancy, regardless of how they responded to the separate yes/no question on lifetime pregnancy. We checked the sensitivity of our results to this coding decision by constructing an alternative version of the outcome measure that ignored this coding rule (Table B.1). To check the reliability of responses to these questions and to minimize any missing data resulting from item nonresponse, we also compared each participant’s response to the lifetime pregnancy question on the 12-month survey with his or her responses to comparable questions asked on the earlier post-test and 6-month surveys. For respondents who skipped the 12-month question on lifetime pregnancy but reported a pregnancy on either the post-test or 6-month surveys, we logically imputed their outcome on the 12-month survey to indicate a pregnancy. For respondents whose answers conflicted across surveys—for example, indicating a lifetime pregnancy on the 6-month survey but not the 12-month survey—we coded the lifetime pregnancy outcome as yes for the 12-month survey. We also checked the sensitivity of our results to this coding decision (Table B.1). 4. STI risk As summarized in chapter III, we constructed three outcomes related to the transmission and incidence of STIs: • Multiple sexual partners. The 12-month survey asked youth how many different people they had sex with in the past three months. The possible response categories ranged from “I B-6 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM have not had sex in the past 3 months” to “6 or more people.” We used responses to this question to create a binary (yes/no) indicator for youth who reported having two or more sexual partners in the past three months. The question was limited to voluntary vaginal intercourse, not involuntary sexual activity or voluntary oral or anal sex. To minimize item nonresponse, for respondents who skipped the question on multiple sexual partners, we logically imputed their outcome as having not had multiple sexual partners if their coded value on the sexual activity outcome indicated that they had not had sex in the past three months. For respondents whose answers to the question on multiple partners conflicted with their reported sexual activity, we recoded the outcome as missing. • Tested for an STI. The 12-month survey asked youth if they had been tested by a doctor or nurse for an STI in the past 12 months. We used responses to this question to create a binary (yes/no) indicator for whether youth reported having been tested. If the response to this question was missing but the youth reported being told by a doctor or nurse that he or she had an STI, we coded the value of this response to yes. If the response to this question was inconsistent with other STI related questions—that is, if the youth reported having been told by a doctor or nurse in the past 12 months that he or she had an STI but also reported not having been tested by a doctor or nurse for an STI—we recoded the outcome as missing. • Diagnosed with an STI. The 12-month survey asked youth if they had been told by a doctor or a nurse in the past 12 months that they had an STI. We used responses to this question to create a binary (yes/no) indicator of STI diagnoses. If the response to this question was inconsistent with other STI related questions—that is, if the youth reported having been told by a doctor or nurse in the past 12 months that he or she had an STI but also reported not having been tested by a doctor or nurse for an STI—we recoded the outcome as missing. For all three of these outcomes, Table B.1 describes our coding rules and sensitivity tests. B. Candidate covariates As discussed in Chapter III, we included several types of baseline covariates in the regression models used to estimate program impacts. We included some of these covariates to account for the stratification used for random assignment. We included others to improve the precision of the impact estimates. To help select the covariates used for precision gains, we used a data-driven stepwise selection procedure developed previously in the literature (Social and Character Development Research Consortium 2010). For this procedure, we considered as candidate covariates both (1) any baseline measure for which the observed difference between the treatment and control groups had a p-value of 0.20 or less based on a two-sided t-test and (2) other baseline variables that other studies have shown to have a strong link with risk sexual behavior and repeat pregnancy. Table B.2 provides a complete list of the variables considered. B-7 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table B.2. Measures of baseline sample characteristics Measure Definition Demographic and personal characteristics Age when entered foster care Continuous variable for age when first entered the foster care system system Age Continuous variable for age at randomization Grade behind for age Binary variable: equals 1 if youth is a grade level behind based on last grade completed and age at baseline; equals 0 if youth has not completed 12th grade and is not a grade level behind based on age, or if 12th is the last grade completed Race and ethnicity Series of binary (yes/no) indicators for the following racial and ethnic categories: (1) Hispanic, (2) non-Hispanic white, (3) non-Hispanic black, (4) Native American, (5) Asian and (6) non-Hispanic other race Aspire to attend college Binary variable: equals 1 if youth reported that he/she is somewhat or very likely to graduate from a 4-year college; equals 0 if youth reported that he or she is not at all likely or a little bit likely to graduate from a 4-year college Ever had sex Binary variable: equals 1 if the youth reported having had sexual intercourse; equals 0 if the youth reported never having sexual intercourse Age at first sex Continuous variable for age at first sex Sex without using a condom Binary variable: equals 1 if youth reported having had sex in the 3 months before completing the baseline survey without using a condom; equals 0 if youth reported not having had sex or always using a condom Sex without using an effective Binary variable: equals 1 if youth reported having had sex in the 3 months method of protection before completing the baseline survey without using any effective method of protection; equals 0 if youth reported abstaining from sex in the past 3 months or always using an effective method of protection Number of different lifetime sexual Categorical variable with categories for (1) youth had one or no sexual partners partners (2) youth had 2 to 5 sexual partners (3) youth had 6 to 12 sexual partners and (4) youth had more than 12 sexual partners Lifetime pregnancy Binary variable: equals 1 if youth reported having ever been pregnant (females) or gotten someone pregnant (males); equals 0 if youth reported having never been pregnant (females) or gotten someone pregnant (males) Recent pregnancy Binary variable: equals 1 if youth reported having been pregnant (females) or gotten someone pregnant (males) in the prior 3 months; equals 0 if youth reported having not been pregnant (females) or gotten someone pregnant (males) in the prior 3 months Tested for STI Binary variable: equals 1 if youth reported being tested for an STI in the 12 months before completing the baseline survey; equals 0 if youth reported not being tested Diagnosed with STI Binary variable: equals 1 if youth reported being diagnosed with an STI in the 12 months before completing the baseline survey; equals 0 if youth reported not being diagnosed STI = sexually transmitted infection. B-8 APPENDIX C SENSITIVITY ANALYSIS This page has been left blank for double-sided copying. FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM The main impact findings presented in Chapter IV of this report derive from a particular set of analytic decisions, ranging from the data cleaning procedures used to construct the outcome measures to the specification of the regression models. We made these decisions in accordance with established research standards and the particular features of our study design. However, we also investigated the sensitivity of our results to alternative analytic decisions. We conducted these sensitivity tests only for the behavioral outcomes (unprotected sex, sexual activity, pregnancy risk, and sexually transmitted infection [STI] risk), not the secondary measures of knowledge, awareness, attitudes, and self-efficacy. For each outcome, we conducted four kinds of sensitivity tests: 1. Coding of the outcome measure. As discussed in greater detail in Appendix B, we constructed our behavioral outcomes to minimize the amount of missing data and to ensure that coded values were consistent across outcomes. For example, for participants who did not respond to the question on sex without a condom, we logically imputed their outcome to no if they reported on a separate question not having had sex in the past three months. We made similar coding decisions for the measures of sexual activity, pregnancy risk, and STI risk. Because of the importance of these outcomes for our analysis, we checked the sensitivity of our results to these coding decisions by creating up to three alternative versions of each outcome. Appendix B describes the details of these alternatives. 2. Specification of the regression model. For the main findings presented in Chapter IV of this report, we specified the regression models using ordinary least squares (OLS) regression and accounting for the clustered design by estimating cluster-robust standard errors. To check the sensitivity of our results to these analytic decisions, we estimated three alternative specifications of the regression model. First, for binary outcome measures, we estimated logistic regression models instead of OLS regression. For this test, we had to replace the covariates for random assignment strata with binary indicator variables for state of residence (California, Maryland, or Oklahoma) because, for some outcomes, a lack of variation in youth outcomes within strata made it impossible for the logistic regression model to estimate coefficients for the strata indicators. Second, as an additional sensitivity test, we accounted for the clustered design using a linear mixed-effects or multilevel regression model instead of cluster-robust standard errors. Third, as an alternative way to account for the clustered design, we conducted a two-staged cluster-level analysis instead of an individual-level regression model (Hayes and Moulton 2009). The interim report provides additional details on the mixed-effects regression models and cluster-level analysis (Goesling et al. 2015). We did not adjust our results for multiple hypothesis testing when conducting these tests. 3. Covariates included in the model. As described in Chapter III, for the main findings presented in Chapter IV of this report, each regression model included the following covariates: (1) a binary indicator for treatment status, (2) binary indicator variables for each of the strata created for random assignment, (3) three key demographic variables that are highly correlated with our key outcomes of interest (age, gender, and race), (4) the baseline measure of the outcome (if available), and (5) an additional set of baseline covariates that we empirically selected through a data-driven forward selection procedure. As a sensitivity test, we estimated comparable regression models with two alternative combinations of covariates: (1) controlling only for treatment status, random assignment strata, and the baseline measure of the outcome (if available); and (2) controlling only for treatment status and random assignment strata with no additional covariates. C-3 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM 4. Method for estimating p-values. For the main findings presented in Chapter IV of this report, we adjusted the reported p-value for each outcome to account for other tests conducted within the same family of outcomes. For example, for the primary measures of unprotected sexual activity, we adjusted the reported p-values to account for our two separate outcomes: (1) sex without a condom and (2) sex without any effective method of protection. Adjusting for separate but correlated outcomes has the practical effect of increasing the reported p-values for certain outcomes, which might lead to more conservative conclusions about program effectiveness. As a sensitivity test, we also calculated unadjusted p-values that treat each outcome as an independent test. The results of these analyses showed that our findings were generally robust to alternative analytic decisions. For our primary measures of unprotected sex (Table C.1), the direction, size, and statistical significance of the impact estimate were generally consistent with our main findings presented in Chapter IV and each of the sensitivity tests we conducted. In eight of the nine sensitivity tests, the size of the impact of the program on the likelihood that youth reported having had sex without using any effective method of protection in the past three months was small (ranging from 0.4 to 1.1 percentage points) and not statistically significant—that is, we cannot reject the hypothesis that the treatment and control groups were equally likely to report having had sex without using any effective method of protection in the past three months. When conducting the cluster level analysis, the size of the impact was larger than what we report in Chapter IV (2.6 versus 0.8 percentage points), but not statistically significant. For the other primary outcome—sex without using a condom—the direction and size of the impact was consistent between our main findings presented in Chapter IV and each of the sensitivity tests. Our findings were also generally robust for the measures of sexual activity, pregnancy risk, and STI risk (Table C.2). For the outcome of recent sexual activity, the impact estimate was small (ranging from -0.7 to 0.4 percentage points) and statistically insignificant across all tests. For the outcome of lifetime pregnancy, the impact remained statistically significant in seven of the nine sensitivity tests, ranging from -4.3 to -6.5 percentage points. For the other outcomes— recent pregnancy, multiple sexual partners, tested for an STI, and diagnosed with an STI—the general magnitude and statistical significance of the impact estimate was consistent between our main findings presented in Chapter IV and each of the sensitivity tests. Table C.1. Sensitivity of impacts on primary outcomes Had sex without any effective . method of protection Had sex without condom . Impact p-value Impact p-value Main findingsa -0.5 >0.99 1.5 >0.99 Alternative coding of outcome: . . . . Alternative 1 -0.7 >0.99 1.8 0.94 Alternative 2 -0.8 >0.99 2.2 0.80 Alternative 3 -0.6 >0.99 2.0 0.84 Specification of regression model: . . . . Logistic model -0.2 >0.99 2.4 0.97 Linear mixed-effects model -0.5 0.82 1.5 0.55 Linear cluster-level analysis 2.9 0.42 5.6 0.13 C-4 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Had sex without any effective . method of protection Had sex without condom . Impact p-value Impact p-value Covariates included in the model: . . . . Strata and baseline outcome only -1.0 >0.99 1.4 >0.99 Strata only -1.1 >0.99 1.3 >0.99 Method for estimating p-values: . . . . Ignore multiple comparisons -0.5 0.82 1.5 0.55 Source: Surveys administered to study participants by the evaluation team. Note: For each outcome, the numbers in the columns labeled “Impact” represent the difference between regression-adjusted predicted values of outcomes at the 12-month follow-up survey between the treatment and control groups. p-values are adjusted for clustering of standard errors and for multiple outcomes measured within a single domain, unless otherwise stated. See Chapter III for a more detailed description of the analytic methods. a The main findings denote the impact estimates presented in Chapter IV of the report. C-5 Table C.2. Sensitivity of impacts on sexual activity, pregnancy risk, and STI risk Recent sexual Lifetime Recent Multiple sexual Diagnosed with . activity pregnancy pregnancy partners Tested for STI STI . Impact p-value Impact p-value Impact p-value Impact p-value Impact p-value Impact p-value Main findingsa -0.5 0.83 -5.1 <0.01 -0.6 >0.99 -0.1 >0.99 2.3 0.97 1.3 >0.99 Alternative coding of outcome . . . . . . . . . . . . Alternative 1 -0.4 0.89 -5.7 <0.01 -0.7 >0.99 -0.4 >0.99 3.4 0.40 -0.6 >0.99 Alternative 2 -0.4 0.86 -4.7 0.3 -0.3 >0.99 . . . . . . Alternative 3 -0.4 0.89 -4.3 0.5 -0.6 >0.99 . . . . . . Specification of regression . . . . . . . . . . . . model Logistic model 0.6 0.86 -4.9 0.05 -0.7 >0.99 -0.5 >0.99 1.8 >0.99 1.6 >0.99 Linear mixed-effects model -0.5 0.83 -5.5 <0.01 -0.6 0.63 -0.1 0.95 2.3 0.32 1.3 0.49 Linear cluster-level analysis 0.6 0.88 -5.4 0.05 -1.3 0.60 0.2 0.95 2.9 0.45 0.1 0.97 Covariates included in the . . . . . . . . . . . . C-6 model Strata and baseline outcome -0.7 0.80 -5.5 <0.01 -0.7 >0.99 0.3 >0.99 2.2 >0.99 0.8 >0.99 only Strata only -0.7 0.81 -6.5 0.02 -0.6 >0.99 0.3 >0.99 0.9 >0.99 1.0 >0.99 Method for estimating p-values: . . . . . . . . . . . . Ignore multiple comparisons -0.5 0.83 -5.1 <0.01 -0.6 0.63 -0.1 0.95 2.3 0.32 1.3 0.49 Source: Surveys administered to study participants by the evaluation team. Note: For each outcome, the numbers in the columns labeled “Impact” represent the difference between regression-adjusted predicted values of outcomes at the 12-month follow-up survey between treatment and control groups. p-values are adjusted for clustering of standard errors and for multiple outcomes measured within a single domain, unless otherwise stated. See Chapter III for a more detailed description of the analytic methods. a The main findings denote the impact estimates presented in Chapter IV of the report. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. STI = sexually transmitted infection. APPENDIX D IMPACT AT EACH FOLLOW-UP This page has been left blank for double-sided copying. FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM As discussed in Chapter I of this report, the study was ultimately designed to assess the program’s success in reducing rate of unprotected sex among study youth at the time of the 12- month follow-up survey. For this reason, the main findings presented in Chapter IV focus on data from the 12-month survey. However, in total we administered surveys to youth at four time points: at baseline, immediately after the intervention, and 6 and 12 months after the program ended. As a result, it is possible that our focus on the 12-month survey led us to miss some meaningful or substantively important program impacts on youth behaviors—for example, if the program had short-term or immediate effects on behavior that faded over time. This appendix presents estimated differences between the treatment and control groups on each behavioral outcome at four points in time. We do not intend to formally estimate repeated measures or growth curve analyses examining trends or trajectories of youth behaviors over time. Instead, we analyze the data for each of the four time points separately, to provide a snapshot of differences between the treatment and control groups at each point in time. We estimated these differences using the analytic approach described in Chapter III. A. Unprotected sex We found no statistically significant difference between the treatment and control groups for our primary measures of unprotected sex at any point in time (Figures D.1 and D.2). At the time of the baseline survey, both groups were equally likely to report having engaged in sex without any effective method of protection (about 24 percent) or without a condom (about 30 percent) in the past 3 months. Rates of unprotected sexual activity declined from the time of the baseline survey to immediately after the intervention, but then increased from the time immediately after the intervention to the 6-month follow-up survey. For example, from the time of the post- intervention survey to the 6-month survey, the likelihood of reporting having sex without using any effective method of protection increased from 18.1 to 31.0 percent among youth assigned to the treatment group and from 17.4 to 35.1 percent among youth assigned to the control group. From the time of the 6-month survey to the 12-month survey, rates of unprotected sexual activity were mostly the same. D-3 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Figure D.1. Differences in rates of sex without any effective method of protection Source: Youth surveys administered by the study team. Figure D.2. Differences in rates of sex without using a condom Source: Youth surveys administered by the study team. 2. Other sexual risk behaviors and reproductive health outcomes We found no statistically significant difference between the treatment and control groups for any other sexual risk behavior or reproductive health outcomes immediately after the intervention or at the time of the 6-month follow-up survey. Youth in the treatment group were just as likely as those in the control group to report having had sex in the past three months, on each survey administered (Figure D.3). At the time of the baseline survey, youth in the treatment group were almost 3 percentage points more likely to report having been pregnant or gotten someone pregnant in the past 3 months (p-value = 0.03), but both groups were equally likely to D-4 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM report a pregnancy in the past 3 months at the time of each follow-up survey (Figure D.4). Youth assigned to the treatment group were less likely to report being pregnant or getting someone pregnant at the time of the 12-month survey compared with youth assigned to the control group (50.8 versus 55.9 percent), as discussed in Chapter IV. Figure D.3. Differences in rates of sexual activity in the past three months Source: Youth surveys administered by the study team. Figure D.4. Differences in rates of pregnancy in the past three months Source: Youth surveys administered by the study team. *Significantly different from zero at the .05 level, two-tailed test. D-5 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Figure D.5. Differences in rates of lifetime pregnancy ** Source: Youth surveys administered by the study team. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. We found no evidence that the PTC program had an impact on sexually transmitted infection (STI) risk factors such as number of sexual partners and STI diagnoses and testing. At the time of each follow-up survey, youth in the treatment group were equally likely to report having multiple sexual partners in the past three months compared with youth in the treatment group (Figure D.6). However, the -3.8 percentage point difference measured immediately after the intervention approaches statistical significance (p-value = 0.08). From the post-intervention survey to the time of the 6-month survey, the likelihood of having multiple sexual partners increased from 12.4 to 34.5 percent among youth in the treatment group and from 16.2 to 31.9 percent among youth in the control group. Note that this outcome was not measured at baseline. Youth in the treatment and control groups were also equally likely to report being tested for (Figure D.7) or diagnosed with an STI (Figure D.8) in the past year at the time of each follow-up survey. For both outcomes, the rates remain largely the same across the follow-up period. D-6 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Figure D.6. Differences in rates of having multiple sexual partners in the past three months Source: Youth surveys administered by the study team. Figure D.7. Differences in rates of being tested for a STI in the past 12 months Source: Youth surveys administered by the study team. D-7 FINAL IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Figure D.8. Differences in rates of being diagnosed with an STI in the past 12 months Source: Youth surveys administered by the study team. D-8 This page has been left blank for double-sided copying. http://www.hhs.gov/ash/oah/