OAH Evaluation Report Impact Report from the Evaluation of Adolescent Pregnancy Prevention Approaches POWER Through Choices August 2015 Purpose statement: This study reports interim findings from a large-scale demonstration project and evaluation of POWER Through Choices, a comprehensive sexual health education curriculum designed specifically for youth in foster care and other out-of-home care settings. The study reports interim impacts of the program on measures of youth knowledge, attitudes, and intentions assessed shortly after the program ended. A future report will examine the program’s longer-term impacts on youth sexual risk behaviors. August 2015 Recommended citation: Goesling, Brian, Reginald D. Covington, Jennifer Manlove, Megan Barry, Roy F. Oman, and Sara Vesely. “Interim Impacts of the POWER Through Choices Program.” Princeton, NJ: Mathematica Policy Research, August 2015. This document is available at http://www.hhs.gov/ash/oah/oah-initiatives/evaluation/federalevals.html Prepared for OAH under contract number: HHSP233201450030A Authored by: Brian Goesling Reginald D. Covington Mathematica Policy Research Jennifer Manlove Megan Barry Child Trends Roy F. Oman Sara Vesely University of Oklahoma Health Sciences Center ACKNOWLEDGEMENTS ACKNOWLEDGEMENTS This evaluation was made possible only through the hard work and support of many people and organizations. First, we wish to acknowledge the support of staff from the Office of Adolescent Health (OAH) in the U.S. Department of Health and Human Services (HHS). We especially thank Amy Farb, the OAH project officer for the Evaluation of Adolescent Pregnancy Prevention Approaches (PPA) study, who has provided outstanding support and guidance throughout the project. We also thank staff from the Family and Youth Services Bureau within the Administration for Children and Families at HHS, especially Itege Bailey and Marc Clark. At the Oklahoma Institute for Child Advocacy (OICA), a special thanks goes to Janene Fluhr, whose tireless commitment to the POWER Through Choices program served as the driving force behind the evaluation and was instrumental to the study’s success. We also thank Sheila Cavallo from OICA, as well as a large team of talented and dedicated data collectors, program facilitators, and administrators from OICA, Kern County Superintendent of Schools, and Planned Parenthood of Maryland. At the University of Oklahoma Health Sciences Center, we thank Jeannie Williams and Jennifer Green for their expert oversight of a highly successful data collection effort. At Mathematica Policy Research, we thank Christopher Trenholm and Alan Hershey for their guidance and support during the initial study design phase. We also thank Melissa Thomas for supporting the data collection effort, Molly Crofton for excellent programming assistance, and Jennifer Baskwell for producing the report. At Child Trends, we thank Kristin Moore for useful comments on an earlier draft of the report and Julia Goodwin for helpful research assistance. Finally, we extend our greatest thanks and gratitude to the youth, program staff, and administrators of the many group homes that agreed to participate in the study. We hope that our report does justice to the time and effort they devoted to making the study possible. Brian Goesling Reginald D. Covington Jennifer Manlove Megan Barry Roy F. Oman Sara Vesely This page has been left blank for double-sided copying. INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM CONTENTS I INTRODUCTION.............................................................................................................................. 1 A. The POWER Through Choices program ................................................................................... 2 B. Evaluation challenges ................................................................................................................ 4 C. Research questions ................................................................................................................... 6 II STUDY DESIGN .............................................................................................................................. 9 A. Recruitment of group homes ..................................................................................................... 9 B. Random assignment procedures............................................................................................. 10 C. Enrollment and retention of individual youth ........................................................................... 11 D. Baseline sample characteristics .............................................................................................. 13 E. Treatment and control conditions ............................................................................................ 15 III DATA, MEASURES, AND ANALYSIS ........................................................................................... 17 A. Outcome measures ................................................................................................................. 17 1. Exposure to information .................................................................................................... 17 2. Knowledge ........................................................................................................................ 19 3. Awareness of available health resources ......................................................................... 19 4. Attitudes toward safe sex and methods of protection ....................................................... 19 5. Perceived self-efficacy to avoid unprotected sex .............................................................. 20 6. Intentions toward sexual activity ....................................................................................... 20 B. Analytic approach .................................................................................................................... 21 IV RESULTS ....................................................................................................................................... 23 A. Exposure to program information ............................................................................................ 23 B. Knowledge and awareness of health resources...................................................................... 24 C. Attitudes and intentions ........................................................................................................... 26 V DISCUSSION AND CONCLUSION ............................................................................................... 29 REFERENCES ............................................................................................................................................ 31 APPENDIX A: NONRESPONSE ANALYSIS ............................................................................................. A.1 APPENDIX B: OUTCOME MEASURES .................................................................................................... B.1 APPENDIX C: SENSITIVITY ANALYSES .................................................................................................C.1 ii This page has been left blank for double-sided copying. INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM I. INTRODUCTION Some of the nation’s most vulnerable youth are those living in foster care and other out-of- home settings. Many such youth have experienced abuse and neglect, face mental health and substance abuse challenges, and struggle with serious behavioral problems (Casanueva et al. 2014). They are more likely than their peers to have academic struggles and less likely to graduate from high school or attend college (Dworsky et al. 2014). In part for these reasons, they also have higher rates of teen pregnancy, sexually transmitted infections (STIs), and associated sexual risk behaviors. Estimates suggest that as many as one in three girls in foster care will become a teen mother—a rate more than double the national average (Dworsky and Courtney 2010). Of those who become mothers before age 18, more than one in four will go on to have another child before aging out of their teens (Putnam-Hornstein and King 2014). Despite these increased risks, many youth in out-of-home care lack timely access to sexual health education and services. Frequent changes in living arrangements can disrupt school attendance and reduce exposure to common school-based programs and services (Becker and Barth 2000; Boonstra 2011). Youth placed in residential group homes in particular report a lack of exposure to sexual health education and difficulty accessing reproductive health information and services (Freundlich 2003; Crottogini et al. 2008). In other cases, youth report that the sexual health education they received came too late, after they had already become sexually active (Love et al. 2005). As a result of these challenges, youth living in out-of-home care also report relatively low levels of knowledge about contraceptive methods and reproductive health (Hudson 2012). Efforts to improve sexual health education and services among these youth have become a growing focus of child welfare and youth-serving organizations in recent years (for example, Leonard and Suellentrop 2013). This report presents interim findings from a large-scale demonstration project and evaluation of the POWER Through Choices (PTC) program, a comprehensive sexual health education curriculum designed specifically for youth in foster care and other out-of-home care settings. The program features ten 90-minute sessions delivered to small groups of youth by trained facilitators in a classroom-based setting. This curriculum-based structure is similar to many teen pregnancy prevention programs delivered to general youth populations in schools and community settings. The PTC program is unique, however, in addressing the needs and risk factors specific to youth in foster care and other out-of-home care settings. Findings from an implementation study of the PTC program were presented in an earlier report (Meckstroth et al. 2014). The present report adds to these findings by describing the short-term impacts of the PTC program on youth knowledge, attitudes, and intentions. A future report will examine the program’s longer-term impacts on youth sexual risk behaviors. The evaluation has involved a unique collaboration and partnership among several organizations. The evaluation was originally designed by the Oklahoma Institute for Child Advocacy (OICA) in collaboration with researchers from the University of Oklahoma Health Sciences Center (OUHSC). In fall 2010, OICA was awarded competitive federal grant funding for the evaluation through the Family and Youth Services Bureau within the Administration for Children and Families of the U.S. Department of Health and Human Services (HHS). In winter 2011, the PTC evaluation was then selected as one of seven sites to participate in the Evaluation of Adolescent Pregnancy Prevention Approaches (PPA) study, a major federal effort to expand 1 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM available evidence on effective ways to prevent and reduce pregnancy and related sexual risk behaviors among teens in the United States. The PPA study is conducted by Mathematica Policy Research and its partners, Child Trends and Twin Peaks Partners, LLC, under contract with the Office of Adolescent Health within HHS. Participating in PPA provided the PTC evaluation additional resources to support data collection and analysis. In addition, researchers from the PPA evaluation team have collaborated with OICA and OUHSC to refine the evaluation design, support data collection, and plan the analysis. Two other regional organizations—the Kern County Superintendent of Schools and Planned Parenthood of Maryland—have also played key roles in supporting program implementation and study data collection, as explained later in the report. The report is divided into five chapters. In the remainder of this chapter, we provide a more detailed description of the PTC program and discuss several practical and technical challenges that shaped the overall evaluation design. Chapters II and III provide a more detailed description of the study design, data, and analytic methods. Chapter IV presents findings from the interim impact analysis, and Chapter V summarizes and discusses the implications of the results. A. The POWER Through Choices program PTC is a comprehensive teen pregnancy, HIV, and STI prevention program designed specifically to address the needs of youth living in foster care and other out-of-home care settings. The ten 90-minute program sessions (Table I.1) are delivered once or twice a week for 5 to 10 weeks. The sessions feature interactive skill-building activities delivered by trained facilitators to small groups of 8 to 20 youth. The program targets male and female youth ages 13 to 18 years. The program developers designed the PTC program for use with youth living in any number of out-of-home care settings, including family foster homes, kinship foster care, residential group homes, transitional living centers, and juvenile justice facilities. The present study focuses on the second edition of the PTC program. An earlier first edition was developed in the mid-1990s by researchers in the Family Welfare Research Group at the University of California, Berkeley, in response to an unmet need for sexual health education among youth in out-of-home care settings (Becker and Barth 2000). The intensive three-year program development effort drew on input from multiple sources, including youth focus groups, interviews with program staff and practitioners, and pilot testing of all program sessions and activities. The program was grounded in four psychosocial theories of individual health behavior: the health belief model, self-regulation theory, theory of reasoned action, and social and cognitive learning theory. Beginning in 2005, OICA launched a five-year effort to update and expand the original curriculum. This second edition of the PTC program maintains the format, goals, and interactive nature of the original curriculum but features two new sessions on reproductive health and STIs, revised role-playing scenarios, and updated data and resource information. The second edition also promotes inclusivity of sexual orientation—for example, through role-playing scenarios on gay and lesbian relationships. The PTC implementation report provides a more detailed description of the program’s origins and development (Meckstroth et al. 2014). 2 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table I.1. POWER Through Choices program sessions Session Title Purpose 1 Introduction to PTC Introduce curriculum, assess participants’ knowledge regarding pregnancy prevention and sex education, and demonstrate role playing 2 Adolescent Reproductive Increase knowledge of male and female reproductive anatomy, the Health Basics process of fertilization and conception, and the menstrual cycle 3 Creating the Future You Want Identify planning involved in practicing positive sexual behaviors, outline individual choices involved in sexual decision making, and discuss abstinence as a viable choice 4 Making Choices Clear Help participants to build assertiveness and communication skills related to sexual activity 5 Understanding STIs and HIV Increase knowledge and understanding of STI/HIV transmission and and How to Reduce Your Risk prevention 6 Increasing Contraceptive Increase knowledge about contraceptive methods Knowledge 7 Practice Makes Perfect Discuss the level of risk associated with various sexual behaviors, use role playing to demonstrate the importance of dual methods, and learn condom use skills 8 Using Resources to Support Discuss ways to improve communication about contraception with foster Your Choices parents, guardians, and group home staff members; learn how to access local sexual and reproductive health resources 9 Making Choices That Fit Your Develop a plan for avoiding unwanted pregnancies and STIs, set short- Lifestyle and long-term goals, and identify choices needed to attain goals 10 Plan + Prepare + Practice = Reinforce themes and messages of the curriculum POWER The program is designed to emphasize two main themes: self-empowerment and the impact of choices. Youth in out-of-home care often lack the positive skills and resources necessary to avoid risky sexual behaviors and teen pregnancy (Becker and Barth 2000). These include dependable family supports and social networks, a strong sense of personal autonomy or control, and the skills necessary to identify and successfully attain personal goals. The PTC program seeks to address these barriers by empowering youth to make informed decisions about their sexual risk behaviors and to help them recognize the potential consequences of these decisions for their future goals. The program challenges youth to envision a positive future for themselves and to make choices about relationships and health behaviors that help promote their future success. The program emphasizes these themes throughout each program session and activity. To help build a sound base of knowledge, the program provides a broad range of factual information on reproductive health, HIV, STIs, and methods of protection. It also seeks to increase awareness of available health resources, such as where to find an adolescent health center or how to identify and communicate with a trusted adult. The program has a strong emphasis on skill building through activities and group discussions intended to promote communication skills, healthy decision making, and goal setting. The program sessions feature an interactive approach that engages youth through role playing, group discussion, and other hands-on activities. Each session also includes time for questions and answers (Q&A). 3 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM These program activities are designed to have impacts that unfold over time on an interrelated, sequential series of outcomes (Figure I.1). In the short run, the program aims to build knowledge, develop skills, increase awareness of available health resources, and promote a greater sense of personal empowerment or self-efficacy. PTC anticipates that youth will feel motivated, prepared, and empowered to make healthy decisions by gaining knowledge, setting goals, planning for the future, and developing communication and other life skills. In turn, these short-term outcomes are thought to have longer-run effects on a range of behavioral outcomes: delayed onset of sexual activity, decreased number of sexual partners, increased use of contraceptive methods, and increased correct and consistent condom use. The program’s ultimate long-term goals are to reduce the incidence of teen pregnancy and STIs among youth in foster care and other out-of-home settings. Figure I.1. POWER Through Choices program logic model B. Evaluation challenges Rigorously evaluating the PTC program presents many practical and technical challenges. An earlier small-scale evaluation of the first edition PTC program found that participants responded positively to the program and reported greater intention to avoid unprotected sex (Becker and Barth 2000). However, this study was conducted with a small sample of 66 youth who participated in the program pilot and was designed only to provide preliminary evidence on the program’s potential for success. In contrast, the present study is designed as a more formal demonstration and random assignment evaluation of the program when implemented on a large scale. Both the need for a large scale and the random assignment evaluation design present unique challenges for the study. Understanding these challenges is necessary for making sense of the more technical study design and analytic issues discussed in the remainder of the report. 4 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM A key early challenge was identifying an appropriate setting for the evaluation. The program developers designed the PTC program to be used with youth living in any number of out-of- home care settings, including family foster homes, kinship foster care, residential group homes, transitional living centers, and juvenile justice facilities. However, the evaluation required implementing the program on a very large scale to hundreds of participating youth. The need to implement the program on such a large scale ruled out common settings such as family foster homes or kinship foster care that could not generate the necessary sample size but are otherwise appropriate for the program. Recognizing this challenge, OICA and OUHSC made the key early decision to conduct the evaluation with youth living in residential group homes. These homes are congregate residential facilities operated or contracted by a state child welfare agency, a state juvenile justice agency, or a private care provider. They typically serve a very high-risk population of youth referred for full-time residential housing and care through the state child welfare (foster care) or juvenile justice systems. The youth placed in these homes typically live on site for a period of several months, often as a result of a court order. The combination of a residential setting and high-risk population made these homes a feasible and important venue for evaluating the PTC program. This focus on residential group homes in turn complicated other aspects of the evaluation design. For example, to provide the most rigorous evidence on the effects of the PTC program, the evaluation was designed as a randomized trial that involved randomly assigning youth to a treatment group that was offered the PTC program or to a control group that was not offered the program (see details in Chapter II). However, it was not feasible to randomly assign youth to treatment and control groups within the same home because the residential setting allowed for potential spillover or “contamination” effects if a youth assigned to the treatment group interacted with youth in the control group. Randomly assigning individual youth also presented a logistic challenge of transporting youth assigned to the treatment group to a central location to receive the intervention. To address these complications, we designed the evaluation so that all youth living in the same group home were assigned to the same research status, either treatment or control. This approach yielded a “cluster” random assignment design in which youth were assigned to the treatment or control conditions in intact clusters or groups. Cluster randomized trials have become relatively common in studies of school-based teen pregnancy prevention programs (Goesling et al. 2014) but not, to our knowledge, in studies involving high-risk youth in residential group homes. In part because of the need for a clustered evaluation design, the study team set very high recruitment targets. The approach of randomly assigning youth in intact clusters or groups introduces a “design effect” that reduces the study’s effective sample size—that is, the statistical precision of the study is weakened, as though the study’s sample size were smaller than it actually is (Hayes and Moulton 2009). This design effect is a function of both the average size of the clusters and the correlation in outcomes among youth within the same cluster. The larger the design effect, the smaller the effective sample size. To help offset this expected design effect, the initial design for the PTC evaluation called for recruiting more than 800 youth from a sample of 32 residential group homes. With additional resources made available through the federal PPA study, the team further increased these targets to more than 1,000 youth from a sample of at least 5 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM 40 residential group homes. Although a sample size of 1,000 is not uncommon in studies involving the general youth population, it is relatively rare in studies of youth in foster care and other out-of-home care settings (Lee et al. 2008; Kinsey and Schlösser 2012). To achieve these recruitment targets, OICA recognized a need to recruit residential group homes across multiple states. OICA planned to lead the recruitment of homes within its home state of Oklahoma. However, the team also recognized that even with strong participation rates, there were not enough homes in Oklahoma (or any other single state) to generate the targeted sample size. As a result, OICA also recruited three partner organizations to help recruit homes and implement the PTC program in other states: the Kern County Superintendent of Schools (California), Planned Parenthood of Maryland, and the Illinois Caucus for Adolescent Health. These three organizations were selected because they all had existing relationships with their state foster care systems as well as history and experience delivering sexual health education (Meckstroth et al. 2014). The Illinois Caucus for Adolescent Health ultimately dropped out of the study because of staff turnover and uncertainty about its prospects for recruiting group homes. Both the Kern County Superintendent of Schools and Planned Parenthood of Maryland remained involved throughout the study period and together generated more than half of the final study sample (see details in Chapter II). The confluence of these factors explains the study design described in the remainder of this report: a large-scale, multi-site evaluation involving 44 residential group homes in three states (California, Maryland, and Oklahoma). OICA played the lead role in coordinating the overall partnership and monitoring program delivery across states. OICA also led the recruitment of group homes and program delivery within the state of Oklahoma. Kern County Superintendent of Schools and Planned Parenthood of Maryland led the recruitment of group homes and program delivery in California and Maryland, respectively. OUHSC oversaw the evaluation activities and had lead responsibility for coordinating sample enrollment, random assignment, and data collection. Members of the federal PPA team supported OUHSC in monitoring the evaluation activities and planning the analysis. C. Research questions This report examines the interim impacts of the PTC program measured immediately after youth completed the 10 program sessions. This focus on very short-term program impacts, measured while most study participants were still living in one of the participating group homes, naturally limits our focus to such outcomes as youth knowledge, attitudes, and intentions. A future report will examine longer-term impacts on youth behavioral outcomes measured 6 and 12 months after enrolling in the study. The specific questions addressed in this interim report are as follows: • Is the PTC program effective in increasing exposure to information on reproductive health, pregnancy and STI prevention, and methods of protection? • Does PTC increase youth knowledge of reproductive health, STIs, and methods of protection? 6 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM • Does PTC increase youth awareness of available health resources and where to get methods of protection? • Does PTC lead to increased support for safe sex and the use of methods of protection? • Is PTC effective in promoting a greater sense of self-empowerment or self-efficacy to avoid unprotected sexual activity? • Does PTC increase youth intentions to avoid unprotected sexual activity? The question of program impacts on rates of unprotected sexual activity and associated sexual risk behaviors will be addressed in a future report, drawing on self-reported data from longer-term surveys conducted 6 and 12 months after study enrollment. 7 This page has been left blank for double-sided copying. INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM II. STUDY DESIGN The study was designed as a cluster randomized trial involving youth recruited from 44 residential group homes across three states: California, Maryland, and Oklahoma. Within each state, about half the group homes were randomly assigned to a treatment group that offered the PTC program. The other homes were assigned to a control group that did not offer the program. This design resulted in about half the study youth being offered the PTC program and half not being offered the program. We calculate program impacts by comparing youth outcomes between the two groups at the end of the program. In this chapter, we begin by describing the recruitment of group homes and random assignment procedures. We then describe the enrollment and retention of individual youth within the group homes as well as the baseline characteristics of the study sample. We end by providing a summary description of the treatment and control conditions. The next chapter describes the data, measures, and analytic methods used to estimate impacts of the PTC program. A. Recruitment of group homes Group homes were recruited on a rolling basis from summer 2011 through fall 2013. This recruitment effort was led by a different organization in each state: the Oklahoma Institute for Child Advocacy recruited homes throughout the state of Oklahoma; the Kern County Superintendent of Schools recruited homes from Kern and San Luis Obispo counties, California; and Planned Parenthood of Maryland recruited homes from seven counties around Baltimore, Maryland. In each state, members of the study team worked with the state and local officials in the child welfare and juvenile justice systems to identify prospective homes. The selected homes were not intended to be a random or representative sample of all group homes in the targeted geographic areas. Rather, the team sought a nonprobability or purposive sample of homes with the capacity and commitment to support the study. A more detailed description of the recruitment process is provided in the PTC implementation report (Meckstroth et al. 2014). These recruitment efforts led to a final sample of 44 group homes spread across three participating states (Table II.1). The homes served a mix of youth from the child welfare and juvenile justice systems. Some of the homes were contracted with only one of the two systems. However, even in these homes, it is possible that some of the resident youth were “dually adjudicated,” meaning they had ongoing court involvement in both the juvenile justice and child welfare systems. The homes ranged in size from small cottage-style homes with eight or fewer residents to larger campus or dormitory settings with dozens of residents. Nearly 60 percent of the homes served only male residents, about 30 percent served only female residents, and a little less than 10 percent were coed. Most homes allowed residents to travel off site for public school attendance. A smaller number offered schooling on site as part of their core program services. The homes also had different policies concerning unsupervised leave for reasons other than school attendance. 9 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table II.1. Group home characteristics, by state California Maryland Oklahoma Total Total number 19 homes 10 homes 15 homes 44 homes (465 youth) (196 youth) (376 youth) (1,037 youth) Contracted to serve youth from . . . . Child welfare (CW) system 0 0 8 8 Juvenile justice (JJ) system 2 0 7 9 Both CW/JJ 17 10 0 27 Gender serveda . . . . Male 11 5 10 26 Female 7 3 4 14 Both 1 2 1 4 Size (number of beds) . . . . 8 or fewer 16 5 0 21 9 to 16 1 0 14 15 17 to 31 0 4 0 4 32 or more 2 1 1 4 Primary setting or type . . . . Cottage/home setting 17 5 0 24 Campus/dorm setting 1 4 15 18 Academy 1 1 0 2 Schooling . . . . On site 3 0 12 15 Off site (public schooling) 16 7 0 23 Both 0 3 3 6 Unsupervised leave allowedb . . . . Yes (contingent on approval/behavioral status) 17 10 0 27 No (not at all) 2 0 15 17 Source: Meckstroth et al. (2014). aHomes that serve pregnant and parenting teens were excluded from the evaluation. bExcluding a pass to visit home. “Unsupervised leave” refers to leave within the community where the group home is located. B. Random assignment procedures The study’s approach to random assignment was shaped by three main factors: 1. Need to assign youth in clusters. As discussed in Chapter I, it was not feasible to randomly assign youth to treatment and control groups within the same home both for logistical reasons and to avoid potential spillover or “contamination” effects. As a result, the study was designed as a cluster randomized trial that assigned all youth living in the same group home to the same research status. 2. Homes with multiple dormitories or living facilities. Some of the group homes recruited for the study had multiple dormitories or living facilities located on a single campus. Some of these dormitories or facilities were split by gender; others were split by different types of care or treatment services. In some cases, these dormitories or facilities were large and separate enough to randomly assign on their own as independent clusters. In part for this reason, some of the 44 group homes recruited for the study yielded more than one cluster of 10 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM youth for the purpose of random assignment. In other cases, the individual dormitories or facilities were too small to randomly assign on their own and were instead combined for random assignment. 3. Rolling recruitment of group homes. Because group homes were recruited into the study on a rolling basis, we did not wait until the end of the recruitment period to conduct random assignment. Rather, homes were randomly assigned in small pairs or groups as new homes were recruited into the study. This rolling approach to random assignment also allowed for the possibility of randomly assigning the same group home more than once, after the population of youth in the home had fully turned over. The opportunity to generate multiple clusters of youth from the same group home further boosted the number of randomized clusters. These features of the design ultimately yielded a sample of 80 clusters of youth for the purpose of random assignment across the 44 group homes (Figure II.1). To help ensure an even balance between the treatment and control groups, as well as to improve the precision of the impact estimates, we divided the clusters into 39 separate strata on the basis of location (California, Maryland, or Oklahoma); recruitment date; size; and gender of youth served. The first four homes recruited in California were randomly assigned as a stratum of four clusters. All other clusters were grouped into matched pairs of two clusters for random assignment. Within each matched pair or stratum, we randomly assigned an equal number of clusters to the treatment and control groups. None of the 80 clusters was lost to follow-up during the study period. C. Enrollment and retention of individual youth Within each home, all resident youth ages 13 to 18 were eligible to participate in the study, contingent on the study’s consent and assent requirements. The study required consent for each youth from a legally authorized representative. The identity of this representative varied across states, depending on state law and regulations. In California, the consent process required a signature from the individual lawyer or probation officer assigned to each youth or, in some cases, a biological parent. In Maryland, officials at the state level had authority to complete much of the consent process without involvement from individual social workers, biological parents, or other representatives. In Oklahoma, for youth in the child welfare system, the consent process required a signature from a caseworker. For youth in the juvenile justice system, consent required signature from a state official or, in some cases, a biological parent. In addition to these consent requirements, the study also required individual youth to assent to participating in the study. In each participating home, the study team began the consent and assent process in advance of random assignment to avoid any bias that might result from knowledge of the random assignment results. All youth enrolled at the beginning of the study were eligible to complete the later follow-up surveys. In the present report, we focus on data from the immediate post-test survey conducted at the end of the 10 program sessions. The study data collectors attempted to administer the surveys as close as possible to each home’s scheduled survey completion date. However, to accommodate scheduling issues and other practical constraints, the data collectors were also allowed a “window” of two weeks before and two weeks after the scheduled completion date to administer the surveys. This approach of allowing a short window for data collection has been used successfully in prior surveys of youth (Oman et al. 2009). Per the data collection protocol 11 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM established at the start of the study, any surveys completed outside the defined window were considered invalid and excluded from our subsequent analyses. A more detailed description of the data collection process is provided in Chapter III. Figure II.1. Flow of group homes and individual youth The resulting consent and assent rates for the study were very high (Figure II.1). The study team obtained consent for almost 98 percent of all youth eligible for enrollment. Only one youth declined to provide assent for the study. The sites achieved these high rates by taking a very individualized, labor intensive approach to consent and assent gathering. They began by working with the relevant state child welfare and juvenile justice agencies to gain buy-in for the study and identify the appropriate process of obtaining consent. For youth that required consent from an individual case worker, the sites had their data collectors visit the case workers in person to explain the study and request consent. For youth that required consent from a biological parent or legal guardian, the sites worked with the administrator in each participating group home to explore the possibility of incorporating the study’s consent form into the home’s existing intake or enrollment materials. The data collectors also relied on staff in the participating group homes 12 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM to help facilitate the process of gathering assent from study youth. The data collectors sought to make a personal connection with eligible participants by visiting the group homes, meeting with individual youth, and clearly describing the purpose of the study and what participation would involve. In part because most study youth were still living in the group homes at the time of the immediate post-test survey, response rates for the survey were very high. Among the 518 youth assigned to the treatment group, all but 49 completed the immediate post-test survey, for a response rate of 91 percent. Among 519 youth assigned to the control group, all but 35 completed the immediate post-test survey, for a response rate of 93 percent. We will report retention rates for the longer-term follow-up surveys as part of a future report. See Appendix A for a nonresponse analysis examining the characteristics of the youth who did not complete the immediate post-test survey. D. Baseline sample characteristics The demographic characteristics of the study sample (Table II.2) reflect the characteristics of the group homes recruited for the study. At the time of the baseline survey, before the start of the program, the participants ranged in age from 13 to 18 years old. The large majority were male (81 percent of the treatment group and 78 percent of the control group). The study participants had diverse racial/ethnic backgrounds. Hispanics represented the largest group (37 percent of the treatment group and 36 percent of the control), but the sample also had sizeable numbers of non-Hispanic blacks, whites, Native Americans, and Asians or Pacific Islanders. Most participants reported having lived in their current group home for less than six months. More than 4 in 10 participants were behind in school relative to their age, and more than a quarter of the study sample did not expect to graduate from high school. The study participants reported high rates of sexual activity and associated risk behaviors at baseline (Table II.3). Nearly 9 in 10 participants reported some lifetime experience with sexual intercourse, compared with only 11 percent who reported never having had sex. One in three participants reported having sexual intercourse for the first time before they turned 13, and a large majority reported having more than three lifetime sexual partners. One-third of the sample reported having had sex without a condom in the past three months. As a result of these risk behaviors, more than one-third of the sample also reported having been pregnant or gotten a partner pregnant at some point in their lives. All of these rates are substantially higher than national averages (Kann et al. 2014). 13 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table II.2. Baseline demographic and personal characteristics Treatment Control Measure group group Difference p-valuea Age in years (%) . . . . Less than 15 years old 11.3 12.2 -0.9 0.59 15 years old 16.6 16.5 0.1 . 16 years old 28.4 27.3 1.1 . 17 years old 32.0 35.1 -3.1 . 18+ years old 11.7 8.9 2.8 . Male (%) 81.2 77.7 3.5 0.18 Race/ethnicity (%) . . . . Hispanic 37.3 36.9 0.4 0.98 Non-Hispanic black 20.5 18.9 1.6 . Non-Hispanic white 20.0 21.2 -1.2 . Non-Hispanic Native Americanb 13.4 14.5 -1.1 . Non-Hispanic Asian or Pacific Islanderb 5.6 5.2 0.4 . Non-Hispanic other 3.2 3.4 -0.2 . Months in current group home (%) . . . . Less than 3 months 45.3 43.8 1.5 0.68 3–6 months 36.1 35.3 0.8 . More than 6 months 18.6 20.9 -2.3 . Behind grade level (%) 41.7 44.6 -2.9 0.36 Highest level of education likely to complete (%) . . . . Less than high school 26.9 29.4 -2.5 0.63 Graduate from high school 17.9 20.1 -2.2 . Some college or technical training 16.4 14.1 2.3 . Graduate from a 2-year college 8.7 7.5 1.2 . Graduate from a 4-year college 30.1 29.0 1.1 . Sample sizec 469 484 . . Source: Baseline surveys administered to study participants before the start of the program. aReported p-values are based on two-tailed t-tests for dichotomous measures and chi-squared tests for categorical measures. bRespondents could report their race in these categories alone or in combination with another race. cReported sample size is the number of participants who completed the immediate post-test survey and are included in the analysis; it does not account for item nonresponse for any measures included in the table. 14 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table II.3. Baseline exposure to information and risk behaviors Treatment Control Measure group group Difference p-valuea In past 12 months, received information on (%) . . . . Relationships, dating, marriage, family life 78.4 81.0 -2.6 0.33 Sexually transmitted infections 65.9 60.2 5.7 0.07 Talking to partner about sex or birth control 59.2 55.0 4.2 0.20 How to say no to sex 56.8 52.4 4.4 0.18 Abstinence from sex 56.0 52.3 3.7 0.25 Where to obtain birth control 46.7 49.6 -2.9 0.39 Methods of birth control 46.1 48.3 -2.2 0.49 Ever had sex (%) 89.5 88.6 0.9 0.65 Age at first sexual intercourse (%) . . . . <13 years old 33.4 33.5 -0.1 0.97 13 or 14 years old 38.8 37.7 1.1 . 15+ years old 17.1 17.4 -0.3 . Never had sexb 10.6 11.5 -0.9 . Lifetime number of sexual partners (%) . . . . 1–3 22.3 20.6 1.7 0.07 4–8 23.7 24.5 -0.8 . 9–14 24.8 18.7 6.1 . 15+ 18.0 24.3 -6.3 . Never had sexb 11.2 11.9 -0.7 . In past three months . . . . Had sex without condom (%) 33.5 34.3 -0.8 0.81 Had sex without any method of protection (%) 26.8 26.8 0.0 0.99 Ever been pregnant or gotten partner pregnant (%) 37.8 38.8 1.0 0.75 Sample sizec 469 484 . . Source: Baseline surveys administered to study participants before the start of the program. aReported p-values are based on two-tailed t-tests for dichotomous measures and chi-squared tests for categorical measures. bReported differences in rates of never had sex reflect differences in item nonresponse across measures. cReported sample size is the number of participants who completed the immediate post-test survey and are included in the analysis; it does not account for item nonresponse for any measures included in the table. E. Treatment and control conditions Youth assigned to the treatment group were offered the 10-session PTC program in their group homes. The program was delivered by teams of two trained facilitators from OICA, Kern County Superintendent of Schools, or Planned Parenthood of Maryland. The facilitators traveled to each group home to deliver the PTC program as a supplement to any other educational programs and services offered in the home. The facilitators worked with program staff in each home to determine the best schedule for delivering the program. Although some group homes chose to have the PTC program delivered once a week for 10 weeks, most completed the program in 5 weeks (two sessions a week). The shorter five-week schedule helped reduce the chances of youth leaving the home before the end of the program. Offering the program twice a week also helped to build rapport quickly between youth and the program facilitators. All program facilitators received extensive training in the PTC program. OICA provided an initial four-day in-person training for facilitators at the start of the study in spring 2011. 15 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Additional in-person training sessions were held again in fall 2012 and fall 2013 to review the program materials and discuss facilitators’ real-world experiences in delivering the program. Between these in-person training sessions, OICA convened periodic group discussions and supplemental training sessions via conference call. Facilitators had the opportunity to practice delivering the curriculum both during the training sessions and during an initial pilot of the evaluation. They received feedback and technical assistance on their performance from both OICA and a designated site coordinator in each state. Our accompanying implementation study of the PTC program found that the program was well implemented (Meckstroth et al. 2014). Despite frequent scheduling constraints, a majority of youth (66 percent) attended all 10 program sessions, and 83 percent attended at least 8 sessions. A large majority of youth gave positive ratings to the program facilitators and reported that the program had been useful. Youth were particularly engaged during the Q&A time at the end of each session. The session length sometimes made it difficult for youth to remain engaged throughout the 90-minute sessions. The facilitators began adding a snack break in the middle of sessions to offset the fatigue. In one site, facilitators noted that some youth, particularly boys, felt uncomfortable engaging in some of the role-playing activities. Such discomfort may have stemmed in part from delivering the program in single-gender settings, which required participants to conduct role plays and other program activities exclusively in same-sex pairs. However, such reports of discomfort were isolated and did not represent the majority experience. A more detailed description of the implementation successes and challenges is provided in the PTC implementation report (Meckstroth et al. 2014). Youth assigned to the control group were not offered the PTC program but retained access to any other existing community and group home services available to them. Interviews with program and group home staff in the control group found that youth generally received limited or no sexual and reproductive health education or services. In the few group homes that did provide other sources of sex education, these programs and services were typically limited to a single- session class or one-on-one counseling provided to youth on an individual, as-needed basis. The other types of educational programs and services offered to youth varied from home to home but typically included a mix of case management, individual or family therapy, substance abuse counseling and treatment, and various types of education and training programs. In both the California and Maryland sites, program and group home staff reported the availability of some outside, community-based services for youth in foster care and other out-of-home care settings. For example, both Bakersfield, California, and Baltimore, Maryland, have dedicated resource centers where youth can access family planning and related support services. In all three participating states, public schools are required to provide education on HIV/AIDS but do not have broader sexual health education requirements mandated for all students. See Meckstroth et al. (2014) for a more detailed description of the programs and services available to youth in the control group. 16 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM III. DATA, MEASURES, AND ANALYSIS The analysis is based on data from two rounds of surveys completed by youth in both the treatment and control groups. An initial baseline survey was administered roughly one week before homes in the treatment group began the program. An immediate post-test survey was administered roughly five or six weeks later, upon completion of the 10-session curriculum. The baseline surveys were administered on site in the group homes by trained data collectors as paper-and-pencil questionnaires. The immediate post-test survey was administered in groups or individually depending on whether the study participants were still residing in participating group homes. The questions and possible responses were read aloud by the data collectors to minimize any problems with reading comprehension or skipped questions. In the remainder of this chapter, we first describe the outcome measures constructed from the post-test survey. We then discuss the analytic methods used to assess the impacts of the PTC program on youth outcomes. For more detailed information on the measures, see Appendix B. A. Outcome measures Drawing on data from the immediate post-test survey, we constructed six groups of outcome measures, each corresponding to one of the study’s research questions: (1) youth exposure to information on reproductive health, pregnancy and STI prevention, and methods of protection; (2) youth knowledge of reproductive health, STIs, and methods of protection; (3) youth awareness of available health resources; (4) youth attitudes toward safe sex and methods of protection; (5) youth perceptions of self-empowerment or self-efficacy; and (6) youth intentions toward sexual activity and contraceptive use. These measures are summarized in Table III.1 and described in greater detail in the remainder of this section. 1. Exposure to information The immediate post-test survey included two questions designed to assess youth exposure to information on reproductive health, pregnancy and STI prevention, and methods of protection. The survey first asked youth whether they had received any information in the past 12 months on topics such as abstinence from sex, methods of birth control, and STIs. We used responses to this question to create a series of seven binary (yes/no) measures of whether youth had received information on each topic. The survey then asked youth how frequently they had received such information from different sources—in a school class, from a doctor or nurse, at a community center or other afterschool activity, in a group home, and so on. Response categories ranged from “never” to “10 or more times.” We used responses to this question to create a series of four continuous measures indicating the frequency of receiving information from each of the following sources: (1) group home; (2) school class; (3) community center or youth organization; and (4) doctor, nurse, or clinic. To examine the robustness of our results to an alternative coding scheme, we also examined the percentage of youth who reported receiving information from each source four or more times. 17 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table III.1. Outcome measures Measure Definition . Exposure to information Receipt of information in Series of seven binary variables: equals 1 if youth reported receiving information the past 12 months on each of seven topics; equals 0 if youth did not receive information. Frequency of receiving Series of four continuous variables: equals number of times youth reported information from different receiving information from a (1) group home; (2) school class; (3) community sources center or youth organization; or (4) doctor, nurse, or clinic. ., Knowledge Knowledge of reproductive Continuous index variable: sum of correct responses to four survey questions; anatomy and fertility variable ranges from 0 to 4, with higher values indicating greater knowledge. Knowledge of HIV and STIs Continuous index variable: sum of correct responses to seven survey questions; variable ranges from 0 to 7, with higher values indicating greater knowledge. Knowledge of methods of Continuous index variable: sum of correct responses to 10 survey questions; protection variable ranges from 0 to 10, with higher values indicating greater knowledge. . Awareness of available health resources Ability to find methods of Binary variable: equals 1 if youth reported feeling “very sure” he/she could find protection place to obtain methods of protection; equals 0 if youth did not feel very sure. Perceived access to Binary variable: equals 1 if youth reported he/she “strongly agrees” that condoms condoms are pretty easy to get; equals 0 if youth did not strongly agree. Perceived access to birth Binary variable: equals 1 if youth reported he/she “strongly agrees” that birth control control is pretty easy to get; equals 0 if youth did not strongly agree. . Attitudes toward safe sex and methods of protection General support for Continuous scale variable: average of responses to six survey questions; variable methods of protection ranges from 1 to 4, with higher values indicating stronger support. Perceived barriers to Continuous scale variable: average of responses to five survey questions; variable methods of protection ranges from 1 to 4, with higher values indicating fewer perceived barriers. Perceived self-efficacy to . avoid unprotected sex Perceived ability to Continuous scale variable: average of responses to three survey questions; communicate with partner variable ranges from 1 to 4, with higher values indicating greater perceived ability. Perceived ability to plan for Continuous scale variable: average of responses to four survey questions; and avoid unprotected sex variable ranges from 1 to 4, with higher values indicating greater perceived ability. Intentions toward sex . Intentions to have sexual Binary variable: equals 1 if youth reported definitely intending to have sexual intercourse intercourse in the next year; equals 0 if youth reported less intention. Intentions to use a condom Binary variable: equals 1 if youth reported definitely intending to use a condom if he/she has sexual intercourse; equals 0 if youth reported less intention. Intentions to use birth Binary variable: equals 1 if youth reported definitely intending to use birth control if control he/she has sexual intercourse; equals 0 if youth reported less intention. Note: Appendix B lists the specific survey questions used to construct each measure. 18 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM 2. Knowledge To measure youth knowledge of reproductive health, STIs, and methods of protection, we constructed three different outcomes: • Knowledge of reproductive anatomy and fertility. The survey asked youth a series of four knowledge questions about reproductive anatomy and fertility. For example, youth were asked to identify the part of the female body where the baby grows during pregnancy, with possible response categories of “cervix,” “uterus,” “vagina,” “ovary,” and “don’t know.” We summed the number of correct responses to these four knowledge questions to create an index of knowledge of reproductive anatomy and fertility. The index ranges from 0 to 4, with higher values indicating greater knowledge. • Knowledge of HIV and STIs. In a separate series of questions, the survey asked youth to respond to a series of seven true/false questions concerning general knowledge of HIV and STIs. For example, one question read “HIV destroys the immune system’s ability to fight off infections and diseases.” The survey asked youth to respond in one of three categories: “true,” “false,” or “don’t know.” We summed the number of correct responses to these seven questions to create an index of knowledge of HIV and STIs. The index ranges from 0 to 7, with higher values indicating greater knowledge. • Knowledge of methods of protection. The survey asked youth to respond to a series of 10 questions concerning their general knowledge of condoms and other methods of protection. For example, one true/false question read, “A condom can be used more than once.” We summed the number of correct responses to these 10 questions to create an index of knowledge of methods of protection. The index ranges from 0 to 10, with higher values indicating greater knowledge. 3. Awareness of available health resources The immediate post-test survey included three different questions concerning youth awareness of available health resources and where to get methods of protection. First, the survey asked youth how certain they felt about performing the following activity: “Find a place in your community to obtain methods of protection from pregnancy and STIs.” The four possible response categories ranged from “very sure” to “very unsure.” We used responses to this question to construct a binary measure comparing youth who felt “very sure” of their ability to youth who felt less certain. Second, the survey asked youth whether they agreed or disagreed with the following statement: “Condoms are pretty easy to get.” The four possible response categories ranged from “strongly agree” to “strongly disagree.” We constructed a binary measure comparing youth who said they “strongly agree” with this statement to youth who did not strongly agree. Third, the survey asked youth whether they agreed or disagreed with the statement “Birth control is pretty easy to get.” We constructed a binary measure comparing youth who said they “strongly agree” with this statement to youth who did not strongly agree. 4. Attitudes toward safe sex and methods of protection We constructed two measures of youth attitudes toward safe sex and methods of protection: • General support for methods of protection. The survey asked youth whether they agreed or disagreed with six statements concerning support for methods of protection, such as “Two 19 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM people having vaginal intercourse should use some method of protection if they are not ready for a child” and “Condoms should always be used if a person your age has sexual intercourse.” The four possible response categories ranged from “strongly agree” to “strongly disagree.” We averaged responses across the six items to create a composite scale of general support for methods of protection. Higher values on the scale indicate stronger levels of support. • Perceived barriers to methods of protection. The survey asked youth whether they agreed or disagreed with five statements concerning possible barriers to using methods of protection, such as “Condoms are a hassle to use” and “Birth control has too many negative side effects.” The four possible response categories ranged from “strongly agree” to “strongly disagree.” We averaged responses across the five items to create a composite scale of youth perceived barriers to methods of protection. Higher values on the scale indicate fewer perceived barriers. 5. Perceived self-efficacy to avoid unprotected sex To assess the program’s success in promoting a greater sense of self-empowerment or self- efficacy to avoid unprotected sex, we constructed two different outcomes: • Perceived ability to communicate with partner about sex. The survey asked youth how certain they felt about performing each of the following three activities: (1) “Tell your partner your feelings about what you do and do not want to do sexually,” (2) “Say “no” if your partner puts pressure on you to be involved sexually,” and (3) “Talk with your partner about methods of protection if you have sex with him/her.” The four possible response categories ranged from “very sure” to “very unsure.” We averaged responses across the three items to create a composite scale of perceived ability to communicate with a partner. Higher values on the scale indicate a greater perceived ability. • Ability to plan for and avoid unprotected sex. The survey asked youth a similar series of four questions about their perceived ability to avoid unprotected sex. In particular, youth were asked how certain they felt about performing activities such as “plan ahead to have some method of protection available.” The four possible response categories ranged from “very sure” to “very unsure.” We averaged responses across the four items to create a composite scale of perceived ability to plan for and avoid unprotected sex. Higher values indicate greater perceived ability. 6. Intentions toward sexual activity The survey included three questions concerning youth intentions toward sexual activity. The first question asked youth whether they intended to have sexual intercourse in the next year. The second asked youth whether they intended to use a condom (or have their partner use a condom) if they had sexual intercourse. The third question asked youth about their intention to use other effective methods of protection, such as birth control pills, the shot (Depo Provera), or intrauterine devices (IUDs). For each question, we constructed a binary measure comparing youth who said they “definitely” intended to engage in the activity to youth who reported a less strong intention. 20 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM B. Analytic approach Our approach to estimating the impacts of the PTC program on youth outcomes was shaped by three key features of our study design. First, as discussed in Chapter II, the design involved randomly assigning clusters of youth, not individual youth, to the treatment and control groups. This method of random assignment introduces a “design effect” that must be captured when estimating standard errors and conducting statistical significance tests. Second, in randomly assigning clusters of youth, we grouped them into matched pairs or strata and randomized an equal number of clusters to the treatment and control groups. This stratification must also be captured in the analysis because it limits the possible combinations of random assignment results. Third, the number of clusters randomly assigned is relatively large (N = 80), allowing for a range of possible analytic approaches. In particular, our sample size exceeds the cutoff of roughly 20 clusters per research arm needed to support a regression-based analysis of program impacts (Hayes and Mouton 2009; Donner and Klar 2000). To incorporate these design features, we used a multivariate regression framework to estimate the impact of the PTC program on each outcome measure. We used logistic regression for binary outcomes and ordinary least squares regression for continuous outcomes. Each regression model included a binary variable for treatment status as well as a series of indictor variables for the matched pairs or strata created for random assignment. To further improve the precision of the impact estimates and adjust for any marginal differences in the characteristics of the treatment and control groups, each regression model also included covariates for age, race, gender, and a baseline measure of the outcome variable (when available). To account for the design effect introduced by randomly assigning youth to the treatment and control conditions in groups, we calculated cluster-robust standard errors based on the 80 clusters of youth that were randomly assigned (White 1984; Liang and Zeger 1986). When reporting results from the logistic regression models, we calculated mean marginal effects to express the impact estimates as percentage-point differences in outcomes between the treatment and control groups. We adjusted the statistical significance tests (p-values) from our regression models to account for multiple hypothesis testing. As discussed earlier in this chapter, we constructed multiple outcomes to answer each of our research questions. For example, we constructed three separate outcomes to assess the impacts of the PTC program on youth knowledge of reproductive health, STIs, and methods of protection. Unless taken into account, this multiplicity can increase the chances of making a false discovery and lead to spurious claims about the program’s effectiveness. We adjusted for multiple hypothesis testing using a procedure outlined by Hothorn et al. (2008) and Schochet (2009a). In brief, this procedure involves adjusting the reported p-value for each outcome to account for other tests conducted within the same “family” of related measures. We began by estimating a separate multivariate regression model for each outcome as planned. To calculate the p-values for the impact estimates, we then compared the estimated t-statistic from each regression model against critical values from a multivariate t distribution determined by the combination of regression models estimated within the same family of outcomes. Similar to other methods of adjusting for multiple hypothesis testing, this procedure yields a 5 percent false positive rate across all outcomes within the same family. However, the procedure is less restrictive than other common adjustment methods (for example, the Bonferroni correction) because it also accounts for any correlation in test statistics across outcomes within the same family. 21 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM We made this adjustment separately for each of the six groups of outcome measures described earlier in this chapter (and presented in Table III.1). That is, we adjusted the p-values accounting for multiple outcomes within each of the six groups of measures, but not for multiple outcomes measured across the different groups. We followed this approach because each group of outcomes aligns with a different research question. We base our substantive conclusions for each question only on the corresponding group of outcome measures. The number of outcomes measured in other groups has no bearing on our substantive conclusions for each question and therefore does not warrant an additional adjustment for multiple hypothesis testing. We conducted several tests to assess the robustness of our results to alternative analytic decisions: (1) calculating p-values without an adjustment for multiple hypothesis testing, (2) estimating impacts using a mixed effects or multi-level regression model, and (3) estimating impacts using a “cluster-level” analysis instead of an individual-level regression model. We report findings from these tests in Appendix C. 22 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM IV. RESULTS The youth assigned to receive the PTC program had more favorable outcomes on nearly all of the short-term measures examined. They were significantly more likely to report receiving information on reproductive health, pregnancy and STI prevention, and methods of protection. They reported increased knowledge of reproductive health, STIs, and methods of protection, as well as increased awareness of available health resources. They reported more supportive attitudes for methods of protection and a greater sense of self-empowerment or self-efficacy to avoid unprotected sex. We did not find a statistically significant difference between the treatment and control groups in the percentage of youth who said they intended to have sex, but we did find a difference in the percentage of youth who said they intended to use a condom when having sex. We detail these findings in the remainder of this chapter. A. Exposure to program information The PTC program had large and statistically significant impacts on youth exposure to information on reproductive health and sexual education topics (Table IV.1). Among youth assigned to the treatment group, more than 90 percent reported receiving information on such topics as where to obtain birth control (94.1 percent); talking to a partner about sex or birth control (93.4 percent); how to say no to sex (91.6 percent); relationships, dating, marriage, and Table IV.1. Impacts on youth exposure to program information Treatment Control Measurea group group Difference p-value Percentage of youth that reported receiving information on the following topics: . . . . Where to obtain birth control 94.1 55.1 39.0** <.01 Talking to partner about sex or birth control 93.4 62.8 30.6** <.01 How to say no to sex 91.6 59.6 32.0** <.01 Relationships, dating, marriage, family life 91.4 76.9 14.5** <.01 Methods of birth control 90.3 57.2 33.0** <.01 Abstinence from sex 89.1 60.1 29.0** <.01 Sexually transmitted infections 88.8 64.9 23.9** <.01 Number of times youth received information from each of the following sources:b . . . . Group home 5.3 2.7 2.6** <.01 School class 4.0 2.5 1.5** 0.01 Community center or youth organization 2.9 1.8 1.1** 0.01 Doctor, nurse, or clinic 3.0 2.9 0.2 1.00 Source: Youth surveys administered by the study team. Notes: For each outcome, the numbers in the columns labeled “treatment group” and “control group” are regression-adjusted predicted values of outcomes at the immediate post-test survey. Sample sizes accounting for item nonresponse range from 728 to 948 depending on the measure. Reported p-values are adjusted for multiple outcomes measured within a single domain. See Chapter III for a more detailed description of the analytic methods. aQuestions refer to information received in the 12 months prior to survey administration. See Appendix B for a more detailed description of each measure. bThe regression models for these outcomes do not control for baseline values of the outcomes because the questions were asked only at the time of the follow-up survey. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. 23 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM family life (91.4 percent); and methods of birth control (90.3 percent). Fewer youth in the control group had received such information. For example, less than 60 percent of youth in the control group reported receiving information on how to say no to sex (59.6 percent), methods of birth control (57.2 percent), or where to obtain birth control (55.1 percent). All of the reported differences are statistically significant. Youth assigned to the treatment group also reported receiving reproductive health information more frequently than did those in the control group. In particular, youth in the treatment group reported receiving reproductive health information in their group homes on average 5.3 times over the past 12 months, an average nearly double the frequency reported by youth in the control group (2.7 times in the past 12 months). The difference between groups remained statistically significant when we instead looked at the percentage of youth who reported receiving information rather than the average frequency of receipt (results not shown). For youth in the treatment group, the average reported frequency of receiving information in a group home (5.3 times) is lower than the total number of sessions offered by the PTC program (10 sessions). However, we did not expect perfect alignment in these numbers. As explained in Chapter II, scheduling and logistical constraints prevented some youth from attending all sessions. In addition, not all of the PTC sessions have an exclusive focus on reproductive health topics. Some of the sessions also cover broader topics such as goal setting and general decision making (see program description in Chapter I). In part for these reasons, we did not expect all youth in the treatment group to self-report on the survey questionnaire receiving reproductive health information in a group home 10 or more times. One unexpected finding is that youth in the treatment group were also more likely to report receiving reproductive health information in school or from a community center or youth organization (Table IV.1). For example, the average youth in the treatment group reported receiving reproductive health information in school 4.0 times in the past 12 months, compared to 2.5 times for the average youth in the control group. The most likely explanation for this finding is that youth interpreted the PTC program as being delivered as part of a school class or community program, perhaps especially in homes that provided on-site schooling. However, another possibility is that participating in the PTC program made youth more likely to notice or seek out reproductive health information in settings beyond their group homes, such as in school or from community organizations. The program did not impact the reported frequency of receiving information from a doctor, nurse, or clinic. B. Knowledge and awareness of health resources The PTC program had statistically significant impacts on youth knowledge of reproductive health, STIs, and methods of protection (Table IV.2). On our four-item index of knowledge of reproductive anatomy and fertility, we found an average score of 2.75 for youth in the treatment group and 2.39 for youth in the control group. This difference represents a roughly 15 percent increase in knowledge for youth in the treatment group relative to the control group (0.35/2.39 = 15 percent). Youth in the treatment group also had higher scores both for knowledge of HIV and STIs and for knowledge of methods of protection. For knowledge of HIV and STIs, we found average scores of 5.33 for the treatment group and 4.51 for the control group—a difference favoring the treatment group by 18 percent (0.82/4.51 = 18 percent). For knowledge of methods 24 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM of protection, we found average scores of 7.90 for the treatment group and 6.19 for the control group—a difference favoring the treatment group by 28 percent (1.71/6.19 = 28 percent). All of these differences are statistically significant. The PTC program also had impacts on youth awareness of available health resources. Among youth assigned to the treatment group, nearly two-thirds (66.0 percent) reported feeling “very sure” where to get methods of protection (Table IV.2). By comparison, about half the youth in the control group (51.1 percent) reported the same level of confidence. The 14.9 percentage point difference in rates (66.0 percent versus 51.1 percent) is statistically significant. For our main impact findings, we did not find statistically significant differences in the percentage of youth who were very likely to agree with the statements “condoms are pretty easy to get” and “birth control is easy to get.” However, we did find statistically significant impacts on these outcomes in one of our sensitivity tests (see Appendix C). Table IV.2. Impacts on knowledge and awareness of health resources Treatment Control Measure group group Difference p-value Knowledge of reproductive anatomy and fertility (index score, ranges 0–4)a 2.75 2.39 0.35** <.01 Knowledge of HIV and STIs (index score, ranges 0–7)b 5.33 4.51 0.82** <.01 Knowledge of methods of protection (index score, ranges 0-10)c 7.90 6.19 1.71** <.01 Percentage of youth reporting they are “very sure” where to get methods of protection 66.0 51.1 14.9** <.01 Percentage of youth reporting they “strongly agree” that Condoms are pretty easy to get 64.1 58.2 5.9 0.31 Birth control is easy to get 31.7 27.8 3.9 0.57 Source: Youth surveys administered by the study team. Notes: For each outcome, the numbers in the columns labeled “treatment group” and “control group” are regression-adjusted predicted values of outcomes at the immediate post-test survey. Sample sizes accounting for item nonresponse range from 906 to 953 depending on the measure. Reported p-values are adjusted for multiple outcomes measured within a single domain. See Chapter III for a more detailed description of the analytic methods and Appendix B for a more detailed description of each measure. aThis index counts the number of correct responses to a series of four knowledge questions. Possible values range from 0 to 4, with higher values indicating a greater number of correct responses. bThis index counts the number of correct responses to a series of seven knowledge questions. Possible values range from 0 to 7, with higher values indicating a greater number of correct responses. cThis index counts the number of correct responses to a series of 10 knowledge questions. Possible values range from 0 to 10, with higher values indicating a greater number of correct responses. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. The findings show room for further improvement in knowledge and awareness among all study youth. In percentage terms, youth in the treatment group provided correct answers on average to 69 percent of the questions on knowledge of reproductive health and anatomy (average score of 2.75 out of 4 items), 76 percent of the questions on knowledge of HIV and STIs (average score of 5.33 out of 7 items), and 79 percent of the questions on knowledge of methods of protection (average score of 7.90 out of 10 items). These average scores are higher 25 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM than those reported for the control group, but they also show room for further improvement in knowledge, even among youth in the treatment group. Similarly, although youth in the treatment group reported greater awareness of available health resources relative to the control group, not everyone on the treatment group felt confident in his or her ability to find and access these resources. C. Attitudes and intentions The PTC program had favorable impacts on youth attitudes toward methods of protection (Table IV.3). On a scale of general support ranging from 0 to 4, with higher values indicating more supportive attitudes, we found average scale scores of 3.48 for youth in the treatment group and 3.32 for youth in the control group. The difference of 0.16 scale points represents an increase favoring the treatment group of roughly 5 percent (0.16/3.32 = 5 percent). We did not find a statistically significant difference in the scale measuring perceived barriers to methods of protection (average scale score of 2.50 for the treatment group versus 2.44 for the control group). Table IV.3. Impacts on youth attitudes and intentions Treatment Control Measure group group Difference p-value General support for methods of protection (scale score, ranges 1–4)a 3.48 3.32 0.16** <.01 Perceived barriers to methods of protection (scale score, ranges 1–4)b 2.50 2.44 0.06 0.33 Perceived ability to communicate with partner (scale score, ranges 1–4)c 3.54 3.28 0.26** <.01 Perceived ability to plan for and avoid unprotected sex (scale score, ranges 1–4)d 3.35 3.08 0.26** <.01 Percentage of youth reporting intentions to engage in the following behaviors in the next 12 months: . . . . Have sexual intercourse 69.9 68.5 1.4 1.00 Use condoms if having sex 57.6 43.9 13.7** <.01 Use other protection method if having sex 45.0 39.7 5.3 0.67 Source: Youth surveys administered by the study team. Notes: For each outcome, the numbers in the columns labeled “treatment group” and “control group” are regression-adjusted predicted values of outcomes at the immediate post-test survey. Sample sizes accounting for item nonresponse range from 915 to 936 depending on the measure. Reported p-values are adjusted for multiple outcomes measured within a single domain. See Chapter III for a more detailed description of the analytic methods and Appendix B for a more detailed description of each measure. aThis scale averages responses to six questions on support for methods of protection. Possible values range from 1 to 4, with higher values indicating greater support. The inter-item reliability (alpha) equals 0.76. bThis scale averages responses to five questions on perceived barriers to methods of protection. Possible values range from 1 to 4, with higher values indicating fewer perceived barriers. The inter-item reliability (alpha) equals 0.68. cThis scale averages responses to three questions on perceived ability to communicate with partner. Possible values range from 1 to 4, with higher values indicating greater perceived ability. The inter-item reliability (alpha) equals 0.81. dThis scale averages responses to four questions on perceived ability to plan for and avoid unprotected sex. Possible values range from 1 to 4, with high values indicating greater perceived ability. The inter-item reliability (alpha) equals 0.81. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. 26 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Consistent with one of the key program messages, youth in the treatment group reported a greater sense of self-efficacy or self-empowerment to avoid unprotected sex (Table IV.3). On our scale of perceived ability to communicate with a partner, we found average scale scores of 3.54 for the treatment group and 3.28 for the control group. The difference of 0.26 scale points represents an increase favoring the treatment group by roughly 8 percent (0.26/3.28 = 8 percent). We found similar results for our scale of perceived ability to plan for and avoid unprotected sex: average scale scores were 3.35 for the treatment group and 3.08 for the control group. For both scales, the reported differences are statistically significant. The findings for the intentions measures are also generally consistent with the program messages. Slightly more than two-thirds of both the treatment and control groups said they planned to have sex in the next 12 months (69.9 percent for the treatment group and 68.5 percent for the control group). This finding is not surprising given that most sample members had already had sex before enrolling in the study (see Chapter II). In addition, although the program teaches abstinence as the only way to fully avoid the risk of pregnancy and STIs, it does not promote or value abstinence over other methods of protection. The program places a strong emphasis on condom use—both as a method to reduce pregnancy risk and especially for protection against STIs. Our program impact estimates reflect this strong emphasis: youth in the treatment group were more likely than those in the control group to say they planned to use a condom when having sex (57.6 percent versus 43.9 percent). Youth in the treatment group were also more likely to say they planned to use other measures of protection, such as birth control pills or IUDS, though the difference from the control group is not statistically significant (45.0 percent versus 39.7 percent). 27 This page has been left blank for double-sided copying. INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM V. DISCUSSION AND CONCLUSION This report presents interim impacts of the POWER Through Choices program, a comprehensive sexual health education curriculum designed specifically for youth in foster care and other out-of-home care settings. Prior research indicates that youth in out-of-home care are at particularly high risk for teen pregnancy, STIs, and associated sexual risk behaviors (Dworsky and Courtney 2010). However, many of these youth have limited access to timely sexual health education and services. The PTC program is one of the only sexual health education programs designed to address the needs and risks specific to this population. The study focused on a predominately male, high-risk sample of teens aged 13 to 18 living in residential group homes. Upon enrolling in the study, nearly 9 in 10 participants reported some lifetime experience with sexual intercourse, and more than one third of the sample reported having been pregnant or gotten a partner pregnant at some point in their lives. Four in 10 sample members were behind grade level in school, and more than one-quarter did not expect to complete high school. Such educational deficits are often linked with teen pregnancy and may put these youth at further risk. Our findings show that the PTC program was successful in changing participants’ short-term outcomes measured just after the end of the 10 program sessions. Consistent with the program logic model and goals, the PTC program was effective in increasing youth exposure to information on reproductive health, pregnancy and STI prevention, and methods of protection. The program increased youth knowledge of reproductive health, STIs, and methods of protection. It also increased youth awareness of available health resources and where to get methods of protection. The PTC program led to increased support for safe sex and a greater sense of self-empowerment or self-efficacy to avoid unprotected sex. It also increased youth intentions to avoid unprotected sex by using condoms. These findings are consistent with our earlier study that documented the implementation of the PTC program as part of this large-scale demonstration project and evaluation (Meckstroth et al. 2014). The implementation study found that the PTC program was well implemented and delivered in accordance with the program model. The sessions were well attended and delivered by qualified, trained facilitators. The interim impact findings described in the present report extend that story by finding that youth also received the intended program messages and achieved the desired improvements in knowledge, awareness, and attitudes. This interim report does not answer the question of whether the program’s success in changing short-term outcomes results in longer-term changes in sexual risk behaviors. Because this report focused on data from an immediate post-test survey administered shortly after the end of the 10 program sessions, we focused on very proximal outcomes such as exposure to program information, knowledge of reproductive health topics, attitudes, and intentions. A future report will examine the longer-term impacts of the PTC program on sexual risk behaviors measured 6 and 12 months after study enrollment. In that future report, we will also examine whether the observed program impacts on knowledge, awareness, attitudes, and intentions were sustained in the months after the program ended. 29 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM The findings presented in this report may not generalize to other parts of the country or to all youth living in foster care or other out-of-home care settings. By design, this evaluation focused on a specific sample of high-risk youth living in a select sample of residential group homes in three states (California, Maryland, and Oklahoma). The selected homes were not drawn as a representative sample but instead on the basis of their capacity and commitment to support the program and study activities. These design decisions were governed by the steep practical and logistical challenges involved in conducting a large-scale demonstration project and evaluation among youth in out-of-home care settings. It may be possible for organizations to successfully replicate the positive results presented in this report when implementing the program on a smaller scale or within a single community. However, factors such as the capacity and commitment of the host organization, quality of implementation, and characteristics of the target population likely all play a role in determining how well our results generalize to other settings. The results of this study indicate that the PTC program has the potential for positive short-term impacts when implemented with high quality in a supportive environment. Successfully replicating these positive outcomes likely requires the same close attention to the design and delivery of the program. 30 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM REFERENCES Becker, M. G., and R. P. Barth. “POWER Through Choices: The Development of a Sexuality Education Curriculum for Youths in Out-of-Home Care.” Child Welfare, vol. 79, no. 3, 2000, pp. 269–282. Boonstra, H. “Teen Pregnancy Among Young Women in Foster Care: A Primer.” Guttmacher Policy Review, vol. 14, no. 2, 2011, pp. 8–19. Casanueva, C., S. Tueller, K. Smith, M. Dolan, and H. Ringeisen. “NSCAW II Wave 3 Tables.” OPRE Report #2013-43. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2014. Crottogini, J., J. Fajardo, E. Villasenor, and K. Ward. “Understanding the Sexual Health Needs of Bay Area Foster and Former Foster Care Youth: A Collaborative Project Between Health Initiatives for Youth (HIFY) and San Francisco State University.” San Francisco, CA: San Francisco State University, 2008. Donner, A., and N. Klar. Design and Analysis of Cluster Randomization Trials in Health Research. London, UK: Arnold, 2000. Dworsky, A., and M. E. Courtney. “The Risk of Teenage Pregnancy Among Transitioning Foster Youth: Implications for Extending State Care Beyond Age 18.” Children and Youth Services Review, vol. 32, no. 10, 2010, pp. 1351–1356. Dworsky, A., C. Smithgall, and M.E. Courtney. “Supporting Youth Transitioning out of Foster Care, Issue Brief 1: Education Programs.” OPRE Report #2014-66. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2014. Freundlich, M. Time Running Out: Teens in Foster Care. New York, NY: Children’s Rights, 2003. Goesling, B., S. Colman, C. Trenholm, M. Terzian, and K. Moore. “Programs to Reduce Teen Pregnancy, Sexually Transmitted Infections, and Associated Sexual Risk Behaviors: A Systematic Review.” Journal of Adolescent Health, vol. 54, no. 5, 2014, pp. 499–507. Hayes, R. J., and L. H. Moulton. Cluster Randomized Trials. Chapman & Hall/CRC Interdisciplinary Statistics Series. Boca Raton, FL: CRC Press, 2009. Hothorn, T., F. Bretz, and P. Westfall. “Simultaneous Inference in General Parametric Models.” Biometrical Journal, vol. 50, no. 3, 2008, pp. 346–363. Hudson, A. L. “Where Do Youth in Foster Care Receive Information About Preventing Unplanned Pregnancy and Sexually Transmitted Infections?” Journal of Pediatric Nursing, vol. 27, no. 5, 2012, pp. 443–450. Kann, L., S. Kinchen, S. L. Shanklin, et al. “Youth Risk Behavior Surveillance—United States, 2013.” Morbidity and Mortality Weekly Report, vol. 63, no. 4, 2014, pp. 1-168. 31 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Kinsey, D. and A. Schlösser. “Interventions in Foster and Kinship Care: A Systematic Review.” Clinical Child Psychology, vol. 18, no. 3, 2012, pp. 429-463. Lee, S., S. Aos, and M. Miller. “Evidence-Based Programs to Prevent Children from Entering and Remaining in the Child Welfare System: Benefits and Costs for Washington.” Olympia, Washington: Washington State Institute for Public Policy, 2008. Leonard, S. M., and K. Suellentrop. “Integrating Pregnancy Prevention in Child Welfare: Supporting Youth in Care to Prevent Teen Pregnancy.” Policy and Practice, August 2013, pp. 24–31. Liang, Kung-Yee, and Scott L. Zeger. “Longitudinal Data Analysis Using Generalized Linear Models.” Biometrika, vol. 73, no. 1, 1986, pp. 13–22. Love, L. T., J. McIntosh, M. Rosst, and K. Tertzakian. “Fostering Hope: Preventing Teen Pregnancy Among Youth in Foster Care.” Washington, DC: National Campaign to Prevent Teen Pregnancy, 2005. Meckstroth, A., M. Barry, B. Keating, E. Kisker, and K. Andrews. “Addressing Teen Pregnancy Risks for Youth Living in Out-of-Home Care: Implementing POWER Through Choices 2010.” Washington, DC: Office of Adolescent Health, U.S. Department of Health and Human Services, 2014. Oman, R. F., S. K. Vesely, et al. “Methodological Considerations in a Community-Based Longitudinal Study.” American Journal of Health Behavior, vol. 33, no. 1, 2009, pp. 58-68. Putnam-Hornstein, E., and B. King. “Cumulative Teen Birth Rates Among Girls in Foster Care at Age 17: An Analysis of Linked Birth and Child Protection Records from California.” Child Abuse & Neglect, vol. 38, no. 4, 2014, pp. 698–705. Schochet, P. Z. “An Approach for Addressing the Multiple Testing Problem in Social Policy Impact Evaluations.” Evaluation Review, vol. 33, no. 6, 2009a, pp. 539–567. Schochet, P. Z. “The Estimation of Average Treatment Effects for Clustered RCTs of Education Interventions.” NCEE 2009-0061 rev. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education, 2009b. White, H. Asymptotic Theory for Econometricians. Orlando, FL: Academic Press, 1984. 32 APPENDIX A NONRESPONSE ANALYSIS This page has been left blank for double-sided copying. INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM This appendix examines the characteristics of the study participants lost to follow-up at the time of the immediate post-test survey (N = 84). Because these participants did not complete a post-test survey, we excluded them from the program impact analyses presented in this report. We compared the baseline characteristics of these nonrespondents to the larger sample of youth that completed the post-test survey and thus make up our analysis sample (N = 953). We compared the groups on a total of 19 measures of demographic characteristics, personal characteristics, exposure to information, and sexual risk behaviors (Tables A.1 and A.2). On the basis of this comparison, we found only three statistically significant differences: the nonrespondents had a lower proportion of males (70.2 percent versus 79.4 percent), were more likely to report having received information on STIs (77.1 percent versus 63.0), and were more likely to report having received information on talking to a partner about sex or birth control (71.1 percent versus 57.0 percent). We also found differences between the groups in demographic and personal characteristics such as age, months in current group home, and behind in grade level. However, these differences were not statistically significant at the 5 percent level. Table A.1. Baseline demographic and personal characteristics Measure Respondents Nonrespondents Difference p-valuea Age in years (%) . . . . Less than 15 years old 11.8 13.1 -1.3 0.32 15 years old 16.6 10.7 5.9 . 16 years old 27.8 27.4 0.4 . 17 years old 33.6 32.1 1.5 . 18+ years old 10.3 16.7 -6.4 . Male (%) 79.4 70.2 9.2* 0.05 Race/ethnicity (%) . . . . Hispanic 37.1 36.1 1.0 0.99 Non-Hispanic black 19.7 21.7 -2.0 . Non-Hispanic white 20.6 19.3 1.3 . Non-Hispanic Native Americanb 13.9 14.5 -0.6 . Non-Hispanic Asian or Pacific Islanderb 5.4 4.8 0.6 . Non-Hispanic other 3.3 3.6 -0.3 . Months in current group home (%) . . . . Less than 3 months 44.5 33.8 10.7 0.17 3–6 months 35.7 41.3 -5.6 . More than 6 months 19.8 25.0 -5.2 . Behind grade level (%) 43.2 33.7 9.5 0.10 Highest level of education likely to complete (%) . . . . Less than high school 28.2 21.4 6.8 0.51 Graduate from high school 19.0 17.9 1.1 . Some college or technical training 15.2 15.5 -0.3 . Graduate from a 2-year college 8.1 7.1 1.0 . Graduate from a 4-year college 29.5 38.1 -8.6 . Sample sizec 953 84 . . Source: Baseline surveys administered to study participants before the start of the program. aReported p-values are based on two-tailed t-tests for dichotomous measures and chi-square tests for categorical measures. bRespondents could report their race in these categories alone or in combination with another race. cReported sample size does not account for item nonresponse for any measures included in the table. *Significantly different from zero at the .05 level, two-tailed test. A.3 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Table A.2. Baseline exposure to information and risk behaviors Measure Respondents Nonrespondents Difference p-valuea In past 12 months, received information on (%) . . . . Relationships, dating, marriage, family life 79.7 74.4 5.3 0.25 Sexually transmitted infections 63.0 77.1 -14.1** 0.01 Talking to partner about sex or birth control 57.0 71.1 -14.1** 0.01 How to say no to sex 54.6 57.8 -3.2 0.57 Abstinence from sex 54.1 61.0 -6.9 0.23 Where to obtain birth control 48.2 58.5 -10.3 0.07 Methods of birth control 47.2 55.0 -7.8 0.18 Ever had sex (%) 89.0 94.1 -5.1 0.15 Age at first sexual intercourse (%) . . . . <13 years old 33.4 36.1 -2.7 0.54 13 or 14 years old 38.2 38.6 -0.4 . 15+ years old 17.3 19.3 -2.0 . Never had sexb 11.1 6.0 5.1 . Lifetime number of sexual partners (%) . . . . 1–3 21.4 24.1 -2.7 0.65 4–8 24.1 27.9 -3.8 . 9–14 21.7 20.3 1.4 . 15+ 21.2 21.5 -0.3 . Never had sexb 11.6 6.3 5.3 . In past three months . . . . Had sex without condom (%) 33.9 30.5 3.4 0.53 Had sex without any method of protection (%) 26.9 27.2 -0.3 0.96 Ever been pregnant or gotten partner pregnant (%) 38.3 45.2 -6.9 0.21 Sample sizec 953 84 . . Source: Baseline surveys administered to study participants before the start of the program. aReported p-values are based on two-tailed t-tests for dichotomous measures and chi-square tests for categorical measures. aReported differences in rates of never had sex reflect differences in item nonresponse across measures. bReported sample size does not account for item nonresponse for any measures included in the table. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. A.4 APPENDIX B OUTCOME MEASURES This page has been left blank for double-sided copying. INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM This appendix provides more detailed information on the outcome measures used to assess the impacts of the PTC program. As discussed in Chapter III, we examined program impacts on six groups of outcome measures: (1) youth exposure to information on reproductive health, pregnancy and STI prevention, and methods of protection; (2) youth knowledge of reproductive health, STIs, and methods of protection; (3) youth awareness of available health resources; (4) youth attitudes toward safe sex and methods of protection; (5) youth perceptions of self- empowerment or self-efficacy; and (6) youth intentions toward sexual activity and contraceptive use. In the remainder of this appendix, we detail how we constructed the outcome measures for each of these groups. A. Exposure to information The immediate post-test survey included two questions designed to assess youth exposure to information on reproductive health, pregnancy and STI prevention, and methods of protection. The survey first asked youth whether they had received any information in the past 12 months on the following topics: • Relationships, dating, marriage, or family life • Abstinence from sex • Methods of birth control • Where to get birth control • Sexually transmitted infections, also known as STIs • How to talk to your partner about whether to have sex or whether to use birth control • How to say no to sex For each topic, youth answered “yes” or “no” to having received information in the past 12 months. We used responses to this question to create a series of seven binary (yes/no) measures of whether youth had received information on each topic (yes = 1, no = 0). For each topic, youth who did not respond to the question were coded as missing. The survey then asked youth how frequently they had received such information from different sources. For this analysis, we focused specifically on the following four sources: • Group home • School class • Community center, youth organization, or afterschool activity • Doctor, nurse, or clinic For each source, the survey asked youth to respond in one of the following four categories: “never,” “1–3 times,” “4–9 times,” or “10 or more times.” We assigned numerical values to each response category as follows: “never” = 0; “1–3 times” = 2 (the midpoint value); “4–9 times” = 6.5 (the midpoint value); “10 or more times” = 10. We used these numerical values as a continuous measure of the frequency of receiving information from each of the following sources: (1) group home; (2) school class; (3) community center or youth organization; and B.3 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM (4) doctor, nurse, or clinic. For each source, youth who did not respond to the question were coded as missing. B. Knowledge The survey asked youth 21 questions about their knowledge of reproductive health, methods of protection, and STIs (see Table B.1). The questions were a mix of true/false and multiple choice. We divided the questions into three groups corresponding to different topics taught by the program: (1) knowledge of reproductive anatomy and fertility, (2) knowledge of HIV and STIs, and (3) knowledge of methods of protection. Within each group, we totaled the number of correct responses to create an index score. The maximum score on each index is determined by the total number of questions: a maximum score of 4 for the index of knowledge of reproductive anatomy and fertility, 7 for the index of knowledge of HIV and STIs, and 10 for the index of knowledge of methods of protection. The minimum score of 0 indicates no correct responses. Higher values on each index indicate greater knowledge. We coded missing values and “don’t know” as incorrect responses. Table B.1. Questions used to construct knowledge outcomes Question Response categories . Knowledge of reproductive anatomy and fertility The part of the female body where a baby grows during Cervix, uterus*, vagina, ovary, don’t know pregnancy is the ___? The part of the male body that produces sperm is the___? Testicles*, urethra, penis, prostate, don’t know When is it a possible for a female to become pregnant? The first time she has sex, when she is ovulating, when her partner withdraws before ejaculating, all of the above*, don’t know If a young couple has had unprotected sex a few times and a True, false*, don’t know pregnancy did not occur, then they do not have to worry about her getting pregnant. Knowledge of HIV and STIs . HIV destroys the immune system’s ability to fight off infections True*, false, don’t know and diseases. You cannot tell if a person has HIV by looking at them. True*, false, don’t know HIV is the only sexually transmitted infection that is incurable. True, false*, don’t know All sexually active individuals are at risk for getting HIV. True*, false, don’t know All sexually transmitted infections (STIs) can be cured. True, false*, don’t know You can get the same sexually transmitted infection (STI) twice. True*, false, don’t know You can get a sexually transmitted infection (STI) from having True*, false, don’t know oral sex. Knowledge of methods of protection . Of the following statements about methods of protection, which You can get them with a prescription from a one is false? doctor; you can buy them at a local store in the drug or pharmacy section; some require a prescription and others do not, depending on the type of method; you must have your parent’s/guardian’s permission to get them if you are under age 18*; don’t know B.4 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM Question Response categories A sexually active girl can become pregnant if she forgets to take True*, false, don’t know her birth control bills for several days in a row. Using a condom can help prevent HIV. True*, false, don’t know A condom can be used more than once. True, false*, don’t know If a condom is used, a young man should be careful how he pulls True*, false, don’t know out. Latex condoms are 100% effective in preventing pregnancy and True, false*, don’t know STIs (including HIV). Which of the following methods of protection offers the most Depo-Provera (the shot), vaginal film, protection against HIV and other STIs? condom (rubber)*, birth control pill, don’t know What is the safest and most effective method for avoiding Birth control pill, condom (rubber), Depo- pregnancy and sexually transmitted infections (STIs)? Provera (the shot), abstinence (not having sex)*, don’t know Which one of the following methods listed below do you think is Condom (rubber)*, Depo-Provera (the the most effective for preventing pregnancy? shot)*, rhythm (safe period by calendar), patch (Ortho evra)*, birth control pill*, withdrawal method, don’t know Which one of the following methods listed below do you think is Condom (rubber), Depo-Provera (the shot), the least effective for preventing pregnancy? rhythm (safe period by calendar)*, patch (Ortho evra), birth control pill, withdrawal method*, don’t know * Indicates correct answers. C. Awareness of available health resources The immediate post-test survey included three different questions concerning youth awareness of available health resources and where to get methods of protection. First, the survey asked youth how certain they felt about performing the following activity: “Find a place in your community to obtain methods of protection from pregnancy and STIs.” The possible response categories were: “very sure,” “somewhat sure,” “somewhat unsure,” and “very unsure.” We used responses to this question to construct a binary measure comparing youth who felt “very sure” of their ability to youth who felt “somewhat sure,” “somewhat unsure,” or “very unsure.” We coded youth who did not respond to the question as missing values. Second, the survey asked youth whether they agreed or disagreed with the following statement: “Condoms are pretty easy to get.” The possible response categories were “strongly agree,” “agree,” “disagree,” and “strongly disagree.” We used responses to this question to construct a binary measure comparing youth who said they “strongly agree” with this statement to youth who said they “agree,” “disagree,” or “strongly disagree.” We coded youth who did not respond to the question as missing values. Third, the survey asked youth whether they agreed or disagreed with the statement “Birth control is pretty easy to get.” The possible response categories were “strongly agree,” “agree,” “disagree,” and “strongly disagree.” We used responses to this question to construct a binary measure comparing youth who said they “strongly agree” with this statement to youth who said they “agree,” “disagree,” or “strongly disagree.” We coded youth who did not respond to the question as missing values. B.5 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM For the purpose of our analysis, we treated each question as a separate outcome and estimated program impacts separately for each. We had initially considered the possibility of combining the three questions into a single summary scale or index. However, we ultimately decided against this approach, first, because the response categories were not uniform across the three questions and, second, because an exploratory factor analysis suggested that the combined measure had low reliability. D. Attitudes toward safe sex and methods of protection We constructed two separate measures of youth attitudes toward sex safe and methods of protection: (1) general support for methods of protection and (2) perceived barriers to methods of protection. For the measure of general support for methods of protection, the survey asked youth whether they agreed or disagreed with each of the following six statements: • Two people having vaginal intercourse should use some method of protection if they are not ready for a child. • Using a method of protection is very important. • Condoms should always be used if a person your age has sexual intercourse. • Condoms are important to make sex safer. • Birth control should always be used if a person your age has sexual intercourse. • Birth control is important to make sex safer. For each statement, the possible response categories were “strongly agree,” “agree,” “disagree,” and “strongly disagree.” We used responses to these items to create a composite scale of general support for methods of protection. We calculated the scale for youth who responded to at least five of the six survey questions. For youth meeting this criterion, we calculated their scale score by taking the average value of the non-missing items. The scale ranges from 1 to 4, with higher values indicating strong support. Youth who responded to fewer than five items were coded as missing on the scale. For the measure of perceived barriers to methods of protection, the survey asked youth whether they agreed or disagreed with each of the following five statements: • Condoms are a hassle to use. • Condoms decrease sexual pleasure. • Condoms make sex less exciting. • Birth control is a hassle to use. • Birth control has too many negative side effects. For each statement, the possible response categories were “strongly agree,” “agree,” “disagree,” and “strongly disagree.” We used responses to these items to create a composite scale B.6 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM of perceived barriers to methods of protection. We calculated the scale for youth who responded to at least four of the five survey questions. For youth meeting this criterion, we calculated their scale score by taking the average value of the non-missing items. The scale ranges from 1 to 4, with higher values indicating fewer perceived barriers. Youth who responded to fewer than four items were coded as missing on the scale. We constructed these measures using principal-component factor analysis and reliability testing. We used the results of the initial factor analysis to identify the two distinct scales. We then examined the factor loadings for the individual items to identify the specific questions to include with each scale. We included any item with a factor loading of at least 0.5. We also examined the internal reliability of each scale. The measure of general support for methods of protection had high reliability (alpha coefficient = 0.76). The measure of perceived barriers had slightly lower, and moderate, reliability (alpha coefficient = 0.68). To avoid any correlation between the effects of the PTC program and the process used to construct the outcome measures, we conducted the factor analysis and reliability testing on the full analytic sample using responses from the baseline (not post-test) survey. E. Perceived self-efficacy to avoid unprotected sex We constructed two separate measures related to youth perceived self-efficacy: (1) perceived ability to communicate with a partner and (2) perceived ability to plan for and avoid unprotected sex. For the measure of perceived ability to communicate with a partner, the survey asked youth how certain they felt about performing each of the following three activities: • Tell your partner your feelings about what you do and do not want to do sexually. • Say “no” if your partner puts pressure on you to be involved sexually and you do not want to be involved sexually. • Talk with your partner about methods of protection if you have sex with him/her. The possible response categories were “very sure,” “somewhat sure,” “somewhat unsure,” and “very unsure.” We used responses to these items to create a composite scale of perceived ability to communicate with a partner. We calculated the scale for only those youth who responded to all three questions. For youth meeting this criterion, we calculated their scale score by taking the average value of the three items. The scale ranges from 1 to 4, with higher values indicating greater perceived ability. Youth who did not responded to all three questions were coded as missing on the scale. For the measure of perceived ability to plan for and avoid unprotected sex, the survey asked youth how certain they felt about performing each of the following four activities: • Insist on using a method of protection if you have sex and want to use a method of protection. • Stop and use a method of protection once you are turned on. • Plan ahead to have some method of protection available. B.7 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM • Resist having sex with your partner if he/she does not want to use a method of protection. The possible response categories were “very sure,” “somewhat sure,” “somewhat unsure,” “very unsure.” We used responses to these items to create a composite scale of perceived ability to plan for and avoid unprotected sex. We calculated the scale for only those youth who responded to at least three of the four survey questions. For youth meeting this criterion, we calculated their scale score by taking the average value of the non-missing items. The scale ranges from 1 to 4, with higher values indicating greater perceived ability. Youth who responded to fewer than three items were coded as missing on the scale. We constructed these measures using principal-component factor analysis and reliability testing. We used the results of the initial factor analysis to identify the two distinct scales. We then examined the factor loadings for the individual items to identify the specific questions to include with each scale. We included any item with a factor loading of at least 0.5. We also examined the internal reliability of each scale. The measure of perceived ability to communicate with a partner and the measure of perceived ability to plan for and avoid unprotected sex both had high reliability (alpha coefficient = .81 for both scales). To avoid any correlation between the effects of the PTC program and the process used to construct the outcome measures, we conducted the factor analysis and reliability testing on the full analytic sample using responses from the baseline (not post-test) survey. F. Intentions toward sexual activity The survey included three questions concerning youth intentions toward sexual activity: • Do you intend to have sexual intercourse in the next year? • If you have sexual intercourse in the next year, do you intend to use (or have your partner use) a condom? • If you have sexual intercourse in the next year, do you intend to use (or have your partner use) any of these methods of birth control? Birth control pills, the shot (Depo-Provera), the patch, the ring (NuvaRing), IUD (Mirena or Paragard), implants (Implanon). The response categories for all three questions were “yes, definitely”; “yes, probably”; “no, probably not”; and “no, definitely not.” For each question, we constructed a binary measure comparing youth who responded “yes, definitely” to youth who responded in any of the other three categories. For each question, youth who did not respond to the question were coded as missing. We estimated impacts of the PTC program separately for each measure. B.8 APPENDIX C SENSITIVITY ANALYSES This page has been left blank for double-sided copying. INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM The main findings presented in Chapter IV of this report derive from a particular set of analytic decisions about how to estimate program impacts on youth outcomes in the context of a cluster randomized controlled trial design. In particular, we estimated program impacts using a multivariate regression framework. We accounted for the clustered design by calculating cluster- robust standard errors for regression models estimated with individual-level data. In addition, we adjusted the statistical significance tests (p-values) for the impact estimates to account for multiple hypothesis testing—in particular, our decision to construct multiple outcomes for each study research question. Although we view these analytic decisions as the most suitable and defensible for the specifics of our design, we also assessed the robustness of our results to three alternative analytic decisions: 1. No adjustment for multiple hypothesis testing. As discussed in Chapter III, for the main findings presented in this report, we adjusted the reported p-value for each outcome measure to account for other tests conducted within the same family of outcomes. For example, for the measures of youth knowledge of reproductive health and methods of protection, we adjusted the reported p-values to account for our three separate knowledge indices. Adjusting for separate but correlated indices had the practical effect of increasing the reported p-values for certain outcomes, which may lead to more conservative conclusions about program effectiveness. As a sensitivity test, we also calculated unadjusted p-values that treat each outcome as an independent test. 2. Estimation of mixed-effects regression models. In our main findings, we adjusted for the clustered design by estimating regression models with cluster-robust standard errors. This approach adjusts the standard errors and reported p-values for the impact estimates to account for any correlation in outcomes among youth from the same cluster. As an alternative approach, some researchers choose to account for clustering by estimating mixed-effects regression models, also known as hierarchical linear modeling. The two approaches have slightly different theoretical interpretations and can also yield differences in the reported impact estimates and statistical significance tests (Hayes and Moulton 2009; Schochet 2009b). To estimate the sensitivity of our results to this alternative estimation approach, we estimated mixed-effects regression models using the programming commands—mixed (for continuous outcomes) and melogit (for dichotomous outcomes)— provided in the Stata statistical software package. For dichotomous outcomes, the p-values outputted by the melogit command are associated with the estimated odds ratios from the regression model, not the mean marginal effects we present in the report. We did not adjust p-values for multiple hypothesis testing when conducting this sensitivity test. 3. Cluster-level analysis. Both mixed-effects regression models and the main regression approach we used for this report involve estimating regression models with the individual- level survey data collected from study youth. Prior methodological research suggests that estimating such individual-level regressions models with clustered data has the potential to yield unreliable standard errors and statistical significance tests, especially in studies with only a small number of clusters (Hayes and Moulton 2009; Donner and Klar 2000). For our evaluation of the PTC program, the number of clusters exceeds the cutoff of roughly 20 clusters per research arm needed to support an individual-level regression approach. However, to test the sensitivity of our results to an alternative modeling approach, we also C.3 INTERIM IMPACTS OF THE POWER THROUGH CHOICES PROGRAM conducted a “cluster-level” analysis that does not have the same large sample size requirements (Hayes and Moulton 2009). In brief, this cluster-level analysis involves using the individual-level data to calculate aggregate mean outcomes for each cluster in the study and then comparing the mean cluster-level outcomes between the treatment and control groups. The statistical significance test of the resulting impact estimates is determined using a paired t-test with degrees of freedom equal to the number of matched pairs or strata used for random assignment. We did not adjust p-values for multiple hypothesis testing when conducting this sensitivity test. The results of these analyses (Tables C.1 through C.3) showed that our findings are generally robust to alternative analytic decisions. For all outcomes, the direction of the impact estimate is consistent between our main findings presented in Chapter IV and each of the three sensitivity tests. Our conclusions about the statistical significance of the study findings are robust in two of the three sensitivity tests: the unadjusted p-values and the cluster-level analysis. For the third sensitivity test—the mixed-effects regression models—we reach more favorable conclusions about the statistical significance of the study findings for four outcomes: (1) the percentage of youth who strongly agree that condoms are pretty easy to get (Table C.2), (2) the percentage of youth who strongly agree that birth control is easy to get (Table C.2), (3) the index of perceived barriers to methods of protection (Table C.3), and (4) the percentage of youth reporting intentions to use “other” measures of birth control in the next 12 months (Table C.3). For all these outcomes, the reported impact estimate is statistically significant at the 5 percent level in the mixed-effects regression models. C.4 Table C.1. Sensitivity of impacts on youth exposure to program information Unadjusted Mixed-effects Cluster-level Main findings p-values regression model analysis Measurea Diff. p-value Diff. p-value Diff. p-value Diff. p-value . . . . . . . . Percentage of youth that reported receiving information on the following topics: Where to obtain birth control 39.0** <.01 39.0** <.01 39.0** <.01 39.2** <.01 Talking to partner about sex or birth control 30.6** <.01 30.6** <.01 30.4** <.01 29.7** <.01 How to say no to sex 32.0** <.01 32.0** <.01 32.4** <.01 30.8** <.01 Relationships, dating, marriage, family life 14.5** <.01 14.5** <.01 14.0** <.01 18.3** <.01 Methods of birth control 33.0** <.01 33.0** <.01 32.8** <.01 32.7** <.01 Abstinence from sex 29.0** <.01 29.0** <.01 28.8** <.01 28.8** <.01 Sexually transmitted infections 23.9** <.01 23.9** <.01 23.9** <.01 26.0** <.01 . . . . . . . Number of times youth received information from each of the following sources:b . Group home 2.6** <.01 2.6** <.01 2.6** <.01 2.7** <.01 School class 1.5** 0.01 1.5** <.01 1.2** <.01 1.0** <.01 Community center or youth organization 1.1** 0.01 1.1** <.01 1.2** <.01 1.4** <.01 Doctor, nurse, or clinic 0.2 1.00 0.2 0.75 0.2 0.45 0.2 0.56 C.5 Source: Youth surveys administered by the study team. Notes: For each outcome, the numbers in the columns labeled “Diff.” indicate the regression-adjusted average difference in outcomes between the treatment and control groups. See the accompanying Appendix C text for a more detailed description of the estimation methods. aQuestions refer to information received in the 12 months prior to survey administration. See Appendix B for a more detailed description of each measure. bThe regression models for these outcomes do not control for baseline values of the outcomes because the questions were asked only at the time of the follow-up survey. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. Table C.2. Sensitivity of impacts on knowledge and awareness Unadjusted Mixed-effects Cluster-level Main findings p-values regression model analysis Measure Diff. p-value Diff. p-value Diff. p-value Diff. p-value Knowledge of reproductive anatomy and fertility (index score, ranges 0–4)a 0.35** <.01 0.35** <.01 0.36** <.01 0.35** <.01 Knowledge of HIV and STIs (index score, ranges 0–7)b 0.82** <.01 0.82** <.01 0.81** <.01 0.94** <.01 Knowledge of methods of protection (index score, ranges 0–10)c 1.71** <.01 1.71** <.01 1.73** <.01 1.86** <.01 Percentage of youth reporting they are “very sure” where to get methods of protection 14.9** <.01 14.9** <.01 15.7** <.01 12.2** 0.01 . . . . . . . Percentage of youth reporting they “strongly agree” that . Condoms are pretty easy to get 5.9 0.31 5.9 0.10 6.2** 0.01 7.5 0.10 Birth control is easy to get 3.9 0.57 3.9 0.26 4.9** 0.01 4.3 0.18 Source: Youth surveys administered by the study team. Notes: For each outcome, the numbers in the columns labeled “Diff.” indicate the regression-adjusted average difference in outcomes between the treatment and control groups. See the accompanying Appendix C text for a more detailed description of the estimation methods. C.6 aThis index counts the number of correct responses to a series of four knowledge questions. Possible values range from 0 to 4, with higher values indicating a greater number of correct responses. bThis index counts the number of correct responses to a series of seven knowledge questions. Possible values range from 0 to 7, with higher values indicating a greater number of correct responses. cThis index counts the number of correct responses to a series of 10 knowledge questions. Possible values range from 0 to 10, with higher values indicating a greater number of correct responses. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. Table C.3. Sensitivity of impacts on youth attitudes and intentions Unadjusted Mixed-effects Cluster-level Main findings p-values regression model analysis Measure Diff. p-value Diff. p-value Diff. p-value Diff. p-value General support for methods of protection (scale score, ranges 1–4)a 0.16** <.01 0.16** <.01 0.17** <.01 0.21** <.01 Perceived barriers to methods of protection (scale score, ranges 1–4)b 0.06 0.33 0.06 0.21 0.06 0.05 0.03 0.53 Perceived ability to communicate with partner (scale score, ranges 1–4)c 0.26** <.01 0.26** <.01 0.25** <.01 0.23** <.01 Perceived ability to plan for and avoid unprotected sex (scale score, ranges 1–4)d 0.26** <.01 0.26** <.01 0.26** <.01 0.27** <.01 Percentage of youth reporting intentions to engage in the following behaviors in the next 12 months: . . . . . . . . Have sexual intercourse 1.4 1.00 1.4 0.82 1.5 0.73 3.8 0.26 Use condoms if having sex 13.7** <.01 13.7** <.01 14.0** <.01 9.2* 0.04 Use other protection method if having sex 5.3 0.67 5.3 0.33 5.8* 0.02 1.4 0.76 C.7 Source: Youth surveys administered by the study team. Notes: For each outcome, the numbers in the columns labeled “Diff.” indicate the regression-adjusted average difference in outcomes between the treatment and control groups. See the accompanying Appendix C text for a more detailed description of the estimation methods. aThis scale averages responses to six questions on support for methods of protection. Possible values range from 1 to 4, with higher values indicating greater support. The inter-item reliability (alpha) equals 0.76. bThis scale averages responses to five questions on perceived barriers to methods of protection. Possible values range from 1 to 4, with higher values indicating fewer perceived barriers. The inter-item reliability (alpha) equals 0.68. cThis scale averages responses to three questions on perceived ability to communicate with partner. Possible values range from 1 to 4, with higher values indicating greater perceived ability. The inter-item reliability (alpha) equals 0.81. dThis scale averages responses to four questions on perceived ability to plan for and avoid unprotected sex. Possible values range from 1 to 4, with higher values indicating greater perceived ability. The inter-item reliability (alpha) equals 0.81. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. http://www.hhs.gov/ash/oah/