OAH Evaluation Report Impact Report from the Evaluation of Adolescent Pregnancy Prevention Approaches Teen Options to Prevent Pregnancy July 2016 Purpose statement: This report documents the final findings from a large-scale demonstration project and evaluation of Teen Options to Prevent Pregnancy, an 18-month intervention designed specifically for pregnant and parenting adolescents with three key components: (1) telephone-based care coordination, (2) facilitated access to contraception, and (3) access to a social worker. The study reports the impacts of the program on sexual risk behaviors and repeat pregnancy at the time of program completion. July 2016 Recommended citation: Rotz, Dana, Dara Lee Luca, Brian Goesling, Elizabeth Cook, Kelly Murphy, and Jack Stevens. “Final Impacts of the Teen Options to Prevent Pregnancy Program.” Cambridge, MA: Mathematica Policy Research, July 2016. This document is available at http://www.hhs.gov/ash/oah/oah-initiatives/evaluation/federal-led-evaluation/ ppa-study.html Prepared for OAH under contract number: HHSP233201450030A Authored by: Dana Rotz Dara Lee Luca Brian Goesling Mathematica Policy Research Elizabeth Cook Kelly Murphy Child Trends Jack Stevens Nationwide Children’s Hospital FINAL IMPACTS OF THE T.O.P.P. PROGRAM ACKNOWLEDGEMENTS This evaluation was made possible only through the hard work and support of many people and organizations. First, we wish to acknowledge the support of staff from the Office of Adolescent Health (OAH) in the U.S. Department of Health and Human Services (HHS). We especially thank Amy Farb, the OAH project officer for the Evaluation of Adolescent Pregnancy Prevention Approaches (PPA) study, who has provided outstanding support and guidance throughout the project, and Alexandra Warner of OAH, who provided valuable support in preparing this report. We also thank staff from the Family and Youth Services Bureau within the Administration for Children and Families at HHS, especially Itege Bailey and Marc Clark. At OhioHealth, a special thanks goes to Robyn Lutz, whose careful management of the delivery of the Teen Options to Prevent Pregnancy (T.O.P.P.) program was instrumental to the study’s success. We thank Dr. Ngozi Osuagwu and Angela Taylor at OhioHealth for their commitment and contributions to the project. We also thank Gary Stofle, who provided expert training and technical assistance to OhioHealth program staff on motivational interviewing. Finally, we thank the large team of talented and dedicated nurse educators, social workers, administrators, and other staff from OhioHealth and Nationwide Children’s Hospital who supported the program’s implementation and the evaluation. At Mathematica Policy Research, we thank Christopher Trenholm and Alan Hershey for their guidance and support during the initial study design phase. We also thank Kristin Velyvis and Melissa Thomas for supporting the data collection effort, Kimberly Smith for providing valuable feedback on the report, Malik Khan Mubeen for excellent programming assistance, Dale Anderson for editorial assistance, and Sharon Clark for producing the report. At Child Trends, we thank Kristin Moore and Jennifer Manlove for useful comments on an earlier draft of the report. Finally, we extend our greatest thanks and gratitude to the teen mothers who agreed to participate in the study. Their lives motivated the work reflected in the pages of this report. Dana Rotz Dara Lee Luca Brian Goesling Elizabeth Cook Kelly Murphy Jack Stevens iii This page has been left blank for double-sided copying. FINAL IMPACTS OF THE T.O.P.P. PROGRAM CONTENTS I. INTRODUCTION.............................................................................................................................. 1 A. The T.O.P.P. program ............................................................................................................... 2 1. Telephone-based care coordination ................................................................................... 2 2. Facilitated access to contraceptive services ....................................................................... 3 3. Access to a social worker ................................................................................................... 3 B. Summary of interim impact findings .......................................................................................... 4 C. Research questions ................................................................................................................... 6 II. STUDY DESIGN .............................................................................................................................. 9 A. Sample enrollment and retention............................................................................................... 9 B. Baseline sample characteristics .............................................................................................. 11 C. Treatment and control conditions ............................................................................................ 14 III. DATA, MEASURES, AND ANALYSIS ........................................................................................... 19 A. Outcome measures ................................................................................................................. 19 1. Repeat pregnancy ............................................................................................................. 21 2. Other pregnancy-related outcomes .................................................................................. 22 3. Contraceptive use ............................................................................................................. 23 4. Unprotected sex ................................................................................................................ 23 5. Other sexual risk behaviors .............................................................................................. 24 6. Education .......................................................................................................................... 25 B. Analytic approach .................................................................................................................... 25 IV. RESULTS ....................................................................................................................................... 27 A. Repeat pregnancy and other pregnancy-related outcomes .................................................... 27 B. Contraceptive use and unprotected sex .................................................................................. 28 C. Other sexual risk behaviors ..................................................................................................... 30 D. Educational outcomes ............................................................................................................. 31 V. DISCUSSION AND CONCLUSIONS ............................................................................................. 33 REFERENCES ............................................................................................................................................ 37 APPENDIX A NONRESPONSE ANALYSIS .............................................................................................. 39 APPENDIX B DATA AND MEASURES ..................................................................................................... 45 APPENDIX C 18-MONTH IMPACTS ON MEDIATING FACTORS ........................................................... 55 APPENDIX D SENSITIVITY ANALYSIS.................................................................................................... 61 v This page has been left blank for double-sided copying. FINAL IMPACTS OF THE T.O.P.P. PROGRAM TABLES I.1 Key 6-month findings from the T.O.P.P. interim impact report ........................................................ 5 II.1 Baseline sociodemographic characteristics ................................................................................... 12 II.2 Baseline exposure to reproductive health information ................................................................... 13 II.3 Baseline sexual behaviors ............................................................................................................. 14 II.4 Receipt of T.O.P.P. services .......................................................................................................... 15 II.5 Receipt of T.O.P.P. services, by participant race/ethnicity ............................................................ 16 II.6 Receipt of T.O.P.P. services, by participant relationship status .................................................... 17 III.1 Outcome measures ........................................................................................................................ 20 IV.1 Impacts on repeat pregnancy and related outcomes..................................................................... 28 IV.2 Impacts on contraceptive use and unprotected sex ...................................................................... 29 IV.3 Impacts on other sexual risk behaviors.......................................................................................... 31 IV.4 Impacts on educational outcomes ................................................................................................. 32 FIGURES II.1 Overview of sample enrollment and retention ............................................................................... 11 IV.1 Use of any effective birth control method in the past three months ............................................... 30 vii This page has been left blank for double-sided copying. FINAL IMPACTS OF THE T.O.P.P. PROGRAM I. INTRODUCTION This report presents final impact findings from a demonstration project and evaluation of the Teen Options to Prevent Pregnancy (T.O.P.P.) program, an 18-month clinic-based intervention that aims to reduce rapid repeat pregnancy and promote healthy birth spacing among adolescent mothers. Teen mothers have the highest risk of having a closely spaced repeat pregnancy (Copen et al. 2015), with more than one in three recently pregnant teens experiencing a repeat pregnancy within two years of a previous birth or abortion (Baldwin and Edelman 2013). The majority of these pregnancies are reported as unintended (Mosher et al. 2012). Teen mothers who experience rapid repeat pregnancies are at significantly greater risk of having a stillbirth or preterm birth than mothers who delay subsequent childbearing (Conde-Agudelo et al. 2006). They are also less likely to stay in or complete high school, to work or maintain economic self-sufficiency, and to have children who exhibit school readiness when older (Klerman 2004). The T.O.P.P. program was developed to reduce these risks and to promote more healthy birth spacing among teen mothers through a unique combination of individualized support services. In an earlier report, we found promising short-term effects of the T.O.P.P. program in increasing rates of contraceptive use and reducing rates of unprotected sex among teen mothers (Smith et al. 2015). Drawing on data from a rigorous random assignment evaluation involving nearly 600 low-income adolescent mothers from the Columbus, Ohio, area, our earlier report found that the T.O.P.P. program succeeded in increasing participants’ use of highly effective long-acting reversible contraceptive (LARC) methods and reducing the prevalence of unprotected sexual intercourse. In additional analyses, we found evidence suggesting that the T.O.P.P. program achieved these impacts primarily by increasing participants’ exposure to information on birth control methods and sources and increasing participants’ access to contraceptive services. We measured these interim impacts after participants had taken part in the first 6 months of the full 18-month T.O.P.P. intervention. In the present report, we extend these results by examining the program’s longer-term impacts, as measured at the end of the 18-month intervention. Our analyses focus on whether the T.O.P.P. program achieved its primary goal of reducing rates of rapid repeat pregnancy among low-income adolescent mothers. We also revisit the outcomes assessed in our earlier interim report, to determine the longer-run impacts of T.O.P.P. on contraceptive use and sexual risk behavior and to see whether the favorable impacts we observed 6 months after program enrollment persist 18 months after enrollment. The evaluation has involved a unique collaboration and partnership among several organizations. The demonstration and evaluation was originally designed by the OhioHealth Research and Innovation Institute and Nationwide Children’s Hospital, in collaboration with the OhioHealth Community Partnerships Department, which houses the T.O.P.P. program. In fall 2010, OhioHealth received competitive federal grant funding for the evaluation through the Personal Responsibility Education Innovation Strategies grant program within the Administration for Children and Families (ACF) of the U.S. Department of Health and Human Services (HHS). In winter 2011, the T.O.P.P. evaluation was selected as one of seven sites to participate in the Evaluation of Adolescent Pregnancy Prevention Approaches (PPA), a major federal effort to expand available evidence on effective ways to prevent and reduce pregnancy and related sexual risk behaviors among teens in the United States. The PPA study is conducted 1 FINAL IMPACTS OF THE T.O.P.P. PROGRAM by Mathematica Policy Research and its partners, Child Trends and Twin Peaks Partners, LLC, under contract with the Office of Adolescent Health (OAH) within HHS. Participation in the PPA study provided the T.O.P.P. evaluation additional resources to support data collection and analysis. In addition, researchers from the PPA evaluation team have collaborated with the OhioHealth Research and Innovation Institute and Nationwide Children’s Hospital to refine the evaluation design, support data collection, and plan the analysis. The report is divided into five chapters. In the remainder of this chapter, we provide a more detailed description of the T.O.P.P. program, summarize key findings from our earlier interim report, and enumerate the research questions for the present 18-month impact analysis. In Chapters II and III, we describe the study design, data, and analytic methods. In Chapter IV, we present findings from the final impact analysis, and in Chapter V we summarize and discuss the implications of the results. A. The T.O.P.P. program T.O.P.P. was developed by staff at OhioHealth—a large faith-based health system in Columbus, Ohio—to reduce rapid repeat pregnancy and promote healthy birth spacing among adolescent mothers in the Columbus area. The program features three main components: (1) telephone-based care coordination, (2) facilitated access to contraceptive services, and (3) referrals to social support services. These services are delivered to program participants by nurse educators and a program social worker over an 18-month period. We briefly summarize these program features in the remainder of this section. We provided a more detailed description of the program in our earlier interim impact report (Smith et al. 2015) and an accompanying implementation report (Meckstroth and Berger 2014). 1. Telephone-based care coordination The telephone-based care coordination component of T.O.P.P. involves one-on-one telephone motivational interviewing sessions with a trained nurse educator. Motivational interviewing is an individualized, client-centered, collaborative communication and counseling approach designed to promote behavior change. It focuses on a person’s goals and motivation to change and emphasizes self-efficacy and the relationship between current behaviors and future goals (Barnet et al. 2007; Hettema et al. 2005). For T.O.P.P., nurse educators use motivational interviewing to elicit information about past experiences with and beliefs about contraception and pregnancy, encourage participants to examine their own knowledge base about contraception, provide individualized education about contraceptive methods based on participants’ preferences and interests, and help guide participants toward birth control methods that can be used effectively and consistently. For T.O.P.P., nurse educators deliver motivational interviewing sessions by telephone with a recommended frequency of approximately once per month throughout the 18-month intervention, with greater call frequency during the initial months of the program and periods in which participants are actively seeking and adopting new forms of birth control. This component of T.O.P.P. shares some similarity with home visiting programs such as the long-standing Nurse Family Partnership (NFP) program, which uses trained nurses to provide one-on-one home visits to first-time, low-income mothers over an extended period from early in the woman’s pregnancy and until her child turns 2 years old. T.O.P.P. is similar to NFP and other home visiting programs 2 FINAL IMPACTS OF THE T.O.P.P. PROGRAM both in its relatively long duration and in the approach of using trained nurse educators to provide individualized, one-on-one services to program participants. However, unlike these other programs, T.O.P.P. is designed to rely more on telephone calls and focus more narrowly on promoting healthy birth spacing and use of effective contraception. 2. Facilitated access to contraceptive services In addition to expanding knowledge and motivation to prevent rapid repeat pregnancy, the T.O.P.P. program model also aims to reduce barriers to effective and consistent contraceptive use, such as limited access to reproductive health services. In recent years, the St. Louis-based Contraceptive CHOICE project has received considerable attention for its positive results in increasing rates of LARC use by offering LARCs at no cost (Secura et al. 2014). The CHOICE project focused on female adolescents and women who were not currently using a contraceptive method or had expressed a willingness to switch methods. For our evaluation of T.O.P.P., not all participants were seeking birth control at the time of enrollment. However, they all had access to family planning services, if interested, at no charge through Medicaid. In addition, the T.O.P.P. program also sought to address such practical barriers as a lack of reliable or convenient transportation and poor access to a regular, convenient health care provider. T.O.P.P. provides program participants access to both a program clinic and transportation services. The T.O.P.P. clinic provides participants direct access to contraceptive services, including LARCs. The clinic was originally designed as a mobile clinic that could be stationed in different parts of the program service areas. However, due to insufficient attendance, the mobile clinic was discontinued after a few months and replaced by a stationary clinic in the T.O.P.P. offices. For participants without a reliable or convenient source of transportation, T.O.P.P. also offers access to a van service. For our evaluation of T.O.P.P., the van service was operated by T.O.P.P. program staff free of charge to program participants. The service was available for transportation to any local health care provider, not just the T.O.P.P. program clinic. To increase awareness of the types of contraceptive methods and services available through health care providers, the T.O.P.P. nurse educators also plan to conduct at least one individual, in-person visit with program participants at the participants’ homes, or, in some cases, in community settings. For these in-person visits, the nurse educators bring a “contraceptive bag” with them as a tool to help educate participants about different contraceptive options. The bag contains informational flyers and pamphlets, as well as a range of birth control devices for participants to see and touch, including NuvaRing (birth control ring), Nexplanon (birth control implant), and intrauterine devices (IUDs). The nurse educators ideally schedule these visits within the first few months of the full 18-month program. 3. Access to a social worker The T.O.P.P. program gives participants access to a program social worker who, based on an initial psychosocial assessment of program participants and subsequent identification of service needs by the nurse educators, can refer participants to appropriate support services. The initial risk assessment and ongoing referral services attempt to address a range of other barriers to adoption of and adherence to an effective birth control regimen, such as maternal depression, domestic violence, poverty, and homelessness. 3 FINAL IMPACTS OF THE T.O.P.P. PROGRAM B. Summary of interim impact findings To assess the impacts of the T.O.P.P. program, we conducted a random assignment evaluation involving almost 600 pregnant or postpartum low-income adolescent women recruited from seven OhioHealth women’s clinics and the postpartum units of five OhioHealth hospitals. The study team recruited participants into the study if they met the following eligibility criteria: ages 10 to 19, at least 28 weeks pregnant or less than 9 weeks postpartum, and enrolled in Medicaid. Among the eligible women who agreed to participate in the study, we randomly assigned approximately half to the treatment group, who were eligible to receive the T.O.P.P. program, and half to the control group, who were not eligible for the program. In both research groups, we administered a baseline survey prior to random assignment and follow-up surveys 6 and 18 months after study enrollment. In an earlier report, we used data from the baseline and first follow-up surveys to assess the short-term impacts of the program about 6 months after study enrollment (Smith et al. 2015). Because that earlier report examined the interim impacts of T.O.P.P. after program participants had received the first 6 months of the full 18-month program, we focused on measuring program impacts on intermediate outcomes such as contraceptive use, sexual risk behaviors, exposure to information on reproductive health topics, knowledge about the effectiveness of contraceptive methods, and attitudes and intentions toward pregnancy and birth control use. We did not examine pregnancy outcomes for the interim report because of the short time horizon. The key findings from the interim report are summarized in Table I.1 and discussed in greater detail in the remainder of this section. We found that T.O.P.P. had a large and statistically significant impact on two of its primary targeted outcomes: use of LARC methods and the prevalence unprotected sex (Table I.1). About six months after program enrollment, 38.3 percent of the participants in the treatment group reported using a LARC method in the past three months, compared to 21.4 percent of participants in the control group. For unprotected sex, we found that 14.4 percent of participants in the treatment group reported having had sex without an effective contraceptive method in the past three months, compared with 24.8 percent of the control group. Our interim findings further suggested that the T.O.P.P. program reduced rates of unprotected sex by leading participants to choose highly effective LARC methods over other contraceptive methods rather than by increasing overall rates of contraceptive use. Among women in the control group, over three-quarters (80.7 percent) reported having used some type of effective contraceptive method in the past three months (Table I.1). The percentage was similar for women in the treatment group (84.4 percent). As a result, we found no evidence of a statistically significant program impact on our more broadly defined measure of contraceptive use, despite the large increase in use of LARC methods among participants in the treatment group. 4 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Table I.1. Key 6-month findings from the T.O.P.P. interim impact report Treatment Control Measure group group Difference p-value Percentage of respondents reporting use of the following birth control methods in the past 3 months: LARC methoda 38.3 21.4 16.9** <0.01 Any hormonal method or IUDb 75.3 67.8 7.5 0.18 Any effective method of birth controlc 84.4 80.7 3.6 0.87 Percentage of respondents who reported having unprotected 14.4 24.8 -10.4** <0.01 sex in the past 3 monthsd Percentage of respondents who reported the following in the past 3 months: Had sexual intercourse 82.5 84.7 -2.2 >0.99 Had sexual intercourse without a condom 49.9 52.2 -2.3 >0.99 Number of sexual partners in the past 3 months 0.93 1.02 -0.10 0.76 Percentage of respondents who reported receiving information on the following topics:e Relationships 42.2 34.5 7.7 0.59 Methods of birth control 89.2 76.8 12.3** <0.01 Where to get birth control 88.9 75.0 13.9** <0.01 Abstinence 52.1 36.3 15.8** <0.01 Sexually transmitted infections 77.8 68.9 9.0 0.18 Talking to a partner about sex or birth control 74.8 71.9 2.8 >0.99 How to say no to sex 77.1 71.8 5.3 >0.99 Percentage of respondents who reported receiving information from each of the following sources:e Nurse or doctor during a facility visit 85.8 81.7 4.1 0.51 Health provider during a home visit 68.2 37.5 30.7** <0.01 Percentage of respondents who reported correct knowledge of: Effectiveness of condoms in preventing pregnancy 52.7 53.9 -1.2 >0.99 Effectiveness of birth control pills in preventing pregnancy 48.6 52.0 -3.4 >0.99 Effectiveness of condoms in preventing STIs 28.6 29.7 -1.2 >0.99 Effectiveness of the birth control pills in preventing STIs 65.3 64.6 0.7 >0.99 Perceived ease of access to condomsf 4.44 4.50 -0.05 >0.99 Perceived ease of access to birth control other than condomsf 4.28 4.34 -0.06 >0.99 Perceived trust in birth control providersf 4.08 4.01 0.07 >0.99 Perceived ease of using birth controlg 3.47 3.48 -0.01 >0.99 Perceived need for condomsf 3.58 3.58 0.01 >0.99 Perceived need for birth control other than condomsf 4.32 4.29 0.03 >0.99 Percentage of respondents indicating an intention to avoid pregnancy in the next 12 months 70.6 64.2 6.4 0.18 Percentage of respondents who reported receiving birth 76.5 66.2 10.3** <0.01 control from a doctor or nurse in past 6 months Source: Smith et al. (2015) a Includes the following methods: IUD and implant. b Includes the following methods: birth control pills, shot, patch, ring, IUD, and implant. c Includes the following methods: male condoms, female condoms, birth control pills, shot, patch, ring, IUD, implant, and vasectomy. d Defined as having sexual intercourse without using an effective birth control method in the past 3 months. e Questions refer to information received in the 6 months prior to survey administration. f Based on response to one survey item, range 1 to 5. g Based on average response to two survey items, range 1 to 5. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. IUD = intrauterine device; LARC = long-acting reversible contraceptive; STI = sexually transmitted infection. 5 FINAL IMPACTS OF THE T.O.P.P. PROGRAM We found no evidence that the program’s promotion of highly effective contraceptive methods, such as LARC methods, had any unintended spillover effects on sexual risk behaviors not directly targeted by the program. Some research has suggested that increasing LARC use among adolescents may lead to higher rates of other sexual risk behaviors, such as sex without a condom (Steiner et al. 2016). For T.O.P.P., however, we found that participants in the treatment group were no more likely than those in the control group to report having had sex (82.5 percent versus 84.7 percent) or to report having had sex without a condom (49.9 percent versus 52.2 percent). Participants in both groups also reported a similar average number of sexual partners (an average of 0.93 for the treatment group and 1.02 for the control group). Our exploration of potential mechanisms for the observed impacts suggested that T.O.P.P. may have influenced rates of LARC use and unprotected sex primarily by increasing exposure to key types of sexual and reproductive health information and by improving access to contraceptive services (Table I.1). The T.O.P.P. program had statistically significant impacts on participants’ exposure to information on methods of birth control, where to get birth control, and abstinence. Women in the treatment group were also statistically significantly more likely to report having received information on reproductive health from a health provider during a home visit. Additionally, T.O.P.P. was associated with a 10 percentage point increase in the share of women who reported receiving birth control from a doctor or nurse. Among treatment group participants, 76.5 percent reported receiving birth control from a doctor or nurse in the past six months, compared with 66.2 percent of the control group. Impacts of T.O.P.P. on other outcomes that may mediate behavioral change were smaller and statistically insignificant (Table I.1). In particular, we found no evidence that T.O.P.P. impacted women’s knowledge of the efficacy of condoms and birth control pills, attitudes toward condoms and other methods of birth control, or intentions to avoid a future pregnancy. For the knowledge and attitude measures, the survey questions focused primarily on outcomes related to condoms, birth control pills, and more general birth control methods broadly defined. Because of this more general focus, the survey did not ask questions on knowledge of or attitudes toward LARC methods or other specific methods of birth control that T.O.P.P. nurse educators may have discussed in detail with program participants. C. Research questions The present report adds to these findings by examining T.O.P.P.’s longer-term impacts measured at the end of the 18-month program. Although our interim report found promising short-term effects on rates of LARC use and unprotected sex after program participants had received the first 6 months of the 18-month program, the ultimate measure of the program’s success is whether it achieved its primary goal of reducing rapid repeat pregnancy. To assess the program’s impact on this primary outcome, we use data from an 18-month follow-up survey to measure rates of repeat pregnancy among participants in both the treatment and control groups. We thus assess the following primary research question of interest: • Does T.O.P.P. decrease rates of rapid repeat teen pregnancy among low-income adolescent women? 6 FINAL IMPACTS OF THE T.O.P.P. PROGRAM To provide a comprehensive assessment of the program’s impacts, we also assess several other important pregnancy-related outcomes. As noted earlier in this chapter, programs for adolescent mothers such as T.O.P.P. are motivated in part by the prior research finding that a majority of adolescent pregnancies are reported as unintended (Mosher et al. 2012). By promoting the uptake of highly effective contraceptive methods such as LARCs, these programs aim to decrease the risk of rapid repeat pregnancy primarily or entirely by reducing the number of pregnancies that are reported as unintended. In addition, from a broader public health and policy perspective, the ultimate outcomes of interest are not only those of the mother, but also those of the child. Although our evaluation of T.O.P.P. was not designed to assess the program’s long-term effects on child outcomes, we can assess the program’s impact on reported rates of live births as an outcome separate from the program’s impact on repeat pregnancy rates. To address these additional outcomes of interest, we assess the following research question: • Does T.O.P.P decrease rates of unintended pregnancy, rates of live births, or a woman’s total number of reported pregnancies? We also revisit several key outcomes from our earlier interim report to examine whether and how the program’s longer-term impacts on outcomes such as contraceptive use and sexual risk behaviors may differ from the impacts we observed in the short run. In particular, we assess the following three research questions measured 18 months after program enrollment: • Are T.O.P.P. participants more likely to have used an effective birth control method in the past three months, and to have used a LARC method in particular? • Is T.O.P.P. successful in reducing rates of unprotected sex within the past three months? • Does participation in T.O.P.P. have an impact on sexual risk behaviors not directly targeted by the program, such as rates of sexual activity and condom use, and number of sexual partners? Finally, although T.O.P.P. emphasizes the prevention of repeat pregnancies and promotion of highly effective contraceptive methods, it is possible that the program has spillover effects on other important outcomes, such as educational attainment. In particular, having a rapid repeat pregnancy may act as a barrier for a young woman to completing high school or undertaking further education. We thus also assess the following research question: • Does T.O.P.P increase school enrollment or educational attainment? 7 This page has been left blank for double-sided copying. FINAL IMPACTS OF THE T.O.P.P. PROGRAM II. STUDY DESIGN This study was designed as a randomized controlled trial involving low-income adolescent women recruited through OhioHealth hospitals and clinics. Among eligible women who agreed to participant in the study, about half were randomly assigned to a treatment group that was offered the T.O.P.P. program and half were assigned to a control group not offered the program. Both treatment and control group participants had access to existing, standard-of-care reproductive health services available through the OhioHealth system and other providers. Furthermore, after enrollment in the study, all participants from both groups received educational handouts on birth control, sexually transmitted infections (STIs), and birth spacing. In this report, we calculate impacts of the T.O.P.P. program by comparing outcomes for the treatment and control groups about 18 months after study enrollment. In this chapter, we begin by describing the enrollment and retention of study participants. We then discuss the baseline characteristics of the study participants included in our 18-month analysis. We end by providing a summary of the treatment and control conditions, including how service receipt varied across treatment group members with different characteristics. In Chapter III, we describe the data, measures, and analytic methods used to estimate program impacts. A. Sample enrollment and retention The study population was composed exclusively of low-income expectant or newly parenting adolescent women in the Columbus, Ohio, area. Low-income adolescents were selected because they were expected to benefit from the T.O.P.P. transportation assistance to a greater extent than higher-income adolescents. As noted in Chapter I, participants were recruited from seven OhioHealth women’s clinics and the postpartum units of five OhioHealth hospitals. These facilities serve seven central Ohio counties: Fairfield, Franklin, Delaware, Licking, Madison, Pickaway, and Union. All of the participating clinics and hospitals but one are located in Franklin County; the exception is a hospital located in Delaware County. To be eligible for the study, women had to be ages 10 to 19, at least 28 weeks pregnant or less than 9 weeks postpartum, and enrolled in Medicaid. Due to the telephone-based nature of the intervention, women also had to have regular telephone service and to speak English. Sample enrollment began in October 2011 and continued until January 2014. To identify eligible women, program staff conducted regular queries in OhioHealth’s electronic scheduling system, producing lists of potentially eligible women and their next appointments for health care at the participating clinics and hospitals, including prenatal and postnatal appointments at the clinics and postpartum appointments in the maternity wards of hospitals. After potentially eligible patients were identified, an OhioHealth standard-of-care provider approached them during their next scheduled clinic appointment or in the postpartum unit of the hospital and told them about the opportunity to learn about the study. Some women agreed to learn more about the study, and T.O.P.P. program or local evaluation staff followed up, providing more detailed information about the study, the potential opportunity to participate in the T.O.P.P. program, and consent procedures. To participate in the study, all adolescents were required to provide written consent and complete a paper-and-pencil baseline survey questionnaire. For most women, this follow-up occurred while they were still at the OhioHealth clinic or hospital. Participants under age 18 had to provide consent from a parent or legal guardian. The study procedures and consent 9 FINAL IMPACTS OF THE T.O.P.P. PROGRAM forms were approved by the institutional review boards of OhioHealth, Nationwide Children’s Hospital, and the Ohio Department of Health. Sample enrollment and random assignment were managed through a secure web-based system. OhioHealth staff entered participants into the system on a rolling basis as they were recruited into the study. After participants were registered as having provided consent and having completed the baseline survey questionnaire, the system randomly assigned them to either the treatment group or the control group. We programmed the system to conduct random assignment using a permuted block design, a method that helps ensure an even balance of participants between the treatment and control groups throughout the study period (Matts and Lachin 1988; Schulz and Grimes 2002). For this study, we specified a variable block size of up to six characters and a one-to-one allocation of participants between the treatment and control groups. We also stratified the random assignment by recruitment location and age group (under age 18 versus age 18 or 19) to avoid the possibility of a chance imbalance in these characteristics between the treatment and control groups. Shortly after random assignment, an OhioHealth nurse educator was assigned to each treatment group participant and began delivering the intervention on an individual basis. The nurse educator assignments were determined based on existing caseloads. The sample enrollment process yielded a total sample of 598 study participants (Figure II.1). This study sample was obtained from a larger sample of 978 women who were identified as potentially eligible for the study through the OhioHealth electronic scheduling system. Of these potentially eligible women, 380 (39 percent) were excluded from the study, with the most common reasons being lack of interest (n = 166) or leaving the hospital or clinic before receiving information on the study (n = 91). Because of these exclusions, the study sample is not intended to be a random or representative sample of all women who were potentially eligible. Of the 598 young women who agreed to participate in the study, roughly half were randomly assigned to the treatment group (297 participants) and the other half were randomly assigned to the control group (301 participants). The retention rate for the study was high (Figure II.1). Of the 297 women randomly assigned to the treatment group, 237 (80 percent) completed the 18-month survey featured in this report. Of the 301 women randomly assigned to the control group, 235 (78 percent) completed the 18- month survey. See Appendix A for a nonresponse analysis examining the characteristics of participants who did and did not complete the 18-month survey. 10 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Figure II.1. Overview of sample enrollment and retention Identified as potentially eligible (n = 978) Excluded (n = 380) Left hospital before contacted (n = 91) Enrollment Declined to receive any information on the study from T.O.P.P. staff (n = 166) Declined to participate after learning about the study from T.O.P.P. staff (n = 47) Did not complete enrollment process (n = 64) Identified after study enrollment had ended (n = 12) Randomized (n = 598) Allocation Allocated to treatment group Allocated to control group (n = 297) (n = 301) Follow-Up Lost to follow-up (n = 60) Lost to follow-up (n = 66) Withdrew consent (n = 1) Withdrew consent (n = 0) Did not respond to 18-month Did not respond to 18-month follow-up survey (n = 59) follow-up survey (n = 66) Analysis Included in primary analysis Included in primary analysis (n = 237) (n = 235) B. Baseline sample characteristics We examined several characteristics of the treatment and control groups at baseline to characterize the sample and ensure that random assignment resulted in comparable study groups. Differences between the treatment and comparison groups were mostly small and statistically insignificant. The sociodemographic characteristics of the study sample were consistent with those of the population targeted by the T.O.P.P. program (Table II.1). At the time of the baseline survey, the mean age of participants at baseline was about 18 years. The majority of study participants were 18 or 19 years old. Less than half of all study participants had received a high school diploma or equivalency credential. About 90 percent of participants were receiving some kind of public 11 FINAL IMPACTS OF THE T.O.P.P. PROGRAM assistance at baseline, primarily through the Supplemental Nutrition Assistance Program (SNAP) or the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC). The racial characteristics of the population reflect those of the area T.O.P.P. serves; the majority of sample members were non-Hispanic whites and non-Hispanic blacks. Table II.1. Baseline sociodemographic characteristics Treatment Control group group Variable mean mean Difference p-value Age at random assignment (years) 18.43 18.36 0.08 0.20 Highest level of education completed (%) No high school 4.1 7.1 -3.0 0.17 Some high school 48.8 47.6 1.2 0.78 High school graduate or GED 37.9 37.4 0.5 0.90 Any postsecondary education 7.0 7.6 -0.6 0.81 Other 2.2 0.4 1.9 0.11 Economic situation (%) Household received SNAP or WIC in past 30 days 91.4 89.7 1.8 0.60 Household received TANF in past 30 days 25.2 26.0 -0.8 0.86 Household received other assistance in past 30 days 24.5 25.7 -1.2 0.81 Race/ethnicity (%) White, non-Hispanic 47.5 46.5 1.0 0.83 Black, non-Hispanic 35.2 37.3 -2.1 0.62 Hispanic 6.8 7.5 -0.7 0.78 Other race/ethnicity or multiracial 10.5 8.7 1.8 0.55 Pregnancy 26.7 23.9 2.8 0.39 Pregnant at time of baseline survey (%) 37.4 35.8 1.6 0.73 Has been pregnant multiple times (%) 1.47 1.44 0.03 0.73 Number of times pregnant (including most recent) Current relationship with baby’s father (%) Married or engaged 25.5 20.3 5.2 0.21 Dating (seriously or casually) 50.6 44.9 5.6 0.29 Other (no contact, have contact but not romantically 23.9 34.8 -10.8* 0.03 involved, or other relationship specified) Family structure (%) Lives with both biological parents 9.9 12.2 -2.3 0.42 Lives with one biological parent 37.5 41.3 -3.8 0.40 Lives with neither biological parent 52.6 46.5 6.1 0.16 Sample sizea 237 235 Source: Baseline survey administered to study participants before the start of the program. Notes: Reported means are from regressions that control for random assignment strata. P-values are adjusted for clustering of standard errors at the randomization block level. See Appendix B for a description of the measures. a Reported sample size is the number of participants who completed the 18-month follow-up survey and are included in the analysis. *Significantly different from zero at the .05 level, two-tailed test. GED = General Educational Development certification; SNAP = Supplemental Nutrition Assistance Program; TANF = Temporary Assistance for Needy Families; WIC = Special Supplemental Nutrition Program for Women, Infants, and Children. 12 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Consistent with program eligibility criteria, roughly one-quarter of participants were pregnant at the time of study enrollment; the remainder had given birth just prior to enrollment. Slightly more than one-third of participants had been pregnant more than once, and the mean number of pregnancies at baseline was 1.5 in the treatment group and 1.4 in the control group. About half of study participants lived with neither biological parent at baseline. Most participants reported having a relationship with their baby’s father at the time of the baseline survey; about half reported being in a dating relationship and an additional one-quarter (treatment group) or one-fifth (control group) reported being either engaged or married to their baby’s father. Treatment group members were less likely than control group members to report having another type of relationship with their baby’s father. In the control group, 34.8 percent of women reported not currently dating, being married to, or being engaged to their baby’s father, compared to 23.9 percent of women in the treatment group (a difference of 10.8 percentage points, p = 0.03). Participants reported mixed levels of exposure to information on reproductive health topics at the time of study enrollment (Table II.2). More than 80 percent of participants said they had received at least some information in the past 12 months on birth control methods, sources of birth control, and STIs. About three-quarters said they had also received information on talking to a partner about sex or birth control and how to say no to sex. Fewer had received more general information on relationships (roughly 65 percent) or abstinence (roughly 50 percent). Table II.2. Baseline exposure to reproductive health information Treatment Control Variable group mean group mean Difference p-value In past 12 months, received information on (%) Relationships 63.8 67.3 -3.5 0.47 Methods of birth control 86.2 85.6 0.6 0.86 Where to get birth control 84.0 84.1 -0.1 0.99 Abstinence 45.5 53.6 -8.1 0.12 Sexually transmitted infections 81.2 84.0 -2.9 0.50 Talking to a partner about sex or birth control 74.0 74.7 -0.7 0.87 How to say no to sex 73.4 77.1 -3.7 0.41 Sample sizea 237 235 Source: Baseline survey administered to study participants before the start of the program. Notes: Reported means are from regressions that control for random assignment strata. P-values are adjusted for clustering of standard errors at the randomization block level. See Appendix B for a description of the measures. a Reported sample size is the number of participants who completed the 18-month follow-up survey and are included in the analysis; it does not account for item nonresponse for any measures included in the table. Use of effective contraceptive methods was relatively low among study participants during the three months before the start of their pregnancy. Although almost all study participants reported some lifetime experience with an effective method of birth control (not shown), only 1 or 2 percent of participants reported using a LARC method, and less than one-third reported using a hormonal method or IUD in the three months before becoming pregnant (Table II.3). Less than 70 percent reported using an effective birth control method, with condoms being the most commonly used method. More than two-thirds of participants reported having unprotected sex in the three months before becoming pregnant, and nearly 90 percent reported having had sex 13 FINAL IMPACTS OF THE T.O.P.P. PROGRAM without a condom. Participants also reported having several sexual partners in their lifetime; the average for participants in the treatment group was 5.08 lifetime partners and the average for women in the control group was 4.80 partners. Table II.3. Baseline sexual behaviors Treatment Control Variable group mean group mean Difference p-value In 3 months prior to becoming pregnant (%): Used a LARC methoda 1.5 0.4 1.1 0.26 Used a hormonal method of birth control or IUDb 32.9 29.1 3.8 0.36 Used an effective method of birth controlc 65.8 69.5 -3.7 0.41 Had unprotected sexual intercoursed 75.9 71.2 4.7 0.29 Had sexual intercourse without a condom 91.2 85.8 5.4 0.08 Lifetime number of sexual partners 5.08 4.80 0.29 0.62 Sample sizee 237 235 Source: Baseline surveys administered to study participants before the start of the program. Notes: Reported means are from regressions that control for random assignment strata. P-values are adjusted for clustering of standard errors at the randomization block level. See Chapter III for a description of the measures. a Includes the following methods: IUD and implant. b Includes the following methods: birth control pills, shot, patch, ring, IUD, and implant. c Includes the following methods: male condoms, female condoms, birth control pills, shot, patch, ring, IUD, implant, and vasectomy. d Defined as having sexual intercourse without using an effective birth control method. e Reported sample size is the number of participants who completed the 18-month follow-up survey and are included in the analysis; it does not account for item nonresponse for any measures included in the table. IUD = intrauterine device; LARC = long-acting reversible contraceptive. C. Treatment and control conditions Participants assigned to the control condition were not offered the T.O.P.P. program but did retain access to any existing standard-of-care services offered through the OhioHealth system and other providers. In the Columbus area, existing pregnancy and reproductive health services include those provided though health care organizations (such as OhioHealth, Nationwide Children’s Hospital, Planned Parenthood, Columbus Neighborhood Health Centers, and Columbus Public Health Women’s Health Center), home visiting programs (such as NFP), and other community-based organizations. Study participants randomly assigned to the treatment condition were offered the 18-month T.O.P.P. program as delivered by trained OhioHealth nurse educators. Because the intervention was delivered one-on-one to program participants and featured individualized motivational interviewing sessions, the T.O.P.P. nurse educators customized the program services to each participant’s unique needs and circumstances. Nearly all (96.0 percent) T.O.P.P. participants received at least one phone call (Table II.4). A majority of participants also received a psychosocial assessment (83.8 percent) and referrals from a social worker (60.3 percent). Fewer participants received a home visit from a T.O.P.P. nurse educator (44.1 percent), had a community visit with a nurse educator (36.7 percent), or used the T.O.P.P. van service (32.0 percent). This variation across different parts of the program arose naturally from the varying 14 FINAL IMPACTS OF THE T.O.P.P. PROGRAM interests and needs of each program participant. For example, participants who already had access to regular transportation or desired clinical services had no need to use the comparable services offered through T.O.P.P. Following the patient-centered principles of motivational interviewing, the T.O.P.P. nurse educators did not require or push participants to use any particular combination of program services. Table II.4. Receipt of T.O.P.P. services Received service Average number of Average duration of each Program service at least once (%) times receiving service service received (in minutes) Phone call 96.0 8.2 12.3 Psychosocial assessment 83.8 0.9 17.2 Referral for social services 60.3 1.5 10.2 Home visit 44.1 0.7 47.1 Community visit 36.7 0.7 55.2 Van service 32.0 0.7 67.4 Clinic visit 25.3 0.4 58.5 Text message 7.7 0.1 5.6 All services 13.3 275.4 Source: Program administrative data collected by the T.O.P.P. nurse educators. Note: We calculated these estimates across all 297 study participants assigned to the treatment condition. In calculating the estimates, we excluded any phone calls or text messaging lasting two minutes or less. The intervention as delivered also involved a high degree of overlap in services between the three main program components. As discussed in Chapter I, the T.O.P.P. program model features a combination of (1) telephone-based care coordination, (2) facilitated access to contraceptive services, and (3) access to a social worker. The program is delivered using motivational interviewing techniques designed to help participants identify and realize their personal goals for contraceptive use and birth spacing. The T.O.P.P. nurse educators had initially planned to deliver the motivational interviewing sessions primarily through the telephone-based care coordination component of the program. However, in practice the nurse educators discovered that additional opportunities for motivational interviewing sessions occurred during the T.O.P.P. home visits, community visits, and van services. Additionally, they discovered that a small percentage of participants preferred to communicate through text messaging instead of phone calls. Nurses thus also used text messaging for motivational interviewing. In part because of this overlap in services between the three main program components, the intervention as delivered had relatively less emphasis on telephone conversations than program staff had initially expected. As discussed in Chapter I, the program model called for the nurse educators to deliver motivational interviewing sessions by telephone with a recommended frequency of approximately once per month for 18 months. In practice, participants received an average of just over eight phone calls, with the average call lasting 12 minutes (Table II.4). If we also account for participation in the T.O.P.P. home visits, community visits, and other program activities, the total intensity or dosage of the program is higher: an average of just over 13 contacts per participant, or roughly one contact every 6 weeks (Table II.4). These contacts amounted to an average of about four and a half hours (275 minutes) of individualized interaction between each program participant and T.O.P.P. staff members. 15 FINAL IMPACTS OF THE T.O.P.P. PROGRAM We also found that program participation rates varied according to certain participant demographic and personal characteristics. In particular, non-Hispanic black participants tended to use more T.O.P.P. services than did non-Hispanic whites (Table II.5). On average, black participants had about three more total contacts with T.O.P.P. staff than did white participants, and the total dosage of the program ranged from a high of about five and a half hours (327 minutes) for black participants to a low of just under four hours (233 minutes) for whites. These differences are largely driven by higher participation rates among blacks in the T.O.P.P. van service, home visits, clinic visits, and referral contacts (Table II.5). Table II.5. Receipt of T.O.P.P. services, by participant race/ethnicity White, non- Black, non- Other race or Measure of service receipt Hispanic Hispanic multiple races Percentage of participants receiving at least one: Phone call 95.6 96.3 96.2 Psychosocial assessment 78.8 88.8 83.5 Referral for social services 51.1 72.0** 60.8 Home visit 38.0 52.3* 44.3 Community visit 30.7 43.0 36.8 Van service 21.9 43.9** 32.6* Clinic visit 17.5 34.6** 25.4 Text message 6.6 8.4 7.6 Total number of times receiving services 11.9 15.0* 13.8 Total duration of services received (minutes) 232.7 326.7* 299.5 Sample size 137 107 47 Source: Program administrative data collected by the T.O.P.P. nurse educators. Note: Estimates exclude any phone calls or text messaging lasting two minutes or less. *Statistically significantly different from non-Hispanic white participants at the 0.05 level, two-tailed test. **Statistically significantly different from non-Hispanic white participants at the 0.01 level, two-tailed test. In addition to these racial/ethnic differences, we found that program participation rates also varied according to the participant’s relationship with the baby’s father (Table II.6). In particular, participants who were not romantically involved with the baby’s father were nearly twice as likely as participants who were married or engaged to the baby’s father to use the T.O.P.P. van service. They were also more likely than married or engaged participants to receive a psychosocial assessment from the T.O.P.P. social worker. We found no evidence of statistically significant differences in program participation rates based on age, education level, economic situation, living circumstances, baseline attitudes toward contraception, baseline pregnancy intentions, or past contraceptive use behaviors (results not shown). Our accompanying implementation study (Meckstroth and Berger 2014) found that many participants reported having a positive experience with the program. Participants stated that the program helped them create and follow a birth control plan. One participant noted that seeing different contraception samples during an in-person visit with a nurse educator helped her better understand her birth control options and ultimately develop her birth control plan. Another participant said that T.O.P.P.’s van service was the main reason she was able to start using contraception because the nurse educator drove her directly to her office appointments. Some participants also reported that they particularly appreciated the monthly phone calls from nurse 16 FINAL IMPACTS OF THE T.O.P.P. PROGRAM educators, as those calls helped them forge a connection with the nurse educator. This type of connection was important, and unique, for some participants, especially those who did not have many other consistent personal relationships in their lives. This observation is consistent with research showing that social isolation can be an important determinant of sexual risk behavior (for example, Biggs et al. 2013; Taylor‐Seehafer et al. 2010). Table II.6. Receipt of T.O.P.P. services, by participant relationship status Married or Other engaged to Dating baby’s relationship with Measure of service receipt baby’s father father baby’s father Percentage of participants receiving at least one: Phone call 94.4 96.5 97.4 Psychosocial assessment 77.8 81.7 92.3* Referral for social services 59.7 54.2 71.8 Home visit 43.1 45.1 43.6 Community visit 33.3 35.9 39.7 Van service 23.6 28.9 43.6** Clinic visit 22.2 21.8 35.9 Text message 9.7 9.2 3.8 Total number of times receiving services 12.8 12.7 14.6 Total duration of services received (minutes) 263.3 266.5 304.3 Sample size 72 142 78 Source: Program administrative data collected by the T.O.P.P. nurse educators. Note: Estimates exclude any phone calls and text messaging last two minutes or less. *Statistically significantly different from married or engaged participants at the 0.05 level, two-tailed test. **Statistically significantly different from married or engaged participants at the 0.01 level, two-tailed test. According to the implementation study, some participants also expressed ways they felt the program could be improved. For example, some participants indicated that they would have preferred more home visits. They felt that seeing the nurse educator face-to-face was more personal than speaking on the phone. In addition, some participants reported that they already knew all the information provided by the program and so did not learn anything new. A small number of participants felt that the timing or frequency of phone calls from the nurse educators could be improved, and two participants expressed discomfort talking to the nurse educators. Although such reported feelings of discomfort were uncommon, they were not unexpected given the sensitive nature of the topics discussed. These findings on participants’ experiences are based on a performance measure survey administered to program participants about nine months after they had started the program. A total of 49 participants answered the survey and provided open-ended responses to questions on what participants liked best about the program; what T.O.P.P. taught them that they did not already know; what they would change about T.O.P.P. if they could; how they would rate their overall experience in T.O.P.P.; and, if they developed a plan to use birth control, what factors made it easier or more difficult to carry out their plan. A more detailed description of the data and participants’ reported experiences is provided in our accompanying implementation study (Meckstroth and Berger 2014). 17 This page has been left blank for double-sided copying. FINAL IMPACTS OF THE T.O.P.P. PROGRAM III. DATA, MEASURES, AND ANALYSIS The impact analysis presented in this report is based mainly on data from two rounds of surveys completed by study participants in the treatment and control groups. As discussed in Chapter II, participants were required to complete a paper-and-pencil baseline survey questionnaire upon enrollment in the study. The survey was administered by OhioHealth program staff and collected a broad range of information on participants’ demographic and personal characteristics, family relationships, attitudes, sexual risk behaviors, and pregnancy histories. A follow-up survey was administered roughly 18 months later by trained data collection staff from Mathematica and Nationwide Children’s Hospital. The data collection staff did not know which participants were assigned to the treatment and control groups. Each participant received a $25 gift card for completing the follow-up survey. For each participant, the data collectors began efforts to administer the follow-up survey 18 months after the initial random assignment date. However, as a result of the time required to locate the participants and schedule the surveys, the average timing of survey completion was just under 20 months after random assignment, with no statistically significant difference in survey timing between the treatment and control groups. Staff administered 74 percent of the completed 18-month follow-up surveys by telephone. For hard-to-reach cases—those unable to be reached by telephone—data collectors administered the survey in person, using a hard copy of the questionnaire. In the remainder of this chapter, we first describe the outcome measures constructed from the survey data. We then discuss the analytic methods used to assess the impacts of the T.O.P.P. program on participant outcomes. For more detailed information on the measures, see Appendix B. A. Outcome measures Drawing on data from the 18-month follow-up survey, we constructed six groups of outcome measures, each corresponding to one of the study’s research questions: (1) repeat pregnancy, (2) other pregnancy-related outcomes, (3) contraceptive use, (4) unprotected sex, (5) other sexual risk behaviors, and (6) education. These measures are summarized in Table III.1 and described in greater detail below. 19 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Table III.1. Outcome measures Measure Definition Repeat pregnancy Repeat pregnancy Binary variable: equals 1 if participant reported having had a repeat pregnancy in the past 18 months or was currently pregnant; equals 0 if she reported not having had a repeat pregnancy. Other pregnancy-related outcomes Unintended pregnancy Binary variable: equals 1 if participant reported having had a repeat pregnancy in the past 18 months that she did not want or had wanted to have later; equals 0 if she reported not having had a repeat pregnancy or having become pregnant and wanted to be pregnant. Repeat pregnancy ending in a live Binary variable: equals 1 if participant reported having had a repeat birth pregnancy in the past 18 months that ended in a live birth; equals 0 if she reported not having had a repeat pregnancy, having had a repeat pregnancy that did not end in a live birth, or being currently pregnant and having not had another repeat pregnancy in the past 18 months that ended in a live birth. Number of lifetime pregnancies Continuous variable: Number of times the participant reported having ever been pregnant. Contraceptive use Use of a LARC method Binary variable: equals 1 if participant reported using a LARC method (IUD or implant) in the past 3 months; equals 0 if she did not report use of a LARC method. Use of a hormonal method or IUD Binary variable: equals 1 if participant reported using one of the following methods in the past 3 months: birth control pills, shot, patch, ring, IUD, or implant; equals 0 if she did not report use of any of these methods. Use of any effective birth control Binary variable: equals 1 if participant reported using a condom (male or method female), hormonal method, IUD, or male vasectomy in the past 3 months; equals 0 if she did not report use any of these methods. Unprotected sex Incidence of unprotected sex Binary variable: equals 1 if participant reported having had sexual intercourse without using an effective birth control method in the past 3 months; equals 0 if she reported not having had intercourse or having always used an effective birth control method during intercourse. Other sexual risk behaviors Sexual activity Binary variable: equals 1 if participant reported having sexual intercourse in the past 3 months; equals 0 if she did not report having intercourse. Had sexual intercourse without a Binary variable: equals 1 if participant reported having sexual intercourse condom without a condom in the past 3 months, equals 0 if she reported not having intercourse or having always used a condom. Number of partners Continuous variable: Number of reported sexual partners in the past 3 months. Education Currently enrolled in school Binary variable: equals 1 if participant reported being currently enrolled in school; equals 0 if she did not report being currently enrolled. Completed high school or GED Binary variable: equals 1 if participant reported that she had completed high school or attained a GED; equals 0 if she reported completing less than twelfth grade or that she had not completed high school or attained a GED. GED = General Educational Development certification; IUD = intrauterine device; LARC = long-acting reversible contraceptive. 20 FINAL IMPACTS OF THE T.O.P.P. PROGRAM 1. Repeat pregnancy As discussed in Chapter I, our primary outcome of interest is whether a woman experienced a repeat pregnancy within 18 months of study enrollment. To assess this outcome, the 18-month follow-up survey asked participants the following yes/no question: “You were pregnant 18 months ago, right before or when you filled out a paper survey similar to this one for this same study. Please think back to that pregnancy you experienced 18 months ago. Have you been pregnant again since that pregnancy ended?” We used responses to this question to create a binary (yes/no) indicator of whether the participant had experienced a repeat pregnancy in the past 18 months. Both to check the reliability of responses to this question and to minimize any missing data resulting from item nonresponse, we compared each participant’s response to this question with her response to a similar question asked on the follow-up survey conducted 6 months after study enrollment. If a woman reported having had a repeat pregnancy on the 6-month survey but did not respond to the repeat pregnancy question on the 18-month survey, we logically imputed her response for the 18-month survey as having had a repeat pregnancy. If a woman reported having had a repeat pregnancy on the 6-month survey but then contradicted this response on the 18- month survey by reporting not having had a repeat pregnancy, we considered her responses inconsistent and set the variable to missing. A total of 9 participants were recoded as missing values for this reason. Because of the importance of this outcome for the analysis, we checked the robustness of our results to these coding decisions by creating three alternative versions of the same outcome: • Alternative 1. For this version of the outcome, we looked only at responses to the repeat pregnancy question on the 18-month follow-up survey without checking the consistency of responses against data from the earlier 6-month follow-up survey. A total of 9 participants had different values on this alternative version of the outcome than on our primary version. These 9 participants were coded as missing values on the primary version and having had a repeat pregnancy on the alternative version. • Alternative 2. For this version of the outcome, we looked at responses to the repeat pregnancy questions on both the 6-month and 18-month follow-up surveys. In addition, we also incorporated information from the 6-month survey on the outcome of the reported repeat pregnancy. For example, the 6-month survey asked women who reported a repeat pregnancy how far along they were in the pregnancy. Examining this additional information helped establish the reliability of responses to the repeat pregnancy question on the 6-month survey. A total of 9 participants had different values on this version of the outcome than on our primary version. These 9 participants were coded as missing on the primary version. For this alternative version, we recoded 7 participants as having had a repeat pregnancy and 2 participants as not having had a repeat pregnancy. • Alternative 3. For this version of the outcome, we incorporated information from a question on total lifetime number of pregnancies that was asked in each round of the survey. In particular, we required consistency in responses between (1) a participant’s answer to the basic yes/no question on repeat pregnancy and (2) the total lifetime number of pregnancies the participant reported across survey waves. For example, if a participant reported having 21 FINAL IMPACTS OF THE T.O.P.P. PROGRAM had a repeat pregnancy on the 18-month survey but not on the 6-month survey, we tried to validate this response by looking in the 18-month survey for a corresponding increase in the total lifetime number of pregnancies reported. If the response to the question on total lifetime number of pregnancies did not confirm the repeat pregnancy, we considered the participant’s responses inconsistent and set the outcome variable to missing. A total of 20 participants had different values on this version of the outcome than on our primary version. Of these 20 participants, 11 were coded as having had a repeat pregnancy on the primary version and missing values on the alternative version. Another 9 participants were coded as missing values on the primary version and not having had a repeat pregnancy on the alternative version. As we show later in Chapter IV, we find similar results across all of these different versions of the outcome measure, giving confidence that our results are not an artifact of our particular coding decisions. 2. Other pregnancy-related outcomes To provide a comprehensive assessment of the program, we constructed measures of three additional pregnancy-related outcomes in addition to our primary outcome of repeat pregnancy. These three additional measures are as follows: • Unintended pregnancy. The 18-month survey asked participants who reported a repeat pregnancy whether they had (1) wanted to become pregnant at the time, (2) wanted to become pregnant later, or (3) not wanted to become pregnant at all. We used responses to this question to create a binary (yes/no) indicator for whether the participant reported having had an unintended pregnancy—defined as either wanting to become pregnant later or not wanting to become pregnant at all. This measure was set to zero for women who did not report a repeat pregnancy (according to our primary measure). The resulting outcome measure indicates the percentage of women who reported having an unintended pregnancy in the 18 months after study enrollment. • Repeat pregnancy ending in a live birth. As discussed in Chapter I, although our evaluation of T.O.P.P. was not designed to assess the program’s long-term effects on child outcomes, we can assess the program’s impact on reported rates of live births as an outcome separate from the program’s impact on repeat pregnancy rates. The survey asked participants who reported a repeat pregnancy about the outcome of their most recent pregnancy. We created a binary (yes/no) indicator for whether the participant reported that her pregnancy had ended in a live birth. This measure was set to zero for women who did not report a repeat pregnancy (according to our primary measure) or who were still pregnant at the time of the survey. We also explored the possibility of creating an outcome for the number of repeat pregnancies ending in an abortion. However, the reported prevalence of this outcome was very low and therefore could not be used in the analysis. There were seven reported abortions among control group participants and none in the treatment group. We cannot say whether these numbers reflect actual rates or reflect an underreporting of abortions on the survey; prior research has shown a pattern of underreporting of abortions on self-reported surveys (for example, Fu et al. 1998; Jones and Kost 2007). 22 FINAL IMPACTS OF THE T.O.P.P. PROGRAM • Number of lifetime pregnancies. The survey asked participants how many times they had ever been pregnant, including their initial pregnancy reported at baseline. We used responses to this question to create a count variable capturing the reported total number of lifetime pregnancies. 3. Contraceptive use The 18-month survey asked participants a series of questions about their use of specific birth control methods over the past three months. The list of birth control methods was designed to be as comprehensive as possible, covering methods ranging from more traditional methods, such as fertility awareness, to more modern methods such as the contraceptive implant and IUDs (see Appendix B for a complete list). For contraceptive shots, vaginal rings, IUDs, and the contraceptive implant, the survey listed specific brand names in addition to a generic name. For example, the survey listed Depo-Provera as well as contraceptive shots as a generic method. To measure the impacts of the T.O.P.P. program on contraceptive use, we used responses to these individual survey questions to construct the following measures: • Any use of a LARC method. To measure the impact of the program on LARC use, we created a binary (yes/no) indicator for whether the participant reported ever using an IUD or contraceptive implant in the past three months. • Any use of a hormonal method or IUD (nonbarrier method). To measure the impacts of the program on nonbarrier methods of contraceptive use, we created a binary (yes/no) indicator for whether the participant reported ever using at least one of the following methods of contraception in the past three months: birth control pills, contraceptive shots, hormonal patches, vaginal rings, IUDs, or the contraceptive implant. This outcome differs from our measure of LARC use by also accounting for use of birth control pills, contraceptive shots, vaginal rings, and hormonal patches. • Any use of an effective birth control method. To broadly assess the program’s impacts on a very general measure of contraceptive use, we created a binary (yes/no) indicator for whether the participant reported ever using at least one of the following methods of contraception in the past three months: male condom, female condom, birth control pills, contraceptive shots, hormonal patches, vaginal rings, IUDs, the contraceptive implant, or vasectomy. This outcome differs from our measure of non-barrier methods of contraceptive use by also accounting for use of male condoms, female condoms, and male vasectomy. This measure spans contraceptive methods with a very wide range of effectiveness rates. All three of these outcomes match the contraceptive use measures featured in our earlier interim impact report (Smith et al. 2015). 4. Unprotected sex In the interim impact report, we found evidence that the T.O.P.P. program reduced the incidence of unprotected sex at the time of the 6-month follow-up survey (Smith et al. 2015). To examine whether this effect persisted at the end of the 18-month program, we used responses from the 18-month survey to construct the same measure of unprotected sex featured in our earlier report. For this measure, the survey asked participants whether they had sexual 23 FINAL IMPACTS OF THE T.O.P.P. PROGRAM intercourse in the past three months without using any effective contraceptive method during at least one encounter with a sexual partner. The question was limited to vaginal intercourse and did not include oral or anal intercourse. The survey defined effective contraceptive methods as comprising condoms, birth control pills, the shot, the patch, the ring, an IUD, or the contraceptive implant. Based on responses to this question, we created a binary (yes/no) indicator for whether the participant reported having unprotected sex. Participants who reported abstaining from sexual intercourse in the past three months were retained in the analysis by coding them as protected and combining them with respondents who reported always using an effective contraceptive method. 5. Other sexual risk behaviors To examine whether the program’s emphasis on promoting the use of highly effective contraceptive methods, such as LARCs, had any unintended spillover effects on other types of sexual risk behaviors not directly targeted by the program, we constructed three different outcomes: • Had sexual intercourse. The survey asked participants whether they had had sexual intercourse in the past three months. Based on responses to this question, we created a binary (yes/no) indicator for whether a participant reported having had sexual intercourse. The question was limited to vaginal intercourse and did not include oral or anal intercourse. However, in additional exploratory analyses not presented in this report, we found substantively similar results for comparable measures of oral and anal intercourse. • Had sexual intercourse without using a condom. The survey asked participants whether they had had vaginal intercourse without using a condom at least once in the past three months. Based on responses to this question, we created a binary (yes/no) indicator for whether the participant reported having had sexual intercourse without a condom in the past three months. Participants who reported abstaining from sexual intercourse in the past three months were retained in the analysis by coding them as protected and combining them with respondents who reported always using a condom when they had sexual intercourse. The question was limited to vaginal intercourse and did not include oral or anal intercourse. However, in additional exploratory analyses not presented in this report, we found substantively similar results for comparable measures of unprotected oral and anal intercourse. • Number of sexual partners. For respondents who reported being sexually active, the survey asked them to report the number of different sexual partners they had in the past three months. The question was limited to vaginal intercourse and did not include oral or anal intercourse. Based on responses to this question, we created a continuous variable for the number of recent sexual partners. Respondents who reported abstaining from sexual intercourse in the past three months were retained in the analysis by coding them as having zero sexual partners. We examined the same three outcomes in our earlier interim report and found no evidence of unintended spillover effects 6 months after study enrollment (Smith et al. 2015). In the present report, we revisit the same outcomes using longer-term data from the 18-month follow-up survey. 24 FINAL IMPACTS OF THE T.O.P.P. PROGRAM 6. Education Although the emphasis of T.O.P.P. was on the prevention of repeat pregnancy and promotion of highly effective contraceptive methods, it is possible that the program had spillover effects on other important outcomes, such as educational attainment. In particular, having a rapid repeat pregnancy may act as a barrier for a young woman to completing high school or pursuing any further education. To measure whether T.O.P.P. had spillover effects on educational persistence or attainment, we created the following two binary variables: • Currently enrolled in school. The 18-month survey asked participants about their current school status. We created a binary (yes/no) indicator for whether the participant was currently enrolled in school or studying school subjects either at home, online, or elsewhere. • Completed high school or GED. The survey asked participants the highest grade in school they had completed. We created a binary (yes/no) indicator for whether the participant had completed high school or attained a GED. B. Analytic approach We used a multivariate regression framework to analyze the impact of T.O.P.P. on each outcome. A regression framework is appropriate for this study because it allows us to account for the stratification and permuted block design used for random assignment (discussed in Chapter II). It also allows us to improve the precision of our impact estimates by statistically adjusting for any baseline covariates that are strongly correlated with our outcome measures. This approach of adjusting for baseline covariates can help achieve precision gains in the impact estimates by reducing the amount of residual variation in the outcome measures. We estimated a separate regression model for each outcome. For our main analyses, presented in Chapter IV, we estimated an ordinary least-squares (OLS) regression model for each outcome. We adjusted the standard errors of the impact estimates to account for the permuted block random assignment design (Matts and Lachin 1988). As a sensitivity test, we also estimated impacts using logistic regression models for binary outcomes (for example, repeat pregnancy within 18 months of a prior pregnancy). Appendix D presents the results of this sensitivity test as well as other possible specifications of the regression model. Each regression model included the following covariates: (1) a binary indicator for treatment status, (2) binary indicator variables for each of the strata created for random assignment, (3) two key demographic variables that are highly correlated with our key outcomes of interest (age and race), (4) a continuous variable measuring the number of months between when the baseline and follow-up surveys were administered to the participant, (5) the baseline measure of the outcome (if available), and (6) an additional set of baseline covariates that were empirically selected through a data-driven forward selection procedure because of their strong predictive power and potential to improve the precision of the impact estimates (described below). For all these covariates, we used dummy variable adjustment to avoid losing any cases on account of missing baseline data (Puma et al. 2009). For the forward selection procedure, we used a data-driven stepwise procedure developed previously in the literature (Social and Character Development Research Consortium 2010). For this procedure, we considered as candidate covariates a variety of baseline variables that have 25 FINAL IMPACTS OF THE T.O.P.P. PROGRAM been shown in other studies to have a strong link with risky sexual behavior and repeat pregnancy. Appendix B provides a complete list of the covariates considered. From this list of candidate covariates, the forward selection procedure involves gradually adding covariates to the model in order from most to least predictive of the outcome (as defined by the t-statistic on each covariate’s regression coefficient). We conducted the selection procedure separately for each outcome. We then compared the selection results across outcomes and identified those covariates meeting either one of two conditions: (1) the covariate was selected by the stepwise procedure for at least 60 percent of the outcomes or (2) the covariate was selected for only one outcome but the observed baseline difference between treatment and control groups in that covariate had a p-value of 0.20 or less. From among the full list of candidate covariates listed in Appendix B, only the covariates meeting these conditions were selected for inclusion in the impact analysis. We used the same list of covariates for estimating impacts on each outcome. Appendix D explores the robustness of our results to the use of models that exclude covariates selected by this procedure. We adjusted the statistical significance tests (p-values) from our regression models to account for multiple hypothesis testing. As discussed earlier in this chapter, our analysis uses multiple outcomes to answer some of the key research questions. For example, we constructed three separate measures of contraceptive use. Unless we account for this multiplicity, it could increase the chances of making a false discovery and lead to spurious claims about the program’s effectiveness. Researchers often declare a finding statistically significant if the probability of falsely rejecting the null hypothesis of no impact is less than 5 percent. However, when conducting separate tests arising from multiple outcomes, the probability of falsely rejecting the null hypothesis in at least one of them can be much higher than 5 percent. To correct for this increased probability, we apply a multiple hypothesis testing procedure outlined by Hothorn et al. (2008) and Schochet (2009). This procedure involves adjusting the reported p-value for each test to account for other tests conducted within the same “family” of related measures. Similar to other common methods of adjusting for multiple hypothesis testing, this procedure yields a 5 percent false positive rate across outcomes within the same family. However, the procedure is less conservative than other common adjustment methods, such as the well-known Bonferroni correction, because it also accounts for any correlation in test statistics among outcomes within the same family. We made this adjustment separately for each of the six groups of outcome measures described earlier in this chapter (and presented in Table III.1). That is, we adjusted the p-values accounting for multiple outcomes within each of the six groups of measures, but not for multiple outcomes measured across the different groups. We followed this approach because each group of outcomes aligns with a different research question. We base our substantive conclusions for each question only on the corresponding group of outcome measures. The number of outcomes measured in other groups has no bearing on our substantive conclusions for each question and therefore does not warrant an additional adjustment for multiple hypothesis testing. 26 FINAL IMPACTS OF THE T.O.P.P. PROGRAM IV. RESULTS The T.O.P.P. program succeeded in accomplishing its primary goal of reducing the incidence of rapid repeat pregnancy. Participants assigned to the treatment group were 17.4 percentage points less likely to report a repeat pregnancy within 18 months of their prior pregnancy, a large and statistically significant difference. The program also led to significant reductions in other important pregnancy outcomes, including the likelihood of having an unintended pregnancy and the total number of pregnancies. In addition, our evidence suggests that 18 months after program enrollment, T.O.P.P. decreased the sexual risk behaviors that it targets without leading to any negative effects on sexual risk behaviors that had not been targeted. Participants assigned to the treatment condition were more likely to report use of both LARC methods and effective contraceptive methods defined more broadly. Furthermore, participants assigned to the treatment group reported a similar number of sexual partners, were no more likely than those in the control group to report having sexual intercourse, and were less likely to report having had sex without a condom in the three months prior to the 18-month survey. A. Repeat pregnancy and other pregnancy-related outcomes Our evidence suggests that the T.O.P.P. program had a large and statistically significant impact on the percentage of women reporting a repeat pregnancy at the end of the 18-month intervention (Table IV.1). Among participants in the treatment group, 20.8 percent reported having had a repeat pregnancy at the time of the 18-month follow-up survey, compared to 38.2 percent of participants in the control group. The difference of 17.4 percentage points is statistically significant. As discussed in Chapter III, we also examined the robustness of our results to the coding decisions we used to construct the outcome measure. We found similar impact estimates for all three alternative codings we considered (impacts of 16.6 percentage points, 16.9 percentage points, and 17.5 percentage points, respectively). All of these impact estimates reached statistical significance at the 1-percent level. We found that the T.O.P.P. program also had large and statistically significant impacts on other pregnancy-related outcomes (Table IV.1). Treatment group members were less likely than control group members to report both an unintended repeat pregnancy (16.9 percent versus 34.8 percent) and a repeat pregnancy ending in a live birth (10.3 percent versus 20.0 percent). Additionally, participants in the treatment group reported fewer lifetime pregnancies on the 18- month follow-up survey than did participants in the control group (an average of 1.6 versus 1.9). Differences across study groups in all three of these outcomes are statistically significant. In both the treatment and control groups, the percentage of study participants reporting a repeat pregnancy resulting in a live birth is lower than the percentage reporting any repeat pregnancy for two reasons. First, some women were currently pregnant at the time of the 18- month follow-up survey (8.3 percent for the treatment group and 12.2 percent for the control group). The difference between groups in the percentage of women currently pregnant at the time of the 18-month follow-up survey is not statistically significant (unadjusted p-value = 0.21). Second, some women reported that their most recent repeat pregnancy had ended in a miscarriage or stillbirth (3.8 percent for the treatment group and 4.5 percent for the control 27 FINAL IMPACTS OF THE T.O.P.P. PROGRAM group). The difference between groups in the percentage of women reporting a miscarriage or stillbirth is not statistically significant (unadjusted p-value = 0.71). As discussed in Chapter III, few women reported repeat pregnancies that had ended in abortions (seven participants in the control group and none in the treatment group). Table IV.1. Impacts on repeat pregnancy and related outcomes Treatment Control Measure group group Difference p-value Percentage of respondents reporting a repeat 20.8 38.2 -17.4** <0.01 pregnancy in the past 18 months Percentage of respondents who reported the following in the past 18 months: Unintended repeat pregnancy 16.9 34.8 -17.9** <0.01 Repeat pregnancy resulting in a live birth 10.3 20.0 -9.7* 0.03 Total lifetime number of pregnancies 1.6 1.9 -0.3** <0.01 Source: Surveys administered to study participants by evaluation team. Note: For each outcome, the numbers in the treatment group and control group columns are regression-adjusted predicted values of outcomes at the 18-month follow-up survey. P-values are adjusted for clustering of standard errors at the randomization block level. Sample sizes accounting for item nonresponse range from 424 to 461 depending on the measure. See Chapter III for a detailed description of each measure and the analytic methods. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. In additional exploratory analyses (see Appendix C), we also examined whether the observed program impacts on repeat pregnancy were likely to extend beyond the 18-month follow-up survey, or whether participants in the treatment group were likely to “catch up” with those in the control group soon after the program ended. To provide exploratory evidence on this issue, the 18-month follow-up survey asked participants whether they would be trying to get pregnant again or trying to avoid getting pregnant in the next year. Nearly two-thirds of the participants in both study groups (62.4 percent for treatment group and 60.0 for the control group) said they would be trying to avoid getting pregnant again in the next year. The reported differences between groups was small and not statistically significant (unadjusted p-value = 0.57). These exploratory findings suggest the potential for longer-term impacts of the program beyond the time of the 18-month follow-up survey. However, a more definitive test of the program’s longer-term effects requires a follow-up analysis of participants’ actual behaviors and outcomes, not just their intentions as reported in the 18-month survey. B. Contraceptive use and unprotected sex As discussed in Chapter I, our earlier interim report found promising short-term effects of the T.O.P.P. program in increasing rates of contraceptive use and reducing rates of unprotected sex after the first six months of the intervention (Smith et al. 2015). In particular, we found that study participants in the treatment group were statistically significantly more likely than those in the control group to report having used a LARC methods in the past three months (38.3 percent versus 21.4 percent) and less likely than those in the control group to report having had unprotected sex in the past three months (14.4 percent versus 24.8 percent). 28 FINAL IMPACTS OF THE T.O.P.P. PROGRAM We found similar results when looking at rates of contraceptive use and unprotected sex at the end of the 18-month intervention (Table IV.2). Among participants in the treatment group, 40.9 percent reported using LARC methods in the three months prior to the 18-month survey, compared to 25.9 percent of the control group. The difference of 15.0 percentage points is statistically significant. Moreover, the impacts of the program extended to more broadly defined measures of contraceptive use, including increased use of any hormonal birth control method or IUD (70.7 percent versus 59.2 percent) and increased use of any effective birth control method (82.9 percent versus 74.5 percent). Our findings also suggest that T.O.P.P. led to a statistically significant, 11.5 percentage point reduction in the incidence of unprotected sex. Among participants in the treatment group, 22.4 percent reported having unprotected sex in the past three months, compared to 34.0 percent in the control group. Table IV.2. Impacts on contraceptive use and unprotected sex Treatment Control Measure group group Difference p-value Percentage of respondents reporting use of the following birth control methods in the past 3 months: LARC methoda 40.9 25.9 15.0** <0.01 Any hormonal method or IUDb 70.7 59.2 11.5* 0.02 Any effective method of birth controlc 82.9 74.5 8.3 0.06 Percentage of respondents who reported having had 22.4 34.0 -11.5** <0.01 unprotected sex in the past 3 monthsd Source: Surveys administered to study participants by evaluation team. Note: For each outcome, the numbers in the treatment group and control group columns are regression-adjusted predicted values of outcomes at the 18-month follow-up survey. P-values are adjusted for clustering of standard errors at the randomization block level and for multiple outcomes measured within a single domain. Sample sizes accounting for item nonresponse range from 465 to 470 depending on the measure. See Chapter III for a detailed description of each measure and the analytic methods. a Includes the following methods: IUD and implant. b Includes the following methods: birth control pills, shot, patch, ring, IUD, and implant. c Includes the following methods: male condoms, female condoms, birth control pills, shot, patch, ring, IUD, implant, and vasectomy. d Defined as having sexual intercourse without using an effective birth control method in the past 3 months. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. IUD = intrauterine device; LARC = long-acting reversible contraceptive. A comparison of findings across the 6- and 18-month surveys suggests declining rates of contraceptive use among study participants in both the treatment and control groups (Figure IV.1). In the treatment group, the percentage of women who reported having used any effective birth control method in the past three months declined from 84.4 percent at the time of the 6- month survey to 82.9 percent at the time of the 18-month survey. In the control group, the percentage declined from 80.7 percent to 74.5 percent. At least part of this decline likely reflects women who stopped using contraception after experiencing a repeat pregnancy. Because reported rates of repeat pregnancy were higher in the control group (see Table IV.1), it follows that the control group also had a somewhat larger decline in contraceptive use rates. The decline in contraceptive use over time for both the treatment and control groups is also consistent with 29 FINAL IMPACTS OF THE T.O.P.P. PROGRAM the observed tendency of teen mothers to adopt contraception immediately following a birth but gradually discontinue use over time (Templeman et al. 2000). Figure IV.1. Use of any effective birth control method in the past three months 100 90 84.4 80.7 82.9 80 74.5 70 Percent of women 60 50 40 30 20 10 0 6-month follow-up 18-month follow-up Treatment Control Source: Surveys administered to study participants by evaluation team. Note: Percentages are regression-adjusted predicted values of outcomes at the 6- or 18-month follow-up survey. Measure includes any use of the following methods: male condoms, female condoms, birth control pills, shot, patch, ring, IUD, implant, and vasectomy. See Chapter III for a detailed description of the measure and analytic methods. IUD = intrauterine device. However, despite evidence of a decline in overall rates of contraceptive use, the prevalence of LARC use increased across the 6- and 18-month follow-up surveys. Among study participants in the treatment group, the percentage of young women who reported having used a LARC method in the past three months increased from 38.3 percent in the 6-month follow-up survey to 40.9 percent in the 18-month follow-up survey. Among participants in the control group, the percentage increased from 21.4 percent to 25.9 percent. As a result, the treatment-control difference in rates of LARC use held relatively steady over time: a difference of 16.9 percentage points at the time of the 6-month survey and 15.0 percentage points at the time of the 18-month survey. C. Other sexual risk behaviors Although T.O.P.P. focused on the promotion of highly effective contraceptive methods, such as LARC methods, we found no evidence that this focus led to any unintended spillover effects on sexual risk behaviors not directly targeted by the program. In particular, differences between the treatment and control groups in the prevalence of sexual activity and the number of 30 FINAL IMPACTS OF THE T.O.P.P. PROGRAM sexual partners in the past three months were small and did not reach statistical significance (Table IV.3). Most study participants were sexually active at the time of the 18-month follow-up survey (82.2 percent of participants in the treatment group and 85.4 percent of participants in the control group). Participants in both the treatment and control groups reported having on average one sexual partner in the past three months. Although T.O.P.P. did not explicitly target increasing condom use, our results suggest that the program reduced rates of sex without a condom by 10.3 percentage points (Table IV.3). In particular, 54.2 percent of participants in the treatment group reported having had sexual intercourse without a condom in the past three months, compared with 64.5 percent of participants in the control group. The higher rate of unprotected sex among control group participants may partly reflect women who stopped using condoms after experiencing a repeat pregnancy. Table IV.3. Impacts on other sexual risk behaviors Treatment Control Measure group group Difference p-value Percentage of respondents who reported the following in the past 3 months: Had sexual intercourse 82.2 85.4 -3.2 >0.99 Had sexual intercourse without a condom 54.2 64.5 -10.3* 0.04 Number of sexual partners in the past 3 months 1.0 1.0 0.0 >0.99 Source: Surveys administered to study participants by the evaluation team. Note: For each outcome, the numbers in the treatment group and control group columns are regression-adjusted predicted values of outcomes at the 18-month follow-up survey. P-values are adjusted for clustering of standard errors at the randomization block level and for multiple outcomes measured within a single domain. Sample sizes accounting for item nonresponse range from 455 to 465 depending on the measure. See Chapter III for a detailed description of each measure and the analytic methods. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. D. Educational outcomes Finally, we examined whether the program’s focus on reducing rates of repeat pregnancy had any impacts on the educational outcomes of program participants. Although educational attainment was not an outcome directly targeted by T.O.P.P., it may be an outcome of policy relevance since there is well-documented evidence on the economic benefits of higher levels of schooling. On the basis of data from the 18-month follow-up survey, we found no statistically significant differences in school enrollment or high school completion rates between the treatment and control groups (Table IV.2). Just over a third of the study participants in both groups were currently enrolled in school (39.3 percent for the treatment group and 35.4 percent for the control group). Over half the participants in both groups had completed high school or received a GED (56.1 for the treatment group and 61.5 percent for the control group). 31 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Table IV.4. Impacts on educational outcomes Treatment Control Measure group group Difference p-value Percentage of respondents who reported the following: Currently enrolled in school 39.3 35.4 3.9 0.70 Completed high school or GED 56.1 61.5 -5.5 0.24 Source: Surveys administered to study participants by the evaluation team. Note: For each outcome, the numbers in the treatment group and control group columns are regression-adjusted predicted values of outcomes at the 18-month follow-up survey. P-values are adjusted for clustering of standard errors at the randomization block level and for multiple outcomes measured within a single domain. Sample size accounting for item nonresponse is 466 for both measures. See Chapter III for a detailed description of each measure and the analytic methods. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. GED = General Educational Development certification. 32 FINAL IMPACTS OF THE T.O.P.P. PROGRAM V. DISCUSSION AND CONCLUSIONS This report presents postintervention impacts of the Teen Options to Prevent Pregnancy (T.O.P.P.) program, an innovative 18-month clinic-based program that aims to reduce rapid repeat pregnancies among low-income adolescent mothers. Prior research indicates that adolescent mothers are at high risk of experiencing a rapid repeat pregnancy, the vast majority of which are reported to be unintended (Mosher et al. 2012). The T.O.P.P. program seeks to promote healthy birth spacing and decrease rates of rapid repeat pregnancy through a unique combination of telephone-based care coordination and individualized support services. The program is delivered by trained nurse educators using patient-centered motivational interviewing techniques and emphasizes the use of highly effective contraceptive methods, such as LARCs, for patients who want to delay future childbearing. Drawing on data from a rigorous random assignment evaluation involving a sample of nearly 600 low-income adolescent mothers from the Columbus, Ohio, area, our findings indicate that the T.O.P.P. program was highly successful in reducing rates of rapid repeat pregnancy at the end of the full 18-month intervention. Access to the program reduced the chances of reporting a repeat pregnancy by nearly half, from 38.2 percent for young women in the control group to 20.8 percent for those in the treatment group. We also found evidence of large and statistically significant program impacts on other key pregnancy-related outcomes, including the rate of unintended repeat pregnancies (16.9 percent for the treatment group versus 34.8 percent for the control group) and the percentage of women who reported giving birth (10.3 percent for the treatment group versus 20.0 percent for the control group). The reported birth rate is lower than the reported rate of repeat pregnancies in part because some women were still pregnant at the time we administered our survey. Additional analyses suggest the program achieved these results primarily by increasing rates of LARC use among program participants and reducing the incidence of unprotected sexual intercourse. In an earlier interim report examining the short-term effects of the T.O.P.P. program 6 months after study enrollment, we found that access to the program increased the share of women who reported using LARCs and reduced reported rates of unprotected sexual intercourse (Smith et al. 2015). In the present report, we found similar results when looking at rates of unprotected sex and LARC use at the end of the full 18-month intervention. Among participants in the treatment group, 40.9 percent reported LARC use in the three months prior to the 18- month survey, compared to 25.9 percent of the control group. For unprotected sex, 22.4 percent of participants in the treatment group reported having had sex without any effective method of birth control in the three months prior to the survey, compared to 34.0 percent of participants in the control group. We found no evidence that the program’s focus on reducing barriers to highly effective contraceptive methods, such as LARCs, had any unintended spillover effects on other types of sexual risk behaviors. In particular, participants assigned to the treatment group were no more likely than those in the control group to report having had sexual intercourse or a greater number of sexual partners. Moreover, participants in the treatment group were less likely than those in the control group to report having had sex without a condom in the past three months. 33 FINAL IMPACTS OF THE T.O.P.P. PROGRAM These findings extend prior research by demonstrating that the combination of motivational interviewing and reducing barriers to highly effective contraceptive methods, such as LARCs, may help to reduce rates of unintended and rapid repeat pregnancy, even among a population of women not actively seeking to adopt birth control or avoid a future pregnancy. In recent years, the St. Louis-based Contraceptive CHOICE project has received considerable attention for its positive results in reducing adolescent pregnancy rates relative to national averages (Secura et al. 2014). The CHOICE project sought to improve access to highly effective contraceptive methods primarily by offering LARCs at no cost to adolescents seeking birth control. The T.O.P.P. program sought to address barriers in a different way—namely, through a mix of individualized motivational interviewing sessions, transportation services, and optional access to clinical services. The T.O.P.P. program is also notable because it did not screen out or exclude study participants based on contraceptive use behaviors or intentions, making the program and study sample reflective of a typical clinical environment. Future research could usefully explore the longer-term impacts of the T.O.P.P. program. Our findings indicate that women who received access to the program were less likely to report having had a repeat pregnancy at the end of the full 18-month intervention. However, we do not know whether or how this impact might change their future pregnancy and childbearing trajectories, or whether or how it might change the outcomes for their children. In additional exploratory analyses, we found that women in the treatment group were just as likely as those in the control group to say they wanted to avoid getting pregnant in the year after the 18-month follow-up survey. This exploratory finding suggests that the difference in repeat pregnancy rates documented in this report has the potential to persist over a longer period. However, a more definitive assessment of the program’s longer-term impacts requires longer-term follow-up data and additional research. It is also unclear which component of the T.O.P.P. program is driving our observed results. Program effects may be driven by the motivational interviewing sessions, by the program’s efforts to reduce the logistical barriers to contraceptive services, or by the referral aspects of the program. Alternatively, it might be that all three programmatic components are required to successfully reduce rates of repeat pregnancy. The findings of our earlier interim report suggested that participants’ (1) increased exposure to information on birth control methods and sources and (2) increased access to contraceptives services both played a role in promoting the use of highly effective contraceptive methods and reducing rates of unprotected sex (Smith et al. 2015). Future research could explore the relative importance of different program components through a more formal analysis of the program mechanisms or pathways. The findings presented in this report may not necessarily generalize to populations or settings outside our study sample. By design, the evaluation focused on a specific set of low- income adolescent mothers living in the Columbus, Ohio, area—those who were enrolled in Medicaid and had recently delivered a baby or received prenatal care at an OhioHealth facility at the time of study recruitment. Within this target population, the study was further limited to the subset of women who were successfully contacted before leaving the health care facility and agreed to participate in the study. This particular sample of adolescent mothers may differ from those in other parts of the country or even from other adolescent mothers in the same area. In addition, the study was conducted within a setting—the OhioHealth hospital system—that was particularly well suited to implement this type of program model. T.O.P.P. program staff 34 FINAL IMPACTS OF THE T.O.P.P. PROGRAM leveraged and drew heavily on existing OhioHealth personnel, systems, and infrastructure to identify program participants and deliver key program support services. For example, having access to OhioHealth’s electronic scheduling system greatly helped program staff identify and recruit program participants. The program also benefited from the ability to provide contraceptive services directly to program participants through the T.O.P.P. clinic. Other health care providers or organizations seeking to replicate the positive outcomes presented in this report must think carefully about their own local context and the availability of a comparable mix of supports and resources. 35 This page has been left blank for double-sided copying. FINAL IMPACTS OF THE T.O.P.P. PROGRAM REFERENCES Baldwin, Maureen K., and Alison B. Edelman. “The Effect of Long-Acting Reversible Contraception on Rapid Repeat Pregnancy in Adolescents: A Review.” Journal of Adolescent Health, vol. 52 , no. 4, 2013, pp. S47–S53. Barnet B., J. Liu, M. DeVoe, A.K. Duggan, M.A. Gold, and E. Pecukonis. “Motivational Intervention to Reduce Rapid Subsequent Births to Adolescent Mothers: A Community Based Randomized Trial.” Annals of Family Medicine, vol. 7, 2007, pp. 436–445. Biggs, M. Antonia, Sarah Combellick, Abigail Arons, and Claire D. Brindis. “Educational Barriers, Social Isolation, and Stable Romantic Relationships Among Pregnant Immigrant Latina Teens.” Hispanic Health Care International, vol. 11, no. 1, 2013, pp. 38–46. Conde-Agudelo, Agustin, Anyeli Rosas-Bermudez, and Ana Cecilia Kafury-Goeta. “Birth Spacing and Risk of Adverse Perinatal Outcomes: A Meta Analysis.” Journal of the American Medical Association, vol. 295, 2006, pp. 1809–1823. Copen, Casey E., Marie E. Thoma, and Sharon Kirmeyer. “Interpregnancy Intervals in the United States: Data from the Birth Certificate and the National Survey of Family Growth.” National Vital Statistics Reports, vol. 64, no 3. Hyattsville, MD: National Center for Health Statistics, 2015. Fu, Haishan, Jacqueline E. Darroch, Stanley K. Henshaw, and Elizabeth Kolb.“Measuring the Extent of Abortion Underreporting in the 1995 National Survey of Family Growth.” Family Planning Perspectives, vol. 20, no. 3, 1998, pp. 128–138. Hettema, Jennifer, Julie Steele, and William R. Miller. “Motivational Interviewing.” Annual Review of Clinical Psychology, vol. 1, 2005, pp. 91–111. Hothorn, Torsten, Frank Bretz, and Peter Westfall. “Simultaneous Inference in General Parametric Models.” Biometrical Journal, vol. 50, no. 3, 2008, pp. 346–363. Jones, Rachel K., and Kathryn Kost. “Underreporting of Induced and Spontaneous Abortion in the United States: An Analysis of the 2002 National Survey of Family Growth.” Studies in Family Planning, vol. 38, no. 3, 2007, pp. 187–197. Matts, John P., and John M. Lachin. “Properties of Permuted-Block Randomization in Clinical Trials.” Controlled Clinical Trials, vol. 9, no 4, 1988, pp. 327–344. Meckstroth, Alicia, and Amanda Berger. “Preventing Rapid Repeat Teen Pregnancy with Motivational Interviewing and Contraceptive Access: Implementing Teen Options to Prevent Pregnancy (T.O.P.P.).” Princeton, NJ: Mathematica Policy Research, 2014. Mosher, William D., Jo Jones, and Joyce C. Abma. “Intended and Unintended Births in the United States: 1982–2010.” National Health Statistics Reports, no. 55. Hyattsville, MD: National Center for Health Statistics, 2012. 37 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Schochet, Peter Z. “An Approach for Addressing the Multiple Testing Problem in Social Policy Impact Evaluations.” Evaluation Review, vol. 33, no. 6, 2009, pp. 539–567. Schultz, Kenneth F., and David A. Grimes. “Generation of Allocation Sequences in Randomised Trials: Chance, Not Choice.” Lancet, vol. 359, 2002, pp. 515–519. Secura, Gina M., et al. “Provision of No-Cost, Long-Acting Contraception and Teenage Pregnancy.” New England Journal of Medicine, vol. 371, 2014, pp. 1316–1323. Smith, Kimberly, Dana Rotz, Brian Goesling, Elizabeth Cook, Kelly Murphy, and Jack Stevens. “Interim Impacts of the Teen Options to Prevent Pregnancy Program.” Princeton, NJ: Mathematica Policy Research, December 2015. Social and Character Development Research Consortium. “Efficacy of Schoolwide Programs to Promote Social and Character Development and Reduce Problem Behavior in Elementary School Children.” (NCER 2011-2001.) Washington, DC: National Center for Education Research, Institute of Education Sciences, U.S. Department of Education, 2010. Steiner, Riley J., Nicole Liddon, Andrea L. Swartzendruber, Catherine N. Rasberry, and Jessica M. Sales. “Long-Acting Reversible Contraception and Condom Use Among Female US High School Students: Implications for Sexually Transmitted Infection Prevention.” JAMA Pediatrics, 170, no. 5, 2016, pp. 428–434. Taylor‐Seehafer, Margaret, and Lynn Rew. “Risky Sexual Behavior Among Adolescent Women.” Journal for Specialists in Pediatric Nursing, vol. 5, no. 1, 2000, pp. 15–25. Templeman, Claire L., Vernon Cook, L. Jane Goldsmith, Jacqueline Powell, and S. Paige Hertweck. “Postpartum Contraceptive Use Among Adolescent Mothers.” Obstetrics and Gynecology, vol. 95, no. 5, 2000, pp. 770-776. 38 FINAL IMPACTS OF THE T.O.P.P. PROGRAM APPENDIX A NONRESPONSE ANALYSIS 39 This page has been left blank for double-sided copying. FINAL IMPACTS OF THE T.O.P.P. PROGRAM In this appendix, we examine the characteristics of the study participants lost to follow-up at the time of the 18-month follow-up survey. As reported in Chapter II, among the 598 young women who enrolled in the study and were randomly assigned to the treatment and control groups, 472 completed the 18-month follow-up survey, for an overall response rate of 79 percent. The remaining 126 participants did not complete the 18-month follow-up survey and were therefore excluded from the impact analyses presented in this report. To better understand the characteristics of the study participants lost to follow-up, we used data from the baseline survey to compare the samples of follow-up survey respondents (n = 472) and nonrespondents (n = 126). The characteristics of the survey nonrespondents were generally similar to those of the participants who responded to the survey. The two groups had similar average ages, had similar levels of economic disadvantage, had similar racial and ethnic backgrounds, reported similar numbers of prior pregnancies, and had similar family relationships (Table A.1). The two groups were also similar on baseline measures of exposure to information on reproductive health topics (Table A.2) and most sexual risk behaviors (Table A.3). Among all the demographic and personal characteristics examined, we found three statistically significant differences. Compared to survey respondents, nonrespondents were (1) less likely to report having any postsecondary education, (2) less likely to report their education level in the other category, and (3) less likely to report using an effective method of birth control in the three months prior to becoming pregnant. The difference in birth control use suggests that the nonrespondents may have engaged in more risky sexual behavior prior to becoming pregnant relative to the overall study sample. This difference may affect the external validity or generalizability of our study findings—meaning that our results may not necessarily generalize to the young women with the highest rates of risk sexual behavior prior to becoming pregnant. However, none of the observed differences present a threat to the internal validity of our impact estimates. As shown in Chapter II, all measures of sexual behaviors were similar for survey respondents in the treatment and control groups. 41 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Table A.1. Baseline sociodemographic characteristics Respondent Nonrespondent Variable mean mean Difference p-value Age at random assignment (years) 18.39 18.42 -0.03 0.84 Highest level of education completed (%) No high school 5.6 4.8 0.8 0.74 Some high school 48.3 55.2 -6.9 0.15 High school graduate or GED 37.6 36.8 0.8 0.88 Any postsecondary education 7.3 3.2 4.1* 0.04 Other 1.3 0.0 1.3* 0.01 Economic situation (%) Household received SNAP or WIC in past 30 days 90.4 93.1 -2.7 0.32 Household received TANF in past 30 days 25.4 26.6 -1.2 0.80 Household received other assistance in past 30 days 24.1 22.8 1.3 0.77 Race/ethnicity (%) White, non-Hispanic 47.0 50.8 -3.8 0.38 Black, non-Hispanic 36.4 35.5 0.9 0.85 Hispanic 7.1 4.0 3.1 0.12 Other race/ethnicity or multiracial 9.5 9.7 -0.2 0.96 Pregnancy Pregnant at time of baseline survey (%) 24.8 20.2 4.7 0.26 Has been pregnant multiple times (%) 36.7 42.3 -5.6 0.32 Number of times pregnant (including most recent) 1.46 1.41 0.04 0.51 Current relationship with baby’s father (%) Married or engaged 23.1 19.7 3.4 0.39 Dating (seriously or casually) 47.5 46.7 0.8 0.90 Other (no contact; have contact but not romantically 29.4 33.6 -4.2 0.38 involved; or other relationship specified) Family structure (%) Lives with both biological parents 11.0 8.7 2.3 0.43 Lives with exactly one biological parent 39.4 48.4 -9.0 0.10 Lives with neither biological parent 49.6 42.9 6.7 0.21 Sample sizea 472 126 Source: Baseline survey administered to study participants before the start of the program. Notes: Reported means are from regressions that control for random assignment strata. P-values are adjusted for clustering of standard errors at the randomization block level. See Appendix B for a description of the measures. a Reported sample size does not account for item nonresponse for any measures included in the table. *Significantly different from zero at the .05 level, two-tailed test. GED = General Educational Development certification; SNAP = Supplemental Nutrition Assistance Program; TANF = Temporary Assistance for Needy Families; WIC = Special Supplemental Nutrition Program for Women, Infants, and Children. 42 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Table A.2. Exposure to reproductive health information Responden Nonrespondent Variable t mean mean Difference p-value In past 12 months, received information on: (%) Relationships 65.5 61.8 3.7 0.49 Abstinence from sex 49.6 50.8 -1.3 0.81 Methods of birth control 85.8 85.5 0.3 0.93 Where to get birth control 84.1 86.7 -2.6 0.41 Sexually transmitted infections 82.9 76.4 6.5 0.08 Talking to a partner about sex or birth control 74.4 68.9 5.5 0.20 How to say no to sex 75.3 68.9 6.4 0.16 Sample sizea 472 126 Source: Baseline survey administered to study participants before the start of the program. Notes: Reported means are from regressions that control for random assignment strata. P-values are adjusted for clustering of standard errors at the randomization block level. See Appendix B for a description of the measures. a Reported sample size does not account for item nonresponse for any measures included in the table. Table A.3. Baseline sexual behaviors Respondent Nonrespondent Variable mean mean Difference p-value In 3 months prior to becoming pregnant: Used a LARC methoda 1.0 1.9 -0.9 0.52 Used a hormonal method of birth control or IUDb 30.7 29.4 1.3 0.79 Used an effective method of birth controlc 67.5 55.9 11.6* 0.03 Had unprotected sexual intercoursed 73.5 68.0 5.5 0.31 Had sexual intercourse without a condom 88.5 86.5 2.0 0.61 Lifetime number of sexual partners 4.93 5.65 -0.72 0.49 Sample sizee 472 126 Source: Baseline surveys administered to study participants before the start of the program. Notes: Reported means are from regressions that control for random assignment strata. P-values are adjusted for clustering of standard errors at the randomization block level. See Appendix B for a description of the measures. a Includes the following methods: IUD and implant. b Includes the following methods: birth control pills, shot, patch, ring, IUD, and implant. c Includes the following methods: male condoms, female condoms, birth control pills, shot, patch, ring, IUD, implant, and vasectomy. d Defined as having sexual intercourse without using any effective birth control method. e Reported sample size does not account for item nonresponse for any measures included in the table. *Significantly different from zero at the .05 level, two-tailed test. IUD = intrauterine device; LARC = long-acting reversible contraceptive. 43 This page has been left blank for double-sided copying. FINAL IMPACTS OF THE T.O.P.P. PROGRAM APPENDIX B DATA AND MEASURES 45 This page has been left blank for double-sided copying. FINAL IMPACTS OF THE T.O.P.P. PROGRAM In this appendix, we provide more detailed information on the survey data collection and measures. We begin by describing the survey design and administration. We then provide a more detailed description of how we constructed several key outcome measures. We end by listing the baseline measures considered as candidate covariates for the regression models. A. Survey design and administration As discussed in Chapter III, the impact estimates presented in this report are based mainly on survey data collected at two points: a baseline survey administered upon enrollment in the study and a follow-up survey administered about 18 months after study enrollment. For the baseline survey, OhioHealth program staff distributed to each participant a self-administered paper-and-pencil interviewing (PAPI) questionnaire. For the follow-up surveys, trained data collection staff from Mathematica and Nationwide Children’s Hospital administered the surveys either by telephone or in person. The baseline and follow-up surveys followed a similar structure and were designed to capture a broad range of measures of family background and demographic characteristics, views and attitudes, sexual activity, past pregnancies, and future intentions. The surveys were developed by members of the PPA research team in coordination with OhioHealth and Nationwide Children’s Hospital. They drew on items found in well-established surveys such as the National Longitudinal Study of Adolescent Health, National Longitudinal Survey of Youth, Youth Risk Behavior Survey, and National Survey of Family Growth. In some cases, we had to adapt the questions to fit our PAPI and telephone survey modes. We also made minor changes to question wording and response categories to align with our target population of expectant or parenting young women. As with any self-reported survey, the survey responses may be subject to reporting bias, and the risk of bias may differ between the treatment and control groups. For example, participants in the treatment group may have been less likely to report a repeat pregnancy or having had unprotected sex because they were embarrassed to admit to a behavior the program discourages. Such underreporting could lead to a spurious finding of lower rates of repeat pregnancy or contraceptive use among young women in the treatment group. Conversely, another possibility is that the program made participants feel more comfortable talking about their reproductive health behaviors and therefore more likely to report a repeat pregnancy or involvement in risk sexual behaviors. This type of effect could lead to a spurious finding of higher rates repeat pregnancy or contraceptive use among young women in the treatment group. We took several steps to minimize these risks. To help encourage honest reporting, the follow-up surveys were administered by independent field staff trained and employed by the study team, not OhioHealth program staff or anyone else personally connected to the study participants. In addition, the data collectors did not know the respondents’ treatment status, to avoid any intentional or unintentional difference in data collection procedures between the treatment and control groups. As a final precaution, we had the telephone interviewers use a standardized script to administer the follow-up surveys to ensure both uniformity in the data collection procedures and objectivity in the question wording. The interviewers reminded participants that their answers would be kept confidential and encouraged them to respond truthfully to the questions. 47 FINAL IMPACTS OF THE T.O.P.P. PROGRAM B. Outcome measures Our impact analysis focused on six different groups of outcomes, each corresponding to one of the study’s research questions: (1) repeat pregnancy, (2) other pregnancy-related outcomes, (3) contraceptive use, (4) unprotected sex, (5) other sexual risk behaviors, and (6) education. We provide a summary description of these measures in Chapter III of the report. In this section, we provide additional detail on how we constructed the measures of contraceptive use and sexual risk behaviors. 1. Contraceptive use We constructed three variables capturing the use of different contraceptive methods. The 18- month follow-up survey asked participants to report their recent (occurring within the past three months) use of each of 14 different traditional and modern contraceptive methods. For each method, the survey asked participants whether they had used the method none of the time, some of the time, half of the time, most of the time, or all of the time. The survey also included an open-ended response field that allowed participants to provide alternative names or labels for different contraceptive methods. We used responses to these questions to construct three binary (yes/no) indicator variables: (1) any use of a long-acting reversible contraceptive (LARC) method, (2) any use of a hormonal method of contraception or IUD (nonbarrier method), and (3) any use of any effective birth control method. Table B.1 shows the contraceptive methods that each variable included. We also back-coded any relevant responses from the open-ended survey question. For all three measures, we coded a participant as having used the method if she reported using it some of the time, half of the time, most of the time, or all of the time. The resulting variables therefore reflect measures of any use, not consistency or duration of use. Table B.1. Methods included in summary measures of contraceptive use Use of a LARC Use of a hormonal Use of any effective birth Type of contraceptive method method method or IUD control method Condoms X Birth control pills X X The shot or Depo-Provera X X The patch X X The ring or NuvaRing X X An IUD such as Mirena or Paragard X X X An implant such as IMPLANON X X X Male vasectomy X IUD = intrauterine device. 2. Sexual risk behaviors We constructed four separate measures of sexual risk behaviors. To determine whether T.O.P.P. was successful in reducing rates of unprotected sex, we constructed a binary (yes/no) indicator for whether the study participant reported having sex in the past three months without using any effective contraceptive method. To examine whether the program’s emphasis on promoting the use of highly effective contraceptive methods, such as LARCs, had any 48 FINAL IMPACTS OF THE T.O.P.P. PROGRAM unintended spillover effects to other types of sexual risk behaviors not directly targeted by the program, we also constructed three other measures of sexual risk behaviors: (1) a binary (yes/no) indicator of whether the participant reported having had sexual intercourse in the past three months, (2) a binary (yes/no) indicator of whether the participant reported having had sexual intercourse without a condom in the past three months, and (3) a continuous variable measuring the number of sexual partners in the past three months. We constructed these variables in a stepwise fashion from the following series of four sexual behavior questions included on the survey: 1. Please think about the past three months, that is, from [date three months ago] until today. In the past three months, have you had sexual intercourse, even once? 2. In the past three months, how many different people have you had sexual intercourse with, even once? 3. In the past three months, have you had sexual intercourse without you or your partner using a condom? 4. In the past three months, have you had sexual intercourse without you or your partner using any of these methods of birth control: condoms; birth control pills; the shot or Depo- Provera; the patch; the ring or NuvaRing; an IUD such as Mirena or Paragard; or implants such as IMPLANON? Using responses to these questions, we first constructed the indicator variable for whether the participant reported having had sexual intercourse in the past three months. We then constructed the variables for number of partners, sex without a condom, and sex without any effective contraceptive method. If participants reported being abstinent in the past three months, we retained them in the analysis and assigned them a value of zero on all four outcomes. In constructing these outcomes, we accounted for any observed inconsistent or discrepant responses across different items—for example, participants who reported having had sex in the past three months but also reported that they had had no sexual partners in that time period. We checked for inconsistencies across the four main survey questions listed above as well as across additional survey questions that asked about the frequency of sexual intercourse, sex without a condom, and unprotected sex. To resolve any inconsistent responses across all the related items in the survey, we developed the following set of rules and procedures: • Resolve inconsistencies in responses related to sexual intercourse of any type. We first examined inconsistencies between responses to questions about having had any sexual intercourse, the frequency of sexual intercourse, and the number of partners. We found 10 cases in which two or more of these variables conflicted. We classified 8 of these responses as indicating that the participant did have sex, and 1 as indicating that she did not have sex. For the remaining case, there was not sufficient information and we thus set all outcome measures related to recent sexual activity to missing. • Resolve inconsistencies in responses related to sexual intercourse without a condom. We examined inconsistencies between questions asking about having any sex without a condom, the frequency of sex without a condom, and use of condoms. There were 54 cases 49 FINAL IMPACTS OF THE T.O.P.P. PROGRAM of conflict across these variables. We recoded 16 of these responses as indicating that the participant had sex without a condom and 6 as indicating that she did not do so. We coded 32 cases as missing, as there was no strong evidence in either direction. • Resolve inconsistencies in responses related to sexual intercourse without any birth control method. We examined inconsistencies between questions asking about any sex without the use of birth control, frequency of sex without birth control, and use of various birth control methods. There were 95 cases of conflict across these survey items. We recoded 21 cases as indicating that the participant had sex without birth control, 65 as indicating that she did not have sex without birth control, and 9 cases as missing. C. Candidate covariates As discussed in Chapter III, we included several types of baseline covariates in the regression models used to estimate program impacts. Some of these covariates were included to account for the stratification used for random assignment. Other covariates were included to improve the precision of the impact estimates. To help select the covariates used for precision gains, we used a data-driven stepwise selection procedure developed previously in the literature (Social and Character Development Research Consortium 2010). For this procedure, we considered as candidate covariates a variety of baseline variables that have been shown in other studies to have a strong link with sexual risk behavior and repeat pregnancy. Table B.2 provides a complete list of the variables considered. 50 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Table B.2. Measures of baseline sample characteristics Measure Definition Demographic and personal characteristics Age Continuous variable for age at randomization. Education level Categorical variable with categories for (1) no high school, (2) some high school, (3) high school graduate or GED, (4) any postsecondary education, and (5) other educational attainment. Race/ethnicity Categorical variable with categories for (1) Hispanic, (2) non-Hispanic white, (3) non-Hispanic black, and (4) non-Hispanic other race. Main language spoken at Binary variable: equals 1 if participant reported primarily speaking a language other home is not English than English at home; equals 0 if participant reported speaking primarily English at home. Importance of religion Binary variable: equals 1 if participant reported that religion is very important in her life; equals 0 if participant reported religion is somewhat important or not at all important. Religious attendance Binary variable: equals 1 if participant reported attending religious services once per week or more often; equals 0 if participant reported attending religious services less than once per week. Pregnant at baseline Binary variable: equals 1 if participant was pregnant when the baseline survey was administered; equals 0 if participant was postpartum when the baseline survey was administered. Number of times pregnant Count variable for number of times a participant has been pregnant in the past, including the current pregnancy. Repeat pregnancy Binary variable: equals 1 if current or most recent pregnancy at baseline was not the respondent’s first pregnancy; equals 0 if current or most recent pregnancy at baseline was the respondent’s first pregnancy. Family structure Living situation Categorical variable with categories for (1) lives with both biological parents, (2) lives with exactly one biological parent, (3) lives with neither biological parent. Biological parents’ marital Categorical variable with categories for (1) biological parents are married, (2) status biological parents were previously married, (3) biological parents were never married. Biological parents Binary variable: equals 1 if biological parents currently live together; equals 0 if cohabitation biological parents do not currently live together (or one or both biological parents has passed away). Relationship with baby’s Categorical variable with categories for (1) married or engaged, (2) dating, and (3) father at conception other. Relationship with baby’s Categorical variable with categories for (1) married or engaged, (2) dating, and (3) father at survey other. Economic situation Receive SNAP or WIC Binary variable: equals 1 if anyone in household received transfers from the Supplemental Nutrition Assistance Program (SNAP, otherwise known as food stamps) or Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) in the 30 days prior to survey; equals 0 if no one received such assistance. Receive TANF Binary variable: equals 1 if anyone in household received transfers from the Temporary Assistance for Needy Families (TANF) program in the 30 days prior to survey; equals 0 if no one received such assistance. Receive other assistance Binary variable: equals 1 if anyone in household received Unemployment Insurance or Social Security Disability Income; equals 0 if no one received such assistance. 51 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Measure Definition Receipt of information on sexual and reproductive health Received information on Binary variable: equals 1 if participant received any information in the past 12 relationships months on relationships, dating, marriage, or family life; equals 0 if participant did not receive this information. Received information on Binary variable: equals 1 if participant received any information in the past 12 abstinence from sex months on abstaining from sex; equals 0 if participant did not receive this information. Received information on Binary variable: equals 1 if participant received any information in the past 12 methods of birth control months on methods of birth control; equals 0 if participant did not receive this information. Received information on Binary variable: equals 1 if participant received any information in the past 12 where to get birth control months on where to get birth control; equals 0 if participant did not receive this information. Received information on Binary variable: equals 1 if participant received any information in the past 12 STIs months on STIs; equals 0 if participant did not receive this information. Received information on Binary variable: equals 1 if participant received any information in the past 12 how to talk to partner about months on how to talk to partner about whether to have sex or use birth control; sex equals 0 if participant did not receive this information. Received information on Binary variable: equals 1 if participant received any information in the past 12 how to say no to sex months on how to say no to sex; equals 0 if participant did not receive this information. Number of issues received Count variable for the number of above topics on which a participant received information on information on in the past 12 months; variable ranges from 0 to 7. Knowledge Correct knowledge of Binary variable: equals 1 if participant reported that condoms can prevent efficacy of condoms to pregnancy “a lot;” equals 0 if participant responded to question in any other way. prevent pregnancy Correct knowledge of Binary variable: equals 1 if participant reported that condoms can prevent efficacy of condoms to transmission of HIV, Chlamydia, and gonorrhea “a lot;” equals 0 if participant prevent STI transmission responded to question in any other way. Correct knowledge of Binary variable: equals 1 if participant reported that birth control pills can prevent efficacy of birth control pills pregnancy “a lot;” equals 0 if participant responded to question in any other way. to prevent pregnancy Correct knowledge of Binary variable: equals 1 if participant reported that birth control pills can “not at all” efficacy of birth control pills prevent transmission of HIV and “not at all” prevent transmission of Chlamydia and to prevent STI transmission gonorrhea; equals 0 if participant responded to these questions in any other way. Knew benefits of birth Binary variable: equals 1 if participant reported awareness of the benefits of birth spacing at baseline spacing; equals 0 otherwise. Attitudes Birth control is pretty easy to Response to a single survey question; values range from 1 to 5 with higher get numbers indicating stronger agreement. Condoms are pretty easy to Response to a single survey question; values range from 1 to 5 with higher get numbers indicating stronger agreement. Women can trust what Response to a single survey question; values range from 1 to 5 with higher doctors say about birth numbers indicating stronger agreement. control Perceptions of ease of use Average of responses to two survey questions; values range from 1 to 5 with of birth control higher numbers indicating stronger agreement. Perceptions about need for Response to a single survey question; values range from 1 to 5 with higher condoms numbers indicating stronger agreement. Perceptions about need for Average of responses to two survey questions; values range from 1 to 5 with birth control other than higher numbers indicating stronger agreement. condoms 52 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Measure Definition Intentions Intention to avoid pregnancy Binary variable: equals 1 if participant reported she will be trying to avoid becoming pregnant within the next 18 months; equals 0 if participant did not report she will be trying to avoid becoming pregnant within the next 18 months or is unsure. Intention to avoid Binary variable: equals 1 if participant reported she will definitely not have sex, unprotected sex definitely not have sex without a condom, or definitely not have sex without some other form of birth control in the next 18 months; equals 0 if none of the above apply but associated survey items were not left blank. Preferred time to next Categorical variable with categories for (1) would like to become pregnant less pregnancy than 6 months after birth, (2) would like to become pregnant between 6 and 18 months after birth, (3) would like to wait more than 18 months to become pregnant again, (4) do not want to become pregnant again. Partner’s intention for Binary variable: equals 1 if participant reported that her partner will be trying to get pregnancy her pregnant in the next 18 months; equals 0 if participant has no partner or has a partner who will not be trying to get her pregnant in the next 18 months. GED = General Educational Development certification; SNAP = Supplemental Nutrition Assistance Program; STI = sexually transmitted infection; TANF = Temporary Assistance for Needy Families; WIC = Special Supplemental Nutrition Program for Women, Infants, and Children. 53 This page has been left blank for double-sided copying. FINAL IMPACTS OF THE T.O.P.P. PROGRAM APPENDIX C 18-MONTH IMPACTS ON MEDIATING FACTORS 55 This page has been left blank for double-sided copying. FINAL IMPACTS OF THE T.O.P.P. PROGRAM In this appendix, we provide evidence on the 18-month impacts of T.O.P.P. on four types of intermediate or potential mediating outcomes: (1) exposure to information on sexual and reproductive health topics, (2) knowledge about birth control methods, (3) attitudes and intentions, and (4) access to contraceptive services. As discussed in Chapter I, we analyzed these outcomes in our earlier interim report using data from the 6-month follow-up survey (Smith et al. 2015). The results of that analysis found statistically significant impacts of the program on participants’ exposure to information on sexual and reproductive health topics, as well as on participants’ access to contraceptive services. We found no statistically significant impacts on the measures of knowledge, attitudes, or intentions at the time of the 6-month follow-up survey. In this appendix, we examine the program’s longer-run impacts on these same outcomes using data from the 18-month follow-up survey. We estimated the impacts using the same multivariate regression framework described earlier in Chapter IV. See Smith et al. (2015) for a more detailed description of the measures. A. Exposure to information on sexual and reproductive health topics For the 18-month follow-up survey, the T.O.P.P. program had statistically significant impacts on participants’ exposure to information on two specific sexual and reproductive health topics: (1) methods of birth control and (2) abstinence (Table C.1). Among participants in the treatment group, 83.7 percent reported receiving information on methods of birth control in the past 12 months, compared to 72.3 percent of participants in the control group. The program had a similar impact on receipt of information about abstinence (52.9 percent for the treatment group and 35.9 percent for the control group received this information in the past 12 months). Impacts on exposure to all other topics were not statistically significant. The program increased the chances of receiving reproductive health information from a practitioner during a home visit but not at a health care facility (Table C.1). The majority of participants in both study groups reported receiving information on reproductive health topics from a nurse, doctor, or other professional in a health facility (77.2 percent for the treatment group and 72.4 percent for the control group). For this outcome, the reported difference between groups is not statistically significant. In contrast, participants in the treatment group were more than twice as likely as control group participants to report having received information from a doctor or nurse during a home visit (40.4 percent for the treatment group and 17.8 percent for the control group). This large reported difference (22.6 percent points) is statistically significant and likely attributable to the home visiting component of the T.O.P.P. program. 57 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Table C.1. Impacts on exposure to information on reproductive health topics Treatment Control Measure group group Difference p-value Percentage of respondents who reported receiving information on the following topics:a Relationships 44.8 47.3 -2.6 >0.99 Methods of birth control 83.7 72.3 11.4* 0.02 Where to get birth control 83.3 74.6 8.7 0.12 Abstinence 52.9 35.9 17.0** <0.01 Sexually transmitted infections 73.8 65.5 8.3 0.24 Talking to a partner about sex or birth control 75.1 68.5 6.6 0.48 How to say no to sex 74.5 68.5 6.1 0.77 Percentage of respondents who reported receiving information from each of the following sources:a Nurse or doctor during a facility visit 77.2 72.4 3.0 0.93 Health provider during a home visit 40.4 17.8 22.6** <0.01 Source: Surveys administered to study participants by the evaluation team. Note: For each outcome, the numbers in the treatment group and control group columns are regression-adjusted predicted values of outcomes at the 18-month follow-up survey. P-values are adjusted for clustering of standard errors at the randomization block level and for multiple outcomes measured within a single domain. Sample sizes accounting for item nonresponse range from 420 to 470 depending on the measure. a Questions refer to information received in the 12 months prior to survey administration. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. B. Knowledge about birth control methods We found no evidence that T.O.P.P. affected participants’ knowledge of the effectiveness of condoms and birth control pills at the time of the 18-month follow-up survey (Table C.2). In both study groups, levels of knowledge of these topics were relatively low. Slightly more than half the participants in both the treatment and control groups responded correctly to questions about the efficacy of condoms and birth control pills in preventing pregnancy. Levels of knowledge about STI prevention were low, with under 30 percent of participants in both study groups responding correctly to a question about the efficacy of condoms in preventing STIs. None of the treatment- control group differences in these outcomes are statistically significant. As discussed in our interim report (Smith et al. 2015), the survey did not measure knowledge of contraceptive methods other than condoms and birth control pills. Therefore, we cannot draw conclusions about whether the program impacted knowledge of LARC methods in particular or contraceptive methods more broadly. 58 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Table C.2. Impacts on knowledge about birth control methods Treatment Control Measure group group Difference p-value Percentage of respondents who reported correct knowledge of: Effectiveness of condoms in preventing pregnancy 55.5 57.5 -1.9 >0.99 Effectiveness of birth control pills in preventing pregnancy 55.5 53.2 2.3 >0.99 Effectiveness of condoms in preventing STIs 28.8 29.4 -0.6 >0.99 Effectiveness of the birth control pills in preventing STIs 71.3 68.1 3.1 >0.99 Source: Surveys administered to participants by the evaluation team. Note: For each outcome, the numbers in the treatment group and control group columns are regression-adjusted predicted values of outcomes at the 18-month follow-up survey. P-values are adjusted for clustering of standard errors at the randomization block level and for multiple outcomes measured within a single domain. Sample sizes accounting for item nonresponse range from 469 to 472 depending on the measure. STI = sexually transmitted infection. C. Attitudes and intentions The T.O.P.P. program did not have statistically significant impacts on participants’ attitudes toward birth control or intentions to avoid pregnancy (Table C.3). For each of the six attitude measures examined, the reported differences between the treatment and control groups are small and not statistically significant. Two of these outcomes measured attitudes toward condoms specifically, whereas the other outcomes measured attitudes about more general birth control methods other than condoms. The survey did not measure participants’ attitudes toward LARCs or specific contraceptive methods other than condoms. For the measure of intentions, a majority of participants in both study groups reported that they intended to avoid pregnancy in the next 12 months (62.4 percent of participants in the treatment group and 60.0 percent of participants in the control group). The difference between groups is not statistically significant. Table C.3. Impacts on attitudes and intentions Treatment Control Measure group group Difference p-value Perceived ease of access to condoms (single item, range: 4.56 4.57 -0.01 >0.99 1–5) Perceived ease of access to birth control other than 4.31 4.29 0.02 >0.99 condoms (single item, range: 1–5) Perceived trust in birth control providers (single item, range: 4.03 3.94 0.10 >0.99 1–5) Perceived ease of using birth control (single item, range: 1– 3.48 3.45 0.03 >0.99 5) Perceived need for condoms (average of two items, range: 4.63 4.57 0.06 >0.99 1–5) Perceived need for birth control other than condoms (single 4.22 4.17 0.05 >0.99 item, range: 1–5) Percentage of respondents indicating an intention to avoid 62.4 60.0 2.3 0.57 pregnancy in the next 12 months Source: Surveys administered to study participants by the evaluation team. Note: For each outcome, the numbers in the treatment group and control group columns are regression-adjusted predicted values of outcomes at the 18-month follow-up survey. P-values are adjusted for clustering of standard errors at the randomization block level and for multiple outcomes measured within a single domain. Sample sizes accounting for item nonresponse range from 464 to 470 depending on the measure. 59 FINAL IMPACTS OF THE T.O.P.P. PROGRAM D. Access to contraceptive services At the time of the 18-month follow-up survey, T.O.P.P. program participants reported increased receipt of birth control from a health care provider. Among treatment group participants, 72.3 percent reported receiving birth control from a doctor or nurse in the past six months, compared with 58.5 percent of the control group (Table C.4). The reported difference of 13.8 percentage points is statistically significant. The finding of a statistically significant impact on this outcome is consistent with T.O.P.P.’s emphasis on increasing access to medical providers offering contraceptive services. Table C.4. Impacts on access to birth control Treatment Control Measure group group Difference p-value Percentage of respondents who reported receiving 72.3 58.5 13.8** <0.01 birth control from a doctor or nurse in past 12 months Source: Surveys administered to study participants by the evaluation team. Note: The numbers in the treatment group and control group columns are regression-adjusted predicted values of outcomes at the 18-month follow-up survey. The p-value is adjusted for clustering of standard errors at the randomization block level and for multiple outcomes measured within a single domain. Sample size accounting for item nonresponse is 472. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. 60 FINAL IMPACTS OF THE T.O.P.P. PROGRAM APPENDIX D SENSITIVITY ANALYSIS 61 This page has been left blank for double-sided copying. FINAL IMPACTS OF THE T.O.P.P. PROGRAM The main impact findings presented in Chapter IV of this report are derived from a particular set of analytic decisions, ranging from the data cleaning procedures used to construct the outcome measures to the specification of the regression models. We made these decisions in accordance with established research standards and the particular features of our study design. However, we also investigated the sensitivity of our results to alternative analytic decisions. In this appendix, we present findings from four types of sensitivity tests: • Coding of the outcome measure. As discussed in greater detail in Chapter III, we coded our primary measure of repeat pregnancy accounting for a participant’s responses to survey questions from both the 6-month and 18-month follow-up surveys. If a woman reported having had a repeat pregnancy on the 6-month survey but did not respond to the repeat pregnancy question on the 18-month survey, we logically imputed her response for the 18- month survey as having had a repeat pregnancy. If a woman reported having had a repeat pregnancy on the 6-month survey but then contradicted this response on the 18-month survey by reporting not having had a repeat pregnancy, we considered her responses inconsistent and set the variable to missing. Because of the importance of this outcome for our analysis, we checked the sensitivity of our results to these coding decisions by creating three alternative versions of the same outcome. We describe the details of these alternative versions in Chapter III. • Specification of the regression model. For the main findings presented in Chapter IV of this report, we specified the regression models using ordinary least squares (OLS) regression. Each regression model included the following covariates: (1) a binary indicator for treatment status, (2) binary indicator variables for each of the strata created for random assignment, (3) two key demographic variables that are highly correlated with our key outcomes of interest (age and race), (4) a continuous variable measuring the number of months between when the baseline and follow-up surveys were administered to the participant, (5) the baseline measure of the outcome (if available), and (6) an additional set of baseline covariates that were empirically selected through a data-driven forward selection procedure. To test the sensitivity of our results to an alternative specification, we estimated comparable models using logistic regression for binary variables (for example, repeat pregnancy) and Poisson regression for count variables (for example, total number of pregnancies). We also tested the sensitivity of our results to alternative combinations of covariates—namely, (1) controlling only for random assignment strata and the baseline measure of the outcome (if available), (2) controlling only for random assignment strata with no additional covariates, and (3) dropping the control variable measuring the number of months between the baseline and follow-up survey administration dates. • Threshold for covariate selection procedure. As described in Chapter III, some of the baseline covariates were selected through a data-driven forward stepwise procedure developed previously in the literature (Social and Character Development Research Consortium 2010). At each step of the stepwise procedure, the variable with the smallest p- value below a preset threshold level was included in the model while variables already selected were evaluated to see if any could be removed; any variable with a p-value greater than the critical value of 0.32 and whose removal would least lower the adjusted R2 was removed. The critical p-value was set at 0.32 to correspond to a t-statistic of 1, which is the smallest value of the t-statistic at which the addition of a variable in the model increases the 63 FINAL IMPACTS OF THE T.O.P.P. PROGRAM adjusted R2 value. For a sensitivity test, we selected covariates using alternative values of 0.20 and 0.10 as the threshold p-value. • Method for estimating p-values. For the main findings presented in Chapter IV of this report, we adjusted the statistical significant tests (p-values) to account for two statistical issues. First, we adjusted the standard errors to account for the blocked random assignment design. As described in Chapter II, instead of randomly assigning each participant as an independent observation, we used permuted block random assignment to keep an even balance between the numbers of participants assigned to the treatment and control groups. To account for this design feature, we allowed for clustering of our standard errors at the randomization block level (Matts and Lachin 1988). Second, we also adjusted our p-values to correct for multiple hypothesis testing within domain, using a procedure outlined by Hothorn et al. (2008) and Schochet (2009).To examine the sensitivity of our results to these adjustments, we estimated comparable regression models under three alternative conditions: (1) no adjustment for clustering, (2) no adjustment for multiple hypothesis testing, and (2) no adjustment for clustering or multiple-hypothesis testing. The results of these analyses (Table D.1) showed that our findings are generally robust to alternative analytic decisions. For our primary measure of repeat pregnancy, the direction and statistical significance of the impact estimate is consistent between our main findings presented in Chapter IV and each of the sensitivity tests we conducted. The magnitude of the estimated impact is also consistent across tests, ranging from -16.6 percentage points to -18.4 percentage points. For the other pregnancy-related outcomes — unintended pregnancy, live birth, and number of pregnancies — the direction and magnitude of the impact estimate is consistent between our main findings presented in Chapter IV and each of the sensitivity tests. The statistical significance of the impact estimate is also consistent except in two cases: for the measure of number of pregnancies, the impact estimate loses statistical significance when limiting the covariates to (1) random assignment strata and baseline outcome measure or (2) random assignment strata alone. Our findings are also generally robust for the measures of contraceptive use and sexual risk behaviors (Table D.2). For the three measures of contraceptive use—use of a LARC method, use of any hormonal method or IUD, and use of any effective birth control method—the direction and magnitude of the impact estimates for all of the sensitivity tests is consistent with our main findings presented in Chapter IV. The statistical significance of the impact estimates is consistent for the measures of (1) use of a LARC method and (2) use of any hormonal method or IUD. For the outcome measuring use of any effective birth control method, the impact estimate gains statistical significance at the 5-percent level in seven of the nine sensitivity tests. The direction of the impact estimates for the sexual risk behavior outcomes—unprotected sex, sexual intercourse, and sexual intercourse without a condom—is also consistent with our main findings reported in Chapter IV. For the outcome of unprotected sex, the magnitude of the impact ranges from -8.2 percentage points to -11.8 percentage points, and the impact estimate is statistically significant across eight of the nine tests. For the outcome of any sexual intercourse, the impact remains statistically insignificant in each of the sensitivity tests. The impact estimate and statistical significance for the measure of sexual intercourse without a condom varies relatively more across tests, with the estimate ranging from -5.3 percentage points to -11.3 percentage points, and the p-values ranging from 0.01 to 0.76. 64 FINAL IMPACTS OF THE T.O.P.P. PROGRAM Table D.1. Sensitivity of impacts on pregnancy outcomes Repeat Unintended Live Number of pregnancy pregnancy birth pregnancies Impact p-value Impact p-value Impact p-value Impact p-value Main findingsa -17.4** <0.01 -17.9** <0.01 -9.7* 0.03 -0.3** <0.01 Alternative coding of outcome: Alternative 1 -16.6** <0.01 Alternative 2 -16.9** <0.01 Alternative 3 -17.5** <0.01 Specification of regression model: Logistic or Poisson model -17.8** <0.01 -19.0** <0.01 -10.1* 0.02 -0.3** <0.01 Controls only for strata and -17.5** <0.01 -17.2** <0.01 -10.3* 0.02 -0.2 0.09 baseline outcome Controls only for strata -17.8** <0.01 -17.7** <0.01 -10.3* 0.02 -0.2 0.19 No control for months between -17.6** <0.01 -18.4** <0.01 -10.0* 0.02 -0.3** <0.01 surveys Threshold for covariate selection: p-value = 0.2 -16.6** <0.01 -15.9** <0.01 -10.4* 0.01 -0.2* 0.02 p-value = 0.1 -18.4** <0.01 -17.1** <0.01 -10.4* 0.01 -0.2* 0.04 Method for estimating p-values: Ignore clustering -17.4** <0.01 -17.9** <0.01 -9.7* 0.01 -0.3** 0.01 Ignore multiple comparisons -17.4** <0.01 -17.9** <0.01 -9.7** 0.01 -0.3** <0.01 Ignore both -17.4** <0.01 -17.9** <0.01 -9.7** <0.01 -0.3** <0.01 Source: Surveys administered to study participants by the evaluation team. Note: For each outcome, the numbers in the impact columns represent the difference between regression- adjusted predicted values of outcomes at the 18-month follow-up survey between treatment and control groups. P-values are adjusted for clustering of standard errors at the randomization block level and for multiple outcomes measured within a single domain, unless otherwise stated. See Chapter III for a more detailed description of the analytic methods. a The main findings denote the impact estimates presented in Chapter IV of the report. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. 65 Table D.2. Sensitivity of impacts on contraceptive use and sexual risk behaviors Use of any Use of any Had sexual Use of LARC hormonal method effective birth Had unprotected Had sexual intercourse method or IUD control method sex intercourse without a condom Impact p-value Impact p-value Impact p-value Impact p-value Impact p-value Impact p-value Main findingsa 15.0** <0.01 11.5* 0.02 8.3 0.06 -11.5** <0.01 -3.2 >0.99 -10.3* 0.04 Specification of regression Logistic model 16.7** <0.01 12.9** <0.01 10.5* 0.03 -11.5** <0.01 -4.4 0.80 -11.3* 0.01 Controls only for strata 13.5** 0.01 11.7* 0.02 8.5 0.08 -8.6* 0.04 -2.0 >0.99 -5.3 0.73 and baseline outcome Controls only for strata 13.4** 0.01 12.4* 0.01 8.1 0.10 -8.2 0.05 -1.8 >0.99 -5.3 0.76 No control for months 15.5** <0.01 12.0* 0.02 8.7* 0.05 -11.8** <0.01 -3.2 >0.99 -10.2* 0.04 between surveys Threshold for covariate p-value = 0.2 14.8** <0.01 12.3* 0.01 9.1* 0.03 -9.1* 0.02 -1.4 >0.99 -6.9 0.34 p-value = 0.1 14.6** <0.01 12.6** 0.01 9.7* 0.02 -9.0* 0.02 -3.0 >0.99 -7.7 0.25 Method for estimating p- 66 values: Ignore clustering 15.0** <0.01 11.5* 0.01 8.3* 0.05 -11.5** <0.01 -3.2 0.96 -10.3* 0.01 Ignore multiple 15.0** <0.01 11.5** 0.01 8.3* 0.02 -11.5** <0.01 -3.2 0.36 -10.3* 0.01 Ignore both 15.0** <0.01 11.5** <0.01 8.3* 0.01 -11.5** <0.01 -3.2 0.32 -10.3* 0.05 Source: Surveys administered to study participants by the evaluation team. Note: For each outcome, the numbers in the impact columns represent the difference between regression-adjusted predicted values of outcomes at the 18- month follow-up survey between treatment and control groups. P-values are adjusted for clustering of standard errors at the randomization block level and for multiple outcomes measured within a single domain, unless otherwise stated. See Chapter III for a more detailed description of the analytic methods. a The main findings denote the impact estimates presented in Chapter IV of the report. *Significantly different from zero at the .05 level, two-tailed test. **Significantly different from zero at the .01 level, two-tailed test. IUD = intrauterine device. This page has been left blank for double-sided copying. http://www.hhs.gov/ash/oah/