Findings from the Replication of an Evidence-Based Teen Pregnancy Prevention Program Evaluation of the Children’s Aid Society (CAS)-Carrera Adolescent Pregnancy Prevention Program in Chicago, IL Final Impact Report for Children’s Home + Aid February 29, 2016 Prepared by Scott Herrling, MS 16 Main St. Accord, NY 12404 Recommended citation: Herrling, S. (2015). “Evaluation of the Children’s Aid Society (CAS)-Carrera Adolescent Pregnancy Prevention Program in Chicago, IL: Findings from the Replication of an Evidence-Based Teen Pregnancy Prevention Program.” Accord, NY: Philliber Research & Evaluation. Acknowledgements: We would like to thank the staff at Children’s Home + Aid and at the Children’s Aid Society-Carrera Program for their ongoing support and dedication to this evaluation. We also would like to thank the students who participated in the evaluation. Without their continued participation, the evaluation would not have been possible. A special thank you goes out to Edward Parker at Philliber Research & Evaluation for his assistance in STATA. This publication was prepared under Grant Number TP1AH000047 from the Office of Adolescent Health, U.S. Department of Health & Human Services (HHS). The views expressed in this report are those of the authors and do not necessarily represent the policies of HHS or the Office of Adolescent Health. i Table of Contents Introduction ......................................................................................................................1 A. Introduction and study overview .............................................................1 B. Primary research questions......................................................................3 C. Secondary research questions..................................................................4 I. Treatment and Control Programming ...................................................................4 A. Description of treatment program as intended ........................................4 B. Description of control condition .............................................................9 II. Study Design ........................................................................................................11 A. Sample recruitment .................................................................................11 B. Study design ............................................................................................12 C. Data collection.........................................................................................13 1. Impact evaluation ........................................................................13 2. Implementation evaluation ..........................................................15 D. Outcomes for impact analyses .................................................................16 E. Study sample ...........................................................................................17 F. Baseline equivalence ...............................................................................18 G. Methods ...................................................................................................19 1. Impact evaluation ........................................................................20 2. Implementation evaluation ..........................................................21 III. Study Findings ......................................................................................................21 A. Implementation study findings .................................................................22 1. Adherence to program model .......................................................22 2. Quality of implementation ............................................................24 3. Experiences of the control group ..................................................25 4. Context .........................................................................................26 B. Impact study findings ...............................................................................26 IV. Conclusion ............................................................................................................28 V. References ............................................................................................................31 Appendix A: CAS-Carrera Logic Model .........................................................................32 Appendix B: Study sample ...............................................................................................33 Appendix C: Implementation evaluation data collection .................................................34 Appendix D: Impact model specification .........................................................................35 Appendix E: Missing data ................................................................................................36 Appendix F: Implementation evaluation methods ...........................................................37 Appendix G: Sensitivity analyses .....................................................................................38 ii Evaluation of the Children’s Aid Society (CAS)-Carrera Adolescent Pregnancy Prevention Program in Chicago, IL I. Introduction A. Introduction and study overview Children’s Home + Aid (CH+A) is a leading child and family service agency in Illinois. Each year, it protects, educates, and counsels more than 40,000 children, youth and families across the state. In 2010, this agency was one of 16 grantees to receive funding from the Office of Adolescent Health (OAH) to replicate and rigorously evaluate evidence-based teen pregnancy prevention programs. This agency focused on the Englewood neighborhood of southwest Chicago, and chose to implement the Children’s Aid Society (CAS)-Carrera Adolescent Pregnancy Prevention Program. The CAS-Carrera program uses a holistic approach to empower youth, to help them develop personal goals and the desire for a productive future, to develop their sexual literacy, and to educate them about the consequences of sexual activity. The program works to develop a participant’s capacity and desire to avoid pregnancy. At the time, the incidence of teen births in Englewood was nearly three times as high as the rate for the City of Chicago as a whole. In Englewood in 2006, 12% of all births occurred to teens under age 18, while in the City of Chicago, births to teens under age 18 accounted for just 4.3% of all births. 1 This report describes the methods and results of the evaluation of the CAS-Carrera program as implemented by CH+A in Englewood, Chicago. This evaluation focuses on the impact of the CAS-Carrera after-school program on middle school youth, as well as the challenges of 1 Illinois Department of Public Health, “Illinois Teen Births by County, 2006-2007,” http://www.idph.state.il.us/health/teen/teen0607.htm. 1 implementing the program in this neighborhood. Funding for this program was made possible by a 2010 grant from the Office of Adolescent Health (Tier 1). Prior to this evaluation, the primary evidence of the CAS-Carrera program’s effectiveness was from a multi-site randomized controlled trial at six urban community centers in New York City during 1997-2000. This study involved 600 teens aged 13 to 15, who were not parenting or pregnant at baseline. These teens were randomly assigned to either the CAS-Carrera program or the control group which offered an alternative, less intensive program. Nearly all of the teens were African-American (56%) or Hispanic (42%). Among the 484 youth who were followed after three years (242 treatment and 242 control), results showed that compared to control youth, program youth were significantly less likely to have had sex and significantly less likely to have become pregnant. 2 These significant findings were largely a result of outcomes among females. The study found no statistically significant program impacts on pregnancy or sexual activity among males. It was suggested that better outcomes among males might be achieved if they were reached at a younger age. This study met the 2010 Department of Health and Human Services Teen Pregnancy Prevention Evidence Review criteria for a high study rating, indicating it was a well-implemented randomized controlled trial. 3 CH+A chose the CAS-Carrera program because it is a holistic youth development program delivered in an afterschool setting. Since CH+A already had the Community Schools partnerships in place, it made sense to choose an afterschool model, where both programs could be delivered on school grounds. Additionally, the CAS-Carrera program was aligned with 2 Philliber, S., Williams Kaye, J., Herrling, S., & West, E. (2002). Preventing pregnancy and improving health care access among teenagers: An evaluation of the Children’s Aid Society–Carrera Program. Perspectives on Sexual and Reproductive Health, 34(5), 244–251. 3 http://tppevidencereview.aspe.hhs.gov/document.aspx?rid=3&sid=35&mid=1 2 CH+A’s philosophy, because it is designed to meet all of the youth’s needs rather than just focusing on sexual activity. Lastly, the Chicago replication targeted a younger sample of youth since it was postulated in the previous study that better outcomes might be achieved among males if they were engaged at an earlier age. B. Primary research questions The current evaluation tested the extent to which the CAS-Carrera program, when replicated with fidelity, produced impacts on ever having sex and incidence of sexual intercourse without use of effective contraception during the three months prior to the final follow-up survey. The primary research questions were: (1) What is the impact of the CAS-Carrera after-school program relative to a community after-school program at the end of four years of programming (fall 2011 to spring 2015) on the incidence of students in grades 7 and up ever having sex? (2) What is the impact of the CAS-Carrera after-school program relative to a community after-school program at the end of four years of programming (fall 2011 to spring 2015) on the incidence of sexual intercourse without use of effective contraception in the past three months among students in grades 7 and up? The analyses of these questions will provide confirmatory evidence about the CAS-Carrera program on ever having sex and intercourse without use of effective contraception in the past three months for CH+A’s replication. It should be noted that these research questions are only asked of youth in grades 7 and up. 4 At baseline, the majority of study youth were in grades 4 through 6. Thus, baseline levels of these indicators do not exist for the majority of study 4 OAH stipulated that questions pertaining to sexual activity, contraception, and pregnancy behaviors could not be asked of youth in grades under 7. 3 participants. Impact analyses focus on youth who completed a final spring 2015 follow-up survey. C. Secondary research questions Secondary research questions consist of the two primary research questions analyzed by gender subgroups. In other words, the primary research questions are analyzed for males and females separately. The original randomized controlled trial of the CAS-Carrera program (1997- 2000) found significant effects for females but not males. The analyses of these questions will provide evidence about the CAS-Carrera program’s effectiveness for males and females separately. However, it should be noted that due to the relatively small sample sizes for the subgroup analysis, the ability to precisely detect smaller differences in outcomes between the treatment and control groups is diminished. II. Treatment and Control Programming Both the CAS-Carrera program (the treatment condition) and the control condition (the Community Schools program) were delivered by CH+A staff. CH+A was the lead partnering agency throughout this study. Other partners include Chicago Public Schools, which housed the treatment and control groups, ACCESS Community Health Network which provided medical services for program youth, and other local health and dental providers. A. Description of treatment program as intended The CAS-Carrera program is an adolescent pregnancy prevention program that uses a holistic approach to empower youth, help them develop personal goals and the desire for a productive future, develop their sexual literacy, and educate them about the consequences of sexual activity. The CAS-Carrera program works to develop a participant’s capacity and desire 4 to avoid pregnancy. The program model provides opportunities for young people to discover interests and develop talents. To accomplish its goals, the program uses seven components, including (1) an academic component, (2) comprehensive family life and sexuality education (FLSE), (3) a mental health component called Power Group, (4) an employment component called Job Club, (5) an arts component called Self-Expression, (6) a lifetime individual sports component, and (7) comprehensive medical and dental care. This after-school program typically occurs over a 45-week period, five days per week (Monday through Friday). Saturday programming when possible is also encouraged. This program was delivered in Englewood, Chicago but was adapted from the original after- school design. In Chicago Public Schools, the program was typically offered four days per week (Monday through Thursday) for 39 weeks per year. Job Club, mental health, and FLSE sessions were offered weekly, academic sessions were offered daily, and lifetime individual sports and self-expression sessions were offered weekly for 6 to 8 weeks per year. Each session lasted approximately 50 minutes. Descriptive information about each component follows: 1. Academic: Daily homework help, remediation, and enrichment with trained teachers and tutors driven by Individual Academic Plans for each participant. This component aligns with the school day by reinforcing what is taught daily. 2. Family Life and Sexuality Education (FLSE): Weekly comprehensive, medically accurate sexuality education sessions taught in an age-appropriate fashion by a trained professional. FLSE goals include: developing and increasing student sexual literacy; encouraging and supporting decisions to practice sexual abstinence; linking students to reproductive and 5 sexual health care and mental health services when necessary; helping students understand their attitudinal, behavioral, and normative beliefs regarding healthy sexual behaviors; teaching students the skills they need to develop and foster healthy relationships with family, friends, and romantic partners; and honing negotiation skills to ensure that young people will reach their goals of attaining higher education, fulfilling employment, and ultimately, a productive adulthood. The curriculum has been copyrighted by CAS. 3. Mental Health Services/Power Group: Weekly discussion groups led by licensed social workers; individual counseling, case management, and crisis intervention as needed. This component uses the K-5th grade Second Step training and 6th-8th grade Power Group curriculum provided by Children’s Aid Society. 4. Employment/Job Club: Weekly exposure to the “world of work,” including earning stipends for attending Job Club and Job Club events, opening bank accounts, exploring career choices, and participating in entrepreneurial projects. This component uses the Junior Achievement curriculum. 5 5. Self-Expression: Multiple exposures to music, dance, writing and drama workshops led by theater and art professionals, where children can discover talents and build self-esteem. There are protocols regarding component content but no formal curriculum has been created. Instead, this component focuses on social-emotional learning through interactive enrichment group activities. 6. Lifetime Individual Sports: Multiple exposures to a fitness program emphasizing non-team sports that build self-discipline and can be played throughout life, including golf, squash, 5 The Junior Achievement curriculum is designed to prepare young people for the real world by showing them how to generate wealth and effectively manage it, how to create jobs that make their communities more robust, and how to apply entrepreneurial thinking to the workplace. 6 swimming, and others. There are protocols regarding component content but no formal curriculum has been created. 7. Full Medical and Dental Care: Comprehensive medical and dental services provided in partnership with local providers. A mobile dental van visits youth on a quarterly basis, vision appointments are schedule twice per year, and medical appointments are scheduled as needed. The CAS-Carrera logic model (see Appendix A), hypothesizes that the implementation of all seven program components will prevent early pregnancy and births. Positive youth outcomes are realized when the program is fully implemented. The sum of all of the components working together contributes to positive outcomes related to the primary research questions. It is the expectation that FLSE knowledge scores/other positive behaviors will increase as a function of program dosage. There is no “set dose” of the CAS-Carrera program. Programming usually begins in the middle school grades and ends at high school graduation. However, the Chicago replication targeted a younger sample of youth since it was postulated in the previous randomized controlled trial that better outcomes might be achieved among males if they were engaged at an earlier age. The CAS-Carrera program offered two sessions per day, four days per week. Table II.2 depicts how much of each component was offered in the CH+A replication. Table II.1. CAS-Carrera Program Offerings per Academic Year Component Frequency # of sessions/hours of dosage Academic 4x per week @39 weeks 156 (range of 140-156 hrs) FLSE 1x per week @39 weeks 39 (range of 35-39 hrs) Mental Health 1x per week @39 weeks 39 (range of 35-39 hrs) Job Club 1x per week @39 weeks 39 (range of 35-39 hrs) Self-Expression 2 exposures per year @ 6 – 8 weeks @ 1 hr per week 12 (range of 12-16 hrs) Lifetime Individual Sports 2 exposures per year @ 6 – 8 weeks @ 1 hr per week 12 (range of 12-16 hrs) Medical and Dental Annual visits/screenings (medical, dental, and vision); . follow up as needed Source: Data provided by CAS-Carrera staff. 7 Saturday programming and “maintenance activities” offered during school and/or holiday breaks are considered to be best practices for ongoing youth engagement. Given the mandatory school closings and building use restrictions dictated by Chicago Public Schools, the CH+A CAS-Carrera programs have offered limited maintenance activities but not in a traditional scheduled Saturday school program format. Instead, CH+A organized occasional off-site activities such as movies, field trips, and other short events to ensure student connections when schools were unavailable for routine maintenance offerings. Youth participating in the CH+A CAS-Carrera replication had the potential to receive four years of programming. Initially, programming occurred after school in three middle schools 6 in Englewood, Chicago (generally between 3 and 5 PM). After the first year, one site was split into two sites. In year three, a fifth after-school site was created for treatment youth who had aged out of the study schools (9th graders in 2013 and 9th and 10th graders in 2014).These treatment youth were picked up from their homes via CH+A vans, and driven to a centrally located community center in Englewood daily (Monday through Friday, with occasional Saturday programming). Thus, all treatment youth had an opportunity to attend the CAS-Carrera program for four years. Nothing was created for control youth who aged out of the study sites (once a control youth reached the 9th grade, there was no intervention to attend). Thus, control youth who enrolled as 6th or 7th graders did not have an opportunity to attend the Community Schools program for four years. The CAS-Carrera program was delivered by trained program facilitators who received ongoing staff development as prescribed by the evidence-based model. Program facilitators received training from CAS-Carrera staff on many topics including component-specific 6 These middle schools served youth in grades K through 8. 8 information, the CAS philosophy and culture, performance measures, de-escalation, trauma, child abuse, suicide prevention, bullying, and other topics. Program staff also received training from Planned Parenthood of Illinois, as well as from OAH webinars and trainings. CAS-Carrera, in partnership with CH+A organized orientations and ongoing trainings to ensure that (new) staff were appropriately acclimated to the model’s core principles and philosophy and were sufficiently trained to be able to execute the model as prescribed by their roles. In addition, CAS-Carrera staff provided technical assistance for effective program delivery and overall problem solving. B. Description of control condition The control condition is the CH+A Community Schools program which provides supportive services to children. Community Schools programming was established by CH+A and existed in these study sites/schools prior to fall 2011 when this study began. The program promotes academic achievement and social-emotional development. It consists of a core instructional program (academic support), enrichment (arts and sports), and social-emotional services (mental health services). Specifically: 1. Academic support includes academic remediation, homework help and academic enrichment. 2. Enrichment activities include activities involving art, theater, leadership, community service, sports, dance, martial arts, music, creative writing and gardening. 3. Health services are provided to students and families through health workshops. Clinical counseling services, including group and individual therapy are also provided. 4. Special events for families and the community are held monthly and can include field trips, health/resource fairs and other family activities. 9 Students in the Community Schools after-school program met daily, for two hours, four times per week, for a minimum of 39 weeks. They received a minimum of 12 hours of educational programming, five hours of parent programming, and 12 hours of enrichment programming per month. Each session was one hour in length. There is no standard curriculum for the Community School program. Each after-school program is based on the needs of the school. Each program provides educational learning based on the school’s focus and alignment to the school day. Each program provides social-emotional learning and Common Core learning through enrichment programming (i.e., ballet, science clubs, drama, etc.). The specific activities vary per school. This program was delivered after school in the same three Chicago Public Schools in the Englewood neighborhood as the CAS-Carrera program. As mentioned previously, after the first year, one school was split into two sites. CH+A facilitators delivered the program. These program staff received training and funding from Chicago Public Schools-Community School Initiative, the Illinois Federation for Community Schools and the Illinois School Board of Education. It should be noted that the Community Schools components are very similar to the CAS- Carrera components. The academic component is nearly identical to the CAS-Carrera academic component, enrichment activities mirror the CAS-Carrera self-expression and lifetime individual sports components, health services provide counseling as does the CAS-Carrera mental health component, and CAS-Carrera staff hold special events for treatment youth which are similar to those for control youth. Because the control condition components were very similar to the treatment condition components, it can be argued that the main contrast being tested in this evaluation is the effect of the FLSE, Job Club, and medical care components. 10 III. Study Design An individual-level randomized controlled trial design was used to estimate the impact of the CAS-Carrera program on reducing sexual risk-taking behaviors among youth in the Englewood neighborhood of Chicago. It is hoped that in a well implemented randomized controlled trial there will be no differences in baseline characteristics (demographic or outcome variables) prior to program implementation. Any difference in outcomes between the treatment and control groups can thus be causally attributed to the intervention. A. Sample recruitment In collaboration with principals at three target schools in the Englewood neighborhood of Chicago, during the spring of 2011 letters were sent to parents of 547 children who, in fall 2011, were expected to be in grades 4 through 7 informing them of the CAS-Carrera program (treatment group) and the Community Schools program (control condition). During the summer of 2011, CH+A program staff contacted parents identified through the principals at the target schools. Parents received detailed information about the two after-school programs (treatment and control) and the evaluation process. There were no inclusion/exclusion criteria for sample enrollment, other than being in grades 4 through 7 in fall 2011. Parents who agreed to their children’s participation were offered a chance to enter a lottery. Winners received a gift card valued between $25 - $50. In August 2011, 338 youth had submitted active parental consent for participation in the study. The other 209 youth did not submit consent forms. After obtaining active parental consent in August 2011, all 338 youth were randomized into either the treatment group (n=166) or the control group (n=172). It should be noted that the Community Schools program began at one of the study schools in 2004 and in the other two study schools in 2008. Thus, it is possible that 11 some youth in both the treatment group and the control group may have attended the Community Schools program prior to randomization. B. Study design This study was an individual-level randomized controlled trial. Randomization occurred over a two-week period in August 2011 after parental consent was obtained. Randomization to either the treatment group or the control group occurred at the same time and separately within each school/site. Randomization at each school/site was completed in a day or two depending on student attendance. Students self-randomized by clicking a computer key on a Philliber laptop computer. An Excel program then placed them randomly into either the treatment group or control group. Students were notified immediately of group assignment. Philliber staff supervised the entire process at each school. The only stratification or blocking that occurred was based on same-household living situations. If more than one student at a school was living in the same household, then all those in that household were assigned to the same group (either treatment or control) due to logistical concerns and the potential for contamination. The number of same-household situations did not differ significantly between the treatment and control groups. In the treatment group, 72 participants comprised 34 same-household situations and in the control group, 81 participants comprised 36 same-household situations. No stratification or blocking occurred by school/site because there were only three sites at randomization. The probability of being assigned to either the treatment group or the control group was .5. However, a perfect 50/50 split between groups was not achieved. At the end of randomization, 12 49% of the youth were assigned to the treatment group and 51% were assigned to the control group. 7 Active parental consent was obtained annually from the parents or guardians of all eligible youth each summer over the four years of this study. The same materials, approaches, and procedures were used each year. All consent forms included consent for both program participation and participation in the evaluation. In addition to parental consent, student assent was also collected annually. Parental consent and student assent forms state that participation in the evaluation research is voluntary and participants can change their minds at any time. During the administration of the surveys, subjects are instructed that they can refuse to answer any questions that make them feel uncomfortable. C. Data collection Impact evaluation data were collected via student surveys at eight points over this four-year study: baseline in fall 2011 and follow-up every six months thereafter. Data on program fidelity were gathered on an ongoing basis throughout the study period to document program implementation and provide context for the impact findings. 1. Impact evaluation Philliber evaluation staff collected all youth survey data. Baseline data were collected from 99.4% of those who were randomly assigned to either the treatment or control groups. Final follow-up data were collected from 81.0% of the study youth (83.1% of treatment youth and 79.1% of control youth). Table B.1 in Appendix B outlines the response rates at all data collection points. The majority of survey data were collected in one-week scheduled data collections. These 7 Randomization results for each site were as follows: site 1 (49% treatment, 51% control); site 2 (52% treatment, 48% control; site 3(47% treatment, 53% control). Thus, overall: 49% treatment, 51% control. 13 scheduled data collections took place during normal after-school program hours. Students who were not present during the scheduled one-week data collections were surveyed through the use of local Philliber-trained trackers who lived in the Englewood neighborhood and had a history of working with youth and youth programs. These trackers visited the homes of the absent students and collected needed surveys. There were no differences in data collection methods between treatment and control groups. All survey data were gathered via paper and pencil – whether it was during scheduled data collections or by local trackers. All survey items were read to participants – either in a group classroom setting during scheduled data collections or one-on-one by local trackers. The baseline data collection took 5 weeks to complete due to 5% of the participants who were randomized and then left the schools. Each follow-up data collection took approximately 3 months due to the transient nature of the student population. 8 This data collection interval was consistent between treatment and control conditions. Surveys included contact data: phone numbers, addresses, email addresses, Facebook addresses, and contact data from other family/close friends. This enabled Philliber to document where these students lived and helped to ensure high follow-up rates over the duration of this study. Many of the survey items were required by OAH such as demographic characteristics (age, grade, gender, ethnicity, and race) and outcome variables related to sexual activity, contraception, and pregnancy. There were also many survey items that were not required by OAH that are not included in this report. 9 8 As time went by, a growing number of youth transferred from the study schools or moved locally, thereby increasing the amount of time it took to locate youth for survey completion. 9 Non-OAH survey items measured socio-economic status, health care, educational aspirations and other school- related items, drug use, violence and delinquency, and family life and sexuality education (FLSE) knowledge. The FLSE items were developed by Dr. Carrera and senior FLSE staff to track the dimensions of the FLSE curriculum offered in the program. These items included 12 true/false items measuring knowledge about puberty, peer pressure, male and female anatomy, gender roles, relationships, pregnancy, abstinence, and menstruation. 14 There were two different versions of the survey instrument. Students in grades 7 or higher completed the survey as described above. Students below grade 7 completed a similar survey without the questions pertaining to sexual activity, contraception, birth control, and pregnancy. Participation in the follow-up survey process was recognized with a $20 cash incentive which began at first follow-up in spring 2012. At final follow-up (spring 2015), the cash incentive was raised to $50 to ensure a strong final follow-up rate. 2. Implementation evaluation Fidelity to the CAS-Carrera model was assessed through measures of adherence and quality. To assess adherence, the program staff collected and submitted daily attendance data consisting of the number of sessions scheduled and attended for each of the six non-service components (academic, FLSE, mental health, Job Club, Self-Expression and lifetime individual sports). Staff also submitted general daily attendance data (whether a child was there for any part of a given day but did not attend component activities). 10 These attendance data were sent to Philliber on a monthly basis. 11 Records of the seventh component (medical/dental care) were kept on-site by program staff. Attendance data for the control group consisted of general daily attendance and was also submitted to Philliber on a monthly basis from CH+A staff. The quality of program implementation was assessed by gathering observation data from a convenience sample of 10% of the treatment group sessions. 12 These data were gathered by a CAS-Carrera Fidelity Manager who completed observation forms mandated by OAH to ensure 10 Youth may have stopped by during snack time but did not stay for component activities. 11 Attendance data were collected for 44 months, from November 2011 through June 2015 (the duration of the study). 12 This is a limitation, since the sessions viewed may not be representative of the entire number of sessions received. 15 program fidelity. These data were sent to CH+A staff for review on a monthly basis and then forwarded to Philliber for entry and analyses. Additionally, Philliber staff shadowed the Fidelity Manager one week per year in years one, two, and three to observe sessions jointly. These Philliber observations were paired with those of the Fidelity Manager to ensure inter-rater reliability. There were no significant differences between observers in scores for the 20 sessions viewed jointly by Philliber staff and the CAS- Carrera Fidelity Manager. Philliber staff collected additional data regarding other teen pregnancy prevention programming which may have been available to both the treatment group and control group through follow-up survey questions starting in fall 2013. 13 Additionally, monthly phone meetings with evaluation staff, CH+A staff, and CAS-Carrera staff provided information on any external events and/or challenges that might have affected implementation. Table C.1 in Appendix C provides detailed descriptions of all implementation data collected. D. Outcomes for impact analyses Two primary research questions are answered with two single-item dichotomous measures (see Table III.1) from the baseline and follow-up surveys completed by youth in grades 7+: • “Have you ever had sexual intercourse?” and • “In the past 3 months, have you had sexual intercourse without you or your partner using any of these methods of birth control? [Methods include: condoms, birth control pills, the shot (Depo Provera), the patch, the ring (NuvaRing), IUD (Mirena or Paragard), implants].” 13 In the second year of this study, it was determined that the Teen Outreach Program (TOP) was being offered in some schools in the Chicago area and that it was possible that youth who had transferred to other non-study schools could have been exposed to this curriculum. TOP is a curriculum-based teen pregnancy prevention program offered to high school youth in Chicago. 16 The measure of ever having had sex captures the effect of offering the CAS-Carrera program on the full sample of youth, whether they were sexually active at baseline or not. In other words, we can compare the rate of ever having had sex at final follow-up (spring 2015) between treatment and control youth, regardless of whether they completed this survey question at baseline or not. 14 This same logic applies to the question about sexual intercourse without use of effective contraception. The secondary research questions look at these same outcomes among subsamples of male and female youth separately. Both of the primary outcomes are dichotomous measures and were constructed as dummy variables where youth who responded “yes” to the question were coded as “1” and those who responded “no” were coded as “0”. Table III.1. Behavioral outcomes used for primary and secondary impact analyses Timing of measure Outcome name Description of outcome relative to program Ever had sexual “Have you ever had sexual intercourse?” Asked on all surveys of intercourse This is a dichotomous measure where yes responses are youth in grades 7+ coded as “1” and no responses are coded as “0”. Use of “In the past 3 months, have you had sexual intercourse Asked on all surveys of contraception in the without you or your partner using any of these methods of birth youth in grades 7+ past 3 months control? [methods include: condoms, birth control pills, the shot (Depo Provera), the patch, the ring (NuvaRing), IUD (Mirena or Paragard), implants]“ This is a dichotomous measure where yes responses are coded as “1” and no responses are coded as “0”. E. Study sample Table B.1 in Appendix B depicts the flow of sample members from the beginning of the study throughout the final follow-up survey that were used to address the primary and secondary research questions. As described in section III.A, 338 youth from three schools were randomly assigned to either the treatment condition (n=166) or the control condition (n=172). All 338 youth completed a parental consent form in summer of 2011 prior to their baseline survey. 14 Youth in grades 4 through 6 were not asked this question at baseline. Only youth who were in grade 7+ were asked this question. Thus, a study participant may not have been first asked this question until far after baseline, depending on grade level at baseline. 17 The baseline survey occurred in fall 2011 and was completed by 99.4% of the study youth (treatment group n=165 and control group n=171). Students completed follow-up surveys every spring and fall after baseline. In spring 2015, 81% of these youth (n=274) completed a final follow-up survey (treatment group n=138 and control group n=136). These youth comprise the final analytic sample. Approximately nine out of ten students in the final analytic sample were Black/African- American. Just over half were female and the average age was 11.1 years. Nearly all (98%) spoke English at home. Among those who were in grades seven or higher at baseline (n=55), 16% reported ever having had sexual intercourse. None reported having had sexual intercourse without use of an effective method of contraception. The reader is reminded that only youth in grades seven or higher were asked questions about sexual activity. Fifty-four youth were in grade seven at baseline and one reported being in the 8th grade at baseline. F. Baseline equivalence Baseline equivalence tests were conducted on the final analytic sample to assess whether attrition affected the comparability of the treatment and control groups. 15 Since baseline levels of the two primary outcomes of interest do not exist for 80% of the final analytic sample, tests were conducted for the following demographic variables only: age, gender, and race/ethnicity. Dummy variables for school (the three sites) and same-household living situations at baseline were also used. Table III.2 summarizes the key baseline measures for the final analytic sample, which consists of youth who responded to the survey questions on which the primary and secondary outcome measures are based at final follow-up (spring 2015). There were no significant 15 The attrition rate met the Teen Pregnancy Prevention evidence review threshold for low attrition. 18 differences (p < .05) between the treatment and control groups on the key baseline demographic characteristics. A regression approach that accounted for the random assignment design was used in calculating these percentages; these are adjusted means. Table III.2. Summary statistics of key baseline measures for youth completing final spring 2015 follow-up survey Intervention Intervention versus versus Intervention mean Control mean or control control p- or % (standard % (standard mean value of Baseline measure deviation) deviation) difference difference Age (years) 11.02 (1.24) 11.23 (1.27) -0.21 0.203 Gender (female) 58.9 50.2 8.7 0.183 Race/ethnicity: Black 92.2 88.1 4.1 0.021 Race/ethnicity: American Indian or 9.3 5.0 4.3 0.029 Alaska Native Race/ethnicity: Hispanic 6.3 4.1 2.2 0.444 Race/ethnicity: White 2.6 2.6 0 0.983 Race/ethnicity: Asian .9 1.5 -.6 0.521 Race/ethnicity: Other 3.8 1.9 1.9 0.231 Ever had sex A 14.8 18.8 -4.0 0.698 Use of contraception in the past 3 0 0 0 -- months A Sample size 138 136 . . A These numbers are based on the subsample of 55 7th graders (23 treatment and 32 control) at baseline who were asked this question. The majority of youth were in grades 4 through 6 at baseline and were not asked this question. Source: Baseline survey data. G. Methods To estimate the impact of the CAS-Carrera program, relative to the control group on ever having had sex and sexual intercourse without use of effective contraception, an intent-to-treat design was used. An intent-to-treat design estimates the program’s impact on all possible youth who were enrolled in the treatment group, regardless of the level of participation. In other words, any youth in the treatment group who completed a final follow-up survey was included in the analysis, regardless of whether they were actively attending CAS-Carrera programming. Additionally, any youth in the control group who completed a final follow-up survey was included in the analysis, regardless of whether they were actively attending the Community Schools program. 19 1. Impact evaluation The impact estimate is the regression-adjusted difference between the average outcomes among treatment youth and control youth. Since there are two primary research questions, impact estimates with p-values less than 0.025 (two-tailed test) are considered statistically significant and provide evidence that there are likely true differences between the groups as a result of the CAS-Carrera program. Linear regression analysis was used to estimate program impacts regarding the two primary research questions. The dependent variables consisted of the values of ever having had sex and sexual intercourse without use of effective contraception in the past three months at final follow- up. The independent demographic variables were baseline values of age, gender, and race/ethnicity. These regressions also included two dummy variables for school. Since there were three schools which took part in the random assignment, the school with the largest number of youth served as the reference category and the two remaining schools were included in the equations as separate (0,1) variables. Participants in same-household situations were weighted in the analysis (1/n, where n = the number of ‘siblings’ in the analytic sample). Regressions also included a variable to control for same-household situations at baseline. A detailed description of the regression model used can be found in Appendix D. All of the above analyses were conducted for subsamples of male and female youth to address the study’s secondary research questions. Where missing data occurred, values were imputed by consulting all previous follow-up surveys. For example, if a youth was Hispanic on her first seven surveys, she was coded Hispanic on her final survey. Listwise deletion was the benchmark approach used to handle 20 missing data. However, pairwise deletion was also used as part of a sensitivity analysis. See Appendix E for further details regarding missing data. 2. Implementation evaluation Adherence to the CAS-Carrera program model was largely measured by attendance data. The total number of sessions attended was calculated from these data. However, specific component content was not captured. To determine who delivered material to youth, a snapshot count of program staff at the end of the study was used to determine average program staff size. The quality of staff-participant interactions and youth engagement with the program were measured using program observation forms. Sessions deemed of high quality were scored with a 4 or 5 on five-point scales on items dealing with clear expectations, time management, perceptions of understanding, participation, and ratings of facilitators on knowledge, enthusiasm, confidence, rapport, and the ability to answer questions. 16 There was also an overall rating of the program session. The percentage of 4s and 5s reported are presented. Experiences of the control group, other teen pregnancy prevention programming that may have been available to study participants, external events, and unplanned adaptations are presented as frequencies or counts. Table F.1 in Appendix F provides a list of implementation evaluation methods. IV. Study Findings The two goals of the evaluation were to (1) determine if the CAS-Carrera program had favorable impacts on students’ levels of ever having had sex and sexual intercourse without use 16 These scales were not uniform. For example, the time management scale ranged from 1 ‘not on time’ to 5 ‘well on time’, while the perceptions of understanding scale ranged from 1 ‘little understanding’ to 5 ‘good understanding’. However, in each case, the most favorable ratings had values of 4 or 5. 21 of effective contraception in the past three months, and to (2) understand how the CAS-Carrera program was implemented to provide context for the impact findings. The following section presents the results of the implementation study, followed by findings from the impact analysis. A. Implementation study findings The implementation study focused on how much of the CAS-Carrera program was received, the extent to which the program was delivered with fidelity, the quality of the sessions delivered, and any circumstances that may have affected program implementation. While there were some elements of programming in Chicago that adhered to the CAS-Carrera model, the demonstration overall was not delivered with fidelity, due in large part to the instability of the of the Chicago Public School (CPS) system. As it relates to the delivery of programming, observation data gathered by a CAS-Carrera Fidelity Manager suggest favorable ratings. However, these data represent a convenience sample of less than 10% of all sessions delivered, and thus they may not be representative of all sessions. Additionally, attendance data show that only 12% of program youth attended at least 75% of scheduled sessions (the initial goal). 1. Adherence to program model Table IV.1 depicts how much of each component was offered and received in this replication (except medical and dental visits which occur as needed). Generally, program sessions were offered four days per week and were 50 minutes in length. In some cases there were longer sessions, depending on session content (field trips, events, etc.). The original goal was for treatment youth to attend at least 75% of scheduled sessions. However, the overall percentage of scheduled sessions attended was 42%, far lower than original expectations. As expected, the academic component was offered most often, while Job Club, Mental Health, and FLSE were offered once per week. Sports and self-expression were offered slightly less often. 22 Table IV.1. Attendance data among treatment youth in analytic sample Component Avg. # of sessions scheduled Avg. # of sessions attended (%) Academic 446.1 192.0 (43%) FLSE 144.6 57.7 (40%) Mental Health 144.3 60.2 (42%) Job Club 128.1 54.0 (42%) Self-Expression 113.7 44.8 (39%) Lifetime Individual Sports 82.3 31.2 (38%) Total (n=138) 1,059.2 443.1 (42%) Source: Attendance data collected throughout study. Many treatment youth received very little programming. Three youth never attended a single program session, 29% attended between 1% and 25% of the scheduled sessions, and just 12% attended at least 75% of the scheduled sessions (Table IV.2). Table IV.2. Scheduled sessions attended among treatment youth Scheduled sessions attended N (%) 0% 3 (2%) 1% to 25% 40 (29%) 26% to 50% 41 (30%) 51% to 75% 37 (27%) 76% to 100% 17 (12%) Source: Attendance data collected throughout study. It is important to note that the CAS-Carrera model in Englewood was not fully delivered in a fashion consistent with a typical CAS-Carrera after-school implementation. The CAS-Carrera program is typically a 45-week model. Programming in Englewood averaged 37 to 43 weeks per year due to program start delays, schools cancelling programming due to testing, and building closures. The CAS-Carrera after-school model typically is delivered five days per week (not including Saturdays). In Englewood, the schools did not offer Friday programming. Additionally, virtually no Saturday programming was offered, and very little summer programming was offered, which is not the case in a typical after-school CAS-Carrera program. Also, required mandatory participation of some CAS-Carrera students in school-led Supplemental Education Services also impacted program participation and subsequent program dosage. Student mobility increased each study year and also contributed to lower attendance rates. For example, in the final year (2014-15), 41% of the treatment youth no longer attended the study schools due to transfers/moves/or other reasons. 23 The sites experienced significant staff turnover. Just one program supervisor remained throughout this four-year study. 17 The staff in this position changed at the other sites, and in 2013-14, this position changed twice at one site. There were numerous other staff changes at the curriculum-delivery level as well. In spring 2015, there were 32 CH+A staff involved in CAS- Carrera program delivery. However, overall 82 had been hired since the program began implementation. Such high levels of turnover may have a deleterious effect on programming and lead to a necessity to build new relationships with school staff and, more importantly, with the program participants and their parents. Staff turnover in school administration at two of the schools also caused programmatic delays. 2. Quality of implementation Table IV.3 shows the number and percent of observations with scores of 4 or 5 (the most favorable responses). At least 80% of the observations indicated the program youth understood the presentation materials and/or actively participated in group discussions and activities. Program implementers received favorable scores about 80% of the time as well. However, the overall program session quality ratings were slightly less favorable (74%). 17 The program supervisor is responsible for day-to-day operations of the program, including guidance, training, and supervision of program staff. This person insures the program is delivered with fidelity. 24 Table IV.3. Observation data on program quality (n=726) % of observations rated favorably In general, how clear were the program implementer’s explanations of activities? 78% To what extent did the implementer keep track of time during the session and activities? 87% To what extent did the presentation of materials seem rushed or hurried? 93% To what extent did the participants appear to understand the material? 80% How actively did the group members participate in discussions and activities? 82% On the following scale, rate the implementer on the following qualities: . Knowledge of the program 76% Level of enthusiasm 87% Poise and confidence 83% Rapport and communication with participants 86% Effectively addressed questions/concerns 89% Rate the overall quality of the program session: 74% Source: Observation data collected throughout study by CAS-Carrera Fidelity Manager. Note: These percentages reflect the percentage of ratings receiving a 4 or 5. For example, of the 726 ratings for the overall quality of the program session, 74% of the ratings were 4s or 5s (26% of the ratings were 1s, 2s, or 3s). 3. Experiences of the control group Thirty-eight percent (n=52) of the control group youth said they had received sexuality education and 11% (n=15) said they had been enrolled in the Teen Outreach Program (TOP). Additionally, the strength of the control condition was stronger than had initially been anticipated. Control youth received team sports at all sites, financial literacy at one site, and academic assistance at all sites including daily homework help, tutoring, and other academic enrichment. Chicago Public Schools requires dental exams for all students in the 6th grade each school year and vision exams for 8th graders each school year. In 2013, the vision program was expanded to provide glasses free of charge to any student who previously failed a vision exam but still did not have glasses. Health, wellness, and nutrition education was provided to youth at two of the sites, and one site provided education on anatomy, sexuality, and healthy relationships for two months in 2012-13. Additionally, all sites provided some form of self-expression programming in the form of ballet, art, music, drama, and/or poetry. CAS-Carrera FLSE program staff provided support to all youth and parents (treatment and control) in the form of wellness events (vaccines for students/parents, diabetes/hypertension screenings, and flu shots for parents/staff) and coordinated dental screenings with dental 25 providers. Workshops were also provided on suicide prevention and financial literacy to all parents (treatment and control). In the final study year (2014-15), CH+A replaced their Community Schools program (the control condition) with the CAS-Carrera program (the treatment condition). Thus, in the last year of the study, 13 control youth (those who were still attending the study sites and had not aged out) received all CAS-Carrera services directly from program staff. 4. Context There were two unplanned adaptations during this evaluation study. In the beginning of year two (fall 2012) a teacher strike in Chicago Public Schools delayed the start of programming for two full months. In summer 2013, two of the study schools closed and moved about one mile away due to district consolidation. When programming resumed in fall 2013, 14% of the youth chose not to attend these new locations. Due to safety concerns (gang violence) and liability issues, if a student transfers from a study school to a neighboring school, he/she cannot return to the study school for any reason without explicit permission from the principal. Study school principals refused to assume liability for visiting students. Thus any student (treatment or control) who transferred out of a study school could not continue to receive programming. B. Impact study findings Table IV.4 shows the estimated effect of the CAS-Carrera program on the two primary outcome measures. There is no evidence that the CAS-Carrera program caused changes in the likelihood of ever having had sex or sexual intercourse without use of effective contraception during the three months prior to final follow-up. At final follow-up, 32% of the treatment youth reported ever having had sex compared to 26% of the control youth. This difference was not statistically significant (p = 0.232). Additionally, 4% of the treatment youth reported sexual 26 intercourse without effective use of contraception during the past three months compared to 2% of the control youth. Again, this difference was not statistically significant (p = 0.389). Table IV.4. Post-intervention estimated effects using data from spring 2015 follow-up survey to address the primary research questions Intervention compared to control mean difference (p-value Outcome measure Intervention % Control % of difference) Ever had sex 32.1 25.7 6.4 (0.232) Had sexual intercourse without effective 3.8 2.0 1.8 (0.389) use of contraception in past 3 months Sample Size 138 136 Source: Spring 2015 final follow-up data. Notes: See Table III.1 for a more detailed description of each measure. Table IV.5 shows the estimated effect of the CAS-Carrera program on the primary outcome measures for males and females separately. There is no evidence that the CAS-Carrera program caused changes in either outcome among males or females. The reader is reminded that these analyses are based on very small sample sizes and should be viewed with caution. Table IV.5. Post-intervention estimated effects using data from spring 2015 follow-up survey to address the secondary research questions Intervention compared with control Mean difference (p-value Outcome measure Intervention % Control % of difference) Ever had sex (among males) 41.4 38.9 2.5 (0.868) Ever had sex (among females) 20.3 13.8 6.5 (0.557) Had sexual intercourse without effective use of contraception in past 3 months 6.5 8.8 -2.3 (0.789) (among males) Had sexual intercourse without effective use of contraception in past 3 months 5.8 1.4 4.4 (0.414) (among females) Sample Size males n=57, males n=67, females n=81 females n=69 Source: Spring 2015 final follow-up data. Notes: See Table III.2 for a more detailed description of each measure. Additional analyses were conducted to see if results were sensitive to the benchmark analysis chosen here (listwise deletion). Analyses were run using pairwise deletion as well. Included in all equations were two school dummy variables as well as an independent variable 27 representing same-household situations to see if these had any effect on outcomes. Appendix G provides more detail on these additional analyses. V. Conclusion This study is the first rigorous evaluation of the CAS-Carrera after-school program since the original randomized controlled trial found favorable impacts on teen pregnancy among adolescents about 15 years ago (Philliber et al., 2002). Since that time, the program has expanded to numerous locations nationwide: Georgia, West Virginia, Michigan, Delaware, New York, New Jersey, Florida, Maryland, Ohio, New Mexico, and Washington DC. Many of these expansions were OAH funded, but none included randomized controlled trials. This study took place in the Englewood neighborhood of Chicago from 2011 to 2015. The program was delivered with many adaptations due to unplanned external problems. Based on a sample of 274 youth, we found no evidence that the CAS-Carrera program impacted youth rates of ever having sex or having sexual intercourse without use of effective contraception after four years of programming. These outcomes are inconsistent with those of the original study where treatment youth were deemed significantly less likely than controls to have had sexual intercourse and to have ever been pregnant. These original findings were largely due to results found among females. It should be noted that these two studies were markedly different. The first study occurred from 1997-2000 and involved 484 youth (242 treatment and 242 control) tracked over three years in New York City. These NYC youth were nearly two years older, 56% were African- American and 42% were Hispanic. The vast majority (87%) of Chicago study youth were African-American. Additionally, the number of youth tracked in Chicago was much lower than 28 in the original study (274 in Chicago vs. 484 in NYC). Thus, the smaller sample size in Chicago makes it more difficult to detect significant differences. The NYC study occurred in six youth-serving agencies such as Boys & Girls Clubs, where the control condition was largely either another less intensive youth program offered at the site or business as usual. Many control NYC youth chose not to attend any agency offerings. In Chicago, the control condition was comprised of a program providing over half of the CAS- Carrera components, with many control youth receiving services directly from program staff. In fact, in the final year, the control condition was completely replaced by the CAS-Carrera program. Where NYC youth attended school had little bearing on CAS-Carrera program participation. In Chicago, this became paramount because the program was housed on public school grounds. If a youth transferred to a non-study school he/she was not permitted to attend either treatment or control programming due to safety and liability concerns by school administration. The Englewood neighborhood was a very challenging neighborhood in which to work. Over the course of this study, a growing number of youth transferred out of the study schools. In the final year, over two-fifths of the treatment youth were not actively attending programming. In the NYC study, 79% of treatment youth were still involved at some level in their CAS-Carrera program after three years. Forty-eight percent were actively involved and 31% had contact with program staff outside of the weekday, after-school schedule. In Chicago, those who left the treatment program had little to no contact with CAS-Carrera program staff. In the NYC study, the treatment group was offered programming six days per week, including Saturday and summer programming. In Chicago, the program was offered four days per week, with no Friday programming, and little to no Saturday and/or summer programming. 29 In Chicago, it would appear that more services were offered to control youth than in the NYC study, given the workshops that were made available to all parents and youth. Limitations of this study include low external validity and contamination of some control group members within the school study sites. Since the school sites were not a representative sample of all schools in Chicago, these results cannot be generalized beyond the specific sites in Englewood. Secondly, since the control condition was replaced with the CAS-Carrera program by CH+A in the final year of this study, a small number of control youth (those who had not aged out of the study sites) received all program intervention in the last year. Given all of these factors, the failure of the impact analyses to detect significant differences between treatment and control youth may not be surprising. 30 VI. References Philliber, S., Williams Kaye, J., Herrling, S., & West, E. (2002). Preventing pregnancy and improving health care access among teenagers: An evaluation of the Children’s Aid Society– Carrera Program. Perspectives on Sexual and Reproductive Health, 34(5), 244–251. Illinois Department of Public Health, “Illinois Teen Births by County, 2006-2007,” http://www.idph.state.il.us/health/teen/teen0607.htm. U.S. Department of Health and Human Services. Teen Pregnancy Prevention Evidence Review. http://tppevidencereview.aspe.hhs.gov/document.aspx?rid=3&sid=35&mid=1 31 Appendix A: CAS-Carrera Logic Model CAS-Carrera Program Logic Model Activities Program Service Shorter-term Objectives Longer-term Objectives Objectives CAS-Carrera staff Relative to Short-term Outcomes Relative to comparison are trained in comparison youth, youth, program youth program theory, program youth will: Relative to comparison youth, program will have: philosophy, logic • Have bank youth will: • Lower rates of teen model, and full accounts. • Gain greater knowledge about pregnancy. curricula for each • Have improved sexuality and reproduction. • Lower rates of teen component and receipt of • Gain greater financial literacy. births. philosophy medical care including annual Mid-term Outcomes In future years beyond Youth will receive all comprehensive the current research, core components of medical, dental, Relative to comparison youth, program program youth will the program reproductive, youth will: have: including: and specialty • Be more likely to be sexually • Higher rates of • Academic care abstinent. high school assistance psychosocial • If sexually active, be more likely to graduation. • Family life and assessments, use protection from pregnancy and • Higher rates of sexuality education Hep B and HPV sexually transmitted infections. college admission. • Mental health vaccinations, • Improve their academic • Job Club and care from performance. • Self-expression sites other than • Engage in fewer risk behaviors • Lifetime individual a hospital such as fighting, drug or alcohol sports emergency use, or arrests. • Medical and dental room. care 32 Appendix B: Study sample Table B.1. Youth sample sizes by intervention status across all data collection points Total Interventio Total Interventio Compariso Number of Time sample n sample Comparison response n response n response youth Period size size sample size rate % rate % rate % Assigned to condition August 2011 338 166 172 N/A NA N/A Contributed Sept. 2011 baseline survey – Oct. 2011 336 165 171 99.4 99.4 99.4 Contributed 1st Apr. 2012 – follow-up survey Jul. 2012 319 159 160 94.3 95.8 93.0 Contributed 2nd Nov. 2012 – follow-up survey Jan. 2013 300 153 147 88.7 92.2 85.4 Contributed 3rd Apr. 2013 – follow-up survey Jul. 2013 303 154 149 89.6 92.8 86.6 Contributed 4th Oct. 2013 – follow-up survey Jan. 2014 284 142 142 84.0 85.5 82.6 Contributed 5th Apr. 2014 – follow-up survey Jul. 2014 275 139 136 81.3 83.7 79.1 Contributed 6th Oct. 2014 – follow-up survey Jan. 2015 255 127 128 75.4 76.5 74.4 Contributed final 7th follow-up Apr. 2015 – survey Jul. 2015 274 138 136 81.0 83.1 79.1 33 Appendix C: Implementation evaluation data collection Table C.1. Data used to address implementation research questions Types of data used to assess whether the element of the intervention was Frequency/sampling of Party responsible for Implementation element implemented as intended data collection data collection Adherence: How often Daily paper attendance Attendance records Program staff collect data were sessions offered? records. collected daily. and send to evaluation How many were offered? staff monthly. All sessions are approximately 50 minutes. Adherence: What and how Daily paper attendance Attendance is captured on Program staff collect data much was received? records. daily records. and send to evaluation staff monthly. Adherence: What content Type of sessions recorded Session attendance is Program staff collect data was delivered to youth? on daily paper attendance captured on daily records. and send to evaluation records. Specific session staff monthly. content is not recorded. Adherence: Who delivered List of staff members hired Data on all staff members Program staff. material to youth? and trained to implement are available to program program. Background staff. qualifications of staff members from staff applications. Quality: Quality of staff- Observations of interaction Convenience sample of CAS-Carrera Fidelity participant interactions quality using paper OAH 10% of classroom Manager. Evaluation staff program observation form sessions were selected for monitor. for TPP grantees. observations. Quality: Quality of youth Observations of interaction Convenience sample of CAS-Carrera Fidelity engagement with program quality using paper OAH 10% of classroom Manager. Evaluation staff program observation form sessions were selected for monitor. for TPP grantees. observations. Control: Experiences of Survey items on baseline Every Fall and Spring. Evaluation staff. comparison condition and follow-up surveys (paper and pencil). Context: Other TPP Standard school Ongoing as needed. Program staff. programming available or programming. offered to study Survey questions. Every Fall and Spring. Evaluation staff. participants (both intervention and comparison) Context: External events 2 school relocations. Once in start of year 3. Program staff. affecting implementation Context: Substantial Program staff turnover/loss Moderate/high. Program staff. unplanned adaptation(s) of staff. 1 teacher strike (delayed Once in start of year 2. programming). OAH = Office of Adolescent Health; TPP = Teen Pregnancy Prevention. 34 Appendix D: Impact model specification The impact model for the primary and secondary research questions is described below. where represents that the outcome of interest observed at final follow-up, is the difference in the observed prevalence rates of the outcome (in percentage points) across the treatment and control conditions, 𝑇𝑇 is a treatment indicator variable, and is a vector of covariates. Covariates include: age, gender, and race/ethnicity. Dummy variables for site were included and a variable representing same-household situations was included as well. This model was repeated for males and females for the secondary research questions. 35 Appendix E: Missing data There were two types of inconsistent or missing data: within surveys and across surveys. However, the prevalence of both was minimal. In fact, the range of missing item rates was 0 to 1 (no youth had missing data on more than one baseline covariate). Since we had surveys completed from youth twice per year over five years, Philliber was able to impute all values that were possible. For example, if the data on ever had sexual intercourse and had sexual intercourse without effective use of contraception were inconsistent (denies sex but uses contraception) we consulted older surveys and used a weight of evidence approach in dealing with such inconsistencies. Additionally, if a child was Hispanic on his first seven surveys, he was coded Hispanic on his final survey. Philliber also had access to program staff who could help fill in missing demographic data if needed such as birth dates, gender, and grade. Table F1. Prevalence of missing data for baseline covariates prior to cleaning Baseline Covariates % missing total (n=274) % missing CAS-Carrera (n=138) % missing control (n=136) Age 1.09% .72% 1.47% Gender 0% 0% 0% Race/ethnicity 1.82% 2.17% 1.47% 36 Appendix F: Implementation evaluation methods Table F.1. Methods used to address implementation research questions Implementation element Methods used to address each implementation element Adherence: How often were sessions offered? Program component attendance data were combined to produce an overall average How many were measure of attendance. offered? Adherence: What and Average number of sessions attended is calculated as the average of the number of how much was sessions that each student attended. received? Adherence: What content was delivered to Specific component curriculum content was not captured. youth? Total number of staff delivering the program is a count of staff members implementing Adherence: Who the program. A snapshot count based on the end of the program is used to determine delivered material to average program staff size. 100% of program staff is trained prior to program youth? implementation. Additionally, program staff receives ongoing staff development. An indicator of staff-participant interactions is calculated as the % of observed Quality: Quality of staff- interactions where the CAS-Carrera Fidelity Manager scored the interaction as “high participant interactions quality” (score of 4 or 5). Note: Because a convenience sample of observations is used to capture staff-participant interaction quality, this measure may not be representative of all possible interactions. Quality: Quality of youth A benchmark of the quality of youth engagement is calculated as the % of observed engagement with interactions where the CAS-Carrera Fidelity Manager scored the overall program program session quality as “high quality” (score of 4 or 5). Note: Because a convenience sample of observations is used to capture overall program session quality, this measure may not be representative of all possible interactions. Control: Experiences of Survey data are presented as means, counts or scores as applicable. control condition Context: Other TPP programming available All of the TPP programming available to both intervention and control groups as or offered to study described by program staff and based on survey questions are listed in this report. participants (both intervention and control) Context: External events affecting Occurrences of school relocations are indicated in this report. implementation Context: Substantial Instances of delayed programming due to teacher strikes and/or staff loss/staff turnover unplanned adaptation(s) which led to interrupted programming are indicated in this report. OAH = Office of Adolescent Health; TPP = Teen Pregnancy Prevention. 37 Appendix G: Sensitivity analyses To test whether the results presented in this report were sensitive to researcher decisions about how data were analyzed, an additional sensitivity analyses was conducted. Tables G1 and G2 show the impact analyses using the benchmark approach (listwise deletion handling missing data), as well as a secondary approach (pairwise deletion handling missing data). Results were consistent using both approaches. Table G.1. Sensitivity of impact analyses using data from final spring 2015 follow-up survey to address the primary research questions Pairwise Pairwise Benchmark Benchmark deletion deletion approach approach p- approach approach p- Intervention compared with comparison difference value difference value Ever had sex 0.065 0.232 0.068 0.210 Had sexual intercourse without effective use of 0.018 0.389 0.017 0.390 contraception in past 3 months Source: Spring 2015 final follow-up data. Notes: See Table III.1 for a more detailed description of each measure and Section III.G for a description of the impact estimation methods. Table G.2. Sensitivity of impact analyses using data from final spring 2015 follow-up survey to address the secondary research questions Pairwise Pairwise Benchmark Benchmark deletion deletion approach approach approach approach p- Intervention compared with comparison difference p-value difference value Ever had sex among males 0.025 0.868 0.088 0.346 Ever had sex among females 0.066 0.557 0.056 0.379 Had sexual intercourse without effective use of -0.024 0.789 -0.000 0.989 contraception in past 3 months among males Had sexual intercourse without effective use of 0.044 0.414 0.031 0.266 contraception in past 3 months among females Source: Spring 2015 final follow-up data. Notes: See Table III.2 for a more detailed description of each measure and Section III.G for a description of the impact estimation methods. 38