Findings from an Innovative Teen Pregnancy Prevention Program Evaluation of Alaska Promoting Health Among Teens, Comprehensive Abstinence and Safer Sex (AKPHAT) in Alaska Final Impact Report for State of Alaska Department of Health and Social Services October 6, 2015 Prepared by Stephanie Martin, Alexandra Hill, Margaret Nye, and Kathryn Hollman-Billmeier Institute of Social and Economic Research University of Alaska Anchorage Martin, S., Hill, A., Nye M., Hollman-Billmeier, K. (2015) Evaluation of Alaska Promoting Health Among Teens, Comprehensive Abstinence and Safer Sex (AKPHAT) in Alaska. Institute of Social and Economic Research, University of Alaska Anchorage. About the authors: Stephanie Martin is an Assistant Professor of Economics and Public Policy at the Institute of Social and Economic Research (ISER) at the University of Alaska, Anchorage. Her work includes evaluations of adult and juvenile criminal justice programs, programs for homeless youth, seat belt use, and food relief and development programs in Africa and Asia. She has also conducted household surveys in Alaska and internationally. She holds a Ph.D. in Political Economy from the University of Texas at Dallas, an M.Pl. in Urban Planning from the University of Southern California, and a B.A. with a Philosophy major from Occidental College, Los Angeles. Alexandra Hill is a Research Associate at the Institute of Social and Economic Research at the University of Alaska, Anchorage. Her work includes studies of indigenous education policy, teacher supply and demand, and education program evaluations. She has also worked on economic impact studies of recreational areas and fiscal policy studies of Alaska’s state government. She holds a master’s degree in Public Administration and Urban and Regional Planning from Princeton University, an M.S. in Business Administration from Boston University, and a B.A. in Engineering Science from Dartmouth College. Margaret Nye is a Research Associate the Institute of Social and Economic Research at the University of Alaska, Anchorage. She evaluates educational and public health programs and supervises data collection on several projects. She holds an M.P.H. in Public Health Practice from the University of Alaska Anchorage and a B.A. with a Psychology major from the University of Michigan. Kate Hohman-Billmeier is a Research Associate the Institute of Social and Economic Research at the University of Alaska, Anchorage. She conducts qualitative research on development and gender issues and teaches at the University of Alaska Anchorage in the Political Science Department and Women's Studies Program. She holds a Ph.D. in Development Studies from the School of Oriental and African Studies University of London, an M.A. in Women's Studies from George Washington University, and a B.A with a Political Science and International Affairs major from University of Mary Washington. Contact information for lead author: Stephanie Martin, slmartinak@gmail.com Acknowledgements: The evaluation team would like to thank program staff at the Alaska Department of Health and Social Services, Division of Public Health: Jenny Baker, Mollie Rosier, and Sophie Wentzel; as well as local non- profit staff and leadership: Marlene McCabe, Kelsey Kaiser, and Angela Peacock at Cook Inlet Tribal Council; Anna Meredith and Doug Koester at Kachemak Bay Family Planning; Chris Mortenson, Laura Herman, and Heather Harris at Alaska Youth Advocates; and Eileen Arnold at Tundra Women's Coalition. We also thank ISER staff members Caleb Billmeier, Glenda Swoop, Susan Powell, Katherine Jackstadt, Katie Cueva, and Leah Coppola who were part of the evaluation team. Finally, we are especially thankful for sage advice from Russell Cole at Mathematica, Inc. and patient guidance from Jacquie McCain at the Office of Adolescent Health. This publication was prepared under Grant Number TP2AH000008 from the Office of Adolescent Health, U.S. Department of Health & Human Services (HHS). The views expressed in this report are those of the authors and do not necessarily represent the policies of HHS or the Office of Adolescent Health. EVALUATION OF ALASKA PROMOTING HEALTH AMONG TEENS, COMPREHENSIVE ABSTINENCE AND SAFER SEX (AKPHAT) IN ALASKA: FINDINGS FROM AN INNOVATIVE TEEN PREGNANCY PREVENTION PROGRAM I. Introduction A. Introduction and study overview The President’s budget for Fiscal Year (FY) 2010 included a new Teenage Pregnancy Prevention (TPP) initiative to address high teen pregnancy rates by replicating evidence-based models (Tier 1) and producing new evidence by developing, refining, and testing innovative strategies (Tier 2). TPP target populations include at-risk, vulnerable, and culturally underrepresented youth populations, including youth in foster care, homeless youth, youth with HIV/AIDS, pregnant women or mothers who are under 21 years of age and their partners, and youth residing in areas with high birth rates for youth. Alaska Department of Health and Social Services (DHSS) saw this funding stream as an opportunity to address Alaska's combination of high teen pregnancy rates, high sexually transmitted infection (STI) rates, and lack of access to services. The birth rate for teens 15-19 in Alaska was 42.7 per 1,000 in 2008 (prior to the start of the program) (State of Alaska 2015). Within the state, rates were as high as 109.2 per 1,000 for Alaska Native youth in some rural areas (State of Alaska 2015). These teen birth rates for Alaska compare to 40.2 per 1,000 for the U.S. in the same year (Kost and Henshaw 2012). Alaska ranked first or second in the country in chlamydia rates from 2000-2010 (CDC 2015). Alaska has minimal health education standards, requiring only one credit for health or physical education as a high school graduation requirement (Alaska State Statute 4 AAC 06.075). In remote isolated rural settings, with very small communities (most with populations of a few hundred people), health care workers are related to most community members and fears about confidentiality keep most teens from accessing information and services locally. Alaska was one of 19 grantees to receive Tier 2 funding from the U.S. Department of Health and Human Services (HHS), Office of Adolescent Health (OAH), to implement an adaptation of Promoting Health Among Teens (PHAT), an evidence-based TPP program (HHS 2015). The Alaska version of the program, called AKPHAT, was adapted to address the unique needs of Alaska. The goal of AKPHAT was to provide programming in Anchorage and rural areas across the south-central and western parts of the state. AKPHAT in Anchorage reached local youth, as well as many who migrate from rural Alaska and often end up homeless. Programming in rural areas provided services directly where it is needed most. Statewide, the program targeted detention facilities, non-traditional schools, foster care, and mental health service providers. AKPHAT programming took place after school, during holidays, and on weekends. It was delivered in shelter facilities, schools, mental health facilities, community meeting rooms, and conference rooms of non-profit partners. B. Primary research questions 3 This report assesses the impact of AKPHAT on measures of sexual activity and sexual risk behaviors. The primary research questions are: What is the impact of AKPHAT compared to no program on recent sexual intercourse six months after the end of the program? What is the impact of AKPHAT compared to no program on the recent use of condoms during sexual intercourse six months after the end of the program? C. Secondary research questions The secondary research questions focus on the effects of attending the two modules deemed critical by PHAT program developers. The developers assert that attending Modules 10 and 12, which focus on condom use and role-playing, respectively, is essential for behavior to change. The secondary research questions are: What is the impact of AKPHAT on recent sexual intercourse six months after the end of the program on youth who attended modules 10 and 12 compared to youth who did not attend these two modules (i.e., youth in the control group and youth assigned to AKPHAT who did not attend both modules 10 and 12)? What is the impact of AKPHAT on the recent use of condoms during sexual intercourse six months after the end of the program on youth who attended modules 10 and 12 compared to youth who did not attend these two modules? II. Program and control programming A. Description of program as intended AKPHAT is the State of Alaska version of the evidence-based program PHAT Comprehensive. PHAT Comprehensive, in turn, is a combination of two existing evidence-based programs: Making Proud Choices!, a safer sex intervention, and PHAT - Abstinence-Only, an abstinence intervention. 1 AKPHAT contains 12 one-hour modules. According to developers, delivery of all 12 modules may be as intensive as over two days or spread out over up to three weeks. Modules teach safer sex and abstinence and include information about STIs, HIV, risky sexual behavior, birth control, and pregnancy. According to developers, attending modules 10 and 12 is essential for behavioral change. Module 10 teaches condom use skills and was designed to use a penis model for demonstration and practice. Module 12 involves role-plays, which allow youth to practice the communication and negotiation skills they learned in the other modules. Tier 2 funding required testing a major adaptation to the evidence-based program. AKPHAT made several adaptations to PHAT: 1 Initially, Making Proud Choices! was selected for implementation. However, in April 2011, then-governor Sean Parnell suspended work funded under this cooperative agreement and required an abstinence-primary program to be used instead. 4 The first was to use peer-educators, rather than adult facilitators, to deliver the program. The four implementing non-profits sought true peers of program participants, ages 16 to 21, and provided training developed by Select Media. The peer educators worked in teams of two to deliver the curriculum to groups of 4 to 10. Adult supervisors kept attendance, timed and monitored module completion, and provided support in case of behavioral issues. The second adaptation was to implement AKPHAT with rural and at-risk Alaska youth, many of whom are Alaska Native people. PHAT and Making Proud Choices! were originally tested with urban African-American youth in middle schools. The third adaptation was to implement with older youth. PHAT and Making Proud Choices! were originally designed for and implemented with 11- to 13-year-olds. Because teen pregnancy and STI rates are highest in Alaska among older teens, and program developers noted that PHAT was effective with older teens, Alaska initially identified 11- to 19-year-olds as the target age group. However, the governor ultimately restricted program participation to youth ages 14 years and older. Curriculum-level adaptations included (1) use of talking circles and talking sticks, and (2) modifications to the condom demonstration module. A talking circle is a Native American ritual used to discuss important issues. Participants sit in a circle to avoid hierarchy. A talking stick is a wooden staff used to identify the speaker and help ensure that everyone is allowed a chance to participate. Only the person with the talking stick may speak, and when done, that person passes the talking stick to the left. AKPHAT also substituted the use of fingers in place of a penis model for the condom demonstration in module 10. B. Description of counterfactual condition No program was provided for youth in the control group. Youth in the control group received whatever services were available to them through school or the community. III. Study design A. Sample recruitment Target population. AKPHAT targeted all youth ages 14 and older served by one of four non-profit groups. Alaska Youth Advocates serves homeless and at-risk youth in Anchorage. Cook Inlet Tribal Council serves all Alaska Native youth in Southcentral Alaska and is one of the largest service providers in the state for Alaska Native people. 2 Kachemak Bay Family Planning clinic serves the youth of Homer, Alaska and rural communities on the Kenai Peninsula. Tundra Women’s Coalition serves youth (nearly all are Alaska Native peoples) in Bethel and across rural western Alaska. Non-profit staff members recruited using youth outreach workers, who are key nodes in the local at-risk teen population. Non-profit administrators recruited by providing information to leaders of alternative schools, mental health organizations, 2 Southcentral Foundation, a tribal health organization representing study participants in Anchorage and southcentral Alaska and with whom we have signed a research agreement, requested that publications use ‘Alaska Native people’ in lieu of ‘Alaska Natives’. ‘Alaska Native’ is a U.S. Census recognized racial category. However, ‘Native’ has historically unfavorable connotations in Alaska. 5 Alaska Native tribal organizations, and other community organizations. All non-profits also displayed posters and notifications in local businesses, alternative schools, and mental health and community centers. Sample formation. Youth recruited through the four non-profits became participants in the evaluation when they and a parent or guardian (in the case of youth under 18) returned signed consent/assent forms, or provided witnessed consent over the telephone. The program was delivered to youth in cohorts, which were defined as the group of youth who were randomized on the same date at the same site. Because AKPHAT targeted at-risk and rural youth who live in small, widely dispersed communities and program implementation was divided among four non- profits statewide, AKPHAT was delivered to 31 cohorts, each comprising 4-10 youth. Consent process. The consent process was the same for treatment and control groups (it took place before randomization) and was uniform across all non-profits. The process was thoroughly reviewed and refined by the University of Alaska Institutional Research Board (IRB), Alaska Area (Indian Health Service) IRB, and Southcentral Foundation and Yukon Kuskokwim Health Corporation (both tribal health organizations). No incentives were provided for consent or assent. Consent and assent forms were provided to youth and parents or guardians prior to program implementation (as a hard copy and/or via email). Parental consent was obtained in two ways: (1) by returning signed forms, or (2) a member of the program or evaluation staff and a second witness called the parent or guardian, checked that they have received the form, answered any questions from the parent or guardian, and recorded the consent or non-consent. B. Research design This study is a randomized control trial with individual-level random assignment within each cohort. The evaluation team conducted randomizations. Youth who provided proof of consent were eligible to be randomized. Randomization occurred while youth took the baseline survey. For each cohort, baseline surveys were administered to the entire group at once. The randomization method was revised after the first year of implementation. At the beginning of the project, as youth submitted consent forms, a member of the evaluation team entered their names into a Microsoft Excel spreadsheet. The Excel random number function (rand) assigned a random number between 0 and 1 to each person. Youth with random numbers less than 0.5 were assigned to AKPHAT. Youth with random numbers greater than or equal to 0.5 were assigned to the control group. Because cohorts were usually small, assignment of random numbers was repeated until half (or half plus 1 in the case of an odd number of youth in a cohort) were assigned to treatment. This process was amended starting July 25, 2013, because continued assignment of the “extra” person to treatment was creating a treatment sample that was more than half of the total. The process, as revised, was used for cohorts with an odd number of members. Before randomization, a coin toss determined whether the extra person would be assigned to treatment or control. Youth were then randomized according to the process described above. The analytic approach differs for the two sets of research questions. To address the primary research questions, we use an intent-to-treat (ITT) analysis that compares outcomes for youth randomly assigned to AKPHAT (i.e., the treatment group) to outcomes for youth randomly 6 assigned to the control group, regardless of the extent of the treatment group’s participation in the program. To answer the secondary research questions, we use a treatment-on-the-treated (TOT) analysis. In the TOT analysis, youth who attended the two modules deemed critical by PHAT program developers (Modules 10 and 12) are compared to youth who did not attend these two modules (i.e., youth in the control group and youth assigned to AKPHAT who did not attend both modules 10 and 12). C. Data collection 1. Impact evaluation The evaluation team administered survey instruments that built on instruments developed by Select Media for use with PHAT. The original Select Media surveys included questions designed to measure attitudes, beliefs, and knowledge. Surveys for the AKPHAT evaluation had additional questions about sexual intercourse, condom, and birth control use. OAH developed the supplemental questions, which were included in all Tier 2 program evaluations. Youth received a $25 gift certificate for each completed survey. Baseline and end-of-program surveys were in pencil-and-paper format. For follow-up surveys (3, 6, and 12 months after the end of the program), youth chose either paper or online surveys. For follow-up surveys outside of Anchorage, locally based evaluation staff administered surveys or Anchorage-based evaluators traveled to the sites. Anchorage based evaluators administered surveys in Anchorage. Each cohort was surveyed five times per cohort starting with randomization and ending 12 months later. Youth provided updated contact information prior to each survey. The evaluation team conducted 155 survey rounds—five survey administrations for each of the 31 cohorts. The first round of randomization and baseline surveys (cohort 1) took place on August 7, 2012. The final round of randomization and baseline surveys (cohort 31) took place on July 30, 2014. Six- month follow-up data collection started on February 5, 2013 with cohort 1 and ended January 30, 2015 with cohort 31. 2. Implementation evaluation The project also collected data to monitor and evaluate AKPHAT adherence to the curriculum and quality of implementations. Peer educators, adult supervisors, program participants, and evaluation staff all provided information. Data were collected using curriculum monitoring logs, program attendance logs, evaluator observation forms, and participant and peer educator debriefs. OAH reviewed and approved all data collection instruments. Information about other programming in the areas where AKPHAT was implemented (context) came from reviewing school district curricula and conversations with staff at Planned Parenthood of the Great Northwest. Appendix A contains a detailed description of implementation evaluation data collection. D. Outcomes for impact analyses The two behavioral outcomes are (1) recent sexual intercourse, measured as sexual intercourse in the past 3 months, and (2) recent unsafe sex, measured as not using a condom during sexual intercourse in the past 3 months. Behavioral outcomes for primary and secondary 7 research questions are computed from baseline and six-month follow-up survey data. Table III.1 describes the outcome measures. Table III.1. Behavioral outcomes used for primary and secondary impact analyses research questions Timing of measure Outcome name Description of outcome relative to program Six months after program Sexual intercourse The variable is a yes/no measure of whether a person has ends in the past 3 had sexual intercourse during the 3 months prior to the months survey. The measure is taken directly from the survey question: In the past 3 months, have you had sexual intercourse, even once? The variable is constructed as a dummy variable where 'yes' responses are coded as 1 and 'no' responses are coded as 0. Six months after program Unsafe sex in the The variable is a measure of condom use. It comes from two ends past 3 months survey questions: In the past 3 months, have you had sexual intercourse without you or your partner using a condom?” In the past 3 months, have you had sexual intercourse, even once? The variable is constructed as a dummy variable. 'Yes' responses to the first question are coded as 1 and 'no' responses to the first question are coded as 0. Where data for the first question are missing, 'no' responses to the second question are coded as 0. E. Study sample We planned to recruit 605 youth each for treatment and control groups and estimated an attrition rate of 25% between baseline and six-month follow-up. The resulting 464 in each group would have provided sufficient statistical power to detect a difference of about 9 percentage points between the treatment and control groups in the proportion reporting sexual intercourse, given a baseline proportion of 35 percent (as reported on baseline surveys). Due to several unforeseen events, our final sample sizes were much smaller. Of 302 total randomly assigned youth, 155 were assigned to AKPHAT and 147 were assigned to the control group. About 40% of each group provided both baseline and six-month follow-up surveys (70 treatment students, 54 control). The overall attrition rate is 59%. Appendix B presents the sample flow from randomization through six-month follow-up. Analytic samples were even smaller: 105 youth (59 treatment and 46 control) provided responses to the sexual intercourse outcome question in baseline and six-month follow-up surveys; 86 youth (50 treatment and 36 control) provided responses to the unsafe sex question in both surveys. Most of the limitations to reaching our target population came from outside the study and were in the form of restricting the eligible population. The first limitation was a federal HHS 8 regulation restricting implementation in detention facilities, the original target setting, unless specific conditions are met for the study. 3 Program eligibility was further curtailed by the then- governor. Besides requiring a program with promoting abstinence as the main objective, he limited the eligible population to youth 14 and older, excluding 11- to 13-year-olds. Later in the project, a suspension from the Alaska Area Institutional Review Board (AAIRB) due to complications around an adverse event 4 halted all recruitment and implementation activity for 14 weeks from November 19, 2013 to February 28, 2014. Recruitment in small rural communities, many with fewer than 20 youth ages 14-19, was difficult for several reasons. In post-implementation interviews with tribal council members, rural implementation staff, and administrators at tribal non-profits, several people told us that tribal organizations were reluctant to implement a randomized study 5. They felt that dividing youth into treatment and control groups could be perceived by their constituents as denying services to some of the population. The combination of randomization and a sensitive topic caused many tribal leaders to decline. In some places AKPHAT was not well received because of the long history of studying and providing programming for Alaska Native people. Alaska Native people reportedly were tired of being research subjects. Recruiting was also problematic because we did not offer programs for both treatment and control groups. In small communities, with few teenagers, everyone usually participates in all activities. Delivering AKPHAT after school, and replacing an activity like basketball, meant that half of the youth had nothing to do. AKPHAT also saw a sharp increase in participation among villages after randomization was completed. Tribal council members and tribal non-profit administrators and staff said that many youth decided not to participate when they learned that they could be in the control group, especially when it meant being separated from a friend. Several factors affected attrition. For some youth, assignment to the control group was emotionally upsetting. Even though they signed consent forms, some cried when they heard their assignment. This could be one reason why attrition was higher for youth in the control group 3 In late August 2012 following their initial review and approval, University of Alaska Institutional Review Board (UAAIRB) found they had not adequately considered federal HHS regulations pertaining to incarcerated youth and that AKPHAT was out of compliance with federal regulations because it was not providing a comparable program to the control group. HHS regulation 45 CFR 45 Subpart C, para 46.306(iv) requires that the control group benefit from the research, and if that is not the case, an exception needs to be listed in the federal register and the Secretary must approve the research. Alaska DHSS staff and evaluation team members worked with non-profits to identify possible programs, staff and training needs, and to develop estimates of increased youth served. For the purposes of the evaluation, the same alternative program would have to have been provided in all sites. After several rounds of discussion among the evaluation team, non-profit providers, Alaska DHSS, the HHS Office of Adolescent Health, and Mathematica (the evaluation technical assistance provider for the grant), detained and incarcerated populations were removed from the eligible pool. Staff training, funding, and space constraints were among the concerns of non-profit providers. 4In November 2013, AKPHAT was awaiting renewal of AAIRB approval. At that time the project was implementing with two of the four non-profits, two that were under the jurisdiction of the UAAIRB, and AKPHAT was not being implemented with tribal non-profits. In December 2013 a breach of confidentiality occurred during an implementation. The evaluation team immediately notified both the UAAIRB and AAIRB. The AAIRB immediately suspended all implementations, saying that AKPHAT was out of compliance regardless of the fact that they were not implementing with any tribal non-profits at the time. The AAIRB understood their jurisdiction to be any agency that serves Alaska Native people and the project should not have been operating at all while waiting for an annual renewal. 5Tribal approval is required before AKPHAT can be implemented in small rural villages. In Alaska, nearly every village has tribal status. 9 than in AKPHAT. Cohorts that were due for follow-up surveys during the Alaska Area IRB suspension were not surveyed or, if surveyed, their data were destroyed. All data, including contact information, were destroyed for two cohorts that were randomized and completed baseline surveys during the suspension. Baseline data were destroyed and no follow-ups were conducted for the implementation in a detention facility. However, the biggest factor in attrition was inability to reach youth. Despite collecting detailed and extensive contact information at baseline, in many cases, neither youth nor any of their contacts could be reached for follow-up. F. Baseline equivalence Analytic samples for baseline equivalence are baseline data for youth who provided responses to sexual behavior questions on baseline and 6-month follow-up surveys. To assess baseline equivalence of treatment and control groups we regressed demographic variables (age, gender, and race) and the baseline measures of outcome variables (sexual intercourse during the past 3 months and unsafe sex during the past 3 months) on dummy variables for treatment and cohort. Racial categories are white, Alaska Native, and other. Alaska Native includes youth reporting their race as Alaska Native alone, or in combination with one or more other races. The treatment dummy variable differs between the ITT analysis used to address the primary research questions and the TOT analysis used to address the secondary research questions. For the ITT analysis, the treatment dummy equals one if the youth was randomly assigned to the AKPHAT group and zero if the youth was randomly assigned to the control group. For the TOT analysis, the treatment dummy equals one if the youth attended AKPHAT modules 10 and 12 and zero if the youth did not attend these modules; thus, both youth randomly assigned to the control group and youth randomly assigned to the AKPHAT group who did not attend both modules 10 and 12 are coded as zero. As noted earlier, randomization occurred within cohort (i.e., each cohort is a stratum). The probability of random assignment (for cohorts randomized before July 25, 2013) varied slightly because of how the odd number in a cohort was treated (with the “extra” always being assigned to treatment). We adjusted for this in the baseline equivalence and impact analyses by including dummy variables for cohort as independent variables. For all equations, we used the STATA Regress command to run the following ordinary least squares (OLS) regression models: P (Y = 1) = β 0 + β1 ∗ T + C j + u for dichotomous variables Y = β 0 + β1 ∗ T + C j + u for continuous variables Where, In the case of dichotomous variables (female, race indicators, and baseline measures of outcome variables), P(Y = 1) is the probability that variable Y = 1; for a continuous variable (age), Y is a measure of the variable (age in years). 10 is the intercept. T is a dummy variable for treatment (for ITT analysis, T = 1 if randomly assigned to AKPHAT, T = 0 if randomly assigned to control; for TOT analysis, T = 1 if attended AKPHAT modules 10 and 12, T = 0 if did not attend AKPHAT modules 10 and 12). Cj is a set of cohort dummy variables. u is a disturbance term. We ran four rounds of estimation equations to assess baseline equivalence, one round for each of the primary and secondary research questions. Each round contained six equations, one for each baseline characteristic. We did not adjust for multiple comparisons because adjustments increase the probability of a Type II error (mistakenly finding no difference between treatment and control groups). 6 Tables III.2 through III.5 show that despite high attrition rates, the data do not show statistically significant differences between treatment and control groups on baseline measures. However, in the analytic sample for estimating recent sexual intercourse in the ITT analysis, there is a notable difference in the proportion of females in treatment and control groups. The analysis controls for this statistically by including female as a covariate in the regression equations. Table III.2. Summary statistics of key baseline measures for youth completing sexual intercourse questions in AKPHAT baseline and 6-month follow-up surveys, Intent-to-treat analysis Treatment Treatment Control Treatment versus mean or % mean or % versus control p- (standard (standard control mean value of Baseline measure deviation) deviation) difference difference Age 16.64 (1.15) 16.79 (1.16) -0.15 0.510 Female 62.57 74.09 -11.52 0.238 Race . . . . White 39.26 36.60 2.65 0.759 Alaska Native (alone or in 44.84 51.18 -6.33 0.415 combination with another race) Other 15.90 12.21 3.69 0.592 Sexual intercourse in past 3 months 34.14 32.30 1.84 0.851 Sample size 59 46 105 . 6 Cole et al., 2013. 11 Table III.3. Summary statistics of key baseline measures for youth providing valid data for unsafe sex questions in AKPHAT baseline and 6-month follow-up surveys, Intent-to-treat analysis Treatment Treatment Control Treatment versus mean or % mean or % versus control control (standard (standard mean p-value of Baseline measure deviation) deviation) difference difference Age 16.64 (1.19) 16.59 (1.20) 0.05 0.867 Female 63.80 69.73 -5.93 0.590 Race . . . . White 42.46 35.47 6.98 0.485 Alaska Native (alone or in 47.70 50.41 -2.71 0.758 combination with another race) Other 9.83 14.11 -4.27 0.546 Sex without a condom in past 3 25.32 20.38 4.95 0.629 months Sample size 50 36 86 . Table III.4. Summary statistics of key baseline measures for youth completing sexual intercourse questions in AKPHAT baseline and 6-month follow-up surveys, treatment-on-treated analysis Treatment Treatment Treatment versus versus mean or % Control mean control control p- (standard or % (standard mean value of Baseline measure deviation) deviation) difference difference Age 16.66 (0.021) 16.75 (0.024) -0.08 0.725 Female 64.76 70.77 -6.01 0.547 Race . . . . White 39.23 36.85 2.38 0.787 Alaska Native (alone or in 45.55 49.90 -4.35 0.584 combination with another race) Other 15.22 13.26 1.97 0.799 Sexual intercourse in past 3 months 34.36 32.87 1.49 0.893 Sample size 55 50 105 . Source: AKPHAT, 2015. Baseline and six-month follow-up surveys, and attendance records. 12 Table III.5. Summary statistics of key baseline measures for youth providing valid data for unsafe sex questions in AKPHAT baseline and 6-month follow-up surveys, treatment-on-treated analysis Treatment Treatment Treatment versus versus mean or % Control mean control comparison (standard or % (standard mean p-value of Baseline measure deviation) deviation) difference difference Age 16.71 16.51 0.20 0.471 Female 68.02 64.27 3.75 0.740 Race . . . . White 42.48 36.14 6.34 0.537 Alaska Native (alone or in 48.23 49.37 -1.14 0.899 combination with another race) Other 8.15 15.63 -7.48 0.301 Unsafe sex in past 3 months 24.75 21.54 3.21 0.760 Sample size 46 40 86 . Source: AKPHAT, 2015. Baseline and six-month follow-up surveys, and attendance records. G. Methods 1. Impact evaluation The analytic samples for each of the primary research questions include youth who have 6- month follow-up survey data with valid responses to outcome questions and corresponding baseline data. The analysis data sets contain data for all sites and cohorts. The benchmark analysis is a complete case analysis, where youth who are missing data on outcome variables or covariates are omitted. As a sensitivity analysis, we also conducted multiple imputations to adjust for attrition, and results were similar. Those results are presented in Appendix D. We used the STATA Regress command to run OLS to estimate linear probability models. Models to address the primary and secondary research questions are the same, except the treatment variable is defined differently. As discussed in the preceding section, the ITT analysis addressing the primary research questions defines treatment based on random assignment. The TOT analysis addressing the secondary research questions defines treatment based solely on attendance of the AKPHAT modules 10 and 12. The models are structured as follows: Outcome variables (Y) Sexual intercourse = sexual intercourse in past three months (1 = yes, 0 = no) Unsafe sex = sexual intercourse without a condom past three months (1 = yes, 0 = no) Where, Y is the outcome variable, either sexual intercourse or unsafe sex 13 P (Y = 1) is the probability that variable Y = 1 β 0 is the intercept T is a dummy variable for treatment (for ITT analysis, T = 1 if assigned to AKPHAT, T = 0 if randomly assigned to control; for TOT analysis, T = 1 if attended AKPHAT modules 10 and 12, T = 0 if did not attend AKPHAT modules 10 and 12). C j is a set of cohort dummy variables β k are coefficients X k is a set of demographic characteristics (age, race, and gender) and baseline value of the outcome variable u is a disturbance term. 2. Implementation evaluation The implementation evaluation summarizes fidelity monitoring in order to provide context for impact findings. AKPHAT collected data to measure adherence to the program as it was designed, quality of implementations, the counterfactual, and the context in which AKPHAT operated, with special attention to elements that program designers deemed essential for success. Information about context and the counterfactual came from personal conversations and school district websites. Appendix C describes implementation data elements in detail. Adult supervisors monitored adherence. Every implementation had an adult supervisor in attendance. Adult supervisors timed sections within each module and monitored how well peer educators completed required activities for each module. They also kept attendance records. Measures of implementation quality come from evaluator program observations and participant debriefs. Four indices measure program quality. For the first index, evaluators rated peer educators on their rapport and communication with participants. Data for the other three indices come from participant debriefs. For the second index participants reported their perceptions of peer educator qualifications and credibility. The third index is participants’ reported engagement with the program. The fourth is participants’ reported comfort level with the program. IV. Study findings A. Implementation study findings Adherence. Peer educators implemented AKPHAT with 31 cohorts. Of the 31 cohorts, implementation data were collected for 29 cohorts. Data were destroyed for the other two in accordance with IRB instructions. Attendance records show that all youth assigned to AKPHAT attended at least one module. About two-thirds (68%) attended all 12 modules, 79% attended 75% or more, and 84% and 82% attended modules 10 and 12, respectively. 14 In terms of duration, except for module 12, the implementation adhered to the program design. Each module was designed to last 60 minutes. The average duration was 53 minutes. Module 12 took longer, averaging 77 minutes. All modules were implemented as designed with two peer educators and within the recommended number of days. Of the 29 cohorts, more than half (17) were delivered over 3 days, 7 were delivered over 4 days, 3 over 5 days, and 1 over 3 weeks. Overall, 90% of the modules were completed, either as designed or with adaptations. Ten percent of modules were flagged as 'incomplete'. An 'incomplete' designation means activities were omitted. This happened most often when multiple modules were delivered in one meeting. Every module starts and ends with a talking circle. Peer educators dropped talking circles when they would have taken place in the middle of a meeting. For example if modules 4 and 5 were delivered sequentially in one meeting, talking circles at the end of module 4 and the beginning of module 5 were omitted. The same was true for reviewing homework. If homework was assigned in module 4, and modules 4 and 5 were delivered together, discussion of homework was moved to the start of the next meeting. Adaptation rates were highest for modules 4, 12, and 8. Within module 4, seven implementations skipped the last role-play due to lack of time. Within module 12, in 5 implementations, youth refused to do unscripted role-plays at the end of the module. Peer educators reported that youth felt they had done enough role-plays. Within module 8, four peer educations modified a basketball-like game where teams score points for correct answers. Adaptations were changes to scoring and using a waste-basket as a basketball hoop. In cases where modules took longer than 60 minutes to implement, the most common adaptation was to extend the length of time for the activity, or to complete the activity at the start of the next meeting. Quality. Quality of AKPHAT implementations has two components: quality of peer educator-participant interactions, and quality of youth engagement with the program. Two indices measure each component. The first index measuring the quality of peer educator-participant interactions used data from participant debriefs. Questions asked about whether peer educators know about what they are teaching, are good role models, well prepared, and so forth (questions are listed in Appendix A). The response scale ranged from 1 (disagree strongly) to 5 (agree strongly). Peer educators averaged 4.5 out of 5. The second index measuring the quality of peer educator-participant interactions used data from evaluator observation forms. Evaluators ranked peer educators' 'rapport and communication with participants' on a scale from 1 (doesn't remember names, doesn't connect with participants, acts distant) to 5 (gets participants very excited, very friendly, uses people's names when appropriate, seems to understand the community and its needs). Peer educators averaged 4.2 out of 5. Both indices measuring the quality of youth engagement with the program used data from participant debriefs. The first index measured participant engagement with the program on a scale from 1 (not at all) to 4 (very much). Peer educators averaged 3.5 out of 4. The second index measured participant comfort with the program, using a scale from 1 (very uncomfortable) to 5 (very comfortable). Participants rated their comfort with the program as 4.1. 15 Counterfactual. AKPHAT did not provide a program for the control group. Data from participants indicate that some control group members received AKPHAT information. Spillovers were difficult to avoid because we were working in small communities and within close-knit groups. Among AKPHAT participants who completed program debrief surveys, 34% reported that they talked with ‘someone else who was recruited but not in the program about what they learned’. Of those, most reported that they talked about condom use, STI and pregnancy prevention. A similar share, 34%, reported that someone else in their household was in the program, either as a control or treatment, and 20% reported that they had a relative in the program. Context. To evaluate context we used information from Anchorage, Kenai, Bethel, and Yupiit school district websites and conversations with other agencies implementing programs in the same geographic areas in which AKPHAT operated. During the 2013-2014 and 2014-2015 school years, one high school in Anchorage had a pilot elective course, “Healthy Relationships/Sexuality Education.” No other services were offered through the schools. Planned Parenthood of the Great Northwest was implementing an evidence-based TPP program, the Teen Outreach Program, in Anchorage. In the 2013-14 school year, 52 students were enrolled. In 2014-2015, 44 students were enrolled. We do not know if any AKPHAT youth also participated in the Teen Outreach Program (Personal communication with Lacey Moran, Alaska Education Coordinator, Planned Parenthood of the Great Northwest). B. Impact study findings Our research showed that results appear to be trending in the right direction, but are not statistically significant. The data failed to show a statistically significant difference between treatment and control groups on either outcome measure. Table IV.1. Post-treatment estimated effects using data from AKPHAT to address the primary research questions, intent-to- treat analysis Treatment compared to controls difference (p-value of Outcome measure Treatment % Control % difference) Sexual intercourse in past three months 39.03 43.42 -4.38 (0.633) (n=105) Unsafe sex in past three months (n=86) 24.14 24.80 -0.66 (0.938) Source: AKPHAT, 2015. Baseline and six-month follow-up surveys. Notes: The intent-to-treat analysis compares outcomes for youth randomly assigned to AKPHAT (the treatment group) versus youth randomly assigned to the control group regardless of the extent of the treatment group’s participation in the program. See Table III.1 for a more detailed description of each measure and Section III for a description of the impact estimation methods. Secondary research questions allow us to address one of the assertions of the program developers—that attendance at modules 10 and 12 sessions is essential for behavior change. Our study failed to show that among youth who attended modules 10 and 12, AKPHAT was effective in reducing sexual intercourse or unsafe sex (Table IV.2). 16 Table IV.2. Post-intervention estimated effects using data from AKPHAT to address the secondary research questions, treatment-on-treated analysis Treatment compared to control difference (p-value of Outcome measure Treatment % Control % difference) Sexual intercourse in past three months 38.24 43.94 -5.70 (0.537) (n=105) Unsafe sex in past three months (n=86) 24.58 24.24 -0.34 (0.969) Source: AKPHAT, 2015. Baseline and six-month follow-up surveys. Notes: The treatment-on-treated analysis compares outcomes for youth who attended AKPHAT modules 10 and 12 versus youth who did not attend both of these modules, regardless of the group to which the youth were randomly assigned. See Table III.1 for a more detailed description of each measure and Section III for a description of the impact estimation methods V. Conclusion AKPHAT failed to demonstrate an effect on sexual intercourse or condom use. The study sample was much smaller than expected, and political issues altered intended plans. Unfortunately, we cannot explain why the program did not produce a measurable effect. AKPHAT adaptations to PHAT were: to use peer educators to deliver the program, to implement with 14- to 19-year-olds instead of 11- to 13-year-olds, and to implement with rural and at-risk youth in Alaska. The curriculum was modified to use talking circles and talking sticks and to demonstrate condom use without a penis model. The combination of low recruitment and high attrition limited our sample size, and thus statistical power. Delays and suspensions were the major reason for the small sample. The combination of delays and suspensions meant 10 months when we could not recruit, 61 surveys that were destroyed, and close to 200 surveys that were destroyed or could not be administered. The AKPHAT design and the decision to not provide programming for control group members also limited our ability to work in small communities, and reach our recruitment goals. Disallowing the use of a penis model in training, which was deemed essential by program developers, may have lowered the AKPHAT effectiveness. We also do not know if the program did not have an effect because it was not implemented with 11- to 13-year-olds. These two factors became unintended program adaptations. 17 VI. References State of Alaska, Department of Health and Social Services. Data and statistics. http://dhss.alaska.gov/dph/VitalStats/Pages/data/default.aspx) updated January 26, 2015 accessed September 19, 2015 Kathryn Kost and Stanley Henshaw U.S. teenage pregnancies, births and abortions, 2008: national trends by age, race and ethnicity. Guttmacher Institute. February 2012. Centers for Disease Control. 2013: sexually transmitted diseases surveillance. Table 3. Chlamydia - reported cases and rates of reported cases by state/area and region in alphabetical order, United States and outlying areas, 2009-2013. http://www.cdc.gov/std/stats13/tables/3.htm. Updated December 16, 2014, Accessed September 19, 2015. US Department of Health and Human Services. TPP evidence-based programs. http://www.hhs.gov/ash/oah/oah-initiatives/teen_pregnancy/db/ Updated April 29, 2015. Accessed September 19, 2015. Colman, S. Estimating program impacts for a subgroup defined by post-intervention behavior: Why is it a problem? What is the solution? Evaluation Technical Assistance Brief for OAH & ACYF Teenage Pregnancy Prevention Grantees. Brief 2 December 2012. Jemmott, L.S., Jemmott, J.B. Sexual knowledge, attitudes, and risky sexual behavior among inner-city black male adolescents. Journal of Adolescent Research 1990; 5: 346. Deke, J., Puma, M. Coping with missing data in randomized controlled trials. Evaluation Technical Assistance Brief for OAH & ACYF Teenage Pregnancy Prevention Grantees Brief 3 May 2013. Puma, M.J., Olsen, R.O., Bell, S.H., Price C. What to do when data are missing in group randomized controlled Trials (NCEE 2009-0049). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. 2009. Little, R.J., Rubin, D.B. Annual causal effects in clinical and epidemiological studies via potential outcomes: concepts and analytical approaches. Review of Public Health. 2000. 21:121- 45. Cole, R., Deke, J., Zief, S. Teen pregnancy prevention evaluation technical assistance – analysis plan Frequently Asked Questions. Mathematica Policy Research. May 17, 2013. Dunn, O.J. Multiple comparisons among means, Journal of the American Statistical Association. 1961. 18 Appendix A: Implementation evaluation data collection Table A.1. Data used to address implementation research questions Implementation Types of data used to assess whether the element of the Frequency/sampling of data Party responsible element intervention was implemented as intended collection for data collection Total number of cohorts provides background information on Every cohort is reported in MIS. Adult supervisors Adherence: How program implementation. often were sessions offered? How many Length in minutes of each module is reported in curriculum Adult supervisors 1 complete were offered? monitoring logs. This provides further descriptive information and is curriculum-monitoring logs at the used to compare AKPHAT implementation to the program as it was end of each cohort. designed. These data also describe variation in delivery. According to program Adult supervisors record delivery developers, implementations could range from 2 days to several time for each module weeks. Session-level attendance from the program attendance log. Attendance is reported for every Adult supervisors. Adherence: What module. and how much was The evaluation used the same data to report the percentage of received? youth assigned to AKPHAT who attended 100% of sessions, and the percentage of youth assigned to AKPHAT who attended the session containing modules 10 and 12. Program developers identified attending Modules 10 and 12 as essential for behavior change. The analysis provides descriptions of peer educators' ability to Reports for each module on Adult supervisors Adherence: What complete modules on time, deliver modules within the intended activities completed. Reports are content was amount of time, and make adaptations to modules, as well as peer completed at the end of each delivered to youth? educator perceptions of which modules were the most and least session. successful. This information comes from curriculum monitoring logs. The logs contain a list of modules and for each module the list of required activities. 1 Sessions are led by peers, but an adult supervisor attends each peer-led session. 19 Implementation Types of data used to assess whether the element of the Frequency/sampling of data Party responsible element intervention was implemented as intended collection for data collection AKPHAT was delivered by peer educators with oversight by adult Data are collected every time a Program staff Adherence: Who supervisors. Information about which peer educator delivered each peer education is hired. Data are delivered material to module comes from the curriculum monitoring logs. also collected every time a peer youth? Information about qualifications of peer educators and adult educator completes Collaborative supervisors comes from program documentation and quarterly Institutional Training Initiative reports. (CITI) human subjects protection training. Each peer educator is required to fulfill training obligations prior to implementing, and to receive ongoing enrichment training applicable to population. Data on all staff members are available to program staff. After completion of Module 12 Adult supervisors Quality: Quality of This measure is an index from seven questions from the participant for each cohort collect data from peer educator- debrief. All are scored 1 to 5, disagree strongly to agree strongly. program participants participant 1. My peer educator really knows what he or she is teaching interactions 2. My peer educator is a good role model for me. 3. My peer educator really understands youth my age. 4. My peer educator shows respect for the group’s feelings. 5. My peer educator is very friendly. 6. My peer educator was well prepared. 7. My peer educator knows a lot about life. All participants who attend the final day of the program complete the survey. A limitation is that only youth who attended the final module complete the survey. 20 Implementation Types of data used to assess whether the element of the Frequency/sampling of data Party responsible element intervention was implemented as intended collection for data collection Data come from program observation forms. Reports from classroom Evaluation staff Rank peer educators' 'rapport and communication with observations (10% convenience participants'. Responses categories are 1-5, where 1 = poor, 3 = sample of sessions) average, 5 = excellent. Examples of 'poor' and 'excellent': 1 Doesn’t remember names; does not “connect” with participants; acts distant or unfriendly. 5 Gets participants talking and excited; very friendly; uses people’s names when appropriate; seems to understand the community and its needs. Data come from participant debrief surveys. Participants report on Participant debrief surveys are at Adult supervisors Quality: Quality of their engagement with the material, and on their reactions to the end of every cohort, all youth in collect data from youth engagement peer educators and to the curriculum. The response scale is 1 to 4, attendance complete surveys. program participants with program from 'not at all' to 'very much'. • How much did you get into the group activities? • How much did you talk and share your thoughts in the group? Participant debriefs also ask youth to rate their comfort with the program. The response scale is 1 to 5 very uncomfortable) to 5 very comfortable. • How comfortable did you feel talking and sharing your thoughts in the group? • How comfortable did you feel during the exercises, games, and role-playing? Business as usual is the counterfactual condition of AKPHAT. Participant debrief surveys are at Adult supervisors Counterfactual: AKPHAT did not provide any program to the control group. end of every cohort, all youth in collect data from Experiences of Surveys do not ask about other programming. Generally, there is no attendance complete surveys. program participants control condition sex education programming in Alaska. Information from participant debrief surveys about communication between participants in the AKPHAT program and the control group. 21 Implementation Types of data used to assess whether the element of the Frequency/sampling of data Party responsible element intervention was implemented as intended collection for data collection Descriptions of programming offered in schools and by other Once per year Evaluation staff Context: Other TPP programs similar to AKPHAT, obtained from Planned Parenthood programming staff and school district websites. available or offered to study participants (both treatment and control) Minutes from teleconferences and IRBnet (online IRB document Bi-weekly minutes, IRBnet Program staff, Context: External management system). Local provider staff, evaluation, staff, and irregular intervals evaluation staff events affecting state project administrators attend all teleconferences. Evaluators implementation and the rest of the implementation team learn about external events from provider staff during via bi-weekly meetings. Events affecting implementation are highly idiosyncratic, such as the death of a relative of a peer educator. Information about these events is through word of mouth and not usual media sources. AKPHAT implemented in small settings instead of schools, so information comes from implementing partners. IRBnet provides dates of IRB related suspensions and instructions to destroy data. Curriculum monitoring log, work plan, six-month progress report, Twice a year Program staff, project Context: Substantial annual progress report director, evaluation unplanned staff adaptation(s) AKPHAT = Alaska Promoting Health Among Teens. MIS = Management Information System. TPP = Teen Pregnancy Prevention. 22 Appendix B: Study sample Table B.1 Youth sample sizes by Treatment status Total Treatment Control Total Treatment Control . Time Period sample size sample size sample size response rate response rate response rate Number of Youth . . . . . . . 07-Aug-2012 to 29- 1. Assigned to condition . Jul-2014 302 155 147 . . 2. Contributed a baseline 07-Aug-2012 to 29- survey Jul-2014 302 155 147 1 1 1 3. Contributed a follow-up Immediately post- survey programminga na 122 na na .787 na 4. Contributed a follow-up 3-months post- survey programming 155 79 76 .513 .510 .517 6-months post- 5. Contributed a follow-up programming .367 survey 124 70 54 .411 .452 na = not applicable. a The immediately post-programming survey was administered only to the treatment group. 23 Appendix C: Implementation evaluation methods Table C.1. Methods used to address implementation research questions. Implementation element Methods used to address each implementation element Adherence How often were Sum of the cohorts captured in the MIS. Each cohort received 12 modules. sessions offered? How Average module duration is the average of module length (12 modules per cohort) many were offered? measured in minutes. What and how much Average attendance per session. was received? The percentage of youth attending 100% of sessions (12 modules) equals the number of youth assigned to AKPHAT attending all sessions divided by the total number of youth assigned to AKPHAT. A limitation is that a session can contain 1 or more modules. We did not keep track of tardiness or participants leaving before the end of a module. Our estimates may overstate how much program content was received. The percentage of youth attending module 12. This is equal to the number of youth assigned to AKPHAT who attended module 12 divided by the total number of youth assigned to AKPHAT. What content was The percentage of modules completed equals the number of modules completed delivered to youth? divided by the total number of modules delivered. 'Completed' includes 'completed as designed', and 'completed with adaptations'. Within module completion rates: For each module, total activities completed divided by total activities in the module. Module adaptation rates: For each module, total activities completed with changes divided by total activities in the module. We reviewed data and summarized completion rates by type of activity such as homework, role-plays, and information. A possible limitation of these data is that they are self-reported by peer educators and descriptions may not be standard across all peer educators Who delivered material Total number of peer educators delivering the program is count of peer educators to youth? implementing the program. Quality 24 Implementation element Methods used to address each implementation element Quality of peer- The first measure is the mean of responses to seven questions from the participant educator-participant debrief. All are scored 1 to 5, disagree strongly to agree strongly. interactions 1. My peer educator really knows what he or she is teaching 2. My peer educator is a good role model for me. 3. My peer educator really understands youth my age. 4. My peer educator shows respect for the group’s feelings. 5. My peer educator is very friendly. 6. My peer educator was well prepared. 7. My peer educator knows a lot about life. The second measure is the mean of responses to question 6d in the evaluator observation report. Evaluators are asked to rank peer educators' 'rapport and communication with participants'. Responses categories are 1-5, where 1=poor, 3=average, 5=excellent. 1. Doesn’t remember names; does not “connect” with participants; acts distant or unfriendly. 2. Gets participants talking and excited; very friendly; uses people’s names when appropriate; seems to understand the community and its needs. A limitation is that this is a convenience sample and may not be representative 2. Quality of youth The first measure is of youth engagement. It is the mean of responses 1 to 4 (not at all engagement with to very much) to the following questions: program 1. How much did you get into the group activities? 2. How much did you talk and share your thoughts in the group? The second measure assesses participant comfort with the program. It the mean of scores (1 to 5, very uncomfortable to very comfortable). Index is the mean of scores. 1. How comfortable did you feel talking and sharing your thoughts in the group? 2. How comfortable did you feel during the exercises, games, and role-playing? Counterfactual 2 Evaluators developed decision rules about sessions to observe based on whether travel to the site was involved. If travel was involved and evaluators were only in town for baseline surveys, randomization, and the initial session, then they observed the initial session. If travel was involved and evaluators were in town for first and last sessions, then they observed the last session. If travel was not involved (all sessions in Anchorage), neither the first nor last session was observed. Selection also depended on evaluators' schedules and availability. 25 Implementation element Methods used to address each implementation element Experiences of We did not provide any program for the control group. Because we are implementing counterfactual outside of schools, the control group did not receive any instruction or activity during condition AKPHAT implementation. Additionally, we did not survey the control group at the end of the program. No data were collected on the control group until 3 months after the baseline. We do not ask about exposure to programs with similar content. Across our program areas, there is one elective program that covers content similar to that in AKPHAT in one Anchorage high school. Youth in AKPHAT may have been exposed to this program, but we have no way of knowing if that was the case. However we did ask participants (treatment) if they had discussed the program with any members of the control group. Spillover effects: Percentage responding 'yes' to question 22 from the participant debrief: Have you discussed the program with a teenager who was in a group that was recruited but did not take the curriculum? One limitation of these data is that we cannot link the respondent to the person they talked to. The second is that it was asked at the end of the program—a narrow time frame. Summary of qualitative responses to question 23: If you have discussed the program with a teenager who was in a group that was recruited but did not take the curriculum, what did you tell them you learned? One limitation of these data is that we cannot link the respondent to the person they talked to. The second is that it was asked at the end of the program—a narrow time frame Summary of qualitative responses to question 24: If you have discussed the program with a teenager who was in a group that was recruited but did not take the curriculum, what did they tell you? One limitation of these data is that we cannot link the respondent to the person they talked to. The second is that it was asked at the end of the program—a narrow time frame Context Other TPP programming All of the TPP programming available in areas where AKPHAT was implemented was available or offered to obtained from discussions with Planned Parenthood of the Great Northwest and from study participants (both participant debriefs. Treatment and A limitation is that we don't know whether anyone in AKPHAT participated. counterfactual) External events The number of days lost due to external influences (AAIRB, UAAIRB and government affecting shutdowns) implementation 26 Implementation element Methods used to address each implementation element Substantial unplanned Summary of restrictions from the Governor’s office which became unplanned adaptation(s) adaptations. The number of cohorts that were conducted with one peer educator as opposed to two (as intended). The number of unplanned changes in curriculum implementation. AAIRB = Alaska Area Institutional Review Board. AKPHAT = Alaska Promoting Health Among Teens. UAAIRB=University of Alaska Institutional Review Board. TPP = Teen Pregnancy Prevention. 27 Appendix D: Sensitivity analyses Table D.1 presents sensitivity analysis done to address missing data issues in addressing the primary research questions (ITT analysis). Multiple imputations compensate for missing response data from six month follow-up surveys. The imputation model included the same covariates as the impact estimation models. Missing outcome data were imputed separately for the treatment and control groups. Logit models were used for imputations. For the first outcome variable, ‘sexual intercourse in the past three months’, 77 values were imputed. The benchmark approach included 105 observations. The multiple imputations approach included 182 observations. For the second outcome, the baseline analysis included 86 observations. The multiple imputations approach included 138 observations, 52 values were imputed. There were no statistically significant differences on any baseline variables when analyzed using the multiply imputed data (results available from the author by request). In estimating the effect of AKPHAT on sexual intercourse and unsafe sex, the two methods for addressing missing data produce nearly identical results. Results are not statistically significant using either method. Table D.1. Sensitivity of impact analyses using data from AKPHAT to address missing data the primary research questions, intent-to-treat analysis Benchmark approach, Complete case Multiple analysis Imputation p- p- Diff. value Diff. value Treatment compared with Control Sexual intercourse in -4.38 0.633 -.5.37 0.496 past three months Unsafe sex in past three -0.66 0.938 0.61 0.922 months Source: AKPHAT, 2015. Baseline and six-month follow-up surveys. Notes: The intent-to-treat analysis compares outcomes for youth randomly assigned to AKPHAT (the treatment group) versus youth randomly assigned to the control group regardless of the extent of the treatment group’s participation in the program. For the first outcome variable, ‘sexual intercourse in the past three months’, the benchmark approach included 105 observations and the multiple imputations approach included 182 observations (77 values were imputed). For the second outcome, the baseline analysis included 86 observations, the multiple imputations approach included 138 observations (52 values were computed). See Table III.1 for a more detailed description of each measure and Section III for a description of the impact estimation methods. AKPHAT = Alaska Promoting Health Among Teens. 28