Contract Number: Making a Plan and Sticking to It: HHSP23320082911YC Implementing an Enhanced Version Mathematica Reference Number: of HealthTeacher in Chicago 06549.090 Submitted to: Final Report Office of Adolescent Health Department of Health & Human August 15, 2013 Services 1101 Wootton Parkway, Suite 700 Rockville, MD 20852 Rachel Shapiro (Mathematica Policy Research) Ellen Kisker (Twin Peaks Partners, LLC) Project Officer: Amy Farb Submitted by: Mathematica Policy Research P.O. Box 2393 Princeton, NJ 08543-2393 Telephone: (609) 799-3535 Facsimile: (609) 799-0005 Project Director: Alan Hershey Implementing an Enhanced Version of HealthTeacher in Chicago Mathematica Policy Research CONTENTS INTRODUCTION ........................................................................................................................1 IMPLEMENTATION OF HEALTHTEACHER ..............................................................................4 CPS Enhanced HealthTeacher to Meet Its Needs ......................................................4 Implementation Schedules Were Set by Schools and Adapted by Teachers ..............................................................................................................5 CPS Training Was Generally Valued by Teachers ......................................................7 ADHERENCE TO THE IMPLEMENTATION PLAN .....................................................................9 STUDENT PARTICIPATION AND ENGAGEMENT ..................................................................12 IMPLEMENTATION SUCCESSES, CHALLENGES, AND INSIGHTS FOR THE IMPACT STUDY .......................................................................................................14 Successes ................................................................................................................14 Challenges ................................................................................................................15 Insights Relevant for Interpreting HealthTeacher Impacts ......................................... 16 LOOKING FORWARD: LESSONS TO INFORM FUTURE IMPLEMENTATION EFFORTS .................................................................................................................18 APPENDIX A POPULATION SERVED AND SAMPLE CHARACTERISTICS .......................... 19 APPENDIX B IMPLEMENTATION STUDY DATA SOURCES AND METHODOLOGY .....................................................................................................24 APPENDIX C SUMMARY OF HEALTHTEACHER LESSONS ................................................. 29 iii Implementing an Enhanced Version of HealthTeacher in Chicago TABLES 1 Planned Time Lines for Implementation of Enhanced HealthTeacher Curriculum ..................................................................................................................5 2 Time Line for Program Delivery ..................................................................................6 3 Average Daily Attendance Rates During the Period of Program Implementation .........................................................................................................12 A.1 National School Lunch Program Qualification Status of Students in Treatment Schools, FY 2011 ....................................................................................20 A.2 Sample Characteristics .............................................................................................21 B.1 Data Sources ............................................................................................................26 C.1 Summary of HealthTeacher Lessons ........................................................................30 FIGURES 1 Logic Model of the HealthTeacher Intervention ...........................................................2 iv Implementing an Enhanced Version of HealthTeacher in Chicago INTRODUCTION In 2010, Chicago Public Schools (CPS) was facing a dilemma—how to deliver a consistent sex education program in a complex and diverse school district. CPS wanted schools to comply with the Chicago Board of Education’s 2006 policy calling for students in grades 5–12 to receive family life and comprehensive sex education. CPS was particularly interested in teaching sex education to youth before they entered high school, with a goal of reducing rates of teen births and sexually transmitted infections (STIs). To satisfy the 2006 policy and to reach pre-high school age youth, CPS wanted schools serving 7th grade youth to use programs it recommended; however, CPS did not have the staff or funding to monitor whether or how programs were actually offered or delivered in schools. Mathematica Policy Research found that 7th grade students received little, if any, sex education in the 2009–2010 academic year. CPS recommended three ways to meet the 2006 mandate: (1) HealthTeacher, an online comprehensive K-12 health education curriculum through which teachers can access lesson plans and materials; (2) the Family Life and AIDS Education handbook, developed by CPS; and (3) programs offered by several CPS-approved organizations—the Stakeholders Collaboration to Improve Student Health, Communities in Schools in Chicago, and the Chicago Department of Public Health (CDPH). 1 CPS was particularly interested in the use of the HealthTeacher program as a way to promote adherence to its policy. However, resources to monitor and support the use of HealthTeacher district-wide were not available; district support was limited to paying Relegent (the curriculum developer) for the site license to use HealthTeacher, and providing training to classroom teachers in HealthTeacher. Still, CPS wanted to determine whether consistently implementing the sex education portion of HealthTeacher could delay sexual activity and prevent sexual risk behavior. Evidence to support this might influence CPS’s future resource availability and allocation. HealthTeacher Evaluation—A Snapshot • Part of the national multiyear Evaluation of Adolescent Pregnancy Prevention Approaches o Funded by the Office of Adolescent Health, U.S. Department of Health and Human Services o Conducted by Mathematica Policy Research o Assessing effectiveness of seven programs • Seventeen schools recruited and randomly assigned in Chicago, focusing on schools serving a high-risk teen population with weak or no existing/planned use of HealthTeacher in the 7th and 8th grades o Nine schools delivered intervention in winter and spring 2011 to 7th-grade students o Eight schools assigned to control status (programming as usual) • No similar programming available or provided to 7th-grade students at program or control schools • Ten teachers in nine program schools trained to deliver HealthTeacher and given technical assistance • Twelve lessons delivered at program schools over approximately 16 class periods o Nine lessons from online HealthTeacher curriculum: Recognizing Respect, Changing Minds, Changing Bodies, Menstruation and Sperm Production, Looking to My Future, Looking at Barriers, Abstinence, It’s Okay to Say No, Preventing STDs/HIV o Three lessons developed by CPS and University of Chicago staff: Contraceptives, Sexuality, and Gender • Two lessons monitored at each school by CPS staff • Program impacts measured by two follow-up surveys in fall 2011 and spring 2012 1 The Stakeholders Collaboration and Communities in Schools match schools with CPS-approved health education providers to meet their needs. The CDPH’s Adolescent Health Program provides CPS with health education and testing for HIV/STIs. 1 Implementing an Enhanced Version of HealthTeacher in Chicago Motivated by this perspective, CPS agreed to participate in the Evaluation of Adolescent Pregnancy Prevention Approaches (PPA), a national evaluation to study the effectiveness of various teen pregnancy approaches in seven sites. The study is designed to provide rigorous evidence about program impacts, document program implementation, and generate insights about the successes and challenges of program delivery. The evaluation in Chicago focuses on the implementation and impact of an enhanced version of the HealthTeacher curriculum for 7th grade students in nine schools in winter and spring 2011 (10 teachers delivered the curriculum). The evaluation tested nine lessons from the family health and sexuality module of HealthTeacher for middle school youth, supplemented by three lessons developed by CPS with support from the University of Chicago. The evaluation will test whether the HealthTeacher curriculum is effective at delaying sexual activity and/or reducing risky sexual behavior among youth who are sexually active (Figure 1 presents a summary of program components, the targeted mediating factors, and intermediate and primary outcomes). Figure 1. Logic Model of the HealthTeacher Intervention Program Components Mediating Factors Interim Goals Long- Term Goals  12 classroom-based  Knowledge about:  Increased prevalence of  Reduction in the lessons of 45 to 90 -Reproductive health abstinence among incidence of teen minutes each -Physical, emotional sexually active teens pregnancy  Group work and and social risks of teen discussions sexual behavior  Decreased number of  Reduction in the -Contraceptive methods sexual partners incidence of STDs  Videos and visual aids -STD transmission and prevention  Decreased frequency of Enrichment activities: -Access to community unprotected sex  Varies by teacher resources  Refusal skills  More open communication about sex  Improved peer relationships HealthTeacher was implemented, and the evaluation conducted, in schools serving predominantly disadvantaged youth. More than 90 percent of students in the program schools in the 2010–2011 academic year qualified for free or reduced-price lunches. Baseline survey data indicated that three-quarters of the students were Hispanic, and 5 percent were white, non- Hispanic. Before experiencing HealthTeacher, most of the students (between 72 and 90 percent) did not have accurate knowledge of contraceptive methods and the risk of pregnancy and STIs. However, prior to the intervention, few of the 7th grade students (7 percent) reported being sexually active. Overall levels of risk behavior were also moderate. For example, about a third (34 percent) of students reported they had ever had an alcoholic beverage, while approximately 15 percent reported ever using tobacco or drugs. (Additional details on the students served by the intervention are in Appendix A.) To document and assess implementation of the enhanced HealthTeacher curriculum, Mathematica first focused on understanding the planned implementation, and then detailed the level of adherence to that plan across the study schools. The implementation analysis addressed the following research questions: 2 Implementing an Enhanced Version of HealthTeacher in Chicago • How was the HealthTeacher curriculum implemented in Chicago? • Can a large urban school district achieve consistent implementation of prescribed curriculum materials for middle school youth using an online resource such as HealthTeacher? • How did students respond to the curriculum? Did they participate in class? Were they engaged by the material? • What successes and challenges emerged from the implementation of the enhanced HealthTeacher curriculum? Did HealthTeacher present a strong enough contrast to existing programs to help inform the evaluation’s impact analysis? • What lessons learned from this study may be relevant for future program implementation and replication efforts? To address these questions, Mathematica conducted several types of data collection. A two- person research team conducted two site visits, including face-to-face interviews with CPS staff and other stakeholders, focus group discussions with teachers (from intervention and control schools) and students (from intervention schools), and classroom observations. The team also held telephone interviews with stakeholders and analyzed written feedback from teachers at intervention schools and CPS monitoring forms (details on data sources and methodology for the implementation study are provided in Appendix B). The implementation analysis conducted Why Was Implementing an Enhanced by Mathematica included multiple Version of HealthTeacher in CPS of Interest components. The research team used for the PPA Evaluation? qualitative analysis software to conduct descriptive analyses of site visit, focus group, • School districts are often interested in using an off-the-shelf sex education curriculum and observation data. The team also used similar to HealthTeacher. implementation benchmarks (Appendix B) to assess adherence to the implementation plan. • HealthTeacher, which is highly structured This report presents findings concerning basic and provides online access to lesson plans and supplementary materials, is easy for program implementation (including adherence districts and teachers to implement. to the implementation plan, and student engagement and participation) and assessments of the success and challenges of implementation. The report also discusses how these insights will support and inform Mathematica’s assessment of program impacts 2 on (1) sexual risk outcomes, which include both measures of sexual behaviors and their consequences, most notably pregnancy; and (2) intermediate outcomes, which correspond to the mediating factors through which the program would most likely have an impact on behavior (see Figure 1 above). The report concludes with tentative lessons about implementing programs such as HealthTeacher. 2 To estimate program impacts, Mathematica conducted surveys of youth shortly before the start of program delivery, and approximately 9 and 14 months after the program start—or 6 and 12 months after the end of program delivery. 3 Implementing an Enhanced Version of HealthTeacher in Chicago IMPLEMENTATION OF HEALTHTEACHER The implementation of the enhanced HealthTeacher curriculum can be summarized with reference to three general questions: (1) What was the curriculum and how was it developed? (2) What was the format and schedule of program delivery? (3) How were teachers prepared and supported? CPS Enhanced HealthTeacher to Meet Its Needs HealthTeacher is an online health education curriculum composed of structured, scripted materials. 3 The middle school family health and sexuality module of the curriculum consists of nine lessons, 45–90 minutes in length, covering respectful behavior, adolescence, female and male reproductive anatomy, menstruation and sperm production, goal setting, barriers, abstinence, refusal skills, and STI/HIV prevention. The lesson plans include teacher-led discussions as well as student participation activities such as role plays, small group discussions, games, and exercises involving developing information materials on specific topics (such as pamphlets about menstruation and sperm production). For example, one activity is a card distribution game to help students understand STI/HIV risk and exposure. Another has students develop pamphlets about menstruation and sperm production. Each lesson requires a projector to display transparencies, and some lessons require art and poster supplies. Teachers use the HealthTeacher website to download and print their lessons plans and materials they will use in the classroom. CPS decided to enhance the curriculum to address gaps in HealthTeacher identified by the Illinois Campaign for Responsible Sex Education 4 in the areas of contraception, sexuality, and gender. CPS engaged staff from the University of Chicago Pediatric and Adolescent HIV Team, as well as district health teachers and Coordinated School Health staff, to What District Support Was Necessary to Implement develop three supplemental lessons in the Enhanced Version of HealthTeacher? these areas. The 90-minute contraception lesson involves small group activities to • Payment for a site license for HealthTeacher (approximately $90,000 per year, included access for learn about contraception methods. The teachers throughout CPS) two lessons on sexuality and gender • Initial two-day training for teachers at study schools, were adapted from the Dealing with conducted by CPS staff Difference curriculum and an associated • Two observations at each school to monitor fidelity, DVD. 5 During the sexuality and gender conducted by CPS staff lessons, each 45 minutes in length, students watch a scene from the DVD 3 The HealthTeacher curriculum is accessible at http://www.healthteacher.com/. 4 Illinois Campaign for Responsible Sex Education. “Curriculum Content Review: An in-depth look at sex education curricula in use in Illinois classrooms.” http://icah.org/sites/icah.org/files/docs/Sex%20Education% 20Curriculum%20Content%20Review%2007_0.pdf. Chicago: Illinois Caucus for Adolescent Health, 2007. The Illinois Campaign for Responsible Sex Education is a partnership of the Illinois Caucus for Adolescent Health, Planned Parenthood/Chicago Area, and the Illinois Planned Parenthood Council. 5 CPS provided each treatment school with a copy of the Dealing with Difference DVD, which, along with the curriculum, is available from Human Relations Media. 4 Implementing an Enhanced Version of HealthTeacher in Chicago and discuss what they learned. Relegent was not involved in developing the three additional lessons, but had no objections to their addition, as it views HealthTeacher as a supplemental curriculum and allows school districts to create a custom scope and sequence for their teachers. Mathematica defined some tools and rules for use of the enhanced HealthTeacher curriculum for the evaluation. Relegent created a special website, accessible only to teachers in the study schools assigned to implement the intervention, which included the nine core and three additional lessons. The website provided teacher guides and handouts for each lesson, and some lessons included links to supplementary materials approved by CPS for use in the intervention. To ensure that impact results would pertain to a consistently defined intervention, CPS and Mathematica agreed that teachers would be directed to use only materials available on or through the special HealthTeacher website. Appendix C provides a brief overview of the scope and sequence of the lessons in the enhanced HealthTeacher curriculum and lists the key elements of the curriculum available to teachers for each lesson. Implementation Schedules Were Set by Schools and Adapted by Teachers The study schools determined the program implementation schedule to fit their needs. CPS did not require teachers to implement the HealthTeacher lessons across a full 45-minute period; however, the teachers were expected to cover as many elements of each lesson plan as possible within the time allotted. Schools planned to implement the curriculum in different classes (reading and writing, science, enrichment, physical education) and on varying schedules between January and May 2011 (Table 1). The time allotted for implementation of the curriculum varied. Some schools allocated two weeks, while others planned to spread the lessons over three to four months. In seven of the nine treatment schools, the time lines planned in each school were sufficient to deliver the intended 12 hours of instruction. In two schools, plans appeared to allow for 9 to 11 hours of instruction because the time available for such instruction was limited. CPS did not expect this shortened length to have an impact on the overall implementation of the curriculum. Table 1. Planned Time Lines for Implementation of Enhanced HealthTeacher Curriculum Planned Number Planned Number of Minutes per Treatment School Class of Weeks Days per Week Class Period School A Reading and Writing 17 1–3 40 School B Science 6 3 80 School C Enrichment 9 2 40 School D Physical Education 3 5 35 School E Science 7 2 80 School F Enrichment 14 1 40 School G Science 4 5 40 School H Physical Education 17 1 40 School I Science 6 1–3 40 Source: CPS teachers implementing the curriculum, Mathematica site liaison. 5 Implementing an Enhanced Version of HealthTeacher in Chicago The schools’ varied time lines could be, and were, adapted further. Five of the schools adhered to or did not stray far from the original time line. Three of these five schools completed the lessons in the planned time line (Table 2); of these schools, one (D) found it necessary to divide the 7th grade students into two groups. A fourth school (A) completed the lesson in a shorter time frame (three as compared to four months), and, at a fifth school (C), the curriculum lasted one week longer than planned. The remaining four schools experienced more substantial delays. Two schools (G and I) started program implementation late. At School I, actual program implementation time was comparable to the original plan (6 weeks), while at School G it took almost double the time because the class was able to meet only every other week, and not every week as planned. Schools B and H, which began implementation close to the planned start date, experienced more substantial delays and finished the curriculum approximately one month later than planned. The length of the class period directly affected the duration of program delivery. In most schools, teachers reported that the constraint of 35–40 minute class periods forced deviations from the planned time lines. When students raised questions during lessons, it became more difficult to complete lessons in the allotted periods; the questions extended the overall program delivery time. The teachers in the two schools with 80-minute class periods were able to fit an entire lesson into one class period, but other teachers had trouble doing so. It appeared that the large number of student questions that arose during class periods regarding menstruation and sperm production, sex, pregnancy, contraception, and gender and sexuality (as reported by teachers in written feedback and focus group discussions) often made it difficult to fit an entire lesson into one class period. School-wide events also affected program delivery schedules. The May administration of the Illinois Standardized Achievement Test, school assemblies and holidays, and a blizzard (which led to the closing of schools for two days in January 2011) caused additional delays. Table 2. Time Line for Program Delivery Planned Actual Treatment Duration Duration Program Program Number of Days per School (Weeks) (Weeks) Start End Week School A 17 11 As Planned Early 1–3 School B 6 13 Early Late 3 School C 9 10 As Planned Late 1–2 School D 3 3 As Planned As Planned 5 School E 7 7 As Planned As Planned 2 School F 14 14 As Planned As Planned 1 School G 4 7 Late Late 2 (every other week) School H 17 21 As Planned Late 1 School I 6 6 Late Late 2–3 Source: CPS teachers implementing the curriculum, Mathematica site liaison. 6 Implementing an Enhanced Version of HealthTeacher in Chicago CPS Training Was Generally Valued by Teachers In December 2010, the 10 teachers at the study schools participated in two days (approximately 12 hours) of required training. The training was adapted from the annual three- day training on sex education materials usually conducted by the CPS specialist, although that regular training is not required of CPS teachers. The training focused on helping teachers become comfortable talking about the subject matter. The first day provided an overview of CPS policy on family life and AIDS education, a discussion of statistics and figures relating to sexual behaviors, and a discussion of condom use. The Director of Clinical Operations and Outreach for the University of Chicago Pediatric and Adolescent HIV Team also provided detailed information on HIV/AIDS. The second day of training covered the enhanced HealthTeacher curriculum, with a focus on the lessons on puberty, contraception, and HIV/STIs. The CPS specialist also suggested strategies for teaching the curriculum, and addressed teachers’ comfort level with both teaching the subject matter and using the HealthTeacher website to download lessons. After the training, teachers had access to technical assistance. The CPS specialist who provided training and the director of the University of Chicago team were available to address questions on the curriculum content and assist teachers. A Mathematica liaison and staff from Relegent, the curriculum developer, were available to help teachers access the HealthTeacher website. CPS monitored delivery of lessons through classroom observations and teacher feedback. A CPS specialist and an intern planned to (and did, with one exception) observe two lessons conducted at each of the schools and collect data on curriculum delivery, adherence to the lesson plan, and creation of a comfortable classroom environment. Teachers were expected to use a standardized form to submit written feedback for five of the lessons, documenting: (1) whether they covered all of the topics and activities in the lesson plans and, if not, why not; (2) supplemental materials they used and why; (3) and changes they would suggest to the lesson plans. Teachers found the training useful. The eight teachers who took part in focus group discussions reported that they found the training sufficient and helpful, particularly four teachers who had no prior experience teaching sexual health. Six teachers observed in CPS monitoring felt the training was sufficient for teaching the observed lesson. While a few teachers expressed discomfort with talking about some of the subject matter (for instance, male and female anatomy), the CPS specialist conducting the training did not sense any such discomfort. Most teachers appeared interested in the training. Most of them seemed engaged, according to the CPS trainer and a liaison from Mathematica who observed the training. The director of the University of Chicago team reported that teachers appeared interested in the materials and seemed to understand what they were taught. Teachers appeared most attentive when asked to participate actively, and less attentive when discussions were longer than 20 minutes. A few distracted teachers used cell phones throughout the training. Several teachers felt the training could have been improved, but in somewhat conflicting directions. One felt the training was “minimalist”—too focused on how to use HealthTeacher— and would have preferred more time to explore the lessons and ask questions. She did not recall explicit instructions to use only HealthTeacher and supplemental materials referred to in each 7 Implementing an Enhanced Version of HealthTeacher in Chicago lesson plan as the only teaching resources. Another teacher said the training did not prepare her fully for teaching about contraceptives, but otherwise found the lesson plans detailed and easy to follow. Two other teachers thought the training could have been limited to one day because of the self-explanatory and comprehensive nature of the curriculum; they felt the second day was not helpful. Using the website tools to retrieve curriculum materials posed a problem for several teachers. They were uncomfortable initially using computers and navigating the website to download materials. During the training, several teachers had trouble creating a new account on the HealthTeacher website, but the CPS trainer helped resolve that problem. Toward the end of the intervention period, all teachers reported that they were able to access the website and download lesson plans. However, for any program implementation that relies on a web-based curriculum, this experience underscores the importance of providing basic guidance on how to find and download necessary materials. 8 Implementing an Enhanced Version of HealthTeacher in Chicago ADHERENCE TO THE IMPLEMENTATION PLAN The major question CPS confronted in its HealthTeacher implementation plan was whether it could increase the consistency of sex education instruction. Across CPS, HealthTeacher had been available to teachers and the stated policy required schools to use it or other options. However, CPS had provided only brief training, which was tailored to its family life and AIDS education policy and included only a short section on how to access HealthTeacher. CPS had not provided any clear prescription of what material to teach or conducted systematic monitoring to assess curriculum delivery. The implementation plan developed as part of CPS’s entry into the PPA evaluation addressed these past issues. It included prescribed curriculum modules and clear instructions about delivering them. CPS personnel put in place two processes: (1) one for teachers to report on completion of five of the lessons and their component activities, and (2) one for observing all of the teachers conducting at least two lessons from the HealthTeacher curriculum. By choosing HealthTeacher for this effort, CPS was in effect deciding to retain a curriculum that promised to give teachers all they needed to conduct their classes. The evaluation team examined how teachers received the curriculum, how fully they Was the Enhanced Version of HealthTeacher relied on it, and what factors might have Implemented with Fidelity? contributed to departures in classroom delivery from the highly specific guidance the curriculum • HealthTeacher was implemented by CPS with high fidelity; all teachers implemented provided. Upon assessing these issues and all HealthTeacher lessons in the correct implementation benchmarks (defined in order, in the time allotted, and as Appendix B), the evaluation team determined prescribed. Six teachers reported making minor modifications to the lesson plans. that HealthTeacher was implemented in Chicago with high fidelity. Teachers taught all of the lessons and delivered them as prescribed; six teachers made very minor modifications. The detailed lesson plans and materials were geared toward teachers without experience, and provided welcome support. One of the more inexperienced teachers said, “I was [more prepared to teach] because the lesson plans were so detailed, it just helped me, I thought.” Two of the more experienced teachers felt that the curriculum could help teachers without prior experience or comfort with the topic; one said, “You have to be comfortable with the topic to begin with because there’s a lot of stuff in here, that if you’re not comfortable with, it could be an uncomfortable situation. The HealthTeacher curriculum did help because it is broken down for you; how you’re going to do things step by step … I think this would help if you had any kind of anxiety about it.” For the most part, teachers delivered lessons as prescribed. Observed lessons were carried out as defined in the curriculum materials. However, six teachers reported that, for some lessons that were not observed, they deviated from the lessons plans for several reasons. Some cited time constraints that prevented them from completing all lesson components, sometimes because they spent more time than planned on some topics answering students’ questions. Three teachers added external materials (diagrams) to help teach a concept (menstruation and sperm production). Three teachers noted that they were unable to obtain art and poster supplies required for some activities. 9 Implementing an Enhanced Version of HealthTeacher in Chicago Some experienced teachers felt confined by the instruction to stick to the curriculum materials, and a few made minor modifications. Two experienced teachers expressed frustration with the materials, which they felt were too prescribed and limiting; they wanted to augment the lessons with other materials they had used in the past. Most teachers understood that, to ensure that the evaluation tested a consistent intervention, they were not to draw creatively on outside materials, and CPS monitors did not observe teachers supplementing lessons with external materials. However, some teachers reported to the evaluation team that in other lessons they used materials that they found better or clearer in some way. Two teachers substituted menstruation and sperm production diagrams from other websites, and a third teacher substituted a textbook diagram to explain menstruation and sperm production. One teacher said, “For the menstruation [lesson], I went through the oldest science book I’ve ever seen because they had this old transparency which was the best thing I ever saw to demonstrate the menstrual cycle…it was perfect.” Another teacher felt that the sperm production diagram provided by HealthTeacher was too “generic,” and substituted a diagram from a website that provided detailed information about elements of sperm production. From the perspective of these teachers, the insistence on consistency may have gone too far. In the same spirit, a few teachers modified some lesson plans. One teacher who had rearranged the scope and sequence of lesson plans said, “I had to cut and paste all the time, and do my own thing.” Mathematica researchers also observed a change in lesson format by one teacher who said she adapted the observed contraception lesson to make it more “kid friendly” and fit it into one class period. Instead of creating “stations” with information about different contraceptive methods, the teacher assigned groups of students to learn about one contraceptive method, write up a description of the method (including statistics on effectiveness and use) on poster-sized paper, and then report to the class. Most teachers were comfortable with HealthTeacher and able to handle Did Teachers Need Any Particular Background or Experience to Implement HealthTeacher? students’ questions. All students interviewed by Mathematica looked forward • Because of the structured nature of the to attending the class in which the enhanced HealthTeacher curriculum, it was not important for teachers to be certified health teachers, have HealthTeacher curriculum was taught, and prior experience teaching sex education, or all but one said that their teachers appeared dedicate much time before class to preparing comfortable teaching the material. Students material. appreciated their teachers’ openness about • CPS teachers who implemented HealthTeacher the subject matter; one said the teacher was had a variety of backgrounds (physical education, science, English, guidance “really open and wasn’t hesitant about any counseling) and experience with teaching sex of the things we talked about and it kind of education. helped that the teacher wasn’t hesitant about • Although teachers’ levels of comfort with the anything because it helped the students be material varied, all of them accessed and more open.” Another student said her completed the HealthTeacher curriculum, with only a few reporting minor modifications. teacher described her own experience related to the topics covered in class and “didn’t hesitate about anything. She’s not shy about anything. She can answer anything you want if it’s not inappropriate.” During classroom observations, CPS monitors reported that only a few teachers appeared to struggle with answering students’ questions. One monitor reported only one occasion when she felt compelled to help answer students’ questions, when the teacher asked the monitor to talk to a student 10 Implementing an Enhanced Version of HealthTeacher in Chicago privately to answer questions (related to STIs) that were beyond the scope of the teacher’s knowledge. Teachers’ comfort teaching program material hinged on their prior experience. HealthTeacher includes some activities such as demonstrating use of a condom that made some teachers uncomfortable, even though CPS did not require teachers to include these activities. Two inexperienced teachers reported that they were not comfortable demonstrating how to use a condom. One teacher, teaching sex education for the first time, told a CPS monitor that she was very uncomfortable with the material, and this discomfort was evident to the CPS monitor when she observed this teacher in the classroom. The teacher had a hard time getting the students to settle down and do their work; the students did not appear to be paying attention to the teacher and did not ask many questions during the observed session. In a focus group discussion, this teacher expressed her discomfort with some of the more explicit material: “Some [lessons], like contraceptives, I was so glad the students had to do it on their own—they have to find out what’s going on because I was not really comfortable. . . I’m not going to do any demonstrations. These are out of my comfort zone.” One student in a focus group discussion noted he was aware of his teacher’s inexperience with the subject matter, and would have preferred being taught by an expert. 11 Implementing an Enhanced Version of HealthTeacher in Chicago STUDENT PARTICIPATION AND ENGAGEMENT If a program is to affect behavior, the intended participants must be present and open to the ideas on which the program engages them. Part of the implementation analysis of HealthTeacher therefore focused on documenting attendance. In addition, classroom observations assessed informally students’ engagement in the classes as well as their apparent and reported interest in the material presented. Most students attended most lessons. Although student attendance data for specific class periods were not available, attendance at school provides a good proxy measure, since in the middle schools there is little if any skipping of individual class periods. School attendance data show that average daily attendance rates in the treatment schools during the period of program implementation were high, ranging from 94 to 98 percent (Table 3). These high attendance rates suggest that students were seldom absent from classes in which the enhanced HealthTeacher curriculum was taught. Table 3. Average Daily Attendance Rates During the Period of Program Implementation Treatment School Average Daily Attendance Rate School A 94.8 School B 97.8 School C 96.0 School D 95.2 School E 95.2 School F 95.9 School G 96.2 School H 95.8 School I 93.9 Source: Chicago Public Schools. Note: HealthTeacher was implemented at treatment schools on varying schedules between January and June 2011. These data are applicable to all students in the treatment schools; they are not limited to the 7th grade students enrolled in the intervention. Students appeared engaged in and receptive to the material, and had good interactions with their teachers. Both monitors and teachers reported substantial conversation. One CPS monitor observed students expressing their thoughts and asking good questions in class. One of the more experienced teachers reported that she had good discussions with her students (and was happy to find that they were not yet ready to have sex). Another experienced teacher reported that her students were well-behaved and that she had no problems teaching the material. A third teacher said the students responded better to the lessons than she thought they would, and described the HealthTeacher curriculum as “very good and kid friendly.” Students were comfortable asking questions and reported that they could ask questions about a variety of topics. One student commented, “We can really ask questions about ourselves and the opposite sex or whatever. [The teacher] was really open and it was a fun class.” 12 Implementing an Enhanced Version of HealthTeacher in Chicago In all schools, teachers encouraged open student expression by enabling students to ask questions anonymously. According to one CPS monitor, students were not required to ask questions orally, but could instead write questions down and submit them anonymously to the teacher if they wished. This appeared to be true in all schools. Nine students at one school, as well as several students at four other schools, said they were not ashamed of asking questions because they could do so anonymously. One student said of this process, “There were some questions that [the teacher] didn’t know [how to answer] and she found out and told us the next day. But what she did know she told us and that made us feel better. We had somebody to tell us what’s wrong and what’s right and how we get things and how we do things.” Clearly, however, some students were not fully comfortable with all of the HealthTeacher material. Several students were uncomfortable discussing contraception and menstruation/sperm production. One student reported that she was overwhelmed by the amount of information about contraception, presented as students went to different stations to learn about each Was CPS Able to Reach All Targeted Students type of contraception. She also found it and Retain Them Throughout Program awkward for the boys and girls in the class Implementation? to go around to the stations, some of • Implementing HealthTeacher during the school which had contraceptives used by females day ensured good reach and high retention of and others by males; she said, “it was students. uncomfortable to talk about it with the • HealthTeacher was implemented across 7th guys there. We couldn’t ask as many grade in all participating schools; only students without parental consent did not participate. questions as we wanted.” The menstruation and sperm production lesson • Attendance rates in the schools implementing HealthTeacher were very high, suggesting also appeared to cause a bit of discomfort students seldom missed HealthTeacher lessons. among students. One of the teachers • Students appeared to be engaged in the lessons reported that the girls in her class were and receptive to the material. initially “horrified” that she was discussing the menstrual cycle with them, but acclimated to the class over time, and asked numerous questions on the subject. One student said she was uncomfortable learning and talking about menstruation and sperm production, which she found “kind of gross,” but was eager to learn about all of the other topics. Some students may simply have been too immature to participate in a class like HealthTeacher. Several teachers reported that, when sex-related topics (including menstruation) were discussed, some students tended to giggle and whisper more. One teacher speculated that some students were not ready, as 7th graders, for this kind of material. He and several other teachers reported sending disruptive students out of class a few times. One student felt that teachers should have more proactively removed immature students from class, with “a separate class for people who take [the sex education class] seriously and a separate class for people who don’t take [the class] seriously.” Disruptions did not appear to be limited to girls or boys; teachers noticed disruptive behavior in both groups. 13 Implementing an Enhanced Version of HealthTeacher in Chicago IMPLEMENTATION SUCCESSES, CHALLENGES, AND INSIGHTS FOR THE IMPACT STUDY The implementation evaluation shows that a large urban school district can consistently implement an online sex education curriculum, with some inevitable lapses or deficiencies. District staff provided training that teachers found useful, teachers completed the curriculum with their students, and students engaged in learning the material. However, teachers faced challenges in implementing the curriculum: insufficient time to complete all lessons, some difficulty maintaining continuity between lessons, and unclear directions for some lesson activities. More experienced teachers expressed frustration with restrictions on incorporating materials from outside the curriculum, while less experienced teachers would have liked more direction. How Well Did the Enhanced Version of HealthTeacher Fit with CPS Policies and Climate? • It was a good fit with CPS policies. o It provided a way to meet the district’s sex education mandate. o It could be implemented without a lot of district support. • It was well-implemented despite changes at CPS. o In January 2011, the staff person who led development of the implementation plan left CPS and was replaced by a CPS specialist who oversaw implementation and monitoring. o In 2011, CPS department goals and funding were redeveloped and realigned, with diminished interest in continuing funding for sex education programs. Successes Teachers found the curriculum easy to use. The teachers who delivered the enhanced HealthTeacher curriculum generally liked the lessons, the topics they covered, and their sequence. One teacher said that the lesson plans kept her on track and focused. She contrasted the modest preparation time required with the amount of time it normally takes her to gather material for a class without a prescribed curriculum. Teachers felt the curriculum helped them address students’ questions and misconceptions. One teacher liked the role-play activities associated with the lesson on refusal skills because it gave her students ideas about how to get out of difficult situations. She liked the lesson on goal- setting because it “snapped everything in focus” for the students by pointing out the negative consequences of teen pregnancy or STIs. Three teachers felt the contraception lesson was very effective and helped to clear up students’ misconceptions about the effectiveness of various methods of contraception, such as Plan B and condoms. Another teacher was enthusiastic about the DVD 6 that accompanied the sexuality and gender lessons because it helped answer students’ questions about lesbian, gay, bisexual, and transsexual (LGBT) issues. 6 This DVD was not from the HealthTeacher curriculum. It was from a separate curriculum, Dealing with Difference, and was included in two lessons on gender and sexuality that were added to the HealthTeacher curriculum by CPS staff. 14 Implementing an Enhanced Version of HealthTeacher in Chicago The curriculum filled a programming gap in the study schools. Four schools had not previously provided sex education in 7th grade, and the enhanced HealthTeacher curriculum filled this gap. All students in the focus groups felt that sex education was valuable, and that it should continue to be taught. Teachers noted the value of teaching sex education, especially because students were not comfortable talking about sex education topics with their parents and appeared to be comfortable in the classroom, where they could speak freely with their teachers about these topics. One teacher said, “I don’t think they can talk…about [sex] with their parents, or their parents are…not open-minded…So, [students’] being able to express themselves with me and ask me questions and stuff, and…being able to speak freely, I think that helped them, absolutely.” Teachers said they would like to continue to use the curriculum—either as presented or with adaptations. Two of the more experienced teachers said they might adapt the curriculum with supplementary materials or field trips (for example, a class trip to a local health center that provides health education). The CPS specialist in charge of monitoring the curriculum implementation hoped that teachers who implemented the enhanced HealthTeacher curriculum could be used as advocates to influence other teachers or schools to use the curriculum. Challenges Teachers reported difficulty completing some lessons in the time allotted. Seven teachers (from schools with 35- or 40-minute class periods) reported that they needed more time to cover most lessons fully. Occasionally, time ran short because of students’ questions. One teacher reported that as a result she extended a 40-minute lesson to two or three 40-minute class periods. Another teacher wished that the class had met every day, for more than 40 minutes. The teacher’s class met once a week; he felt that the students would have paid more attention and learned more if they had met more often. Several teachers felt a lack of context and connection between lessons. CPS monitors reported that several teachers felt lessons were not in the ideal order, and sometimes did not provide the background and context to help them move from one point to the next. A few teachers also felt the lessons would have been better taught in a different order—for example, teaching students about sex and contraception before teaching them about making good decisions, so that students fully comprehend why they should make specific choices. One teacher reported changing the order of lesson exercises to provide a better flow for the lessons. Teachers found some lessons confusing and some directions unclear. Eight teachers found some diagrams unclear, and three substituted diagrams from other sources. While four teachers liked having students play a game in the lesson covering STI/HIV prevention, they were confused by the game’s directions. Two teachers found the contraception lesson too complex and the time allowed for students to complete their activity insufficient. 7 Three teachers reported that the birth control comparison chart presented too much information in a small format. 7 This lesson was created by University of Chicago and CPS staff. 15 Implementing an Enhanced Version of HealthTeacher in Chicago Inexperienced teachers sometimes felt they needed more direction. Two teachers felt that they would have benefited from having an “expert” (such as a nurse) in the classroom when they covered contraception because they lacked adequate knowledge on the topic. They expressed discomfort with demonstrating how to use a condom. One CPS monitor noticed teacher discomfort with some lessons, and said, “Teachers who hadn’t used [HealthTeacher] before and whose first exposure to sex ed[ucation] teaching was at the training had a tougher time.” Some more experienced teachers felt hindered by instructions to stick to the HealthTeacher materials. One CPS monitor felt that the more experienced teachers she observed were “struggling” to get the most out of the materials without supplementing the curriculum with materials they were accustomed to using when teaching sex education. She said, “The teachers [who] told me they have the most trouble were really dynamic teachers who I could see had rapport with the students and who had taught sex ed[ucation] before.” Some teachers were hindered by a lack of resources and/or technology. Although the HealthTeacher curriculum is available online, teachers still had to print out all of the materials (including lesson plans, transparencies, and handouts) and, according to CPS monitors, several teachers did not have access to printers or paper. One of the CPS monitors reported that one teacher without access to the internet during a lesson (a requirement of that particular lesson) had to resort to using a textbook. The CPS monitor also reported that another teacher did not have access to an LCD projector, and so was unable to show transparencies during class. Use of HealthTeacher requires yearly renewal of and payment (approximately $90,000 per year) for a site license. For financial reasons, CPS has not renewed its site license agreements with Relegent for use of HealthTeacher, so if the curriculum is used in the future, it would be because the teachers have hard copies of the lessons; they have lost access to the HealthTeacher website. Insights Relevant for Interpreting HealthTeacher Impacts Despite the aforementioned challenges, the implementation of the enhanced HealthTeacher curriculum in nine schools in Chicago provides a reasonable test of the curriculum. Across all schools, teachers were able to consistently implement the curriculum; they generally adhered to the planned scope and sequence of lessons and appeared to present the lessons as directed in the curriculum. Short class periods and unexpected student questions led some teachers to teach some lessons over two or three class periods, but they were still largely able to cover all of the exercises in the lessons. Unplanned adaptations appeared to be limited to substitution of diagrams (for menstruation and sperm production) and some rearrangement of activities. Most students attended the majority of lessons, with the exception of some disruptive students, whom teachers removed from class. The enhanced HealthTeacher curriculum provided in treatment schools created a clear contrast to what was available in control schools. During the 2010–2011 academic year (the year in which the evaluation took place), teachers at control schools reported that none of the schools in the control group provided sex education to their 7th grade students (either during or after school), and treatment teachers and students reported that 7th grade students did not receive any sex education programming in addition to HealthTeacher. Teachers and students reported that several assemblies were offered at the schools enrolled in the study on potentially related topics, including child abuse prevention and STIs. However, none of the schools received sex education 16 Implementing an Enhanced Version of HealthTeacher in Chicago programming from external organizations. Moreover, teachers and students were not aware of any after-school or community programs covering sex education or related topics, indicating that the availability of services or programs for middle school students in the community is limited. (In the 2009–2010 academic year, schools recruited into the study either did not teach 7th grade students sex education or spent 5 days or fewer covering sex education-related topics, such as anatomy and STDs.) In this study, then, the estimated impacts of the enhanced HealthTeacher curriculum will reflect the effects of implementing the enhanced HealthTeacher curriculum with 7th grade students, largely as planned, relative to not providing a sex education curriculum for 7th grade students. 17 Implementing an Enhanced Version of HealthTeacher in Chicago LOOKING FORWARD: LESSONS TO INFORM FUTURE IMPLEMENTATION EFFORTS The implementation of this enhanced version of the middle school family health and sexuality module of the HealthTeacher curriculum in Chicago has implications for other school districts. Other districts may be interested in implementing a similar version of HealthTeacher, or using other curricula of similar focus, intensity, and delivery approach. Lessons for future efforts include the following: • Schedules should allow ample time for students’ questions. The subjects discussed in the HealthTeacher curriculum are likely to elicit numerous student questions. Lessons originally scheduled to take 45 minutes might be better spread over two class periods, and lessons scheduled for 90 minutes might be better spread over three 45- minute class periods. • Teachers should consider implementing lessons on a daily basis, or at least more frequently than once a week, to promote continuity between lessons. Teaching lessons between two and five times a week would keep the materials fresh in the students’ minds, keep students focused, and foster more dynamic discussions during class. • A balance can be struck between striving for consistency and allowing teachers to exercise creativity. Teachers would benefit from expanded opportunities to supplement the curriculum. Teachers value the ability to use materials that they feel best demonstrate concepts; they do not like to be limited to one set of materials when they have used what they consider to be “better” materials in the past. For example, supplementing the menstruation and sperm production lesson with clearer diagrams of male and female anatomy might help the teachers better explain concepts to students. However, since an important motivating factor for CPS and possibly other districts is to establish some consistency in curriculum and delivery, clear guidelines would be useful. • Staff overseeing implementation of the curriculum in schools should provide more support to new teachers, who may not be familiar or comfortable with the content area. For these teachers, discussions about such topics as male and female anatomy, contraception, HIV/STIs, and risky sexual behaviors can prove quite daunting. These teachers would benefit from regular check-ins from supervisory staff to ensure that they remain comfortable with and able to deliver the material confidently to students in class. 18 APPENDIX A POPULATION SERVED AND SAMPLE CHARACTERISTICS Implementing an Enhanced Version of HealthTeacher in Chicago The tables in this appendix provide data on population served and sample characteristics. The characteristics of the control group will be documented in the short-term impact report. Data on the fiscal year (FY) 2011 National School Lunch Program qualification (for all students served in the nine treatment schools) are from the Illinois State Board of Education. Characteristics for students in the study sample were collected through a baseline instrument administered to students in treatment schools between November 2010 and January 2011. Table A.1. National School Lunch Program Qualification Status of Students in Treatment Schools, FY 2011 Percentage of Number of Students Students Qualifying Number of Students Qualifying for for Free or Treatment Qualifying for Free Reduced-Price Reduced-Price Schools Student Enrollment Lunch Lunch Lunch School A 1,231 1,068 88 93.9 School B 1,291 1,201 64 97.9 School C 1,460 1,342 70 96.7 School D 1,294 1,060 125 91.6 School E 424 286 62 82.1 School F 1,146 1,044 52 95.6 School G 893 596 95 77.4 School H 1,355 1,055 142 88.3 School I 1,232 1,179 32 98.3 Source: Illinois State Board of Education. Note: These data are applicable to all students in the treatment schools; they are not limited to the 7th grade students enrolled in the intervention. 20 Implementing an Enhanced Version of HealthTeacher in Chicago Table A.2. Sample Characteristics Percentage of Treatment Group Students (unless noted) Demographic and Background Characteristics Age in Years (mean) 12.42 Female 51.65 Language Spoken at Home English only 31.73 Spanish only 27.47 English and Spanish 34.34 a Other 5.07 Missing 1.37 Race/Ethnicity White Non-Hispanic 4.67 Black Non-Hispanic 12.77 Hispanic 74.31 Other (including multiple) 8.24 Lives with Both Biological Parents 85.89 Parents’ Employment Mother currently employed 71.36 Father currently employed 89.55 Relationship with Parents Feels very close to mother 70.56 Feels very close to father 58.42 Considers Religion Very Important in His/Her Life 41.62 Attends Religious Services/Activities at Least Once a Week 27.20 Levels of Risky Behavior Tobacco, Alcohol, and Drug Use Ever smoked a cigarette 15.68 Smoked in last 30 days 5.44 Ever had an alcoholic beverage 33.91 Had alcoholic beverage in last 30 days 15.62 Binge drinking in last 30 days 6.52 Ever used an illicit substance (including prescription drugs and inhalants) 16.67 Initiation of Sexual Activity Ever had sexual intercourse 5.70 Ever had oral sex 3.43 Ever had anal sex 2.85 Ever had any type of sexual activity (intercourse, oral sex, or anal sex) 6.99 Use of Condoms in Past 3 Months Had sexual intercourse without a condom 0.44 Had oral sex without a condom 0.86 21 Implementing an Enhanced Version of HealthTeacher in Chicago Table A.2 (continued) Percentage of Treatment Group Students (unless noted) Had anal sex without a condom 0.14 Had intercourse, oral sex, or anal sex without a condom 1.33 Frequency of Sexual Activity in Past 3 Months (Mean) Frequency of sexual intercourse 0.09 Frequency of oral sex 0.07 Frequency of anal sex 0.13 Number of Partners (mean) Number of sexual intercourse partners 0.18 Number of oral sex partners 0.08 Number of anal sex partners 0.07 Ever Been/Gotten Someone Pregnant 0.14 Perceived Peer Pressure to Engage in Sexual Intercourse Feels a lot of pressure 8.35 Feels any pressure 68.41 Parents’ Attitude About Child Having Sex and Having a Baby at this Time Mother disapproves of sex and baby 81.80 Father disapproves of sex and baby 81.64 Would Not Feel Upset if Got/Got Someone Pregnant at this Time 28.17 Knowledge and Previous Receipt of Sex Education Received Any Information About the Following Topics in the Last 12 Months Relationships, dating, marriage, or family life 77.33 Abstinence from sex 44.73 Methods of birth control 23.58 Sexually transmitted diseases 58.81 Talking to partner about sex or use of birth control 21.21 How to say no to sex 58.82 Behavioral Expectations Expects to have oral sex next year 9.08 Expects to have sexual intercourse next year 14.01 Expects to have sexual intercourse before marriage 38.45 Knowledge Related to Contraceptive Effectiveness and Risk of Pregnancy and HIV/STIs Condoms Decrease the Risk of Pregnancy Not at all 7.05 A little 31.58 A lot 27.60 Don’t know 27.40 Missing 6.37 Condoms Decrease the Risk of HIV/AIDS Not at all 8.90 A little 22.67 A lot 23.77 22 Implementing an Enhanced Version of HealthTeacher in Chicago Table A.2 (continued) Percentage of Treatment Group Students (unless noted) Don’t know 37.95 Missing 6.71 Birth Control Pills Decrease the Risk of Pregnancy Not at all 4.25 A little 27.40 A lot 24.59 Don’t know 36.23 Missing 7.53 Birth Control Pills Decrease the Risk of HIV/AIDS Not at all 16.71 A little 18.70 A lot 10.00 Don’t know 47.19 Missing 7.40 Birth Control Pills Decrease the Risk of Chlamydia or Gonorrhea Not at all 10.07 A little 14.66 A lot 7.88 Don’t know 59.79 Missing 7.60 Oral Sex Increases the Risk of STIs Yes 42.60 No 8.08 Don't know 43.29 Missing 6.03 b Sample Size 728 Source: Student surveys administered by study team, November 2010—January 2011. a Other languages spoken at home includes the following categories: (1) English, Spanish, and another language (unspecified); (2) English and Chinese; (3) English and another language (unspecified); and (4) another language (unspecified). b Indicates number of students who completed the baseline survey. The sample size for each variable differs due to item nonresponse. 23 APPENDIX B IMPLEMENTATION STUDY DATA SOURCES AND METHODOLOGY Implementing an Enhanced Version of HealthTeacher in Chicago Three data sources provided the information for this report: (1) site visits and telephone interviews, (2) CPS monitoring forms and written teacher feedback, and (3) survey and administrative data (a baseline survey administered by Mathematica and administrative records from the Illinois Board of Education and CPS). The remainder of this appendix provides information about our data collection activities and analyses of these data sources. Site Visits and Telephone Interviews A two-person research team conducted two site visits to Chicago, as well as telephone interviews with multiple stakeholders, to collect in-depth data on: (1) the planned intervention, (2) adherence to the planned intervention, (3) delivery of the teacher training and curriculum, (4) participants’ responsiveness to the curriculum, and (5) successes and challenges encountered during program implementation. During the site visits, which took place in January and May 2011, the researchers (1) conducted in-person interviews with CPS staff, University of Chicago staff, and a Mathematica liaison; and (2) observed four classrooms in which one of the lessons from the enhanced curriculum was being taught. In May 2011, researchers conducted focus group discussions with six teachers from six control schools (two focus groups of three teachers each), eight teachers from eight treatment schools (one focus group with three teachers, and a second with five teachers), and 28 students from five treatment schools (two focus groups with 9 students each, and one focus group with 10 students). Telephone interviews were held with staff from CPS, organizations providing health services to Chicago schools, and Relegent, the developer of HealthTeacher. Table B.1 details the sources for the data collected, the time period during which these data were collected, and topics covered. Analysis Approach. Qualitative analysis of the site visit and telephone interview data was an iterative process using thematic analysis and triangulation of data sources (Patton 2002; Ritchie and Spencer 2002). Because of the number of interviews and focus groups conducted, we used a qualitative analysis software package, Atlas.ti (Scientific Software Development 1997), to facilitate organizing and synthesizing the qualitative data. First, we developed a coding scheme for the study, organized according to key research questions. Within each question, we defined codes for key themes and subtopics we expected to cover in the interviews. Then, we applied the codes to passages in the interview and focus group notes. To ensure accurate and consistent coding, both members of the site visit team coded the notes independently, then met to reconcile any differences in coding. To address the research questions, we used the software to retrieve relevant passages and examined the patterns of responses across respondents and identified themes emerging from the responses. Monitoring Forms To determine whether teachers adhered to the planned time line and duration of lessons, followed the prescribed scope and sequence of lessons, and used approved materials, we analyzed CPS monitoring forms and written teacher feedback, which CPS provided to Mathematica. 25 Table B.1. Data Sources Adherence to Participants’ Challenges Planned Training Planned Responsive- and Data Source Number Date(s) Context Intervention and TA Intervention ness Successes In-Person Interviews CPS CSH specialists 2 Jan, May 2011 X X X X X X CPS CSH intern 1 Jan, May 2011 X X X X Staff at University of Chicago 1 Jan 2011 X X CPS teacher 1 Jan 2011 X Mathematica liaison 1 Jan, May 2011 X X X X X Telephone Interviews CPS CSH specialist 1 Jan 2011 X X Staff at community organizations 5 April 2011 X Relegent staff 1 Oct 2011 X Focus Group Discussions 26 Treatment group teachers 8 May 2011 X X X X X (2 groups) Treatment group students 28 May 2011 X X X (3 groups) Control group teachers 6 May 2011 X (2 groups) Classroom Observations Classroom observations 4 Jan, May 2011 X X Fidelity and Monitoring Forms Written teacher feedback 45 Jan –May 2011 X X CPS monitoring forms 9 Mar – May 2011 X X X CPS = Chicago Public Schools; CSH = Coordinated School Health; TA = technical assistance. Implementing an Enhanced Version of HealthTeacher in Chicago CPS Monitoring Forms. CPS staff planned to monitor two lessons for each teacher, and they reported that they succeeded in observing two sessions in all but one of the treatment schools (one school received only one monitoring visit), despite many logistical challenges. However, half of the monitoring forms were lost by CPS, and our analysis is based on a total of nine monitoring forms—for 7 of the 10 treatment teachers—with only one form available for 5 of the 7 teachers with monitoring forms. Written Teacher Feedback. Teachers were asked to provide written feedback for five lessons. Although all of the treatment teachers submitted written feedback, two teachers submitted feedback for four lessons and one submitted feedback for two lessons. Our analysis is based on 45 feedback reports. Analysis Approach. We established implementation fidelity benchmarks for the enhanced HealthTeacher curriculum based on the theory of change and available data from the monitoring and fidelity forms. Benchmarks included whether: (1) there were deviations from the planned duration of implementation and number of sessions (monitoring forms and written teacher feedback); (2) teachers created a comfortable classroom environment (monitoring forms); (3) teachers answered students’ questions accurately in class (monitoring forms); (4) lessons were delivered in the correct order, in the time allotted, and as prescribed—and, if not, why (monitoring forms and written teacher feedback); (5) teachers used approved supplementary or other materials (monitoring forms and written teacher feedback); and (6) teachers made substantive changes to the lesson plans (written teacher feedback). After CPS provided us with their monitoring forms and the written teacher feedback, we populated a spreadsheet with data on these benchmarks from each school, and tabulated total responses to each of the benchmark fields; responses from monitoring forms and written teacher feedback were tabulated separately. We defined “high fidelity” as implementing all 12 of the lessons contained in the enhanced version of the HealthTeacher curriculum, with only minor modifications (for example, using supplemental diagrams to help teach a concept). We defined “low fidelity” as implementing the lessons with major modifications (for example, omitting lessons, skipping most of the components in lessons, or using materials that changed the content or message of the lessons). Survey and Administrative Data Data on the population served by the intervention were gathered from several sources. The baseline instrument collected data on demographic and background characteristics, risk-taking behavior, previous receipt of sex education, and knowledge and attitudes toward sexual activity and contraceptive use of consented students. It was administered to consented students from the 17 treatment and control schools between November 2010 and January 2011; the data in this report are from the 728 students in treatment schools who completed the baseline survey. Data on FY 2011 National School Lunch Program qualification came from the Illinois State Board of Education 8, and data on student attendance during the period of program performance came from CPS. The data from these two sources are for all students in the treatment schools; they are not limited to the 7th grade students enrolled in the intervention. 8 Available at http://www.isbe.net/nutrition/htmls/eligibility_listings.htm. 27 Implementing an Enhanced Version of HealthTeacher in Chicago REFERENCES Patton, Michael Quinn. Qualitative Research and Evaluation Methods: Third Edition. Thousand Oaks, CA: Sage Publications, 2002. Ritchie, Jane, and Liz Spencer. “Qualitative Data Analysis for Applied Policy Research.” In The Qualitative Researcher’s Companion, edited by A. Michael Huberman and Matthew B. Miles. Thousand Oaks, CA: Sage Publications, 2002. Scientific Software Development. “Atlas.ti: Visual Qualitative Data Analysis Management Model Building in Education Research and Business.” Berlin, Germany: SSD, 1997. 28 APPENDIX C SUMMARY OF HEALTHTEACHER LESSONS Table C.1. Summary of HealthTeacher Lessons Student and Teacher Lesson Description Duration Materials Required Supplies Background Links 1. Recognizing Students use a prepared 45 Worksheets: Expressing Overhead projector None Respect story as a basis for identifying minutes Thoughts and Feelings; ways in which respect for self William Speaks; Checklist and others can be for Respect communicated. Students Handout: A Family Tale enact scripts that identify respectful behavior. Teacher Guide: Respect for Self and Others 2. Changing Students “vote with their feet” 45 Worksheet: What Influences Agree/Disagree sign HealthTeacher links: Minds to indicate whether or not minutes You Emotional and Social they agree with a list of Handout: Now Things are Changes statements about Different adolescence. Students Teacher Guide: It is Normal discuss some of the changes that occur during adolescence, and use a 30 handout to guide a discussion of what influences their feelings, beliefs, and behaviors. 3. Changing Students use valid 45 Worksheets: Changes; Transparencies, HealthTeacher links: Bodies information sources (e.g., minutes Males and Females overhead projector Identifying Valid Sources of from health care personnel, Visual Aids: Learning about Information community care agencies, Male Anatomy; Learning volunteer health agencies, about Female Anatomy government agencies) to Teacher Keys: Learning research male and female about the Physical Changes reproductive anatomy and of Adolescence physiology, and use this Females and Males; information to complete Learning Anatomy worksheets. Table C.1 (continued) Student and Teacher Lesson Description Duration Materials Required Supplies Background Links 4. Menstruation Students work in small 90 Worksheet: How Much Do Transparencies, HealthTeacher links: and Sperm groups to order a number of minutes You Know? overhead projector, Reproduction: Menstruation Production cards about menstrual cycle Visual Aids: The Menstrual scissors and Sperm Production and sperm production. Cycle; Groups then develop The Production of Sperm; pamphlets for other Order Me Around adolescents about the two topics. 5. Looking to My Students review the steps of 45 Worksheets: My Goals; Transparencies/over HealthTeacher links: Goal- Future the goal-setting process, minutes Tasha’s Goals -head projector setting practice setting goals based Visual Aids: Reach Your on a given situation, and then Goal; Ready, Set, Goal write their own goals for Teacher Guide: Setting specified time periods. Goals for Adolescence 6. Looking at Students set goals for their 45 Worksheets: When I’m 25; Scissors None Barriers future, discuss the influences minutes Take My Advice; This is on achieving their goals, and Your Choice 31 identify ways in which Teacher Guide: Influences unprotected sexual and Barriers intercourse would be a barrier to goal achievement. 7. Abstinence Students identify physical, 45 Worksheets: Why Marbles, two jars, art None emotional, and social reasons minutes Abstinence; Dear Teen and poster supplies, why abstinence is a good (or can Advisor access to the choice for adolescents. be Teacher Guide: Abstinence Centers for Disease Students role-play being a extended Control and teen advisor who answers to 90 Prevention letters from other adolescents minutes) publication Youth who are making a decision Risk Behavior about sexual abstinence. Surveillance-United States Table C.1 (continued) Student and Teacher Lesson Description Duration Materials Required Supplies Background Links 8. It’s Okay to Say Students identify some of the 90 Worksheet: How Would You Transparencies, HealthTeacher links: No adolescent health risks of minutes Refuse?; Saying No Role- overhead projector Refusals; Influences on sexual behavior and discuss Plays Decisions; Media Violence; ways to “say no” to Visual Aid: Saying No Sexual Behavior adolescent sex. Students Teacher Guide: Saying No Link to DHHS on resisting write examples of what they Role-Plays peer pressure: would say/do for at least 6 of http://store.samhsa.gov/home the 12 refusal strategies, and Additional links: work in groups to create role The Media Literacy Online plays that demonstrate Project effective use of refusal skills (http://interact.uoregon.edu/M applied to adolescent sex. ediaLit/mlr/home/index.html); Media Literacy: Internet (http://www.media- awareness.ca/english/) 9. Preventing Students play a card 90 Handouts: Sexually Names, addresses Identifying valid sources of STD/HIV distribution game to help minutes Transmitted Diseases; How and phone numbers information (click on the link, 32 them understand HIV/STD Would You Feel?; for local clinics that then click on “accessing risk and exposure. They also STD Facts provide information information” on the left play an STD concentration Games: Transmission about and treatment toolbar) game to compete in Impossible; of HIV/STD, 3 x 5 answering questions about STD Concentration Game colored index cards STDs. They then discuss the Teacher Guide: HIV/STD (four per student), 3 transmission and prevention Teacher Key: STD x 5 white index cards of HIV/STDs, as well as the Concentration Game (four per student), symptoms, consequences, poster board, and treatments for HIV/STDs. markers, Students also identify glue/staples, colored community resources for construction paper, help. scissors Table C.1 (continued) Student and Teacher Lesson Description Duration Materials Required Supplies Background Links 10. Contraceptives Students work in small 90 Worksheet: Contraception Folders for http://www.orthoevera.com; groups and discuss minutes Information – Lab Materials information packets, http://www.plannedparenthoo contraceptive methods. Visual Aids: Birth Control anatomical models d.org; Students also review a series Comparison Chart – (if available), access http://www.cancer.org; of handouts on health English/Spanish to the internet to http://www.americanwomenss activities on contraceptives browse and collect ervices.com; and personal health. information from http://www.cityofchicago.org websites listed in background links 11. Sexuality Students complete a survey 45 Worksheets: Talking the Cardstock, scissors, Gay, Lesbian and Straight on attitudes toward minutes Talk; Attitudes Towards DVD player, and TV Education Network differences, and score Difference (http://www.glsen.org); themselves to find out where DVD: Dealing with Gay-Straight Alliance Network they fall on the acceptance Difference DVD (Human (http://www.gsanetwork.org); spectrum. Students are given Relations Media) National Youth Advocacy a sexuality/gender Homework: What’s in a Coalition term/definition and are Name (http://www.nyacyouth.org); 33 matched with another Queer America student, and read the (http://www.queeramerica.co term/definition to the class. m); Students watch a scene from IYG National Hotline for LGBT the Dealing with Difference Youth (1-800-347-TEEN) DVD and are asked questions about the attitudes and behavior displayed in the scene (about joining a gay- straight alliance). Table C.1 (continued) Student and Teacher Lesson Description Duration Materials Required Supplies Background Links 12. Gender Students watch a scene from 45 Worksheets: Gender Talk; Popular magazines, Gay, Lesbian and Straight the Dealing with Difference minutes Undoing Institutionalized cardstock, scissors, Education Network DVD about attitudes and Heterosexism in schools DVD player, and TV (http://www.glsen.org); behaviors toward LGBT DVD: Dealing with Gay-Straight Alliance Network youth. Students cut out Difference DVD (Human (http://www.gsanetwork.org); phrases, words, and images Relations Media) National Youth Advocacy from magazines and tape Coalition them to the board under (http://www.nyacyouth.org); different categories (male, Queer America gender-neutral, female), and (http://www.queeramerica.co discuss messages and m); societal expectations related IYG National Hotline for LGBT to LGBT people. Youth (1-800-347-TEEN) Source: Chicago Public Schools. 34