Contract Number: Balancing Fidelity and Flexibility: HHSP23320082911YC Implementing the Gen.M Program Mathematica Reference Number: 06549.092 in Texas Submitted to: Office of Adolescent Health Final Report Department of Health & Human Services 1101 Wootton Parkway, Suite 700 Rockville, MD 20852 August 5, 2013 Project Officer: Amy Farb Submitted by: Rachel Shapiro Mathematica Policy Research P.O. Box 2393 Princeton, NJ 08543-2393 Telephone: (609) 799-3535 Facsimile: (609) 799-0005 Project Director: Brian Goesling This page has been left blank for double-sided copying. Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas CONTENTS INTRODUCTION ............................................................................................................. 1 FRAMEWORK FOR PROGRAM DELIVERY .................................................................. 5 Gen.M Links Gender Norms and Inequalities to Sexual Behavior Outcomes .. 5 Gen.M Has Three Interactive Components ...................................................... 6 Partners Helped EngenderHealth Recruit Youth and Deliver the Program ...... 8 FACILITATORS’ TRAINING AND PROGRAM DELIVERY ............................................. 9 Training Gave an Overview of the Curriculum and Hands-On Practice ........... 9 Training and Technical Assistance Emphasized Fidelity .................................. 9 Facilitators Were Not Fully Prepared to Maintain Fidelity ............................... 10 EngenderHealth Made Adjustments in Response to Feedback ..................... 10 These Adjustments Led to Better Program Delivery and Fidelity ................... 11 Facilitators’ Confidence and Program Delivery Skills Increased Over Time ... 11 PARTICIPANTS’ ENGAGEMENT AND UNDERSTANDING OF THE MATERIAL ....... 13 Attendance Among Participants Was High..................................................... 13 Participants Were Engaged, Active, and Valued ............................................ 13 Program Environment Encouraged Sharing of Opinions and Open Questioning.............................................................................................. 14 Core Messages Resonated with Participants ................................................. 14 Perceptions of Core Messages Differed Between Males and Females .......... 15 Age Differences Were Reflected in Responses to the Material ...................... 15 LOOKING FORWARD: LESSONS FOR FUTURE IMPLEMENTATION ....................... 17 REFERENCES .............................................................................................................. 19 APPENDIX A: ................................................................................................................ 21 APPENDIX B: ................................................................................................................ 27 APPENDIX C: ............................................................................................................... 33 APPENDIX D: ............................................................................................................... 41 APPENDIX E: ................................................................................................................ 51 iii This page has been left blank for double-sided copying. Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas TABLES A.1 Sample Characteristics .................................................................................. 23 B.1 Data Sources ................................................................................................. 30 C.1 Summary of Gen.M Lessons .......................................................................... 35 FIGURES 1 Logic Model of the Gen.M Intervention............................................................. 2 2 Travis County Youth Demographics ................................................................. 3 v This page has been left blank for double-sided copying. Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas INTRODUCTION In 2012, EngenderHealth, a nonprofit organization focused on improving sexual and reproductive health around the world, brought Gender Matters to Travis County, Texas. This sex education program, referred to more often as Gen.M, aims to reduce teen pregnancy by (1) challenging commonly held perceptions of gender roles and their association with sexual behaviors, (2) promoting healthy, equitable relationships; and (3) providing high-quality comprehensive sex education. EngenderHealth chose to offer the program in Travis County because the county’s teen birth rates are among the highest in the state (U.S. Census Bureau 2013). Gen.M is being implemented as part of the Evaluation of Adolescent Pregnancy Prevention Approaches (PPA). PPA is a national evaluation funded by the U.S. Department of Health and Human Services, Office of Adolescent Health, to study the effectiveness of various teen pregnancy prevention approaches in seven sites. The study is designed to provide rigorous evidence about program impacts, document program implementation, and generate insights about the successes and challenges of program delivery. The evaluation in Travis County focuses on the first ever implementation of Gen.M, although the program was derived from earlier EngenderHealth curricula on gender roles. The evaluation will test the impact of Gen.M on youth ages 14 to 16 in three cohorts. It will test whether the Gen.M program is effective at delaying sexual activity and/or reducing risky sexual behavior among youth who are sexually active. Figure 1 presents a summary of program components, the targeted mediating factors, and intermediate and primary outcomes. Gen.M Evaluation—A Snapshot • Part of the national multiyear Evaluation of Adolescent Pregnancy Prevention Approaches: o Funded by the Office of Adolescent Health, U.S. Department of Health and Human Services o Conducted by Mathematica Policy Research, with Child Trends and Twin Peaks Partners, LLC o Assessing effectiveness of seven programs • Approximately 1,140 youth ages 14 to 16 will be recruited and randomly assigned—half to a program group and half to a group that does not receive Gen. M: o Program will be delivered to program group youth in three cohorts o Sample intake will occur annually, February–July of 2012–2014 • Three components: o Five 4-hour sessions presented on consecutive days in July or August by a male/female pair of facilitators to small groups of 8 to 16 youth o Participants are paid $150 for completing the sessions o Social media (SMS texting and Facebook) campaign in August–December o Film about each cohort’s experience is shown to youth in the fall • Topics covered: Gender roles, healthy relationships, making decisions about sexual activity, and skills for preventing pregnancy through use of condoms and other contraception • Impacts will be measured by follow-up surveys 6 and 18 months after the end of the program • Summer 2012 implementation: o Training and technical assistance provided to14 facilitators o Two rounds of workshops served 154 youth o Fidelity monitoring of workshops 1 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Gen.M is not the only teen pregnancy prevention program in Travis County. Real Talk, an abstinence-based program that uses It’s Your Game: Keep it Real for middle school students and Reducing the Risk for high school students, is offered in Austin Independent School District middle and high schools by a partnership of LifeWorks and Planned Parenthood of the Texas Capital Region. In prior research, both It’s Your Game and Reducing the Risk showed evidence of favorable program impacts on youth sexual risk behaviors. In addition, Austin LifeGuard Character and Sexuality Education, an abstinence-based program, is being implemented by Austin LifeCare to middle and high school youth groups. It is likely that some youth in the Gen.M evaluation sample have been or will be exposed to one or more of these other curricula. Figure 1. Logic Model of the Gen.M Intervention Program Components Mediating Factors Intermediate Outcomes Primary Outcomes • Workshop (20-hour • Conceptions of • Delayed onset of sexual • Reduction in the incidence curriculum) masculinity and femininity activity of teen pregnancy • Social media (Facebook related to sexual behaviors group, text messaging) and pregnancy • Increased use of effective • Film screening contraceptive methods, • Attitudes toward gender including hormonal birth Curriculum topics: roles in intimate control methods and the • Gender norms relationships IUD • Healthy relationships • Teen parenting and skills • Communication, • Increased consistency in to delay sexual activity negotiation, and refusal the use of condoms • Pregnancy, STIs, and skills condom use • Sexual and reproductive • Knowledge of health services contraceptive methods and STIs • Knowledge and use of sexual and reproductive health services Gen.M’s sponsors aimed to serve racially and ethnically diverse youth from low-income areas of Travis County, some of whom would likely have already had sexual experience and for whom accurate understanding of sex would be important for preventing pregnancy. Among youth enrolled in Gen.M in summer 2012, one-third (31 percent) reported having been sexually active, and 20 percent reported having had sexual intercourse in the past three months. Before experiencing Gen.M, between a third and half of youth (36 to 55 percent) had accurate knowledge of contraceptive methods and the risk of pregnancy and sexually transmitted infections (STIs). Figure 2 presents detail on participants’ age and race/ethnicity, as well as socioeconomic status of Travis County youth. Additional details on the program participants are in Appendix A. 2 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Figure 2. Travis County Youth Demographics Gen.M Participants' Age Gen.M Participants' Race/Ethnicity Other White 4% Non- Hispanic Age 16 5% Age 14 24% 30% Hispanic Black 35% Non- Age 15 Hispanic 46% 57% Socioeconomic Status of Travis County Youth Below Age 18 Below Poverty Level, 22% Above Poverty Level, 78% Source: Youth surveys administered by the PPA evaluation team in June and July 2012; U.S. Census Bureau, 2007–2011 American Community Survey. A program such as Gen.M, implemented in this context, raises several questions: • Who developed Gen.M and why? Who implemented and delivered the program? How was the program delivered? • How were program facilitators trained to deliver Gen.M? How did they respond to training? How well did they deliver the program? • How did youth respond to the program? Did they participate in the workshops? Were they engaged by the material and facilitators? Did they understand the key messages? • What lessons were learned from this study? How are they relevant for future program implementation and replication efforts? To address these questions, the PPA evaluation team (Mathematica Policy Research, Child Trends, and Twin Peaks Partners LLC), in collaboration with Columbia University’s Mailman School of Public Health, conducted several types of data collection. The Mathematica team completed two visits to conduct face-to-face interviews with EngenderHealth staff and other stakeholders, focus group discussions with Gen.M facilitators, and classroom observations. A Columbia research assistant conducted face-to-face interviews with Gen.M facilitators and 3 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas participants, as part of Columbia’s role as local evaluator under contract to EngenderHealth. The team also analyzed monitoring logs completed by facilitators, observation forms completed by EngenderHealth and Columbia staff, and surveys completed by facilitators and participants. Details on data sources and methodology for the implementation study are provided in Appendix B. The PPA implementation analysis includes multiple components. The research team used qualitative analysis software to conduct descriptive analyses of site visit, focus group, and observation data. The team also used implementation benchmarks (Appendix B) to assess adherence to the implementation plan in the first year of program implementation. Columbia University will assess and report on implementation of Gen.M beyond the first year. This report presents findings from the first year of Gen.M program implementation. The report describes the program’s design, facilitators’ training in and delivery of the program, and youth engagement in and understanding of the material. 1 It concludes with tentative lessons about implementing programs such as Gen.M. 1 Because this report only analyzes the implementation experiences in the first year of the Gen.M intervention, is too early to make conclusions about how these insights will inform Mathematica’s assessment of program impacts on (1) sexual risk outcomes, which include both measures of sexual behaviors and their consequences, most notably pregnancy; and (2) intermediate outcomes, which correspond to the mediating factors through which the program would most likely have an impact on behavior. 4 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas FRAMEWORK FOR PROGRAM DELIVERY The implementation of the Gen.M program in summer 2012 can be summarized with reference to three general questions: (1) Who developed Gen.M and why? (2) How was the program structured? (3) How was the program delivered, and by whom? Gen.M Links Gender Norms and Inequalities to Sexual Behavior Outcomes Gen.M was developed by EngenderHealth in 2010. The Gen.M curriculum grew out of the Men as Partners® (MAP) program developed in the early 2000s by EngenderHealth to address gender inequalities in sexual and reproductive decision making. The MAP program uses interactive workshops, public education campaigns, and advocacy networks to deliver public health messages about gender equality and sexual decision making to young men in third-world countries. The goal of the program is to change men’s perceptions of gender roles so that they view themselves as equal partners (with women) in establishing and maintaining healthy sexual relationships. Like the MAP program, Gen.M uses an interactive workshop approach. Unlike the MAP program, Gen.M serves youth of both sexes, with a more restricted focus on the relationship between perceptions of gender roles and associated risks of teen pregnancy. The link between gender norms and sexual behavior outcomes that underlies the Gen.M curriculum is supported by an extensive literature. Studies show that adolescent males who hold stereotypical attitudes toward masculinity use condoms less consistently, have lower use of health services, and have greater belief that pregnancy validates masculinity (Kandrack et al. 1991; Pleck et al. 1993; and Courtenay 2000). Adolescent females who hold conventional views on femininity are more likely to accommodate men’s interests, use condoms less consistently, and become pregnant at an early age (Connell 1987; Stewart 2003; Ickovics and Rodin 1992). EngenderHealth anticipates that learning about the influence of gender norms on sexual and behavior outcomes will motivate and prepare youth to delay sexual activity and/or reduce risky sexual behavior. Three psychosocial behavioral change theories guide this expectation. First, the Gen.M curriculum applies elements of social cognitive theory. Youth learn behaviors by observing their peers and practicing their knowledge and skills in their own environments. Second, the curriculum incorporates the theory of gender and power. Youth examine how gender norms and power dynamics in relationships influence sexual risk behaviors and teenage pregnancy. Third, Gen.M uses social norm theory. The program explores, questions, and attempts to change participants’ perceptions about gender and pregnancy. In addition, the curriculum addresses other determinants of teen sexual behaviors: decision making, communication, and negotiation skills and knowledge about effective birth control methods. 5 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Gen.M Has Three Interactive Components Gen.M is a community-based intervention with three interactive components. Facilitators deliver a 20-hour educational curriculum in weeklong workshops, held in two rounds each summer, one in July and one in August. After the workshops are completed, EngenderHealth staff facilitate a social media (SMS texting and Facebook) campaign from August through December. EngenderHealth staff also bring the participants together in the fall to view a film composed of video clips of the youth participating in the summer workshops. The social media campaign and film screening are designed to support and reinforce key messages delivered in the summer workshops. 2 EngenderHealth piloted all three components of the Gen.M program in 2011, and revised them based on the pilot experience and feedback from participants and facilitators. Workshops. The Gen.M workshop curriculum builds on five themes that emphasize understanding gender, using protection/birth control, and building relationships. The themes are: (1) gender norms and their influence on sexual behavior; (2) healthy relationships; (3) challenges of being a teen parent and skills to delay sexual activity; (4) pregnancy, STIs, and how to prevent Gen.M’s Key Messages both through condom use; and (5) taking action— • I am the boss of me. family planning services and individual behavior change plans to prevent pregnancy. Activities • I decide what being a man or a woman means to me. implemented during the workshops emphasize and repeat at least one of Gen.M’s six key messages (see • I treat others in the way I want to be treated. text box). These messages were developed based on the five themes. They are posted on the classroom • I make have own decision about if and when to my sex. wall during each workshop. EngenderHealth program leaders expect that sustained exposure to these clear • I use protection every time I have sex. messages about positive behaviors will help youth • I go to the clinic to get tested and protected. remember the messages and make the behaviors part of their own lives. The curriculum engages youth as active participants. Facilitators lead discussions to introduce themes and messages. Participants work independently and produce individual work (for example, assessments and writing assignments) in which they use critical thinking and analysis skills to apply the themes and messages to their lives. Group activities (such as icebreakers, games, small group discussions, and role plays) encourage expression and communication. (Appendix Table C.1 provides a more detailed summary of the Gen.M lessons.) At the end of each session, facilitators videotape participants’ reflections on the day’s activities—to be incorporated later into the video component of Gen.M. 2 This report focuses on the implementation of the workshop curriculum, and does not cover implementation of the social media campaign and film screening. Subsequent reports by Columbia University will document the role of these two later components in the program. 6 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Examples of Group Activities Sexual Decision Making: Gender Fishbowl: Condom Obstacles: 1. Participants brainstorm in 1. Females sit in a circle and are 1. Participants brainstorm about male- and female-only groups videotaped as they discuss why teens choose not to use about why some teens choose experiences related to being condoms consistently (5 to have sex and others do not, female; males observe/listen minutes). and examine differences based (18 minutes). on gender (25 minutes). 2. Small groups identify ways to 2. Males sit in a circle and are overcome barriers to using 2. Participants discuss case videotaped as they discuss condoms (10 minutes) studies of teen couples and experiences related to being whether they are ready to have male; females observe/listen 3. Groups display and review sex (25 minutes). (18 minutes). barriers and solutions (5 minutes). 3. Small groups draft letters 3. Participants discuss what they providing advice on how to learned (4 minutes). 4. Participants discuss barriers cope with peer pressure to and which solutions are most have sex (20 minutes). realistic (10 minutes). To keep sessions lively, facilitators use a variety of props: • Flipcharts to facilitate group discussions. For example, in the condom obstacles activity, participants record on a flipchart barriers to using condoms, along with their proposed solutions to overcome them, and the group discusses the barriers and solutions. • Handouts to engage students. In the sexual decision-making activity, facilitators distribute case studies and participants discuss which factors in the stories influence the characters’ decisions about sex. • Penis model to demonstrate the eight steps of condom use. Participants then break into groups to practice their own skills using the same kind of model. • Laptops or tablets to connect to the internet. Facilitators use the laptops or tablets to display examples of gender messages in music videos and movie and television clips, and to look up medically accurate information and answer participants’ questions. • Handheld video camera to videotape participants at the end of each day. For instance, during the Gender Fishbowl activity, the facilitator records the group discussion about what participants have learned about being male or female. Social Media Campaign. After the summer workshops, program staff reinforce curriculum messages in a four-month social media campaign. Throughout the workshops, facilitators invite participants to join a private, supervised Facebook group. After the workshop ends, participants can access information and share thoughts in this group, while also maintaining connections formed with other participants during the workshop. Each month from August through December, EngenderHealth staff focus their postings on one of the program’s key messages, with a goal of stimulating discussion among group participants. Film Screening. To further reinforce key messages and foster continued participant relationships, the program staff invite participants to a film screening. The 20-minute film compiles the components of the lessons during which participants were videotaped. In these video segments, participants (either in group discussions or individually) detail what they have 7 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas learned, and how they intend to use this information in their lives. During summer 2012, EngenderHealth hired a production company to videotape the youth and create the film, which can be used as an example for replications. However, EngenderHealth acknowledges the video can be produced in a satisfying and engaging way (with a handheld camera, following the recommended storyboard) without the cost of outside contractors. Partners Helped EngenderHealth Recruit Youth and Deliver the Program To implement Gen.M, EngenderHealth needed a strategy for recruiting participants and competent facilitators. EngenderHealth partnered with a local youth development program to recruit youth. The Travis County Summer Youth Employment Program (SYEP) places youth at local job sites for one of two five-week sessions during the summer. Not all youth who apply for SYEP are able to receive job assignments because of limitations in the number and size of job sites. EngenderHealth recruited youth who had applied for SYEP, including some who were placed at SYEP job sites and some who did not receive job assignments. 3 To attract youth to participate in the program, EngenderHealth offered youth an incentive payment. This incentive was equal to the weekly payment youth received for participating in SYEP ($150 per week). Only those youth who attended all five days (20 hours) of programming received the entire incentive payment. Facilitators’ Professional Backgrounds SafePlace, a center that provides services to victims of domestic violence and sexual assault, • 5 licensed professional counselors provided facilitators (selected by SafePlace and • 3 licensed social workers approved by EngenderHealth). SafePlace shares • 1 prevention education manager EngenderHealth’s goals of promoting healthy teen relationships and improving participants’ understanding • 1 chemical dependency counselor of gender and gender stereotypes. The facilitators from • 1 community educational specialist SafePlace brought to the program past training in • 3 youth program coordinators/staff counseling, social work, and related fields. They had worked with youth as support group facilitators or educators for SafePlace’s Expect Respect program, which offers school-based support groups and counseling, youth leadership activities, and educational programs in school and community settings. In their earlier work, all facilitators had discussed healthy and unhealthy relationships (with respect to dating violence) with youth, but had not taught youth about sexual and reproductive health. SafePlace paired facilitators (one male and one female) for both rounds of summer 2012 workshops. In the first-round workshop, 12 facilitators conducted the workshop for six workshop groups. Two of these facilitators (from different facilitator pairs) were unavailable to lead the second round of workshops, and were replaced by two other facilitators. These facilitators had been previously recruited and approved by SafePlace and EngenderHealth as alternates. 3 Details on sample enrollment and random assignment of youth are provided in the PPA Evaluation Design Report (Smith and Colman 2012). 8 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas FACILITATORS’ TRAINING AND PROGRAM DELIVERY Although the workshop facilitators for Gen.M had prior experience in similar positions at SafePlace, they had not implemented such a prescribed curriculum or received formal training on leading group discussions and facilitating interactive elements such as role plays and icebreakers. Furthermore, the facilitators were not certified as sex educators. To improve their skills and ability to deliver the curriculum with fidelity, EngenderHealth offered the facilitators training on the curriculum, classroom management, and sexual and reproductive health. All facilitators attended the training. Implementing Gen.M twice in the same summer, after each of the SYEP sessions, showed that experience helps. By the second workshop, facilitators had improved their program delivery and achieved a higher level of proficiency with the material. In addition, EngenderHealth modified its approach to monitoring program delivery, reducing the number of sessions monitored to give facilitators a greater sense of autonomy. Training Gave an Overview of the Curriculum and Hands-On Practice Facilitators received a five-day (40-hour) training in spring 2012 that oriented them to the curriculum and classroom management. On the first and second days, EngenderHealth staff provided a brief overview of the Training Components Gen.M program and the evaluation design, and demonstrated several • Overview of Gen.M program activities. On the third and fourth • Introduction to evaluation days, facilitators presented selected activities and received feedback on • Review of research on gender norms and teen pregnancy how they might improve their • Introduction to fidelity and its importance in the program facilitation techniques. On the fifth • Demonstration and teach-backs of curriculum activities day, a consultant taught facilitators • Review of group management skills how to effectively manage groups. • Introduction to reproductive and sexual health issues EngenderHealth held a supplemental training later in the spring to give facilitators more practice teaching youth about sexual and reproductive health. Facilitators wanted more practice opportunities than were available during the five-day training period. They reported that it would have been easier to implement Gen.M if the training had included modeling of all of the curriculum activities. They wanted to see the curriculum facilitated “the right way” and wanted more time to practice program delivery. One facilitator said, “I felt more confident doing the sessions that I had already seen or done” in the training; he felt modeling all of the sessions would have boosted his confidence in his facilitation skills throughout the workshop. Training and Technical Assistance Emphasized Fidelity During the training and subsequent technical assistance (described below) available to facilitators, EngenderHealth repeatedly emphasized fidelity to the curriculum. Early in the training, EngenderHealth defined expectations for maintaining fidelity to the curriculum and described it as “critical” to the evaluation. EngenderHealth defined fidelity as “replicating an intervention as it is written so that its core components are not compromised,” and defined core 9 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas components as “essential features of an intervention that are responsible for its effectiveness.” EngenderHealth stressed that facilitators should document and alert EngenderHealth staff to any modifications made to activities during program delivery. After training was completed, EngenderHealth held monthly meetings with facilitators to review logistics for the workshops and further discuss fidelity. EngenderHealth also monitored program delivery for fidelity, through both classroom observations and a review of written feedback from facilitators. (An example of the fidelity log forms used to monitor program delivery is in Appendix D. An example of the program observation log form is in Appendix E.) When EngenderHealth staff noticed changes in lesson plans, they reviewed these modifications with facilitators and gave them feedback on how to maintain fidelity, as well as how to improve program delivery. Facilitators Were Not Fully Prepared to Maintain Fidelity Implementing a prescribed curriculum with fidelity was largely unfamiliar and uncomfortable territory for facilitators. By emphasizing fidelity, the training led some facilitators astray. They interpreted maintaining fidelity to the curriculum as reading from the curriculum word-for-word (without modifications for cultural- or age-appropriateness). Facilitators believed the directions they received from EngenderHealth did not allow them to insert their own phrasing or interpretations of the key messages. Several facilitators were concerned and anxious about being rated for fidelity; they were apprehensive about EngenderHealth’s review of forms submitted by teachers that noted any minor or major changes to the activities. The way the training was received illustrated some of the differences between EngenderHealth and SafePlace. The two organizations focus on different concerns. EngenderHealth implements sexual and reproductive health programming, while SafePlace implements programs that address domestic violence. In addition, whereas EngenderHealth is accustomed to observing its staff to ensure that program models are implemented with fidelity, SafePlace does not regularly observe staff members’ interactions with youth, and encourages its team members to create their own activities and discussion topics based on youths’ needs. EngenderHealth Made Adjustments in Response to Feedback The two organizations’ divergent practices and the facilitators’ initial discomfort prompted EngenderHealth to rework the original curriculum sequence. EngenderHealth shifted the placement of several activities to improve the flow of the curriculum, as well as the facilitators’ and participants’ comfort. For example, because EngenderHealth felt the gender fish bowl activity (in which participants discuss their views of gender messages) would be more effective if participants had already achieved a degree of familiarity amongst themselves, the activity was moved later in the curriculum. Icebreakers were added to the first day so that youth could have more time to get to know one another and become engaged in the program. EngenderHealth also decreased its emphasis on a strict interpretation of fidelity, and allowed facilitators to make some types of adaptations to the curriculum. Any modifications were documented systematically in fidelity log forms and program observation log forms. Facilitators could make modifications for age and culture, and could also substitute or modify activities, provided EngenderHealth agreed that they covered the same topics as the activities that were replaced or modified and emphasized the key messages. In adapting the curriculum, facilitators made judgments about what would engage youth better. A few facilitators added activities to keep the participants engaged and moving. Some activities were modified by EngenderHealth 10 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas based on feedback from SafePlace. Facilitators felt the revised activities—such as asking participants to say how to refuse unwanted sexual advances, or testing participants on their knowledge of male and female reproductive anatomy—helped participants apply and retain the curriculum’s key messages. EngenderHealth worked with the facilitators to make greater use of skits in order to make the material more “alive” and keep participants actively engaged. In some cases, activities were rewritten to allow participants to act out how they would handle certain situations (as opposed to writing down their thoughts, as originally recommended in the curriculum). For example, in one session, participants produced a talk show, during which a “host” provided advice to different “guests” on how to make sexual decisions. In another session, participants created a public service announcement, during which they provided information about recognizing and dealing with unhealthy behavior. Some facilitators covered material in greater detail to enhance the key messages. For example, a few facilitators wanted participants to have full knowledge of their contraceptive options, should they choose to have sex. Hence, rather than limit the number of contraceptive methods reviewed, they spent additional time reviewing all available contraceptive methods. Because facilitators provided participants with more, not less, information than in the lesson plan, EngenderHealth welcomed this modification. These Adjustments Led to Better Program Delivery and Fidelity Facilitators felt they could do a better job delivering the program once EngenderHealth revised expectations for fidelity and rearranged curriculum activities. All of the combined changes enabled the facilitators to be less concerned about following the curriculum verbatim. Rather, they were more focused on drawing out main themes. EngenderHealth reported that facilitators Gen.M was Implemented with High Fidelity to delivered the curriculum with fidelity. Facilitators the Implementation Plan were able to complete all activities, although • Activities delivered as prescribed, with very some had to cut group discussions or minor modifications brainstorming sessions short if time was tight. • Comfortable environments created Only one facilitator pair reported there was not enough time to complete one activity. • Program messages communicated clearly EngenderHealth asserted that any adaptations or • Participants’ questions answered effectively modifications to the lesson plan did not alter (and, • All of the activities taught in each session in some cases, actually enhanced) the key messages. Facilitators’ Confidence and Program Delivery Skills Increased Over Time Confidence with the material grew over time. Initially, facilitators lacked comfort with sexual and reproductive health topics. They were apprehensive about facilitating youth groups on these topics. This resulted from the facilitators’ lack of training in health education. However, as the facilitators were immersed in the curriculum materials, they became more attuned to the curriculum content. In addition, the technical assistance provided by EngenderHealth (which addressed any questions on content or program delivery) boosted the facilitators’ self-assurance 11 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas in their own program delivery skills. After implementing one workshop, most facilitators were more at ease with the material. During the workshops, facilitators built on pre-existing skills in working with youth to educate them about gender messages and relationships. Indeed, despite their lack of comfort with some of the material, most facilitators worked well in teams and built a rapport with Gen.M participants. Two facilitators were particularly at ease in the groups because they were in their early 20s, close in age to the youth participating in Gen.M. Both PPA site visitors and EngenderHealth monitors observed facilitators who appeared to have clear control of the youth in each workshop, covering all activities in full (stating key messages throughout) and welcoming participants’ questions and contributions to the discussions. 12 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas PARTICIPANTS’ ENGAGEMENT AND UNDERSTANDING OF THE MATERIAL For a program to change participants’ behavior, it is important for enrolled participants to attend all sessions and be engaged by both the material and the environment in which the material is presented. Participants should leave the program with the ability to apply what they have learned. In the case of Gen.M, this translates into self-confidence in making sexual decisions, as well as knowing how to prevent teen pregnancy. To determine the extent of participants’ attendance, engagement, and understanding, the analysis of the first year of Gen.M implementation documented attendance and assessed participants’ engagement in class and their understanding of the material. Attendance Among Participants Was High Most of the enrolled youth attended the workshops. Of the 154 youth enrolled in the treatment group, 126 (81 percent) completed the 20-hour program. Three participants dropped out after one day (no reason given), two were asked to leave after the second day because they were disruptive and disrespectful, and nine missed two or three sessions because of other commitments. Fourteen youth did not participate in the program. On the one hand, the high attendance could be a result of the $150 incentive participants received for attending all five sessions. On the other hand, youth appeared quite engaged (described in detail below). Hence, the incentive may have been important initially in enticing youth to come to the first session, but was not necessarily the critical element in engaging them for the remainder of the week. Participants Were Engaged, Active, and Valued According to facilitators, participants became increasingly connected to the material over the course of the workshop. On the first day of the program, facilitators noticed that the participants had a hard time understanding various concepts and engaging with the material. However, by the end of the workshop, the youth appeared to have retained the key messages. In interviews with Columbia University staff, participants easily recalled the take-home messages on how to use and access contraceptive methods, prevent STIs, and control their own actions. Facilitators reported that participants engaged with the material most when they were active. Youth were most engaged in skits and role plays. One facilitator commented, “The days that we [had] more interactive things…the kids were pretty well engaged.” For example, during the sexual decision-making activity on the third day, when participants are placed in small groups to draft letters providing advice on how to cope with peer pressure to have sex, two facilitator pairs recollected that youth connected with the material and appeared to express their emotions honestly. One participant remarked that she found it “cool” to interact with the facilitators and other participants through role plays. Participants were less focused when lessons involved lectures, or question-and-answer sessions, possibly because these activities felt more like school. In interviews, participants said they felt that facilitators treated them like adults and valued their opinions and answers to questions. As a result, participants were not shy; they actively voiced their opinions and enjoyed hearing from others, particularly when hearing about how males and females perceive themselves. One participant said, “I learned that my opinions about women weren’t all right, but the girls’ opinions of men weren’t right either.” Still, facilitators reported that participants’ concentration and engagement seemed to lapse (if only briefly) again on the last day of the workshop, when they appeared to be more focused on getting paid. 13 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Program Environment Encouraged Sharing of Opinions and Open Questioning Participants appeared to respect the facilitators. Both PPA site visitors and EngenderHealth monitors found that facilitators created a safe environment in which participants could express their opinions. Facilitators peppered sessions with semi-open-ended questions to foster group discussions (for example, “Was it easier to come up with reasons for not using condoms or solutions to the reasons? Why?”). They encouraged participants to speak up and share their answers and opinions. Participants commented that they did not feel nervous about opening up to the facilitators. One participant said, “I felt like I could say whatever and [facilitators were] okay with me saying it.” Participants also had high regard for their peers. Most youth interviewed by Columbia University researchers felt that similarities in backgrounds between themselves and their peers allowed them to connect with each other and freely share opinions. One participant stated, “We all had the same kind of questions [and] experiences.… So that’s why we all could relate to each other.” Participants said they would like to maintain friendships, but thought that this might be difficult given that they are dispersed across Travis County and attend different schools. The openness between facilitators and participants may have contributed to participants’ self-assurance in asking questions during the workshop. If participants felt they might be embarrassed by asking the questions aloud, they could ask questions anonymously by inserting a sheet of paper with their question in a box. One facilitator commented, “I was impressed by the students’ desire to understand the materials.” Participants’ questions generally showed that they were thinking through what they were learning. Students were curious about reproductive anatomy (“How do you know when you are ovulating?”), how to access birth control, and transmission of STIs/HIV (“What are all the legal consequences of knowingly transmitting STIs/HIV to other people?” “Is HIV/AIDS curable?”). Facilitators recognized that some of the students’ questions reflected their ignorance regarding sexual health. However, the facilitators encouraged these questions so they could dispel any myths the participants might believe. For example, some participants asked whether drinking a lot of water and urinating would prevent pregnancy if they did not use protection. In response to this question, the facilitators effectively talked through the use of birth control, and cleared up this case of misinformation. Core Messages Resonated with Participants Participants understood the core messages about using contraceptives and going to clinics. Facilitators felt that, during the workshops, the participants displayed clear understanding of how to prevent pregnancy (by abstaining from sex or, if sexually active, using contraceptives). In interviews with Columbia University researchers, participants identified concrete sexual health knowledge they gained from the workshops. Topics that made an impression included male and female reproductive anatomy, how to use a condom and have safe sex, and how STIs are transmitted. Interviewed participants viewed information about condoms, contraceptive methods, and where to get them as new and important knowledge. For example, one participant said that prior to the workshop she “had no idea [about] any of the birth control methods except for the pill,” but was thankful that she had learned about other available contraceptives and that she could go to a clinic to get them. In a survey conducted by Columbia University, participants 14 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas reported that, as a result of what they learned in the workshops, they were more likely to use a condom or other methods of birth control if they have sex in the next year. Perceptions of Core Messages Differed Between Males and Females Several of the core messages meant something different to males and females. Both males and females valued being in control of their own actions and said they would use the strategies Gen.M taught to “say no” to sex. However, males and females had different perceptions of what this meant with regard to sex and dating. Females understood that they could defend themselves against peer pressure to date and have sex. On the other hand, males heard that they could make individual choices about sex (that is, to wait to have sex and not to pressure a partner into having sex). Several male participants said that, after the program, they had acquired a new understanding of how to have a healthy relationship that involves trust, communication, support, and consent. Age Differences Were Reflected in Responses to the Material Older participants (ages 15 or 16) were able to connect with the material. Facilitators found they did not have to work hard to get the messages across to these youth, perhaps because these youth were more likely than younger youth to have had sexual experiences or to know about these experiences from their friends. One facilitator said these older youth were “phenomenal” and were highly engaged with the material, moving through the topics with relative ease. In addition to finding the material useful, older participants were not afraid to speak about their own sexual experiences and voice their opinions. Younger participants (age 14) had difficulty understanding some of the material because they had less experience with sex and dating. As a result, they were less comfortable discussing these topics. One facilitator said, “They just hadn’t had as many of those [sexual] experiences, so they weren’t attached to it….There wasn’t anything that was concrete in their lives.” Some facilitators commented that, although they emphasized the connection between gender and sexual decision making during the workshop, it just “didn’t make sense” to the youth. One facilitator felt the youth were simply repeating “sound bites” and did not seem to be enlightened in regard to gender messages. Indeed, Columbia University staff reported that the younger youth seemed less likely to relate to the curriculum’s messages in their own lives; they “were generally optimistic about their ability to carry out the behaviors learned in Gen.M in a way that betrayed a kind of naiveté.” Although they were less familiar with the subject matter, younger participants appeared to be more engaged in the material when they were in a group with older youth. In these groups, the older participants were able to provide their younger counterparts with real world examples, helping them to understand the key messages with regard to pregnancy prevention and reproductive health. One facilitator reported that the group he facilitated with 14 to 16 year olds “went amazingly because there [were] people that were willing to speak and people with experiences of things that had happened, people with friends who were going through teen pregnancy. They had experiences of things, so it actually applied to them emotionally, and they were learning from it because they were attached to it.” 15 This page has been left blank for double-sided copying. Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas LOOKING FORWARD: LESSONS FOR FUTURE IMPLEMENTATION In the first year of implementation, EngenderHealth experienced both achievements and challenges in getting the Gen.M program off the ground. EngenderHealth crafted a methodical and scripted curriculum that (1) enabled facilitators to deliver the program effectively, and (2) actively engaged youth. However, EngenderHealth also faced a challenge in balancing fidelity of implementation with discretion of facilitators in program delivery. In addition, younger participants struggled to grasp some of the concepts. EngenderHealth’s early experience has immediate implications for the continued implementation of Gen.M in Travis County, Texas. As a result of the growing pains in the first year of operations, EngenderHealth has modified its approach. Facilitators will be more actively involved in planning for the implementation of future workshops. EngenderHealth will use the facilitators’ feedback to define allowable further adaptations to program delivery. There are also implications for future program implementation by other organizations (discussed below). Emphasis on fidelity should be balanced with facilitator discretion. EngenderHealth initially faced difficulties in striking the right balance between providing guidance on maintaining fidelity to the program model and allowing facilitators to adapt elements of the curriculum in small ways. Throughout training, EngenderHealth stressed the importance of maintaining fidelity. In turn, facilitators interpreted fidelity as maintaining strict adherence to the program’s lesson plans. This misunderstanding was corrected over the course of the summer, as EngenderHealth (1) provided more guidance on options for modifying language in the curriculum for age and culture, and (2) allowed facilitators to introduce approved adaptations that did not alter core messages. In future implementation efforts, and specifically in the training that introduces facilitators to the program, organizations should clearly state expectations for maintaining fidelity, and should provide guidance on what types of adaptations are allowed. Younger and older adolescents might need to be taught separately. Younger teens are less likely than their older peers to have had sexual experiences. In addition, they are less likely to be comfortable talking about the possibility of having such experiences. Hence, these youth may have trouble relating to some of the Gen.M material and applying the core messages to their daily lives. There are several ways for organizations to deal with this. Organizations could choose simply to focus on older teens, who are more likely to have had sexual experiences and can relate to the key messages in the curriculum’s current form. Organizations might also choose to separate younger youth from their older peers and serve each group separately. In this case, organizations could adapt the curriculum to account for the younger teens’ relative inexperience, while not modifying the curriculum for the older youth. Active engagement during the workshops supports learning. The curriculum’s approach enables youth to open up, participate actively in sessions, and engage with the material. In particular, icebreakers (at the beginning of the workshop and interspersed throughout the five days) help participants to connect with one another. These activities facilitate participants’ trust of the facilitators. Sessions that included a lot of participant interaction (for example, sessions that mix icebreakers, role plays, and group discussions) seem to work best because they keep youth moving and interested in the subject matter. Finally, the repetition of the six key messages, 17 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas combined effectively with role plays and skits, helps youth remember the messages (especially those related to pregnancy prevention) after the workshop is completed. An initial incentive may help to attract youth. A 20-hour, five-day program can be a large commitment for youth. It is only feasible in the summer if the program is to retain its intensive character. EngenderHealth addressed this by offering participants a $150 incentive payment if they attended all five 4-hour program sessions. In summer 2012, this amounted to close to $20,000 across all program participants—a large sum that many organizations cannot afford. Another, less costly approach is to offer youth a more modest sum to attend the first day of the program. This approach might prove just as effective in attracting youth, if the interactive nature of the program model can engage youth and motivate them to attend the remaining program sessions. 18 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas REFERENCES Connell, R. Gender and Power: Society, the Person and Sexual Politics. Palo Alto, CA: University of California Press, 1987. Courtenay, W.H. “Constructions of Masculinity and Their Influence on Men’s Well-being: A Theory of Gender and Health.” Social Science and Medicine, vol. 50, no. 10, 2000, pp. 1385–1401. Ickovics, J.R., and R. Rodin. “Women and AIDS in the United States; Epidemiology, National History and Meditation Mechanisms.” Health Psychology, vol. 11, 1992, pp. 1–16. Kandrack, M.A., K.R. Grant, and A. Segall. “Gender Differences in Health Related Behavior: Some Unanswered Questions.” Social Science and Medicine, vol. 32, no. 5, 1991, pp. 579– 590. Patton, Michael Quinn. Qualitative Research and Evaluation Methods: Third Edition. Thousand Oaks, CA: Sage Publications, 2002. Pleck, J.H., F.L. Sonenstein, and L.C. Ku. “Masculinity Ideology: Its Impact on Adolescent Males’ Heterosexual Relationships.” Journal of Social Issues, vol. 43, no. 3, 1993, pp. 11– 29. Ritchie, Jane, and Liz Spencer. “Qualitative Data Analysis for Applied Policy Research.” In The Qualitative Researcher’s Companion, edited by A. Michael Huberman and Matthew B. Miles. Thousand Oaks, CA: Sage Publications, 2002. Scientific Software Development. “Atlas.ti: Visual Qualitative Data Analysis Management Model Building in Education Research and Business.” Berlin, Germany: SSD, 1997. Smith, Kimberly, and Silvie Colman. “Evaluation of Adolescent Pregnancy Prevention Approaches: Design of the Impact Study. Final Report.” Princeton, NJ: Mathematica Policy Research, October 2012. (Supporting authors: Christopher Trenholm, Alan Hershey, Brian Goesling, Anastasia Erbe, Caitlin Davis, and Brice Overcash [Mathematica]; and Kristine Andrews, Amanda Berger, Lori Ann Delale-OConnor, and Mindy Scott [Child Trends].) Available at: [http://www.hhs.gov/ash/oah/oah-initiatives/assets/ppa_design_report.pdf]. Stewart, J. “The Mommy Track: The Consequences of Gender Ideology and Aspirations on Age at First Motherhood.” Journal of Sociology and Social Welfare, vol. 30, no. 2, 2003, pp. 3– 30. U.S. Census Bureau. “State & County Quick Facts.” 2013. Available at: http://quickfacts.census.gov/qfd/states/48000.html. Accessed January 25, 2013. 19 This page has been left blank for double-sided copying. APPENDIX A SAMPLE CHARACTERISTICS This page has been left blank for double-sided copying. Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Table A.1. Sample Characteristics Percentage of Treatment Group Students Demographic and Background Characteristics Age in Years 14 29.6 15 46.1 16 24.3 Female 51.3 Language Spoken at Home English only 84.4 Spanish only 1.9 English and Spanish 13.0 a Other 0.6 Race/Ethnicity White Non-Hispanic 4.5 Black Non-Hispanic 56.5 Hispanic 35.1 Other (including multiple) 3.9 Lives with Both Biological Parents 30.9 Parents’ Employment Mother currently employed 83.6 Father currently employed 85.1 Relationship with Parents Feels very close to mother 65.1 Feels very close to father 38.6 Considers Religion Very Important in His or Her Life 83.6 Attends Religious Services/Activities at Least Once a Week 37.5 Levels of Risky Behavior Alcohol and Drug Use Ever had an alcoholic beverage 55.2 Had alcoholic beverage in past 30 days 27.0 Binge drinking in past 30 days 14.0 Ever smoked marijuana 38.7 Smoked marijuana in past 30 days 22.3 Ever used an illicit substance (including prescription drugs and inhalants) 20.7 Ever had sexual intercourse 31.1 Number of lifetime sexual intercourse partners 0 68.9 1 11.3 2 7.3 3 or more 12.6 23 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Percentage of Treatment Group Students Frequency of sexual intercourse in past three months (number of times) 0 79.6 1 8.8 2 to 5 7.5 6 or more 4.1 Had sexual intercourse without a condom (past three months) Yes 6.8 No 13.6 Did not have sexual intercourse 79.6 Had sexual intercourse without using any effective birth control method (past three months) Yes 6.1 No 14.3 Did not have sexual intercourse 79.6 Ever Been/Gotten Someone Pregnant 3.9 Perceived Peer Pressure to Engage in Sexual Intercourse Feels a lot of pressure 5.8 Feels any pressure 39.6 Parents’ Attitude About Child Having Sex and Having a Baby at This Time Mother disapproves of sex and having a baby at this time 67.1 Father disapproves of sex and having a baby at this time 60.0 Would Not Feel Upset if Got/Got Someone Pregnant at This Time 34.6 Behavioral Expectations Expects to have sexual intercourse next year 47.3 Expects to have sexual intercourse before marriage 68.6 Knowledge Related to Contraceptive Effectiveness and Risk of Pregnancy and HIV/STIs Condoms Decrease the Risk of Pregnancy Not at all 3.9 A little 24.0 A lot 50.0 Completely 6.5 Don’t know 14.3 Missing 1.3 Condoms Decrease the Risk of HIV/AIDS Not at all 11.0 A little 25.3 A lot 36.4 Completely 8.4 Don’t know 13.6 Missing 5.2 24 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Percentage of Treatment Group Students Birth Control Pills Decrease the Risk of Pregnancy Not at all 1.9 A little 20.8 A lot 44.8 Completely 11.7 Don’t know 19.5 Missing 1.3 Birth Control Pills Decrease the Risk of HIV/AIDS Not at all 55.2 A little 8.4 A lot 7.8 Completely 3.9 Don’t know 19.5 Missing 5.2 b Sample Size 154 Source: Youth surveys administered by the PPA evaluation team in June and July 2012. a Other languages spoken at home include Korean and Patois. b Indicates number of students who completed the baseline survey. The sample sizes for each variable differ due to item nonresponse and logical skips, and ranged from 121 to 154. 25 This page has been left blank for double-sided copying. APPENDIX B IMPLEMENTATION STUDY DATA SOURCES AND METHODOLOGY This page has been left blank for double-sided copying. Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Three data sources provided the information for this report: (1) site visits, (2) fidelity and performance monitoring data, and (3) survey data (a baseline survey administered by Mathematica, pre- and post-tests administered to facilitators before and after they received training on the Gen.M curriculum, and a participant satisfaction survey, administered to youth participating in Gen.M sessions on the last day of the workshop). Site Visits Two researchers each conducted a site visit (for a total of two site visits) to Austin to collect in-depth data on: (1) the planned intervention, (2) adherence to the planned intervention, (3) delivery of the facilitator training and curriculum, (4) participants’ responsiveness to the curriculum, and (5) successes and challenges encountered during program implementation. During the site visits, which took place in August 2012, the researchers (1) conducted in-person interviews with staff from EngenderHealth, Columbia University, SafePlace, and the Travis County Summer Youth Employment Program (SYEP); (2) conducted a focus group discussion with 12 facilitators; and (3) observed three facilitated sessions of the curriculum. In July and August 2012, a Columbia graduate student conducted interviews with 12 facilitators (six interviews with two facilitators each) who taught the first workshop and 24 Gen.M participants (24 separate interviews) who completed the second workshop. Table B.1 details the sources for the data collected, the time period during which these data were collected, and topics covered. Analysis Approach. Qualitative analysis of the site visit data involved an iterative process using thematic analysis and triangulation of data sources (Patton 2002; Ritchie and Spencer 2002). Because of the number of interviews conducted, we used a qualitative analysis software package, Atlas.ti (Scientific Software Development 1997), to facilitate organizing and synthesizing the qualitative data. First, we developed a coding scheme for the study, organized according to key research questions. Within each question, we defined codes for key themes and subtopics we expected to cover in the interviews. Then, we applied the codes to passages in the interview and focus group notes. To ensure accurate and consistent coding, an analyst and a research assistant/programmer independently coded site visit data, and a researcher (a member of the site visit team) reviewed the coded documents and reconciled any differences in coding. To address the research questions, we used the software to retrieve relevant passages, and then examined the patterns of responses across respondents and identified themes emerging from the responses. Fidelity and Performance Measure Data To determine whether facilitators adhered to the planned time line and duration of lessons, and followed the prescribed scope and sequence of lessons, we analyzed fidelity and observation data, which EngenderHealth provided to Mathematica. 29 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Table B.1. Data Sources Topic Areas Adherence to Challenges Planned Training Planned Participants’ and Data Source Number Date(s) Context Intervention and TA Intervention Responsiveness Successes In-Person Interviews EngenderHealth staff 3 Aug. 2012 X X X X X X Columbia graduate student 1 Aug. 2012 X X X X Oct. 2011, SYEP staff 2 X X Aug. 2012 SafePlace staff 1 Aug. 2012 X X X SafePlace facilitators 12 (6 groups) July 2012 X X X X Gen.M participants 24 July, Aug. 2012 X X Focus Group Discussions 30 SafePlace facilitators 12 (1 group) Aug. 2012 X X X X X Classroom Observations Classroom observations 3 Aug. 2012 X X Fidelity and Monitoring Program fidelity logs 39 July, Aug. 2012 X Program observation logs 7 July, Aug. 2012 X X Surveys Facilitator training pre-tests 15 Apr. 2012 X Facilitator training post-tests 15 Apr. 2012 X Facilitator survey 24 July, Aug. 2012 X X X Participant satisfaction 136 July, Aug. 2012 X X surveys TA = technical assistance. Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Program Fidelity Logs. Observers from EngenderHealth and Columbia completed fidelity logs for each of the sessions they observed. (An example of the fidelity log forms used to monitor program delivery is in Appendix D.) In these logs, observers reported on the number of activities scheduled and completed in each session, and any changes made to activities or reasons for non-completion of activities. During the first cohort, EngenderHealth staff planned to observe each session conducted by the six facilitator pairs (for a total of 30 observed sessions), and succeeded in observing all but one session. Most (24) of the facilitated sessions in the first workshop had one observer; four sessions had two observers. During the second workshop, the observation schedule was revised such that observations were spaced out (not all sessions were observed) and there was only one observer in a classroom. A total of nine observations were conducted in the second workshop. Program Observation Logs. In addition to monitoring activity completion in the fidelity logs, EngenderHealth and Columbia staff monitored seven sessions for seven pairs of facilitators to report on adherence to the planned intervention, quality of the observed session, facilitators’ comfort with the material, facilitator-youth interactions, and the engagement and receptiveness of youth to the material (An example of the program observation log form is in Appendix E). Analysis Approach. We established implementation fidelity benchmarks for the Gen.M program based on the theory of change and available data from program fidelity and observation logs. Information on the benchmarks varied in completeness. The primary implementation fidelity benchmark, based on data available from the 39 program fidelity logs, was whether sessions and activities were delivered in the correct order, in the time allotted, and as prescribed—and, if not, why. Secondary benchmarks, based on data available from seven observation logs, included whether: (1) facilitators created a comfortable workshop environment, (2) facilitators had good knowledge of the program and were able to communicate session goals effectively (observation logs), and (3) facilitators answered youths’ questions effectively in the workshop (observation logs). Because more data were available from the program fidelity logs (for all of the sessions in the first cohort and about a third of the sessions in the second cohort), while data from program observation logs were limited (available for only 12 percent of the sessions across both cohorts), our assessment of implementation fidelity was largely determined by a tabulation of site observers’ assessments of whether sessions and activities were delivered in the correct order and as prescribed. We found that facilitators clearly hit this benchmark, and implemented the observed sessions in the correct order and as prescribed, with minor (green-light) adaptations. Although data for the remaining three benchmarks were more limited, tabulations of these data consistently showed that facilitators were able to create comfortable environments, had good knowledge of the program, and answered questions effectively. Because the data on these secondary benchmarks were consistent with the data from the primary benchmark, we concluded that, in summer 2012, Gen.M was implemented with fidelity to the implementation plan. Survey and Administrative Data Population Served. Data on the population served by the intervention were gathered from several sources. The baseline instrument collected data on demographic and background characteristics, risk-taking behavior, previous receipt of sex education, and knowledge and attitudes toward sexual activity and contraceptive use of consented youth. It was administered to consented youth in June and July 2012; the data in this report are from the 154 youth who 31 Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas participated in Gen.M and completed the baseline survey. Data on youth attendance in the Gen.M workshops came from EngenderHealth. Training. Data on reactions to the week-long curriculum and facilitator training provided by EngenderHealth were gathered from a pre- and post-test. These surveys collected information from facilitators about their understanding of the key curriculum messages and importance of fidelity, the usefulness of the training, their level of confidence in their ability to facilitate the curriculum, and suggestions for improvement of the training. The pre-test was administered to 15 facilitators at the start of the first day of training. The post-test was administered to the same 15 facilitators at the end of the fifth, and final, day of training. Facilitator Feedback. Data on facilitators’ satisfaction with the Gen.M curriculum were gathered from a facilitator survey administered at the end of the two cohorts in July and August 2012. The survey collected data on facilitators’ level of confidence and effectiveness facilitating workshops, and assessment of the workshops. Participant Satisfaction. Data on participant satisfaction with the Gen.M curriculum were gathered from a participant survey administered to 136 youth at the end of the two workshops in July and August 2012. The survey collected data on participants’ feelings about their level of engagement, the facilitators who led the sessions, and other participants in their sessions. The survey also collected data on participants’ likelihood to use a condom and abstain from sex, and changes in their opinions about gender behavior. 32 APPENDIX C SUMMARY OF GEN.M LESSONS This page has been left blank for double-sided copying. Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Table C.1. Summary of Gen.M Lessons Session Session Description Materials Activity Lengtha Methods Activity Objectives 1. Understanding Gender Helps youth become aware of, • Markers, pens, pencils, 1.1 Welcome and 60 • Minilecture • Identify names of other participants in question, and redefine gender masking tape, three overview minutes • Large-group the group norms in ways that build balls discussion • Explain Gen.M goals equitable relationships and • Flip chart, box for index • Ice breaker • Agree upon shared norms for promote well-being. cards • Game participating in the group • Participant handouts • Video camera, microphone 1.2 Values 25 • Forced choices • Examine individual attitudes about clarification minutes • Large-group gender differences, roles, double discussion standards, and inequalities • Question how individual attitudes about gender affect behaviors 1.3 Gender 70 • Minilecture • Describe the difference between sex, messages minutes • Brainstorming gender, and sexual orientation • Critical thinking • Identify: at least three gender • Large-group messages that define acceptable discussion gender roles for both men and women in U.S. society; at least two messages for each gender that are harmful; at least one way that harmful gender messages contribute to increasing risk for unintended teen 35 pregnancy; at least one way that promoting positive and equitable gender messages can reduce teen pregnancy 1.4 Gender in the 50 • Minilecture • Analyze images and messages about media minutes • Small-group work gender that are perpetuated in (analysis of popular American culture images) • Explain how harmful gender • Large-group messages portrayed in the media can discussion negatively affect young men and women 1.5 Video review: 20 • Video review • Identify at least one progressive it’s about me minutes message about gender that participants want to embrace Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Session Session Description Materials Activity Lengtha Methods Activity Objectives 2. Healthy Relationships Helps youth understand the • Markers, pens, pencils, 2.1 Check-in 20 • Minilecture • List Session 2 agenda items characteristics of healthy and masking tape, boxes minutes • Ice breaker • Describe how to sign on to the Austin unhealthy relationships and • Flip chart, box for index Gen.M Facebook group page builds skills to ensure that their cards own relationships are fulfilling, • Participant handouts enjoyable, and healthy. • Participant incentives • Laptop or tablet computer with internet access • Video camera, microphone 2.2 Healthy 50 • Brainstorming • Name healthy and unhealthy relationships minutes • Large-group sort behaviors that exist within and deal • Large-group relationships breakers discussion • State important characteristics of a • Individual work healthy relationship (self-assessment) • State a deal breaker behavior that would cause participants to end a relationship • Describe steps to guide a person when faced with unhealthy behaviors in a relationship 2.3 Assertive 65 • Minilecture • Describe the difference between communication minutes • Demonstration of passive, aggressive, and assertive 36 refusal skill communication styles • Role plays of • List the five steps that can be used in refusal skill an effective refusal • Large-group • Demonstrate effective refusal to discussion unwanted sex in a role play 2.4 What is 50 • Minilecture • Define the concept of sexual consent consent? minutes • Individual analysis • Apply the definition of consent to (scenario practical, real-life situations analysis) • Identify strategies to establish • Large-group consent for sexual activity brainstorm and • Identify strategies for respecting a discussion partner’s sexual limits • Identify how gender norms influence people’s ability to ask for consent and to respect a partner’s sexual limits 2.5 Video review: 40 • Handout with • Demonstrate understanding of creative minutes questions concepts expressions • Video Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Session Session Description Materials Activity Lengtha Methods Activity Objectives 3. Big Decisions Helps youth understand the • Markers, masking tape 3.1 Check-in 25 • Minilecture • List Session 3 agenda items challenges of being a teen • Flip chart, box for index minutes • Ice breaker parent and build skills in cards (simulation) making decisions about sexual • Participant handouts activity. • Participant incentives Video camera, microphone 3.2 Life changes 65 • Survey • List the ways that pregnancy would minutes • Brainstorming affect participants’ lives • Small-group work • Identify how boys and girls might • Large-group experience parenthood similarly and discussion differently • Individual work • Describe how participants would feel (letter to parent) about becoming pregnant and how they would tell their parents/guardian 3.3 Sexual decision 70 • Brainstorming • Identify reasons why some teens making minutes • Small-group work choose to have sex and some (case studies) choose not to have sex • Large-group • Make decisions about engaging in discussion sexual activity • Individual • Set personal limits around sexual reflection activity 3.4 Ways to show 25 • Brainstorming • List alternatives to engaging in sex 37 you care minutes • Large-group discussion 3.5 Video review: 40 • Gender fishbowl • Describe at least two significant gender fishbowl minutes • Video gendered experiences participants’ opposite-sex peers have • Describe at least two significant gendered experiences that participants and their same-sex peers typically have • Identify at least one thing participants can do to better support their peers around resisting and/or changing harmful gender norms Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Session Session Description Materials Activity Lengtha Methods Activity Objectives 4. Skills for Preventing Teaches youth about • Markers, masking tape, 4.1 Check-in 30 • Minilecture • Understand the importance of talking Pregnancy pregnancy and STIs and builds art supplies minutes • Ice breaker about what participants want when it their skills in preventing both • Flip chart, box for index comes to issues related to abstinence through the consistent and cards and sexual activity correct use of condoms. • Penis models, condoms, paper towels, hand sanitizer • Participant handouts • Participant incentives Video camera, microphone 4.2 Keeping the egg 20 • Minilecture (with • Identify and describe basic elements and sperm apart minutes anatomy and of female and male reproductive physiology flip anatomy and physiology chart) • Describe how fertilization and pregnancy occur 4.3 Burning 15 • Large-group • Identify basic information about STIs questions about minutes discussion and ways to prevent acquiring STIs STIs 4.4 How to use 45 • Condom lineup • List, in order, the eight steps to condoms minutes • Demonstration of correctly use a condom condom use • Demonstrate the correct use of a • Condom skill condom on a model 38 practice • Identify reasons for incorrect condom • Large-group use discussion • Express greater familiarity and comfort in using a condom • Identify reasons why teens do not 4.5 Condom 30 • Brainstorming consistently use condoms obstacles minutes • Small-group work • Generate ideas on how to overcome • Gallery walk barriers to condom use • Large-group • Assess how different barriers may be discussion influenced by factors related to gender 4.6 Negotiating 40 • Minilecture • Demonstrate effective negotiation of condom use minutes • Demonstration of condom use negotiating skill • Role play of negotiating skill • Large-group discussion 4.7 Video review: 45 • Small-group • Compose a slogan that emphasizes condom slogan minutes artwork (condom the importance of using condoms slogans) • Describe at least two reasons for • Video using condoms Balancing Fidelity and Flexibility: Implementing the Gen.M Program in Texas Session Session Description Materials Activity Lengtha Methods Activity Objectives 5. Taking Action to Prevent Teaches youth about the most 5.1 Check-in 25 • Minilecture • List Session 5 agenda items Teen Pregnancy widely accessible hormonal minutes and long-acting contraceptives • Ice breaker and where to obtain them. Asks youth to identify personal behaviors that they intend to sustain or change in order to prevent pregnancy. 5.2 Birth control 50 • Brainstorming • Describe how contraceptive methods scavenger hunt minutes • Small learning are used and the advantage of using groups them • Large-group • Identify possible barriers to using discussion contraception and possible solutions to overcoming these barriers • Identify ways that men can support the consistent and correct use of female-focused contraceptives 5.3 The clinic 40 • Brainstorming • State location of a teen clinic minutes • Clinic telephone • Describe the services provided at a calls family-planning clinic • Minilecture • Demonstrate how to call a family- • Large-group planning clinic and make an discussion appointment 39 5.4 Game show 50 • Group game • Recall important information from all review minutes five workshop sessions 5.5 Video review: 40 • Individual work • Identify one value or belief that has making a minutes (commitment changed as a result of the workshop commitment worksheet) • Cite one behavior that participants • Large-group intend to change as a result of the discussion workshop • Video • Describe one action that participants intend to take to prevent teen pregnancy 10 • Minilecture 5.6 Future Gen.M minutes • Describe the text, Facebook, and activities movie premiere components of the Gen.M program • Sign on to the Gen.M Facebook page 5.7 Closing activity: 10 • Spider web • Express appreciation for fellow spider web minutes participants • Recognize positive characteristics of fellow participants • Receive affirmation from fellow participants Source: “Gen.M: A Gender Transformative Teenage Pregnancy Prevention Curriculum – Draft.” New York: Engender Health, June 2012. a Each session is 4 hours long, with 3 hours and 75 minutes of scheduled programming and 15 minutes for breaks between activities. STI = sexually transmitted infection. This page has been left blank for double-sided copying. APPENDIX D GEN.M FIDELITY MONITORING LOG FORM This page has been left blank for double-sided copying. Curriculum Fidelity Monitoring Log Your Name: _________________________ Co-facilitator’s Name: _________________________ Workshop session dates: _________________________ PURPOSE OF THIS LOG The curriculum fidelity monitoring log assesses whether the core components of the Gender Matters curriculum have been fully implemented as written, and to gathers information about any changes made to the curriculum during implementation. DIRECTIONS Please complete the appropriate section for each workshop day after you have facilitated that day. It is best to complete the form right after the workshop to minimize recall errors. For each of the activities in the session, please indicate whether you have completed it as described in the curriculum, made any changes, or did not complete the activity. Describe any changes that you have made to the activity, however small you feel they may have been (i.e., using pairs instead of small groups or eliminating discussion questions due to lack of time). You will submit the log to project staff at the completion of the entire workshop week. Group: _______ Observer Name: _________________________ GENM WORKSHOP SESSION ONE: UNDERSTANDING GENDER Facilitator Names: ____________________________ Date: __________ # of participants __________ Did you complete each activity below? Activity 1: Activity 2: Activity 3: Activity 4: Activity 5: Welcome and Overview Values Clarification Gender Messages Gender in the Media Video Review – Gender Fishbowl □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ No □ No □ No □ No □ No If you made and changes please describe them here, or if you did not complete an activity, please describe why here. _____ total # of activities completed out of 5 _____ total # of activities not completed Group: _______ Observer Name: _________________________ GENM WORKSHOP SESSION TWO: HEALTHY RELATIONSHIPS Facilitator Names: ____________________________ Date: __________ # of participants __________ Did you complete each activity below? Activity 1: Activity 2: Activity 3: Activity 4: Activity 5: Session 2 Check-in Healthy Relationships Assertive Communication What is Consent? Video Review – and Deal Breakers Creative Expressions □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ No □ No □ No □ No □ No If you made and changes please describe them here, or if you did not complete an activity, please describe why here. _____ total # of activities completed out of 5 _____ total # of activities not completed Group: _______ GENM WORKSHOP SESSION THREE: BIG DECISIONS Observer Name: _________________________ Facilitator Names: ____________________________ Date: __________ # of participants __________ Did you complete each activity below? Activity 1: Activity 2: Activity 3: Activity 4: Activity 5: Session 3 Check-in Life Changes Sexual Decision-Making Ways to Show You Care Video Review – Bag of Sentences □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ No □ No □ No □ No □ No If you made and changes please describe them here, or if you did not complete an activity, please describe why here. _____ total # of activities completed out of 5 _____ total # of activities not completed Group: _______ Observer Name: ______________________ GENM WORKSHOP SESSION FOUR: SKILLS FOR PREVENTING PREGNANCY Facilitator Names: ____________________________ Date: __________ # of participants __________ Did you complete each activity below? Activity 1: Activity 2: Activity 3: Activity 4: Activity 5: Activity 6: Activity 7: Session 4 Check-in Keeping the Egg Burning Questions How to Use Condom Obstacles Negotiating Video Review – and Sperm Apart About STIs Condoms Condom Use Condom Slogan □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ No □ No □ No □ No □ No □ No □ No If you made and changes please describe them here, or if you did not complete an activity, please describe why here. _____ total # of activities completed out of 7 _____ total # of activities not completed Group: _______ Observer Name: _________________________ GENM WORKSHOP SESSION FIVE: TAKING ACTION TO PREVENT TEEN PREGNANCY Facilitator Names: ____________________________ Date: __________ # of participants __________ Did you complete each activity below? Activity 1: Activity 2: Activity 3: Activity 4: Activity 5: Activity 6: Activity 7: Session 5 Check-in Birth Control The Clinic Game Show Video Review – Future Gen.M Closing Spider Web Scavenger Hunt Review Making a Activities Commitment □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes Completely □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ Yes with changes □ No □ No □ No □ No □ No □ No □ No If you made and changes please describe them here, or if you did not complete an activity, please describe why here. _____ total # of activities completed out of 7 _____ total # of activities not completed This page has been left blank for double-sided copying. APPENDIX E GEN.M OBSERVATION LOG FORM This page has been left blank for double-sided copying. Program Observation & Quality Assessment Form Introduction The purpose of this observation form is to measure the fidelity and quality of program implementation. Please use the guidelines below when completing the observation form and do not change the scoring provided; for example, do not circle multiple answers or score a 1.5 rather than a 1 or a 2. You should complete the observation form after viewing the entire session. Prior to observation, you should read through the questions to become familiar with them. It is also helpful to take notes during your observation; for example, for Question 1, each time an implementer gives explanations, place a checkmark next to the appropriate rating. Observation and Assessment Form At-A-Glance A quick guide to what you are watching for during presentations and scoring afterwards 1. Facilitators’ preparation for the session 2. Facilitators’ explanation of activities 3. Facilitators’ ability to keep track of time 4. Presentation of materials seemed rushed or hurried 5. Participants’ appear to understand the material 6. Group members’ participation in discussions and activities 7. Facilitators manage youth participation 8. Facilitators’ use the space around them 9. Facilitators support and encourage youth 10. Facilitators a. Knowledge of the program b. Ability to communicate session goals c. Level of enthusiasm d. Poise and confidence e. Rapport and communication with participants f. Effectively addressed questions/concerns g. Provided a welcoming atmosphere h. Effectively transitioned from one activity to another i. Effectively managed the emotional climate Observer: Facilitators: Observation Date: Group #: Component / Session Number: Instructions The following questions assess the overall quality of the program session and delivery of the information. Use your best judgment and do not circle more than one response. 1. How prepared were the facilitators for the session? 1 2 3 4 5 Not prepared Somewhat prepared Very prepared 1- Materials not ready or not available, facilitators did not arrive early, facilitators were not familiar with activities 3- Some materials ready, facilitators not fully prepared to start session on time, facilitators somewhat familiar with 5- activities Materials ready and set up for activities, facilitators early to workshop, facilitators very familiar with activities 2. In general, how clear were the program facilitators’ explanations of activities? Q1 1 2 3 4 5 Not clear Somewhat clear Very clear 1- Most participants did not understand instructions and could not proceed, many questions asked 3- About half of the group understood, while the other half asked questions for clarification 5- 90-100% of the participants began and completed the activity/discussion with no hesitation and no questions 3. To what extent did the facilitators keep track of time during the session and activities? Q2 1 2 3 4 5 Not on time Some loss of time Well on time 1- Facilitator did not have time to complete the material (particularly at the end of the session), regularly allowed discussions to drag on (e.g., participants seem bored or began discussing non-related issues in small groups) 3- Misses a few points, sometimes allowed discussions to drag on 5- Completed all content of the session, completed activities and discussions in a timely manner (using the suggested time limitations in the program manual, if available 4. To what extent did the presentation of materials seem rushed or hurried? Q3 1 2 3 4 5 Very rushed Somewhat rushed Not rushed 1- Facilitators didn’t allow time for discussion, didn’t have time for examples, told participants that they were in a hurry, body language suggested stress or hurry 3- Some deletion of discussion/activities, sometimes stated but does not explain material 5- Didn’t rush participants or speech but still completed all the materials, appeared relaxed 5. To what extent did the participants appear to understand the material? Q4 1 2 3 4 5 Little understanding Some understanding Good understanding Use your best judgment based on participant conversations and feedback. Roughly: 1 - Less than 25% seemed to understand; 3 - About half; 5 - 75-100% understood 6. How actively did the group members participate in discussions and activities? Q5 1 2 3 4 5 Little participation Some participation Active participation Use your best judgment based on listening to the discussions and feedback. Roughly, 1 - Less than 25% participated; 3 - About half participated; 5 - 75-100% participated 7. How well did the facilitators manage youth participation? 1 2 3 4 5 Poor Average Excellent 1 - Did not elicit participation from quiet youth, did not manage dominant talkers, group was loud and unfocused 5 - Able to draw out quiet youth, manage dominant talkers, group remained focused and on task 8. How well did facilitators manage and use the space around them? 1 2 3 4 5 Poor Average Excellent 1 - Visual aids and flipcharts difficult to see or absent, GM declaration and group norms not posted visibly, work space not conducive to learning, facilitators sat to facilitate activities, facilitators not physically present during activities 5 - Visual aids and flipcharts easy to see, effectively used room (in small groups, large seating, discussions), tables and desks used when needed, facilitators shared the space equally, facilitators limited sitting and sat only when appropriate, facilitators were present during activities 9. How well did the facilitators support and encourage youth. 1 2 3 4 5 Poor Average Excellent 1 – During activities, facilitators not actively involved with youth, did not support contributions or accomplishments of youth, very limited use of open-ended questions 3 – Limited use of open-ended questions, used subjective or evaluative language (I like it, you’re smart) 5 – During activities, facilitators were actively involved with youth, supported contributions or accomplishments with specific and nonevaluative language (I can tell that you put a lot of thought into your project), frequently used open-ended questions 10. Using the scale provided, rate the facilitators on the following qualities: a) Knowledge of the program Q6a 1 2 3 4 5 Poor Average Excellent 1 – Could not answer questions, mispronounced names; read from the manual 5 - Provided information above and beyond what’s in the manual, seemed very familiar with the concepts, answered questions with ease b) Ability to communicate goals and “key points” of the session 1 2 3 4 5 Poor Average Excellent 1 – Didn’t state the goals of the session, didn’t summarize key messages for activities, or they were unclear 5 – Clearly stated the goals of the session, provided key take away messages for each activity c) Level of enthusiasm Q6b 1 2 3 4 5 Poor Average Excellent 1 - Presented information in a dry and boring way, lacked personal connection to material, appeared “burned out” 5 - Made clear that the program is a great opportunity, got participants talking and excited, was outgoing d) Poise and confidence Q6c 1 2 3 4 5 Poor Average Excellent 1 - Appeared nervous or hurried, did not have good eye contact 5 - Did not hesitate in addressing concerns, was well organized, not nervous e) Rapport and communication with participants Q6d 1 2 3 4 5 Poor Average Excellent 1 – Didn’t remember names, did not “connect” with participants, acted distant or unfriendly 5 - Got participants talking and excited, very friendly, used people’s names when appropriate, seemed to understand the community and its needs f) Effectively addressed questions/concerns Q6e 1 2 3 4 5 Poor Average Excellent 1 - Engaged in “power struggles,” responded negatively to comments, gave inaccurate information, didn’t direct participants elsewhere for further information if needed 5 - Answered questions of fact with information and questions of value with validation, if didn’t know the answer was honest about it and directed them elsewhere g) Provided a welcoming atmosphere 1 2 3 4 5 Poor Average Excellent 1 – Did not greet youth, little to no eye contact, arms crossed, stiff instead of animated body language, negative tone of voice, disrespectful language 5 – Greeted all youth, made eye contact with youth when addressing them, smiled, showed open body language, stands by youth to quiet them, circles the room during group work, used warm tone of voice, used respectful language h) Effectively transitioned from one activity to another 1 2 3 4 5 Poor Average Excellent 1 – Did not use transitions or transitions were unclear and confusing. 5 – Used transitions to connect one activity to another, transitions were smooth and clear. i) Effectively managed the emotional climate 1 2 3 4 5 Poor Average Excellent 1 – Emotional climate was negative, negative behaviors were not mediated, comments with bias were not addressed 5 – Emotional climate was positive, respectful and fun; negative behaviors and biased comments were mediated, or there was no evidence of bias but rather a mutual respect and inclusion of others 11. Rate the overall quality of the program session. Q7 1 2 3 4 5 Poor Average Excellent Summary measure of all the preceding questions - assesses the extent of material covered and performance of the facilitator. Excellent sessions look like: • Participants are doing rather than talking about activities • Non-judgmental responses to questions • Answering questions of fact with information, questions of value with validation • Good time management and well organized • Adequate pacing—not too fast and did not drag • Using effective checks for understanding. • Supportive and encouraging towards youth. Poor sessions look like: • Lecture-style presenting of content • Reading the content from the notebook • Stumbling along with the content and failing to make connections to what has been discussed previously or what participants are contributing. • Uninvolved participants • Getting into power struggles with participants about the content. • Judgmental responses • Flat affect and boring style • Unorganized and random • Loses track of time. 12. Please note at least one major strength of the session and/or facilitators’ delivery of the material.