02

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/285392953

Schools as venues for prevention programming

CHAPTER · DECEMBER 2015




READS
26


4 AUTHORS, INCLUDING:

Kevin Sutherland
Virginia Commonwealth Uni…
46 PUBLICATIONS  681 CITATIONS


Katherine A Taylor
Virginia Commonwealth Uni…
11 PUBLICATIONS  22 CITATIONS







Available from: Kevin Sutherland Retrieved on: 16 January 2016

Chapter 10
Schools as Venues for Prevention Programming

Terri N. Sullivan, Kevin S. Sutherland, Albert D. Farrell and Katherine A. Taylor






Schools are frequently the setting for the implementation of programs focused on reducing the frequency of problem behaviors such as aggression, substance use, and truancy (Botvin et al. 2006; Espelage et al. 2013; MVPP 2009; Vo et al. 2012). The high prevalence rates and negative outcomes associated with risk behaviors among youths demonstrate the need for prevention programs. For example, annual preva- lence rates for alcohol and marijuana use in a national survey of 8th, 10th, and 12th graders conducted in 2012 were 44 and 25 %, respectively (Johnston et al. 2013). Results from a national survey of high school students conducted in 2011 indicated that in the past 12 months, 33 % of students had been in a physical fight and 20 % had been bullied on school grounds (Centers for Disease Control and Prevention 2011). Negative consequences of risk behaviors may include physical injury, sub- stance abuse, poor academic performance, and early school leaving (Guerra and Bradshaw 2008). These statistics highlight the potential benefits of researchers and schools forming strong partnerships to effectively address risk behaviors.
Schools represent a particularly appropriate setting for implementing universal prevention programs. From a practical standpoint, they enable access to large num- bers of students from early childhood to late adolescence and provide an opportuni- ty to follow students over time (Farrell et al. 2001). Moreover, school staff members are trained to support youth development, and the public generally supports this


T. N. Sullivan () · A. D. Farrell · K. A. Taylor
Department of Psychology, Virginia Commonwealth University, 810 West Franklin Street,
Richmond, VA 23284-9045, USA
e-mail: tnsulliv@vcu.edu
A. D. Farrell
e-mail: afarrell@vcu.edu
K. A. Taylor
e-mail: taylorka7@vcu.edu
K. S. Sutherland
Special Education and Disability Policy, School of Education, Virginia Commonwealth University, 1015 West Main Street, Richmond, VA 23284-9045, USA
e-mail: kssuther@vcu.edu
© Springer Science+Business Media New York 2015
K. Bosworth (ed.), Prevention Science in School Settings,
Advances in Prevention Science, DOI 10.1007/978-1-4939-3155-2_10

















201


focus (Gottfredson 2001). Although students spend only about 18 % of their waking hours at school, the school context has a powerful influence on their development (Gottfredson and Gottfredson 2007). Peer groups typically form within the school context and often exert their influence during the school day. Schools may also place youths in contact with peers who have a variety of backgrounds and different values. Depending on the norms and attitudes of peers within various groups, peer influences may either encourage or discourage risk behaviors. Within the school en- vironment, informal social norms may also support engagement in problem behav- iors; for example, using aggression as a means to gain status and correct perceived injustices (Fagan and Wilkinson 1998). Other risk factors may also be present with- in the school environment. For example, although the most serious incidents of victimization (such as homicides) typically occur outside of school, youths are at elevated risk for experiencing a broader range of criminal victimization (e.g., theft, assault) when they are at school or on their way to and from school (Gottfredson and Gottfredson 2007). As Gottfredson (2001) noted, some of the likely causes of problem behavior at school are related, and many of these factors can be altered to reduce students’ involvement in these behaviors.


10.1 The Focus of School-Based Prevention Efforts

School-based prevention programs have the potential to influence multiple levels of the social systems that influence students’ behavior. Programs focused on indi- vidual students attempt to increase students’ knowledge base and enhance skills such as communication, problem-solving, and emotion management that have been associated with specific risk behaviors (Botvin et al. 2006; Espelage et al. 2013; Farrell et al. 2001). For example, Second Step: Student Success Through Prevention is a violence prevention program for middle school students focused on enhancing empathy, communication, problem-solving, and anger management skills (Espel- age et al. 2013). Another example of an individual-level skill-building program is the Life Skills Training Program, which addresses substance use prevention for elementary to high school students (Botvin et al. 2006). The Life Skills Training Program aims to improve youths’ self-esteem and skills related to stress and coping, emotion management, resistance against peer and media pressure to use drugs, and social competence (Botvin and Tortu 1988). Implementing these types of programs in school settings provides students opportunities to learn, model, and apply tar- geted skills with peers on a day-to-day basis. Improvements in social and emotional learning may have broader benefits, as was documented by a recent meta-analysis showing that school-based interventions that concentrate on social and emotional learning significantly increased academic achievement (Durlak et al. 2011).

Programs that focus on the classroom level are designed to improve teachers’ classroom management skills and effectiveness in dealing with students’ disruptive or aggressive behavior, which can negatively affect academic success and class- room climate (van Lier et al. 2004; Vo et al. 2012). As an example, BEST in CLASS



is a manualized program designed for preschool children at risk for emotional or behavioral disorders. This program uses a teacher training and coaching model to provide teachers direct assistance in learning and effectively implementing instruc- tional and classroom management skills with the goal of fostering prosocial behav- ior and engagement in learning (Vo et al. 2012). The PAX Good Behavior Game focuses more generally on creating a positive and prosocial classroom, and thereby decreasing off-task and disruptive or aggressive behaviors. Students develop ex- pectations for classroom behavior with their teacher, then the team of students who best meets these expectations wins the game and earns a reward. This program also incorporates evidence-based behavioral strategies designed to increase students’ en- gagement in learning (Embry 2003). Prevention programs focused on the classroom level may not only improve students’ behavior and academic achievement but also enhance positive relationships with peers and with teachers.
Prevention programs targeting the school environment often aim to create a warm and responsive school climate with high expectations for prosocial behavior and academic achievement (Embry et al. 1996; Olweus 2004). A key focus is on fostering positive relationships. Recognition of the transactional nature of social interactions among students and teachers, and of the effect of these interactions    in supporting prosocial behavior, is central to achieving a positive school climate (Conroy et al. 2009). School environment programs may implement interventions at multiple levels of the school’s social system. For example, the Olweus Bullying Prevention Program (Olweus 2004) includes a component to recognize individual students’ prosocial behaviors and address individual bullying behaviors; a class- room-level component to foster positive student–student and student–teacher rela- tionships; and a school-level component focused on adults’ consistent implementa- tion of school rules, an “on-the-spot” intervention to address bullying behaviors, and awareness and monitoring of “hot spots” where these behaviors are most likely to occur. A common thread woven through each of these components is parental involvement. Efforts to address broader influences through community outreach are also included. Ideally, such a comprehensive approach creates a protective environ- ment that supports adaptive development and deters risk behaviors.


10.2 Complexities in Implementing Prevention Programs in Schools

Regardless of the specific focus of a school-based prevention program, schools present unique implementation challenges for researchers. Schools are complex sys- tems containing youths with diverse needs, and school contexts are often changing and unpredictable (Forman et al. 2013). Researchers have found that the quality of implementation of prevention programs in schools is often low (Durlak 2010; Dur- lak and DuPre 2008; Forman et al. 2013; Gottfredson and Gottfredson 2002). This is unfortunate in that high-quality implementation (e.g., the implementer’s skills and completeness of delivery) appears necessary for positive outcomes (Durlak and


DuPre 2008; Wilson et al. 2003). The replication and sustainability of research- based prevention programs in day-to-day practice also remain a concern. Variations in both the quality of implementation and in outcomes associated with evidence- based prevention programs have contributed to an increased focus on implementa- tion science, which seeks to explore and explain how and why interventions work in real-world contexts (Kelly and Perkins 2012).
In the following sections, we use the health promotion intervention life cycle (Bopp et al. 2013) as the framework for addressing several aspects of partnering with schools as venues for prevention programs. This model is divided into three phases: (a) program development, adoption, and implementation; (b) sustainability, institutionalization, or termination; and (c) diffusion and dissemination. For this chapter, we focus primarily on the first and second phases, as diffusion and dis- semination are addressed in Chap. 9. It is important to note that although we present phases of the model in a linear fashion, they are actually iterative in nature. For example, decisions made in the development and implementation of a school-based prevention program directly inform its sustainability, and factors that may enhance sustainability should be considered during development and implementation.


10.3 Program Development and Adoption

A key initial step for school staff and researchers is identifying the most appropriate prevention program for a particular school or set of schools. This involves consid- eration of characteristics of both the intervention and the setting where it will be implemented. The prevention program needs to be evidence-based, culturally and developmentally relevant, and focused on key risk and protective factors for the risk behavior being targeted within the student population. The capacity or readiness of the school to support the prevention program must also be considered. In this sec- tion, we discuss three issues that inform both program development and  adoption:
(a) identifying the needs of the schools, (b) selecting the most appropriate interven- tion, and (c) assessing the school’s capacity or readiness for the program.


10.3.1 Identifying the Needs of the Schools

A key consideration in selecting a prevention program is how well it addresses   the needs of the school. Interventions differ in their goals, the specific risk and protective factors they target, their intended population, the intervention strategies they use, and the resources they require for successful implementation (Farrell and Vulin-Reynolds 2007). Establishing a match between these features and the needs and characteristics of a particular school is critical to ensuring an intervention’s success. School interventions are not likely to be successful without the support of teachers and other school staff, and such support is not likely to be present without


a clear sense that the intervention addresses an important need. A logical starting point may involve conducting a needs assessment to identify the key concerns of teachers, school administrators, and other stakeholders, or collecting data that iden- tify problems within the school (e.g., truancy rates, student reports of drug use). Specific goals might involve reducing a specific problem behavior (e.g., aggression, drug use, early school leaving), addressing a constellation of problem behaviors, or promoting positive development.


10.3.2 Selecting the Most Appropriate Intervention Approach

Interventions are not likely to have their desired impact if they do not address the factors responsible for the development or maintenance of the problem they are attempting to change (Coie et al. 1993). These factors are likely to vary across schools. Support for this notion was provided by a recent literature review that iden- tified student (e.g., gender, initial level of aggression, parental monitoring), school (e.g., norms for aggression), and community (e.g., level of poverty and crime) char- acteristics that moderate the impact of school-based violence prevention programs (Farrell et al. 2013). Thus, the selection of appropriate prevention strategies must take into account the dynamics of the student population and the school and their specific profile of risk and protective factors (Farrell and Vulin-Reynolds 2007), as different risk factors may operate in different socio-ecological contexts. This real- ity underscores the need for assessing levels of risk and protective factors present within a given school and using this information to identify relevant intervention strategies. The Communities That Care model, which involves selecting interven- tions based on their match to data obtained from a community assessment of risk factors, provides an excellent example of this approach (Hawkins et al. 2012).

The comprehensiveness of an intervention approach is also important in that multiple interventions may be needed in the school setting, or across multiple set- tings, to address key risk and protective factors associated with a risk behavior and to provide adequate dosage to produce desired outcomes (Nation et al. 2003). Far- rell and Camou (2006) provided a framework for comprehensive youth violence prevention programs based on a grid model that incorporated social setting (e.g., school, family, and community), developmental stage, and level of risk. Although many schools implement multiple programs, these do not necessarily represent a coordinated effort. Gottfredson and Gottfredson (2001), in their national survey of principals representing 635 schools, found that schools implemented a median of 14 different prevention activities. Although this level of activity may be of benefit, they suggested that such diverse efforts may reduce the impact of intervention by spreading resources too thin. Domitrovich et al. (2010) similarly noted that although schools often implement multiple prevention programs, these efforts are not always complementary. They discussed advantages of using theory and data to system- atically integrate prevention programs (i.e., blending overlapping components and retaining unique elements) to address intervention targets. Integrated prevention ef-


forts may cover a wider range of risk and protective factors (e.g., to systematically address a particular risk behavior or target multiple risk behaviors) and have both interactive and additive effects (Domitrovich et al. 2010). A comprehensive effort to address problem behaviors is more likely to be successful if it involves interven- tions that focus on multiple settings, both within and outside of the school context. According to this model, school-based prevention programs represent only one part of a larger effort.


10.3.3 Assessing School Capacity or Readiness for a Program

Another consideration for program adoption is the school’s capacity or readiness for the program. Capacity represents “the skills, motivations, knowledge, and attitudes necessary to implement innovations, which exist at the individual, organization, and community levels,” and is similar to the construct of readiness, which denotes the school’s preparedness to begin and successfully complete program implementation (Flaspohler et al. 2008, p. 183). The literature on implementation science identifies numerous aspects of capacity (Bosworth et al. 1999), three of which appear particu- larly critical. First, it is important to differentiate between two implementation sci- ence frameworks. Research-to-practice models focus on the transfer of innovative, evidence-based programs that have been validated in controlled experimental set- tings into real-world settings, such as classrooms and schools (Aarons et al. 2010; Fixsen et al. 2005). In contrast, community-centered models begin by identifying community-based capacities and the ways in which novel prevention programs or innovations may be adapted to meet community needs, with the improvement of current practice superseding the introduction of innovations (Wandersman 2003).

Flaspohler et al. (2008) identified innovation-specific and general capacity factors that reflect a school’s organizational capacity. Innovation-specific capac- ity, which is evaluated more often in research-to-practice models, encompasses program fit. Fit represents the degree to which the program (a) dovetails with the school’s needs, goals, and day-to-day practice and (b) is socioculturally and devel- opmentally relevant (Nation et al. 2003). For example, interventions need to address specific risk factors that emerge during specific stages of youth development (Far- rell and Camou 2006). Programs that cover multiple grades may therefore require a different focus at each grade. For example, interventions focused on early grades may address factors such as impulse control, whereas those focused on secondary school students may address factors related to dating violence. Additional factors within innovation-specific capacity include school and, especially administrative, buy-in and support needed to supply sufficient time and resources and create a cli- mate conducive to successful program implementation as well as the capability for ongoing training and program evaluation.
Compared to research-to-practice models, community-centered models may place more emphasis on an organization’s general capacity, which consists of “lead- ership, organizational structure, management style, organizational climate, resource


availability, staff capacity, and external relationships” with community members (Flaspohler et al. 2008, p. 191). In contrast to innovation-specific capacities (i.e., those needed to incorporate a specific innovation into the school setting), general capacities represent the broad-spectrum infrastructure, mission, and capabilities of the organization (Halgunseth et al. 2012). School personnel may experience ten- sion between wanting to be receptive to prevention research innovations but at the same time needing to be confident that such innovations will fit within the school’s general capacities. Flaspohler et al. (2008) also highlighted the need for researchers to more carefully consider general capacities prior to implementing research-to- practice models because general capacities influence implementation effectiveness. Several researchers have developed measures to assess school capacity (e.g., Bo- sworth et al. 1999; Roberts-Gray et al. 2007). For example, Bosworth et al. (1999) created a model to assess the probability of successful implementation of school- based prevention efforts. In consultation with a panel knowledgeable about pre- vention programs and intervention theory, they identified several factors that may predict successful school-based implementation. Factors included the complexity of teaching and implementing the program, the plan for facilitating implementation (e.g., training, coaching, and fidelity monitoring), implementer characteristics (e.g., commitment and ability to implement the program based on other job duties), and the program’s compatibility with school needs. They also listed broader factors re- lated to leadership at the individual school and school district levels, external envi- ronment (e.g., policy and procedural support at the school district level and support from parents and the community), and availability of resources needed to deliver the program (Bosworth et al. 1999). Roberts-Gray et al. (2007) used these factors to assess the implementation success of the Texas Tobacco Prevention Initiative in 47 schools. Aggregated scores for the eight capacity factors predicted the quality of program adherence and the quantity of program activities implemented. Thus, this type of assessment may be helpful in evaluating the likelihood of program success
in specific schools.


10.3.4 Collaborating with Schools on Intervention Selection and Development

Another issue related to selecting a specific intervention concerns how best to in- volve teachers, school administrators, and other key stakeholders in the decision- making process. Their active involvement is often key to obtaining support for pro- gram implementation. They may also have valuable insights into what may or may not meet the needs of their school, and the degree to which specific programs are feasible and acceptable to the community. Although actively involving teachers and administrators in intervention development and implementation planning can in- crease their buy-in, it can also pose some difficulties. There are numerous examples in the literature of well-intentioned, intuitively appealing prevention efforts that have ultimately been found to have limited effects, or in some cases negative effects


(Dishion et al. 1999). Many programs such as Drug Abuse Resistance Education (D.A.R.E.), Scared Straight, and boot camps continue to have widespread support despite evidence questioning their effectiveness (U.S. Surgeon General 2001). This underscores the need to involve key stakeholders in intervention selection where possible, but to limit choices to intervention strategies with research support for their effectiveness (Hawkins et al. 2012).

As this section illustrates, the process of program development and adoption requires considerable effort to ensure that it leads to a carefully developed inter- vention plan. This plan must begin with a careful consideration of the goals and characteristics of the settings where the intervention will be implemented, includ- ing information regarding the factors responsible for maintaining the problem be- ing addressed. As we noted, the selection of an intervention requires identifying complementary components that have potential to accomplish these goals. Finally, the potential for success will also depend on the school’s capacity to implement  the program successfully. Case study 1 describes a collaborative adoption process whereby specific school needs were matched to evidence-based programming in order to maximize the likelihood of program effectiveness.


Case Study 1: Concept Mapping

This case study illustrates the use of a concept-mapping approach to involve key stakeholders in the identification of relevant issues related to youth vio- lence prevention programs. The information gathered informed the interven- tion approach for a research project that focused on developing and testing the feasibility of a middle school violence prevention program for youth with and without disabilities (Sullivan et al. 2013). In the planning phase of this research project, concept mapping was used to identify the needs related to youth violence prevention at participating middle schools. Concept map- ping, which has been promoted as a useful participatory research method for program planning, encompasses multiple steps: (a) brainstorming project- relevant ideas; (b) sorting and rating ideas based on similarity, importance, and feasibility; (c) visually representing stakeholders’ ideas in the form of a concept map; (d) interpreting the map; and (e) utilizing the map for project planning (Trochim 1989).
The first component of the concept-mapping process involved a brainstorm- ing activity where stakeholders were asked to generate goals for the project by listing statements in response to the open-ended prompt: “An important goal for school-based violence prevention programs for middle school students is….” A total of 57 adults participated, including middle school teachers, administrators, other school staff, researchers, community workers, parents, a school board member, and 16 middle school students with and without dis- abilities. Adult participants were given the option of completing brainstorm- ing  activities  anonymously  using  a  web-based  concept-mapping program



(Concept Systems Incorporated) or completing a paper-and-pencil version  of the activity. Students completed the paper-and-pencil version. This task generated 245 goals. Combining similar responses and eliminating redundant responses reduced the final number of goals to 85.
During the second stage of the project, 22 school staff, including some who had participated in the brainstorming activity, sorted the 85 goals into piles they thought were similar or related and rated each goal’s importance and feasibility on a five-point scale where higher scores indicated greater importance or feasibility. A concept map was then generated based on par- ticipants’ responses to the brainstorming and sorting and rating tasks. The concept map was a two-dimensional scatter plot of goals, where those that were closer together were considered more similar. Procedures described in previous research (e.g., Sutherland and Katz 2005; Trochim 1989) were then used to identify clusters that represented similar sets of goals. This led to   the identification of eight discrete clusters of goals that related to (a) cop-  ing and conflict resolution skills, effective nonviolent alternatives, and prob- lem recognition; (b) educating students about different forms of violence and teaching social skills/anger management; (c) safety, positive development, and specific aspects of violence prevention programs; (d) school policies and school climate and physical environment; (e) role models and mentors, out- side influences on student behavior, and student engagement at school and in the community; (f) citizenship and relationships; (g) interdependence among the school, parents, and students, positive relationships between school staff and students, and strategies for teachers; and (h) community and parent involvement. Participants rated goals in the last cluster (community and par- ent involvement) as the most difficult to address.
Final clusters and cluster maps were then presented to the stakeholders involved in the concept-mapping process in order to facilitate the interpreta- tion of the results and ensure their accuracy. Focus groups were also conducted to identify the potential supports and challenges to implementing program goals in the participating schools. Discussion centered on several topics that were useful in the assessment of each school’s capacity and readiness for program implementation, including aspects of the school climate, resources, effective teaching methods, and parental support. Themes such as parent and community involvement, school climate, and students’ social and emotional development echoed researchers’ thoughts that prevention programs should address risk behaviors at multiple levels of the school ecology and in multiple settings (Nation et al. 2003).
This concept-mapping project was a first step toward engaging principal stakeholders in the selection of an intervention approach for middle school violence prevention. Ultimately, the ideas generated during this  process were used to determine important and feasible areas for intervention. Taken together, the results suggested that the prevention program should address




10.4 Implementation Phase

Having identified an appropriate intervention approach, the next step involves its implementation. Implementation is defined as the set of activities designed to put a program into practice in an organization such as a school (Forman et al. 2013). The goal of this phase is to have all components of an evidence-based program in place and integrated into practice, organizational and community structures, and policies. There are a myriad of interconnected aspects to implementing a new school-based prevention program, such as the capacity and culture of the school and the ongoing skill development of implementers (e.g., teachers; Fixsen et al. 2005). The unique interplay between practitioners’ fear of change, their investment in the status quo, and the complexities of a particular environment create a tenuous context for imple- mentation, particularly when implementing a new program.
Berkel et al. (2011) proposed a working model of prevention that highlights the relations between several dimensions of implementation and desirable outcomes among youths. To illustrate, they argued that the behaviors of program implement- ers (e.g., teachers) as they implement and adapt core components are associated with the behaviors of participants (e.g., students), such as attendance and engage- ment. From their perspective, a critical aspect of implementation research is how programs promote high levels of youth engagement, and how this engagement me- diates program effectiveness. This model highlights the importance of both high- quality training and ongoing support for implementers (e.g., coaching of teachers, performance feedback), which can increase the likelihood of high-quality imple- mentation as well as implementers using instructional behaviors associated with youth engagement (Aarons et al. 2010; Conroy et al. 2009; Domitrovich et al. 2012; Dunst and Trivette 2012; Fixsen et al. 2005; Vo et al. 2012).

One critical factor related to the successful implementation of a school-based intervention is getting the full support and cooperation of teachers and administra- tors. School staff members must perceive the need for the innovation before they will implement it (Gottfredson 2001). As previously noted, most schools have a his- tory of implementing a variety of different prevention strategies (Gottfredson and Gottfredson 2001), which may not always have involved evidence-based interven- tions or been implemented with sufficient dosage or fidelity to produce their desired effects (Gottfredson and Gottfredson 2002). A long history of limited success may


lead teachers and administrators to be quite skeptical that the next strategy they at- tempt will be any more successful. This may make it difficult to generate the degree of enthusiasm, support, and cooperation needed for successful implementation. It may also create negative expectations for success.
Efforts to obtain teacher and school administrator support for prevention pro- gramming are often hampered by the practice of having outside staff implement prevention programs. The use of outside interventionists under the direct control and supervision of researchers may increase the likelihood that an intervention will be conducted with fidelity. However, it also decreases the involvement of teachers and other key staff. Teachers play a critical role in the climate of a school, and their interactions with students and with each other provide opportunities for them to model the skills targeted by many intervention programs. Using outside interven- tionists may limit teachers’ support for the intervention and the degree to which it is infused into the curriculum and other aspects of the school. One example of how in- terventions can increase the involvement of teachers and administrators is provided by the Olweus Bullying Prevention Program (Olweus 2004), which establishes a Bullying Prevention Coordinating Committee composed of teachers, administra- tors, and other school staff. In the research project discussed in case study 2, this committee receives performance-based feedback regarding teachers’ adherence to and competence with intervention delivery, and based on this feedback they identify areas for improvement (and positive reinforcement for implementation successes). Another factor related to successful implementation concerns the extent to which teachers and administrators feel that they “own” the intervention. Schools, particu- larly those located near a university, may have a history of researchers coming into the school to implement an intervention, collecting data, and disappearing when the project ends. Teachers may question the extent to which those outsiders, regardless of their academic credentials, have a clear understanding of their school and the issues students and teachers face. They may also feel disrespected if their opinions are not actively sought. Although their knowledge and insights may be of value in tailoring an intervention to meet the specific needs of a school, involving teachers and school staff in tailoring intervention strategies may be challenging. Adapting an intervention requires a clear understanding of which aspects must be implemented with fidelity and which can be modified to be more culturally or contextually ap- propriate to specific groups of students, or to meet practical constraints such as class schedules (Meyer et al. 2000). Whereas adaptations have historically been viewed as deviations from program core components that threaten internal validity, more recent work has focused upon how thoughtful adaptations may strengthen the effectiveness of prevention programs (Durlak and DuPre 2008; Fixsen et al. 2005; Forman et al. 2013). In their review of implementation research, Durlak and DuPre (2008) reported that of three studies assessing program adaptation, all found adaptation had a positive effect on outcomes. However, without strong measures of treatment fidelity, the core components of prevention programs cannot be mea- sured, limiting our ability to identify effective versus harmful adaptations. Thus, an important focus in implementation research must be the monitoring of fidelity to


the prevention program in order to assess adaptations in the school context (Bopp et al. 2013).
Providing quality training is critical to the success of a prevention program, regardless of whether researchers or school staff are responsible for implementa- tion. High-quality training has been associated with better program implementa- tion (Durlak and DuPre 2008). Research suggests that professional development in the form of one-time training does not result in proficient delivery of practices in authentic settings (Becker and Domitrovich 2011; Sholomskas et al. 2005), high- lighting the need for implementers to receive ongoing coaching and performance feedback (Vo et al. 2012). A large literature suggests that coaching strategies such as collaborative decision-making, modeling, observation and performance feedback, and opportunities to problem solve enhance and sustain teacher delivery of pre- vention components (Han and Weiss 2005; Reinke et al. 2009; Reyes et al. 2012). Forman et al. (2013) noted that multiple models of implementation emphasize the importance of not only developing competent implementers (training and coaching) but also rewarding implementation and providing feedback to implementers on the process of implementation (e.g., fidelity) as well as program outcomes.
Ensuring that prevention programs are implemented as intended increases the likelihood that they will produce positive outcomes (Hulleman et al. 2013). Treat- ment fidelity, referring to the degree to which a program is delivered as intended, has three components—treatment adherence, treatment differentiation, and competence (McLeod et al. 2009). As it relates to prevention programs delivered in schools by teachers, treatment adherence refers to the extent to which the teacher delivers the program as designed (i.e., prescribed practices). Treatment differentiation refers to the extent to which treatments being implemented differ along appropriate lines defined by the treatment protocol (e.g., do not have protocol violations). Finally, competence refers to the level of skill and degree of responsiveness the teacher demonstrates when delivering the evidence-based instructional practices prescribed by the protocol. Each component captures a unique aspect of treatment fidelity that is important to assess in prevention research (Carroll and Nuro 2002).
A number of factors within schools can influence implementation fidelity, includ- ing the diverse training backgrounds of teachers (Kam et al. 2003), level of teacher training (Pianta and Rimm-Kaufman 2006), and resource restrictions (Domitrovich et al. 2010). More proximal factors associated with teacher and student behavior, such as teacher relationships with students and level of student involvement in classroom activities, may also influence outcomes. Indeed, prevention programs will not be effective if youths do not bond with teachers or actively participate in classroom activities (McLeod et al. 2009). It is plausible that such contextual fac- tors may influence fidelity and thereby outcomes, so it is critical to assess imple- mentation fidelity during the implementation phase.
Because a key part of implementation involves training and supervising teachers to deliver the instructional practices specified for a prevention program, ascertain- ing the extent to which these practices are delivered according to the treatment pro- tocol is of critical importance (Sutherland et al. in press). Doing so requires assess- ing all three treatment fidelity components (McLeod et al. 2013). Indeed, it is nec-


essary to assess implementation fidelity so that researchers can determine whether any “failure” to produce a desired outcome is due to the prevention program or its implementation. If it is due to the program (i.e., implementation is sufficient), this implies the need to adapt the program or select an alternative. In contrast, if the pro- gram was not appropriately implemented, this suggests the need for improvements in teacher training and support. Fidelity measurement of a prevention program may also assist researchers in identifying the components of an intervention that are most related to desirable outcomes, allowing for more efficient delivery. In addition, identifying the core components of a prevention program may assist in identifying those components that are amenable to adaptation, increasing the acceptability of the program. Unfortunately, the science and measurement of treatment fidelity is underdeveloped in school-based prevention (Sanetti and Kratochwill 2009). Most studies focus on adherence, which leaves competence of delivery and treatment dif- ferentiation largely unstudied (Sanetti and Fallon 2011).
The measurement of treatment fidelity is clearly important during implementa- tion, and researchers and program purveyors strive for the highest levels of imple- mentation fidelity possible. At the same time, full implementation is often not a realistic goal, and research has suggested that implementation fidelity ranging from 60 to 80 % is associated with positive youth outcomes (Durlak and DuPre 2008). Thus, whereas new implementers can present challenges, particularly ones associ- ated with the fidelity of implementation, they can also present unique opportunities to innovate and refine practices based upon school and programmatic needs (Fixsen et al. 2005). Several researchers (Dissemination Working Group 1999; Fixsen et al. 2005; Winter and Szulanski 2001) recommended innovation after full implementa- tion with fidelity in order not to “escape the scrutiny of fidelity” (Fixsen et al. 2005,
p. 17), noting that at some point if innovations to implementation are significant, they may warrant future scientific study. At the same time, the ability to innovate may enhance program acceptance, and if researchers have clearly identified the core components of their program that require implementation with fidelity, practitio- ners may feel some freedom to adapt a particular evidence-based program to their context without compromising the fidelity of implementation (Durlak 2010; Harn et al. 2013). In the next section, a case study of an ongoing research project is used to illustrate how adaptation and performance-based feedback provided to schools implementing the Olweus Bullying Prevention Program (Olweus 2004) are used  to maintain and improve the fidelity of implementation of classroom meetings by teachers.




it is to enhance school safety through monitoring and overseeing positive changes to the school environment and the application of school rules. The classroom component consists of weekly classroom meetings, where students and teachers engage in discussion and activities related to preventing bullying behavior.
The first six classroom meetings are scripted in order to provide structure around establishing school-wide rules and program expectations. Beyond these six meetings, teachers are expected to discuss bullying-related topics  in the classroom meetings, but specific scripted meetings are not provided.  A large number of teachers in the two middle schools in this case preferred more structured guides for classroom meetings, so we worked in partnership with teachers to identify relevant topics (e.g., problem-solving, emotional regulation, cyberbullying) then created meeting plans to address these. Teach- ers used their knowledge of their students and school culture to assist us in enhancing the cultural relevance of these meetings by adapting role plays, examples, and language. Such activities integrated the teachers as stakehold- ers in the adaptation process (Domenech-Rodriquez et al. 2011) and are in line with recommendations to make programs more culturally relevant (Ber- nal et al. 1995; Nation et al. 2003). Research also suggests that local adapta- tions may enhance the sustainability of intervention programs (Berkel et al. 2011; Rogers 2003).
Because teachers conduct the classroom meetings, measuring the extent  to which these meetings are delivered according to the meeting plans is of critical importance for monitoring implementation fidelity. Thus, we assess both the adherence and competence of teacher delivery of classroom meet- ings as well as more general teacher instructional practices (e.g., providing youth with opportunities to respond, reinforcement procedures) and student engagement. Becker and Domitrovich (2011) have pointed out that although researchers collect data on treatment fidelity and program implementation, they typically do not share these data with the school where they are col- lected. We address this concern by collecting data on implementation fidel- ity on a weekly basis in at least 20 % of the classrooms implementing the Olweus program, and using these data to inform implementation procedures at the schools. Observational data are compiled and shared with each school’s Bullying Prevention Coordinating Committee, which is made up of teachers, administrators, parents, and university partners. These presentations highlight both successes (e.g., improvements in teachers providing feedback to stu- dents) and areas for improvement (e.g., low rates of teacher reinforcement) while focusing on the links between teacher behaviors and youth engage- ment during the classroom meetings. In addition, twice a year these data are compiled and shared with the entire school faculty, again highlighting both strengths of program delivery and areas for improvement.


The work of the Bullying Prevention Coordinating Committee at each school, and the ongoing support and collaboration with university partners in this process, has promise for increasing the likelihood of both high-quality implementation and sus- tainability of the program.
The amount of time it takes to move from adoption to quality implementation  of a major school-wide reform has not yet been established, with estimates ranging from a minimum of 1 year (e.g., Fixsen et al. 2005) to at least 3 years (e.g., Felner et al. 2001). Our use of a multiple baseline design provides a rare opportunity to ex- amine changes in implementation fidelity over the course of the project. Our design involves implementing the intervention in three different schools but uses random- ization to determine when intervention activities are initiated at each school. Fol- lowing a year of baseline data collection, intervention implementation was started at school A in year 2 of the project and at school B in year 3; implementation will begin in school C at the end of the final year of the project. We are also engaged in efforts to sustain intervention activities at the conclusion of the project. This design allows us to examine changes related to implementation across school years. We are also able to benefit from our experience implementing the intervention at school A as we work with schools B then C.
Fidelity data from the first 2 years of our project provide some insight into the implementation of the Olweus program. At school A, adherence and competence both increased modestly from year 1 to year 2 and mean scores fell in the acceptable range. These data suggest that teachers are implementing the program in Year 2 at least as extensively and competently as they did in Year 1. Comparing competence data for the first year of implementation in school B with those for the second year of implementation in school A reveals that almost all time points for both schools were in the acceptable to excellent range. At school A, some increases in compe- tence of delivery occurred over time, perhaps reflective of the teachers’ experi- ence in implementing the program the previous year, their approval of adaptations to classroom meetings made in the previous year, ongoing fidelity monitoring, or some combination thereof. Meanwhile, teachers in school B were energized about the program at the beginning of their first year of implementation. Interestingly, both schools have some fluctuations in competence that merit further examination to potentially link them with events and other activities going on in the schools. Our long-term goal is to use the ongoing data to plan for sustainability.


10.5 Sustainability

In the health promotion intervention life cycle, a critical point for many prevention programs is the time frame after implementation. At this point, the program may be sustained and move toward being institutionalized within a school or, as is more often the case, terminated and sometimes replaced by a new program (Bopp et al. 2013). Although the successful implementation of evidence-based prevention pro- grams has great value, the ultimate goal is their sustainability in schools.   Sustain-


ability is the ability to maintain and support the long-term survival and effectiveness of an evidence-based program at a particular intervention site (Fixsen et al. 2005). Ensuring that intervention activities will continue beyond the end of a research project requires that sustainability be considered from the onset through each phase of the project. Some of the issues we highlighted in the sections on program devel- opment and adoption (e.g., ensuring the program meets the school’s needs and gar- nering program support) and implementation (e.g., providing ongoing training and coaching and tailoring the program as allowable to be more relevant for students and staff) enhance the likelihood of sustainability. Bopp et al. (2013) identified fac- tors that promote sustainability, including characteristics of the prevention program, its implementers, and the implementation process, as well as the organization’s gen- eral capacity and support for the program. In this section, we first present concerns related to difficulties in sustaining school-based prevention programs, then review factors that may promote sustainability in more detail.
The sustainability of prevention programs delivered in schools is a concern of prevention researchers and practitioners (Fixsen et al. 2009; Han and Weiss 2005; Reyes et al. 2012). Whereas there is substantial evidence that school-based preven- tion can have positive effects on students’ academic and behavioral functioning (e.g., Durlak et al. 2011), the degree to which school-based implementers (e.g., teachers and other school personnel) can sustain high-quality implementation re- mains an open and vexing question. Fixsen et al. (2009) pointed out that efforts to implement evidence-based programs over the past two decades suggest that “re- search results are not being used with sufficient quantity and quality to impact hu- man services and, therefore, have not provided the intended benefits to consumers and communities” (p. 531).
Several overarching factors influence the sustainability of school-based preven- tion programs. At the macro level, the availability of resources and the priorities and policies of the school district, which are influenced by resources and priorities at the state and federal levels, impact the agenda for prevention programs (Han and Weiss 2005). At the school level, some indicators of sustainability can be found by examining a school’s general capacity (Halgunseth et al. 2012). The life span of a prevention program is influenced by the school’s overall management, leadership, and organizational structure as well as administrative policies within the school that support the program and the availability of resources needed to sustain it (Fla- spohler et al. 2008; Johnson et al. 2004). For grant-funded prevention programs,   a resource gap often exists in that the resources available to initiate the program during the life cycle of the grant exceed those that remain after the grant ends. As Gottfredson (2001) noted, “Programs operated as part of research endeavors are generally implemented under unusual conditions” (p. 232).
A key concern for sustainability is ensuring that a program will continue to be implemented at the required level of fidelity, as in a national survey Gottfredson and Gottfredson (2002) found that the quality of school-based prevention practices implemented in the typical school was generally low. Their analysis of correlates of prevention quality suggested that implementation could be improved through bet- ter integration of prevention activities into school operations; more extensive local


planning and involvement in selecting interventions; stronger organizational sup- port in terms of high-quality training, supervision, and principal support; and use of more standardized program materials and methods.
Factors related to a particular school-based prevention program may also influ- ence its sustainability. Universal programs aimed at changing the school environ- ment generally have a framework that supports their integration into school op- erations, which can facilitate sustainability (Olweus 2004). Clearly, the degree to which the program is high quality, effective, and congruent with school needs and goals plays a role in sustainability. In addition, the presence of a school champion for the program and its perceived support by the principal are critical. In particular, school staff may use the principal’s support as a gauge of the program’s priority within the school; whether they will be held accountable for its implementation; and whether sufficient resources, time, and energy will be devoted to it both during the initial implementation and in the future (Han and Weiss 2005). One challenge is that interventions often must be in place for some time before they produce their intended effects. School climate is usually well established and somewhat resistant to change. Prevention programs targeted across multiple grade levels may need    to be implemented for several years before effects on school climate are realized, allowing new students to enter a school in which the majority of students in older grades have participated in previous years of the intervention. Unfortunately, teach- ers and administrators may lose patience waiting to see signs of success and may be tempted to try something different. This has led to some school systems adopting a new program(s) in lieu of or in addition to existing programs.
Similar to the implementation phase, the extent to which the prevention program allows for adaptations can have implications for whether it is accepted by school staff and sustained by a school. Given that greater resources often exist when a prevention program is initially implemented than in subsequent years, adaptations that enhance the feasibility of continued implementation without damaging the in- tegrity of the intervention are important. In fact, implementation research suggests that when programs are adapted locally, they may have a greater likelihood of being sustained (Berkel et al. 2011; Rogers 2003). Ensuring that program adaptations do not detract from the integrity of the intervention and monitoring program fidelity on an ongoing basis are critical if the prevention program is to maintain its evidence- based status as it is sustained in the school context (Aarons et al. 2010; Domitro- vich et al. 2012). Moreover, researchers must have sustainability in mind as they develop and test prevention programs. The early identification of core components, development of fidelity measures that assess the adherence to and competence of delivery of core components, and assessment of adaptations (either through direct observations or via teacher focus groups) are all critically important to developing prevention programs that have a higher than average likelihood of sustainability.
Implementer characteristics—including attitudes toward the intervention, teach- ing ability, self-efficacy, commitment and motivation to change, and perceptions of program benefits—may also influence the sustainability of a prevention program. Initiating and maintaining ongoing collaborative work with teachers and other school personnel to ensure that prevention programs are feasible and acceptable can


lead to greater implementation and sustainability. Prevention researchers have em- phasized that if evidence-based prevention programs are to be delivered by teachers effectively, feedback from teachers should be used to address potential resistance to implementation. Such feedback can be incorporated into the design of program content as well as implementation of instructional practices (Becker et al. 2009; Greenberg et al. 2003). In addition, the ongoing measurement of treatment fidelity at multiple levels, such as training, delivery, coaching, and teacher implementation (adherence and competence of delivery) can assist in both planning for sustain- ability as well as making the program more flexible and adaptable. As Becker and Domitrovich (2011) note, researchers collect data on treatment fidelity and program implementation but typically do not share these data with the school in which they are collecting the data. By collecting fidelity data on a regular basis and using these data to inform implementation procedures, researchers can facilitate both the it- erative development process and adaptations, with the ultimate goal of increasing program sustainability.
Johnson et al. (2004) emphasize that sustainability planning is an iterative pro- cess that continues over the life span of a prevention program. They identified two building blocks for sustainability; namely, ensuring that (a) the organization builds the infrastructure capacity to sustain the prevention program and (b) the program continues to meet shareholder needs, be implemented with fidelity, and maintain its effectiveness. They detailed five sustainability actions: assessing the organization’s infrastructure capacity and readiness for the prevention program; developing a sus- tainability plan to target issues related to capacity and readiness; and executing, evaluating, and revising this plan as needed. These sustainability actions then lead to the proximal outcome of “sustainability readiness” and distal outcomes in terms of benefits for students and school staff when effective and relevant programs are well integrated within the organizational structure. Case study 3 describes efforts to increase the implementation fidelity, and subsequently the sustainability, of a pro- gram designed to prevent the development of emotional and behavioral disorders in young, high-risk children.
The last case study describes the BEST in CLASS project to exemplify issues  of implementation and sustainability. Recall that BEST in CLASS is a secondary- level intervention that targets problem behaviors in preschool children at risk for emotional and behavioral disorders.




Supervision, (e) Opportunities to Respond and Instructional Pacing, (f) Instructive and Corrective Feedback, (g) Home–School Communication, and
(h) Linking and Mastery. The efficacy of BEST in CLASS is currently being investigated in a multisite randomized controlled trial, and preliminary data suggest that the program had an impact on observed teacher instructional and child behaviors (Conroy et al. 2013) and on standardized measures of child behavior (Vo et al. 2012).
BEST in CLASS was developed with an eye toward sustainability. Specifi- cally, program administrators initially approached university researchers for assistance with an increasing number of young children arriving in preschool with high rates of problem behavior. Through a collaborative process, BEST in CLASS was developed with feasibility of implementation in mind. Accord- ingly, teachers deliver evidence-based instructional practices (i.e., core com- ponents) during their classroom instructional activities. Program development was guided by an iterative process (see Vo et al. 2012, for a description) wherein teachers implemented aspects of the program; direct observations   of implementation were conducted focusing on adaptations, adherence, and competence of delivery; and focus groups were held to identify barriers and supports to program delivery.
In order to plan for implementation, BEST in CLASS was developed according to a consultation model. Existing literature and data collected dur- ing the development project (e.g., focus groups with teachers) guided devel- opment of a structured, practice-based coaching model whereby coaches provided teachers with weekly performance feedback on their implementa- tion. In addition, observations and focus groups with teachers and coaches led to adaptations in both component delivery and coaching, adaptations which were integrated into training materials and procedures. These adaptations are in line with implementation literature which suggests that incorporating the knowledge program providers (e.g., teachers, program administrators) have about their particular context can lead to innovations that improve interven- tion (Durlak and DuPre 2008) and in turn increase the likelihood of sustain- ability (Berkel et al. 2011; Rogers 2003).
The identification of core program components also contributed to the devel- opment of a fidelity measure, the BEST in CLASS Adherence and Competence Scale (BiCACS; Sutherland et al. in press). This direct observational tool assesses the extensiveness (adherence) and quality (competence) of delivery of each of the eight core components of BEST in CLASS using a Likert-type scale. Two addi- tional items (child responsiveness; engagement) assess children’s responses to the program components. The BiCACS was found to have promising reliability and validity in an initial psychometric study (Sutherland et al. in press). This is important (and novel) as researchers have highlighted the need for reliable mea- sures of integrity to advance the science of prevention (Durlak 2010; Sanetti and Kratochwill 2009; Wolery 2011).




In summary, the development and ongoing adaptation of BEST in CLASS, particu- larly during the project phase, resulted in a program designed in collaboration with teachers and program administrators that has promise for quality implementation and long-term sustainability. Initial data from the efficacy trial suggest that teachers implemented the program with increasing adherence and competence across time, and the use of an observational integrity tool, the BiCACS, with promising psycho- metrics to measure implementation fidelity has much potential for determining the core components of the program.


10.6 Conclusion

Schools represent a key context for youths’ social, emotional, and academic de- velopment and are a natural fit for prevention programs that seek to strengthen these areas of development, thereby decreasing the incidence of problem behaviors. Strong partnerships between researchers and school staff are essential for the effec- tive implementation of evidence-based programs. Prior to program implementation, it is important for researchers to identify the prevalence of the problem behavior   in the target school population and the constellation of risk and protective factors that influence it. A thorough understanding of these risk and protective factors can then guide the selection of the evidence-based program or programs that most ef- fectively target them. The school’s readiness to implement the prevention program must also be considered. In this regard, the fit between the prevention program  and the school’s general capacity in terms of its organizational structure,   mission,


goals, and staff “buy-in” or support for the program should be addressed. Overall, it is essential that the prevention program meet the school’s needs by (a) effectively targeting the risk behavior and (b) being feasible to implement.
During implementation, an understanding of which are the core components    of the prevention program that must be implemented to preserve its integrity and which can be adapted to make the program more relevant and feasible to imple- ment is needed. Program adaptations that maintain a program’s integrity have been shown to enhance the likelihood of desired outcomes. However, the collection of fi- delity data is necessary to ensure that these adaptations are well documented and do not alter the implementation of the program’s core components. The benefits of pre- vention programs in mitigating serious negative outcomes for youth are well worth the trade-off in energy, time, and money spent on their implementation. However, unless researchers and school staff can improve the sustainability of these programs in schools, their true capacity to benefit youth over the course of development may remain unknown.
It is again worth emphasizing that schools are dynamic entities containing het- erogeneous student populations embedded within diverse community contexts. Therefore, the process of implementing interventions within schools is necessarily iterative to meet the numerous needs extant in an individual school. Although many promising prevention programs have been developed (U.S. Surgeon General 2001), there is considerable room for improvement. The best of our current programs are based on an incomplete understanding of the myriad risk factors that influence and maintain problem behaviors in school-aged youth and of potential promotive and protective factors that could enhance their positive development (Farrell and Vulin- Reynolds 2007). Further progress in identifying these factors will guide the de- velopment of more effective prevention strategies. Further work is also needed to improve the effectiveness and efficiency of delivery of current interventions. There is growing evidence to suggest that the effectiveness of many school-based inter- ventions varies across individuals, schools, and settings (Farrell et al. 2013). This implies a need to tailor interventions to best meet the needs of a target population. Unfortunately, we currently have limited information to determine which aspects of interventions can be adapted without reducing their impact. Finally, the dissemi- nation of interventions and their sustainability continue to pose major challenges (Forman et al. 2013). Considerable efforts are needed to ensure that schools are able to devote their limited resources to programs that have the greatest likelihood of effectiveness, to implement these programs with sufficient integrity to ensure that they produce their intended effects, and to sustain quality implementation with their available resources. These efforts will require a sustained iterative effort whereby interventions are continuously (a) developed to address emerging information about the nature of the problem being targeted and the factors that maintain it, (b) evalu- ated to determine their impact and the conditions under which they are effective, and (c) disseminated to establish the factors that facilitate their widespread imple- mentation (Gottfredson 1984).


References

Aarons, G. A., Glisson, C., Hoagwood, K., Kelleher, K., Landsverk, J., & Cafri, G. (2010). Psy- chometric properties and U.S. national norms of the Evidence-Based Practice Attitude Scale (EBPAS). Psychological Assessment, 22(2), 356–365. doi:10.1037/a0019188
Becker, K. D., & Domitrovich, C. E. (2011). The conceptualization, integration, and support of evidence-based interventions in the schools. School Psychology Review, 40(4), 582–589.
Becker, K. D., Nakamura, B., Young, J., & Chorpita, B. (2009). What better place than here? What better time than now? ABCT’s burgeoning role in the dissemination and implementation of evidence-based practices. Behavior Therapist, 32(5), 89–96.
Berkel, C., Mauricio, A., Schoenfelder, E., & Sandler, I. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Sciences, 12(1), 23–33. doi:10.1007/ s11121-010-0186-1
Bernal, G., Bonilla, J., & Bellido, C. (1995). Ecological validity and cultural sensitivity for outcome research: Issues for the cultural adaptation and development of psychosocial treatments with Hispanics. Journal of Abnormal Child Psychology, 23(1), 67–82. doi:10.1007/BF01447045
Bopp, M., Saunders, R. P., & Lattimore, D. L. (2013). The tug-of-war: fidelity versus adaptation throughout the health promotion program life cycle. Journal of Primary Prevention, 34(3), 193–207. doi:10.1007/s10935-013-0299-y
Bosworth, K., Gingiss, P., Potthoff, S., & Roberts-Gray, C. (1999). A Bayesian model to predict the success of the implementation of a health education innovation. Evaluation & Program Planning, 22(1), 1–11.
Botvin, G. J., & Tortu, S. (1988). Preventing adolescent substance abuse through life skills train- ing. In R. H. Price, E. L. Cowen, R. P. Lorion, & J. R. McKay (Eds.), Fourteen ounces of pre- vention: A casebook for practitioners (pp. 98–110). Washington, DC: American Psychological Association.
Botvin, G. J., Griffin, K. W., & Nichols, T. D. (2006). Preventing youth violence and delinquency through a universal school-based prevention approach. Prevention Science, 7(4), 403–408. doi:10.1007/s1121-006-0057-y
Carroll, K. M., & Nuro, K. F. (2002). One size cannot fit all: A stage model for psychotherapy manual development. Clinical Psychology: Science & Practice, 9(4), 396–406. doi:10.1093/ clipsy.9.4.396
Centers for Disease Control and Prevention. (2011). Web-based injury statistics query and report- ing system (WISQARS). Atlanta: National Center for Injury Prevention and Control. http:// www.cdc.gov/injury/wisqars/index.html.
Coie, J. D., Watt, N. F., West, S. G., Hawkins, J. D., Asarnow, J. R., Markman, H. J., Ramey, S. L., Shure, M. B., & Long, B. (1993). The science of prevention: A conceptual framework and some directions for a national research program. American Psychologist, 48(10), 1013–1022. doi:10.1037/0003-066X.48.10.1013
Committee for Children. (2008). Second Step: Student success through prevention program. Se- attle: Author.
Conroy, M. A., Sutherland, K. S., Haydon, T., Stormont, M., & Harmon, J. (2009). Preventing and ameliorating young children’s chronic problem behaviors: An ecological classroom-based ap- proach. Psychology in the Schools, 46(1), 3–17. doi:10.1002/pits.20350
Conroy, M. A., Sutherland, K. S., Vo, A., Carr, S. E., & Ogston, P. (2013). Early childhood teach- ers’ use of effective instructional practices and the collateral effects on young children’s behav- ior. Journal of Positive Behavioral Interventions. doi:10.1177/1098300713478666
Dishion, T. J., McCord, J., & Poulin, F. (1999). When interventions harm: Peer groups and prob- lem behavior. American Psychologist, 54(9), 755–764. doi:10.1037/0003-066X.54.9.755
Dissemination Working Group. (1999). Common elements of developing and implementing pro- gram models. Calgary: FYI Consulting, Ltd.


Domenech Rodriguez, M. M., Baumann, A., & Schwartz, A. (2011). Cultural adaptation of an empirically supported intervention: From theory to practice in a Latino/a community context. American Journal of Community Psychology, 47(1–2), 170–186. doi:10.1007/s10464-010- 9371-4
Domitrovich, C. E., Bradshaw, C. P., Greenberg, M. T., Embry, D., Poduska, J. M., & Ialongo, N.
S. (2010). Integrated models of school-based prevention: Logic and theory. Psychology in the Schools, 47(1), 71–88. doi:10.1002/pits.20452
Domitrovich, C. E., Moore, J. E., & Greenberg, M. T. (2012). Maximizing the effectiveness of social-emotional interventions for young children through high-quality implementation of evidence-based interventions. In B. Kelly & D. F. Perkins (Eds.), Handbook of implementation science for psychology in education (pp. 207–229). Cambridge: Cambridge University Press.
Dunst, C. J., & Trivette, C. M. (2012). Meta-analysis of implementation practice research. In B. Kelly & D. F. Perkins (Eds.), Handbook of implementation science for psychology and educa- tion (pp. 68–91). Cambridge: Cambridge University Press.
Durlak, J. (2010). The importance of doing well in whatever you do: A commentary on the special section: “Implementation research in early childhood education.” Early Childhood Research Quarterly, 25(3), 348–357.
Durlak, J. A, & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350. doi:10.1007/s10464-008-9165-0
Durlak, J. A., Weissberg, R. P.,  Dymnicki, A. B., Taylor,  R. D., & Schellinger, K. B. (2011).  The impact of enhancing students’ social and emotional learning: A meta-analysis of school- based universal interventions. Child Development, 82(1), 405–432. doi:10.1111/j.1467- 8624.2010.01564.x
Embry, D. D. (2003). PAX Good Behavior teacher’s guide. Center City: Hazelden.
Embry, D. D., Flannery, D. J., Vazsonyi, A. T., Powell, K. E., & Atha, H. (1996). PeaceBuilders: A theoretically driven, school-based model for early violence prevention. American Journal of Preventive Medicine, 12, 91–100.
Espelage, D. L., Low, S., Polanin, J. R., & Brown, E. C. (2013). The impact of a middle school pro- gram to reduce aggression, victimization, and sexual violence. Journal of Adolescent Health, 53(2), 180–186. doi:10.1016/j.jadohealth.2013.02.021
Fagan, J., & Wilkinson, D. (1998). Social contexts and functions of adolescent violence. In D. El- liott, B. Hamburg, & K. Williams (Eds.), Violence in American schools (pp. 55–93). New York: Cambridge University Press.
Farrell, A. D., & Camou, S. (2006). School-based interventions for youth violence prevention. In
J. Lutzker (Ed.), Preventing violence: Research and evidence-based intervention strategies
(pp. 125–145). Washington, DC: American Psychological Association.
Farrell, A. D., & Vulin-Reynolds, M. (2007). Violent behavior and the science of prevention.   In
D. J. Flannery, A. T. Vazsonyi, & I. D. Waldman (Eds.), The Cambridge handbook of violent behavior and aggression (pp. 767–786). New York: Cambridge University Press.
Farrell, A. D., Meyer, A. L., Kung, E. M., & Sullivan, T. N. (2001). Development and evaluation of school-based violence prevention programs. Journal of Clinical Child Psychology, 30(2), 207–220. doi:10.1207/S15374424JCCP3002_8
Farrell, A. D., Henry, D. B., & Bettencourt, A. (2013). Methodological challenges examining sub- group differences: Examples from universal school-based youth violence prevention trials. Prevention Science, 14(2), 1–13. doi:10.1007/s11121-011-0200-2
Felner, R. D., Favazza, A., Shim, M., Brand, S., Gu, K., & Noonan, N. (2001). Whole school im- provement and restructuring as prevention and promotion: Lessons from STEP and the Project on High Performance Learning Communities. Journal of School Psychology, 39(2), 177–202. doi:10.1016/S0022-4405(01)00057-7
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Louis de la Parte Florida Mental Health Institute, National Implementation Research Network (FMHI Publication #231). Tampa: University of South Florida. http://nirn.fmhi.usf.edu/resources/publications/Monograph/pdf/monograph_full.pdf


Fixsen, D. L., Blase, K. A., Naoom, S. F., & Wallace, F. (2009). Core implementation components.
Research on Social Work Practice, 19(5), 531–540. doi:10.1177/ 1049731509335549 Flaspohler, P., Stillman, L., Duffy, J. L., Wandersman, A., & Maras, M. (2008). Unpacking capac-
ity: The intersection of research to practice and community centered models. American Journal of Community Psychology, 41(3–4), 182–196. doi:10.1007/s10464-008-9162-3
Forman, S. G., Shapiro, E. S., Codding, R. S., Gonzales, J. E., Reddy, L. A., Rosenfield, S. A., Sanetti, L. M., & Stoiber, K. C. (2013). Implementation science and school psychology. School Psychology Quarterly, 28(2), 77–100. doi:10.1037/spq0000019
Gottfredson, D. C. (1984). Environmental change strategies to prevent school disruption. Report prepared for the Conference on Student Discipline Strategies of the Office of Educational Re- search and Improvement. Washington, DC: U.S. Department of Education.
Gottfredson, D. C. (2001). Schools and delinquency. New York: Cambridge University Press.
Gottfredson, G. D., & Gottfredson, D. C. (2001). What schools do to prevent problem behavior and promote safe environments. Journal of Educational & Psychological Consultation, 12(4), 313–344. doi:10.1207/S1532768XJEPC1204_02
Gottfredson, D. C., & Gottfredson, G. D. (2002). Quality of school-based prevention programs: Results from a national survey. Journal of Research in Crime and Delinquency, 39(1), 3–35. doi:10.1177/002242780203900101
Gottfredson, G. D., & Gottfredson, D. C. (2007). School violence. In D. J. Flannery, A. T. Vazso- nyi, & I. D. Waldman (Eds.), Cambridge handbook  of  violent  behavior  and  aggression (pp. 344–358). New York: Cambridge University Press.
Greenberg, M. T., Weissberg, R. P., O’Brien, M. U., Zins, J. E., Fredericks, L., Resnik, H., & Elias, M. J. (2003). Enhancing school-based prevention and youth development through coor- dinated social, emotional, and academic learning. American Psychologist, 58(6–7), 466–474. doi:10.1037/0003-066X.58.6-7.466
Guerra, N. G., & Bradshaw, C. P. (2008) Core competencies to prevent problem behaviors and promote positive youth development. Hoboken: John Wiley & Sons.
Halgunseth, L. C., Carmack, C., Childs, S. S., Caldwell, L., Craig, A., & Smith, E. P. (2012). Us- ing the interactive systems framework in understanding the relation between general program capacity and implementation in afterschool settings. American Journal of Community Psychol- ogy, 50(3–4), 311–320. doi:10.1007/s10464-012-9500-3
Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based men-  tal health programs. Journal of Abnormal Child Psychology, 33(6), 665–679. doi:10.1007/ s10802-005-7646-2
Harn, B., Parisi, D., & Stoolmiller, M. (2013). Balancing fidelity with flexibility and fit: What  do we really know about fidelity of implementation in schools? Exceptional Children, 79(2), 181–193.
Hawkins, J. D., Oesterle, S., Brown, E. C., Monahan, K. C., Abbott, R. D., Arthur, M. W., & Catalano, R. F. (2012). Sustained decreases in risk exposure and youth problem behaviors after installation of the Communities That Care prevention system in a randomized trial. Archives of Pediatrics & Adolescent Medicine, 166(2), 141–148. doi:10.1001/archpediatrics.2011.183
Hulleman, C. S., Rimm-Kaufman, S. E., & Abry, T. (2013). Innovative methodologies to explore implementation: Whole-part-whole—construct validity, measurement, and analytical issues for intervention fidelity assessment in education research. In T. Halle, A. Metz, & I. Marti- nez-Beck (Eds.), Applying implementation science in early childhood programs and systems (pp. 65–93). Baltimore: Brookes.
Johnson, K., Hays, C., Center, H., & Daley, C. (2004). Building capacity and sustainable preven- tion innovations: A sustainability planning model. Evaluation and Program Planning, 27(2), 135–149.
Johnston, L. D., O’Malley, P. M., Bachman, J. G., & Schulenberg, J. E. (2013). Monitoring the future national results on drug use: 2012 overview, key findings on adolescent drug use. Ann Arbor: Institute for Social Research, University of Michigan.


Kam, C., Greenberg, M. T., & Walls, C. T. (2003). Examining the role of implementation qual-  ity in school-based prevention using the PATHS curriculum. Prevention Science, 4(1), 55–63. doi:10.1023/A:1021786811186
Kelly, B., & Perkins, D. F. (Eds.). (2012). Handbook of implementation science for psychology in education. Cambridge: Cambridge University Press.
McLeod, B. D., Southam-Gerow, M., & Weisz, J. R. (2009). Conceptual and methodological is- sues in treatment integrity measurement. School Psychology Review, 38(4), 541–546.
McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodríguez, A., & Smith, M. M. (2013). Mak- ing a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science & Practice, 20(1), 14–32. doi:10.1111/cpsp.12020
Meyer, A. L., Farrell, A. D., Northup, W. B., Kung, E. M., & Plybon, L. (2000). Adaptation of RIPP for cultural and community differences. In A. L. Meyer (Ed.), Promoting nonviolence in early adolescence (pp. 97–115). New York: Springer.
Multisite Violence Prevention Project (MVPP). (2009). The ecological effects of universal and selective violence prevention programs for middle school students: A randomized trial. Journal of Consulting & Clinical Psychology, 77(3), 526–542. doi:10.1037/a0014395
Nation, M., Crusto, C., Wandersman, A., Kumpfer, K. L., Seybolt, D., Morrissey-Kane, E., & Davino, K. (2003). What works in prevention: Principles of effective prevention programs. American Psychologist, 58(6–7), 449–456. doi:10.1037/0003-066X.58.6-7.449
Olweus, D. (2004). The Olweus Bullying Prevention Programme: Design and implementation issues and a new national initiative in Norway. In P. K. Smith, D. Pepler, & K. Rigby (Eds.), Bullying in schools: How successful can interventions be? (pp. 13–36). Cambridge: Cambridge University Press.
Pianta, R. C., & Rimm-Kaufman, S. (2006). The social ecology of the transition to school: Class- rooms, families, and children. In K. McCarthy & D. Phillips (Eds.), Handbook of early child development (pp. 490–507). Oxford: Blackwell.
Reinke, W. M., Sprick, R., & Knight, J. (2009). Coaching classroom management. In J. Knight (Ed.), Coaching: Approaches and perspectives (pp. 91–112). Thousand Oaks: Corwin Press.
Reyes, M. R., Brackett, M. A., Rivers, S. E., Elbertson, N. A., & Salovey, P. (2012). The interaction effects of program training, dosage, and implementation quality on targeted student outcomes for the RULER approach to social and emotional learning. School Psychology Review, 41, 82–99.
Roberts-Gray, C., Gingiss, P. M., & Boerm, M. (2007). Evaluating school capacity to implement new programs. Evaluation and Program Planning, 30(3), 247–257.
Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: Free Press.
Sanetti, L. M. H., & Fallon, L. M. (2011). Treatment integrity assessment: How estimates of adher- ence, quality, and exposure influence interpretation of implementation. Journal of Educational and Psychological Consultation, 21(3), 209–232. doi:10.1080/10474412.2011.595163
Sanetti, L. M. H., & Kratochwill, T. R. (2009). Treatment integrity assessment in the schools: An evaluation of the treatment integrity planning protocol. School Psychology Quarterly, 24(1), 24–35. doi:10.1037/a0015431
Sholomskas, D. E., Syracuse-Siewert, G., Rounsaville, B. J., Ball, S. A., Nuro, K. F., &   Carroll,
K. M. (2005). We don’t train in vain: A dissemination trial of three strategies of training clini- cians in cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology, 73(1), 106–115. doi:10.1037/0022-006X.73.1.106
Sullivan, T. N., Sutherland, K. S., Farrell, A. D., Taylor, K. A., Doyle, S. T., & Ulmer, L. J. (2013). Evaluation of combined individual-level skill-building and school environment interventions: The role of gender and disability status. Manuscript in preparation.
Sutherland, S., & Katz, S. (2005). Concept mapping methodology: A catalyst for organizational learning. Evaluation and Program Planning, 28(3), 257–269.
Sutherland, K. S., McLeod, B. D., Conroy, M. A., & Cox, J. R. (in press). Measuring treatment integrity in the implementation of evidence-based programs in early childhood settings: Con- ceptual issues and recommendations. Journal of Early Intervention.


Trochim, W. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12(1), 1–16. doi:10.1016/0149-7189(89)90016-5
U.S. Surgeon General. (2001). Youth violence: A report of the surgeon general. Washington, DC:
U.S. Department of Health and Human Services.
van Lier, P. A., Muthén, B. O., van der Sar, R. M., & Crijnen, A. A. (2004). Preventing disrup- tive behavior in elementary schoolchildren: Impact of a universal classroom-based interven- tion. Journal of Consulting and Clinical Psychology, 72(3), 467–478. doi:10.1037/0022- 006X.72.3.467
Vo, A., Sutherland, K. S., & Conroy, M. A. (2012). Best in Class: A classroom-based model for ameliorating problem behavior in early childhood settings. Psychology in the Schools, 49(5), 402–414. doi:10.1002/pits.21609
Wandersman, A. (2003). Community science: Bridging the gap between science and practice with community-centered models. American Journal of Community Psychology, 31(3–4), 227–242. doi:10.1023/A:1023954503247
Wilson, S. J., Lipsey, M. W., & Derzon, J. H. (2003). The effects of school-based intervention programs on aggressive behavior: A meta-analysis. Journal of Consulting and Clinical Psy- chology, 71(1), 136–149. doi:10.1037/0022-006X.71.1.136
Winter, S. G., & Szulanski, G. (2001). Replication as strategy. Organization Science, 12(6), 730– 743. doi:10.1287/orsc.12.6.730.10084
Wolery, M. (2011). Intervention research: The importance of fidelity measurement. Topics in Early Childhood Special Education, 31(3), 155–157. doi:10.3102/0034654307313793