A Program Evaluation of the Effectiveness of Turnaround’s Model at Implementing and Operating Clear and Consistent Protocols within Schools

NOTE* Due to inability to insert graphs, the results analysis and appendices have been removed

 

Question: Does implementing school protocols help school staff feel better prepared to manage students with mental health/behavioral problems?

Hypothesis: No, protocols in themselves cannot be effective if school staff do not have the resources to operationalize the protocols well.

Turnaround for Children utilizes an entire school transformation model to improve poorly performing public schools in New York City and Washington D.C.  One aspect of this model requires the school to define clear school-wide protocols to manage student behavior.  When clear protocols are operationalized, a more positive school climate can be achieved (Cohen, McCabe, Michelli, & Pickeral, 2009). Students better understand how they should behave and what the consequences are when they do not follow the rules, and teachers are better aware of how they should manage problem behaviors. Problem behaviors are often linked to behavioral/mental health issues and diagnosis and require appropriate interventions (Center for Mental Health in Schools at UCLA, 2008).  Clear protocols also lead to earlier identification of those students that require extra support for these issues (SAMHSA, 2011). Earlier identification means the students are better served and can increase capacity and learning at school.

Because of the importance that having clear protocols to manage student behavior has, it is necessary to follow-through and make sure that these protocols are being operationalized consistently. The existence of these protocols may have no meaning if they are not being used appropriately. Issues with the operation of school-wide protocols may include a lack of resources.  For example, student support workers (ie: school social workers and guidance counselors) are necessary in supporting teachers when a child is disrupting a class and the teacher does not have the ability to attend to the needs of the one student.  Often, schools do not employ enough student support workers to adequately provide services to all the students that need it within the school (NASW Center for Workforce Studies, 2010). Other times, teachers are not able to attend trainings that explain school protocols to refer students to mental health services.  Administrators need to take these issues into consideration in order to promote positive climate in their schools.

Turnaround for Children (TFC) has gathered data via a school staff survey in order to evaluate how their model works within schools, and if school staff feel any improvements while undergoing the transformation process. This survey was collected via surveymonkey.com, sent to all known staff at each of the schools TFC works with in New York City and Washington D.C.

In order to answer my research question, I focused on those questions from the school staff survey that operationalize the presence and implementation of consistent and clear protocols (also known as the Input) and the questions that operationalize the operations within the school that are consistent with the protocols (also known as the Output), measured against how many years the school has been using the Turnaround model, the size of the school, and the level of the school.

Using the answers to the above questions, I have conducted statistical testing using SPSS statistical analyzing software.  Using t-tests and Anova significance testing I can compare means that will show the difference between how school staff feel about their schools’ protocols, how the protocols are being operationalized, and how school staff feel about their own ability to manage behavioral/mental health problems.

Data and Methods

Data was collected by survey (via Survey Monkey) from 26 different public schools across New York City and Washington D.C. The surveys included 27 questions for all school staff members to complete (School Staff Survey, Appendix A).  If the survey-taker identified their role as teacher, they were asked to complete a supplemental questionnaire pertaining to their experience in the classroom.  Each question is answered on a four point Likert-type scale for Never, Occasionally, Usually, Always; or a 10 point Likert-type scale where respondents are asked to rate how they feel from Very Poor to Excellent.  The intent of the survey is to evaluate how effective the Turnaround model is at improving the functioning of the school.  Not every survey question pertains to the variables in this study, so only pertinent data will be analyzed to answer the research question.

The independent variable in this study is the “Input” of school-wide expectations and protocols.  Turnaround for Children trains their field staff to go into their partner schools and to introduce these protocols as a tool to manage student misbehavior. These protocols are based on research conducted by the agency and are consistent across each school, with variations depending on the type of school (elementary, middle, middle/high, or high school) and the needs of the individual school. The dependent variable, the “Output”, is how efficiently these protocols are implemented within the schools; if the school staff feel confident and comfortable using the protocols to manage student misbehavior.  Indices were created using survey questions that operationalize these concepts.

In order to determine which questions from the School Staff Survey were going to be used as indices for the two variables, a factor analysis was conducted in SPSS (Appendix B, Goldstein[1]). The factor analysis pulled out 12 questions as creating indices that hang together well, 7 in the Input index, and 5 in the Output (See Descriptive Statistics in Appendix C). The Input consists of the following questions: There are clear protocols defining responses to student behavior? Administrators respond consistently to student problem behavior? Student expectations are taught to all consistently? There are  clear school-wide expectations for student behavior? Staff consistently reinforce school-wide aspirations for positive behavior for all students? There are clear aspirations for positive behavior? There are clear protocols resulting in office referrals? These seven questions test together with a Cronbach’s Alpha of .902. The Output consists of these questions: Follow-Up information about students referred is given? Referring staff are supported to fulfill role in student intervention plans? Student referrals to behavioral support systems are easy to complete? Referring staff know role in student intervention plans? Do staff meet to discuss social, emotional, and mental health needs of students? These four questions test together with a Cronbach’s Alpha of .873.

When combined, these indices create variables with significantly different means, proven by conducting a t-test in SPSS with the result of p<.001.

 

Conclusion:

Based on the data collected this year, we see that there is a small, but significant, decrease of school staff satisfaction when viewing the clarity of protocols taught to them at the beginning of the year, and the way these protocols are consistently used throughout the year. This can be caused by any number of reasons, lack of follow-through by school staff, not every staff member was well-educated on their role within the protocol system, a lack of urgency taught around the need to remain consistent throughout the year, or no discourse when the system breaks down.  The most significant differences in means are between small and large schools and between the first and second year of engagement.  These differences may lead to different protocols needing to be established in schools based on the size of their schools or a different approach to schools based on their year of engagement.

In order to change the current Turnaround model in an effective and meaningful way to improve these scores for the future and lessen the discrepancy between how satisfied school staff are with the establishment of protocols and how well these protocols are operationalized, a study has to be done.  Qualitative interviews with staff members asking questions that have been developed based on the results of this analysis would be beneficial in knowing why school staff feel this way and what should be done to change it.

 

 

 

References

Center for Mental Health in Schools in UCLA. (2008). Conduct and behavior problems:

intervention and resources for school aged youth. University of California Los Angeles.

            Los Angeles, CA.

 

Cohen, J., McCabe, E., Michelli, N., & Pickerel, T. (2009). School climate: Research, policy,

practice, and teacher education. Teacher’s College, Columbia University. 3:1. 180-213.

 

National Association of Social Workers for Workforce Studies and Social Work Practice. (2010).

Social workers in schools: Occupational profile. NASW. Washington, D.C.

 

Substance Abuse and Mental Health Services Administration. (2011). Identifying mental health

and substance abuse problems of children and adolescents: A guide for child-serving

organizations.  Office of Child, Adolescent, and Family Branch, Substance Abuse and

            Mental Health Services Administration. Rockville, MD.

 

[1] Leah Goldstein, LMSW, supervised this statistical test.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s