Two published articles are available regarding the results from the initial Beacons demonstration project that was implemented in four elementary schools. The references for these articles are:
Walker, B., Cheney, D., Stage, S., & Blum, C. (2005). Schoolwide screening and positive behavior support: Identifying and supporting students at risk of school failure. Journal of Positive Behavior Interventions, 7, 194-204.
Cheney, D., Blum, C., & Walker, B. (2004). An analysis of leadership teams' perceptions of positive behavior support and the outcomes of typically developing and at-risk students in their schools, Assessment for Effective Intervention, 30, 7-24.
School-Wide Evaluation Tool (SET)
The SET is designed to assess and evaluate the critical features of school-wide positive behavior support (PBS). These critical features include: defining expectations, teaching behavioral expectations, developing an ongoing system of rewarding behavioral expectations, responding to behavioral violations, monitoring and decision-making, management, and district level support. The SET was administered yearly to schools in the Beacons Outreach Project to assess each school's progress with implementing the essential features of school-wide positive behavior support. Project staff visited schools during the winter of each school year and completed the SET.
Schools were divided into two cohorts based on their entry into the project. Twelve schools began the project in the fall of the 2003-2004 school year (Cohort 1). Cohort 1 attended initial staff training in August of 2003, and the schools level of implementation of PBS practices was assessed (using the SET) in the winter of the 2003-2004 school year after their initial training. Thus, the 2003-04 data that are displayed are initial post-training data and do not represent a true baseline. Level of PBS implementation was again assessed for all Cohort 1 schools in the winter of the 2004-2005 and 2005-2006 school years. Several Cohort 1 schools also agreed to participate in a follow-up year of data collection, during the 2006-2007 school year. School progress for these years is displayed in the figure below.
A second group, of eight schools, began the project in the fall of the 2004-2005 school year (Cohort 2). Cohort 2 attended their initial staff training in August of 2004, but their first SET was done prior to training. Thus, their first displayed data represent a true baseline on the SET. Implementation was again measured for all Cohort 2 schools in the winter of the 2004-2005 and 2005-2006 school years. Several Cohort 2 schools also agreed to participate in a follow-up year of data collection, during the 2006-2007 school year. The data below display their progress on implementation of PBS features for these years.
SET results for these 2 cohorts during years of project participation are reported below. A total score of 80 or above indicates successful implementation of school-wide PBS practices. Results show that both cohorts had reached acceptable levels of implementation of PBS practices during years 2 and 3 of the project.
Scores on the SET for Cohort 1 During Involvement With The Beacons Project
| SET Feature | 2003-04 | 2004-05 | 2005-06 | 2006-07 |
| Expectations defined | 81 | 88 | 97 | 94 |
| Expectations taught | 73 | 91 | 95 | 93 |
| Reward system in place | 72 | 90 | 80 | 96 |
| Responding to problem beh. | 79 | 79 | 87 | 89 |
| Monitoring | 77 | 92 | 98 | 78 |
| Management | 80 | 89 | 92 | 70 |
| District support | 88 | 100 | 100 | 39 |
| Total | 78 | 90 | 93 | 80 |
Scores on the SET for Cohort 2 During Involvement With The Beacons Project
| SET Feature | 2003-04 | 2004-05 | 2005-06 | 2006-07 |
| Expectations defined | 36 | 77 | 88 | 96 |
| Expectations taught | 38 | 85 | 83 | 95 |
| Reward system in place | 53 | 80 | 86 | 97 |
| Responding to problem beh. | 58 | 84 | 82 | 86 |
| Monitoring | 44 | 92 | 98 | 96 |
| Management | 46 | 93 | 90 | 91 |
| District support | 56 | 99 | 100 | 58 |
| Total | 45 | 90 | 89 | 88 |
Self-Assessment
The BEACONS Project at the University of Washington has developed a PBS Self-Assessment Survey (Cheney & Walker, 2003), which is intended to cover evidence-based practices across all three levels of the PBS model. This tool is in its third iteration after being used in previous projects dating to 1995. The current version of the survey has 10 subscales regarding evidence-based practices. The practices that are measured include policies and procedures, prevention and early identification, staff development, behavioral expectations, procedures for discipline referrals, systematic support for students with behavior problems, Functional Behavior Assessment procedures, data-based decision making, family supports, and intensive level support. Each subscale includes a range of items or indicators (5-8 per subscale) related to the practice or systems supports at levels of the PBS model.
The survey was designed for use by school leadership teams who are charged with implementing PBS systems in their schools. Team members are first asked to individually complete ratings on all indicators of PBS implementation using a five point Likert scale. They then meet as a team to discuss their findings and determine a team rating for each indicator. Indicators are summed within each subscale for a subscale score. Raw scores are then transformed to percentages of implementation for each subscale and those percentages are charted yearly. The results provide feedback to the Leadership team on how successful they perceive their implementation of PBS is in their school. The results also help identify areas for school improvement efforts.
The PBS Self-Assessment Survey has been demonstrated to be a efficient tool to complement other schoolwide data such as SET or ODR results The survey was shown to have adequate reliability and validity in a recent dissertation conducted by Bridget Walker (The Initial Psychometric Assessment Of The BEACONS Positive Behavior Support Individual and Team Self-Assessment and Program Review, University of Washington, 2005).
The results below indicate how schools rated themselves in each of the 10 subscales during each year of participation in the project. Scores that exceed 70% suggest that the school’s leadership team have rated the practice to be mostly in place. As a group, schools in the project rated 9 of the 10 practices to be mostly in place in the final year of the project. The results also suggest that the schools rated their growth consistently across the 10 years of the project to attain the 70% proficiency level. The one practice that was rated lowest and below the mostly in place level was that of Staff Development. This practice asked the team if they have a plan in place and regular inservice training for staff to increase their knowledge and skills for children with emotional or behavioral problems. Even though this project provided ongoing training on this practice, staff still thought more staff development was needed through inservice training. This finding demonstrates the challenges in a school when students have difficult behavior in general or special education classrooms. Further ongoing training and support is needed in these schools to increase staff’s competence for working with children with challenging behavior.
| Survey | 2003-04 | 2004-05 | 2005-06 |
| Policy/Procedure | 74 | 85 | 93 |
| Prevention/early id | 62 | 76 | 84 |
| Staff development | 45 | 65 | 68 |
| Clear expectations | 69 | 85 | 92 |
| Response to discipline | 65 | 85 | 88 |
| Systematic supports | 60 | 78 | 78 |
| FBA | 47 | 64 | 84 |
| Data based | 58 | 81 | 88 |
| Families | 59 | 75 | 78 |
| Intensive support | 56 | 75 | 83 |
| TOTAL | 59 | 77 | 84 |
Results are displayed as the percentage of the subscale that the leadership team perceived as fully implemented at their school. For example, if a subscale had a total raw score of 35 (7 items with a high rating of 5), and a lead team rated each item as a 4 (almost fully implemented), they would have a raw score of 28 for the subscale and a percentage score of 80 for the subscale (28/35).
Office Discipline Referrals (ODRs)
Each school monitored their ODR by using the School-Wide Information System® (SWIS). SWIS is a web-based computer application for entering, organizing, and reporting office discipline referrals. SWIS can be used to create numerical reports or charts to reflect individual or school-wide discipline reports. This information can be used for internal decision making on school-wide discipline issues, to support development of an individual behavior support plan, or for creating school, district, or state discipline reports.
The national average of office discipline referrals for schools in the PBS network in 2006 was .37 ODR per day per 100 students (University of Oregon's PBS Center, 2006). Over time schools report decreasing numbers of ODR yearly.
During participation in the Beacons project, schools used SWIS to track ODR. Results are reported below for Cohort 1 and Cohort 2 (Note: Cohort 2 did not use SWIS during the 2003-04 school year). By the conclusion of the project, school averages were below the national average of .39. Data from participating schools is also presented below from the follow-up year of data collection (2006-7).
Office Discipline Referrals per day per 100 students for Cohort 1 & Cohort 2
Cohort 1: 2003-04 (.22), 2004-05 (.28), 2005-06 (.30), 2006-07 (.28)
Cohort 2: 2004-05 (.41), 2005-06 (.36), 2006-07 (.38)
ODR results for two schools over 3 years of the Beacons Project
School 1: 2003-04 (.35). 2004-05 (.26), 2005-06 (.23)
School 2: 2003-04 (.07). 2004-05 (.04), 2005-06 (.02)
Social Skill Rating System (SSRS - Gresham & Elliott, 1990)
The SSRS was used to assess individual student progress for students who were identified through SSBD screening as demonstrating challenging behaviors. Students who were identified in the screening and who had parent permission were included in the targeted group intervention. The SSRS is a 50-item rating system completed by teachers. It assesses three domains that include school social skills, problem behaviors, and academic competence of students. Because the project focused on improving students' social behavior, we report results for two subscales of the SSRS: Social Skills and Problem Behavior. Results are reported below for 126 students who were involved in the Behavioral Education Program (BEP) during the 2005-06 school year.
We hypothesized that teachers would perceive students enrolled in a targeted group intervention as improving on social skills and decreasing problem behavior. The data reported show the group changes for the 126 students who were enrolled in the BEP (targeted group intervention). The SSRS was completed by teachers in May 2005, prior to implementation of BEP, and again in May 2006 after one year of implementing the BEP. Using a repeated measure ANOVA for each scale of the SSRS, results were significant for both scales (Social Skills, F=5.22, p= .02; Problem Behavior, F=21.15, p<.0001). Teachers perceived their students to become more competent in their use of social skills and to decrease their problem behaviors in school. In addition to being statistically significant, the groups also moved into the normative range of functioning (within one standard deviation from the mean) on the SSRS.
Results are reported as standard scores (X=100; SD=15) for the group of 126 students who were enrolled in the Behavior Education Program
