Succeeds Prize Selection Process

Overview of the Selection Process for Impact Awards

Colorado Succeeds developed a three-step process by which the Selection Committee selected The Succeeds Prize winners.

• The first step in the selection process used publicly available performance data to develop the Succeeds Index, later used to identify 25 top performing schools for each award.
• Next, the committee narrowed down the top 25 schools in each category to a top three through a deliberation process that was informed by additional surrounding data.
• Lastly, members of the Selection Committee visited each of the finalists and reconvened to select a winner based on the data identified in step two and what was observed during the site visits.

Though the selection process was organized by Colorado Succeeds, decisions within each step were driven by members of the Selection Committee. Additionally, Colorado Succeeds hired Augenblick, Palaich and Associates (APA Consulting) to support the process.

Step One: Development of the Succeeds Prize Index

The Succeeds Prize index was developed to identify a group of 25 schools for each award to serve as the basis for the second step in the selection process. The index uses data from the Colorado Department of Education’s School Performance Framework, which provides a validated source of data that is well understood across the state. Specifically, the index utilizes growth measured by median growth percentiles, mean scores in achievement (CMAS and ACT), and high school graduation rates (counted as part of achievement).

The index is composed of three parts. The first measures growth, the second measures overall academic achievement, and the third accounts for achievement gaps within a school, where gaps are weighted negatively to select for schools with small or no achievement gaps.

As the Committee reviewed the initial set of data, they recommended a change to the input data used for the gaps composite measure. The original gaps composite measure used the difference between the school average performance and average performance of minority (non-white) students in a school. The committee recommended use of the median growth percentile for free and reduced lunch eligible students (a standard measure of poverty) as the input for the gaps measure. This change was made before identifying the group of 25 schools for each award.

The basic formula is the same for each prize. However, the input data used for each composite measure varied between prizes.

In the Transformational Impact awards the following data were used for the index:

• The achievement composite was the average CMAS scale scores for English language arts (ELA) and math
  • The high school level award also included:
    • ACT composite score
    • Average weighted 4, 5, 6, and 7-year graduation rate
  • The growth composite was consisted of the ELA and math median growth percentiles
  • The gaps composite used the ELA and math median growth percentiles for students who qualified for free and reduced lunch.

The Excellence in STEM award’s index used the following data:

• The achievement composite was the average of CMAS scale scores in math and science
• The growth composite consisted of median growth percentile in math
• The gaps composite used the median growth percentile in math for students who qualify for free and reduced lunch.

Step Two: Selecting the Top Three Schools for Each Prize Category

The Selection Committee was responsible for narrowing the top 25 schools as identified using the index to a top three schools in each category.

Surrounding Data

A key tool for this process was gathering surrounding data to provide additional information on each schools’ contexts. The surrounding data provided information on the following components of school context:

• School ratings;
• Student demographics;
• School performance; and
• District context.

The full list of surrounding data is provided in a chart at the bottom of this page.

During this process, the names of the top 25 schools for each award were masked. This was done to support the Selection Committee’s use of common data for the selection, instead of personal impressions or connections to schools or districts. Schools were identified with pseudonyms.

Process of Identifying Top Schools

The Selection Committee met to determine the top five schools in each of the four award categories using the information explained above. They debated the merits of various schools based on that information, and ultimately voted via blind online ballot, provided by APA Consulting. Selection Committee members voted for their top five schools within each category. Then, APA informed the group of the top five schools and the Selection Committee members again discussed those five schools. Finally, Selection Committee members ranked the top five campuses identified by the group.

Upon reflection on the top campuses, some members of the Selection Committee expressed concern over the lack of geographic diversity in the top three finalists. Colorado Succeeds then organized a series of conference call whereby the Committee decided it would revisit its decision and looked at the top six to eight campuses based on the initial online ballot. This group was much more geographically diverse, and after another round of discussion, the Selection Committee re-ranked their top five campuses in each category.

Step Three: School Visits and Final Selection

With the Selection Committee done ranking their top five campuses in each award category, APA Consulting unmasked the actual names of the schools that had been chosen. While the Selection Committee would visit only three campuses for each award, the remaining two of the top five identified in step two would be recognized as “Honorable Mention” schools, given that they were also selected by the Committee for their excellence.

The Selection Committee split up to visit all twelve of the finalist campuses before reconvening to inform one another about what they had seen. To make the site visit process as consistent as possible, members of the Selection Committee formed sub-committees for each award, so that the same groups would visit the schools with in a singular award category.

During the school visits, Selection Committee members interviewed school administrators, teachers, parents, and students, in addition to observing classroom teaching. Throughout the visits Selection Committee members gathered evidence as to why these finalist campuses were so successful and what learning could be shared with educators across the state.

The final Selection Committee meeting was held to share what each sub-committee had learned from their site visits and ultimately to vote on a winner in each category. As before, Selection Committee members debated, asked questions, and shared why they believed certain schools offered a model of excellence that was relevant to share statewide. After the discussion on each award, APA Consulting collected rankings of the finalist campuses by private ballot. This final vote determined the winners in each category. The winners were not revealed to the Selection Committee prior to the live-reveal at The Succeeds Prize on Oct. 3.

Next Steps

The finalist and winning schools have all agreed to help share their stories with other schools across the state. We will share more about these excellent schools in the coming weeks and months.

Colorado Succeeds and APA Consulting will work with members of the Selection Committee to collect feedback on the inaugural year. Throughout the process, the team has identified areas of growth for the awards, including building in opportunities for schools to submit additional information, collecting supporting documentation for specific programming, collection of community or partner collaboration, and facilitating a process where the Selection Committee can learn earlier in the process about other indicators outside the School Performance Framework. Any changes to the selection process will be updated on our website prior to school identification.

Appendix A: Indicators in the Surrounding Information Provided to the Selection Committee

Background and Demographic Information

• Rating assigned by Great Schools
• Grade assigned by Colorado Succeeds as part of Colorado School Grades
• Proportion of students eligible for free and reduced priced lunch, a measure of poverty
• Proportion of students who are learning how to speak English
• Proportion of students who have an individual education plan which indicates they qualify for special education services
• Proportion of non-white students
• Proportion of students that are black
• Proportion of students that are Hispanic
• Proportion of students that are Asian or Pacific Islander
• Proportion of students that are White
• Proportion of students that are 2 or more races
• Unduplicated count of students in the school who moved into the district divided by total enrollment
• Count of students enrolled

Achievement Composite

• Mean scale score for English Language Arts, all students
• Percentile ranking based on English Language Arts mean scale score
• Test participation rate, English Language Arts
• Mean scale score for Math, all students
• Percentile ranking based on Math mean scale score
• Test participation rate, Math
• Mean scale score for Science all students
• Percentile ranking based on Science mean scale score
• Test participation rate, Science
• Median growth percentile, English Language Arts
• Median growth percentile, Math

Gaps Composite

• Mean scale score for English Language Arts, students eligible for free or reduced lunch
• Percentile rank based on the FRPL student’s mean scale score
• Mean scale score for Math, students eligible for free or reduced lunch
• Percentile rank based on the FRPL student’s mean scale score
• Mean scale score for Science, students eligible for free or reduced lunch
• Percentile rank based on the FRPL student’s mean scale score
• Mean scale score for English Language Arts, English language learner students
• Percentile rank based on the English Language Learner students’ mean scale score
• Mean scale score for Math, English language learner students
• Percentile rank based on the English Language Learner students’ mean scale score
• Mean scale score for Science, English language learner students
• Percentile rank based on the English Language Learner students’ mean scale score
• Mean scale score for English Language Arts, minority students
• Percentile rank based on the minority students’ mean scale score
• Mean scale score for Math, minority students
• Percentile rank based on the minority students’ mean scale score
• Mean scale score for Science, minority students
• Percentile rank based on the minority students’ mean scale score
• Mean scale score for English Language Arts, students with disabilities
• Percentile rank based on the students with disabilities mean scale score
• Mean scale score for Math, students with disabilities
• Percentile rank based on the students with disabilities mean scale score

Geographic and Funding Information

• Region in the state based on CDE’s 8 state regions
• Total per pupil revenue
• Total enrollment in the district
• School locale based on Census Bureau designations