Spring training has ended, and the regular season has begun for my Colorado Rockies and Kansas City Royals. The games, and the stats, now matter.

In our schools, High Stakes Testing Season has begun, and the results of these assessments tend to matter much more than all the others we have administered during the first 3/4 of the school year.

Schools, like professional sports, are data driven institutions. But as was pointed out in a Wired Magazine article a few years ago, the numbers crunchers don’t always “get it right.”

Why Quants Don’t Know Everything

Schools exist to create successful students, students don’t attend to create successful schools. If all students are successful, it follows that the school will also be see as a “high performing” school. The oxymoron, however, is that high achieving schools are not always filled with a high percentage of successful students. Test scores are neither the only nor the best measures of a successful school, they are just the easiest measure to derive. The growth of individual students cannot be reduced to, or expressed by a single score. Higher test scores alone do not meet the test of continuous improvement.

We tend to work with three types of assessments: comparison; instruction; and evaluation. Comparison assessments look at score-based differences between individual students or among groups of students. This naturally leads to classification. Instructional assessments tell us how well the students are doing so instructional decisions can be made on how to improve instruction.  Evaluative assessments help us answer the question of how well did it work.

Comparison dominates during this season. The results provide few instructional insights and lead to inaccurate evaluations of instructional quality.

I have seen too many schools aggregate the data, then analyze the results. In my elite schools, we analyzed the data, then aggregated. This let us find patterns with individual students rather than find patterns within the groups. Instructional evaluation starts with the individual student, not with the group. This perspective kept us from falling into the trap of seeing the scores of our lowest performing students staying static, scores of the highest performers increasing, deluding us into believing that our test scores were rising. Yet the gains did not happen for all students. In other words, not every boat rose as the level of the water rose.

Ok, Gary, that is clear as mud. What would you look at if you were looking at the performance of my school? How would you determine if we are a successful school for all students, and meeting our stated goal of continuous improvement?

Most of us have swallowed the RTI kool-aid. I would look at:

• How many students are at the lowest level for multiple years? Which students are at the lowest level for multiple years?
• How many students drop a level between spring testing one year and fall testing the subsequent year? Is this an acceptable number?
• What is the correlation between end of year course grades and high stakes assessment scores in the tested subject areas? If the high stakes assessments truly reflect the curriculum being taught, should there be a correlation?
• Look at scores through the filter of quartile of age. What percentage of the lowest performing groups are also in the youngest quartile at that grade level? Are the low scores a function of age? Particularly important for those of us who screen our entering kindergarten students, is their performance a function of academic preparation, developmental age, or both? How do we know?

Looking beyond the model:

• How successful are students the next year? Particularly between the transition years – e.g. 5th to 6th between elementary and middle school, 8th to 9th between middle school and high school. Are the high performers in 8th grade still the high performers in 9th? Prepare to be surprised when you dig into this one.
• In larger schools where multiple teachers teach the same class, compare student performance by teacher. Would a high performing student in Teacher A class also be a high performing student in Teacher B class. Sounds simple, again, prepare to be surprised.
• Continue to correlate course grades to other assessment scores. At the secondary level, is there a correlation between ACT, SAT, PSAT, and AP scores and course grades in the tested subjects? Should there be?

But Gary, that’s all about test scores, and you certainly don’t believe that test scores alone are the measure of a successful school. What else do we need to look at?

During the 2003-2004 school year, I was the Director of High School Programs at the Central Educational Center in Newnan, Georgia. That year, we were named a National Model High School by the International Center for Leadership in Education, Willard Daggett’s group. Over 300 schools from across the country were nominated, 30 were selected. To quote from the summary report prepared by ICLE, “The 30 high schools included in the Bringing Best Practices to Scale initiative have provided great insight into how American high schools can help all students complete an academically rigorious and relevant curriculum. Especially insightful is the comparison of findings from these 30 high schools to the many good high schools that will need to make some further changes before they can be classified as great in terms of all students’ academic success.”

After we were notified that we had been nominated by our State Superintendent, we had some work to do in compiling data to submit to ICLE as part of their evaluation of our performance. Interestingly, SAT/ACT participation rates and results on prior year state exam results were the only “test score” data points on a very extensive list.

So in addition to the basic demographic data of student enrollment, student racial/ethnic origin, free/reduced lunch count, ELL learners, students with disabilities, per pupil expenditures and staffing, here is the “what else” I would look at:

• Analysis of curriculum using the Rigor/Relevance Framework
• Evidence that all students have access to a challenging curriculum
• Degree of focus and connection of curriculum to real-world requirements and performance levels
• Evidence that assessment practices are varied, collaborative and shared within the school
• Instructional practices/strategies
• Literacy development, which is very different than looking at reading scores or grades in English classes
• Evidence of instruction driving the schedule, organization of teachers, class assignments of students, and professional development activities
• Evidence of clear expectations and accountability
• Internships
• Community service
• Availability, training in, and use of technology in the school. Selling out to Google, chromebooks, apps and google docs doesn’t work for this one.
• Use of data
• Professional development
• Extent of professional collaboration focused on student learning
• Extent the ongoing development of the school is sustained
• Evidence of a clear, focused, well articulated vision and mission
• Evidence of high expectations for achievement of all students
• School policies that relate to student performance
• Evidence of sustained student relationships with adults
• Evidence that students are supported to achieve at high levels
• Evidence that the environment is authorities, safe, ethical, and studious. The staff teaches, models, and expects  responsible behavior; relationships are based on mutual respect
• Evidence of meaningful opportunites for all members of the school to demonstrate leadership
• Evidence that leadership encourages the sharing of knowledge, and the development of new knowledge, within and among learning community members
• Evidence that’s the principal seeks, shares, and promotes leadership among staff members in the quest for continual improvement in School effectiveness
• Evidence that acquiring and developing a high-quality staff is considered a high priority
• Types and extent of parent and community partnerships

Quite a laundry list to look at. Data was collected during the site visit through up to 15 classroom observations, interviews, and a review of curriculum and student work. The review team then completed a rubric based on 10 key components fundamental to school improvement.

Obviously, we “made the team.”

We were a public charter high school with an enrollment at that time of  1,123 students, 14 certificate programs, 18 industry mentors, and a 5 year average graduation rate of 98%. We also “guaranteed” our graduates to post-secondary programs and industry, but that is another story for another day.

We all pay lip service to continuous improvement, yet look only to measures of student growth based on test scores to measure our progress. It is more complex than a single set of numbers can measure. Let’s spend more time looking at individual student performance in all the skills and competencies required for high performance once they leave us and become employees rather than just a snapshot taken over a few days of testing. We owe it to our kids.

And so it goes…