Educational Outcomes and Indicators

Suppose that you go to your doctor to learn the results of your annual physical examination and blood tests. She warns you that your blood pressure is high and so is your cholesterol. She checks your weight and asks about your diet and exercise regime. She warns you that unless you make some serious lifestyle changes, you are at risk of a heart attack. You take her advice seriously and at your follow-up visit some time later, all of your risk indicators are back to a healthy level.

Contrast this scenario with the typical response to wide-scale national and international assessments like the Pan-Canadian Assessment Program (PCAP) and the Programme for International Student Assessment (PISA). In Manitoba, where our latest rankings, compared to other Canadian provinces, are seen to have slipped, the Winnipeg Free Press editorial argued, “Teachers’ salaries don’t match results” (September 4, 2015). Others point the finger at other supposed provincial culprits, such as the absence of provincial standardized tests or “educational fads, like discovery learning.”¹

In the face of disappointing results, what should we do? Pay teachers less? Should we go back to the basics, have kids write more standardized tests, or remedy child poverty? It’s a little like arguing whether a fatal heart attack was caused by hereditary or lifestyle factors – either way, you are already dead! Basically there are precious few lessons to be learned from provincial and national rankings alone. If we want to improve our education systems, we need to look to other indicators, ones that: 1) reflect the range of important goals we have for our schools and 2) have predictive value.

A comprehensive set of indicators

It goes without saying that we – individual parents and students, teachers, administrators, politicians, and all members of the public – need to have reliable accounts of students’ individual and collective experiences in schools. It is also widely accepted that the academic well-being of students is a core responsibility of schools and something that needs to be properly accounted for. But there is far less agreement on the extent to which we can, or should, take national and international assessments, collected at specific times for specific subject areas and specific grade levels, as either appropriate indicators of “student success” or, more broadly, as indicators of overall school quality.

Shifting our focus to leading indicators

PCAP and PISA scores are highly influential examples of trailing or lagging achievement/assessment indicators; high school graduation rates are another. These may be important measures of student achievement across schools and school systems in selected academic subjects and they may report on important school and school system goals, but they are summative assessments and invariably the data from them – even where it is reported in a form that could be useful – arrives too late to help individual students who may be struggling.

While the media and politicians may pay more attention to these high-profile, narrowly focused, comparative data sets, educators and parents may be better served by focusing more of their attention on developing and sharing leading indicators. Leading indicators are progress-oriented measures that precede the eventual achievement gain/educational outcome. They tell us whether or not these gains are likely to occur in the future – and, critically, they are formative and have far more potential for individual teachers, with the support of school and school system administrators, to create positive futures for the actual students the data is being collected from. Here are five examples of what we mean.

1. Early Reading Proficiency/Reading for Pleasure

We know that if children aren’t reading well by the end of Grade 3 they will struggle throughout their remaining school years. We can predict that and we need to do something about it, both before and after Grade 3. We also know that students who read for pleasure are stronger readers than those who don’t. Both these indicators have strong predictive value and they give us information we can do something about. They help us know what we should focus on.

2. Over-age/Under Credited Students – Grade 9 Credits

Failing to pass even a single Grade 9 course and having to repeat it with students who are a year younger is highly predictive of failing to graduate. Using this measure provides high schools with a pretty good indication, within the first few months of Grade 9, of who will struggle to graduate four years later. With this data, we have the opportunity to provide extra help, opportunities to complete missing work, or a chance to complete the work in summer school in order to keep the student “on track” for graduation.

3. Student Intellectual Engagement

The level of a student’s intellectual engagement is highly predictive of student attainment. Intellectual engagement – by which we mean a personal commitment to and investment in learning – is nurtured by learning activities that are both challenging and appropriate to the student’s current skill level. Simply put, students who are more engaged work harder and learn more. They aren’t bored and they aren’t anxious. Schools can measure engagement. Teachers can simply observe it, and more importantly they can challenge bored students, support anxious students, and do more of what engages all students.

4. Extra-Curricular Involvement

Schools should prepare students for life. Interestingly, extra-curricular involvement is a better predictor of success in life than grades. It’s really not all that surprising that students who are passionate and involved, who can manage their time and juggle competing demands, grow up to be successful adults. Can we expand opportunities for students to develop these important life skills?

5. Special Education Enrollment

An indicator of a whole school system’s effectiveness is how it is doing with regards to students with special needs. Are children with special needs identified and helped to succeed early in their education, as they are in Finland, which is the Western world’s top school system according to PISA results? Or do special education student numbers grow and grow, with few of them ever flourishing? The goal of extra help for students should be to increase their success rather than to limit their future. An indicator of a school system’s effectiveness is that students receiving special needs support become successful, not dependent.

Schools and school systems need to find a balance between leading and trailing indicators. Outcomes like having all children read fluently with understanding and having all children graduate are the destinations we want to reach. Reading for pleasure and getting full course credits in Grade 9 are the signposts that tell us we’re on the right track to reach those destinations. We need both destinations and signposts.

If we only know that a school or a province has high standardized test scores, we don’t know what is working and why. And if the scores are low, we don’t know how to improve them. Or, as Ellen Foley and her colleagues wrote:

Collecting information only on lagging [trailing] indicators… is like ‘playing with the scoreboard off.’ When the buzzer goes at the end of the game, you flip the scoreboard on and say ‘Wait a minute. I thought we were ahead.’ ²

And finally, we must recognize that key to student success is the daily work of schools and teachers asking, “What do we need to do to get this student back on the path to success?” The key to improvement lies in doing this daily work with skill and with heart. We cannot wait for the end to poverty and racism, and we cannot put our faith in simple solutions.

En bref: Dans cet article, les auteurs font ressortir les limites découlant du fait d’accorder trop d’attention aux comparaisons provinciales et nationales des résultats des évaluations à grande échelle comme le Programme pancanadien d’évaluation (PPCE) et le Programme international pour le suivi des acquis des élèves (PISA). Plutôt, selon eux, si nous désirons améliorer nos systèmes d’éducation, nous devons tenir compte d’un ensemble élargi d’indicateurs qui 1) reflètent l’éventail des objectifs importants que nous établissons pour nos écoles et 2) comportent une certaine valeur prédictive et nous permettent d’intervenir dans l’apprentissage des élèves de façon à promouvoir leur réussite. Le défi consiste à travailler avec un ensemble robuste d’indicateurs de « tendance » et de « suivi » – un équilibre d’objectifs visés et de marqueurs.

Photo: iStock

First published in Education Canada, March 2016

1 Anna Stokke, “No surprise in Manitoba students’ poor math showings,” CBC News, Oct. 8, 2014. www.cbc.ca/news/canada/manitoba/no-surprise-in-manitoba-students-poor-math-showings-1.27925422

2 E. Foley, J. Mishooke, J. Thompson, M. Kubiak, J. Supovitz, and M. Rhude-Faust, Beyond Test Scores: Leading indicators for education (Providence, RI: Annenberg Institute for School Reform at Brown University, 2008). http://annenberginstitute.org/sites/default/files/product/206/files/LeadingIndicators.pdf

Meet the Expert(s)

Jon Young

Senior Scholar, Faculty of Education, University of Manitoba

Jon Young is a Professor in Educational Administration at the Faculty of Education, University of Manitoba.

Read More

Brian O’Leary

Superintendent, Seven Oaks School Division

Brian O’Leary is Superintendent of Seven Oaks School Division in Manitoba.

Read More