Evaluating Best Practices in Large-Scale Assessment

As Canadian education systems contemplate how they will reliably and validly measure the learning that goes on in the increasingly complex classrooms within their jurisdiction, it is important that insights, ideas, experiences and research from all regions in the country be examined and shared.

All Canadian provincial jurisdictions develop and administer large-scale assessment programs in education. A variety of factors must be considered at the development stage of the program as a jurisdiction decides what information needs to be collected, how it should be collected and why. In Canada, despite many similarities between provincial large-scale assessment programs, different approaches have been taken recently as jurisdictions make changes, or consider making changes, to these programs. These changes reflect ongoing discussion about the purpose of large-scale assessment.

Generally, in Canada, students are tested in mathematics and language arts in early and late elementary grades as well as in high school. Provincial ministry or department websites cite various purposes for these tests. Data is used to identify areas of need at all levels: provincial, board, school, classroom, and individual students. At the high school level, marks from large-scale examinations contribute to a student’s final course mark in some provinces, while other provinces administer a literacy and/or numeracy test as a graduation requirement. Data is also used to report to the public on the effectiveness of the system, with aggregated results made available to school boards or regions and often posted publicly. Individual student results have been made available to schools, students and/or parents.

Recent changes

Over the last few years, many provinces have implemented or are publicly considering implementing significant changes in their large-scale assessment programs. Although these changes are different in nature, they all reflect a desire on the part of jurisdictions to clarify the purpose and improve the value of large-scale assessment as part of education reform initiatives.

Examples of changes at the elementary level include Alberta’s Student Learning Assessment in Grade 3 which, since 2015, can be administered at any time during the school year and is not used for accountability purposes. The Alberta Achievement Tests in Grades 6 and 9, however, continue to be written at the end of the year and provide data to “report to Albertans how well students have achieved provincial standards at given points in their schooling.”¹

As of 2018, the Foundation Skills Assessment in British Columbia includes collaboration and self-reflection activities in addition to traditional written questions.² At the high school level, B.C.’s end-of-course examination program is being phased out and replaced with a literacy and a numeracy assessment that are requirements for graduation.³

In Nova Scotia, the Grade 12 end-of-course examinations in literacy and in mathematics have been moved to the Grade 10 level.⁴

In March 2018, the Ontario Ministry of Education released a report of an independent review of the province’s assessment, evaluation, and reporting practices. The review included a close look at the Education Quality and Accountability Office (EQAO), the agency responsible for developing the province’s large-scale assessments. In a letter published at the beginning of the report, the assessment review committee summarizes: “We propose a system of assessment that prioritizes classroom assessments to support each student’s learning and development, engage parents/guardians meaningfully in knowing about their child’s achievements and progress, and enable educators to develop and share their professional practices.”⁵ The report recommends increased focus on high-quality classroom diagnostic, formative and summative assessment to provide information about individual students, and suggests that schools and teachers should no longer use large-scale individual student data for diagnostic purposes: “ … student reports should clarify this is a snapshot of performance on a system assessment and is not intended for diagnostic or evaluative purposes.”⁶ It goes on to recommend the discontinuation of large-scale assessment in Grade 3 and the development of a new high school test which would no longer be a graduation requirement. It is recommended that large-scale assessment data from testing all students continue to be collected to identify the needs of groups of students who require further support as well as to report to the public about system performance. A shift in the role of large-scale assessment in Ontario is recommended:

“We propose large-scale provincial assessments that provide public information about the performance of Ontario’s education system overall and to inform future improvements to benefit all students to succeed, including identifying inequities in outcomes for groups of students whose diverse experiences and needs require further attention.”⁷

These examples of change in various provinces, despite their differences, all indicate a desire to clarify the role of large-scale assessment and to create new developmentally and pedagogically appropriate ways of assessing students in a large-scale format.

Clarifying the purpose of large-scale assessment

As the value of large-scale assessment is widely discussed and as provincial governments implement, or consider implementing, educational reforms, two opposing sides seem to have emerged. On the one hand are those who argue that provincial assessments serve important purposes, including holding school systems to account and providing information about where supports should be placed to improve student achievement. Education department or ministry business plans cite provincial assessment data as a critical measure of the success of important initiatives. School improvement can be measured using provincial data. The public can be informed about how well the education system is doing by reporting aggregated provincial data. Finally, provincial tests provide data on how individual students are doing in relation to provincial standards.

On the other side are those who argue that teachers know their students best and that large-scale assessment data is only a snapshot in time that may or may not reflect individual student performance. Furthermore, large-scale assessments take time and attention away from the real business of classroom teaching, and teachers spend too much time preparing for and administering tests which do not reflect classroom practice. Finally, teachers feel pressure to improve scores, and they may not see any link between improving scores and improving student learning.

Two surveys in public attitudes towards education indicate that these two sides have become unnecessarily polarized. In Public Education in Canada: Fact, Trends and Attitudes, a 2007 nationwide survey of attitudes towards education by the Canadian Education Association (CEA), 77 percent of Canadians agreed that high school students should be assessed using province-wide tests.⁸ In Public Attitudes Towards Education in Ontario, a recent OISE survey of Ontarians’ attitudes towards education, 66 percent agreed that each secondary student should be assessed using a province-wide test.⁹ This survey shows somewhat less support for testing students at the elementary level than at the high school level, although a majority still support testing at this level, with 49 percent agreeing that “every student should be tested” and 19 percent that “a sample of students should be tested.”

While many Canadians see value in provincial large-scale testing, they also value teachers’ work. Seventy percent of Canadians agreed that “teachers are doing a good job,” and 60 percent agreed that high school grades should mainly reflect teachers’ assessments. In the OISE survey, Just over half of Ontarians reported being somewhat satisfied or satisfied with the job elementary teachers are doing, and half were somewhat satisfied or satisfied with the job high school teachers are doing. Interestingly 20 percent responded that they are neither satisfied nor dissatisfied with the job teachers are doing. Fifty-five percent of Ontarians agreed that “high school students’ final grades should mainly reflect their teachers’ assessments, not the results of province-wide tests.” Once again, 20 percent neither agreed nor disagreed.

In general, Canadians see value in large-scale testing and at the same time they value teachers’ professional judgments in determining student achievement. Each has an important role to play and they are not necessarily mutually exclusive. The main role of large-scale assessment is to provide consistent province-wide data that can be tracked over time and, at the high school level, to provide a province-wide measure of each student’s achievement in key areas. The main role of teachers’ classroom assessment practices is to provide detailed achievement information that can be used to plan instructional strategies for individual students over the course of an academic year or term. As Lorna Earle writes: “Large-scale assessments and classroom assessments done by teachers both make important contributions to continuous improvement in education. It is important that we continue to support both approaches and ensure that both forms of assessment provide high-quality information that the public can have confidence in and value as fair representations of students’ learning.”¹⁰ Similarly, in the OISE report the authors summarize: “… although most want EQAO testing retained as a way of monitoring outcomes, there is little support for ‘high stakes’ province-wide testing that would determine the advancement of individual students. In other words, both province-wide testing and teacher assessments are valued for different reasons.”

Considerations going forward

Canada’s educational jurisdictions are attempting to clarify the purpose of large-scale assessment programs in different ways. As one province implements new large-scale assessments as a graduation requirement, another considers phasing out an older assessment with a similar requirement. As one province implements significant changes to its Grade 3 assessment program, another considers phasing out its 20-year-old Grade 3 assessment program entirely. It is likely that most provinces are having internal conversations about the purpose of their programs and discussing potential avenues for change.

As jurisdictions consider making changes, it is important not to lose sight of the fact that the value of large-scale assessment data increases over time. Data collected yearly over 20 years is rich in information, since trends can be identified, monitored and acted upon only after several years of data is available. For this reason, changes in a large-scale assessment program must be carefully planned and must take into consideration its value over the long term.

It is also important to note that the cost of developing a large-scale assessment is not tied to the number of students who take the tests. The resources required for the development of high-quality assessment tools are the same for all jurisdictions, regardless of their size. Requirements include subject-matter and psychometric expertise as well as robust item banking, field testing and standard-setting procedures. It takes two years or longer to develop a high-quality test, whether it is administered to 10,000 students or 130,000 students. Furthermore, this development process must be repeated regularly to create new questions for each administration of the test.

As provinces consider changes, best practices in large-scale assessment should be identified so that all Canadian students can benefit from innovative assessment practices tailored to their individual needs, and so that residents of all jurisdictions can rely on high-quality data about their education systems. As Canadian education systems contemplate how they will reliably and validly measure the learning that goes on in the increasingly complex classrooms within their jurisdiction, it is important that insights, ideas, experiences and research from all regions of the country be examined and shared.

Photo: iStock

First published in Education Canada, March 2019

Notes

1 https://education.alberta.ca

2 https://curriculum.gov.bc.ca/assessment-reporting/new-foundation-skills-assessment

3 https://curriculum.gov.bc.ca/provincial-assessment/graduation/literacy

4 https://plans.ednet.ns.ca

5 C. Campbell, J. Cinton, M. Fullan, et al., Ontario, A Learning Province: Findings and recommendations from the independent review of assessment and reporting (Province of Ontario, March 2018), 2. https://www.oise.utoronto.ca/preview/lhae/UserFiles/File/OntarioLearningProvince2018.pdf

6 Ibid, 70.

7 Ibid, 3.

8 Public Education in Canada: Fact, Trends and Attitudes 2007, Canadian Education Association (2007), 8.
www.edcan.ca/articles/public-education-in-canada-facts-trends-and-attitudes-2007

9 Doug Hart and Arlo Kempf, Public Attitudes Towards Education in Ontario 2018: The 20^th OISE survey of educational issues (The Ontario Institute for Studies in Education/University of Toronto, 2018), 30. www.oise.utoronto.ca/oise/UserFiles/Media/Media_Relations/OISE-Public-Attitudes-Report-2018_final.pdf

10 Public Education in Canada, p. 8

Meet the Expert(s)

Dr. Vera Grayson Kocay

Literacy Evaluation Coordinator, Nova Scotia Department of Education and Early Childhood Development / Coordinatrice d'évaluation en littératie, Ministère de l’Éducation et du Développement de la petite enfance de la Nouvelle-Écosse

Dr. Vera Grayson is currently Literacy Evaluation Coordinator at the Nova Scotia Department of Education and Early Childhood Development, and previously worked for the Education Quality and Accountabi...

Read More