Effects of School Disruptions in the Years 2020 and 2021 on the Learning of Quebec Children

COVID-19 and the measures put in place to limit its spread have affected the quantity and quality of teaching and services offered by schools. These measures include the complete closure of Canadian schools as of mid-March 2020, emergency distance learning and support, the creation of classroom bubbles, and the temporary closure of classes to control outbreaks. According to the Conseil supérieur de l’éducation du Québec (2021), school disruptions could lead to an impoverishment of the knowledge and skills acquired by children, particularly among the most vulnerable. From an economic perspective, learning delays among elementary and secondary school children in Canada could result in losses amounting to nearly $2,500 billion in GDP over 80 years (Hanushek & Woessmann, 2020).

A growing body of data regarding the effects of academic disruptions on learning became available during the years 2021-2023. A pre-recorded systematic review on the topic indicates that outcomes vary according to the social and school context in which the disruptions occurred, and how they were managed. According to this meta-analysis of 42 studies in 15 countries, a substantial overall learning deficit (Cohen’s d = -0.14, 95% confidence interval -0.17 to -0.10) emerged early in the pandemic and persisted over time. Learning deficits are particularly significant among children from underprivileged socioeconomic backgrounds. These deficits are also higher in mathematics than in reading, and in middle-income countries than in high-income countries.

Although the effects of COVID-19 on learning are of concern to the Canadian government, there appears to be little national data available. In Quebec, the Observatory for Children’s Health and Education (OPES), in collaboration with the Ministère de l’Éducation, collected data to measure the reading skills of Quebec children completing Grade 4 of elementary school in 2021. Between March 2020 and May 2022, these children experienced 15 months of school disruptions caused by the health measures that were implemented to counter the spread of COVID-19.

The goal was to compare the learning level of children in 2021 to that of Grade 4 children in 2019 (who had not been exposed to the pandemic). We tested the possibility that learning gaps between cohorts can vary by child gender, reading-risk status and socioeconomic background, and by the number of days classes were closed.

Methodology

In April 2021, all French-language and public school service centres (n=60) in the province were invited to participate in a study to understand the impact of school closures on reading performance in Grade 4. School participation was voluntary.

The analyses included 10,317 students in 2021 and 13,669 students in 2019 from the same schools, for a total of 23,986 students.

Learning was measured using the compulsory French test (language of instruction). This was the same test that was administered prior to the COVID-19 pandemic, i.e., the June 2019 test.

The Grade 4 reading assessment featured two tasks: the literary text task and the ordinary text task. Only one of the two tests was administered in this study: the ordinary text test. This is a 2.5-hour test in which students read an ordinary 1,000-word text and respond to 12 questions with short answers. The tests were corrected centrally, with each copy being corrected twice by employees of the Direction de la sanction des études (DSE) of the Ministère de l’Éducation du Québec.

The child’s gender and the school’s Socio-Economic Environment Index11 (IMSE) were used as independent variables.The number of class closure days during the 2020-2021 school year (fewer than 15 days closed or more than 15 days) was used in the analyses.

Calculation of learning gaps

To estimate learning gaps between 2019 and 2021, we used a linear model that included school fixed effects. The model is as follows:

Yiet=ɑ +β Cohortt2021 + γBoyi + θe + εiet

where Yiet is the score of student i in school e in year t. The term Cohortt2021 is an indicator variable equal to one in 2021 and zero in 2019. School fixed effects are collected by θe. The student’s gender is controlled via Boyi and εiet is the error term. Standard deviations were calculated to account for the strongest correlation in outcomes among children in the same school (cluster analysis). Thus, the coefficient β captures the effects of school disruptions under certain assumptions. The possibility of interactions between IMSE, performance and gender was investigated.

Our results

There was an average 8.4 percentage point decline in reading between June 2019 and June 2021. The size of this gap varied according to the children’s test performance: it was high for the bottom decile of performance (20 pp); medium for the middle deciles (10 pp at the 4th decile); and zero for the top 2 deciles of performance. Boys had slightly more pronounced losses than girls.

School staff and promotion of equal opportunities

We compared reading skills as measured through a ministry test (n=10,880, 9-10-year-old students) with children in the same schools (n=13,669) that had administered the same test in 2019. Results indicated an 8.4% difference in student reading scores between 2019 (pre-pandemic) and 2021 (after 15 months of exposure to the pandemic). Thus, while the average score was 77.7% in 2019, it was 69.3% in 2021. The size of this standard deviation varied depending on how well the children performed on the test. It was high for the bottom decile of performance (20 pp2); medium for the middle deciles (10 pp at the 4th decile); and zero for the top two performance deciles. These results suggest that children who were already strong in reading, i.e., those with scores in the top 20%, had not experienced learning losses 15 months after the onset of the pandemic. On the other hand, children who were weak in reading, i.e., those with scores in the bottom 20%, did experience significant learning losses (15-20 pp). The performance gaps were larger (by 1.3 pp) for boys.

The results suggest that students with academic difficulties are particularly in need of the specialized and structured environment offered by the school to support them in their learning. They highlight the crucial role of teachers and school professionals as advocates for equal opportunity. Moreover, studies of classroom management practices in a context of uncertainty (such as during a pandemic) suggest that less attention is paid to vulnerable groups during times when everyone is experiencing a low sense of safety. As the years 2020-2021 have placed unusual demands on school staff, it is possible to anticipate that the return to greater normality in 2022 will provide the support needed for everyone’s success.

The results of Betthauser et al. (2023) indicate greater losses for children from disadvantaged backgrounds. Although our results show that children from disadvantaged schools had lower scores than others in both 2019 and 2021, we did not detect that learning losses were greater for children from disadvantaged schools, but rather for children who had lower reading levels. Note that since the disadvantage index is correlated with performance, it is plausible that the differential results by performance obscured the difference in the disadvantage index.

It is also possible that we did not find an impact from deprivation due to the fact that the index to which we had access was measured at the school level and not at the child/family level. The composition of schools is in fact very heterogeneous in terms of family disadvantage. Thus, even in disadvantaged neighbourhoods, the variable may not capture individual differences in the level of deprivation, whereas the performance variable does. This explanation also applies to the income threshold, which is a school-level variable.

Special considerations and recommended follow-ups

This study has important methodological strengths, including the standardized nature of the test, the large sample size (n=23,986), the diversity and representativeness of the socioeconomic status of participating schools, and the comparison of children in the same schools in 2019 and 2021. The within-school comparison type controls for a large number of confounding variables, including school management, school staff and the children who attend the schools.

The results captured the full direct and indirect effects of pandemic disruption on the acquisition of reading skills, not just the effects of academic disruption. It should be noted that the context in which the tests were taken may have influenced the results: the test in 2021 was not compulsory and was not entered on report cards. Although teachers were instructed to prepare their students for the test by following standard practices, children’s motivation and stress levels may have been different in 2021 than in 2019. The lower level of stress may have had a positive effect on the performance of some children. For others, the fact that grades were not reported on the report card may have decreased motivation.

The results underscore the importance of conducting a long-term review of the learning trajectories of Canadian students following the 2019-2021 academic disruptions. Researchers have shown that a 0.2 standard deviation increase in academic achievement is associated with higher earnings (2.6% over a life course) and better labour market participation (Chetty, Friedman, & Rockoff, 2014a, 2014b). It should be noted that this study shows greater effects for the most vulnerable students, suggesting increased social inequalities in academic performance. Monitoring of standardized test scores over the next several years is necessary to quantify changes in gaps, if any, and to identify strategies to narrow them.

We would like to thank our collaborators: Karine Trudeau, PhD, and William Sauvé.

References

Blainey, K. (2020). The impact of lockdown on children’s education: a nationwide analysis. chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://www.risingstars-uk.com/media/Rising-Stars/Assessment/Whitepapers/RS_Assessment_white_paper_1.pdf

Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014a). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104(9), 2593-2632.

Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014b). Measuring the impacts of teachers II: Teacher value-added and student outcomes in adulthood. American Economic Review, 104(9), 2633-2679.

Engzell, P., Frey, A., & Verhagen, M. D. (2021). Learning loss due to school closures during the COVID-19 pandemic. Proceedings of the National Academy of Sciences, 118(17), e2022376118.

Goldhaber, D., Imberman, S. A., et al. (2022). To what extent does in-person schooling contribute to the spread of Covid-19? Evidence from Michigan and Washington. Journal of Policy Analysis and Management, 41(1), 318-349.

Haelermans, C., Jacobs, M., et al. (2021). A full year COVID-19 crisis with interrupted learning and two school closures: The effects on learning growth and inequality in primary education. doi.org/10.31219/osf.io/78fje

Hanushek, E. A., & Woessmann, L. (2020). The economic impacts of learning losses. OECD Work document on education, 225, OECD Edition, Paris.

Maldonado, J. E., & De Witte, K. (2022). The effect of school closures on standardized student test outcomes. British Educational Research Journal, 48(1), 49-94.

Milanovic, K. (2022). The residual impact of educational disruption on primary school attainment by spring 2022. 22 RS Assessment, School Dash. https://risingstars-uk.com/media/Rising-Stars/Assessment/Whitepapers/Spring22_RS_Assessment_white_paper.pdf

Quebec, É. (2021). GDUNO. https://prod.education.gouv.qc.ca/gdunojrecherche/rechercheOrganisme.do;jsessionid=3sZe_Q8EMu68DHnFStbDpDDcFQszamaRzuTZLx4hSeKmIfn7AGRh!1100596104!-1697553619?methode=recherche

Quebec, M. d. l. e. d. (2021). Indice de milieu socio-économique (IMSE). www.education.gouv.qc.ca/enseignants/aide-et-soutien/milieux-defavorises/agir-autrement/indice-de-milieu-socio-economique-imse

Photo: iStock
First published in Education Canada, April 2023

Meet the Expert(s)

Dr. Sylvana Côté

Professor and Researcher, University of Montreal and CHU Sainte-Justine

Sylvana Côté, PhD, is a psychologist by training, a full Professor in the School of Public Health at the Université de Montréal and a researcher at CHU ...

Read More

Dr. Simon Larose

Professor, Université Laval

Simon Larose, PhD, is a full Professor in the Faculty of Education at Université Laval and is responsible for the Education axis of the Observatory for Chi...

Read More

Dr. Catherine Haeck

Professor, Université du Québec à Montréal

Catherine Haeck, PhD, is a full Professor in the Department of Economics at the Université du Québec à Montréal. She specializes in the economics of edu...

Read More