Rethinking international educational comparisons of non-cognitive measures: from league tables of means to slopes
Large-scale international assessments such as the Programme for International Student Assessment (PISA) provide valuable insights into both cognitive (e.g., mathematics, science, reading) and non-cognitive (e.g., motivation, well-being) student outcomes. While league tables of country means are commonly used, they may oversimplify the complexities of non-cognitive constructs. Using data from PISA 2018 (612,004 students across 80 countries), we examined how non-cognitive measures vary across the student (L1), school (L2), and country (L3) levels.
We conducted simple correlations and multilevel modeling to explore associations among non-cognitive variables, cognitive performance, socioeconomic status (SES), and gender. Although countries differed markedly in cognitive achievement, between-country variance in non-cognitive measures was small. Notably, the relationships between non-cognitive and cognitive variables reversed across levels—for instance, students who enjoyed reading performed better in reading; yet, countries with higher average reading enjoyment had lower national reading scores.
These reversals suggest potential ecological fallacies, response style differences, or omitted variables at the country level. We advocate moving beyond league tables of means to slope comparisons—examining how associations (slopes) with SES, gender, achievement, or resilience vary across countries. Our findings highlight within-school reference effects and offer guidelines for analyzing non-cognitive data in international and smaller-scale research.
Speaker:
Kit-Tai Hau — Research Professor, Emeritus Professor, Department of Educational Psychology, School of Education, The Chinese University of Hong Kong, Hong Kong SAR