Bridging Time and Context: Longitudinal Comparability of Background Scales in International Assessments (COMPASS)

International Large-Scale Assessments (ILSAs) such as TIMSS, PIRLS and ICCS occupy a central position in educational research and policy. Beyond measuring student achievement in core domains (mathematics, science, reading and civic knowledge), they generate extensive background scales that capture constructs ranging from students’ motivation, attitudes and self-efficacy to teachers’ job satisfaction, instructional practices and professional development, as well as school climate, leadership and resources. These measures are indispensable for investigating the mechanisms underlying achievement differences, identifying equity gaps and informing policy interventions.
A persistent methodological challenge concerns the cross-country and cross-cycle comparability of these background scales. Recent large-scale assessment research has employed alignment methods to establish comparability when full measurement invariance cannot be guaranteed. However, most applications have relied on confirmatory factor analysis for continuous indicators—an approach ill-suited to the predominantly ordinal (four-category) response formats used in ILSA background questionnaires. Fitting continuous models to ordinal data can underestimate parameters, distort group comparisons and produce inverted, artifactual or undetected relationships.
The present study therefore proposes the use of confirmatory factor analysis for ordinal indicators (CFAord). Measurement invariance will first be diagnosed via a model-based approach. Where evidence of non-invariance emerges, partially invariant solutions will be sought through alignment optimisation conducted within the CFAord framework, thereby reducing inferential error in cross-country and cross-cycle comparisons. The empirical focus is the background questionnaire scales of ICCS, examined across countries and assessment cycles. The principal output will be an overview of the extent to which these scale scores can validly support comparisons across nations and over time.

International surveys such as TIMSS, PIRLS and ICCS measure student achievement and also collect background information on motivation, teaching practices and school conditions. This information helps explain differences in results and guide improvements.
The difficulty is whether these background measures can be fairly compared across countries and over time. Earlier methods often used unsuitable statistics for the ordered answer choices in the questionnaires, risking inaccurate results.
This study will use a more appropriate statistical approach to check and, where needed, carefully adjust the measures so scores can be compared reliably. Focusing on the ICCS questionnaires, it will show how many of these measures can safely support cross-country and over-time comparisons.