Students ratings their open classroom discussion

Open classroom discussion (OPD) is a recognized school practice, that promotes civic knowledge on students. However, the study of its effectiveness includes various methodological challenges. OPD items are reference-shift items, and if their rater-response nature is ignored, researchers may specify a compositional model leading to underestimation. Moreover, OPD scores of schools are subject to students inter rater variability. Common advice in this regard is to exclude groups with low inter-rater agreement. Nevertheless, this recommendation can result into a considerable loss of sample. In this paper, we argue that a within-between model specification is needed to address the first problem. For the second problem, it is proposed to use a dispersion effect model. This later model studies OPD relations to civic knowledge, at conditional levels of students’ lack of agreement on OPD ratings. Caveats on the use of students as raters of school practices are discussed.