How to build more generalizable models for collaboration quality?

This talk presented a research paper at the 14th International Conference of Learning Analytics & Knowledge organized at Arizona State University, Arizona. The paper included results from a study exploring methodologies to build generalizable collaboration prediction models for authentic classroom settings.

You can read more about the presented work here: Paper.

Presentation slides: Slides

Abstract

Multimodal learning analytics (MMLA) research for building collab- oration quality estimation models has shown significant progress. However, the generalizability of such models is seldom addressed. In this paper, we address this gap by systematically evaluating the across-context generalizability of collaboration quality mod- els developed using a typical MMLA pipeline. This paper further presents a methodology to explore modelling pipelines with differ- ent configurations to improve the generalizability of the model. We collected 11 multimodal datasets (audio and log data) from face-to- face collaborative learning activities in six different classrooms with five different subject teachers. Our results showed that the models developed using the often-employed MMLA pipeline degraded in terms of Kappa from Fair (.20 < Kappa < .40) to Poor (Kappa < .20) when evaluated across contexts. This degradation in performance was significantly ameliorated with pipelines that emerged as high- performing from our exploration of 32 pipelines. Furthermore, our exploration of pipelines provided statistical evidence that often- overlooked contextual data features improve the generalizability of a collaboration quality model. With these findings, we make recommendations for the modelling pipeline which can potentially help other researchers in achieving better generalizability in their collaboration quality estimation models