Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research

Diego Campos · Tim Fütterer · Thomas Gfrörer · Rosa Ellen Lavelle-Hill · Kou Murayama · Lars König +3 more

11 citations0/0 files stored

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Meta-analysis and systematic reviews Information Retrieval and Search Behavior Computational and Text Analysis Methods

Abstract

Systematic reviews and meta-analyses are crucial for advancing research; yet, they are time-consuming and resource-demanding. Although machine learning and natural language processing algorithms may reduce this time and these resources, their performance has not been tested in the field of education and educational psychology, and there is a lack of clear information on when researchers should stop the reviewing process. In this study, we conducted a retrospective screening simulation using 27 systematic reviews in education and educational psychology. We evaluated the recall, work saved over sampling, and the estimated time savings of several active learning screening algorithms and heuristic stopping criteria. The results showed on average a 50% (SD = 20%) reduction in screening workload when using active learning algorithms for abstract screening and an estimated time savings of 1.64 days (SD = 1.78). The learning algorithm Random Forests with Sentence Bidirectional Encoder Representations from Transformers outperformed other algorithms—a finding that emphasizes the importance of incorporating semantic and contextual information during feature extraction and modeling in the screening process. Furthermore, we found that 95% of all relevant abstracts within a given dataset can be retrieved using heuristic stopping rules. Specifically, an approach that stops the screening process after consecutively classifying 7% of irrelevant papers yielded the most significant gains in terms of works savings over sampling (M = 41%, SD = 26%). However, the performance of the heuristic stopping criteria depended on the active learning algorithm used, the length and the proportion of relevant papers in a database. Overall, our study provides empirical evidence on the performance of machine learning screening algorithms for abstract screening in systematic reviews in education and educational psychology.