Article2020

Comparing machine and human reviewers to evaluate the risk of bias in randomized controlled trials

Susan L. Armijo-Olivo · Rodger Craig · Sandy Campbell

30 citations0/0 files storedResearch Synthesis Methods

No readable copy stored yet — this record is metadata only.

Meta-analysis and systematic reviews scientometrics and bibliometrics research Delphi Technique in Research

Abstract

BACKGROUND: Evidence from new health technologies is growing, along with demands for evidence to inform policy decisions, creating challenges in completing health technology assessments (HTAs)/systematic reviews (SRs) in a timely manner. Software can decrease the time and burden by automating the process, but evidence validating such software is limited. We tested the accuracy of RobotReviewer, a semi-autonomous risk of bias (RoB) assessment tool, and its agreement with human reviewers. METHODS: Two reviewers independently conducted RoB assessments on a sample of randomized controlled trials (RCTs), and their consensus ratings were compared with those generated by RobotReviewer. Agreement with the human reviewers was assessed using percent agreement and weighted kappa (κ). The accuracy of RobotReviewer was also assessed by calculating the sensitivity, specificity, and area under the curve in comparison to the consensus agreement of the human reviewers. RESULTS: The study included 372 RCTs. Inter-rater reliability ranged from κ = -0.06 (no agreement) for blinding of participants and personnel to κ = 0.62 (good agreement) for random sequence generation (excluding overall RoB). RobotReviewer was found to use a high percentage of "irrelevant supporting quotations" to complement RoB assessments for blinding of participants and personnel (72.6%), blinding of outcome assessment (70.4%), and allocation concealment (54.3%). CONCLUSION: RobotReviewer can help with risk of bias assessment of RCTs but cannot replace human evaluations. Thus, reviewers should check and validate RoB assessments from RobotReviewer by consulting the original article when not relevant supporting quotations are provided by RobotReviewer. This consultation is in line with the recommendation provided by the developers.

Citations in this library

Stored only when both papers are in the corpus — 30 citations exist upstream.

References

1977The Measurement of Observer Agreement for Categorical Data 2011The Cochrane Collaboration's tool for assessing risk of bias in randomised trials 1968Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.2005Understanding interobserver agreement: the kappa statistic.1995Empirical Evidence of Bias 1995Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials 1998Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?2008Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study

Cited by

2024Zero‐ and few‐shot prompting of generative large language models provides weak assessment of risk of bias in clinical trials 2022Accuracy and Efficiency of Machine Learning–Assisted Risk-of-Bias Assessments in “Real-World” Systematic Reviews 2025Digital Tools to Support the Systematic Review Process: An Introduction 2025Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2

Imported from OpenAlex (CC0) — nothing here is generated.