Reviewgold2025

Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers

Tue Helms Andersen · T. M. Marcussen · Anne‐Ditte Termannsen · Trine Witzner Hessel Lawaetz

35 citations0/0 files storedCochrane Evidence Synthesis and Methods

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Meta-analysis and systematic reviews Artificial Intelligence in Healthcare and Education Computational and Text Analysis Methods

Abstract

ABSTRACT Background Systematic reviews are essential but time‐consuming and expensive. Large language models (LLMs) and artificial intelligence (AI) tools could potentially automate data extraction, but no comprehensive workflow has been tested for different review types. Objective To evaluate Elicit's and ChatGPT's abilities to extract data from journal articles as a replacement for one of two human data extractors in systematic reviews. Methods Human‐extracted data from three systematic reviews (30 articles in total) was compared to data extracted by Elicit and ChatGPT. The AI tools extracted population characteristics, study design, and review‐specific variables. Performance metrics were calculated against human double‐extracted data as the gold standard, followed by a detailed error analysis. Results Precision, recall and F1‐score were all 92% for Elicit and 91%, 89% and 90% for ChatGPT. Recall was highest for study design (Elicit: 100%; ChatGPT: 90%) and population characteristics (Elicit: 100%; ChatGPT: 97%), while review‐specific variables achieved 77% in Elicit and 80% in ChatGPT. Elicit had four instances of confabulation while ChatGPT had three. There was no significant difference between the two AI tools' performance (recall difference: 3.3% points, 95% CI: –5.2%–11.9%, p = 0.445). Conclusion AI tools demonstrated high and similar performance in data extraction compared to human reviewers, particularly for standardized variables. Error analysis revealed confabulations in 4% of data points. We propose adopting AI‐assisted extraction to replace the second human extractor, with the second human instead focusing on reconciling discrepancies between AI and the primary human extractor.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 35 citations exist upstream.

References

2019Cochrane Handbook for Systematic Reviews of Interventions 2016The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses 2017Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry 2006Single data extraction generated more errors than double data extraction in systematic reviews 2015Automating data extraction in systematic reviews: a systematic review 2024Can large language models replace humans in systematic reviews? Evaluating GPT ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages 2017Frequency of data extraction errors and methods to increase data extraction quality: a methodological review 2024Data extraction for evidence synthesis using a large language model: A proof‐of‐concept study

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.