Reviewgold2025

Accelerating the pace and accuracy of systematic reviews using AI: a validation study

Jiada James Zhan · Kara Suvada · Muwu Xu · Wenya Tian · Kelly Copeland Cara · Taylor C. Wallace +1 more

12 citations0/0 files storedSystematic Reviews

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Meta-analysis and systematic reviews Information Retrieval and Search Behavior Artificial Intelligence in Healthcare and Education

Abstract

BACKGROUND: Artificial intelligence (AI) can greatly enhance efficiency in systematic literature reviews and meta-analyses, but its accuracy in screening titles/abstracts and full-text articles is uncertain. OBJECTIVES: This study evaluated the performance metrics (sensitivity, specificity) of a GPT-4 AI program, Review Copilot, against human decisions (gold standard) in screening titles/abstracts and full-text articles from four published systematic reviews/meta-analyses. RESEARCH DESIGN: Participant data from four already-published systematic literature reviews were used for this validation study. This was a study comparing Review Copilot to human decision-making (gold standard) in screening titles/abstracts and full-text articles for systematic reviews/meta-analyses. The four studies that were used in this study included observational studies and randomized control trials. Review Copilot operates on the OpenAI, GPT-4 server. We examined the performance metrics of Review Copilot to include and exclude titles/abstracts and full-text articles as compared to human decisions in four systematic reviews/meta-analyses. Sensitivity, specificity, and balanced accuracy of title/abstract and full-text screening were compared between Review Copilot and human decisions. RESULTS: Review Copilot's sensitivity and specificity for title/abstract screening were 99.2% and 83.6%, respectively, and 97.6% and 47.4% for full-text screening. The average agreement between two runs was 95.4%, with a kappa statistic of 0.83. Review Copilot screened in one-quarter of the time compared to humans. CONCLUSIONS: AI use in systematic reviews and meta-analyses is inevitable. Health researchers must understand these technologies' strengths and limitations to ethically leverage them for research efficiency and evidence-based decision-making in health.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 12 citations exist upstream.

References

2016Rayyan—a web and mobile app for systematic reviews 2012Interrater reliability: the kappa statistic 2007The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement 2021An open source machine learning framework for efficient and transparent systematic reviews 2019Toward systematic review automation: a practical guide to using machine learning tools in research synthesis 2012Deploying an interactive machine learning system in an evidence-based practice center 2023Artificial intelligence in systematic reviews: promising when appropriately used 2022Using artificial intelligence methods for systematic review in health sciences: A systematic review

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.