Articlegold2024

Evaluating the effectiveness of large language models in abstract screening: a comparative analysis

Michael Li · Jian‐Ping Sun · Xianming M. Tan

126 citations0/0 files storedSystematic Reviews

No readable copy stored yet — this record is metadata only.

Meta-analysis and systematic reviews Information Retrieval and Search Behavior Artificial Intelligence in Healthcare and Education

Abstract

OBJECTIVE: This study aimed to evaluate the performance of large language models (LLMs) in the task of abstract screening in systematic review and meta-analysis studies, exploring their effectiveness, efficiency, and potential integration into existing human expert-based workflows. METHODS: We developed automation scripts in Python to interact with the APIs of several LLM tools, including ChatGPT v4.0, ChatGPT v3.5, Google PaLM 2, and Meta Llama 2, and latest tools including ChatGPT v4.0 turbo, ChatGPT v3.5 turbo, Google Gemini 1.0 pro, Meta Llama 3, and Claude 3. This study focused on three databases of abstracts and used them as benchmarks to evaluate the performance of these LLM tools in terms of sensitivity, specificity, and overall accuracy. The results of the LLM tools were compared to human-curated inclusion decisions, gold standard for systematic review and meta-analysis studies. RESULTS: Different LLM tools had varying abilities in abstract screening. Chat GPT v4.0 demonstrated remarkable performance, with balanced sensitivity and specificity, and overall accuracy consistently reaching or exceeding 90%, indicating a high potential for LLMs in abstract screening tasks. The study found that LLMs could provide reliable results with minimal human effort and thus serve as a cost-effective and efficient alternative to traditional abstract screening methods. CONCLUSION: While LLM tools are not yet ready to completely replace human experts in abstract screening, they show great promise in revolutionizing the process. They can serve as autonomous AI reviewers, contribute to collaborative workflows with human experts, and integrate with hybrid approaches to develop custom tools for increased efficiency. As technology continues to advance, LLMs are poised to play an increasingly important role in abstract screening, reshaping the workflow of systematic review and meta-analysis studies.

Citations in this library

Stored only when both papers are in the corpus — 126 citations exist upstream.

References

2009Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement 2009Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement 2008Cochrane Handbook for Systematic Reviews of Interventions 2019Cochrane Handbook for Systematic Reviews of Interventions 2017AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both 2009Latent Class and Latent Transition Analysis: With Applications in the Social, Behavioral, and Health Sciences 1987Latent Class Analysis 1974Exploratory Latent Structure Analysis Using Both Identifiable and Unidentifiable Models

Cited by

2024Leveraging artificial intelligence to enhance systematic reviews in health research: advanced tools and challenges 2025Streamlining systematic reviews with large language models using prompt engineering and retrieval augmented generation 2025Large language models for automated scholarly paper review: A survey 2025Chatbots for conducting systematic reviews in pediatric dentistry 2025Leveraging open-source large language models (LLMs) in scoping reviews: a case study on disability and AI applications

Imported from OpenAlex (CC0) — nothing here is generated.