Evaluating the Accuracy of scite, a Smart Citation Index

Caitlin Bakker · Nicole R Theis-Mahon · Sarah Jane Brown

10 citations0/0 files storedHypothesis Research Journal for Health Information Professionals

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

scientometrics and bibliometrics research Academic Writing and Publishing Meta-analysis and systematic reviews

Abstract

Objectives: Citations do not always equate endorsement, therefore it is important to understand the context of a citation. Researchers may heavily rely on a paper they cite, they may refute it entirely, or they may mention it only in passing, so an accurate classification of a citation is valuable for researchers and users. While AI solutions have emerged to provide a more nuanced meaning, the accuracy of these tools has yet to be determined. This project seeks to assess the accuracy of scite in assessing the meaning of citations in a sample of publications. Methods: Using a previously established sample of systematic reviews that cited retracted publications, we conducted known item searching in scite, a tool that uses machine learning to categorize the meaning of citations. scite's interpretation of the citation's meaning was recorded, as was our assessment of the citation’s meaning. Citations were classified as mentioning, supporting or contrasting. Recall, precision, and f-measure were calculated to describe the accuracy of scite's assessment in comparison to human assessment. Results: From the original sample of 324 citations, 98 citations were classified in scite. Of these, scite found that 2 were supporting and 96 were mentioning, while we determined that 42 were supporting, 39 were mentioning, and 17 were contrasting. Supporting citations had high precision and low recall, while mentioning citations had high recall and low precision. F-measures ranged between 0.0 and 0.58, representing low classification accuracy. Conclusions: In our sample, the overall accuracy of scite's assessments was low. scite was less able to classify supporting and contrasting citations, and instead labeled them as mentioning. Although there is potential and enthusiasm for AI to make engagement with literature easier and more immediate, the results generated from AI differed significantly from the human interpretation.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 10 citations exist upstream.

References

2015Using text mining for study identification in systematic reviews: a systematic review of current approaches 2019Toward systematic review automation: a practical guide to using machine learning tools in research synthesis 2015RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials 2004Citation Analysis and Discourse Analysis Revisited 2022Using artificial intelligence methods for systematic review in health sciences: A systematic review 2017Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach 2021scite: A smart citation index that displays the context of citations and classifies their intent using deep learning 2021An open source machine learning framework for efficient and transparent systematic reviews

Cited by

2024Enhancing the quality and efficiency of regulatory science literature reviews through innovation and collaboration with library and information science experts 2023The relationship between methodological quality and the use of retracted publications in evidence syntheses

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.