The Use of Bayesian Networks to Assess the Quality of Evidence from Research Synthesis: 2. Inter-Rater Reliability and Comparison with Standard GRADE Assessment

Alexis Llewellyn · Craig J. Whittington · Gavin Bruce Stewart · Julian P. T. Higgins · Nick Meader

16 citations0/0 files storedPLoS ONE

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Meta-analysis and systematic reviews Clinical practice guidelines implementation Delphi Technique in Research

Abstract

BACKGROUND: The grades of recommendation, assessment, development and evaluation (GRADE) approach is widely implemented in systematic reviews, health technology assessment and guideline development organisations throughout the world. We have previously reported on the development of the Semi-Automated Quality Assessment Tool (SAQAT), which enables a semi-automated validity assessment based on GRADE criteria. The main advantage to our approach is the potential to improve inter-rater agreement of GRADE assessments particularly when used by less experienced researchers, because such judgements can be complex and challenging to apply without training. This is the first study examining the inter-rater agreement of the SAQAT. METHODS: We conducted two studies to compare: a) the inter-rater agreement of two researchers using the SAQAT independently on 28 meta-analyses and b) the inter-rater agreement between a researcher using the SAQAT (who had no experience of using GRADE) and an experienced member of the GRADE working group conducting a standard GRADE assessment on 15 meta-analyses. RESULTS: There was substantial agreement between independent researchers using the Quality Assessment Tool for all domains (for example, overall GRADE rating: weighted kappa 0.79; 95% CI 0.65 to 0.93). Comparison between the SAQAT and a standard GRADE assessment suggested that inconsistency was parameterised too conservatively by the SAQAT. Therefore the tool was amended. Following amendment we found fair-to-moderate agreement between the standard GRADE assessment and the SAQAT (for example, overall GRADE rating: weighted kappa 0.35; 95% CI 0.09 to 0.87). CONCLUSIONS: Despite a need for further research, the SAQAT may aid consistent application of GRADE, particularly by less experienced researchers.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 16 citations exist upstream.

References

1968Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.2011GRADE guidelines 6. Rating the quality of evidence—imprecision 2012Collaborative care for depression and anxiety problems 2015Cognitive behavioural therapy for anxiety disorders in children and adolescents 2014A checklist designed to aid consistency and reproducibility of GRADE assessments: development and pilot validation 2010Intensive case management for severe mental illness 2013The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses 2020Cognitive behavioural therapy for anxiety disorders in children and adolescents

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.