Appraising the Quality of Medical Education Research Methods

David A. Cook · Darcy A. Reed

779 citations0/0 files storedAcademic Medicine

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Meta-analysis and systematic reviews Innovations in Medical Education Health Education and Validation

Abstract

PURPOSE: The Medical Education Research Study Quality Instrument (MERSQI) and the Newcastle-Ottawa Scale-Education (NOS-E) were developed to appraise methodological quality in medical education research. The study objective was to evaluate the interrater reliability, normative scores, and between-instrument correlation for these two instruments. METHOD: In 2014, the authors searched PubMed and Google for articles using the MERSQI or NOS-E. They obtained or extracted data for interrater reliability-using the intraclass correlation coefficient (ICC)-and normative scores. They calculated between-scale correlation using Spearman rho. RESULTS: Each instrument contains items concerning sampling, controlling for confounders, and integrity of outcomes. Interrater reliability for overall scores ranged from 0.68 to 0.95. Interrater reliability was "substantial" or better (ICC > 0.60) for nearly all domain-specific items on both instruments. Most instances of low interrater reliability were associated with restriction of range, and raw agreement was usually good. Across 26 studies evaluating published research, the median overall MERSQI score was 11.3 (range 8.9-15.1, of possible 18). Across six studies, the median overall NOS-E score was 3.22 (range 2.08-3.82, of possible 6). Overall MERSQI and NOS-E scores correlated reasonably well (rho 0.49-0.72). CONCLUSIONS: The MERSQI and NOS-E are useful, reliable, complementary tools for appraising methodological quality of medical education research. Interpretation and use of their scores should focus on item-specific codes rather than overall scores. Normative scores should be used for relative rather than absolute judgments because different research questions require different study designs.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 779 citations exist upstream.

References

1977The Measurement of Observer Agreement for Categorical Data 2010Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses 2000Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs 2003Evaluating non-randomised intervention studies 2008Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study 2007Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography 2007Association Between Funding and Quality of Published Medical Education Research 2001Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies

Cited by

2016The Effectiveness of Blended Learning in Health Professions: Systematic Review and Meta-Analysis 2017The Use of Social Media in Graduate Medical Education: A Systematic Review 2020The effectiveness of the combined problem-based learning (PBL) and case-based learning (CBL) teaching method in the clinical practical teaching of thyroid disease 2020Assessing the quality of studies in meta‐research: Review/guidelines on the most important quality assessment tools

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.