Rigor in Data Synthesis: A Case Study of Reliability in Meta-Analysis

William Albert Stock · Morris Alan Okun · Marilyn J. Haring · Wendy Miller · Clifford Kinney · Robert W. Ceurvorst

68 citations0/0 files storedEducational Researcher

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Meta-analysis and systematic reviews Psychology Research and Bibliometrics Reliability and Agreement in Measurement

Abstract

The subject of this study was the reliability among persons coding information from primary research sources for use in a metaanalysis. Meta-analysis is a quantitative procedure for synthesizing data in primary research reports (Glass, 1976; 1977). As such, it is one of a number of quantitative procedures proposed as alternatives to the traditional, verbal review of literature. This study rests on the assumption that meta-analysis in particular, and quantitative data synthesis procedures in general, should be examined for sources of invalidity as rigorously as is the process of generating primary research. Pillemer and Light (1980) defined data synthesis as, using formal procedures for combining the results from several experiments (p. 177). Until recently, the traditional review of literature has been the primary formal mechanism for data synthesis. However, in the context of burgeoning numbers of primary research sources, alternate quantitative procedures have been proposed (see Pillemer & Light, 1980, for a review). Among the different methods that have been proposed for data synthesis, meta-analysis has been applied most widely. With regard to primary research, numerous texts call attention to sources of invalidity which investigators must try to avoid (Campbell & Stanley, 1966). However, only recently has it been suggested that quantitative data synthesis also is subject to threats to its validity, and that data synthesis studies should be examined for sources of invalidity as carefully as primary research (Glass, McGaw, & Smith, 1981; Cooper, Note 1). Although few empirical efforts have focused directly on assessing the validity of meta-analysis, critics have expressed several objections. First, they have called attention to the lack of a statistical

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 68 citations exist upstream.

References

1963Experimental and Quasi-Experimental Designs for Research 1969The Handbook of Social Psychology 1973Review of research in education 2013Handbook of Social Psychology 1965Educational Research: An Introduction 1981Distribution Theory for Glass's Estimator of Effect Size and Related Estimators 1984Educational Research: An Introduction 1976Primary, Secondary, and Meta-Analysis of Research

Cited by

1985Marital Status and Subjective Well-Being: A Research Synthesis 1992Summarizing Monte Carlo Results in Methodological Research 1993On the Reliability of Meta-Analytic Reviews 2009An Empirical Assessment of Meta-Analytic Practice 1986Chapter 11: Issues in Meta-Analysis 1985Going beyond the narrative summarization of research findings: The meta‐analysis approach 1984A review and application of citation analysis methodology to reading research journal literature

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.