Vote-Counting Procedures In Meta-Analysis

Abstract

As the number of scientific studies continues to grow, it becomes increasingly important to integrate the results from these studies. One simple approach involves counting votes. In the conventional vote-counting procedure, one simply divides studies into three categories: those with significant positive results, those with significant negative results, and those with nonsignificant results. The category containing the most studies is declared the winner. For example, if the majority of studies examining a treatment found significant positive results, then the treatment is considered to have a positive effect. Many authors consider the conventional vote-counting procedure to be crude, flawed, and worthless (see Friedman 2001; Jewell and McCourt 2000; Lee and Bryk 1989; Mann 1994; Rafaeli-Mor and Steinberg 2002; Saroglou 2002; Warner 2001). Take, for example, the title of one article: "Why Vote-Count Reviews Don't Count" (Friedman 2001). We agree that the conventional votecounting procedure can be described in these ways. But all vote-counting procedures are not created equal. The vote-counting procedures described in this chapter are far more sophisticated than the conventional procedure. These more sophisticated procedures can have an important place in the meta-analyst's toolbox. When authors use both vote-counting procedures and effect size procedures with the same data set, they quickly discover that vote-counting procedures are less powerful (see Dochy et al. 2003; Jewell and McCourt 2000; Saroglou 2002). However, vote-counting procedures should never be used as a substitute for effect size procedures. Research synthesists generally have access to four types of information from studies: • a reported effect size, • information that can be used to compute an effect size estimate (for example, raw data, means and standard deviations, test statistic values), • information about whether the hypothesis test found a statistically significant relationship between the independent and dependent variables, and the direction of that relationship (for example, a significant positive mean difference), and • information about only the direction of the relationship between the independent and dependent variables (for example, a positive1 mean difference). These types are rank ordered from most to least in terms of the amount of information they contain (Hedges 1986).2 Effect size procedures should be used for the studies that contain enough information to compute an effect size estimate (see chapters 12 and 13, this volume). Vote-counting procedures should be used for the studies that do not contain enough information to compute an effect size estimate but do contain information about the direction and the statistical significance of results, or that contain just the direction of results. We recommend that vote-counting procedures never be used alone unless none of the studies contain enough information to compute an effect size estimate. Rather, vote-counting procedures should be used in conjunction with effect size procedures. As we describe in section 11.4, effect size estimates and votecount estimates can be combined to obtain a more precise overall estimate of the population effect size. Copyright © 2009 by Russell Sage Foundation. All rights reserved.

Cite this

Brad J. Bushman, Morgan C. Wang (2009). Vote-Counting Procedures In Meta-Analysis. Journal of International Crisis and Risk Communication Research.

@article{j2009votecounting,
  title  = {Vote-Counting Procedures In Meta-Analysis},
  author = {Bushman, Brad J. and Wang, Morgan C.},
  year   = {2009},
  journal = {Journal of International Crisis and Risk Communication Research}
}

Citations in this library

Stored only when both papers are in the corpus — 162 citations exist upstream.

Cited by