A procedure for estimating the reliability of sets of ratings in terms of the intraclass correlation coefficient is discussed. The procedure is based upon the analysis of variance and the estimation of variance components. For the one-way classification the intraclass correlation coefficient defined as the ratio of variances can be interpreted as a correlation coefficient. Caution, however, is urged in the application of the definition to a two-way model, i.e., one in which between-rater variance is removed. It is maintained that the frequent use of the standard definition of the one-way intraclass correlation coefficient applied to the two-way classification is not a proper procedure if in fact the coefficient is to be interpreted as a correlation coefficient. Definitions for reliability obtained from the two-way models are given which can legitimately be considered correlation coefficients.
Stored only when both papers are in the corpus — 2,304 citations exist upstream.