The purpose of the present investigation was to assess the agreement between two independent reviews of each of 278 manuscripts submitted to the Journal of Educational Psychology on an overall recommendation and on specific rating items. The single-reviewer reliability for the overall recommendation, .30, was comparable with the .34 in a previous study with the same journal and with an average value of .27 found in other research. Factor analyses and multitrait-multimethod analyses provided support for four dimensions: Research Methods, Relevance to Readers, Writing Style/Presentation Clarity, and Significance/Importance. However, idiosyncratic response biases (e.g., tendencies toward leniency or harshness in responses) were larger for the specific rating items that defined these dimensions than for the overall recommendation that was better anchored in terms of behavioral consequences. Consequently, agreement between different reviewers on each separate dimension, on the unweighted sum of these dimensions, and on various empirically weighted sums of these dimensions was no better than for the overall recommendation by itself. The findings were discussed in terms of the reliability and validity of the review process and implications for further research.
Stored only when both papers are in the corpus — 120 citations exist upstream.