Determinants of Interrater Reliability of Job Analysis: A Meta-analysis

Olga F. Voskuijl · Tjarda van Sliedregt

63 citations0/0 files storedEuropean Journal of Psychological Assessment

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Reliability and Agreement in Measurement Psychometric Methodologies and Testing Employer Branding and e-HRM

Abstract

Summary: This paper presents a meta-analysis of published job analysis interrater reliability data in order to predict the expected levels of interrater reliability within specific combinations of moderators, such as rater source, experience of the rater, and type of job descriptive information. The overall mean interrater reliability of 91 reliability coefficients reported in the literature was .59. The results of experienced professionals (job analysts) showed the highest reliability coefficients (.76). The method of data collection (job contact versus job description) only affected the results of experienced job analysts. For this group higher interrater reliability coefficients were obtained for analyses based on job contact (.87) than for those based on job descriptions (.71). For other rater categories (e.g., students, organization members) neither the method of data collection nor training had a significant effect on the interrater reliability. Analyses based on scales with defined levels resulted in significantly higher interrater reliability coefficients than analyses based on scales with undefined levels. Behavior and job worth dimensions were rated more reliable (.62 and .60, respectively) than attributes and tasks (.49 and .29, respectively). Furthermore, the results indicated that if nonprofessional raters are used (e.g., incumbents or students), at least two to four raters are required to obtain a reliability coefficient of .80. These findings have implications for research and practice.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 63 citations exist upstream.

References

1977Handbook of Industrial and Organizational Psychology.1988Statistical Methods for Meta-Analysis 1991Meta-Analytic Procedures for Social Research 1994Meta-Analytic Procedures for Social Research.1998The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings.1984Essentials of Behavioral Research: Methods and Data Analysis 1990A Continuum of Impression Formation, from Category-Based to Individuating Processes: Influences of Information and Motivation on Attention and Interpretation 1984Validity and utility of alternative predictors of job performance.

Cited by

2004Easing the Inferential Leap in Competency Modelling: The Effects of Task‐related Information and Subject Matter Expertise*2011The Rise and Fall of Job Analysis and the Future of Work Analysis 2010Work analysis: From technique to theory.2009USING WEB‐BASED FRAME‐OF‐REFERENCE TRAINING TO DECREASE BIASES IN PERSONALITY‐BASED JOB ANALYSIS: AN EXPERIMENTAL FIELD STUDY 2015Reconciling the Two Disciplines of Organisational Science: A Comparison of Findings from Lab and Field Research

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.