Comparative evaluation of three situational judgment test response formats in terms of construct-related validity, subgroup differences, and susceptibility to response distortion.

Winfred Arthur · Ryan M. Glaze · Craig D White · Ira Schurig

61 citations0/0 files storedJournal of Applied Psychology

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Psychometric Methodologies and Testing Psychological Testing and Assessment Medical Education and Admissions

Abstract

As a testing method, the efficacy of situational judgment tests (SJTs) is a function of a number of design features. One such design feature is the response format. However, despite the considerable interest in SJT design features, there is little guidance in the extant literature as to which response format is superior or the conditions under which one might be preferable to others. Using an integrity-based SJT measure administered to 31,194 job applicants, we present a comparative evaluation of 3 response formats (rate, rank, and most/least) in terms of construct-related validity, subgroup differences, and score reliability. The results indicate that the rate-SJT displayed stronger correlations with the hypothesized personality traits; weaker correlations with general mental ability and, consequently, lower levels of subgroup differences; and higher levels of internal consistency reliability. A follow-up study with 492 college students (Study 2; details of which are presented in the online supplemental materials) also indicates that the rate response format displayed higher levels of internal consistency and retest reliability as well as favorable reactions from test takers. However, it displayed the strongest relationships with a measure of response distortion, suggesting that it is more susceptible to this threat. Although there were a few exceptions, the rank and most/least response formats were generally quite similar in terms of several of the study outcomes. The results suggest that in the context of SJTs designed to measure noncognitive constructs, the rate response format appears to be the superior, preferred response format, with its main drawback being that it is susceptible to response distortion, although not any more so than the rank response format.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 61 citations exist upstream.

References

2011Sources of Method Bias in Social Science Research and Recommendations on How to Control It 1977Handbook of Industrial and Organizational Psychology.1991THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META‐ANALYSIS 1993Comprehensive meta-analysis of integrity test validities: Findings and implications for personnel selection and theories of job performance.1990An alternative selection procedure: The low-fidelity simulation.1997Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions.2001Identification and meta-analytic assessment of psychological constructs measured in employment interviews.2006A Meta‐Analytic Investigation of Job Applicant Faking on Personality Measures

Cited by

2017A Call for Conceptual Models of Technology in I-O Psychology: An Example From Technology-Based Talent Assessment 2020Situational Judgment Tests: An Overview of Development Practices and Psychometric Characteristics 2017An Information-Processing-Based Conceptual Framework of the Effects of Unproctored Internet-Based Testing Devices on Scores on Employment-Related Assessments and Tests 2020

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.