A simulation of the impacts of machine learning to combine psychometric employee selection system predictors on performance prediction, adverse impact, and number of dropped predictors

Abstract

Abstract We compare modern machine learning (MML) techniques to ordinary least squares (OLS) regression on out‐of‐sample (OOS) operational validity, adverse impact, and dropped predictor counts within a common selection scenario: the prediction of job performance from a battery of diverse psychometrically‐validated tests. In total, scores from 1.2 billion validation study participants were simulated to describe outcomes across 31,752 combinations selection system design and scoring decisions. The most consistently valuable improvement from adopting MML over traditional regression was from dropping predictors rather than by improving prediction. On average, MML improved prediction of performance from psychometric scale composites only when the ratio of sample size to scale count was less than approximately 3, although algorithm choice, predictor count, and selection ratio affected outcomes as well. We also simulated the effects of design choices when combining item scores, which showed consistent, superior predictive accuracy for several MML algorithms, especially elastic net and random forest, over traditional regression. Given these results, we suggest the potential of machine learning for employee selection is unlikely to be realized in selection systems focusing on the combination of scale composites from previously validated psychometric tests. Instead, it will be realized in unconventional design scenarios, such as the use of individual items to make multiple trait inferences, or with novel data formats like text, image, audio, video, and behavioral traces. We therefore recommend researchers focus on the potential value of MML in future selection contexts rather than continuing to focus on the current value of MML in current selection contexts.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 22 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.