The Idealized Machine Learning Pipeline (IMLP) for Advancing Reproducibility in Machine Learning

Abstract

We investigate the influence of Human Factors on reproducible machine learning, using a novel “Idealized Machine Learning Pipeline” we introduce as a conceptual framework. The study of Human Factors seeks to ensure that systems meet the needs and expectations of people who use and interact with these systems. It also, importantly, seeks to ensure that the capabilities and limitations of those people are accommodated by the system. As increasing the reproducibility of Machine Learning continues to be a community priority, we believe an improved understanding the Human Factors associated with the complex human-machine system of the Machine Learning pipeline could help facilitiate useful steps toward reproducibility. To do this, we first define a practical abstraction of the steps that comprise a typical Machine Learning pipeline for a well-known use case, from raw data through to model estimation used for inference and prediction, that we call the “Idealized Machine Learning Pipeline.” We emphasize that our proposed “Idealized Machine Learning Pipeline” is intended as an abstraction, rather than a directive or description of all Machine Learning workflows, and meant to harmonize and integrate different viewpoints, expertise, and other elements vital to Machine Learning research. Our goal is to enable the research community to coalesce around priorities for theoretical and applied research on reproducible Machine Learning including tools, frameworks, comparisons, and policies, and provide a foundation for communicating and teaching its myriad aspects as an integrated whole. We define the principal steps as follows: 0) Documentation, 1) Problem Definition, 2) Input Data, 3) Data Preparation, 4) Feature Selection, 5) Model Training, 6) Model Evaluation, 7) Preservation & Publication. To then understand barriers and opportunities attending to the adoption of reproducible Machine Learning pipelines, we leverage the “Idealized Machine Learning Pipeline” for our chosen use case to identify and motivate relevant Human Factors that affect various steps in a reproducible Machine Learning pipeline. We find the identified Human Factors fall into three groups: Incentives; Training; and Error-based Human Factors.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 4 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.