Guidelines for Planning and Evidence for Assessing a Well-Designed Experiment

Abstract

Since the design of experiments was first introduced by Fisher 90 years ago, this scientific and statistical approach to system interrogation for acquiring knowledge has enjoyed success across industries and among products, processes, and services. In the last decade, military test organizations have been promoting the use of design of experiments (DOE) as the preferred method of constructing and analyzing test programs. Increasingly, design of experiments is being used to greater effect and its impact is reaching groups less experienced in the method. Stories of successful application continue to have a common thread: detailed, effective planning. But not all organizations have members experienced in DOE test planning. And although planning papers and how-to case studies have appeared in the literature, the volume of these contribution types is dwarfed by theory and methods papers. If an experiment is planned and the planning process is documented, how would one go about assessing that plan? If the desire is to gauge the probability of experiment success as defined by a robust and truthful understanding of the system revealed upon analysis of the data, can the plan be assessed prior to test execution? This article proposes guidelines and evidence that span all phases of the experiment cycle, which can inform assessment of experiment planning soundness. The experiment cycle of plan, design, execute, and analyze (consistent with literature and texts) is used to structure the discussion, geared toward an audience somewhat familiar with the DOE method. Checklists are provided for each experiment phase coupled with descriptions of what would constitute fingerprints of successful implementation.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 28 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.