Statistical Foundations for Model Validation: Two Papers

Abstract

This workshop was sponsored by the Defense Modeling & Simulation Office and other government agencies and hosted by the Johns Hopkins University Applied Physics Laboratory.Its first objective was to produce a clear & comprehensive description of the state of the verification and validation art for modeling and simulation.Complete conference proceedings on CD-ROM can be obtained from the DMSO. OverviewValidation of a computer model means the comparison of computational predictions to physical outcomes of the events being predicted.Conventional validation practice is to use these comparisons to make a pass/fail, valid/not-valid decision about the computer model.I take the view that the purpose of these comparisons is to characterize how well the computer model predicts, first within the realm of conditions in which validation-experiments are conducted, then for real-world applications in which computational predictions are required.Because of limitations on experiments, this latter objective may require an extrapolation, or inference.Basically, the objective is to take what we can learn about a model's predictive capability in an experimental region and use that information, buttressed by scientific understanding of the phenomena being modeled and predicted, to make credible, defensible, communicable statements about predictive capability for pertinent applications of the model that may be outside the realm of experimental or other available data.This is not easy.What we learn about predictive capability depends on the nature and extent of the experiments conducted for that purpose.This means that experimental design, the specification of a suite of experiments and corresponding computational predictions, defines the breadth and depth of what we can potentially earn about predictive capability.Then, the analysis of the predictivecapability data that are obtained from a possibly quite diverse set of experiments summarizes and communicates what we have learned from these experiments and computations.These two elements, experimental design and data analysis, are fundamentally statistical in nature.For this reason the V & V02 Workshop sponsors organized a session on the statistical foundations of model-validation.The first paper, co-authored by James Berger, Duke University, and myself sets up a statistical model for the predictive-capability problem and identifies and discusses various issues that need to be addressed in experimental design and statistical data analysis.To provide concrete illustrations of the methods involved in addressing these issues, two implementation papers, or case studies, were also presented.One, by James Berger and associates from Duke University and the National Institute of Statistical Sciences defines and illustrates a Bayesian analysis of a suite of computational predictions and experimental data.That paper is in the Workshop Proceedings or is available from the authors.The second implementation paper, included in this report, presents my analysis, from what is sometimes called a "classical" or "frequentist" statistical perspective, of a set of Sandia experiments and predictions pertaining to the degradation of polyurethane foam in a thermal environment.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 30 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.