Are environmental models transparent and reproducible enough?

Abstract

Environmental computer models are considered essential tools in supporting environmental decision making, but their main value is that they allow a better understanding of our complex environment.Environmental models are in fact applications of shared theories on how real-world systems are functioning.Just like the underlying scientific theories they need to be evaluated and discussed among peers.To allow proper assessment of the quality and suitability of environmental models peers should be able to trace their results and insights through the model structure to the underlying choices and assumptions.This ideal of model evaluation should take place in the peer-review process before their output and the analysis of their output are published in scientific journals, but is hardly ever realized in reality.It is hypothesized that, despite the numerous attempts to promote good modeling practice and extended peer review, the reproducibility and transparency of environmental models is limited in actual terms.To test the validity of this hypothesis we have reviewed publications, documentation and software of four environmental models.We analyzed to what extent this material provided insight in the model structure and the modeling process and to what extent model findings could be traced back to the underlying choices and assumptions.All four models and their applications have been described in dozens of articles in peer reviewed journals.This indicates that these models and there results and insights are trusted and used.They can be understood as wellestablished models grounded in a scientific theory.However, in our study we found that for at least three of the models reviewers lack information to evaluate their quality or suitability.Neither can they ensure the reproducibility and transparency of the model results and insights, for a number of reasons.First, information on model design is scattered over several sources, which are to a limited extent freely and easily available to reviewers.Second, written model documentation does not provide a sufficient description of the modeled system.Third, written model documentation does not provide a sufficient description of the calculation of model results.Finally, results presented in scientific publications do not necessarily correspond with parameter values or equations in the model source code.Our findings suggest that environmental models lack essential quality characteristics in terms of transparency and reproducibility.This raises the concern that they are being used in applications without respecting and discussing their underlying choices and assumptions.We identify three structural causes for this lack of transparency and reproducibility: (1) the size and complexity of environmental models, (2) a lack of incentives for environmental modelers to be to transparent in the modeling process and (3) the use of computers and the focus on computation and simulation instead of the descriptive side of modeling.We submit that openness in the modeling process can only be achieved with a general change of attitude.On the one hand model developers must become explicit and open about their choices and assumptions.On the other hand peers, stakeholders and journals must request openness and challenge these choices and assumptions.In an operational sense, computers and networks can be turned to their advantage by having them disseminate high-quality model descriptions using shared vocabularies.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 5 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.