A risk-based tool for documenting and auditing the modelling process

Abstract

Integrated modelling literature recognises the need to focus on process in model development, and provide documentation to allow critical review. It should be possible to critique key judgement calls made by the modeller. In order to achieve this, the whole modelling process needs to be documented in sufficient detail that ideally it could be replicated. Existing documentation, in reports and journals, rarely achieves the required level of detail in describing the modelling process, as it represents the authors' necessarily restricted view of the audience's immediate needs. While methods may vary between projects, the same uncertainties are usually encountered along a broadly similar set of model development steps. This paper therefore proposes an approach, and a supporting tool, to systematically document uncertainties rather than tasks. An uncertainty is a reason for doubting that the right choice was made. Uncertainties arise from boundary judgements, i.e. in choosing one course of action and excluding alternatives, there may be a reason that the right or 'true' value may be missed. To manage uncertainty, a set of uncertainty management actions need to be addressed. To identify whether an uncertainty has been satisfactorily managed, the impact of the uncertainty on the end user's objectives, i.e. the corresponding risk, must be acceptable. A hierarchical documentation format is used to provide a consistent but extensible structure, minimising input required and maximising flexibility in manipulating the data to provide output reports. As an audit tool, this approach and tool can be used to evaluate the completeness of documentation, and allow step-by-step review of a modelling process. It can also be used by the modeller as an aid to reflection on the model development process to ensure uncertainty is treated thoroughly. By building a shared catalogue of workflow elements for modelling and uncertainty management, the documentation structure can: alert the modeller to missed uncertainties; inform the development of best practice guidelines; and allow consistent communication of accumulated uncertainties and their implications for even a large multi-disciplinary geographicallydistributed integrated modelling project. The approach and supporting tool is compared with a targeted literature review highlighting the roles in quality assurance documentation of guidelines, tasks vs. boundary judgements and uncertainty. Issues are discussed related to the work involved in detailed documentation, use of the large amount of information collected and adoption of such tools. The argument is made that appropriately designed information technology software (in prototype development) is able to minimise these problems, and help the documentation approach deliver its multiple benefits to both individual modellers and the larger modelling, uncertainty and risk management community. Show more