Reproducibility Debt In Scientific Software

Abstract

Reproducibility of scientific computation is critical in validating its underlying process, but is often elusive. Complexity and continuous evolution in software systems have introduced new challenges for reproducibility across various scientific disciplines, resulting in growing debt. In scientific software, the inability to reproduce results often stems from technical issues and challenges in recreating the full computational workflow used in the original analysis. We conceptualise this problem as Reproducibility Debt (RpD). RpD may accumulate when researchers and scientific software developers engage in suboptimal practices, often for short-term benefits, thereby compromising the reproducibility of results derived from scientific software.

This thesis provides a comprehensive, domain-agnostic study that defines and investigates RpD in scientific software. It establishes an integrated definition of Scientific Software Reproducibility, grounded in evidence synthesised from 104 primary studies, and reconceptualises reproducibility as a system-level property shaped by technical, procedural, and organisational factors. This definition extends beyond code and data availability to include transparent sharing of workflows, documentation, and computational environments, thereby providing a rigorous foundation for analysing reproducibility challenges.

Building on this foundation, the thesis introduces RpD as a distinct extension of the Technical Debt (TD) paradigm tailored to scientific software. While conceptually related to TD, RpD differs in its origins, incentive structures, temporal dynamics, and consequences. It emerges from research-specific pressures such as rapid experimentation, publication-driven incentives, and the prioritisation of short-term scientific outputs over long-term software sustainability, directly impacting scientific validity rather than solely software quality.

The thesis systematically uncovers and classifies the underlying factors contributing to the accumulation of RpD, focusing on both its causes and effects. A Systematic Literature Review (SLR) was conducted to chart the landscape of RpD, organising and synthesising reproducibility issues discussed in existing research. The evidence found in the literature was further investigated and validated through Interviews with practitioners working with scientific software, allowing a deeper understanding of real-world challenges. The insights gained from both studies informed the design of a survey tool, InsightRpD, which was deployed to gather data from global participants.

The triangulation of results from the three studies, SLR, Interviews, and Survey, led to the development of a multidimensional taxonomy of RpD and the Reproducibility Debt Management Framework (RpD-MF). The taxonomy captures data-, code-, documentation-, infrastructure-, versioning-, human-, and legal-related dimensions of reproducibility challenges, while the framework provides structured strategies to identify, prioritise, mitigate, and prevent RpD in scientific and research software projects. Collectively, the findings establish RpD as a systemic and specialised challenge that requires both theoretical grounding and structured management approaches within research software engineering. Show more