Characterising Algorithm Debt in Machine and Deep Learning Systems

Abstract

The integration of Machine and Deep Learning (ML/DL) into modern software systems has transformed application domains such as finance, healthcare, and autonomous technologies. However, the complexity of ML/DL algorithms, the unique development pipeline, and the stochastic training process introduce challenges beyond those found in traditional software development, compromising the long-term reliability of ML/DL systems. These challenges, often due to design choices, lead to Technical Debt (TD). Algorithm Debt (AD) in ML/DL systems is a TD type caused by inefficient algorithmic decisions, leading to poor scalability and model degradation. Despite its impact, AD remains underexplored in ML/DL systems due to its novelty, with limited and fragmented studies, creating a knowledge gap.

To address this gap, this thesis aims to characterise AD in ML/DL systems. It defines AD, investigates its causes, effects, and mitigation strategies, using a sequential mixed-methods approach. The first study, a Systematic Literature Review (SLR), synthesised evidence from 44 included studies published across IEEE, Springer, ACM, and ScienceDirect. This review produced an extended definition for AD in ML/DL systems and revealed that AD manifests in three ways: poor model scalability, inefficiency, and degradation. The SLR also identified nine ``AD smells'' (i.e., indicators) that lay the foundation for subsequent studies.

Next, an experimental study was conducted to explore automated AD detection in code comments using ML/DL approaches. A labelled dataset of self-admitted TD from seven open-source repositories, identified through the SLR, was used to train and evaluate ML/DL classifiers. However, the results were constrained by the limited nature of the available dataset, which was labelled based on pre-existing, narrower definitions of AD. This underscored the inadequacy of current data for understanding AD and justified the subsequent need for comprehensive studies.

Following the ML/DL experiments, two studies were conducted to gain a deeper understanding of AD through the real-world experiences of ML/DL practitioners and researchers. The first study involved semi-structured interviews with 21 participants. The second study used a questionnaire completed by 65 respondents. These studies revealed the causes, effects, and mitigation strategies of AD in ML/DL systems. Key findings showed that awareness of AD among practitioners is limited. Also, poor model scalability was identified as the major effect of AD, with continuous practitioner education emerging as a key mitigation strategy.

In summary, this thesis provides an enhanced definition of AD and characterises its causes, effects, and mitigation strategies in ML/DL systems. It reveals that AD is a socio-technical phenomenon shaped by both intrinsic ML/DL system properties and engineering decisions. It also introduces a novel, empirically grounded framework for AD through methodological triangulation. The findings underscore the need for continuous practitioner education to address knowledge gaps within the software engineering community, alongside proactive strategies for managing AD. Ultimately, this work lays the foundation for building scalable and reliable AI systems, bridging the gap between research and practice in software engineering.

Keywords: Technical Debt, Algorithm Debt, Machine Learning, Deep Learning, Software Engineering, Automated Detection, AD Smells Show more