Reinforcement Learning-based Artificial Pancreas Systems to Automate Treatment in Type 1 Diabetes

Abstract

Type 1 Diabetes (T1D) is a chronic disease, which impairs the glucose homeostasis of the body due to a deficiency in insulin production. People with T1D are life dependent on external insulin administration. Despite advancements in insulin treatment, current methods are still associated with heavy cognitive burden due to the need for manual meal announcement, carbohydrate (CHO) estimation, and insulin dose calculation. To automate treatment, Artificial Pancreas Systems (APS) have been introduced; they consist of a continuous subcutaneous glucose sensor, insulin pump, and control algorithm to estimate the insulin dose. However, the glucoregulatory system is a partially observable complex dynamical system with large inter- and intra-population variability, affected by unknown disturbances related to meals, exercise, and stress among others. This complexity, along with delays in glucose sensing and insulin action, has challenged the performance of existing control algorithms, which still require users to engage manually. Related research has advanced to exploring Machine Learning (ML) for APS and integrating additional physiological signals as Multi-input Artificial Pancreas Systems (MAPS). This thesis presents a body of novel research on using Reinforcement Learning (RL) to address control challenges in APS, where RL-based APS are designed and developed to automate insulin delivery by eliminating CHO estimation and meal announcement. Secondarily, the thesis explores and analyses the use of additional physiological signals for MAPS.

The glucose regulation problem was formulated as a continuing task using the average reward RL framework with new state-space, action-space, and reward function designs addressing real-world challenges associated with the problem. State-of-the- art RL algorithms were first explored to design RL-based APS. A novel algorithm named G2P2C (Glucose Control by Glucose Prediction and Planning) was designed and developed to address the shortcomings identified in state-of-the-art algorithms. G2P2C integrated Proximal Policy Optimisation (PPO) with a model-learning phase as an auxiliary learning task and a planning phase to improve performance and safety. G2P2C did not require any prior announcement of upcoming meals or meal CHO content. Its performance was tested in-silico using an open-source T1D simulator based on the Food and Drug Administration-approved UVA/PADOVA 2008 model and benchmarked against clinical treatment strategies and state-of-the-art RL algorithms. G2P2C achieved a time-in-range of 73% and 64% for adult and adolescent cohorts, respectively. This meant it outperformed basal-bolus clinical treatment strategies in the adult cohort without requiring any meal user input. The control performance and algorithmic characteristics of G2P2C show promise as a candidate algorithm for automating glucose control in APS.

The potential of MAPS was explored further by conducting a systematic literature review of 17 shortlisted publications from Scopus, PubMed, and IEEE Xplore in order to identify and analyse input signals of MAPS. Heart rate, accelerometer readings, energy expenditure, and galvanic skin response were the main found signals that can be captured by non-invasive wearable devices, while evidence was also found for lactate and adrenaline as potential invasive biomarkers of T1D. Finally, review findings were used to analyse the relationship of the above signals with exercise types (e.g., moderate-intensity, high-intensity, and resistance exercise) that affect glucose control in T1D. This analysis validated lactate and noradrenaline as important biomarkers in estimating exercise events. This highlighted their value for next-generation MAPS. This research is expected to be valuable for the T1D diabetes community through solutions to reduce the cognitive burden and exploration of MAPS; and for the RL community through the development of new RL algorithms targeting real-world applications.