Safe Optimal Control of Battery Energy Storage Systems via Hierarchical Deep Reinforcement Learning

Abstract

Effective control of Battery Energy Storage Systems (BESSs) and household appliances is crucial for transitioning toward a sustainable and robust power grid. This paper presents a Hierarchical Reinforcement Learning (HRL) control framework, executed by Deep Reinforcement Learning (DRL) agent to achieve effective control of BESSs. The proposed HRL approach compartmentalizes the control problem into overarching strategic decisions and detailed executable actions. At the higher-level, we secure a set of actions that guarantee the safety of the BESS operations. Building upon this base, the next tier of our approach is dedicated to achieving optimal performance within the confines of this established safety set. We employ two cutting-edge HRL architectures to benchmark against our proposed HRL method. Simulation results indicate that our HRL model outperforms traditional DRL, other HRL techniques, and classical optimization methods like Quantum Delta-Potential-Well-based Particle Swarm Optimization. Our approach achieves the highest reward for the defined BESS challenge and reduces computational time to just 10.7 percent compared to the best-performing DRL agent. These results underscore the proposed HRL's promise as a scalable, efficient controller for both BESS and household utilities.