Explainability of reinforcement learning agents within simulated environments

Abstract

This dissertation explores the use of model‐agnostic Explainable Artificial Intelligence (XAI) techniques for interpreting the behaviour of Reinforcement Learning (RL) agents trained in Unity‐based environments. As RL systems grow in complexity and are applied in critical domains, the need for interpretable decision‐making becomes increasingly important. Traditional Deep Reinforcement Learning (DRL) policies often lack transparency, creating challenges for both developers and end‐users in understanding agent behaviour and ensuring reliability. Improving explainability in this context is essential not only for debugging and validation but also for enhancing trust in autonomous systems. The study involved training agents across three distinct Unity environments using the ML‐Agents toolkit, followed by the application of explainability techniques including surrogate models, SHAP feature attribution, and saliency mapping. Observation–action data was collected during inference and analysed using Python‐based tooling. Custom logic was used to preprocess inputs, train interpretable models, and extract feature attributions aligned with the agents’ input structures. These techniques were tailored to the characteristics of each environment, accounting for differences in action space dimensionality and sensor design. Selected results were integrated into Unity through a seeded post‐hoc visualisation interface to enable real‐time inspection of decision‐making processes using in‐engine UI elements. Results demonstrate that semantic explanations such as SHAP can meaningfully highlight key features driving agent behaviour, and that Unity can serve as a viable platform for embedding visual explanations. Surrogate models offered reliable approximations of discrete agent policies, though they struggled with continuous control due to observation complexity and the noisier nature of regression targets. Decision tree models provided interpretable symbolic representations but were constrained by the high dimensionality of the agent observations. While real‐time explanation remains computationally infeasible, the framework developed in this project lays the groundwork for scalable, interpretable reinforcement learning pipelines. These contributions aim to bridge the gap between algorithmic transparency and user‐facing explainability within high‐fidelity simulation environments.