Extending Goal-Oriented Specification for Deep Reinforcement Learning to High-Dimensional State Spaces

Deep Reinforcement Learning is used in many complex control problems, such as autonomous driving and control of humanoid robots. The design of a reward function is one of the key challenges. It is often unintuitive and especially difficult for users with limited experience in Reinforcement Learning. Specification languages, in particular goal-oriented specification¹, address this problem and represent an alternative approach to classical reward functions. However, goal-oriented specification defines goals as a subspace of the state space. Therefore, in practice, it is limited to intuitive and lower-dimensional state spaces.

This work extends goal-oriented specification to enable its application to high-dimensional state spaces. We introduce two extensions as our main contributions. First, we present an intuitive feature space that is independent of the high-dimensional state space. This allows defining goals as a subspace of the feature space. Second, we introduce a separate training space. The training space is constructed from either both the high-dimensional state space and the feature space or only one of them and is used as input for the agent.

Markov Decision Process composed of initial environment and engineered aspects. Adapted from Schwan et al.¹ Extensions of this work are added in green.

We validate the proposed approach with two case studies, using the Pendulum environment from the Gymnasium framework and the Pong environment from the Arcade Learning Environment. Our results demonstrate that goal-oriented specification can be successfully applied to high-dimensional state spaces. With this, we overcome prior limitations of goal-oriented specification and enable their application in high-dimensional state spaces.

For further information, you can download the publication that resulted from this work at Springer Nature Link. The code is available via Figshare.

¹ Simon Schwan and Sabine Glesner. "Iterative Environment Design for Deep Reinforcement Learning Based on Goal-Oriented Specification:" in: Proceedings of the 17th International Conference on Agents and Artificial Intelligence. Porto, Portugal: SCITEPRESS - Science and Technology Publications, 2025, pp. 240-251.