Mobile robot sequential decision making using a deep reinforcement learning hyper-heuristic approach

Sequential decision making is an important part of robotic problems that is receiving unprecedented attention from both academia and industry. Recently, Deep Reinforcement Learning (DRL) has shown its promising capabilities in decision making problems. However, traditional DRL algorithms directly operate in the space of low-level actions, when it is applied in the domain of robotics, it can easily result in an exponential growth of computational complexity and suffer from the “curse of dimensionality”, becoming less efficient as the dimensionality of the environment increases. To address this issue, a novel DRL hyper-heuristic approach is proposed in this paper. The proposed approach is tailored to align with a problem taken from a real-world competition by taking advantage of well-developed low-level heuristic actions in order to narrow the search space and speed up the convergence. This fundamental contribution is a significant step forward from earlier approaches that directly exploit the entire low-level action domain. A state augmentation scheme and a novel reward design are utilized to further improve the performance of the proposed method. Moreover, a Real-to-Sim based training framework is developed to reduce the cost of acquiring real-time data and improve the robustness of agent's decision-making model. Numerous experimental results demonstrate our proposed method can achieve notable performance gains compared to both competitive DRL baselines and heuristic approaches of the same problem in both known environment and previously unseen scenarios.