Projection Exploration for Multi-agent Reinforcement Learning

Abstract In multi-agent reinforcement learning (MARL), complete exploration is difficult to achieve because of the curse of dimensionality and sparse rewards. Existing methods improve the exploration to some extent, such as noise-based exploration methods and count-based exploration methods. But they don’t evaluate the state exploration value, suffering low data efficiency. In this paper, a projection exploration algorithm (PERL) is designed for MARL. In this method, states with high exploration value are identified in the projected state-action space with maximum distribution entropy through the count-based method. Then the agents are trained to reach these states in a coordinated manner. Meanwhile, comparison experiments are conducted in multi-agent environments of Multi-Particle environments (Pass, Secret-Room) and StarCraftII (3m,2s3z and 3s_vs_5z). Comparison results suggest that PERL consistently outperforms the state-of-the-art baselines in environments with high-dimensional state-action space and sparse rewards.