A Driving Model in the Realistic 3D Game Trackmania Using Deep Reinforcement Learning

Deep Reinforcement Learning has achieved state-of-the-art results in the field of autonomous driving, especially in video game simulations. Trained agents were able to beat the best players achieving supremacy in the field. However, these agents often take in their neural network input the geometry of the track during the learning process. Our approach is to create a learning procedure for autonomous driving agents that is not environment-dependent by design to extend Deep Reinforcement Learning to racing games without racetrack information. Our work builds upon the notorious game Trackmania Nations United Forever, an open access 3D realistic Formula 1 simulation game that offers a high degree of customization. We use the Proximal Policy Optimization algorithm, using hybrid raw pixel data and physics sensors vectors as agent observations. The training track is a custom-made circuit carefully crafted from all available blocks in the game. The training track is considered like a standard dataset in machine learning and thus split in several segments in which the agent is randomly put into. We also created two regularization reward functions for the car’s speed and position to increase robustness of the agent in new scenarios.