Advances and Challenges in Learning from Experience Replay

Abstract From the first theoretical propositions in the ’50s, inspired by the neuroscience and psychology studies about the learning processes in human beings and animals, to its application in real-world problems on learning to action, Reinforcement Learning (RL) is still being a fascinating, rich, and complex class of machine learning algorithms. In particular, we will start reviewing its fundamental principles and develop a discussion about how a technique called Experience Replay (ER) has been of fundamental importance in making a variety of methods in most of the relevant problems and different domains more data efficient, using agent experiences to improve its performance. We present some of the more relevant methods in the literature, which base most recent research on improving RL with ER. Finally, we bring from the recent literature some of the main trends, challenges, and advances focused on reviewing and discussing ER formal basement and how to improve its proposition to make it even more efficient in different methods and domains.