Regularly updated benchmark sets for statistically correct evaluations of AlphaFold applications

AlphaFold2 changed structural biology by providing high-quality structure predictions for all possible proteins. Since its inception, a plethora of applications were built on AlphaFold2, expediting discoveries in virtually all areas related to protein science. In many cases, however, optimism seems to have made scientists forget about data leakage, a serious issue that needs to be addressed when evaluating machine learning methods. Here we provide a rigorous benchmark set that can be used in a broad range of applications built around AlphaFold2/3. Graphical abstract Key Points When building applications on AlphaFold, scientists should consider the possibility of data leakage between AlphaFold training set and the independent test set of their method BETA provides multiple datasets with structures and sequences that were not used during the training of AlphaFold These datasets provide a diverse range of use cases The protocol was applied when building a simple disordered prediction method, showing different parameters required to optimize disordered prediction for proteins not used in AlphaFold training