Molecule Generation for Drug Discovery with New Transformer Architecture

Since the outbreak of the COVID-19 pandemic, the rapid development of new drug candidates has been demanded. To satisfy such demand, various deep generative models have been applied to drug generation techniques, and recurrent neural networks (RNNs) based models have achieved the state-of-the-art performance. Since the RNNs architecture suffers from the long-term dependency problem, recently Transformer-based models have been proposed to address the problem with the self-attention mechanism. However, the Transformer models showed worse performance than the RNNs models in the drug generation task, and we believe it is because the Transformer model is over-parameterized with the over-fitting problem. In this paper, we propose to replace the large decoder with simple feed-forward layers to avoid the problem. In our experiments, we demonstrate that our proposed model outperforms the previous state-of-the-art baseline in major evaluation metrics while preserving other minor metrics with a similar level of performance. Furthermore, when we apply our models to generate candidate molecules against the SARs-CoV-2 (COVID-19) virus, the generated molecules are more effective than the drugs in commercial market like Paxlovid, Molnupiravir, and Remdesivir.