A Code Completion Model based RWKV with Bimodal Pretraining

Intelligent code completion has gained popularity as a technology that accelerates the modern software development process by learning vast amounts of code data and providing programming suggestions to developers. However, traditional RNN-based code completion models encounter issues related to their slow training speed, limited number of network layers, and high computational complexity. These concerns pose a significant challenge for these models transitioning toward big data and larger models, such as the transformer. Therefore, in this paper, we develop a new code completion model called RWKVCC for code comprehension and completion. The model utilizes the RWKV neural network architecture, which boasts faster training speeds, deeper network layers, and shorter transformed abstract syntax tree sequence lengths, resulting in improved expressiveness and performance compared to previous RNN-based code completion models. Experimental results demonstrate the superior code completion effect of RWKVCC, as measured by ACC, EM, and Edit Sim, compared to traditional RNN-based models.