CV classification in a big data environment using BERT and alignment learning

Today, text classification has always been a crucial discipline in the field of text data processing. However, with the advent of Big Data, text classification has reached new heights in terms of the volume of data to be processed and the complexity of the tasks. This task is of great importance in many fields, such as spam detection, sentiment analysis, document categorization, content recommendation and many more. In this work, we focus specifically on resume classification, an area that presents unique challenges due to diverse formats, ambiguous language, and variations in applicants' work experiences. We propose an innovative approach for distributed classification of CVs using contextual alignment techniques with the job offer and the pre-trained language model BERT (Bidirectional Encoder Representations from Transformers) which constitutes a major advance in this field , and using RNN (Recurrent Neural Network) to solve text classification problems. We use a distributed processing approach to leverage the parallel computing power of Spark, enabling large volumes of data to be processed efficiently. The main objective of our study is to improve the relevance of CV classification by leveraging the pre-trained language models and distributed processing power of Spark.