Performance Comparison of Large Language Models, GPT and Gemini on Turkish News Classification Task

Abstract Recently, large language models-LLMs have become very popular in many tasks of natural language processing. Examples of these tasks include text classification, question answering, text summarization, and text generation for natural language processing. Apart from LLMs, GPT and Gemini models are at the top of the list in terms of use for text generation tasks. This study aims to contribute to the literature on the use and comparison of LLMs and text generation models for the Turkish language. To achieve this purpose, the dataset consisting of Turkish news was classified by training BERT, ALBERT, DistilBERT, ELECTRA, XLM-RoBERTA LLMs with fine-tuning. Additionally, GPT-3.5 and Gemini text generation models were used by sending prompts for this classification task, and the success of the models was compared with LLMs. As a result of all analyses, the BERT model gave 97.619% accuracy among LLMs, while Gemini gave 99.167% accuracy among text generation models.