Beyond Binary: A Multi-Class Approach to Large Language Model-Generated Text Percentage Estimation

Beyond Binary: A Multi-Class Approach to Large Language Model-Generated Text Percentage Estimation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Beyond Binary: A Multi-Class Approach to Large Language Model-Generated Text Percentage Estimation Ankush Goel, Vineet Kumar, Saibal Patra, Savita Yadav This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4318347/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 3 You are reading this latest preprint version Abstract The usage of AI in the workplace has increased to 22% of those polled, underscoring the increasing demand for the detection of AI-generated material. With an AUROC of 0.95, DetectGPT beat previous models, demonstrating its efficacy in identifying text written by LLM. Still, there are difficulties in recognizing subtleties that are typical of text generated by language models. This study presents perc-DETECTLLM, a unique method to precisely estimate the percentage of LLM- generated text by combining a zero-shot model with a Bayesian model. perc-DETECTLLM as compared to other models, shows promising results with an accuracy of 72.1% and a recall rate of 81.0%, solving the difficulties in machine-generated text detection. More than 500K samples from different domains makeup constitute our dataset, which makes thorough model training and assessment possible. Perc-DETECTLLM achieves promising text classification results by estimating the overall percentage of LLM-generated text using Bayesian inference. Ensemble techniques perform better and are used as a standard for LLM detection models. Compared to rule-based approaches, perc- DETECTLLM's data-driven methodology improves accuracy and dependability. Overall, all the experiment results are promising and could be extended to other domains, there are still difficulties with dynamic thresholding processes and perturbation function optimization, which points to areas for further study. With perc-DETECTLLM, textual content authentication in the digital age is feasible and is a substantial improvement in LLM-generated text detection. Bay’s probability Ensemble model LLM word embedding zero shot model Figures Figure 1 Figure 2 Figure 3 Figure 4 1 Introduction These recent years have belonged to AI with the advancement of generative AI models in computer vision to large language models (LLMs) in natural language processing (NLP) (Junchao and Yang, et al., 2023) which has sparked the need to detect these AI-generated work and human-generated. According to McKinsey, 22% of people surveyed now use AI in some form for their work. Artificial intelligence (AI) tools can be abused by users for immoral activities like spamming, creating fraudulent product reviews, plagiarism, fake news, and social engineering, all of which can have detrimental effects on society. Numerous basic flaws have been included into news stories that AI has rewritten. Therefore, it's important to make sure that these generative AI technologies are used responsibly. Much current research focuses on identifying AI-generated texts in order to facilitate this CITATION Che231 \l 1033 (Chen, et al., 2023) . Text generated by large language models (LLMs) poses many issues and with the rapid development of these LLMs it is becoming harder to distinguish between human and machine-generated work (Chen, et al., 2023). It poses a threat to many industries like creative arts, education, and journalism. A good LLM text detection algorithm can be a great source of help for publication as well as to maintain authentication in research work. Detecting generated text produced by Large Language Models (LLMs) serves various purposes, driven by concerns related to misinformation, ethics, and security. The ability to detect the percentage of LLM content accurately in such conditions is paramount for several reasons: 1. Misinformation and Fake News: Because LLMs can produce extremely convincing and contextually relevant content, it can be difficult to discern between factual and false information. 2. Ethical Use of Technology: It is critical to ensure that LLMs are used responsibly. When automated text generation is being utilized unethically, for example, in the production of frauds, deepfake text, or fraudulent material, detection algorithms can be employed to help identify these cases. 3. Protecting Against Cyber Threats: Phishing emails, social engineering communications, and other cyber threats can be created by malicious actors using LLMs. 4. Preserving Authenticity in Creative Works: Preserving the authenticity of content created by humans is crucial in fields such as academics, journalism, and literature. Maintaining the integrity of artistic and intellectual works can be facilitated by detection tools. The existing research work addressed the following knowledge gaps: 1. There is no existing model that gives a percentage of LLM-generated text based on the probabilistic approach used by the zero-shot model. 2. The binary classification does not account for the padding of human-generated data with LLM-generated text. 3. There is no existing any robust dataset which encounters a percentage of LLM-generated text as a label. The proposed solutions bridge these existing shortcomings by: 1. Created a model that gives a percentage of LLM-generated text based on the probabilistic approach used by the shot model. 2. By knowing the amount of AI-generated text in the content, since binary classification does not account for padding of human-generated data with LLM-generated text. 3. Created a robust dataset with the percentage of LLM-generated text as a label. Earlier LLM detection was solely focused on paraphrasing which is not a very reliable method taking the rapid growth of the LLM model's progress into account, outfox (Koike, et al., 2024) was one first model that went beyond this and used two-way input stream to train a model which significantly improved the performance. Furthermore, this plagiarism is generated from various LLMs such as ChatGPT and Gemini, with each of them having its unique properties which are fundamentally similar, however, their training model is very different which makes it more challenging to detect LLM-generated text. It is important to note here that “rephrasing” is not the only way used to get away with LLM-generated text (Venkatraman, et al., 2023), people also add some of their text in the work to evade AI detection methods like waterfall, zero-shot (Mitchell, et al., 2023) etc. To counter such practice there is a need for detection of the percentage of LLM content in the given text or string, existing algorithms in this field can binary classify the given text but perform poorly when it comes to finding the percentage of LLM content in the text. Further, there does not exist any good dataset with a percentage label of LLM. To address the above research objectives and questions, firstly, a literature review is undertaken to furnish an overview of LLM-generated text detection, their limitations, and opportunities, as outlined in Section 2. Relevant research methodologies are discussed in Section 3, followed by an investigation of the performance of current state-of-the-art deep learning models for LLM-generated text detection in Section 4. The model pertinent discussion is done in Section 5. Finally, novelty of this research includes proposing using the Bayesian model along with the zero-shot model to enhance the zero-shot model’s probabilistic approach to achieve a better level of performance and evidence-based recommendations are presented in Sections 6 and 7. The findings are promising, and they will lead to more studies in the future, relating to various detection algorithms for text generated by LLMs. 2 Literature review 2.1 Overview of LLM models Generative Pre-trained Transformers, which are known as GPTs, are a series of multimodal which have advanced forms of Artificial Intelligence that are designed to understand and generate responses as Humans do. The model has been used in various Domains like translation, content writing, chatbots, image generation, and video generation. 2.1.1 GPT based models The detection of fake content or misinformation is created by large language models like GPT. The paper (Chen, et al., 2023) discusses whether AI-generated misinformation or fake news was more harmful than human-generated misinformation written by humans. It has been seen that LLM-generated misinformation is more the difficult to detect and investigate. The study also found that AI-generated misinformation was more deceptive and difficult to detect, posing a significant challenge to tackle online safety and public trust websites. “GPT-who” CITATION Ven02 \l 1033 (Venkatraman, et al., 2023) , an LLM-generated text detector based on Uniform information Density (UID), suggested that human generally distribute their information or knowledge evenly during their communication. This concept is used in “GPT-who” to detect the difference between human-generated and AI-generated. The paper claims that it performs better than another state-of-the-art detecting model by around 20% on many performance metrics and provides more interpretable representations of the text. 2.1.2 Gemini model Gemini is an advanced chatbot model built by Google DeepMind. The author (Saeidnia & Hamid, 2023) examined the motivation behind the development of Google’s chatbot Gemini and its potential impact on IT industry. Gemini is designed to provide a better user experience by personalized and relevant information and improving customer services. Model built to transform how users access the internet and information and interact with it. We used Gemini’s API to generate text and also used it for the paraphrasing of our dataset. 2.1.3 Necessity of LLM generated text detection The impact of the LLM model on academic and student learning has been seen in higher education through a content analysis of 100 articles from Australia, New Zealand, the United States, and the United Kingdom (Sullivan, et al., 2023) . It explores various benefits and risks associated with using generative AI tools and the need to change teaching style. The assignment may need to adapt to this new reality as the performance of ChatGPT compared to humans in various expert domains is getting better day by day shown in CITATION Guo231 \l 1033 (Guo, et al., 2023) . The study CITATION Kha23 \l 1033 (Khalil, et al., 2023) evaluates LLMs’ ability to understand the theory of mind task. It suggests that as LLMs improve, their language skills may lead to the emergence of human-like cognitive abilities as discussed by (Junchao and Yang, et al., 2023) , emphasize the importance of building detectors for LLM-generated text from potential misuse and protecting different domains like artistic expression, social network. It discusses recent advancements in detection techniques and the challenges faced, such as out-of-distribution problems and data ambiguity. It is not only necessary to detect generated text but also to detect hallucinations or unreliable answers in LLMs. CITATION Hal23 \l 1033 (Chen, et al., 2023) and (Guo, et al., 2023) talk about how LLM could be deceived to generate false information while forcing on that LLM can also make statistical mistakes and how can they prove to be costly in many fields. The need for a robust model to discern reliable information provided by the LLM model is critical for their safe and effective use. 2.1.4 Challenges in detection of LLM generated text Identifying text written by Large Language Models provided several challenges for researchers and practitioners. First, the out-of-distribution problem results in false negatives because detection models cannot identify a novel text, meaning a text that was not part of their training data. In addition, the text could be deceptive in style (Chen, et al., 2023) similar to human prose, which makes the machine-based system and humans difficult to detect that the text was written by the machine. Ultimately, this becomes more challenging and deceiving if production of misinformation is produced by LLM. Another challenge, and restriction was the ineffectiveness of traditional plagiarism systems (Khalil, et al., 2023) they were unable to detect LLM-generated materials because these were new and original. In this case, adversaries could be detecting a mistake in a text and trick the detection system into thinking that it was not abductive hence, the system misclassifies them because they could classify most of them as shown in CITATION PuJ30 \l 1033 (Pu, et al., 2023) . The limitation in (Guo, et al., 2023) is the high difficulty of measuring the LLM generated text using logistic regression. This was difficult because the content generated by LLMs was complex and confusing therefore, understanding requires profound insights into critical factors that influence performance and the ability to adapt to these factors in various contexts. These challenges necessitate ultra-efficient detection methods that can adapt to the growing capabilities of LLMs. 2.2 Text detection algorithms 2.2.1 Double stream models OUTFOX CITATION Koi24 \l 1033 (Koike, et al., 2024) frame has been used to improve the detection by having detector and attacker model and iteratively learn from each other’s outputs through in-context learning. The detector uses human-generated text (essays), attacker-generates texts, and regular LLM’s generated texts as in-context examples to learn to detect the attacker’s generated texts. The attacker used the detector’s outputs to learn adversarial to generated text which was not easily detected by the detector. They constructed a dataset of 15,400 triplets of essay prompts, human-written essays, and LLM-generated essays to train and evaluate the model. In recent work, (Abburi, et al., 2023) investigated the task of detecting AI-generate text and attributed it to the large language model. They proposed an ensemble that involved passing input text through pre-trained large language models like BERT, RoBERTs, and DeBERTa, etc., and using the output probabilities from the model as features input for traditional machine learning models for binary classifiers such as SVM and Logistic regression. For the detection task, the ensemble approach achieved a micro F1score of 0.733 on English data and 0.649 on Spanish data. The approach performed well on the multiclass task of attributing LLM-generated text to one of six language models, ranking 1st in both micro F1 0.625 in English and micro F1 0.653 in Spanish. 2.2.2 Multi feature detection The concept of multi-feature detection in CITATION WuZ23 \l 1033 (Wu, et al., 2023) , was a complex approach to improve various fields in the accuracy and reliability of the zero-shot detection model. Multi-feature detection model integrates various data attributes and multiple sources using advanced algorithms to analyse and find complex patterns. The zero-shot model used different mathematics approaches like log-likelihood, log-rank, and entropy for better interpretation. This approach was particularly favourable in scenarios where a single feature model may fail or not be enough, such as in a medical field where patients have multiple points and must be considered or in Cybersecurity, where a wide range of indicators need to be analysed to find the thread. 2.2.3 Adversarial learning RADAR (Hu, et al., 2023) aims to identify machine-generated text using a method called learning with adversarial networks (GANs). As a classifier RADAR utilizes a diversity-enhancing GAN that employs a unidirectional LSTM as the discriminator. This unique approach incorporates rewards at both the word and sentence levels during training treating text generation as a decision-making process. It considers the generated text as the state and the next word to be generated as an action utilizing Monte Carlo searches to gather feedback from the discriminator. Conversely, OUTFOX (Koike, et al., 2024) enhances detection by employing both a detector and an attacker model that learn from each other through in-context learning. The detector learns to distinguish between human-generated texts (essays) created by attackers and regular language model-generated texts using examples. In contrast, the attacker leverages feedback, from the detector to create text that evades detection. The authors (Mitchell, et al., 2023) introduced a novel method to detect LLM’s generated text. The authors found a distinctive property of LLM’s probability function to detect. LLM-generated text from the sample generally made negative curvature with the model’s log probability function and they used this insight, to develop a curvature base criterion called “DetectGPT”. It did not require any separate classifier or training dataset to build the model. It only used the log probability function to compute from another pre-trained language model and the result showed that DetectGPT performed better than the existing Zero-shot model. DetectGPT achieved 0.95 AUROC compared to Zero-shot 0.81 AUROC. 2.2.4 Comparative analysis of different multilingual models A recent study by (Orenstrakh, et al., 2023) explored a method to detect the Large language model in the context of academic integrity. For this, the author compiled a dataset of student text along with ChatGPT-generated text and used datasets with widespread powerful large language models like CopyLeaks, GPTKit, GLTR, and GPTZero to do comparative analysis for detection. The most accurate detector was CopyLeaks shown in Table 1 which had the highest precision in identifying the LLM-generated text. GPTKit was for effectively reduced the false positive and GLTR was the most resilient one. However, the study also highlighted concerns regarding the high false positive rates of GPTZero as well as the reduced accuracy of all detectors when faced with code, non-English content, and paraphrased submissions. Table (I): Overall accuracy of LLM – generated text detectors Detectors Human Data ChatGPT Data CopyLeaks 99.12% 95.00% GPT2Detector 98.25% 95.00% CheckForAI 98.25% 95.00% GLTR 82.46% 95.00% GPTKit 100.00% 75.00% OriginalityAI 93.86% 70.00% AI Text Classifier 94.74% 60.00% GPTZero 54.39% 45.00% 3 Methodology and Implementation 3.1 Data Sourcing and Data Statistics In our quest for data robustness and diversity, we strived to encompass a broad spectrum of domains and generation styles by sourcing data from an array of benchmark datasets. Opinion Statements: Data was collected from Yelp reviews dataset. Question Answering: Hello-SimpleAI/HC3 question answering data was also used during model training and evaluation. The dataset contains more than 24K data points, containing questions and answers from the domain, ‘reddit-sli5’, ‘finance’, ’medicine’, ‘open-qa’ ‘wiki-csai’. Opinion Statements: The dataset incorporated 804 opinion statements from the Reddit subcommunity /r/ChangeMyView (CMV) as studied by (Tan, et al., 2016), and 1,000 reviews from the Yelp reviews dataset as analysed by (Zhang, et al., 2015 ). News Article: For news article writing, it comprised 1,000 news articles from XSum as per (Narayan, et al., October 31 - November 4, 2018), and 777 news articles from TLDR_news. Story Generation: For story generation, the dataset contained 1,000 human-written stories based on prompts from the Reddit WritingPrompts (WP) dataset as per (Fan, et al., July 15–20, 2018), and 1,000 stories from ROCStories Corpora (ROC) as per (Mostafazadeh, et al., June 12–17, 2016) . Reasoning: In the commonsense reasoning category, it encompassed 1,000 sentence sets for reasoning from HellaSwag as per (Anon., July 28- August 2, 2019). Knowledge Illustration: For knowledge illustration, it included 1,000 Wikipedia paragraphs from SQuAD contexts as per (Rajpurkar, et al., November 1–4,2016). Scientific Writing: Lastly, for scientific writing, it included 1,000 abstracts of scientific articles from SciGen as per (Moosavi, et al., 2016). Model is based on more than 500K + data collected from above mentioned sources. The data distribution can be shown using the following pie chart, Table (II): Distribution of data on the bases of length of text Data Source Human Written LLM Generated All Average Document Length 225.06 224.31 224.58 Average Sentence Length 15.7 17.5 16.84 Average #sentence per document 14.84 14.59 14.68 3.2 Model Intuition and New Approach, ‘perc-DETECTLLM’ 3.2.1 Model Intuition - perc-DETECTLLM ’s approach to distinguishing between machine-generated and human-written text is particularly relevant in today’s digital age, where AI language models are increasingly used to generate text. This could range from creating content for websites, drafting emails, to even generating news articles. As such, the ability to discern the origin of the text becomes crucial in various contexts, including but not limited to, academic integrity, content credibility, and information security. The perturbation function, a key component of perc-DETECTLLM , is designed to introduce subtle changes to the text while preserving its overall meaning. This is a non-trivial task as it requires a careful balance between making meaningful modifications and maintaining the original intent of the text. The choice of perturbation function can significantly impact the performance of DetectGPT, making it an important area for further research and optimization. The Perturbation Discrepancy Gap Hypothesis, which forms the theoretical foundation of DetectGPT, provides an interesting insight into the characteristics of machine-generated text. It suggests that machine-generated text is more sensitive to perturbations, resulting in larger discrepancies in log probabilities. This characteristic can be attributed to the fact that language models, when generating text, optimize for high-probability outputs, which often lie in areas of negative curvature in the log probability function. perc-DETECTLLM ’s thresholding mechanism is another critical aspect that requires careful calibration. The threshold needs to be set at a level that maximizes the accuracy of distinguishing between machine-generated and human-written text. It is worth noting that this threshold may vary across different language models and datasets, necessitating a dynamic thresholding strategy. perc-DETECTLLM represents a significant advancement in the field of AI language models. By providing a robust and scalable method to distinguish between machine-generated and human-written text, it opens up new possibilities for understanding and regulating the use of AI in text generation. However, like all models, DetectGPT is not without its limitations and areas for improvement, which present exciting opportunities for future research. 3.2.2 Model Methodology – Data Manipulation The first step in our process involved manipulating the data to create a mixed dataset of human-written and machine-generated text. We achieved this by randomly selecting segments within human-written sentences and replacing them with corresponding segments generated by Language Models (LLMs). This process simulates a real-world scenario where human-written content might be interspersed with machine-generated text. Calculation of LLM Percentage Data ( llm_perc_data ) : Once we have our mixed dataset, we calculated a metric called LLM percentage data for each sentence. This metric quantifies the proportion of each sentence that has been generated by an LLM. It is calculated as the ratio of the word count of the LLM-generated segment to the total word count of the sentence: $$llm-percentage-data=\frac{len\left(wordcountofLLM-generated\right)}{len\left(wordcountofthesentence\right)}$$ This gives us a continuous value between 0 and 1, where 0 indicates no LLM-generated content and 1 indicates that the entire sentence is LLM-generated. Utilization of DetectGPT Model Next, we used the DetectGPT model to independently assess the likelihood of each sentence being machine-generated. DetectGPT provided an estimate, also a continuous value between 0 and 1, of how likely it is that a given sentence has been generated by an LLM. Bayesian Model Evaluation Finally, we combined the LLM percentage data and the DetectGPT assessment using Bayesian inference to estimate the overall percentage of LLM-generated text in our dataset. Bayesian inference allowed us to update our prior beliefs about the distribution of LLM-generated text based on the observed data. The Bayesian update was performed using Bayes’ theorem : $$P\left(\right\{LLM-generated\left\} \right| \left\{data\right\}) = \frac{\left\{P\right(\left\{data\right\} \left| \right\{LLM-generated\left\}\right) \times P\left(\right\{LLM-generated\left\}\right)\}}{P\left(\right\{data\left\}\right)}$$ Notation : \(P\left(\right\{LLM-generated\left\} \right| \left\{data\right\})\) is the posterior probability, or our updated belief about the proportion of LLM-generated text after observing the data. \(P\left(\right\{data\left\} \right| \{LLM-generated\})\) is the likelihood, or the probability of observing our data given that a certain proportion of it is LLM-generated. \(P\left(\right\{LLM-generated\left\}\right)\) is the prior probability, or our initial belief about the proportion of LLM-generated text before observing the data. \(P\left(\right\{data\left\}\right)\) is the evidence, or the total probability of observing our data under all possible proportions of LLM-generated text. 4 Results When compared to LLM-generated text, the perc-DETECTLLM model's evaluation produced encouraging results in terms of text classification. With an accuracy of 72.1%, the model demonstrated that it could accurately classify occurrences of both categories of text. Precision was found to be 76.0%, which is the percentage of correctly detected LLM-generated text among all text classified as LLM-generated. This measure shows how well the model can reduce false positives, meaning that the content that is classified as being generated by language models is, in fact, generated by language models. In addition, the recall metric—which measures the percentage of accurately recognized LLM-generated text across all instances of LLM-generated text—was 81.0%. This high recall rate suggests that a considerable amount of the LLM-generated text in the dataset was successfully captured by the model. Tabel 3 shows that zero shot – bayes model has second best accuracy with best recall among other existing models, which provides best overall performance for LLM detection. This is a promising result and could act as a reference in model decision-making for researchers and manufacturers. Table (III): Performance of Large Language Model generated text detection models. Model Accuracy Precision Recall deberta-large 0.620 0.783 0.610 xlm-r-100langs-bert-base-nli-stsb-mean-tokens 0.647 0.782 0.639 roberta-base-openai-detector 0.679 0.805 0.671 xlm-roberta-large-xnli-anli 0.618 0.782 0.608 roberta-large 0.623 0.784 0.613 Ensemble with Voting classifier (𝑃c as input feature) 0.751 0.826 0.745 perc-DETECTLLM 0.721 0.760 0.810 5 Findings and discussion With respect to current techniques for identifying text created by LLM, the perc-DETECTLLM model offers a number of noteworthy benefits. Contrary to rule-based methods(Wu, et al., 2023) that depend on pre-established patterns(Hu, et al., 2023 ) or language characteristics (Junchao and Yang, et al., 2023), perc-DETECTLLM uses a data-driven strategy to improve accuracy and dependability by fusing machine learning with Bayesian inference. This addresses the inherent complexity and diversity in LLM-generated text as discussed by (Khalil, et al., 2023), by allowing the model to dynamically adapt to various language models and datasets. Additionally, perc-DETECTLLM takes into account semantic coherence and contextual relevance, identifying subtle subtleties indicative of machine-generated text, whereas other previous techniques(Koike, et al., 2024 ) (Abburi, et al., 2023), just concentrate on surface-level traits(Chen, et al., 2023) or statistical anomalies (Orenstrakh, et al., 2023). Furthermore, by adding a perturbation function, perc-DETECTLLM may replicate actual fluctuations in LLM-generated material, simulating the difficulties encountered in real-world situations systems(Khalil, et al., 2023) (Guo, et al., 2023). Thorough approach and strong performance of perc-DETECTLLM position it as a promising advancement in the field of LLM-generated text detection. It can be seen in Fig. 3 that models currently in use (Wu, et al., 2023)(Orenstrakh, et al., 2023) perform differently on different parameters, some reach comparable accuracy (Chen, et al., 2023) and recall (Abburi, et al., 2023) levels, while others don't measure up. As an illustration, the accuracy rates of DeBERTa-Large (Abburi, et al., 2023) , XLM-R-100langs-BERT-base-nli-stsb-mean-tokens (Abburi, et al., 2023) , and Roberta-Base-OpenAI-Detector (Abburi, et al., 2023) range from 61.8–67.9%, whereas the recall rates fall between 60.8% and 67.1%. While these models perform rather well, perc-DETECTLLM outperforms them in terms of accuracy and recall. Conversely, ensemble techniques (Wu, et al., 2023) - especially those that use voting classifiers—achieve higher accuracy rates; the highest, 75.1%, is documented. These models' marginally lower precision scores as compared to perc-DETECTLLM, however, suggest that they might forgo precision in favour of enhanced recall (Pu, et al., 2023) . 6 Conclusions In this work, we constructed zero shot and bayes model for Large Language Model generated text detection, by collecting texts from various writing tasks and Gemini API for LLM text. When it comes to detecting machine-generated texts, human annotators are only marginally more accurate than random guessing. The challenge of detecting deepfake text was illustrated by empirical findings on widely used detection techniques. We found that ensemble methods were the best-performing detection methods across all existing models. This work extended the prior existing work on binary classification of LLM generated text to percentage of LLM generated text by employing the Bayesian concept. The perc-DETECTLLM model presents a viable approach to address the increasing demand for textual content authentication in the current digital era. perc-DETECTLLM makes a significant contribution to the development of AI-driven text analysis and guarantees the authenticity and integrity of textual information in a variety of settings. The present research formed foundation for other researchers in generative AI as our constructed database is robust and can be employed in different fields. Limitations : The model does have certain drawbacks and difficulties, though. Variations in language models and datasets require additional optimization of perturbation functions and dynamic thresholding procedures. Furthermore, language models that closely resemble human writing or excessively edited text may have an impact on the model's performance. The perc-DETECTLLM model presents a viable approach to address the increasing demand for textual content authentication in the current digital era. To realize the full potential of the model, more study is needed to improve its performance and investigate its use in practical situations. perc-DETECTLLM makes a significant contribution to the development of AI-driven text analysis and guarantees the authenticity and integrity of textual information in a variety of settings. 7 Future Work The percentage of LLM-generated text that is included as a label in the dataset that was produced by manipulating pre-existing datasets has enormous potential for use in future studies and applications. Researchers can train and assess models for LLM-generated text recognition as well as gain a greater knowledge of the subtleties and features of machine-generated content across a range of fields by annotating the dataset with this vital information. This annotated dataset is an invaluable tool for comparing and benchmarking various detection techniques, facilitating methodical assessments and progress in the field. Moreover, fine-grained analysis is made possible by the labelling of the LLM-generated text percentage, which provides insight into the presence and distribution of machine-generated content across various textual corpora and communication ecosystems. Future research endeavours may investigate novel detection strategies that utilize deep learning and sophisticated machine learning techniques, in addition to utilizing the annotated dataset. Investigating adversarial training techniques, in which models are trained on data that has been intentionally disturbed, is one possible direction. The goal is to improve robustness against highly skilled adversaries who are trying to avoid detection. A different approach involves combining linguistic and contextual clues, like syntactic and semantic characteristics, to identify nuances typical of text produced by language learning models. Furthermore, developing best practices, laws, and ethical standards guiding the proper application and deployment of LLM-generated text detection systems requires cooperation between researchers, industry stakeholders, legislators, and civil society organisations. Researchers should minimise the negative effects and optimise the positive effects of LLM-generated text detection tools on digital communication and information ecosystems by proactively addressing these ethical and societal issues. Declarations No funding was received for conducting this study. All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript Author Contribution All authors contributed equally to every aspect of the research project, from conceptualization to manuscript preparation, ensuring collaborative and balanced effort throughout the entire process. References Abburi, et al., 2023. Generative ai text classification using ensemble llm approaches. arXiv preprint arXiv:2309.07755. Anon., July 28- August 2, 2019. Hellaswag: Can a machine really finish your sentence?. s.l., Association for Computational Linguistics, p. 4791–4800. Chaka & Chaka, 2023. Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: The case of five AI content detection tools. Journal of Applied Learning and Teaching, Volume 6. Chen, Canyu and Shu & Kai, 2023. Can llm-generated misinformation be detected?. arXiv preprint arXiv:2309.13788. Chen, et al., 2023. Hallucination detection: Robustly discerning reliable answers in large language models. s.l., s.n., pp. 245--255. Fan, A., Lewis, M. & Dauphin, Y. N., July 15-20, 2018. Hierarchical neural story generation. s.l., Association for Computational Linguistics., p. 889– 898. Guo, et al., 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597. Huo, Siqing and Arabzadeh, Negar and Clarke & Charles, 2023. Retrieving supporting evidence for generative question answering. s.l., s.n., pp. 11--20. Hu, Xiaomeng and Chen, Pin-Yu and Ho & Tsung-Yi, 2023. RADAR: Robust AI-Text Detection via Adversarial Learning. In: A. Oh and T. Neumann, et al. eds. Advances in Neural Information Processing Systems. s.l.:Curran Associates, Inc., pp. 15077--15095. Junchao and Yang, et al., 2023. A survey on llm-gernerated text detection: Necessity, methods, and future directions. arXiv preprint arXiv:2310.14724. Khalil, Mohammad and Er & Erkan, 2023. Will ChatGPT G et You Caught? Rethinking of Plagiarism Detection. s.l., International Conference on Human-Computer Interaction. Cham: Springer Nature Switzerland., pp. 475--487. Koike, Ryuto and Kaneko, Masahiro and Okazaki & Naoaki, 2024. Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples. s.l., s.n., pp. 21258--21266. Mitchell, et al., 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. s.l.:PMLR. Moosavi, et al., 2016. Learning to reason for text generation from scientific tables. arXiv preprint arXiv:2104.08296. Mostafazadeh, N. et al., June 12-17, 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. s.l., The Association for Computational Linguistics, p. 839–849. Narayan, Shashi and Cohen, Shay B and Lapata & Mirella, October 31 - November 4, 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. s.l., Association for Computational Linguistics, p. 1797–1807. Orenstrakh, et al., 2023. Detecting llm-generated text in computing education: A comparative study for chatgpt cases. Pu, et al., 2023. Deepfake text detection: Limitations and opportunities. s.l., IEEE, pp. 1613--1630. Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P., November 1-4,2016. Squad: 100, 000+ questions for machine comprehension of text. s.l., The Association for Computational Linguistics, p. 2383–2392. Saeidnia & Hamid, R., 2023. Welcome to the Gemini era: Google DeepMind and the information industry. 26 December. Sullivan, Miriam and Kelly, Andrew and McLaughlan & Paul, 2023. ChatGPT in higher education: Considerations for academic integrity and student learning. Journal of Applied Learning & Teaching. Tan, et al., 2016. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. s.l., s.n., pp. 613--624. Tang, Ruixiang and Chuang, Yu-Neng and Hu & Xia, 2023. The science of detecting llm-generated texts. arXiv preprint arXiv:2303.07205. Venkatraman, Saranya and Uchendu, Adaku and Lee & Dongwon, 2023. Gpt-who: An information density-based machine-generated text detector. arXiv preprint arXiv:2310.06202. Wang, et al., 2024. LLM-Detector: Improving AI-Generated Chinese Text Detection with Open-Source LLM Instruction Tuning. arXiv preprint arXiv:2402.01158, p. 17. Wu, et al., 2023. A survey on llm-gernerated text detection: Necessity, methods, and future directions. arXiv preprint arXiv:2310.14724. Wu, Zhendong and Xiang & Hui, 2023. MFD: Multi-Feature Detection of LLM-Generated Text. Zhang, Xiang and Zhao, Junbo and LeCun & Yann, 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, Volume 28. Additional Declarations No competing interests reported. Supplementary Files Supplementaryfile.pdf Cite Share Download PDF Status: Under Review Version 1 posted Editor assigned by journal 02 May, 2024 Submission checks completed at journal 25 Apr, 2024 First submitted to journal 24 Apr, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4318347","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":295240055,"identity":"7bc761e3-8804-41d5-ab07-ca53dca5e2cd","order_by":0,"name":"Ankush Goel","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABJUlEQVRIie2RsUrEMBjHUwLXpTVrSqF9hZSCCB69V2kotMs9wIGDcYlL0fVEfIqDw7GlUJc+QMQDveUmB0d7HGhyUqfWw00wv+H/BZIf+fIFAI3mTzICwGBqAVWOLfS9Aw8q+0w9R5XiF0oVkqJTBkC3PHXaexBdmxcMWltIF0+XG/cNRD4w7V4Tr+ratRuQ3OQlg/bViC5XzTEuQBIweBT3XiMy7hocJERQqeQWXYrpSCowBtAifYYvFadVyvNaKZgu5tlGKueDChFpjW0OIiIM+ZZ3EhIcq8aqQSUQaXJqcxw7OWXlHYs9LKbhSUMeAj6geCINHls+niCzKl9edx8WmmdrMZud+Qg1vUo3OMpkFnIOXcP7D/uRyVfZHTim0Wg0/5JPpuxcuh0c1hMAAAAASUVORK5CYII=","orcid":"","institution":"Netaji Subhas University of Technology","correspondingAuthor":true,"prefix":"","firstName":"Ankush","middleName":"","lastName":"Goel","suffix":""},{"id":295240057,"identity":"e4d2636f-4b1c-4ca3-9123-5b36a2a59d48","order_by":1,"name":"Vineet Kumar","email":"","orcid":"","institution":"Netaji Subhas University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Vineet","middleName":"","lastName":"Kumar","suffix":""},{"id":295240059,"identity":"d9914af6-29be-49aa-b776-293d504840e6","order_by":2,"name":"Saibal Patra","email":"","orcid":"","institution":"Netaji Subhas University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Saibal","middleName":"","lastName":"Patra","suffix":""},{"id":295240063,"identity":"043e98aa-c67f-4b9c-905a-faee555715a9","order_by":3,"name":"Savita Yadav","email":"","orcid":"","institution":"Netaji Subhas University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Savita","middleName":"","lastName":"Yadav","suffix":""}],"badges":[],"createdAt":"2024-04-24 12:48:30","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4318347/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4318347/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":55573029,"identity":"6c849f3b-e163-4ca9-bcd8-0a132b0e4834","added_by":"auto","created_at":"2024-04-30 05:57:38","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":18900,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eDistribution of LLM generated data and Human generated data\u003c/em\u003e\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-4318347/v1/2d983b1e111b404a9aac1afb.png"},{"id":55573031,"identity":"83074ce6-4a8c-44a2-9057-d5df70904138","added_by":"auto","created_at":"2024-04-30 05:57:38","extension":"jpeg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":122272,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eThe workflow diagram of perc-DETECTLLM\u003c/em\u003e\u003c/p\u003e","description":"","filename":"floatimage2.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-4318347/v1/ecbd40188a2e1b667431ad38.jpeg"},{"id":55573030,"identity":"3a0cbb02-aeec-4020-91bf-ccadbc193af1","added_by":"auto","created_at":"2024-04-30 05:57:38","extension":"jpeg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":181285,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003ePerformance of Large Language Model generated text detection models.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"floatimage3.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-4318347/v1/4557f03b82fb55e662007764.jpeg"},{"id":55573032,"identity":"749fdbde-f424-4b88-8f3d-efc157f8be3c","added_by":"auto","created_at":"2024-04-30 05:57:39","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":45600,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eComparison of performance of Large Language Model generated text detection models.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-4318347/v1/54e5034d31664d9444d55f5b.png"},{"id":55573424,"identity":"949078c4-f645-457a-a6fd-c6e0d6b86e9a","added_by":"auto","created_at":"2024-04-30 06:05:44","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":745742,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4318347/v1/4bdebc75-8961-4303-869f-11d1addc291f.pdf"},{"id":55573028,"identity":"c30212b1-897d-480d-bc90-a14b53ef9f38","added_by":"auto","created_at":"2024-04-30 05:57:38","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":27941,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementaryfile.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4318347/v1/b54999509bdcc56539675edd.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Beyond Binary: A Multi-Class Approach to Large Language Model-Generated Text Percentage Estimation","fulltext":[{"header":"1 Introduction","content":"\u003cp\u003eThese recent years have belonged to AI with the advancement of generative AI models in computer vision to large language models (LLMs) in natural language processing (NLP) (Junchao and Yang, et al., 2023) which has sparked the need to detect these AI-generated work and human-generated. According to McKinsey, 22% of people surveyed now use AI in some form for their work. Artificial intelligence (AI) tools can be abused by users for immoral activities like spamming, creating fraudulent product reviews, plagiarism, fake news, and social engineering, all of which can have detrimental effects on society. \u0026nbsp;Numerous basic flaws have been included into news stories that AI has rewritten. Therefore, it\u0026apos;s important to make sure that these generative AI technologies are used responsibly. Much current research focuses on identifying AI-generated texts in order to facilitate this\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element:field-begin'\u003e\u003c/span\u003e\u003cspan lang=EN-US style='mso-ansi-language:EN-US'\u003e\u0026nbsp;CITATION Che231 \\l 1033 \u003c/span\u003e\u003cspan style='mso-element:field-separator'\u003e\u003c/span\u003e\u003c![endif]--\u003e\u003cspan lang=\"EN-US\"\u003e(Chen, et al., 2023)\u003c/span\u003e\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element:field-end'\u003e\u003c/span\u003e\u003c![endif]--\u003e.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eText generated by large language models (LLMs) poses many issues and with the rapid development of these LLMs it is becoming harder to distinguish between human and machine-generated work \u0026nbsp;(Chen, et al., 2023). It poses a threat to many industries like creative arts, education, and journalism. A good LLM text detection algorithm can be a great source of help for publication as well as to maintain authentication in research work.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eDetecting generated text produced by Large Language Models (LLMs) serves various purposes, driven by concerns related to misinformation, ethics, and security. The ability to detect the percentage of LLM content accurately in such conditions is paramount for several reasons:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e1. \u0026nbsp; \u0026nbsp; Misinformation and Fake News: Because LLMs can produce extremely convincing and contextually relevant content, it can be difficult to discern between factual and false information.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e2. \u0026nbsp; \u0026nbsp; Ethical Use of Technology: It is critical to ensure that LLMs are used responsibly. When automated text generation is being utilized unethically, for example, in the production of frauds, deepfake text, or fraudulent material, detection algorithms can be employed to help identify these cases.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e3. \u0026nbsp; \u0026nbsp; Protecting Against Cyber Threats: Phishing emails, social engineering communications, and other cyber threats can be created by malicious actors using LLMs.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e4. \u0026nbsp; \u0026nbsp; Preserving Authenticity in Creative Works: Preserving the authenticity of content created by humans is crucial in fields such as academics, journalism, and literature. Maintaining the integrity of artistic and intellectual works can be facilitated by detection tools.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe existing research work addressed the following knowledge gaps:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e1.\u0026nbsp; \u0026nbsp; \u0026nbsp;There is no existing model that gives a percentage of LLM-generated text based on the probabilistic approach used by the zero-shot model.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e2. \u0026nbsp; \u0026nbsp; The binary classification does not account for the padding of human-generated data with LLM-generated text.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e3. \u0026nbsp; \u0026nbsp; There is no existing any robust dataset which encounters a percentage of LLM-generated text as a label.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe proposed solutions bridge these existing shortcomings by:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e1. \u0026nbsp; \u0026nbsp; Created a model that gives a percentage of LLM-generated text based on the probabilistic approach used by the shot model.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e2. \u0026nbsp; \u0026nbsp; By knowing the amount of AI-generated text in the content, since binary classification does not account for padding of human-generated data with LLM-generated text.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e3. \u0026nbsp; \u0026nbsp; Created a robust dataset with the percentage of LLM-generated text as a label.\u003c/p\u003e\n\u003cp\u003eEarlier LLM detection was solely focused on paraphrasing which is not a very reliable method taking the rapid growth of the LLM model\u0026apos;s progress into account, outfox \u0026nbsp;(Koike, et al., 2024) was one first model that went beyond this and used two-way input stream to train a model which significantly improved the performance. Furthermore, this plagiarism is generated from various LLMs such as ChatGPT and Gemini, with each of them having its unique properties which are fundamentally similar, however, their training model is very different which makes it more challenging to detect LLM-generated text.\u003c/p\u003e\n\u003cp\u003eIt is important to note here that \u0026ldquo;rephrasing\u0026rdquo; is not the only way used to get away with LLM-generated text \u0026nbsp;(Venkatraman, et al., 2023), people also add some of their text in the work to evade AI detection methods like waterfall, zero-shot (Mitchell, et al., 2023) etc. To counter such practice there is a need for detection of the percentage of LLM content in the given text or string, existing algorithms in this field can binary classify the given text but perform poorly when it comes to finding the percentage of LLM content in the text. Further, there does not exist any good dataset with a percentage label of LLM.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTo address the above research objectives and questions, firstly, a literature review is undertaken to furnish an overview of LLM-generated text detection, their limitations, and opportunities, as outlined in Section 2. Relevant research methodologies are discussed in Section 3, followed by an investigation of the performance of current state-of-the-art deep learning models for LLM-generated text detection in Section 4. The model pertinent discussion is done in Section 5. Finally, novelty of this research includes proposing using the Bayesian model along with the zero-shot model to enhance the zero-shot model\u0026rsquo;s probabilistic approach to achieve a better level of performance and evidence-based recommendations are presented in Sections 6 and 7. The findings are promising, and they will lead to more studies in the future, relating to various detection algorithms for text generated by LLMs.\u003c/p\u003e"},{"header":"2 Literature review","content":"\u003cp\u003e\u003cstrong\u003e2.1 Overview of LLM models\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eGenerative Pre-trained Transformers, which are known as GPTs, are a series of multimodal which have advanced forms of Artificial Intelligence that are designed to understand and generate responses as Humans do. The model has been used in various Domains like translation, content writing, chatbots, image generation, and video generation.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.1.1 GPT based models\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe detection of fake content or misinformation is created by large language models like GPT. The paper \u0026nbsp;\u003cstrong\u003e(Chen, et al., 2023)\u003c/strong\u003e discusses whether AI-generated misinformation or fake news was more harmful than human-generated misinformation written by humans. It has been seen that LLM-generated misinformation is more the difficult to detect and investigate. The study also found that AI-generated misinformation was more deceptive and difficult to detect, posing a significant challenge to tackle online safety and public trust websites. \u0026ldquo;GPT-who\u0026rdquo;\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element:field-begin'\u003e\u003c/span\u003e\u003cspan lang=EN-US style='mso-ansi-language: EN-US'\u003e\u0026nbsp;CITATION Ven02 \\l 1033 \u003c/span\u003e\u003cspan style='mso-element:field-separator'\u003e\u003c/span\u003e\u003c![endif]--\u003e\u003cstrong\u003e\u003cspan lang=\"EN-US\"\u003e(Venkatraman, et al., 2023)\u003c/span\u003e\u003c/strong\u003e\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element:field-end'\u003e\u003c/span\u003e\u003c![endif]--\u003e, an LLM-generated text detector based on Uniform information Density (UID), suggested that human generally distribute their information or knowledge evenly during their communication. This concept is used in \u0026ldquo;GPT-who\u0026rdquo; to detect the difference between human-generated and AI-generated. The paper claims that it performs better than another state-of-the-art detecting model by around 20% on many performance metrics and provides more interpretable representations of the text.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.1.2 Gemini model \u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eGemini is an advanced chatbot model built by Google DeepMind. The author \u003cstrong\u003e(Saeidnia \u0026amp; Hamid, 2023)\u003c/strong\u003e examined the motivation behind the development of Google\u0026rsquo;s chatbot Gemini and its potential impact on IT industry. Gemini is designed to provide a better user experience by personalized and relevant information and improving customer services. Model built to transform how users access the internet and information and interact with it. We used Gemini\u0026rsquo;s API to generate text and also used it for the paraphrasing of our dataset.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.1.3 Necessity of LLM generated text detection\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe impact of the LLM model on academic and student learning has been seen in higher education through a content analysis of 100 articles from Australia, New Zealand, the United States, and the United Kingdom \u0026nbsp;\u003cstrong\u003e(Sullivan, et al., 2023)\u003c/strong\u003e. It explores various benefits and risks associated with using generative AI tools and the need to change teaching style. The assignment may need to adapt to this new reality as the performance of ChatGPT compared to humans in various expert domains is getting better day by day shown in \u0026nbsp;\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element:field-begin'\u003e\u003c/span\u003e\u003cspan lang=EN-US style='mso-ansi-language: EN-US'\u003e\u0026nbsp;CITATION Guo231 \\l 1033 \u003c/span\u003e\u003cspan style='mso-element:field-separator'\u003e\u003c/span\u003e\u003c![endif]--\u003e\u003cstrong\u003e\u003cspan lang=\"EN-US\"\u003e(Guo, et al., 2023)\u003c/span\u003e\u003c/strong\u003e\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element:field-end'\u003e\u003c/span\u003e\u003c![endif]--\u003e. The study \u0026nbsp;\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element: field-begin'\u003e\u003c/span\u003e\u003cspan lang=EN-US style='mso-ansi-language:EN-US'\u003e\u0026nbsp;CITATION Kha23 \\l 1033 \u003c/span\u003e\u003cspan style='mso-element:field-separator'\u003e\u003c/span\u003e\u003c![endif]--\u003e\u003cstrong\u003e\u003cspan lang=\"EN-US\"\u003e(Khalil, et al., 2023)\u003c/span\u003e\u003c/strong\u003e\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element:field-end'\u003e\u003c/span\u003e\u003c![endif]--\u003eevaluates LLMs\u0026rsquo; ability to understand the theory of mind task. It suggests that as LLMs improve, their language skills may lead to the emergence of human-like cognitive abilities as discussed by (Junchao and Yang, et al., 2023) , emphasize the importance of building detectors for LLM-generated text from potential misuse and protecting different domains like artistic expression, social network. It discusses recent advancements in detection techniques and the challenges faced, such as out-of-distribution problems and data ambiguity. It is not only necessary to detect generated text but also to detect hallucinations or unreliable answers in LLMs. \u0026nbsp;\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element:field-begin'\u003e\u003c/span\u003e\u003cspan lang=EN-US style='mso-ansi-language: EN-US'\u003e\u0026nbsp;CITATION Hal23 \\l 1033 \u003c/span\u003e\u003cspan style='mso-element:field-separator'\u003e\u003c/span\u003e\u003c![endif]--\u003e\u003cstrong\u003e\u003cspan lang=\"EN-US\"\u003e(Chen, et al., 2023)\u003c/span\u003e\u003c/strong\u003e\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element:field-end'\u003e\u003c/span\u003e\u003c![endif]--\u003e and \u003cstrong\u003e(Guo, et al., 2023)\u003c/strong\u003e talk about how LLM could be deceived to generate false information while forcing on that LLM can also make statistical mistakes and how can they prove to be costly in many fields. The need for a robust model to discern reliable information provided by the LLM model is critical for their safe and effective use.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.1.4 Challenges in detection of LLM generated text\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIdentifying text written by Large Language Models provided several challenges for researchers and practitioners. First, the out-of-distribution problem results in false negatives because detection models cannot identify a novel text, meaning a text that was not part of their training data. In addition, the text could be deceptive in style \u003cstrong\u003e(Chen, et al., 2023)\u003c/strong\u003esimilar to human prose, which makes the machine-based system and humans difficult to detect that the text was written by the machine. Ultimately, this becomes more challenging and deceiving if production of misinformation is produced by LLM.\u003c/p\u003e\n\u003cp\u003eAnother challenge, and restriction was the ineffectiveness of traditional plagiarism systems \u003cstrong\u003e(Khalil, et al., 2023)\u003c/strong\u003e they were unable to detect LLM-generated materials because these were new and original. In this case, adversaries could be detecting a mistake in a text and trick the detection system into thinking that it was not abductive hence, the system misclassifies them because they could classify most of them as shown in \u0026nbsp;\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element: field-begin'\u003e\u003c/span\u003e\u003cspan lang=EN-US style='mso-ansi-language:EN-US'\u003e\u0026nbsp;CITATION PuJ30 \\l 1033 \u003c/span\u003e\u003cspan style='mso-element:field-separator'\u003e\u003c/span\u003e\u003c![endif]--\u003e\u003cstrong\u003e\u003cspan lang=\"EN-US\"\u003e(Pu, et al., 2023)\u003c/span\u003e\u003c/strong\u003e\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element:field-end'\u003e\u003c/span\u003e\u003c![endif]--\u003e. The limitation in \u003cstrong\u003e(Guo, et al., 2023)\u003c/strong\u003e is the high difficulty of measuring the LLM generated text using logistic regression. This was difficult because the content generated by LLMs was complex and confusing therefore, understanding requires profound insights into critical factors that influence performance and the ability to adapt to these factors in various contexts. These challenges necessitate ultra-efficient detection methods that can adapt to the growing capabilities of LLMs.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.2 Text detection algorithms\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.2.1 Double stream models\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOUTFOX \u0026nbsp;\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element: field-begin'\u003e\u003c/span\u003e\u003cspan lang=EN-US style='mso-ansi-language:EN-US'\u003e\u0026nbsp;CITATION Koi24 \\l 1033 \u003c/span\u003e\u003cspan style='mso-element:field-separator'\u003e\u003c/span\u003e\u003c![endif]--\u003e\u003cstrong\u003e\u003cspan lang=\"EN-US\"\u003e(Koike, et al., 2024)\u003c/span\u003e\u003c/strong\u003e\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element:field-end'\u003e\u003c/span\u003e\u003c![endif]--\u003e frame has been used to improve the detection by having detector and attacker model and iteratively learn from each other\u0026rsquo;s outputs through in-context learning. The detector uses human-generated text (essays), attacker-generates texts, and regular LLM\u0026rsquo;s generated texts as in-context examples to learn to detect the attacker\u0026rsquo;s generated texts. The attacker used the detector\u0026rsquo;s outputs to learn adversarial to generated text which was not easily detected by the detector. They constructed a dataset of 15,400 triplets of essay prompts, human-written essays, and LLM-generated essays to train and evaluate the model.\u003c/p\u003e\n\u003cp\u003eIn recent work, \u003cstrong\u003e(Abburi, et al., 2023)\u003c/strong\u003e investigated the task of detecting AI-generate text and attributed it to the large language model. They proposed an ensemble that involved passing input text through pre-trained large language models like BERT, RoBERTs, and DeBERTa, etc., and using the output probabilities from the model as features input for traditional machine learning models for binary classifiers such as SVM and Logistic regression. For the detection task, the ensemble approach achieved a micro F1score of 0.733 on English data and 0.649 on Spanish data. The approach performed well on the multiclass task of attributing LLM-generated text to one of six language models, ranking 1st in both micro F1 0.625 in English and micro F1 0.653 in Spanish.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.2.2 \u0026nbsp;Multi feature detection\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe concept of multi-feature detection in \u0026nbsp;\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element:field-begin'\u003e\u003c/span\u003e\u003cspan lang=EN-US style='mso-ansi-language: EN-US'\u003e\u0026nbsp;CITATION WuZ23 \\l 1033 \u003c/span\u003e\u003cspan style='mso-element:field-separator'\u003e\u003c/span\u003e\u003c![endif]--\u003e\u003cstrong\u003e\u003cspan lang=\"EN-US\"\u003e(Wu, et al., 2023)\u003c/span\u003e\u003c/strong\u003e\n \u003c!--[if supportFields]\u003e\u003cspan style='mso-element:field-end'\u003e\u003c/span\u003e\u003c![endif]--\u003e, was a complex approach to improve various fields in the accuracy and reliability of the zero-shot detection model. Multi-feature detection model integrates various data attributes and multiple sources using advanced algorithms to analyse and find complex patterns. The zero-shot model used different mathematics approaches like log-likelihood, log-rank, and entropy for better interpretation. This approach was particularly favourable in scenarios where a single feature model may fail or not be enough, such as in a medical field where patients have multiple points and must be considered or in Cybersecurity, where a wide range of indicators need to be analysed to find the thread.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.2.3 Adversarial learning\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eRADAR \u003cstrong\u003e(Hu, et al., 2023)\u003c/strong\u003eaims to identify machine-generated text using a method called learning with adversarial networks (GANs). As a classifier RADAR utilizes a diversity-enhancing GAN that employs a unidirectional LSTM as the discriminator. This unique approach incorporates rewards at both the word and sentence levels during training treating text generation as a decision-making process. It considers the generated text as the state and the next word to be generated as an action utilizing Monte Carlo searches to gather feedback from the discriminator. Conversely, OUTFOX \u003cstrong\u003e(Koike, et al., 2024)\u003c/strong\u003eenhances detection by employing both a detector and an attacker model that learn from each other through in-context learning. The detector learns to distinguish between human-generated texts (essays) created by attackers and regular language model-generated texts using examples. In contrast, the attacker leverages feedback, from the detector to create text that evades detection.\u003c/p\u003e\n\u003cp\u003eThe authors \u003cstrong\u003e(Mitchell, et al., 2023)\u003c/strong\u003e introduced a novel method to detect LLM\u0026rsquo;s generated text. The authors found a distinctive property of LLM\u0026rsquo;s probability function to detect. LLM-generated text from the sample generally made negative curvature with the model\u0026rsquo;s log probability function and they used this insight, to develop a curvature base criterion called \u0026ldquo;DetectGPT\u0026rdquo;. It did not require any separate classifier or training dataset to build the model. It only used the log probability function to compute from another pre-trained language model and the result showed that DetectGPT performed better than the existing Zero-shot model. DetectGPT achieved 0.95 AUROC compared to Zero-shot 0.81 AUROC.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.2.4 Comparative analysis of different multilingual models\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eA recent study by \u003cstrong\u003e(Orenstrakh, et al., 2023)\u003c/strong\u003eexplored a method to detect the Large language model in the context of academic integrity. For this, the author compiled a dataset of student text along with ChatGPT-generated text and used datasets with widespread powerful large language models like CopyLeaks, GPTKit, GLTR, and GPTZero to do comparative analysis for detection. The most accurate detector was CopyLeaks shown in Table 1 which had the highest precision in identifying the LLM-generated text. GPTKit was for effectively reduced the false positive and GLTR was the most resilient one. However, the study also highlighted concerns regarding the high false positive rates of GPTZero as well as the reduced accuracy of all detectors when faced with code, non-English content, and paraphrased submissions.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eTable (I): Overall accuracy of LLM \u0026ndash; generated text detectors\u003c/em\u003e\u003c/p\u003e\n\u003cdiv\u003e\n \u003ctable style=\"border: none;\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eDetectors\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eHuman Data\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eChatGPT Data\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eCopyLeaks\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e99.12%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e95.00%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eGPT2Detector\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e98.25%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e95.00%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eCheckForAI\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e98.25%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e95.00%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eGLTR\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e82.46%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e95.00%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eGPTKit\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e100.00%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e75.00%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eOriginalityAI\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e93.86%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e70.00%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eAI Text Classifier\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e94.74%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e60.00%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd\u003e\n \u003cp\u003e\u003cstrong\u003eGPTZero\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e54.39%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd\u003e\n \u003cp\u003e45.00%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e"},{"header":"3 Methodology and Implementation","content":"\u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Data Sourcing and Data Statistics\u003c/h2\u003e \u003cp\u003eIn our quest for data robustness and diversity, we strived to encompass a broad spectrum of domains and generation styles by sourcing data from an array of benchmark datasets.\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eOpinion Statements: Data was collected from \u003cb\u003eYelp\u003c/b\u003e reviews dataset.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eQuestion Answering: Hello-SimpleAI/HC3 question answering data was also used during model training and evaluation. The dataset contains more than 24K data points, containing questions and answers from the domain, \u0026lsquo;reddit-sli5\u0026rsquo;, \u0026lsquo;finance\u0026rsquo;, \u0026rsquo;medicine\u0026rsquo;, \u0026lsquo;open-qa\u0026rsquo; \u0026lsquo;wiki-csai\u0026rsquo;.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eOpinion Statements: The dataset incorporated 804 opinion statements from the Reddit subcommunity /r/ChangeMyView (CMV) as studied by (Tan, et al., 2016), and 1,000 reviews from the Yelp reviews dataset as analysed by (Zhang, et al., \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2015\u003c/span\u003e).\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eNews Article: For news article writing, it comprised 1,000 news articles from XSum as per (Narayan, et al., October 31 - November 4, 2018), and 777 news articles from TLDR_news.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eStory Generation: For story generation, the dataset contained 1,000 human-written stories based on prompts from the Reddit WritingPrompts (WP) dataset as per (Fan, et al., July 15\u0026ndash;20, 2018), and 1,000 stories from ROCStories Corpora (ROC) as per (Mostafazadeh, et al., June 12\u0026ndash;17, 2016) .\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eReasoning: In the commonsense reasoning category, it encompassed 1,000 sentence sets for reasoning from HellaSwag as per (Anon., July 28- August 2, 2019).\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eKnowledge Illustration: For knowledge illustration, it included 1,000 Wikipedia paragraphs from SQuAD contexts as per (Rajpurkar, et al., November 1\u0026ndash;4,2016).\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eScientific Writing: Lastly, for scientific writing, it included 1,000 abstracts of scientific articles from SciGen as per (Moosavi, et al., 2016).\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003cdiv class=\"BlockQuote\"\u003e \u003cp\u003eModel is based on more than 500K\u0026thinsp;+\u0026thinsp;data collected from above mentioned sources. The data distribution can be shown using the following pie chart,\u003c/p\u003e \u003c/div\u003e \u003c/p\u003e\u003cp\u003e \u003cem\u003eTable (II): Distribution of data on the bases of length of text\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Tabb\" border=\"1\"\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eData Source\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHuman Written\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLLM Generated\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAll\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAverage Document Length\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e225.06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e224.31\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e224.58\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAverage Sentence Length\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e15.7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e17.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e16.84\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eAverage #sentence per document\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e14.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e14.59\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e14.68\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Model Intuition and New Approach, \u0026lsquo;perc-DETECTLLM\u0026rsquo;\u003c/h2\u003e \u003cdiv id=\"Sec18\" class=\"Section3\"\u003e \u003ch2\u003e3.2.1 Model Intuition -\u003c/h2\u003e \u003cp\u003e \u003cb\u003eperc-DETECTLLM\u003c/b\u003e\u0026rsquo;s approach to distinguishing between machine-generated and human-written text is particularly relevant in today\u0026rsquo;s digital age, where AI language models are increasingly used to generate text. This could range from creating content for websites, drafting emails, to even generating news articles. As such, the ability to discern the origin of the text becomes crucial in various contexts, including but not limited to, academic integrity, content credibility, and information security.\u003c/p\u003e \u003cp\u003eThe perturbation function, a key component of \u003cb\u003eperc-DETECTLLM\u003c/b\u003e, is designed to introduce subtle changes to the text while preserving its overall meaning. This is a non-trivial task as it requires a careful balance between making meaningful modifications and maintaining the original intent of the text. The choice of perturbation function can significantly impact the performance of DetectGPT, making it an important area for further research and optimization.\u003c/p\u003e \u003cp\u003eThe Perturbation Discrepancy Gap Hypothesis, which forms the theoretical foundation of DetectGPT, provides an interesting insight into the characteristics of machine-generated text. It suggests that machine-generated text is more sensitive to perturbations, resulting in larger discrepancies in log probabilities. This characteristic can be attributed to the fact that language models, when generating text, optimize for high-probability outputs, which often lie in areas of negative curvature in the log probability function.\u003c/p\u003e \u003cp\u003e \u003cb\u003eperc-DETECTLLM\u003c/b\u003e\u0026rsquo;s thresholding mechanism is another critical aspect that requires careful calibration. The threshold needs to be set at a level that maximizes the accuracy of distinguishing between machine-generated and human-written text. It is worth noting that this threshold may vary across different language models and datasets, necessitating a dynamic thresholding strategy.\u003c/p\u003e \u003cp\u003e \u003cb\u003eperc-DETECTLLM\u003c/b\u003e represents a significant advancement in the field of AI language models. By providing a robust and scalable method to distinguish between machine-generated and human-written text, it opens up new possibilities for understanding and regulating the use of AI in text generation. However, like all models, DetectGPT is not without its limitations and areas for improvement, which present exciting opportunities for future research.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec19\" class=\"Section3\"\u003e \u003ch2\u003e3.2.2 Model Methodology \u0026ndash;\u003c/h2\u003e \u003cp\u003e \u003cstrong\u003eData Manipulation\u003c/strong\u003e \u003cp\u003eThe first step in our process involved manipulating the data to create a mixed dataset of human-written and machine-generated text. We achieved this by randomly selecting segments within human-written sentences and replacing them with corresponding segments generated by Language Models (LLMs). This process simulates a real-world scenario where human-written content might be interspersed with machine-generated text.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cspan type=\"ItalicUnderline\" class=\"ItalicUnderline\" name=\"Emphasis\"\u003eCalculation of LLM Percentage Data\u003c/span\u003e \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e(\u003c/span\u003e\u003cspan type=\"ItalicUnderline\" class=\"ItalicUnderline\" name=\"Emphasis\"\u003ellm_perc_data\u003c/span\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e)\u003c/span\u003e: Once we have our mixed dataset, we calculated a metric called LLM percentage data for each sentence. This metric quantifies the proportion of each sentence that has been generated by an LLM. It is calculated as the ratio of the word count of the LLM-generated segment to the total word count of the sentence:\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e\n$$llm-percentage-data=\\frac{len\\left(wordcountofLLM-generated\\right)}{len\\left(wordcountofthesentence\\right)}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eThis gives us a continuous value between 0 and 1, where 0 indicates no LLM-generated content and 1 indicates that the entire sentence is LLM-generated.\u003c/p\u003e \u003cp\u003e \u003cstrong\u003eUtilization of DetectGPT Model\u003c/strong\u003e \u003cp\u003eNext, we used the DetectGPT model to independently assess the likelihood of each sentence being machine-generated. DetectGPT provided an estimate, also a continuous value between 0 and 1, of how likely it is that a given sentence has been generated by an LLM.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eBayesian Model Evaluation\u003c/strong\u003e \u003cp\u003eFinally, we combined the LLM percentage data and the DetectGPT assessment using Bayesian inference to estimate the overall percentage of LLM-generated text in our dataset. Bayesian inference allowed us to update our prior beliefs about the distribution of LLM-generated text based on the observed data.\u003c/p\u003e \u003c/p\u003e \u003cp\u003eThe Bayesian update was performed using \u003cem\u003eBayes\u0026rsquo; theorem\u003c/em\u003e:\u003cdiv id=\"Equb\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equb\" name=\"EquationSource\"\u003e\n$$P\\left(\\right\\{LLM-generated\\left\\} \\right| \\left\\{data\\right\\}) = \\frac{\\left\\{P\\right(\\left\\{data\\right\\} \\left| \\right\\{LLM-generated\\left\\}\\right) \\times P\\left(\\right\\{LLM-generated\\left\\}\\right)\\}}{P\\left(\\right\\{data\\left\\}\\right)}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003e \u003cem\u003eNotation\u003c/em\u003e:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cspan class=\"InlineEquation\"\u003e \u003cspan class=\"mathinline\"\u003e\\(P\\left(\\right\\{LLM-generated\\left\\} \\right| \\left\\{data\\right\\})\\)\u003c/span\u003e \u003c/span\u003eis the posterior probability, or our updated belief about the proportion of LLM-generated text after observing the data.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cspan class=\"InlineEquation\"\u003e \u003cspan class=\"mathinline\"\u003e\\(P\\left(\\right\\{data\\left\\} \\right| \\{LLM-generated\\})\\)\u003c/span\u003e \u003c/span\u003e is the likelihood, or the probability of observing our data given that a certain proportion of it is LLM-generated.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cspan class=\"InlineEquation\"\u003e \u003cspan class=\"mathinline\"\u003e\\(P\\left(\\right\\{LLM-generated\\left\\}\\right)\\)\u003c/span\u003e \u003c/span\u003e is the prior probability, or our initial belief about the proportion of LLM-generated text before observing the data.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cspan class=\"InlineEquation\"\u003e \u003cspan class=\"mathinline\"\u003e\\(P\\left(\\right\\{data\\left\\}\\right)\\)\u003c/span\u003e \u003c/span\u003e is the evidence, or the total probability of observing our data under all possible proportions of LLM-generated text.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"4 Results","content":"\u003cp\u003eWhen compared to LLM-generated text, the perc-DETECTLLM model's evaluation produced encouraging results in terms of text classification. With an accuracy of 72.1%, the model demonstrated that it could accurately classify occurrences of both categories of text. Precision was found to be 76.0%, which is the percentage of correctly detected LLM-generated text among all text classified as LLM-generated. This measure shows how well the model can reduce false positives, meaning that the content that is classified as being generated by language models is, in fact, generated by language models. In addition, the recall metric\u0026mdash;which measures the percentage of accurately recognized LLM-generated text across all instances of LLM-generated text\u0026mdash;was 81.0%. This high recall rate suggests that a considerable amount of the LLM-generated text in the dataset was successfully captured by the model.\u003c/p\u003e \u003cp\u003eTabel 3 shows that zero shot \u0026ndash; bayes model has second best accuracy with best recall among other existing models, which provides best overall performance for LLM detection. This is a promising result and could act as a reference in model decision-making for researchers and manufacturers.\u003c/p\u003e \u003cp\u003e \u003cem\u003eTable (III): Performance of Large Language Model generated text detection models.\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Tabc\" border=\"1\"\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePrecision\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003edeberta-large\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.620\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.783\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.610\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003exlm-r-100langs-bert-base-nli-stsb-mean-tokens\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.647\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.782\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.639\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eroberta-base-openai-detector\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.679\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.805\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.671\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003exlm-roberta-large-xnli-anli\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.618\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.782\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.608\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eroberta-large\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.623\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.784\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.613\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEnsemble with Voting classifier (\u0026#119875;c as input feature)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.751\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.826\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.745\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eperc-DETECTLLM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.721\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.760\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.810\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e"},{"header":"5 Findings and discussion","content":"\u003cp\u003eWith respect to current techniques for identifying text created by LLM, the perc-DETECTLLM model offers a number of noteworthy benefits. Contrary to rule-based methods(Wu, et al., 2023) that depend on pre-established patterns(Hu, et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) or language characteristics (Junchao and Yang, et al., 2023), perc-DETECTLLM uses a data-driven strategy to improve accuracy and dependability by fusing machine learning with Bayesian inference. This addresses the inherent complexity and diversity in LLM-generated text as discussed by (Khalil, et al., 2023), by allowing the model to dynamically adapt to various language models and datasets. Additionally, perc-DETECTLLM takes into account semantic coherence and contextual relevance, identifying subtle subtleties indicative of machine-generated text, whereas other previous techniques(Koike, et al., \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) (Abburi, et al., 2023), just concentrate on surface-level traits(Chen, et al., 2023) or statistical anomalies (Orenstrakh, et al., 2023). Furthermore, by adding a perturbation function, perc-DETECTLLM may replicate actual fluctuations in LLM-generated material, simulating the difficulties encountered in real-world situations systems(Khalil, et al., 2023) (Guo, et al., 2023). Thorough approach and strong performance of perc-DETECTLLM position it as a promising advancement in the field of LLM-generated text detection.\u003c/p\u003e \u003cp\u003eIt can be seen in Fig.\u0026nbsp;3 that models currently in use\u003cb\u003e(Wu, et al., 2023)(Orenstrakh, et al., 2023)\u003c/b\u003e perform differently on different parameters, some reach comparable accuracy\u003cb\u003e(Chen, et al., 2023)\u003c/b\u003e and recall\u003cb\u003e(Abburi, et al., 2023)\u003c/b\u003e levels, while others don't measure up. As an illustration, the accuracy rates of DeBERTa-Large \u003cb\u003e(Abburi, et al., 2023)\u003c/b\u003e, XLM-R-100langs-BERT-base-nli-stsb-mean-tokens \u003cb\u003e(Abburi, et al., 2023)\u003c/b\u003e, and Roberta-Base-OpenAI-Detector\u003cb\u003e(Abburi, et al., 2023)\u003c/b\u003e range from 61.8\u0026ndash;67.9%, whereas the recall rates fall between 60.8% and 67.1%. While these models perform rather well, perc-DETECTLLM outperforms them in terms of accuracy and recall. Conversely, ensemble techniques \u003cb\u003e(Wu, et al., 2023)\u003c/b\u003e- especially those that use voting classifiers\u0026mdash;achieve higher accuracy rates; the highest, 75.1%, is documented. These models' marginally lower precision scores as compared to perc-DETECTLLM, however, suggest that they might forgo precision in favour of enhanced recall \u003cb\u003e(Pu, et al., 2023)\u003c/b\u003e.\u003c/p\u003e "},{"header":"6 Conclusions","content":"\u003cp\u003eIn this work, we constructed zero shot and bayes model for Large Language Model generated text detection, by collecting texts from various writing tasks and Gemini API for LLM text. When it comes to detecting machine-generated texts, human annotators are only marginally more accurate than random guessing. The challenge of detecting deepfake text was illustrated by empirical findings on widely used detection techniques. We found that ensemble methods were the best-performing detection methods across all existing models.\u003c/p\u003e \u003cp\u003eThis work extended the prior existing work on binary classification of LLM generated text to percentage of LLM generated text by employing the Bayesian concept. The perc-DETECTLLM model presents a viable approach to address the increasing demand for textual content authentication in the current digital era. perc-DETECTLLM makes a significant contribution to the development of AI-driven text analysis and guarantees the authenticity and integrity of textual information in a variety of settings. The present research formed foundation for other researchers in generative AI as our constructed database is robust and can be employed in different fields.\u003c/p\u003e \u003cp\u003e \u003cb\u003eLimitations\u003c/b\u003e:\u003c/p\u003e \u003cp\u003eThe model does have certain drawbacks and difficulties, though. Variations in language models and datasets require additional optimization of perturbation functions and dynamic thresholding procedures. Furthermore, language models that closely resemble human writing or excessively edited text may have an impact on the model's performance. The perc-DETECTLLM model presents a viable approach to address the increasing demand for textual content authentication in the current digital era. To realize the full potential of the model, more study is needed to improve its performance and investigate its use in practical situations. perc-DETECTLLM makes a significant contribution to the development of AI-driven text analysis and guarantees the authenticity and integrity of textual information in a variety of settings.\u003c/p\u003e"},{"header":"7 Future Work","content":"\u003cp\u003eThe percentage of LLM-generated text that is included as a label in the dataset that was produced by manipulating pre-existing datasets has enormous potential for use in future studies and applications. Researchers can train and assess models for LLM-generated text recognition as well as gain a greater knowledge of the subtleties and features of machine-generated content across a range of fields by annotating the dataset with this vital information. This annotated dataset is an invaluable tool for comparing and benchmarking various detection techniques, facilitating methodical assessments and progress in the field. Moreover, fine-grained analysis is made possible by the labelling of the LLM-generated text percentage, which provides insight into the presence and distribution of machine-generated content across various textual corpora and communication ecosystems.\u003c/p\u003e \u003cp\u003eFuture research endeavours may investigate novel detection strategies that utilize deep learning and sophisticated machine learning techniques, in addition to utilizing the annotated dataset. Investigating adversarial training techniques, in which models are trained on data that has been intentionally disturbed, is one possible direction. The goal is to improve robustness against highly skilled adversaries who are trying to avoid detection. A different approach involves combining linguistic and contextual clues, like syntactic and semantic characteristics, to identify nuances typical of text produced by language learning models.\u003c/p\u003e \u003cp\u003eFurthermore, developing best practices, laws, and ethical standards guiding the proper application and deployment of LLM-generated text detection systems requires cooperation between researchers, industry stakeholders, legislators, and civil society organisations. Researchers should minimise the negative effects and optimise the positive effects of LLM-generated text detection tools on digital communication and information ecosystems by proactively addressing these ethical and societal issues.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eNo funding was received for conducting this study. All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript\u003c/p\u003e\n\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eAll authors contributed equally to every aspect of the research project, from conceptualization to manuscript preparation, ensuring collaborative and balanced effort throughout the entire process.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAbburi, et al., 2023. Generative ai text classification using ensemble llm approaches. \u003cem\u003earXiv preprint arXiv:2309.07755.\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003eAnon., July 28- August 2, 2019. \u003cem\u003eHellaswag: Can a machine really finish your sentence?. \u003c/em\u003es.l., Association for Computational Linguistics, p. 4791\u0026ndash;4800.\u003c/li\u003e\n\u003cli\u003eChaka \u0026amp; Chaka, 2023. Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: The case of five AI content detection tools. \u003cem\u003eJournal of Applied Learning and Teaching, \u003c/em\u003eVolume 6.\u003c/li\u003e\n\u003cli\u003eChen, Canyu and Shu \u0026amp; Kai, 2023. Can llm-generated misinformation be detected?. \u003cem\u003earXiv preprint arXiv:2309.13788.\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003eChen, et al., 2023. \u003cem\u003eHallucination detection: Robustly discerning reliable answers in large language models. \u003c/em\u003es.l., s.n., pp. 245--255.\u003c/li\u003e\n\u003cli\u003eFan, A., Lewis, M. \u0026amp; Dauphin, Y. N., July 15-20, 2018. \u003cem\u003eHierarchical neural story generation. \u003c/em\u003es.l., Association for Computational Linguistics., p. 889\u0026ndash; 898.\u003c/li\u003e\n\u003cli\u003eGuo, et al., 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. \u003cem\u003earXiv preprint arXiv:2301.07597.\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003eHuo, Siqing and Arabzadeh, Negar and Clarke \u0026amp; Charles, 2023. \u003cem\u003eRetrieving supporting evidence for generative question answering. \u003c/em\u003es.l., s.n., pp. 11--20.\u003c/li\u003e\n\u003cli\u003eHu, Xiaomeng and Chen, Pin-Yu and Ho \u0026amp; Tsung-Yi, 2023. RADAR: Robust AI-Text Detection via Adversarial Learning. In: A. Oh and T. Neumann, et al. eds. \u003cem\u003eAdvances in Neural Information Processing Systems. \u003c/em\u003es.l.:Curran Associates, Inc., pp. 15077--15095.\u003c/li\u003e\n\u003cli\u003eJunchao and Yang, et al., 2023. A survey on llm-gernerated text detection: Necessity, methods, and future directions. \u003cem\u003earXiv preprint arXiv:2310.14724.\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003eKhalil, Mohammad and Er \u0026amp; Erkan, 2023. \u003cem\u003eWill ChatGPT G et You Caught? Rethinking of Plagiarism Detection. \u003c/em\u003es.l., International Conference on Human-Computer Interaction. Cham: Springer Nature Switzerland., pp. 475--487.\u003c/li\u003e\n\u003cli\u003eKoike, Ryuto and Kaneko, Masahiro and Okazaki \u0026amp; Naoaki, 2024. \u003cem\u003eOutfox: Llm-generated essay detection through in-context learning with adversarially generated examples. \u003c/em\u003es.l., s.n., pp. 21258--21266.\u003c/li\u003e\n\u003cli\u003eMitchell, et al., 2023. \u003cem\u003eDetectgpt: Zero-shot machine-generated text detection using probability curvature. \u003c/em\u003es.l.:PMLR.\u003c/li\u003e\n\u003cli\u003eMoosavi, et al., 2016. Learning to reason for text generation from scientific tables. \u003cem\u003earXiv preprint arXiv:2104.08296.\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003eMostafazadeh, N. et al., June 12-17, 2016. \u003cem\u003eA corpus and cloze evaluation for deeper understanding of commonsense stories. \u003c/em\u003es.l., The Association for Computational Linguistics, p. 839\u0026ndash;849.\u003c/li\u003e\n\u003cli\u003eNarayan, Shashi and Cohen, Shay B and Lapata \u0026amp; Mirella, October 31 - November 4, 2018. \u003cem\u003eDon\u0026apos;t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. \u003c/em\u003es.l., Association for Computational Linguistics, p. 1797\u0026ndash;1807.\u003c/li\u003e\n\u003cli\u003eOrenstrakh, et al., 2023. Detecting llm-generated text in computing education: A comparative study for chatgpt cases. \u003c/li\u003e\n\u003cli\u003ePu, et al., 2023. \u003cem\u003eDeepfake text detection: Limitations and opportunities. \u003c/em\u003es.l., IEEE, pp. 1613--1630.\u003c/li\u003e\n\u003cli\u003eRajpurkar, P., Zhang, J., Lopyrev, K. \u0026amp; Liang, P., November 1-4,2016. \u003cem\u003eSquad: 100, 000+ questions for machine comprehension of text. \u003c/em\u003es.l., The Association for Computational Linguistics, p. 2383\u0026ndash;2392.\u003c/li\u003e\n\u003cli\u003eSaeidnia \u0026amp; Hamid, R., 2023. Welcome to the Gemini era: Google DeepMind and the information industry. 26 December. \u003c/li\u003e\n\u003cli\u003eSullivan, Miriam and Kelly, Andrew and McLaughlan \u0026amp; Paul, 2023. ChatGPT in higher education: Considerations for academic integrity and student learning. \u003cem\u003eJournal of Applied Learning \u0026amp; Teaching.\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003eTan, et al., 2016. \u003cem\u003eWinning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. \u003c/em\u003es.l., s.n., pp. 613--624.\u003c/li\u003e\n\u003cli\u003eTang, Ruixiang and Chuang, Yu-Neng and Hu \u0026amp; Xia, 2023. The science of detecting llm-generated texts. \u003cem\u003earXiv preprint arXiv:2303.07205.\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003eVenkatraman, Saranya and Uchendu, Adaku and Lee \u0026amp; Dongwon, 2023. Gpt-who: An information density-based machine-generated text detector. \u003cem\u003earXiv preprint arXiv:2310.06202.\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003eWang, et al., 2024. LLM-Detector: Improving AI-Generated Chinese Text Detection with Open-Source LLM Instruction Tuning. \u003cem\u003earXiv preprint arXiv:2402.01158, \u003c/em\u003ep. 17.\u003c/li\u003e\n\u003cli\u003eWu, et al., 2023. A survey on llm-gernerated text detection: Necessity, methods, and future directions. \u003cem\u003earXiv preprint arXiv:2310.14724.\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003eWu, Zhendong and Xiang \u0026amp; Hui, 2023. MFD: Multi-Feature Detection of LLM-Generated Text. \u003c/li\u003e\n\u003cli\u003eZhang, Xiang and Zhao, Junbo and LeCun \u0026amp; Yann, 2015. Character-level convolutional networks for text classification. \u003cem\u003eAdvances in neural information processing systems, \u003c/em\u003eVolume 28.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"social-network-analysis-and-mining","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"snam","sideBox":"Learn more about [Social Network Analysis and Mining](http://link.springer.com/journal/13278)","snPcode":"13278","submissionUrl":"https://submission.nature.com/new-submission/13278/3","title":"Social Network Analysis and Mining","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Bay’s probability, Ensemble model, LLM, word embedding, zero shot model","lastPublishedDoi":"10.21203/rs.3.rs-4318347/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4318347/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe usage of AI in the workplace has increased to 22% of those polled, underscoring the increasing demand for the detection of AI-generated material. With an AUROC of 0.95, DetectGPT beat previous models, demonstrating its efficacy in identifying text written by LLM. Still, there are difficulties in recognizing subtleties that are typical of text generated by language models. This study presents perc-DETECTLLM, a unique method to precisely estimate the percentage of LLM- generated text by combining a zero-shot model with a Bayesian model. perc-DETECTLLM as compared to other models, shows promising results with an accuracy of 72.1% and a recall rate of 81.0%, solving the difficulties in machine-generated text detection. More than 500K samples from different domains makeup constitute our dataset, which makes thorough model training and assessment possible. Perc-DETECTLLM achieves promising text classification results by estimating the overall percentage of LLM-generated text using Bayesian inference. Ensemble techniques perform better and are used as a standard for LLM detection models. Compared to rule-based approaches, perc- DETECTLLM's data-driven methodology improves accuracy and dependability. Overall, all the experiment results are promising and could be extended to other domains, there are still difficulties with dynamic thresholding processes and perturbation function optimization, which points to areas for further study. With perc-DETECTLLM, textual content authentication in the digital age is feasible and is a substantial improvement in LLM-generated text detection.\u003c/p\u003e","manuscriptTitle":"Beyond Binary: A Multi-Class Approach to Large Language Model-Generated Text Percentage Estimation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-04-30 05:57:34","doi":"10.21203/rs.3.rs-4318347/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorAssigned","content":"","date":"2024-05-02T07:07:46+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-04-25T04:56:47+00:00","index":"","fulltext":""},{"type":"submitted","content":"Social Network Analysis and Mining","date":"2024-04-24T12:43:05+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"social-network-analysis-and-mining","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"snam","sideBox":"Learn more about [Social Network Analysis and Mining](http://link.springer.com/journal/13278)","snPcode":"13278","submissionUrl":"https://submission.nature.com/new-submission/13278/3","title":"Social Network Analysis and Mining","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"6acea62a-eb5c-411f-a33f-bdcdd7a3d5e9","owner":[],"postedDate":"April 30th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2024-04-30T05:57:34+00:00","versionOfRecord":[],"versionCreatedAt":"2024-04-30 05:57:34","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4318347","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4318347","identity":"rs-4318347","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}