{"paper_id":"450fac2e-a98e-4472-8db4-28be7f2fbe1b","body_text":"Fine-grained Insider Threat Detection with Large Language Models: A Comparative Study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Fine-grained Insider Threat Detection with Large Language Models: A Comparative Study Parvin Ahmadi Doval Amiri, Alexis Brissard, Frédéric Cuppens, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7511791/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Insider threats remain a significant challenge in cybersecurity, demanding more effective and efficient detection strategies. The advent of Large Language Models (LLMs) presents new opportunities in Insider Threat Detection (ITD), particularly in monitoring and analyzing behavioral patterns indicative of potential threats. However, LLMs also present limitations, such as the tendency to generate inaccurate or misleading outputs due to their generative nature. This study explores the use of LLMs for ITD by leveraging the CERT r4.2 dataset. We perform a comprehensive comparative analysis of fine-tuned models—specifically BERT, LLaMA 3, and Phi 3—used as classifiers in both binary and multi-class classification tasks, as well as generative models through in-context learning (ICL) techniques. Our findings demonstrate that fine-tuned LLMs achieve high accuracy and stability in detecting insider threats, even across complex multi-class scenarios. These models consistently outperform baseline methods, effectively capturing subtle behavioral cues associated with insider risks. Additionally, we introduce a refined Chain-of-Thought (CoT) prompting method that significantly improves ICL performance, particularly for scenario-specific threat identification. We also investigate the models' ability to manage previously unseen insider behaviors by incorporating a dedicated “Unknown” class. Results reveal that LLMs frequently misclassify these unknown behaviors as benign, especially in high-risk contexts, underscoring the difficulty of detecting novel threats in practical ITD applications. Insider Threat Large Language Models In-Context Learning Fine-tuning Prompting Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {\"props\":{\"pageProps\":{\"initialData\":{\"identity\":\"rs-7511791\",\"acceptedTermsAndConditions\":true,\"allowDirectSubmit\":true,\"archivedVersions\":[],\"articleType\":\"Research Article\",\"associatedPublications\":[],\"authors\":[{\"id\":516560896,\"identity\":\"28193c88-2676-479b-8a2f-ec58c459fbd3\",\"order_by\":0,\"name\":\"Parvin Ahmadi Doval Amiri\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Polytechnique Montréal\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Parvin\",\"middleName\":\"Ahmadi Doval\",\"lastName\":\"Amiri\",\"suffix\":\"\"},{\"id\":516560897,\"identity\":\"7bad6efa-a961-4ead-b4dd-8c7086bbbd3f\",\"order_by\":1,\"name\":\"Alexis Brissard\",\"email\":\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABHUlEQVRIiWNgGAWjYDACdiBOACFmMJdZDogbgAwL3FqY0bQYMzAwArUkSODXAtEF4SY2ENLC38x8TOLBH4Y883b2xx9+7rFO387e2CZ184cEA3/7AaxaJA6zJRsktjEUyxzmMZPseZaeu7PnYJt0DtAWiTMJWLUwHOYxfAB0TOIMZh42Bp4Dh3M33EiEaDFgwK5F/jD/hwMJf0Ba2B9//HPgcLoBXAv/A6xaDA7zMD5IYANpYTCQBtqSgNAigd0Ww8NsxkC/SBRLMPOYScscSDcE+qXZOidNgkfiBnZb5I43P5P88ccmT4L/+OOPbw5Yy5uzNx+8nWNjI8ffj8P70IBDcioDAwuIy4NPPZrvgPH5gXjlo2AUjIJRMAIAAPxDWWdAjvHGAAAAAElFTkSuQmCC\",\"orcid\":\"\",\"institution\":\"Polytechnique Montréal\",\"correspondingAuthor\":true,\"prefix\":\"\",\"firstName\":\"Alexis\",\"middleName\":\"\",\"lastName\":\"Brissard\",\"suffix\":\"\"},{\"id\":516560898,\"identity\":\"b9fcae78-7aef-49b0-a25d-10c0911718e7\",\"order_by\":2,\"name\":\"Frédéric Cuppens\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Polytechnique Montréal\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Frédéric\",\"middleName\":\"\",\"lastName\":\"Cuppens\",\"suffix\":\"\"},{\"id\":516560900,\"identity\":\"88fd89b9-f8e4-408b-96cf-213f1933928d\",\"order_by\":3,\"name\":\"Amal Zouaq\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"Polytechnique Montréal\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Amal\",\"middleName\":\"\",\"lastName\":\"Zouaq\",\"suffix\":\"\"}],\"badges\":[],\"createdAt\":\"2025-09-01 22:53:10\",\"currentVersionCode\":1,\"declarations\":\"\",\"doi\":\"10.21203/rs.3.rs-7511791/v1\",\"doiUrl\":\"https://doi.org/10.21203/rs.3.rs-7511791/v1\",\"draftVersion\":[],\"editorialEvents\":[],\"editorialNote\":\"\",\"failedWorkflow\":false,\"files\":[{\"id\":91938182,\"identity\":\"ceb01e68-6cde-4eed-bc8b-8175d069e48c\",\"added_by\":\"auto\",\"created_at\":\"2025-09-23 03:05:17\",\"extension\":\"pdf\",\"order_by\":0,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":3212483,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"FinegrainedInsiderThreatDetectionwithLargeLanguageModelsAComparativeStudy.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7511791/v1/078356f38f5ff09ac585936f.pdf\"},{\"id\":91938181,\"identity\":\"a27d0c70-4b31-4b4d-bfb0-c61d0dead61b\",\"added_by\":\"auto\",\"created_at\":\"2025-09-23 03:05:16\",\"extension\":\"json\",\"order_by\":1,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":5944,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"3aab7d743ac14e2094a6aca3d7564c5b.json\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7511791/v1/858a9c628072b5e27ca60616.json\"},{\"id\":108181015,\"identity\":\"79b25d56-e89a-49b7-91a2-2ea05ffe03f3\",\"added_by\":\"auto\",\"created_at\":\"2026-04-30 08:56:14\",\"extension\":\"pdf\",\"order_by\":1,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"manuscript-pdf\",\"size\":3699779,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"FinegrainedInsiderThreatDetectionwithLargeLanguageModelsAComparativeStudy.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-7511791/v1_covered_b32e658f-873a-4887-a7aa-da6e918467d5.pdf\"}],\"financialInterests\":\"No competing interests reported.\",\"formattedTitle\":\"Fine-grained Insider Threat Detection with Large Language Models: A Comparative Study\",\"fulltext\":[],\"fulltextSource\":\"\",\"fullText\":\"\",\"funders\":[],\"hasAdminPriorityOnWorkflow\":false,\"hasManuscriptDocX\":false,\"hasOptedInToPreprint\":true,\"hasPassedJournalQc\":\"\",\"hasAnyPriority\":false,\"hideJournal\":true,\"highlight\":\"\",\"institution\":\"\",\"isAcceptedByJournal\":false,\"isAuthorSuppliedPdf\":true,\"isDeskRejected\":\"\",\"isHiddenFromSearch\":false,\"isInQc\":false,\"isInWorkflow\":false,\"isPdf\":true,\"isPdfUpToDate\":true,\"isWithdrawnOrRetracted\":false,\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"researchsquare\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":true,\"externalIdentity\":\"\",\"sideBox\":\"\",\"snPcode\":\"\",\"submissionUrl\":\"/submission\",\"title\":\"Research Square\",\"twitterHandle\":\"researchsquare\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"\",\"reportingPortfolio\":\"\",\"inReviewEnabled\":false,\"inReviewRevisionsEnabled\":true},\"keywords\":\"Insider Threat, Large Language Models, In-Context Learning, Fine-tuning, Prompting\",\"lastPublishedDoi\":\"10.21203/rs.3.rs-7511791/v1\",\"lastPublishedDoiUrl\":\"https://doi.org/10.21203/rs.3.rs-7511791/v1\",\"license\":{\"name\":\"CC BY 4.0\",\"url\":\"https://creativecommons.org/licenses/by/4.0/\"},\"manuscriptAbstract\":\"\\u003cp\\u003eInsider threats remain a significant challenge in cybersecurity, demanding more effective and efficient detection strategies. The advent of Large Language Models (LLMs) presents new opportunities in Insider Threat Detection (ITD), particularly in monitoring and analyzing behavioral patterns indicative of potential threats. However, LLMs also present limitations, such as the tendency to generate inaccurate or misleading outputs due to their generative nature.\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003eThis study explores the use of LLMs for ITD by leveraging the CERT r4.2 dataset. We perform a comprehensive comparative analysis of fine-tuned models—specifically BERT, LLaMA 3, and Phi 3—used as classifiers in both binary and multi-class classification tasks, as well as generative models through in-context learning (ICL) techniques. Our findings demonstrate that fine-tuned LLMs achieve high accuracy and stability in detecting insider threats, even across complex multi-class scenarios. These models consistently outperform baseline methods, effectively capturing subtle behavioral cues associated with insider risks.\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cp\\u003eAdditionally, we introduce a refined Chain-of-Thought (CoT) prompting method that significantly improves ICL performance, particularly for scenario-specific threat identification. We also investigate the models' ability to manage previously unseen insider behaviors by incorporating a dedicated “Unknown” class. Results reveal that LLMs frequently misclassify these unknown behaviors as benign, especially in high-risk contexts, underscoring the difficulty of detecting novel threats in practical ITD applications.\\u003c/p\\u003e\",\"manuscriptTitle\":\"Fine-grained Insider Threat Detection with Large Language Models: A Comparative Study\",\"msid\":\"\",\"msnumber\":\"\",\"nonDraftVersions\":[{\"code\":1,\"date\":\"2025-09-23 03:05:12\",\"doi\":\"10.21203/rs.3.rs-7511791/v1\",\"editorialEvents\":[{\"type\":\"communityComments\",\"content\":0}],\"status\":\"published\",\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"researchsquare\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":true,\"externalIdentity\":\"\",\"sideBox\":\"\",\"snPcode\":\"\",\"submissionUrl\":\"/submission\",\"title\":\"Research Square\",\"twitterHandle\":\"researchsquare\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"\",\"reportingPortfolio\":\"\",\"inReviewEnabled\":false,\"inReviewRevisionsEnabled\":true}}],\"origin\":\"\",\"ownerIdentity\":\"0a91a642-e2c7-4851-810e-887560ee50cc\",\"owner\":[],\"postedDate\":\"September 23rd, 2025\",\"published\":true,\"recentEditorialEvents\":[],\"rejectedJournal\":[],\"revision\":\"\",\"amendment\":\"\",\"status\":\"posted\",\"subjectAreas\":[],\"tags\":[],\"updatedAt\":\"2026-04-24T14:11:16+00:00\",\"versionOfRecord\":[],\"versionCreatedAt\":\"2025-09-23 03:05:12\",\"video\":\"\",\"vorDoi\":\"\",\"vorDoiUrl\":\"\",\"workflowStages\":[]},\"version\":\"v1\",\"identity\":\"rs-7511791\",\"journalConfig\":\"researchsquare\"},\"__N_SSP\":true},\"page\":\"/article/[identity]/[[...version]]\",\"query\":{\"redirect\":\"/article/rs-7511791\",\"identity\":\"rs-7511791\",\"version\":[\"v1\"]},\"buildId\":\"8U1c8b4HqxoKbykW_rLl7\",\"isFallback\":false,\"isExperimentalCompile\":false,\"dynamicIds\":[84888],\"gssp\":true,\"scriptLoader\":[]}","source_license":"CC-BY-4.0","license_restricted":false}