Argumentative essay assessment with LLMs: A critical scoping review

Argumentative essay assessment with LLMs: A critical scoping review | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Argumentative essay assessment with LLMs: A critical scoping review Lucile Favero, Gabrielle Gaudeau, Juan Antonio Pérez-Ortiz, Tanja Käser, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8632843/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Large Language Models are rapidly reshaping Automated Essay Scoring (AES), yet the methodological, conceptual, and ethical foundations of Argumentative Automated Essay Scoring (AAES) remain underdeveloped. This critical review synthesizes 46 studies published between 2022--2025, following PRISMA 2020 guidelines and a preregistered protocol. We map the landscape of LLM-based AAES across six dimensions---datasets, traits, models, methods, evaluation, and analytics. Our findings show that AAES research remains fragmented and insufficiently grounded in argumentation theory. The field relies on non-comparable datasets which vary in availability, prompt diversity, rater configuration, and linguistic background. Trait analysis reveals substantial overrepresentation of rhetorical and linguistic features and sparse coverage of reasoning-oriented constructs (e.g., logical cogency, dialectical quality). Studies mainly rely on proprietary GPT-family models and rubric-based prompting, while only a minority employ fine-tuning, multi-agent approaches, or reasoning LLMs. Evaluation practices remain uneven: although studies report high human-model agreement, robustness analyses expose sensitivity to prompting, score distributions, and learner proficiency. FATEN analyses reveal recurrent concerns regarding fairness (e.g., style and L1 bias), transparency, randomness sensitivity, limited pedagogical alignment, and an absence of work on privacy or deployment safety. Taken together, the evidence suggests that while LLMs can approximate human scoring on several traits, current systems insufficiently model core argumentative reasoning and lack the validity, interpretability, and accountability required for high-stakes assessment. We conclude by proposing a research agenda focused on construct-valid datasets and rubrics, psychometric modeling, transparent evaluation protocols, and responsible design frameworks. Argumentative Automated Essay Scoring Educational Assessment Large Language Models Multi-trait Assessment Fairness Scoring Rubrics Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8632843","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":583205445,"identity":"9ccba513-2e59-4441-a884-ce1590b8465f","order_by":0,"name":"Lucile Favero","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABA0lEQVRIie3QMWrDMBTG8S8I7CXEq0pKfAWFLBmKe5VnBJk6FLp4KI0h4C6BrIGeIjdQCHQy7WpIKQmBTh2ULUMpfTqAcLOVoj8ItPx4TwJCoT9Yj4/Z4Qq9uFMCt7KdRI4QJoiEI8oR0U5A2DBxN4VfEKn3horXQSTEzJ7UOE2eFusdives9JKJMlRvR7xYdTFXcrh82wiF+k77yQ1MXm3zigm6ShIaHclORdq/mCPfL1Mms+MXk7TR8amdlIbcj/XdFMVTwCTzku4Hv+VZD91i/Ut+y6rRI0k1kY8ksT5Ye5+lSfx4OH4WD+mgyffWFnTtI554RF6eaYBzp4RCodD/7Qety0+OsiT2wwAAAABJRU5ErkJggg==","orcid":"","institution":"ELLIS Alicante Foundation","correspondingAuthor":true,"prefix":"","firstName":"Lucile","middleName":"","lastName":"Favero","suffix":""},{"id":583205446,"identity":"3ed87403-ab11-4596-b9fd-20846a61fcd6","order_by":1,"name":"Gabrielle Gaudeau","email":"","orcid":"","institution":"University of Cambridge","correspondingAuthor":false,"prefix":"","firstName":"Gabrielle","middleName":"","lastName":"Gaudeau","suffix":""},{"id":583205447,"identity":"982fceeb-7bea-46ce-bde8-d09ec3aeea83","order_by":2,"name":"Juan Antonio Pérez-Ortiz","email":"","orcid":"","institution":"University of Alicante","correspondingAuthor":false,"prefix":"","firstName":"Juan","middleName":"Antonio","lastName":"Pérez-Ortiz","suffix":""},{"id":583205448,"identity":"063c738d-8de4-48d5-9abb-74ea5ba8147d","order_by":3,"name":"Tanja Käser","email":"","orcid":"","institution":"École Polytechnique Fédérale de Lausanne","correspondingAuthor":false,"prefix":"","firstName":"Tanja","middleName":"","lastName":"Käser","suffix":""},{"id":583205449,"identity":"96ed1f3c-ada3-4115-9da9-291fe545fde1","order_by":4,"name":"Nuria Oliver","email":"","orcid":"","institution":"ELLIS Alicante Foundation","correspondingAuthor":false,"prefix":"","firstName":"Nuria","middleName":"","lastName":"Oliver","suffix":""}],"badges":[],"createdAt":"2026-01-18 17:23:25","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8632843/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8632843/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":102992741,"identity":"3cc92012-9812-4f3e-bcc9-032518b35fa9","added_by":"auto","created_at":"2026-02-19 11:41:10","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1696554,"visible":true,"origin":"","legend":"","description":"","filename":"manuscriptAIreview.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8632843/v1_covered_0d047c95-4658-44aa-8bab-6dfc6146ccf5.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Argumentative essay assessment with LLMs: A critical scoping review","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Argumentative Automated Essay Scoring, Educational Assessment, Large Language Models, Multi-trait Assessment, Fairness, Scoring Rubrics","lastPublishedDoi":"10.21203/rs.3.rs-8632843/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8632843/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Large Language Models are rapidly reshaping Automated Essay Scoring (AES), yet the methodological, conceptual, and ethical foundations of Argumentative Automated Essay Scoring (AAES) remain underdeveloped. This critical review synthesizes 46 studies published between 2022--2025, following PRISMA 2020 guidelines and a preregistered protocol. We map the landscape of LLM-based AAES across six dimensions---datasets, traits, models, methods, evaluation, and analytics. Our findings show that AAES research remains fragmented and insufficiently grounded in argumentation theory. The field relies on non-comparable datasets which vary in availability, prompt diversity, rater configuration, and linguistic background. Trait analysis reveals substantial overrepresentation of rhetorical and linguistic features and sparse coverage of reasoning-oriented constructs (e.g., logical cogency, dialectical quality). Studies mainly rely on proprietary GPT-family models and rubric-based prompting, while only a minority employ fine-tuning, multi-agent approaches, or reasoning LLMs. Evaluation practices remain uneven: although studies report high human-model agreement, robustness analyses expose sensitivity to prompting, score distributions, and learner proficiency. FATEN analyses reveal recurrent concerns regarding fairness (e.g., style and L1 bias), transparency, randomness sensitivity, limited pedagogical alignment, and an absence of work on privacy or deployment safety. Taken together, the evidence suggests that while LLMs can approximate human scoring on several traits, current systems insufficiently model core argumentative reasoning and lack the validity, interpretability, and accountability required for high-stakes assessment. We conclude by proposing a research agenda focused on construct-valid datasets and rubrics, psychometric modeling, transparent evaluation protocols, and responsible design frameworks.","manuscriptTitle":"Argumentative essay assessment with LLMs: A critical scoping review","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-02-02 03:45:38","doi":"10.21203/rs.3.rs-8632843/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"ac8b0eb3-0b51-4a5d-a226-be394be3c599","owner":[],"postedDate":"February 2nd, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-02-19T11:40:09+00:00","versionOfRecord":[],"versionCreatedAt":"2026-02-02 03:45:38","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8632843","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8632843","identity":"rs-8632843","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}