MultiLLM – Self Reflect Iterative Prompt Methodology based Automated Essay Scoring System

MultiLLM – Self Reflect Iterative Prompt Methodology based Automated Essay Scoring System | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article MultiLLM – Self Reflect Iterative Prompt Methodology based Automated Essay Scoring System R. Johnsi, G. Bharadwaja Kumar This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6619776/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Although the use of Large Language Models (LLMs) for essay scoring is not a new concept, these models do not grade in the same manner as humans. This discrepancy arises because humans can adapt their grading patterns based on the specific questions they encounter. In contrast, existing research on this topic typically employs a predefined rubric that fails to address the variability in responses effectively. There has been a noticeable lack of systematic research aimed at defining rubrics and prompts tailored to the responses considered. To address this issue and provide a structured approach to LLM-based grading, this paper suggests a new methodology. We propose the use of multiple LLMs for rubric generation and grading through a process of self-reflection and iteration. The key components of this system include: 1. Developing grading rubrics and prompt patterns that account for both the questions asked and the responses provided. 2. Employing self-reflective iteration rubrics across multiple LLMs to ensure consistent scoring for diverse responses. 3. Implementing verification and validation processes to effectively identify anomalous scores, allowing for re-evaluation and achieving consistency. Experimental evaluations demonstrate that the proposed system offers new insights into the role of LLMs in Automated Essay Scoring (AES). Physical sciences/Mathematics and computing/Information technology Physical sciences/Mathematics and computing/Computer science Natural Language Processing (NLP) Automated Essay Scoring (AES) MultiLLM iterative Prompt Methodology (MIPM) MultiLLM-Self reflect iterative-prompt-methodology (MISM) Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6619776","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":471088471,"identity":"d09bbd6b-859f-45bd-83e7-f08232eebe12","order_by":0,"name":"R. Johnsi","email":"","orcid":"","institution":"Vellore institute of technology","correspondingAuthor":false,"prefix":"","firstName":"R.","middleName":"","lastName":"Johnsi","suffix":""},{"id":471088472,"identity":"97867fc6-488f-4032-a5dd-ce220c123a53","order_by":1,"name":"G. Bharadwaja Kumar","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABF0lEQVRIiWNgGAWjYBACAwkQycbAwNjAwAwW4QcxoYJEapFsYGxsIEoLEEC0GByAWYMDmEs3P/vwo+ywPHP7AWZj3hy7fOMbye0PGGrsGPikseu0nHPMeGbPucOGjT0JzMm825Itt91IBDrsWDIDm8wB7A67kWDMwNt2G+j+BObDvNuYDczOHARqYTvAwCaRgENL+mfGv2237Rv7H4C01BsY94C0/MOnJceYGWhLYuMMsMMOGxiwNzY2MLbh1mI5I6eYWebc/+TGGQ+bDeduO24gcbyxcUZiXzIPLi3mEumbGd+Updlu7E8+LPF2W7UBfzP7gw8fvtnJyc/ArgUODBuQYwOomAe/eiCQJ6hiFIyCUTAKRiwAAH6GXf88EOynAAAAAElFTkSuQmCC","orcid":"","institution":"Vellore institute of technology","correspondingAuthor":true,"prefix":"","firstName":"G.","middleName":"Bharadwaja","lastName":"Kumar","suffix":""}],"badges":[],"createdAt":"2025-05-08 10:38:32","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6619776/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6619776/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":89621648,"identity":"984f7b13-e6b5-4ed8-95f5-8d7fd2a50789","added_by":"auto","created_at":"2025-08-22 04:16:46","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1130072,"visible":true,"origin":"","legend":"","description":"","filename":"MultiLLMSelfReflectIterativePromptMethodologybasedAutomatedEssayScoringSystem.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6619776/v1_covered_36be1898-bd6f-40c7-a635-94f361a57ea5.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"MultiLLM – Self Reflect Iterative Prompt Methodology based Automated Essay Scoring System","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Natural Language Processing (NLP), Automated Essay Scoring (AES), MultiLLM iterative Prompt Methodology (MIPM), MultiLLM-Self reflect iterative-prompt-methodology (MISM)","lastPublishedDoi":"10.21203/rs.3.rs-6619776/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6619776/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAlthough the use of Large Language Models (LLMs) for essay scoring is not a new concept, these models do not grade in the same manner as humans. This discrepancy arises because humans can adapt their grading patterns based on the specific questions they encounter. In contrast, existing research on this topic typically employs a predefined rubric that fails to address the variability in responses effectively. There has been a noticeable lack of systematic research aimed at defining rubrics and prompts tailored to the responses considered. To address this issue and provide a structured approach to LLM-based grading, this paper suggests a new methodology. We propose the use of multiple LLMs for rubric generation and grading through a process of self-reflection and iteration. The key components of this system include: 1. Developing grading rubrics and prompt patterns that account for both the questions asked and the responses provided. 2. Employing self-reflective iteration rubrics across multiple LLMs to ensure consistent scoring for diverse responses. 3. Implementing verification and validation processes to effectively identify anomalous scores, allowing for re-evaluation and achieving consistency. Experimental evaluations demonstrate that the proposed system offers new insights into the role of LLMs in Automated Essay Scoring (AES).\u003c/p\u003e","manuscriptTitle":"MultiLLM – Self Reflect Iterative Prompt Methodology based Automated Essay Scoring System","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-18 15:01:27","doi":"10.21203/rs.3.rs-6619776/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"f703b3d9-b0bc-48d7-97fe-6c0bfba59951","owner":[],"postedDate":"June 18th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":50032994,"name":"Physical sciences/Mathematics and computing/Information technology"},{"id":50032995,"name":"Physical sciences/Mathematics and computing/Computer science"}],"tags":[],"updatedAt":"2025-08-22T04:08:38+00:00","versionOfRecord":[],"versionCreatedAt":"2025-06-18 15:01:27","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6619776","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6619776","identity":"rs-6619776","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}