Architectural Transparency in LLM-Based Cognitive Assessment: A Multidimensional TRACE-ED Evaluation of Single-Agent and Multi-Agent Systems

Architectural Transparency in LLM-Based Cognitive Assessment: A Multidimensional TRACE-ED Evaluation of Single-Agent and Multi-Agent Systems | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Architectural Transparency in LLM-Based Cognitive Assessment: A Multidimensional TRACE-ED Evaluation of Single-Agent and Multi-Agent Systems Dani Chandra Yudho Pranoto, Suhailah Binti Hussien, Sabariah Sabariah, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8985839/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Purpose: The rapid integration of Large Language Models (LLMs) into educational assessment systems has intensified the need for rigorous transparency validation beyond accuracy and score agreement. While prior research primarily evaluates automated grading systems through reliability and human–AI alignment metrics, limited attention has been given to how architectural design influences transparency properties. This study proposes TRACE-ED, a multidimensional transparency framework that operationalizes transparency across five dimensions: Reliability (R), Alignment (A), Claim–Evidence Grounding (C), Explanation Coherence (E), and Disclosure/Auditability (D). Transparency is conceptualized as an architectural vector rather than a scalar attribute. Methods: Using a Design Science Research methodology and a Monte Carlo experimental design, this study compares single-agent and multi-agent LLM architectures across 10 synthetic short-answer responses, three rubric indicators, multiple temperature settings, and 15 repetitions per condition (total runs = 900). Reliability is evaluated using Intraclass Correlation Coefficient ICC(1,k), while grounding and coherence are measured through semantic similarity thresholds and polarity alignment metrics. Statistical comparisons employ Welch’s t-test and Cohen’s d effect sizes. Results: Results demonstrate that architectural modularization preserves high reliability (Multi-Agent ICC(1,k) = 0.9921) while significantly increasing semantic grounding (GR = 0.763; d = 3.72) and explanation coherence (CS = 0.782; d = 7.90). A minor increase in contradiction rate (CR = 0.031; d = 0.43) indicates limited coordination trade-offs. These findings confirm that transparency is multidimensional and architecture-sensitive. Multi-agent systems redistribute transparency components rather than uniformly maximizing them. Conclusion: This study advances explainable AI research in education by formalizing transparency as a measurable architectural vector, introducing Monte Carlo–based stochastic validation for LLM assessment systems, and providing empirical evidence of transparency trade-offs in modular AI design. The TRACE-ED framework offers a scalable and auditable methodology for evaluating AI-driven grading systems in high-stakes educational contexts. Large Language Models Automated Cognitive Assessment Explainable AI Architectural Transparency Multi-Agent Systems Monte Carlo Simulation Intraclass Correlation Coefficient Claim Evidence Grounding Explanation Coherence Educational AI Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8985839","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":601484717,"identity":"b4231814-76b5-415e-a220-8f6d56711f56","order_by":0,"name":"Dani Chandra Yudho Pranoto","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABNklEQVRIie3RPUvDQBjA8acUEoejXe+IjV/hwkEyFr9JglCXCE61g+CVQroUXRWlHyOugUC6nM6CHRIKNzlkkJAhiIngCyQZ3ATznx7C/bjcHUBX118s+Jr65XgKgPHP77bSThAoNgClvyKINhCok8FGmLvZ+VYfq6ssTmmhk9uFTF6LrcPVMIZ4WiNEuJYhIskQerg3rill2n5ksZEnHY4mFOzHGqFPyCRcCZ0VPvE1RKmzxrapER46HFwAx2shbxVxZUUu1vg403BRkuFLO5l7H0SpiK1h1yRpuS/HzbsQMTkz5pchQyKyyrMw4+ZOTLWeJ5mHJQ0azjLYhH7Cs1BXlwsZpzP9AD8vfZIX29HV8ChJ8vqNAeyZn9P3E/QRwGH1S0EDAFBlnfRygHHj6q6urq7/2DsbnHIo9odNhwAAAABJRU5ErkJggg==","orcid":"","institution":"Universitas PGRI Adi Buana Surabaya","correspondingAuthor":true,"prefix":"","firstName":"Dani","middleName":"Chandra Yudho","lastName":"Pranoto","suffix":""},{"id":601484718,"identity":"fca3b3b5-38f7-4232-8702-5a5766f576b4","order_by":1,"name":"Suhailah Binti Hussien","email":"","orcid":"","institution":"International Islamic University Malaysia","correspondingAuthor":false,"prefix":"","firstName":"Suhailah","middleName":"Binti","lastName":"Hussien","suffix":""},{"id":601484719,"identity":"346d2342-094d-49da-abd7-f5fc10e57bb7","order_by":2,"name":"Sabariah Sabariah","email":"","orcid":"","institution":"Universitas PGRI Adi Buana Surabaya","correspondingAuthor":false,"prefix":"","firstName":"Sabariah","middleName":"","lastName":"Sabariah","suffix":""},{"id":601484721,"identity":"fb6f10ec-7928-4d9e-80da-d1ff09843c55","order_by":3,"name":"Adi Bandono","email":"","orcid":"","institution":"Universitas PGRI Adi Buana Surabaya","correspondingAuthor":false,"prefix":"","firstName":"Adi","middleName":"","lastName":"Bandono","suffix":""},{"id":601484723,"identity":"bd9f56c6-bac5-42df-b7b1-8220dd505fe6","order_by":4,"name":"Ahmad Bahrawi","email":"","orcid":"","institution":"Politeknik Penerbangan Surabaya","correspondingAuthor":false,"prefix":"","firstName":"Ahmad","middleName":"","lastName":"Bahrawi","suffix":""}],"badges":[],"createdAt":"2026-02-27 09:10:04","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8985839/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8985839/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":104552350,"identity":"a73d0fdc-8b4d-4f8b-97ab-d24183badc2b","added_by":"auto","created_at":"2026-03-13 08:27:39","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1763529,"visible":true,"origin":"","legend":"","description":"","filename":"SpringerNatureLaTeXTemplate.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8985839/v1_covered_93bde674-6aba-4e97-b4e2-82af76d7e90c.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Architectural Transparency in LLM-Based Cognitive Assessment: A Multidimensional TRACE-ED Evaluation of Single-Agent and Multi-Agent Systems","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Large Language Models, Automated Cognitive Assessment, Explainable AI, Architectural Transparency, Multi-Agent Systems, Monte Carlo Simulation, Intraclass Correlation Coefficient, Claim, Evidence Grounding, Explanation Coherence, Educational AI","lastPublishedDoi":"10.21203/rs.3.rs-8985839/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8985839/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003ePurpose:\u003c/h2\u003e\u003cp\u003eThe rapid integration of Large Language Models (LLMs) into educational assessment systems has intensified the need for rigorous transparency validation beyond accuracy and score agreement. While prior research primarily evaluates automated grading systems through reliability and human\u0026ndash;AI alignment metrics, limited attention has been given to how architectural design influences transparency properties. This study proposes TRACE-ED, a multidimensional transparency framework that operationalizes transparency across five dimensions: Reliability (R), Alignment (A), Claim\u0026ndash;Evidence Grounding (C), Explanation Coherence (E), and Disclosure/Auditability (D). Transparency is conceptualized as an architectural vector rather than a scalar attribute.\u003c/p\u003e\u003ch2\u003eMethods:\u003c/h2\u003e\u003cp\u003eUsing a Design Science Research methodology and a Monte Carlo experimental design, this study compares single-agent and multi-agent LLM architectures across 10 synthetic short-answer responses, three rubric indicators, multiple temperature settings, and 15 repetitions per condition (total runs = 900). Reliability is evaluated using Intraclass Correlation Coefficient ICC(1,k), while grounding and coherence are measured through semantic similarity thresholds and polarity alignment metrics. Statistical comparisons employ Welch\u0026rsquo;s t-test and Cohen\u0026rsquo;s d effect sizes.\u003c/p\u003e\u003ch2\u003eResults:\u003c/h2\u003e\u003cp\u003eResults demonstrate that architectural modularization preserves high reliability (Multi-Agent ICC(1,k) = 0.9921) while significantly increasing semantic grounding (GR = 0.763; d = 3.72) and explanation coherence (CS = 0.782; d = 7.90). A minor increase in contradiction rate (CR = 0.031; d = 0.43) indicates limited coordination trade-offs. These findings confirm that transparency is multidimensional and architecture-sensitive. Multi-agent systems redistribute transparency components rather than uniformly maximizing them.\u003c/p\u003e\u003ch2\u003eConclusion:\u003c/h2\u003e\u003cp\u003eThis study advances explainable AI research in education by formalizing transparency as a measurable architectural vector, introducing Monte Carlo\u0026ndash;based stochastic validation for LLM assessment systems, and providing empirical evidence of transparency trade-offs in modular AI design. The TRACE-ED framework offers a scalable and auditable methodology for evaluating AI-driven grading systems in high-stakes educational contexts.\u003c/p\u003e","manuscriptTitle":"Architectural Transparency in LLM-Based Cognitive Assessment: A Multidimensional TRACE-ED Evaluation of Single-Agent and Multi-Agent Systems","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-06 18:31:51","doi":"10.21203/rs.3.rs-8985839/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"7f4ec5dd-d1cb-4b42-b79d-54521c3ba930","owner":[],"postedDate":"March 6th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-03-13T08:26:10+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-06 18:31:51","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8985839","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8985839","identity":"rs-8985839","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}