Mitigating Structural Hallucination in Large Language Models with Local Diffusion

Mitigating Structural Hallucination in Large Language Models with Local Diffusion | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Mitigating Structural Hallucination in Large Language Models with Local Diffusion Kizuki Kiritani, Tsumugi Kayano This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4678127/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Large language models (LLMs) often produce text with inaccuracies, logical inconsistencies, or fabricated information, known as structural hallucinations, which undermine their reliability and trustworthiness. Implementing local diffusion mechanisms within the Mistral LLM architecture has demonstrated significant potential in addressing these issues, enhancing both the accuracy and coherence of the generated text. The modified model exhibited substantial improvements across various performance metrics, including accuracy, precision, recall, and F1 score, validated through rigorous statistical testing. The architectural adjustments, involving the integration of diffusion layers, facilitated better information propagation and reduced the occurrence of structurally flawed outputs. Quantitative analyses highlighted the model's enhanced performance, while qualitative comparisons revealed its improved structural integrity and factual accuracy. Additionally, error analysis revealed a notable reduction in the frequency of factual and logical errors, further affirming the effectiveness of the local diffusion approach. The findings reveal the transformative potential of local diffusion in mitigating structural hallucinations and advancing the field of natural language processing. Artificial Intelligence and Machine Learning Local diffusion Structural hallucination Model architecture Performance metrics Natural language processing Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4678127","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":322051003,"identity":"fc5db9a0-1d2e-437a-a718-6b8967415214","order_by":0,"name":"Kizuki Kiritani","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABFklEQVRIie3PMUvDQBTA8RcO0uWVrFda4lc4KUgHoV+l0iFTwdFBJCHQDK36BfwQt4lbymFcUlwT6iJCpw4BoTgU8V0EEUludrj/cHkc+XF3ADbbP4wzgJS+COiENHJf1PtiZCbpLzIUwDTh7UQv+hhC9XQmawLtpJd0X1V1+TIAVPE75qPgvnP7KD/OOXjJYtJE+qwj0jTbInSjcIgFnz0slVsu6WI8X8sm4jOX3uIqBM8Jp1jxmSymboFEBM3t5PObKCKBIFIeDKSvyWqu9MWimC420WRjOqUXE1nfEMFV7Nzl/Fjm6mQzEBzb3sKfM1Zd7NUYMHirdtnVkXiKtuXucOp7yXUj+Wmc/tlA4+82m81mM/YFjfFhU7AosmcAAAAASUVORK5CYII=","orcid":"https://orcid.org/0009-0006-8865-6363","institution":"","correspondingAuthor":true,"prefix":"","firstName":"Kizuki","middleName":"","lastName":"Kiritani","suffix":""},{"id":322051004,"identity":"9b43d5d2-5ee9-4b81-9505-080b8620261c","order_by":1,"name":"Tsumugi Kayano","email":"","orcid":"https://orcid.org/0009-0006-3946-9158","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Tsumugi","middleName":"","lastName":"Kayano","suffix":""}],"badges":[],"createdAt":"2024-07-03 06:40:38","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-4678127/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4678127/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":59622438,"identity":"3300e0dc-f23b-414b-b8b2-79291ee0f4fa","added_by":"auto","created_at":"2024-07-04 02:41:06","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":239925,"visible":true,"origin":"","legend":"","description":"","filename":"article.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4678127/v1_covered_0c73f3cb-5407-4695-8312-fab5cb7877b8.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eMitigating Structural Hallucination in Large Language Models with Local Diffusion\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Local diffusion, Structural hallucination, Model architecture, Performance metrics, Natural language processing","lastPublishedDoi":"10.21203/rs.3.rs-4678127/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4678127/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eLarge language models (LLMs) often produce text with inaccuracies, logical inconsistencies, or fabricated information, known as structural hallucinations, which undermine their reliability and trustworthiness. Implementing local diffusion mechanisms within the Mistral LLM architecture has demonstrated significant potential in addressing these issues, enhancing both the accuracy and coherence of the generated text. The modified model exhibited substantial improvements across various performance metrics, including accuracy, precision, recall, and F1 score, validated through rigorous statistical testing. The architectural adjustments, involving the integration of diffusion layers, facilitated better information propagation and reduced the occurrence of structurally flawed outputs. Quantitative analyses highlighted the model's enhanced performance, while qualitative comparisons revealed its improved structural integrity and factual accuracy. Additionally, error analysis revealed a notable reduction in the frequency of factual and logical errors, further affirming the effectiveness of the local diffusion approach. The findings reveal the transformative potential of local diffusion in mitigating structural hallucinations and advancing the field of natural language processing.\u003c/p\u003e","manuscriptTitle":"Mitigating Structural Hallucination in Large Language Models with Local Diffusion","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-07-04 02:33:00","doi":"10.21203/rs.3.rs-4678127/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"2c98dacb-87e6-4836-bae7-383f3d570dae","owner":[],"postedDate":"July 4th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":34052755,"name":"Artificial Intelligence and Machine Learning"}],"tags":[],"updatedAt":"2024-07-04T02:33:00+00:00","versionOfRecord":[],"versionCreatedAt":"2024-07-04 02:33:00","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4678127","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4678127","identity":"rs-4678127","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}