Towards Explainable Language Reasoning via Multi-Modal Knowledge Graphs

Towards Explainable Language Reasoning via Multi-Modal Knowledge Graphs | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Towards Explainable Language Reasoning via Multi-Modal Knowledge Graphs chunyu lu, jun luo, kang yu, tianran chen, xueli wang, feng qian, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7677808/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The increasing complexity of natural language reasoning poses significant challenges for transparency, interpretability, and trustworthiness in artificial intelligence systems. While large-scale language models have demonstrated remarkable success in generating contextually relevant responses, their decision-making processes often remain opaque. To address this gap, we propose a novel framework for explainable language reasoning based on multi-modal knowledge graphs (MMKGs). The framework integrates textual, visual, and structural knowledge sources into a unified graph representation, enabling models to ground language reasoning in explicit, semantically rich relationships. We introduce mechanisms for reasoning over MMKGs to generate interpretable inference paths, thus providing human-understandable justifications for model outputs. Experiments conducted on benchmark datasets demonstrate that our approach achieves competitive reasoning performance while significantly improving explainability, as measured by both automated and human evaluation metrics. The proposed framework contributes to bridging the gap between accuracy and interpretability, offering a pathway toward trustworthy and explainable language reasoning in real-world applications such as question answering, dialogue systems, and decision support. Physical sciences/Mathematics and computing Biological sciences/Psychology Social science/Psychology Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7677808","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":532210644,"identity":"62f2d31a-9f1a-4003-9978-31d3aee93178","order_by":0,"name":"chunyu lu","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"chunyu","middleName":"","lastName":"lu","suffix":""},{"id":532210646,"identity":"b47abe5e-b253-45b7-ac53-740db2f5ebaa","order_by":1,"name":"jun luo","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"jun","middleName":"","lastName":"luo","suffix":""},{"id":532210647,"identity":"faa74232-a05b-4ff0-8d24-b7ac8db61963","order_by":2,"name":"kang yu","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"kang","middleName":"","lastName":"yu","suffix":""},{"id":532210648,"identity":"23a82a88-cc31-4e41-bb33-6ea62e8fac4a","order_by":3,"name":"tianran chen","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"tianran","middleName":"","lastName":"chen","suffix":""},{"id":532210649,"identity":"985714a0-334b-41df-aee6-641557bd7299","order_by":4,"name":"xueli wang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7klEQVRIiWNgGAWjYFACHhDxX46xvf3gAxCXj6AONrAWZmPmnjPJBiAtbMRqSWyfkWAmARYgpIN/fu+xDx8q2Bh7ew6kVX7NsZNhY2B++OgGHi0Sx/iSZ844w8Ms2d547LbstmSgw9iMjXPwWXOMx5iZt02CzRBoy23JbcxALTxs0vi0yIO0/P1nwGN/I8GsWHJbPWEtBiAtjA0JEoxA7zN+3HaYsBbDYznGjD3HDhgwAgNZmnHbcR42ZgJ+kTt8xpjhR82B+kZgVH78ua3anp+9+eFjvN5HBsyQOCJWOQgw/iBF9SgYBaNgFIwYAABsjkZGRzGIrgAAAABJRU5ErkJggg==","orcid":"","institution":"","correspondingAuthor":true,"prefix":"","firstName":"xueli","middleName":"","lastName":"wang","suffix":""},{"id":532210650,"identity":"6d210240-4277-4b82-93ab-709288ed76b3","order_by":5,"name":"feng qian","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"feng","middleName":"","lastName":"qian","suffix":""},{"id":532210651,"identity":"2b4ba09e-45e8-452c-bec9-b96bced12375","order_by":6,"name":"chen xi","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"chen","middleName":"","lastName":"xi","suffix":""}],"badges":[],"createdAt":"2025-09-22 14:08:23","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7677808/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7677808/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":94190694,"identity":"cc2888c7-4ade-4621-a6d8-adeea8f76d41","added_by":"auto","created_at":"2025-10-23 12:00:50","extension":"json","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":8262,"visible":true,"origin":"","legend":"","description":"","filename":"098af4b9da2242b899a165d57c64694c.json","url":"https://assets-eu.researchsquare.com/files/rs-7677808/v1/d1f9675f43e305fc9ab80fe0.json"},{"id":99315954,"identity":"fcea654d-3aa9-432a-bfb0-677ab2cf8bb9","added_by":"auto","created_at":"2025-12-31 16:27:30","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":613201,"visible":true,"origin":"","legend":"","description":"","filename":"TemplateforsubmissionstoScientificReports2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7677808/v1_covered_ec74b54b-f15d-492e-a3c6-bd7a7b6b0411.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Towards Explainable Language Reasoning via Multi-Modal Knowledge Graphs","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-7677808/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7677808/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"The increasing complexity of natural language reasoning poses significant challenges for transparency, interpretability, and trustworthiness in artificial intelligence systems. While large-scale language models have demonstrated remarkable success in generating contextually relevant responses, their decision-making processes often remain opaque. To address this gap, we propose a novel framework for explainable language reasoning based on multi-modal knowledge graphs (MMKGs). The framework integrates textual, visual, and structural knowledge sources into a unified graph representation, enabling models to ground language reasoning in explicit, semantically rich relationships. We introduce mechanisms for reasoning over MMKGs to generate interpretable inference paths, thus providing human-understandable justifications for model outputs. Experiments conducted on benchmark datasets demonstrate that our approach achieves competitive reasoning performance while significantly improving explainability, as measured by both automated and human evaluation metrics. The proposed framework contributes to bridging the gap between accuracy and interpretability, offering a pathway toward trustworthy and explainable language reasoning in real-world applications such as question answering, dialogue systems, and decision support.","manuscriptTitle":"Towards Explainable Language Reasoning via Multi-Modal Knowledge Graphs","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-10-23 12:00:45","doi":"10.21203/rs.3.rs-7677808/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e671b806-beef-4995-bdd5-4be016924a61","owner":[],"postedDate":"October 23rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":56571197,"name":"Physical sciences/Mathematics and computing"},{"id":56571198,"name":"Biological sciences/Psychology"},{"id":56571199,"name":"Social science/Psychology"}],"tags":[],"updatedAt":"2025-12-29T18:08:53+00:00","versionOfRecord":[],"versionCreatedAt":"2025-10-23 12:00:45","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7677808","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7677808","identity":"rs-7677808","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}