Mind the Gap: From Plausible to Valid Self-Explanations in Large Language Models

Mind the Gap: From Plausible to Valid Self-Explanations in Large Language Models | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Mind the Gap: From Plausible to Valid Self-Explanations in Large Language Models Korbinian Randl, John Pavlopoulos, Aron Henriksson, Tony Lindgren This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6263278/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 27 Aug, 2025 Read the published version in Machine Learning → Version 1 posted You are reading this latest preprint version Abstract This paper investigates the reliability of explanations generated by large language models (LLMs) when prompted to explain their previous output. We evaluate two kinds of such self-explanations (SE) – extractive and counterfactual – using state-of-the-art LLMs (1B to 70B parameters) on two different classification tasks (objective and subjective). In line with Agarwal et al. (2024), our findings indicate a gap between perceived and actual model reasoning: while SE largely correlate with human judgment (i.e. are plausible), they do not fully and accurately follow the model’s decision process (i.e. are not faithful ). Additionally, we show that counterfactual SE are not even necessarily valid in the sense of actually changing the LLM’s prediction. Our results suggest that extractive SE providethe LLM’s “guess” at an explanation based on training data. Conversely, counterfactual SE can help understand the LLM’s reasoning: We show that the issueof validity can be resolved by sampling counterfactual candidates at high tem-perature – followed by a validity check – and introducing a formula to estimatethe number of tries needed to generate valid explanations. This simple methodproduces plausible and valid explanations that offer a faster alternative to SHAP. Large Language Models (LLMs) Interpretability Self-Explanations Counterfactuals Gradient-Based Explainability Attention-Based Explainability Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 27 Aug, 2025 Read the published version in Machine Learning → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6263278","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":435657101,"identity":"c7b53aa2-d235-47af-92ce-ce86ecd92770","order_by":0,"name":"Korbinian Randl","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABAklEQVRIiWNgGAWjYBACCQYGNoYEEAUGbBZyDAdI1CJhTJwWBGCTSGwgpEWyvffZg4c7LOTNGXgffi4ok0jvO3/GgPFnG4M9Pw4t0jzHzQ0Sz0gY7mxgN5aecU4id+aNHANm3jaGxJkN2LXISaQBHdMmwbjhABuDNG+bRO6GG2wJzIxtDAkGOFwI02IP1ML8G6gl3eD8sQSww+xxaJGGakkEamED2QI0PPkAA9BhjBtweb/nGLsBUEvyhsNsbNY85yQMZ95IPnAYyEicgcMWieNtbA9/ttXZbjjexnybp8xGnu/8wcaHP8ps7PlxeB8BmJHYBxjgkTsKRsEoGAWjgBwAAHmdUS8M62DAAAAAAElFTkSuQmCC","orcid":"","institution":"Stockholm University","correspondingAuthor":true,"prefix":"","firstName":"Korbinian","middleName":"","lastName":"Randl","suffix":""},{"id":435657102,"identity":"47b3229a-19a0-46b0-afd1-5cf113166368","order_by":1,"name":"John Pavlopoulos","email":"","orcid":"","institution":"Athens University of Economics and Business","correspondingAuthor":false,"prefix":"","firstName":"John","middleName":"","lastName":"Pavlopoulos","suffix":""},{"id":435657103,"identity":"bbd60dbb-5c03-4879-9bbd-a8f7c48cae9a","order_by":2,"name":"Aron Henriksson","email":"","orcid":"","institution":"Stockholm University","correspondingAuthor":false,"prefix":"","firstName":"Aron","middleName":"","lastName":"Henriksson","suffix":""},{"id":435657106,"identity":"6129760b-e7a1-4a73-8dd2-cf26b815fb77","order_by":3,"name":"Tony Lindgren","email":"","orcid":"","institution":"Stockholm University","correspondingAuthor":false,"prefix":"","firstName":"Tony","middleName":"","lastName":"Lindgren","suffix":""}],"badges":[],"createdAt":"2025-03-19 16:08:27","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6263278/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6263278/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s10994-025-06838-6","type":"published","date":"2025-08-27T15:57:25+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":90344893,"identity":"a7998d9b-7f98-4ae3-be30-4a0e763bde0a","added_by":"auto","created_at":"2025-09-01 16:07:19","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1158629,"visible":true,"origin":"","legend":"","description":"","filename":"submission.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6263278/v1_covered_98a42eca-5777-48f6-8169-5a0436d28e3d.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Mind the Gap: From Plausible to Valid Self-Explanations in Large Language Models","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Large Language Models (LLMs), Interpretability, Self-Explanations, Counterfactuals, Gradient-Based Explainability, Attention-Based Explainability","lastPublishedDoi":"10.21203/rs.3.rs-6263278/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6263278/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"This paper investigates the reliability of explanations generated by large language models (LLMs) when prompted to explain their previous output. We evaluate two kinds of such self-explanations (SE) – extractive and counterfactual – using state-of-the-art LLMs (1B to 70B parameters) on two different classification tasks (objective and subjective). In line with Agarwal et al. (2024), our findings indicate a gap between perceived and actual model reasoning: while SE largely correlate with human judgment (i.e. are plausible), they do not fully and accurately follow the model’s decision process (i.e. are not faithful ). Additionally, we show that counterfactual SE are not even necessarily valid in the sense of actually changing the LLM’s prediction. Our results suggest that extractive SE providethe LLM’s “guess” at an explanation based on training data. Conversely, counterfactual SE can help understand the LLM’s reasoning: We show that the issueof validity can be resolved by sampling counterfactual candidates at high tem-perature – followed by a validity check – and introducing a formula to estimatethe number of tries needed to generate valid explanations. This simple methodproduces plausible and valid explanations that offer a faster alternative to SHAP.","manuscriptTitle":"Mind the Gap: From Plausible to Valid Self-Explanations in Large Language Models","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-02 21:12:42","doi":"10.21203/rs.3.rs-6263278/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"44282cb2-cc92-4145-908c-38967afe9531","owner":[],"postedDate":"April 2nd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-09-01T16:00:37+00:00","versionOfRecord":{"articleIdentity":"rs-6263278","link":"https://doi.org/10.1007/s10994-025-06838-6","journal":{"identity":"machine-learning","isVorOnly":false,"title":"Machine Learning"},"publishedOn":"2025-08-27 15:57:25","publishedOnDateReadable":"August 27th, 2025"},"versionCreatedAt":"2025-04-02 21:12:42","video":"","vorDoi":"10.1007/s10994-025-06838-6","vorDoiUrl":"https://doi.org/10.1007/s10994-025-06838-6","workflowStages":[]},"version":"v1","identity":"rs-6263278","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6263278","identity":"rs-6263278","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}