RePCS: Diagnosing Data Memorization in LLM-Powered Retrieval-Augmented Generation

RePCS: Diagnosing Data Memorization in LLM-Powered Retrieval-Augmented Generation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article RePCS: Diagnosing Data Memorization in LLM-Powered Retrieval-Augmented Generation Vu Anh Le, Viet Anh Nguyen, Mehmet Dik, Van Nghia Luong This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6992096/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Retrieval-augmented generation (RAG) has become a common strategy for updating large language model (LLM) responses with current, external information. However, models may still rely on memorized training data, bypass the retrieved evidence, and produce contaminated outputs. We introduce Retrieval-Path Contamination Scoring (RePCS), a diagnostic method that detects such behavior without requiring model access or retraining. RePCS compares two inference paths: (i) a parametric path using only the query, and (ii) a retrieval-augmented path using both the query and retrieved context by computing the Kullback-Leibler (KL) divergence between their output distributions. A low divergence suggests that the retrieved context had minimal impact, indicating potential memorization. This procedure is model-agnostic, requires no gradient or internal state access, and adds only a single additional forward pass. We further derive PAC-style guarantees that link the KL threshold to user-defined false positive and false negative rates. On the Prompt-WNQA benchmark, RePCS achieves a ROC-AUC of 0.918. This result outperforms the strongest prior method by 6.5 percentage points while keeping latency overhead below 4.7% on an NVIDIA T4 GPU. RePCS offers a lightweight, black-box safeguard to verify whether a RAG system meaningfully leverages retrieval, making it especially valuable in safety-critical applications. network state queries data memorization retrieval-augmented generation large language models KL divergence Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6992096","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":478708750,"identity":"f3219380-d327-4960-81bd-7e7aeea378be","order_by":0,"name":"Vu Anh Le","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAtklEQVRIiWNgGAWjYFACHsYDDBUSDGykaGE4wHAGqIUEPUAtjG1Ammgt/LN7DxzmnWchxyffwPjhB8PhPIJaJO6cSzjMu03CGOgwZskehsPFhK25kWNwcOY2icQ2oMOkGRjSEhsI6ZAHa5kD1sL8mygtBkAtBz42gLWwAW2xIazFEKTlwzGQXxLbLHsMiNAidyPH8EFCTZ2cfPPhwzd+VEgQ1oIEGIGKDUhQPwpGwSgYBaMANwAAX5o2Hnn8JpoAAAAASUVORK5CYII=","orcid":"","institution":"Vietnam Academy of Science and Technology","correspondingAuthor":true,"prefix":"","firstName":"Vu","middleName":"Anh","lastName":"Le","suffix":""},{"id":478708751,"identity":"f444503f-0514-494d-bdab-b6ea7617161d","order_by":1,"name":"Viet Anh Nguyen","email":"","orcid":"","institution":"Vietnam Academy of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Viet","middleName":"Anh","lastName":"Nguyen","suffix":""},{"id":478708757,"identity":"5c34e499-1ec2-4ae4-ad88-7588bf604b21","order_by":2,"name":"Mehmet Dik","email":"","orcid":"","institution":"Rockford University","correspondingAuthor":false,"prefix":"","firstName":"Mehmet","middleName":"","lastName":"Dik","suffix":""},{"id":478708765,"identity":"21b22fc7-c4f3-43de-a1b5-f3b3bb6e56b5","order_by":3,"name":"Van Nghia Luong","email":"","orcid":"","institution":"Dong A University","correspondingAuthor":false,"prefix":"","firstName":"Van","middleName":"Nghia","lastName":"Luong","suffix":""}],"badges":[],"createdAt":"2025-06-27 13:38:08","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6992096/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6992096/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":104551031,"identity":"7132a8c1-fae9-4672-a8db-d30c92270d66","added_by":"auto","created_at":"2026-03-13 08:12:33","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":986686,"visible":true,"origin":"","legend":"","description":"","filename":"revisedmanuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6992096/v1_covered_7a38adcf-7855-491b-946d-4d5fbf133e84.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"RePCS: Diagnosing Data Memorization in LLM-Powered Retrieval-Augmented Generation","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"network state queries, data memorization, retrieval-augmented generation, large language models, KL divergence","lastPublishedDoi":"10.21203/rs.3.rs-6992096/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6992096/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Retrieval-augmented generation (RAG) has become a common strategy for updating large language model (LLM) responses with current, external information. However, models may still rely on memorized training data, bypass the retrieved evidence, and produce contaminated outputs. We introduce Retrieval-Path Contamination Scoring (RePCS), a diagnostic method that detects such behavior without requiring model access or retraining. RePCS compares two inference paths: (i) a parametric path using only the query, and (ii) a retrieval-augmented path using both the query and retrieved context by computing the Kullback-Leibler (KL) divergence between their output distributions. A low divergence suggests that the retrieved context had minimal impact, indicating potential memorization. This procedure is model-agnostic, requires no gradient or internal state access, and adds only a single additional forward pass. We further derive PAC-style guarantees that link the KL threshold to user-defined false positive and false negative rates. On the Prompt-WNQA benchmark, RePCS achieves a ROC-AUC of 0.918. This result outperforms the strongest prior method by 6.5 percentage points while keeping latency overhead below 4.7% on an NVIDIA T4 GPU. RePCS offers a lightweight, black-box safeguard to verify whether a RAG system meaningfully leverages retrieval, making it especially valuable in safety-critical applications.","manuscriptTitle":"RePCS: Diagnosing Data Memorization in LLM-Powered Retrieval-Augmented Generation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-03 07:01:49","doi":"10.21203/rs.3.rs-6992096/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"f5ddce4e-52c6-480a-b48e-8c83ff9c2444","owner":[],"postedDate":"July 3rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-03-13T08:10:09+00:00","versionOfRecord":[],"versionCreatedAt":"2025-07-03 07:01:49","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6992096","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6992096","identity":"rs-6992096","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}