Membership Disclosure Evaluation for Synthetic Clinical Text Generated by LLM via Dynamic Few-Shot In-Context Learning

Membership Disclosure Evaluation for Synthetic Clinical Text Generated by LLM via Dynamic Few-Shot In-Context Learning | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Membership Disclosure Evaluation for Synthetic Clinical Text Generated by LLM via Dynamic Few-Shot In-Context Learning Sen Li, Fida K. Dankar, Khaled El Emam This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8997592/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 5 You are reading this latest preprint version Abstract Purpose : LLMs are increasingly used to generate synthetic clinical notes for data sharing, but membership inference attacks (MIAs) can reveal whether specific real notes appeared in prompts or training data. Prior studies often examine static few-shot prompts, i.e., fixed exemplar notes that remain the same across queries. We investigate MIAs on synthetic clinical text datasets as an evaluation metric for membership disclosure risk from the perspective of data custodians, focusing on synthetic outputs generated by dynamic few-shot ICL with LLMs. Our goal is to examine whether it is possible to estimate the membership disclosure risk of the generative pipeline using only the real dataset, by simulating MIAs on synthetic outputs and a limited adversarial dataset. Methods : We develop a custodial auditing framework based on sentence-embedding similarity between real notes and the synthetic corpus, using three features: Max (nearest-neighbor similarity), Top-3 (mean of three strongest similarities), and Mean (centroid alignment). We formalize membership under both static and dynamic ICL and introduce a population-aware partitioning protocol that preserves realistic priors for evaluation. We report F1 and an adjusted metric M=(F1-Fmax)/(1-Fmax), where Fmax denotes the F1 score achieved by the naive adversarial strategy of classifying all records as members. Experiments span static and dynamic ICL with $k \in \{1,2,3\}$ and include extensive resampling to assess stability. The entire pipeline is text-only and requires no access to prompts, gradients, or model weights. Results : Static 1-shot ICL exhibits high membership disclosure risk, with Max and Top-3 achieving strong attack performance (high F1 and large positive $M$), indicating substantial leakage. Increasing the number of shots to 2 or 3 in static ICL markedly reduces risk, yielding $M$ below conservative thresholds and bringing performance close to membership-preserving regimes. Dynamic ICL consistently suppresses disclosure across all $k$, with F1 near chance and $M \approx 0$ or slightly negative, signifying negligible exploitable signal. The largest contrast between regimes occurs at 1-shot, underscoring prompt reuse as the primary driver of risk. Conclusion : Dynamic few-shot ICL offers a practical mitigation for membership disclosure in synthetic clinical text. Static 1-shot prompting should be avoided for public releases; when static prompting is unavoidable, using 2–3 examples substantially mitigates leakage. Our text-only, model-agnostic auditing provides custodians with deployable tools and an interpretable metric to decide release readiness. Ongoing experiments will expand datasets, LLMs, and auxiliary data sizes, and assess robustness to domain drift and prompt formatting changes. MIA Synthetic Note LLM In-Context Learning Few-Shot Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Reviewers agreed at journal 09 Apr, 2026 Reviewers invited by journal 27 Mar, 2026 Editor assigned by journal 06 Mar, 2026 Submission checks completed at journal 06 Mar, 2026 First submitted to journal 28 Feb, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8997592","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":614087300,"identity":"89296a26-210e-47f5-8518-3acaf4df3c10","order_by":0,"name":"Sen Li","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA8UlEQVRIiWNgGAWjYBAC+QYGBmYGAwZ7CLcCiJmZG/BqMTgA0ZIIVnbgDEgLIwEtIDVAnADmHWwDkYS0sPceYGYoYEgwlz58TPrjvNpo/naglh8V23D7pedcAtgvln1paRIHtx3PnXGYsYGx58xt3NbcyDEAaWHccIbHDKjlWG4DUAszYxseLfffIGuZcyx3PkEtN3iQtTTU5G4gpMXgTI7BYQYDicSdPWzJFmeOHcjdCNRyEJ9f5NvPGD7+8cfG3pyH+eCNipq63HnnDx988KMCj8OA4AADgwQ4goDgMEyECADVUkeU4lEwCkbBKBhZAACJlVeKN+ZqXAAAAABJRU5ErkJggg==","orcid":"","institution":"University of Ottawa","correspondingAuthor":true,"prefix":"","firstName":"Sen","middleName":"","lastName":"Li","suffix":""},{"id":614087301,"identity":"81bd9759-7cf8-479f-a4a9-5818d97fccea","order_by":1,"name":"Fida K. Dankar","email":"","orcid":"","institution":"CHEO Research Institute","correspondingAuthor":false,"prefix":"","firstName":"Fida","middleName":"K.","lastName":"Dankar","suffix":""},{"id":614087302,"identity":"6c6d0134-af5d-4956-a774-3895a40c88c8","order_by":2,"name":"Khaled El Emam","email":"","orcid":"","institution":"University of Ottawa","correspondingAuthor":false,"prefix":"","firstName":"Khaled","middleName":"El","lastName":"Emam","suffix":""}],"badges":[],"createdAt":"2026-02-28 19:23:52","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8997592/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8997592/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":105741799,"identity":"4a976285-7814-46cf-8ad9-a175a4cf1afb","added_by":"auto","created_at":"2026-03-30 13:22:47","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":490128,"visible":true,"origin":"","legend":"","description":"","filename":"90MIASubmitAmendMar04.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8997592/v1_covered_d1e437e6-e6f4-43d1-ab84-8c316c707e26.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Membership Disclosure Evaluation for Synthetic Clinical Text Generated by LLM via Dynamic Few-Shot In-Context Learning","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-informatics-and-decision-making","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"midm","sideBox":"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/midm/default.aspx","title":"BMC Medical Informatics and Decision Making","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"MIA, Synthetic Note, LLM, In-Context Learning, Few-Shot","lastPublishedDoi":"10.21203/rs.3.rs-8997592/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8997592/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003ePurpose\u003c/strong\u003e: LLMs are increasingly used to generate synthetic clinical notes for data sharing, but membership inference attacks (MIAs) can reveal whether specific real notes appeared in prompts or training data. Prior studies often examine static few-shot prompts, i.e., fixed exemplar notes that remain the same across queries. We investigate MIAs on synthetic clinical text datasets as an evaluation metric for membership disclosure risk from the perspective of data custodians, focusing on synthetic outputs generated by dynamic few-shot ICL with LLMs. Our goal is to examine whether it is possible to estimate the membership disclosure risk of the generative pipeline using only the real dataset, by simulating MIAs on synthetic outputs and a limited adversarial dataset.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods\u003c/strong\u003e: We develop a custodial auditing framework based on sentence-embedding similarity between real notes and the synthetic corpus, using three features: Max (nearest-neighbor similarity), Top-3 (mean of three strongest similarities), and Mean (centroid alignment). We formalize membership under both static and dynamic ICL and introduce a population-aware partitioning protocol that preserves realistic priors for evaluation. We report F1 and an adjusted metric M=(F1-Fmax)/(1-Fmax), where Fmax denotes the F1 score achieved by the naive adversarial strategy of classifying all records as members. Experiments span static and dynamic ICL with $k \\in \\{1,2,3\\}$ and include extensive resampling to assess stability. The entire pipeline is text-only and requires no access to prompts, gradients, or model weights.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e: Static 1-shot ICL exhibits high membership disclosure risk, with Max and Top-3 achieving strong attack performance (high F1 and large positive $M$), indicating substantial leakage. Increasing the number of shots to 2 or 3 in static ICL markedly reduces risk, yielding $M$ below conservative thresholds and bringing performance close to membership-preserving regimes. Dynamic ICL consistently suppresses disclosure across all $k$, with F1 near chance and $M \\approx 0$ or slightly negative, signifying negligible exploitable signal. The largest contrast between regimes occurs at 1-shot, underscoring prompt reuse as the primary driver of risk.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion\u003c/strong\u003e: Dynamic few-shot ICL offers a practical mitigation for membership disclosure in synthetic clinical text. Static 1-shot prompting should be avoided for public releases; when static prompting is unavoidable, using 2–3 examples substantially mitigates leakage. Our text-only, model-agnostic auditing provides custodians with deployable tools and an interpretable metric to decide release readiness. Ongoing experiments will expand datasets, LLMs, and auxiliary data sizes, and assess robustness to domain drift and prompt formatting changes.\u003c/p\u003e","manuscriptTitle":"Membership Disclosure Evaluation for Synthetic Clinical Text Generated by LLM via Dynamic Few-Shot In-Context Learning","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-30 13:22:32","doi":"10.21203/rs.3.rs-8997592/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewerAgreed","content":"19862378675747268313530638567748061977","date":"2026-04-09T20:45:04+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-03-27T09:06:37+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-03-06T11:53:24+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-03-06T11:47:50+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Medical Informatics and Decision Making","date":"2026-02-28T19:09:48+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-informatics-and-decision-making","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"midm","sideBox":"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/midm/default.aspx","title":"BMC Medical Informatics and Decision Making","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"5186d9b6-4a45-4e01-b1f8-12037ae84489","owner":[],"postedDate":"March 30th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-03-30T13:22:32+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-30 13:22:32","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8997592","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8997592","identity":"rs-8997592","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}