Same Prompt, Different Answer: Exposing the Reproducibility Illusion in Large Language Model APIs

Same Prompt, Different Answer: Exposing the Reproducibility Illusion in Large Language Model APIs | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Same Prompt, Different Answer: Exposing the Reproducibility Illusion in Large Language Model APIs Lucas Rover, Hugo Siqueira, Anibal Azevedo, Eduardo Bacalhau, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9096283/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted You are reading this latest preprint version Abstract The same prompt sent twice to a large language model API under documented "deterministic" settings can return different answers, yet this variation is invisible to users. Here we report 4,104 controlled experiments across eight models and five API providers showing that, under temperature-zero greedy decoding with fixed seeds, API-served models reproduce their own outputs only 22.1% of the time, while locally deployed models achieve 95.6%, a gap exceeding four-fold. Non-determinism persists in multi-turn and retrieval-augmented generation workflows, where one model produces zero exact matches across 50 runs, yet remains hidden because outputs are semantically equivalent (BERTScore F1 > 0.97). A quasi-isolation experiment identifies production infrastructure complexity, rather than cloud deployment itself, as the driver. We provide a lightweight provenance protocol (<1% overhead) that makes this variation detectable, raising a reliability concern for the growing use of LLMs in medicine, physical sciences, and automated data analysis. Scientific community and society/Scientific community/Research data/Databases Business and commerce/Information systems and information technology/Library science Reproducibility Large language models Non-determinism Provenance API inference Full Text Additional Declarations There is NO Competing Interest. Supplementary Files 02SupplementaryInformation.pdf Supplementary Information Cite Share Download PDF Status: Under Review Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9096283","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":604933646,"identity":"76f73a94-7b05-4678-86bb-b18331bcda88","order_by":0,"name":"Lucas Rover","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABIUlEQVRIie3RMUvDQBTA8SsH6fI060kg/QpXAlWw0K9yIZCpQosgjgeFdCm6xsmvEHHp0OHCrYdzBZFKIJNDsqWbl2hRaAodHe4/HG+4H+/gEDKZ/mGnvyMWaLObb5uTIdxCrD8j03eaOlwdR4AeSUjwUW5XQ//eUWXhR29uj8P5Jl2+I7s7pni6aiGh50Ae+g93V8/Ej3KPCujzVF2js8UnxXHeQsbIQUJ6VJ0kmkg/QdCflRFDdK23gGgjeFtpMlKQVTV55PUWTUaHiUVASJcCWM0WLn4IJQcI5INLEKFLlDW4YC/6hdKaxqliQFQ+kfE+sbtB9lqJIdgLnK2LG+n25rOkSJfMtefBUzbZJy3Vf9HhCOr5KPCdJiaTyWTa9QXIv2v4BLnpKwAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0001-6641-9224","institution":"Universidade Tecnológica Federal do Paraná","correspondingAuthor":true,"prefix":"","firstName":"Lucas","middleName":"","lastName":"Rover","suffix":""},{"id":604933647,"identity":"65f68ced-a8b7-4f68-84a2-f1457d1adf64","order_by":1,"name":"Hugo Siqueira","email":"","orcid":"","institution":"UNIVERSIDADE TECNOLÓGICA FEDERAL DO PARANÁ","correspondingAuthor":false,"prefix":"","firstName":"Hugo","middleName":"","lastName":"Siqueira","suffix":""},{"id":604933648,"identity":"66ac66cf-6e14-4c47-bc0b-a6cce1b83f5a","order_by":2,"name":"Anibal Azevedo","email":"","orcid":"","institution":"Universidade de Campinas (UNICAMP)","correspondingAuthor":false,"prefix":"","firstName":"Anibal","middleName":"","lastName":"Azevedo","suffix":""},{"id":604933649,"identity":"41a1af36-3bf5-499f-9d3c-90980716cd61","order_by":3,"name":"Eduardo Bacalhau","email":"","orcid":"","institution":"Universidade Federal do Paraná (UFPR)","correspondingAuthor":false,"prefix":"","firstName":"Eduardo","middleName":"","lastName":"Bacalhau","suffix":""},{"id":604933650,"identity":"8b2c98c7-8960-42fc-b6fd-79c44870a145","order_by":4,"name":"Yara Tadano","email":"","orcid":"","institution":"Universidade Tecnológica Federal do Paraná (UTFPR)","correspondingAuthor":false,"prefix":"","firstName":"Yara","middleName":"","lastName":"Tadano","suffix":""}],"badges":[],"createdAt":"2026-03-11 15:45:14","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9096283/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9096283/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":104781489,"identity":"a4d7157c-a5e1-4bb5-bbcc-a3ad473d6f56","added_by":"auto","created_at":"2026-03-17 07:55:47","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":385341,"visible":true,"origin":"","legend":"Article File","description":"","filename":"01Manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9096283/v1_covered_29b79e2d-4b47-4f02-91bd-2678d0e27b82.pdf"},{"id":104559171,"identity":"cbaf999f-f336-4800-9098-d6b5df6f4724","added_by":"auto","created_at":"2026-03-13 09:52:06","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":319097,"visible":true,"origin":"","legend":"Supplementary Information","description":"","filename":"02SupplementaryInformation.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9096283/v1/943d825672e31b207c1e1167.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Same Prompt, Different Answer: Exposing the Reproducibility Illusion in Large Language Model APIs","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Reproducibility, Large language models, Non-determinism, Provenance, API inference","lastPublishedDoi":"10.21203/rs.3.rs-9096283/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9096283/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"The same prompt sent twice to a large language model API under documented \"deterministic\" settings can return different answers, yet this variation is invisible to users. Here we report 4,104 controlled experiments across eight models and five API providers showing that, under temperature-zero greedy decoding with fixed seeds, API-served models reproduce their own outputs only 22.1% of the time, while locally deployed models achieve 95.6%, a gap exceeding four-fold. Non-determinism persists in multi-turn and retrieval-augmented generation workflows, where one model produces zero exact matches across 50 runs, yet remains hidden because outputs are semantically equivalent (BERTScore F1 \u003e 0.97). A quasi-isolation experiment identifies production infrastructure complexity, rather than cloud deployment itself, as the driver. We provide a lightweight provenance protocol (\u003c1% overhead) that makes this variation detectable, raising a reliability concern for the growing use of LLMs in medicine, physical sciences, and automated data analysis.","manuscriptTitle":"Same Prompt, Different Answer: Exposing the Reproducibility Illusion in Large Language Model APIs","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-13 09:52:02","doi":"10.21203/rs.3.rs-9096283/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-communications","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"NCOMMS","sideBox":"Learn more about [Nature Communications](http://www.nature.com/ncomms/)","snPcode":"","submissionUrl":"https://mts-ncomms.nature.com/","title":"Nature Communications","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Communications","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"37c73265-0b6b-4937-873e-ee8e74914e19","owner":[],"postedDate":"March 13th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":64375982,"name":"Scientific community and society/Scientific community/Research data/Databases"},{"id":64375983,"name":"Business and commerce/Information systems and information technology/Library science"}],"tags":[],"updatedAt":"2026-03-20T12:36:18+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-13 09:52:02","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9096283","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9096283","identity":"rs-9096283","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}