The Dice Roll Method: A Standardized Protocol for Measuring Stochastic Bias in Large Language Model Outputs

The Dice Roll Method: A Standardized Protocol for Measuring Stochastic Bias in Large Language Model Outputs | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article The Dice Roll Method: A Standardized Protocol for Measuring Stochastic Bias in Large Language Model Outputs Dmitrij Żatuchin This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8980233/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Researchers increasingly use repeated identical prompts to measure stochastic bias in large language model (LLM) outputs, yet no standardized protocol exists for determining adequate iteration counts, selecting appropriate stability metrics, or establishing reliability thresholds. This paper formalizes the \emph{Dice Roll Method} as a reusable audit protocol through meta-methodology combining reanalysis of five empirical studies (approximately 190,000 observations across three to five LLMs, 270+ brands, 6 languages, and iteration counts from 5 to 40) with Monte Carlo power simulation (10,000 replications per condition). Our power analysis demonstrates that $n = 5$ iterations achieves adequate statistical power ($> 0.80$) only for large effects (Cohen's $d > 0.8$); medium effects ($d = 0.5$) require $n \geq 15$, and small effects ($d = 0.2$) remain undetectable below $n = 40$. Metric convergence follows a logarithmic curve, with 80% of asymptotic precision achieved by $n = 7$ and 90% by $n = 10$. Test-retest reliability reaches acceptable levels (ICC $\geq 0.70$) at $n \geq 8$ for brand count means. Cross-metric correlation analysis reveals that count-based metrics (coefficient of variation, Gini coefficient) and embedding-based metrics (cosine similarity) capture partially orthogonal information (Spearman $r = 0.31$--$0.47$), supporting complementary metric batteries. We provide power analysis lookup tables, a metric selection decision tree, and a reproducible Python implementation. These findings establish minimum methodological standards for LLM auditing research and enable researchers to justify iteration counts through formal power analysis. Large language models Stochastic bias Power analysis Repeated-query auditing Monte Carlo simulation Reproducibility Full Text Additional Declarations Competing interest reported. D. Zatuchin is affiliated with Rankfor.AI, which develops AI brand intelligence tools. The Dice Roll Method protocol described in this paper is implemented in the open-source Dice Roller package. The research was conducted independently; the company had no influence on study design, methodology, analysis, or conclusions. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8980233","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":611649139,"identity":"b64ad2f7-3f30-45ee-97df-b0286bdaae38","order_by":0,"name":"Dmitrij Żatuchin","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA90lEQVRIiWNgGAWjYDCCA4wNDAwGbBAOY4MEAz8DAzMhLY0NEC3MEC2SDQS1MICsYYBpAdp4gIAWvmuH2x9XFPAxyLefPybxc4dFnvHx5sMGDBU2OLVI3k5sbDwDdBhjTzKbZO8ZiWKzM8eSExjOpOHUYgDS0gDUwsyQzCbN2CaRuO1GjvEBxrbDhLWw8T+GaNk8//3nA4z//hPWwiMBtWWDBA9zAmPDAbx+mQnUwiMh8djYsheoZcaZNGODhGPJOLXw3U5/8LHhzzE5+f7Ehzd+ttUl9rcffizxocYOpxYoOMaDyk8gpIGBoYawklEwCkbBKBi5AAASpFQDP8oRowAAAABJRU5ErkJggg==","orcid":"","institution":"Estonian Entrepreneurship University of Applied Sciences","correspondingAuthor":true,"prefix":"","firstName":"Dmitrij","middleName":"","lastName":"Żatuchin","suffix":""}],"badges":[],"createdAt":"2026-02-26 17:08:18","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-8980233/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8980233/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":107487320,"identity":"8c3020e4-1e1a-4f33-a350-c50459d786ee","added_by":"auto","created_at":"2026-04-22 02:40:40","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":275352,"visible":true,"origin":"","legend":"","description":"","filename":"dicerollmethodsubmission.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8980233/v1_covered_b5fe467a-7180-4d4c-ba9b-197c6fdfbc5a.pdf"}],"financialInterests":"Competing interest reported. D. Zatuchin is affiliated with Rankfor.AI, which develops AI brand intelligence tools. The Dice Roll Method protocol described in this paper is implemented in the open-source Dice Roller package. The research was conducted independently; the company had no influence on study design, methodology, analysis, or conclusions.","formattedTitle":"The Dice Roll Method: A Standardized Protocol for Measuring Stochastic Bias in Large Language Model Outputs","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Large language models, Stochastic bias, Power analysis, Repeated-query auditing, Monte Carlo simulation, Reproducibility","lastPublishedDoi":"10.21203/rs.3.rs-8980233/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8980233/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Researchers increasingly use repeated identical prompts to measure stochastic bias in large language model (LLM) outputs, yet no standardized protocol exists for determining adequate iteration counts, selecting appropriate stability metrics, or establishing reliability thresholds. This paper formalizes the \\emph{Dice Roll Method} as a reusable audit protocol through meta-methodology combining reanalysis of five empirical studies (approximately 190,000 observations across three to five LLMs, 270+ brands, 6 languages, and iteration counts from 5 to 40) with Monte Carlo power simulation (10,000 replications per condition). Our power analysis demonstrates that $n = 5$ iterations achieves adequate statistical power ($\u003e 0.80$) only for large effects (Cohen's $d \u003e 0.8$); medium effects ($d = 0.5$) require $n \\geq 15$, and small effects ($d = 0.2$) remain undetectable below $n = 40$. Metric convergence follows a logarithmic curve, with 80\\% of asymptotic precision achieved by $n = 7$ and 90\\% by $n = 10$. Test-retest reliability reaches acceptable levels (ICC $\\geq 0.70$) at $n \\geq 8$ for brand count means. Cross-metric correlation analysis reveals that count-based metrics (coefficient of variation, Gini coefficient) and embedding-based metrics (cosine similarity) capture partially orthogonal information (Spearman $r = 0.31$--$0.47$), supporting complementary metric batteries. We provide power analysis lookup tables, a metric selection decision tree, and a reproducible Python implementation. These findings establish minimum methodological standards for LLM auditing research and enable researchers to justify iteration counts through formal power analysis.","manuscriptTitle":"The Dice Roll Method: A Standardized Protocol for Measuring Stochastic Bias in Large Language Model Outputs","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-26 19:11:58","doi":"10.21203/rs.3.rs-8980233/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"cd629ee5-1cd4-403a-8ad0-45ba72e849a3","owner":[],"postedDate":"March 26th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-04-20T16:40:11+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-26 19:11:58","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8980233","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8980233","identity":"rs-8980233","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}