Certainty-Validity: A Diagnostic Framework for Discrete Commitment Systems

Certainty-Validity: A Diagnostic Framework for Discrete Commitment Systems | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Certainty-Validity: A Diagnostic Framework for Discrete Commitment Systems Datorien L. Anderson This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8845570/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Standard evaluation metrics for machine learning—accuracy, precision, recall, and AUROC— assume that all errors are equivalent: a confident incorrect prediction is penalized identically to an uncertain one. For discrete commitment systems (architectures that select committed states {−W, 0,+W}), this assumption is epistemologically flawed. We introduce the Certainty- Validity (CVS) Framework, a diagnostic method that decomposes model performance into a 2×2 matrix distinguishing high/low certainty from valid/invalid predictions. This framework reveals a critical failure mode hidden by standard accuracy: Confident-Incorrect (CI) behavior, where models hallucinate structure in ambiguous data. Through ablation experiments on Fashion-MNIST, EMNIST, and IMDB, we analyze the “83% Ambiguity Ceiling”—a stopping point where this specific discrete architecture consistently plateaus on noisy benchmarks. Unlike continuous models that can surpass this ceiling by memorizing texture or statistical noise, the discrete model refuses to commit to ambiguous samples. We show that this refusal is not a failure but a feature: the model stops where structural evidence ends. However, standard training on ambiguous data eventually forces Benign Overfitting, causing a pathological migration from Uncertain-Incorrect (UI) (appropriate doubt) to Confident-Incorrect (CI) (hallucination). We propose that “good training” for reasoning systems must be defined not by accuracy, but by maximizing the Certainty-Validity Score (CVS)—ensuring the model knows where to stop. Artificial Intelligence and Machine Learning Discrete Commitment Systems Certainty-Validity Score (CVS) Ambiguity Ceiling Benign Overfitting Platonic Spike Epistemic Calibration Confident-Incorrect (CI) Uncertain-Incorrect (UI) Hallucination Full Text Additional Declarations The authors declare no competing interests. Supplementary Files certaintyvalidity.py Certainty-Validity Evaluation Tool Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8845570","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":589242365,"identity":"d57dab32-9d7d-4a3c-b001-9149588004e0","order_by":0,"name":"Datorien L. Anderson","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA40lEQVRIiWNgGAWjYDCCAyDCgIGBH0QzNhClhRmiRbIBokWCSC0gXQeI1cJ3/PzBxxUFNnnGx5sffubdcbiOn4H54aMbeLRInklmNjxjkFZsduaYsTTvmcMSkg1sxsY5eLQYHEhmk2wwOJy47UYOGzNv22EJgwM8bNJ4tZx/DNLyP3HzDKgWe4JaboBtOZC4QQJmCwMBLZI3HhsbNhgkJ84A+kVy7pl0yRmHCfiF73ziw4cNf+wS+9ubH354u8Oanx/IeIxPCwpg4mFoZmBgJlY5CDD+YKgjRf0oGAWjYBSMEAAAf5pNSAc88x4AAAAASUVORK5CYII=","orcid":"https://orcid.org/0009-0004-7688-6829","institution":"","correspondingAuthor":true,"prefix":"","firstName":"Datorien","middleName":"L.","lastName":"Anderson","suffix":""}],"badges":[],"createdAt":"2026-02-10 22:41:07","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-8845570/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8845570/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":102745899,"identity":"7474a0d6-aab1-46e0-94b3-c4f3486bb2ae","added_by":"auto","created_at":"2026-02-16 08:54:35","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":715856,"visible":true,"origin":"","legend":"","description":"","filename":"references.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8845570/v1_covered_3f15f625-c808-4802-9f6c-d0c05a130bce.pdf"},{"id":102470202,"identity":"503f91ee-6fde-429f-821c-6729379e4f73","added_by":"auto","created_at":"2026-02-12 03:40:39","extension":"py","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":17369,"visible":true,"origin":"","legend":"\u003cp\u003eCertainty-Validity Evaluation Tool\u003c/p\u003e","description":"","filename":"certaintyvalidity.py","url":"https://assets-eu.researchsquare.com/files/rs-8845570/v1/ee33e7c223132738f14cdc3d.py"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eCertainty-Validity: A Diagnostic Framework for Discrete Commitment Systems\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Discrete Commitment Systems, Certainty-Validity Score (CVS), Ambiguity Ceiling, Benign Overfitting, Platonic Spike, Epistemic Calibration, Confident-Incorrect (CI), Uncertain-Incorrect (UI), Hallucination","lastPublishedDoi":"10.21203/rs.3.rs-8845570/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8845570/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eStandard evaluation metrics for machine learning—accuracy, precision, recall, and AUROC— assume that all errors are equivalent: a confident incorrect prediction is penalized identically to an uncertain one. For discrete commitment systems (architectures that select committed states {−W, 0,+W}), this assumption is epistemologically flawed. We introduce the Certainty- Validity (CVS) Framework, a diagnostic method that decomposes model performance into a 2×2 matrix distinguishing high/low certainty from valid/invalid predictions. This framework reveals a critical failure mode hidden by standard accuracy: Confident-Incorrect (CI) behavior, where models hallucinate structure in ambiguous data. Through ablation experiments on Fashion-MNIST, EMNIST, and IMDB, we analyze the “83% Ambiguity Ceiling”—a stopping point where this specific discrete architecture consistently plateaus on noisy benchmarks. Unlike continuous models that can surpass this ceiling by memorizing texture or statistical noise, the discrete model refuses to commit to ambiguous samples. We show that this refusal is not a failure but a feature: the model stops where structural evidence ends. However, standard training on ambiguous data eventually forces Benign Overfitting, causing a pathological migration from Uncertain-Incorrect (UI) (appropriate doubt) to Confident-Incorrect (CI) (hallucination). We propose that “good training” for reasoning systems must be defined not by accuracy, but by maximizing the Certainty-Validity Score (CVS)—ensuring the model knows where to stop.\u003c/p\u003e","manuscriptTitle":"Certainty-Validity: A Diagnostic Framework for Discrete Commitment Systems","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-02-12 03:40:34","doi":"10.21203/rs.3.rs-8845570/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"0aefec0a-0554-495a-8345-70f7d5ac61bb","owner":[],"postedDate":"February 12th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":62696167,"name":"Artificial Intelligence and Machine Learning"}],"tags":[],"updatedAt":"2026-02-12T03:40:34+00:00","versionOfRecord":[],"versionCreatedAt":"2026-02-12 03:40:34","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8845570","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8845570","identity":"rs-8845570","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}