Sensitivity analysis to estimate bias-corrected validity measures in outcome validation studies under the “all possible cases” assumption in routinely-collected health databases

Sensitivity analysis to estimate bias-corrected validity measures in outcome validation studies under the “all possible cases” assumption in routinely-collected health databases | Authorea try { document.documentElement.classList.add('js'); } catch (e) { } var _gaq = _gaq || []; _gaq.push(['_setAccount', 'G-8VDV14Y67G']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); Skip to main content Preprints Collections Wiley Open Research IET Open Research Ecological Society of Japan All Collections About About Authorea FAQs Contact Us Quick Search anywhere Search for preprint articles, keywords, etc. Search Search ADVANCED SEARCH SCROLL This is a preprint and has not been peer reviewed. Data may be preliminary. 28 March 2025 V1 Latest version Share on Sensitivity analysis to estimate bias-corrected validity measures in outcome validation studies under the “all possible cases” assumption in routinely-collected health databases Authors : Norihiro Suzuki 0000-0001-7913-8595 [email protected] , Masataka Taguri , Koichiro Shiba , Masao Iwagami , and Takuhiro Yamaguchi Authors Info & Affiliations https://doi.org/10.22541/au.174316694.46504024/v1 548 views 170 downloads Contents Abstract Supplementary Material Information & Authors Metrics & Citations View Options References Figures Tables Media Share Abstract Purpose In validation studies to assess algorithms identifying subjects’ outcome status, researchers often use a sampling method with the ”all possible cases” assumption that all true cases in the database are included in the sample. No study has quantitatively assessed how the extent of missing true cases might bias the estimated performance measures. This study aims to quantify the magnitude of biases and propose a sensitivity analysis method. Methods We first formulate the bias in each performance measure under the violation of the assumption. Using these bias formulas, we propose a sensitivity analysis method to quantify the magnitude of biases and compute bias-corrected estimates. Also, we proposed a sampling approach that is helpful for sensitivity analysis. Finally, as motivating examples, we apply our proposed method to the data from two validation studies that evaluated the performance of case-finding algorithms created by medical claims data. Results We showed that the violation of the assumption does not bias positive predictive value ( PPV ), while it leads to overestimated sensitivity ( Se ), specificity ( Sp ), and negative predictive value ( NPV ). Our bias formula and example indicate that Se varies greatly depending on the missed true cases, while Sp and NPV are relatively robust under rare outcome situations. Conclusions The deviation from the assumption provides overestimated validation measure values except PPV . This implies a risk of misleading researchers into overestimating the performance of the algorithms. Our proposed sampling option would be useful to investigate whether the assumption is violated and reliably determine the upper limit of the sensitivity parameter. Supplementary Material File (pds-25-0239-file001.docx) Download 1.55 MB Information & Authors Information Version history V1 Version 1 28 March 2025 Copyright This work is licensed under a Non Exclusive No Reuse License. Keywords all possible cases case definition database study positive predictive value sensitivity sensitivity analysis validation study Authors Affiliations Norihiro Suzuki 0000-0001-7913-8595 [email protected] Tokyo Ika Daigaku View all articles by this author Masataka Taguri Tokyo Ika Daigaku View all articles by this author Koichiro Shiba Boston University Department of Epidemiology View all articles by this author Masao Iwagami Tsukuba Daigaku - Tsukuba Campus Kasuga Chiku View all articles by this author Takuhiro Yamaguchi Tohoku Daigaku View all articles by this author Metrics & Citations Metrics Article Usage 548 views 170 downloads .FvxKWukQNSOunydq8rnd { width: 100px; } Citations Download citation Norihiro Suzuki, Masataka Taguri, Koichiro Shiba, et al. Sensitivity analysis to estimate bias-corrected validity measures in outcome validation studies under the “all possible cases” assumption in routinely-collected health databases. Authorea . 28 March 2025. DOI: https://doi.org/10.22541/au.174316694.46504024/v1 If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download. For more information or tips please see 'Downloading to a citation manager' in the Help menu . Format Please select one from the list RIS (ProCite, Reference Manager) EndNote BibTex Medlars RefWorks Direct import Tips for downloading citations document.getElementById('citMgrHelpLink').addEventListener('click', function() { popupHelp(this.href); return false; }); $(".js__slcInclude").on("change", function(e){ if ($(this).val() == 'refworks') $('#direct').prop("checked", false); $('#direct').prop("disabled", ($(this).val() == 'refworks')); }); View Options View options PDF View PDF Figures Tables Media Share Share Share article link Copy Link Copied! Copying failed. Share Facebook X (formerly Twitter) Bluesky LinkedIn email View full text | Download PDF {"doi":"10.22541/au.174316694.46504024/v1","type":"Article"} Now Reading: Share Figures Tables Close figure viewer Back to article Figure title goes here Change zoom level Go to figure location within the article Download figure Toggle share panel Toggle share panel Share Toggle information panel Toggle information panel Go to previous graphic Go to next graphic Go to previous table Go to next table All figures All tables View all material View all material xrefBack.goTo xrefBack.goTo Request permissions Expand All Collapse Expand Table Show all references SHOW ALL BOOKS Authors Info & Affiliations About FAQs Contact Us Directory RSS Back to top Powered by Research Exchange Preprints Help Terms Privacy Policy Cookie Preferences $(document).ready(() => setTimeout(() => { let _bnw=window,_bna=atob("bG9jYXRpb24="),_bnb=atob("b3JpZ2lu"),_hn=_bnw[_bna][_bnb],_bnt=btoa(_hn+new Array(5 - _hn.length % 4).join(" ")); $.get("/resource/lodash?t="+_bnt); },4000)); (function(){function c(){var b=a.contentDocument||a.contentWindow.document;if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'a00605667ea2df88',t:'MTc3OTU1OTg2NQ=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getElementsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();