PTRR: A Metacognitive Framework for Measuring and Mitigating Automation Bias in AI-Assisted Vulnerability Research

PTRR: A Metacognitive Framework for Measuring and Mitigating Automation Bias in AI-Assisted Vulnerability Research | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article PTRR: A Metacognitive Framework for Measuring and Mitigating Automation Bias in AI-Assisted Vulnerability Research Ziad Salah, Ashraf A. Mohamed This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9247251/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Artificial intelligence is increasingly integrated into professional cybersecurity workflows, yet the cognitive effects of AI assistance on researcher behavior remain poorly understood. This paper introduces the PTRR Framework, a structured metacognitive instrument designed to measure and mitigate Automation Bias in AI-assisted vulnerability research. PTRR comprises four components: Prompts (interaction quality scoring), Time (temporal behavioral analysis), Results (outcome-based severity scoring), and Rubric (process independence and verification criteria). Three derived indices operationalize the core constructs: the Automation Bias Index (ABI), which quantifies uncritical reliance on AI output; the Cognitive Struggle Index (CSI), which measures productive independent effort; and the Tool Integration Intensity Score (TIIS), which captures deliberate multi-tool synthesis. The framework is grounded in Dual Process Theory and Cognitive Load Theory, which together predict that expertise moderates the relationship between AI interaction quality and research output. A preliminary single-participant case study conducted over 60 structured hours provides initial construct validity evidence. The case study documented a shift from a low-to-medium severity finding profile to a critical-dominant profile following PTRR-based workflow adoption, including seven critical-severity findings validated by independent program triage and three multi-layer escalation chains. Total accepted findings increased from 2 to 11 over equivalent time periods. These results motivate the formal multi-participant study design proposed here as future work, which employs a within-subject Phase A versus Phase B design, Hierarchical Linear Modeling, and independent human scoring with pre-defined inter-rater reliability thresholds. Physical sciences/Engineering Physical sciences/Mathematics and computing Biological sciences/Psychology Social science/Psychology Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9247251","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":625993372,"identity":"0dea0a55-a170-47ee-aeb5-01aa9ac99f29","order_by":0,"name":"Ziad Salah","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABB0lEQVRIiWNgGAWjYDCCA2AEAR8SGGyAFGPjAVyqUbWwMTDOSGBIA2lpIKiFAa6FgeEwqiA2wHf87MPDBb9s7OXnNz9seFBx3m5t+2GgLTU20bi0SJ5JNzg8sy8tsbGNzbAh4czt5G1nEoFajqXlNuDQYnAgjeEwb8/hBGY2BvMHiW23k80OALUwNhzGreX8M7AWezY29o8NiW3nks3OPySg5QbQFp4fhxl72HgMgVoO2JndIGCL5A2QLQ1piTPYcgqBfklOMLsBtCUBj1/4zqcxf+b5Awyx5uMbG39U2NmbnU9/+OBDjQ1OLWDA2IZgJ4JVJuBTDgZ/EEx7gopHwSgYBaNgxAEACZdrEjG1YlIAAAAASUVORK5CYII=","orcid":"","institution":"","correspondingAuthor":true,"prefix":"","firstName":"Ziad","middleName":"","lastName":"Salah","suffix":""},{"id":625993410,"identity":"107f5f2f-350e-4911-a2ae-5a570ea3270d","order_by":1,"name":"Ashraf A. Mohamed","email":"","orcid":"","institution":"Elsewedy University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Ashraf","middleName":"A.","lastName":"Mohamed","suffix":""}],"badges":[],"createdAt":"2026-03-27 17:24:27","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9247251/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9247251/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":109513315,"identity":"22e31f54-7555-4f48-b9fd-936350b8809c","added_by":"auto","created_at":"2026-05-19 04:56:03","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":366985,"visible":true,"origin":"","legend":"","description":"","filename":"TemplateforsubmissionstoScientificReports.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9247251/v1_covered_053ebc74-e5f9-4e36-92a3-71a3be37f664.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"PTRR: A Metacognitive Framework for Measuring and Mitigating Automation Bias in AI-Assisted Vulnerability Research","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-9247251/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9247251/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eArtificial intelligence is increasingly integrated into professional cybersecurity workflows, yet the cognitive effects of AI assistance on researcher behavior remain poorly understood. This paper introduces the PTRR Framework, a structured metacognitive instrument designed to measure and mitigate Automation Bias in AI-assisted vulnerability research. PTRR comprises four components: Prompts (interaction quality scoring), Time (temporal behavioral analysis), Results (outcome-based severity scoring), and Rubric (process independence and verification criteria). Three derived indices operationalize the core constructs: the Automation Bias Index (ABI), which quantifies uncritical reliance on AI output; the Cognitive Struggle Index (CSI), which measures productive independent effort; and the Tool Integration Intensity Score (TIIS), which captures deliberate multi-tool synthesis. The framework is grounded in Dual Process Theory and Cognitive Load Theory, which together predict that expertise moderates the relationship between AI interaction quality and research output. A preliminary single-participant case study conducted over 60 structured hours provides initial construct validity evidence. The case study documented a shift from a low-to-medium severity finding profile to a critical-dominant profile following PTRR-based workflow adoption, including seven critical-severity findings validated by independent program triage and three multi-layer escalation chains. Total accepted findings increased from 2 to 11 over equivalent time periods. These results motivate the formal multi-participant study design proposed here as future work, which employs a within-subject Phase A versus Phase B design, Hierarchical Linear Modeling, and independent human scoring with pre-defined inter-rater reliability thresholds.\u003c/p\u003e","manuscriptTitle":"PTRR: A Metacognitive Framework for Measuring and Mitigating Automation Bias in AI-Assisted Vulnerability Research","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-21 14:45:13","doi":"10.21203/rs.3.rs-9247251/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e30a84eb-d9f9-4968-817a-088294a3a241","owner":[],"postedDate":"April 21st, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":66746718,"name":"Physical sciences/Engineering"},{"id":66746719,"name":"Physical sciences/Mathematics and computing"},{"id":66746720,"name":"Biological sciences/Psychology"},{"id":66746721,"name":"Social science/Psychology"}],"tags":[],"updatedAt":"2026-05-19T04:55:06+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-21 14:45:13","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9247251","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9247251","identity":"rs-9247251","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}