Out-of-Distribution Detection with Attention Head Masking for Multi-modal Document Classification

Out-of-Distribution Detection with Attention Head Masking for Multi-modal Document Classification | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Out-of-Distribution Detection with Attention Head Masking for Multi-modal Document Classification Christos Constantinou, Georgios Ioannides, Aman Chadha, Aaron Elkins, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6602601/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 03 Jan, 2026 Read the published version in Scientific Reports → Version 1 posted 10 You are reading this latest preprint version Abstract Detecting out-of-distribution (OOD) data is critical for ensuring the reliability and safety of deployed machine learning systems by mitigating model overconfidence and misclassification. While existing OOD detection methods primarily focus on uni-modal inputs, such as images or text, their effectiveness in multi-modal settings, particularly documents, remains underexplored. Moreover, most approaches prioritize decision mechanisms over optimizing the underlying dense embedding representations for optimal separation. In this work, we introduce Attention Head Masking (AHM), a novel technique applied to Transformer-based models for both uni-modal and multi-modal OOD detection. Our empirical results demonstrate that AHM enhances embedding quality, significantly improving the separation between in-distribution and OOD data. Notably, our method reduces the false positive rate (FPR) by up to 10%, outperforming state-of-the-art approaches. Furthermore, AHM generalizes effectively to multi-modal document data, where textual and visual information are jointly modeled within a Transformer architecture. To encourage further research in this area, we introduce FinanceDocs, a high-quality, publicly available document AI dataset tailored for OOD detection. Our code and dataset are publicly available. Physical sciences/Engineering Physical sciences/Mathematics and computing Physical sciences/Mathematics and computing/Computational science Full Text Additional Declarations No competing interests reported. Supplementary Files supplementartpartOOD.pdf Cite Share Download PDF Status: Published Journal Publication published 03 Jan, 2026 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 10 Jun, 2025 Reviews received at journal 08 Jun, 2025 Reviews received at journal 01 Jun, 2025 Reviewers agreed at journal 01 Jun, 2025 Reviewers agreed at journal 29 May, 2025 Reviewers invited by journal 29 May, 2025 Editor assigned by journal 29 May, 2025 Editor invited by journal 20 May, 2025 Submission checks completed at journal 20 May, 2025 First submitted to journal 06 May, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6602601","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":464554056,"identity":"4a6df852-3ae6-426b-8a8f-c9cb55293457","order_by":0,"name":"Christos Constantinou","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABQUlEQVRIie2QsWrDMBCGLwicRY5XhQx+BRlDoWDiVzljULZ26BJIIZ7cJSVroaGPkdlGkCyBrh082BQ8dWg3Z6uUupA0zt5B3yCd7vjQLwEYDP8RBpCddohephlYei8PLetcwd+TKuhB2WmFQDs5U+BU0XsvzX7u61Cc58cq3zdBCH25qfbN+Cbs2/m7/VLcDpzXvEQYu8AEHims2HJJUUQJFROfYnxHySD27XV9nbKYcITYS5g4fi5nAiSgRGD0agRIogVRhb2W3GLEYghEjSbJHyVvUIZaGTY4b5WVUhyplXmXklGUvUQpTBWtkigFYq3oACfB2JsKRoWIUrWMqNhGC0l9b7WpVbDY58i3Xkrr4+c7T4J8NUEQOurHhk0wix6WO6/8uC+4u8yr8nM6c52+4NCB1dXUyS+PDAaDwXCRb7acanXTyd0uAAAAAElFTkSuQmCC","orcid":"","institution":"University of Bristol","correspondingAuthor":true,"prefix":"","firstName":"Christos","middleName":"","lastName":"Constantinou","suffix":""},{"id":464554057,"identity":"c90c0c6f-0d6c-4997-af63-074ea1e003a5","order_by":1,"name":"Georgios Ioannides","email":"","orcid":"","institution":"Carnegie Mellon University","correspondingAuthor":false,"prefix":"","firstName":"Georgios","middleName":"","lastName":"Ioannides","suffix":""},{"id":464554058,"identity":"06d4603e-0cb4-436f-b72a-3fc324c2e34c","order_by":2,"name":"Aman Chadha","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Aman","middleName":"","lastName":"Chadha","suffix":""},{"id":464554059,"identity":"d3eacfab-3bf9-4c89-b19e-7030072353e8","order_by":3,"name":"Aaron Elkins","email":"","orcid":"","institution":"James Silberrad Brown Center for Artificial Intelligence, San Diego State University","correspondingAuthor":false,"prefix":"","firstName":"Aaron","middleName":"","lastName":"Elkins","suffix":""},{"id":464554060,"identity":"f9b8c363-992f-4331-a34c-282b9583260d","order_by":4,"name":"Edwin Simpson","email":"","orcid":"","institution":"University of Bristol","correspondingAuthor":false,"prefix":"","firstName":"Edwin","middleName":"","lastName":"Simpson","suffix":""}],"badges":[],"createdAt":"2025-05-06 11:38:27","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6602601/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6602601/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-025-32328-9","type":"published","date":"2026-01-03T15:58:10+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":99546123,"identity":"19cb6028-ca4b-40e0-9429-94d139cf5daa","added_by":"auto","created_at":"2026-01-05 16:10:03","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1121673,"visible":true,"origin":"","legend":"","description":"","filename":"OODpaper.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6602601/v1_covered_fa94c945-9ca0-4985-8486-4918fe3c8956.pdf"},{"id":83742788,"identity":"458f1c4d-bec5-43ea-ae42-5c2a406e2463","added_by":"auto","created_at":"2025-06-02 01:56:47","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":14198210,"visible":true,"origin":"","legend":"","description":"","filename":"supplementartpartOOD.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6602601/v1/078712d153149ab9d7530065.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Out-of-Distribution Detection with Attention Head Masking for Multi-modal Document Classification","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-6602601/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6602601/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Detecting out-of-distribution (OOD) data is critical for ensuring the reliability and safety of deployed machine learning systems by mitigating model overconfidence and misclassification. While existing OOD detection methods primarily focus on uni-modal inputs, such as images or text, their effectiveness in multi-modal settings, particularly documents, remains underexplored. Moreover, most approaches prioritize decision mechanisms over optimizing the underlying dense embedding representations for optimal separation. In this work, we introduce Attention Head Masking (AHM), a novel technique applied to Transformer-based models for both uni-modal and multi-modal OOD detection. Our empirical results demonstrate that AHM enhances embedding quality, significantly improving the separation between in-distribution and OOD data. Notably, our method reduces the false positive rate (FPR) by up to 10\\%, outperforming state-of-the-art approaches. Furthermore, AHM generalizes effectively to multi-modal document data, where textual and visual information are jointly modeled within a Transformer architecture. To encourage further research in this area, we introduce FinanceDocs, a high-quality, publicly available document AI dataset tailored for OOD detection. Our code and dataset are publicly available.","manuscriptTitle":"Out-of-Distribution Detection with Attention Head Masking for Multi-modal Document Classification","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-06-02 01:56:42","doi":"10.21203/rs.3.rs-6602601/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-06-10T10:15:23+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-06-09T00:13:22+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-06-02T02:58:10+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"1022675641852748264654800513423982113","date":"2025-06-02T00:19:27+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"23231487657904329059086663583824263386","date":"2025-05-29T19:32:15+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-05-29T11:37:04+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-05-29T11:34:28+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-05-20T08:39:39+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-05-20T05:21:40+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2025-05-06T11:28:12+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"7557749a-2c59-4e28-a670-a9fcdf140e97","owner":[],"postedDate":"June 2nd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":49327288,"name":"Physical sciences/Engineering"},{"id":49327289,"name":"Physical sciences/Mathematics and computing"},{"id":49327290,"name":"Physical sciences/Mathematics and computing/Computational science"}],"tags":[],"updatedAt":"2026-01-05T16:08:26+00:00","versionOfRecord":{"articleIdentity":"rs-6602601","link":"https://doi.org/10.1038/s41598-025-32328-9","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2026-01-03 15:58:10","publishedOnDateReadable":"January 3rd, 2026"},"versionCreatedAt":"2025-06-02 01:56:42","video":"","vorDoi":"10.1038/s41598-025-32328-9","vorDoiUrl":"https://doi.org/10.1038/s41598-025-32328-9","workflowStages":[]},"version":"v1","identity":"rs-6602601","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6602601","identity":"rs-6602601","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}