Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection

Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection Rijha Safdar, Danyail Mateen, Syed Taha Ali, Umer Ashfaq, Wajahat Hussain This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7821055/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 9 You are reading this latest preprint version Abstract AI-based solutions demonstrate re- markable results in identifying vulnerabilities in software, but research has consistently found that this performance does not generalize to unseen codebases. In this paper, we specifically investi- gate the impact of model architecture, parameter configuration, and quality of training data on the ability of these systems to generalize. For this purpose, we introduce VulGate, a high quality state of the art dataset that mitigates the shortcomings of prior datasets, by removing mis- labeled and duplicate samples, updating new vul- nerabilities, incorporating additional metadata, in- tegrating hard samples, and including dedicated test sets. We undertake a series of experiments to demonstrate that improved dataset diversity and quality substantially enhances vulnerability detec- tion. We also introduce and benchmark multiple encoder-only and decoder-only models. We find that encoder-based models outperform other mod- els in terms of accuracy and generalization. Our model achieves 6.8% improvement in recall on the benchmark BigVul dataset and outperforms others on unseen projects, demonstrating enhanced gen- eralizability. Our results highlight the role of data quality and model selection in the development of robust vulnerability detection systems. Our find- ings suggest a direction for future systems with high cross-project effectiveness. LLM fine-tuning Encoder-only Vulnerability Detection Generalizability Codebases Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 11 Feb, 2026 Reviews received at journal 11 Feb, 2026 Reviews received at journal 29 Jan, 2026 Reviewers agreed at journal 10 Jan, 2026 Reviewers agreed at journal 01 Dec, 2025 Reviewers invited by journal 28 Nov, 2025 Editor assigned by journal 16 Oct, 2025 Submission checks completed at journal 16 Oct, 2025 First submitted to journal 09 Oct, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7821055","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":552327997,"identity":"8fc59983-0a69-41c1-876c-97eb662069d4","order_by":0,"name":"Rijha Safdar","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABTElEQVRIie2RMWvCQBTHnwTO5TTrBYv5CncEoqXSz3IiOGWRQklBbESIi21XB/sd7CdoJKBLcE5JKIZCpgx1EbvYJhY7JCIdC80P3hv+jx+8dweQk/NHecdJtzAQru+DwuowInHRrFEY/ygrZ58I9NeKFJj7BJFTSu3BDoyo+wpl727mNs1eVSwuwput3pBlQ5h5GHwlpZy5bdafzK9A8petOl/aijTSVA87bTa1UOsCQ6imFEKg9oYRB+pqKuHXVnPqYuQVTJtTwGoFg91IK+JiPcC7g4J6t89uMex87D65bIibowpobFAyD4opcEpAhZJh8fhBUKJkF9M6/cd7jiXfUercsdnY0ZQKnrfY1EbK+YSG6fPjxZ6MaMOrZW/EXrZ6TxaHi2C97V7K8nAQuJHuMyP7MQkYSCYTkkat48L3Vcc5oeTk5OT8E74A1lt0THt+QcoAAAAASUVORK5CYII=","orcid":"","institution":"National University of Sciences and Technology (NUST)","correspondingAuthor":true,"prefix":"","firstName":"Rijha","middleName":"","lastName":"Safdar","suffix":""},{"id":552327998,"identity":"fe73e949-924c-4221-9924-426f2f6c29cb","order_by":1,"name":"Danyail Mateen","email":"","orcid":"","institution":"National University of Computer and Emerging Sciences","correspondingAuthor":false,"prefix":"","firstName":"Danyail","middleName":"","lastName":"Mateen","suffix":""},{"id":552327999,"identity":"d5fe53f7-1bba-4fd1-bd2b-36c02a73ba6f","order_by":2,"name":"Syed Taha Ali","email":"","orcid":"","institution":"National University of Sciences and Technology (NUST)","correspondingAuthor":false,"prefix":"","firstName":"Syed","middleName":"Taha","lastName":"Ali","suffix":""},{"id":552328000,"identity":"75c233a6-7302-42b7-9171-c174e555b8c5","order_by":3,"name":"Umer Ashfaq","email":"","orcid":"","institution":"COMSATS University Islamabad","correspondingAuthor":false,"prefix":"","firstName":"Umer","middleName":"","lastName":"Ashfaq","suffix":""},{"id":552328001,"identity":"9b60e25a-ab47-4966-b0a2-f79fff22a654","order_by":4,"name":"Wajahat Hussain","email":"","orcid":"","institution":"National University of Sciences and Technology (NUST)","correspondingAuthor":false,"prefix":"","firstName":"Wajahat","middleName":"","lastName":"Hussain","suffix":""}],"badges":[],"createdAt":"2025-10-09 22:08:13","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7821055/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7821055/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":97250025,"identity":"de66f02e-5097-49f0-9dcc-673443e5cdf5","added_by":"auto","created_at":"2025-12-02 13:13:48","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":258872,"visible":true,"origin":"","legend":"","description":"","filename":"IJIS2.docx","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/a0d536c1d99db1622f5ec107.docx"},{"id":97248780,"identity":"765ad97b-84d5-41a3-b521-3649da5b94f0","added_by":"auto","created_at":"2025-12-02 13:07:08","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6689,"visible":true,"origin":"","legend":"","description":"","filename":"776fb8ce66c84f2a93211f94b0cba99a.json","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/278b4b9addf598684c205db9.json"},{"id":97250369,"identity":"12e181f4-1f64-480e-8a7d-074003af07a9","added_by":"auto","created_at":"2025-12-02 13:14:22","extension":"xml","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":192176,"visible":true,"origin":"","legend":"","description":"","filename":"776fb8ce66c84f2a93211f94b0cba99a1enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/e2dcf3f9224d4a219c12c6fd.xml"},{"id":97170314,"identity":"2d5530e2-fa44-4ec0-9750-03db283fc1c2","added_by":"auto","created_at":"2025-12-01 14:36:33","extension":"jpeg","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1074,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/246a3c6698e10a5870a7577d.jpeg"},{"id":97249005,"identity":"b952cf8d-2622-49f5-a9d2-ae5130ee359f","added_by":"auto","created_at":"2025-12-02 13:09:27","extension":"png","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":20568,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage10.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/3bbcb085284af9eb050530cd.png"},{"id":97170322,"identity":"e0c22173-57a8-4e76-904f-e2f550678ede","added_by":"auto","created_at":"2025-12-01 14:36:33","extension":"jpeg","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1167,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage2.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/5ac281ee2db3df974cf3f210.jpeg"},{"id":97250323,"identity":"2a1f52e2-cb38-4f2a-9d94-d06f70586c38","added_by":"auto","created_at":"2025-12-02 13:14:16","extension":"jpeg","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":168011,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage3.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/3ce447d07cfd76f6abcd46cc.jpeg"},{"id":97248981,"identity":"4ba0a9df-df6b-4914-89e9-ee25b8381522","added_by":"auto","created_at":"2025-12-02 13:09:15","extension":"png","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1070,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/e440c6fdb9c2b986eabee1fd.png"},{"id":97170317,"identity":"1a51e521-a004-4b1c-8a20-5bb45288db9d","added_by":"auto","created_at":"2025-12-01 14:36:33","extension":"png","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1056,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/d70823df144da968965ffe26.png"},{"id":97170326,"identity":"ed527b8b-de57-47aa-b2b2-9a6d71d3f482","added_by":"auto","created_at":"2025-12-01 14:36:33","extension":"png","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1070,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/8e9b47f46b27e8b8c74cf503.png"},{"id":97249078,"identity":"5c88ecca-455b-40be-b123-a34c47d5b803","added_by":"auto","created_at":"2025-12-02 13:10:14","extension":"png","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1112,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/fc1bdad59e5c87952a32b9f8.png"},{"id":97250678,"identity":"751e3cd7-2dfe-47b7-9754-5fa4ef3554d6","added_by":"auto","created_at":"2025-12-02 13:14:57","extension":"png","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1070,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/9c4052e87ffb210fdb74c29b.png"},{"id":97248796,"identity":"d147bdb4-95fb-4d78-90ae-527af5e0e642","added_by":"auto","created_at":"2025-12-02 13:07:11","extension":"png","order_by":13,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1059,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage9.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/cc06e15046af5b341e9cf1dd.png"},{"id":97170323,"identity":"96667503-2de2-4306-9792-267558e58344","added_by":"auto","created_at":"2025-12-01 14:36:33","extension":"jpeg","order_by":14,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":10123,"visible":true,"origin":"","legend":"","description":"","filename":"groupimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/c0270c8dd942dee9cb0b6611.jpeg"},{"id":97250041,"identity":"00753a6f-95e4-439a-8df9-80613efa5f12","added_by":"auto","created_at":"2025-12-02 13:13:49","extension":"jpeg","order_by":15,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":311,"visible":true,"origin":"","legend":"","description":"","filename":"groupimage10.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/51c0505287755b767647c391.jpeg"},{"id":97170339,"identity":"b850ac18-eddc-4dae-9c83-7ec42b32dac6","added_by":"auto","created_at":"2025-12-01 14:36:33","extension":"jpeg","order_by":16,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":297,"visible":true,"origin":"","legend":"","description":"","filename":"groupimage11.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/b9c2d799512d13007272664c.jpeg"},{"id":97248810,"identity":"5fe51e31-1f1b-4726-b308-c2e9ab4663b2","added_by":"auto","created_at":"2025-12-02 13:07:25","extension":"jpeg","order_by":17,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":296,"visible":true,"origin":"","legend":"","description":"","filename":"groupimage12.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/14099de70a8606ae85536478.jpeg"},{"id":97248798,"identity":"508ddbdf-11f2-4b6f-b62c-692d6be08872","added_by":"auto","created_at":"2025-12-02 13:07:13","extension":"jpeg","order_by":18,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":311,"visible":true,"origin":"","legend":"","description":"","filename":"groupimage10.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/76271b4416ce284b5d842abd.jpeg"},{"id":97170330,"identity":"b3776445-2505-4220-9db3-924798c0b456","added_by":"auto","created_at":"2025-12-01 14:36:33","extension":"jpeg","order_by":19,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":302,"visible":true,"origin":"","legend":"","description":"","filename":"groupimage3.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/368bae6669914c1848115b7a.jpeg"},{"id":97250182,"identity":"a5cdf0a0-7bca-4d8b-bc64-f0ca63a670d1","added_by":"auto","created_at":"2025-12-02 13:14:04","extension":"jpeg","order_by":20,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":302,"visible":true,"origin":"","legend":"","description":"","filename":"groupimage4.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/b46a1bcf93ce1646df05e8be.jpeg"},{"id":97248838,"identity":"e65c0e67-2605-4a52-946f-1def06c1993d","added_by":"auto","created_at":"2025-12-02 13:07:34","extension":"jpeg","order_by":21,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":293,"visible":true,"origin":"","legend":"","description":"","filename":"groupimage5.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/d591bf2225744bad411b3a3e.jpeg"},{"id":97250049,"identity":"47b5fb8c-c973-4e5c-abdd-a1f2214fa402","added_by":"auto","created_at":"2025-12-02 13:13:49","extension":"jpeg","order_by":22,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":302,"visible":true,"origin":"","legend":"","description":"","filename":"groupimage6.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/1e58e3440a7df993b6810919.jpeg"},{"id":97248855,"identity":"7da5305c-af9e-4819-ac32-15f1b86c4e24","added_by":"auto","created_at":"2025-12-02 13:07:40","extension":"jpeg","order_by":23,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":302,"visible":true,"origin":"","legend":"","description":"","filename":"groupimage3.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/f2000376741b9d8bcb6a79c1.jpeg"},{"id":97250509,"identity":"1a20f006-797b-4c0f-8753-d34a3889802c","added_by":"auto","created_at":"2025-12-02 13:14:38","extension":"jpeg","order_by":24,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":311,"visible":true,"origin":"","legend":"","description":"","filename":"groupimage10.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/897be89aeba78408b332b4df.jpeg"},{"id":97170337,"identity":"527aafd6-b066-4660-81f4-7d2e8117fa59","added_by":"auto","created_at":"2025-12-01 14:36:33","extension":"jpeg","order_by":25,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":302,"visible":true,"origin":"","legend":"","description":"","filename":"groupimage3.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/53f4dfed4551254d281a5b90.jpeg"},{"id":97170333,"identity":"32f6291e-8241-4619-b98d-07163ed6bbbd","added_by":"auto","created_at":"2025-12-01 14:36:33","extension":"png","order_by":26,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":935,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/83631a708abb7ff58c293788.png"},{"id":97248742,"identity":"e9910668-c12b-403a-9c6e-1f9ff1c103ca","added_by":"auto","created_at":"2025-12-02 13:06:48","extension":"png","order_by":27,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":8157,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage10.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/7e44711ca22c0119a2679757.png"},{"id":97170354,"identity":"118ee60f-f60e-475d-b4cf-4375f2622663","added_by":"auto","created_at":"2025-12-01 14:36:34","extension":"png","order_by":28,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":945,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/d85fdcebbea51ff1ca55438f.png"},{"id":97248859,"identity":"16f6d89b-de7c-4bf6-80e2-7521eb36c341","added_by":"auto","created_at":"2025-12-02 13:07:40","extension":"png","order_by":29,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":53542,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/d01481df6844014d5622a3b7.png"},{"id":97250141,"identity":"50449624-9302-4441-9bae-b89626dd8e69","added_by":"auto","created_at":"2025-12-02 13:14:01","extension":"png","order_by":30,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":662,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/44c1b9672661c024cbb11cac.png"},{"id":97249099,"identity":"ceab2061-013e-4dc1-b496-826e041b9964","added_by":"auto","created_at":"2025-12-02 13:10:26","extension":"png","order_by":31,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":659,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/7782f40faa38d89b2ec314cc.png"},{"id":97250046,"identity":"24cac313-b2fe-4afd-9a41-c92f1730a982","added_by":"auto","created_at":"2025-12-02 13:13:49","extension":"png","order_by":32,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":662,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/449cb40556806439b4f6ab4d.png"},{"id":97250133,"identity":"e48429a6-66d6-4095-b1ec-68fc1deaaa48","added_by":"auto","created_at":"2025-12-02 13:13:59","extension":"png","order_by":33,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":675,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/16ab1fc2ab14986fbaad989b.png"},{"id":97250129,"identity":"a342d057-193f-41d7-adfe-5418eb23399a","added_by":"auto","created_at":"2025-12-02 13:13:59","extension":"png","order_by":34,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":662,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/a7ab19c9065f6c948f4477ca.png"},{"id":97250420,"identity":"dea8ec8e-9e43-42d5-9950-98fe49bb67d6","added_by":"auto","created_at":"2025-12-02 13:14:29","extension":"png","order_by":35,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":675,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage9.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/838f4da2594f8a72c1eb320d.png"},{"id":97250566,"identity":"9a1b658b-98b6-4ea5-bd5b-824b90e28d11","added_by":"auto","created_at":"2025-12-02 13:14:42","extension":"png","order_by":36,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5532,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegroupimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/4595ac35e50c7aff28ff3eec.png"},{"id":97250055,"identity":"c489f343-50b8-41f7-b3b0-30e95b0aeb0c","added_by":"auto","created_at":"2025-12-02 13:13:49","extension":"png","order_by":37,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":332,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegroupimage10.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/261e841cfb748e497d304ff5.png"},{"id":97170345,"identity":"c2234cc7-5c9a-44c2-9a64-169161b528ed","added_by":"auto","created_at":"2025-12-01 14:36:34","extension":"png","order_by":38,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":328,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegroupimage11.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/5a2b499819c81408e8de5978.png"},{"id":97170342,"identity":"9064ca66-0406-4012-9316-75c8d68bd19d","added_by":"auto","created_at":"2025-12-01 14:36:33","extension":"png","order_by":39,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":326,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegroupimage12.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/d304bc05eda748a6e9187f90.png"},{"id":97249971,"identity":"303a6ca9-1fed-4c67-a1c3-1f30f69154f4","added_by":"auto","created_at":"2025-12-02 13:13:43","extension":"png","order_by":40,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":332,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegroupimage10.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/da5a59a8ab539444635eafb8.png"},{"id":97248864,"identity":"f1785f37-deb1-4722-bcbc-5a4af6a680ac","added_by":"auto","created_at":"2025-12-02 13:07:42","extension":"png","order_by":41,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":331,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegroupimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/c5827a8a569161f71406adfb.png"},{"id":97248862,"identity":"62ba9b1a-fb2b-4320-b151-1eb0e347ffd8","added_by":"auto","created_at":"2025-12-02 13:07:42","extension":"png","order_by":42,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":326,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegroupimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/19d157732d007e38dc74b77f.png"},{"id":97170350,"identity":"7313ed2b-a25c-4344-a169-5e47fecf0f53","added_by":"auto","created_at":"2025-12-01 14:36:34","extension":"png","order_by":43,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":323,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegroupimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/219439531f9a12015271f46e.png"},{"id":97170346,"identity":"f42e1a89-b39d-4897-9167-3d3a213f41a7","added_by":"auto","created_at":"2025-12-01 14:36:34","extension":"png","order_by":44,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":329,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegroupimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/4210d6dbd823834d57113f29.png"},{"id":97170349,"identity":"f9699ca9-efc0-44ad-a9cc-166748c20be9","added_by":"auto","created_at":"2025-12-01 14:36:34","extension":"png","order_by":45,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":331,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegroupimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/d1d4cef7237897ea5e4b6f5d.png"},{"id":97170343,"identity":"9fbfeb78-887f-4673-b3ee-aec9bf57e4e9","added_by":"auto","created_at":"2025-12-01 14:36:34","extension":"png","order_by":46,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":332,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegroupimage10.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/b223e5943122f714ff9fa5d4.png"},{"id":97250515,"identity":"69c23062-efc7-400e-b420-ec8335137f7f","added_by":"auto","created_at":"2025-12-02 13:14:38","extension":"png","order_by":47,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":331,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegroupimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/dd5a015cc651623021db3619.png"},{"id":97170361,"identity":"adc846b3-49c0-4c56-afb2-e78c8afe20ec","added_by":"auto","created_at":"2025-12-01 14:36:34","extension":"xml","order_by":48,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":190021,"visible":true,"origin":"","legend":"","description":"","filename":"776fb8ce66c84f2a93211f94b0cba99a1structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/0d9f55d0d6c4349ef8828fee.xml"},{"id":97250383,"identity":"e2856b5b-2cf4-45cb-bf22-94b4a5e87463","added_by":"auto","created_at":"2025-12-02 13:14:23","extension":"html","order_by":49,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":207092,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1/f0374f87779f3982dfa47c38.html"},{"id":97367470,"identity":"ad3c0358-1b9d-48f7-8c3e-dcc89d199431","added_by":"auto","created_at":"2025-12-03 16:18:53","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":686288,"visible":true,"origin":"","legend":"","description":"","filename":"IJIS2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7821055/v1_covered_5a85927a-52c4-4685-b05b-b6fe5f9c6d48.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-information-security","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ijis","sideBox":"Learn more about [International Journal of Information Security](http://link.springer.com/journal/10207)","snPcode":"10207","submissionUrl":"https://submission.nature.com/new-submission/10207/3","title":"International Journal of Information Security","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"LLM fine-tuning, Encoder-only, Vulnerability Detection, Generalizability, Codebases","lastPublishedDoi":"10.21203/rs.3.rs-7821055/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7821055/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAI-based solutions demonstrate re- markable results in identifying vulnerabilities in software, but research has consistently found that this performance does not generalize to unseen codebases. In this paper, we specifically investi- gate the impact of model architecture, parameter configuration, and quality of training data on the ability of these systems to generalize.\u003c/p\u003e\u003cp\u003eFor this purpose, we introduce VulGate, a high quality state of the art dataset that mitigates the shortcomings of prior datasets, by removing mis- labeled and duplicate samples, updating new vul- nerabilities, incorporating additional metadata, in- tegrating hard samples, and including dedicated test sets. We undertake a series of experiments to demonstrate that improved dataset diversity and quality substantially enhances vulnerability detec- tion. We also introduce and benchmark multiple encoder-only and decoder-only models. We find that encoder-based models outperform other mod- els in terms of accuracy and generalization. Our model achieves \u003cb\u003e6.8%\u003c/b\u003e improvement in recall on the benchmark BigVul dataset and outperforms others on unseen projects, demonstrating enhanced gen- eralizability. Our results highlight the role of data quality and model selection in the development of robust vulnerability detection systems. Our find- ings suggest a direction for future systems with high cross-project effectiveness.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003c/p\u003e","manuscriptTitle":"Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-01 14:36:28","doi":"10.21203/rs.3.rs-7821055/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-02-11T14:04:02+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-02-11T13:04:04+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-01-29T23:18:29+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"253691445100278653444961834003058108326","date":"2026-01-10T11:51:08+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"334420576161173350496608215022757733780","date":"2025-12-01T13:18:23+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-11-28T13:46:57+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-10-17T03:45:05+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-10-17T03:43:00+00:00","index":"","fulltext":""},{"type":"submitted","content":"International Journal of Information Security","date":"2025-10-09T21:55:02+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-information-security","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ijis","sideBox":"Learn more about [International Journal of Information Security](http://link.springer.com/journal/10207)","snPcode":"10207","submissionUrl":"https://submission.nature.com/new-submission/10207/3","title":"International Journal of Information Security","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"d3809293-9950-481c-8811-26f4e97d3b83","owner":[],"postedDate":"December 1st, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[],"tags":[],"updatedAt":"2026-05-14T06:09:12+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-01 14:36:28","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7821055","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7821055","identity":"rs-7821055","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}