Assessing the Risk of Discriminatory Bias in Classification Datasets

Assessing the Risk of Discriminatory Bias in Classification Datasets | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Assessing the Risk of Discriminatory Bias in Classification Datasets Kejun Dai, Jonathan Kim, Sašo Džeroski, Jörg Wicker, Gillian Dobbie, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6370375/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 06 Aug, 2025 Read the published version in Machine Learning → Version 1 posted You are reading this latest preprint version Abstract Bias in machine learning models remains a critical challenge, particularly in datasets with numeric features where discrimination may be subtle and hard to detect. Existing fairness frameworks rely on expert knowledge of marginalized groups, such as specific racial groups, and categorical features defining them. Furthermore, most frameworks evaluate bias in models rather than datasets, despite the fact that model bias can often be traced back to dataset shortcomings. Our research aims to remedy this gap by capturing dataset flaws in a set of meta-features at the dataset level, and to warn practitioners of bias risk when using such datasets for model training. We neither restrict the feature type nor expect domain knowledge. To this end, we develop methods to synthesize biased datasets and extend current fairness metrics to continuous features in order to quantify dataset-level discrimination risks. Our approach constructs a meta-database of diverse datasets, from which we derive transferable meta-features that capture dataset properties indicative of bias risk. Our findings demonstrate that dataset-level characteristics can serve as cost-effective indicators of bias risk, providing a novel method for data auditing that does not rely on expert knowledge. This work lays the foundation for early-warning systems, moving beyond model-focused assessments toward a data-centric approach. Discriminatory bias Meta-learning Fairness Full Text Additional Declarations Competing interest reported. Dost is supported by the European Union's Horizon Europe research and innovation programme under the Marie Skłodowska-Curie Postdoctoral Fellowship Programme, SMASH co-funded under the grant agreement No. 101081355. The SMASH project is co-funded by the Republic of Slovenia and the European Union from the European Regional Development Fund. Džeroski is supported by the Slovenian Research and Innovation Agency (under grant P2-0103). Kim is employed by Callaghan Innovation, a Crown Research Institute in New Zealand. Supplementary Files dai2025discriminatorybiasappendix.pdf Cite Share Download PDF Status: Published Journal Publication published 06 Aug, 2025 Read the published version in Machine Learning → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6370375","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":438023941,"identity":"1c4da89f-a5ff-4319-9ac0-5d0d3ec540cf","order_by":0,"name":"Kejun Dai","email":"","orcid":"","institution":"University of Auckland","correspondingAuthor":false,"prefix":"","firstName":"Kejun","middleName":"","lastName":"Dai","suffix":""},{"id":438023942,"identity":"ff6049db-a10d-47e3-8ea3-3dfca97ee6b4","order_by":1,"name":"Jonathan Kim","email":"","orcid":"","institution":"Callaghan Innovation","correspondingAuthor":false,"prefix":"","firstName":"Jonathan","middleName":"","lastName":"Kim","suffix":""},{"id":438023943,"identity":"7c73420a-7a9f-4635-bcb8-25308e07efe1","order_by":2,"name":"Sašo Džeroski","email":"","orcid":"","institution":"Jožef Stefan Institute","correspondingAuthor":false,"prefix":"","firstName":"Sašo","middleName":"","lastName":"Džeroski","suffix":""},{"id":438023944,"identity":"b018c265-1b81-47b4-8745-11397441bb94","order_by":3,"name":"Jörg Wicker","email":"","orcid":"","institution":"University of Auckland","correspondingAuthor":false,"prefix":"","firstName":"Jörg","middleName":"","lastName":"Wicker","suffix":""},{"id":438023945,"identity":"944376da-401a-48bc-87ad-d3f63575613c","order_by":4,"name":"Gillian Dobbie","email":"","orcid":"","institution":"University of Auckland","correspondingAuthor":false,"prefix":"","firstName":"Gillian","middleName":"","lastName":"Dobbie","suffix":""},{"id":438023946,"identity":"73ed48b9-e0f9-4e6c-941f-a30581eb9cdc","order_by":5,"name":"Katharina Dost","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7klEQVRIiWNgGAWjYBACCQYGNgjFwGAAxDY8jA1AKgGvFmZkLQlpRGthgGk5TNhhku3njz34uccij7+BeePDnz/OyzD3H2D+8ACPFmmeZHbDnmcSxRIH2IqNeRJu8zDOSGCTwOcwOYZkNgmeAxKJDQd4zKQZwFqAAYJXC/9jNsk/QC3zgVokfySc42EEOQyfFmmJZDZpkC0bgFokeBIOAEMsgQGvwyRnPDaTlgFq2XgY5Je0ZKDDEtvwapE4n/hM8s2BusR5x5s3PvxhY2dv2H/48McfeLQgADOUNmwARyYpQJ5E9aNgFIyCUTD8AQD/2EeG+R/1uAAAAABJRU5ErkJggg==","orcid":"","institution":"Jožef Stefan Institute","correspondingAuthor":true,"prefix":"","firstName":"Katharina","middleName":"","lastName":"Dost","suffix":""}],"badges":[],"createdAt":"2025-04-03 14:53:18","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6370375/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6370375/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s10994-025-06843-9","type":"published","date":"2025-08-06T15:56:52+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":88814063,"identity":"38f97291-bd55-4b26-bf9f-fc53d61550ab","added_by":"auto","created_at":"2025-08-11 16:04:58","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":892585,"visible":true,"origin":"","legend":"","description":"","filename":"dai2025discriminatorybias.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6370375/v1_covered_26302f9e-9a5e-438f-a659-d76e8e37193f.pdf"},{"id":79878217,"identity":"ad1c3e47-c4f6-495f-93b9-6f6c23e667c1","added_by":"auto","created_at":"2025-04-04 03:09:45","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":484733,"visible":true,"origin":"","legend":"","description":"","filename":"dai2025discriminatorybiasappendix.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6370375/v1/913ea1c48d8ae622b36452f9.pdf"}],"financialInterests":"Competing interest reported. Dost is supported by the European Union's Horizon Europe research and innovation programme under the Marie Skłodowska-Curie Postdoctoral Fellowship Programme, SMASH co-funded under the grant agreement\nNo. 101081355. The SMASH project is co-funded by the Republic of Slovenia and the European Union from the European Regional Development Fund.\nDžeroski is supported by the Slovenian Research and Innovation Agency (under grant P2-0103).\nKim is employed by Callaghan Innovation, a Crown Research Institute in New Zealand.","formattedTitle":"Assessing the Risk of Discriminatory Bias in Classification Datasets","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Discriminatory bias, Meta-learning, Fairness","lastPublishedDoi":"10.21203/rs.3.rs-6370375/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6370375/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Bias in machine learning models remains a critical challenge, particularly in datasets with numeric features where discrimination may be subtle and hard to detect. Existing fairness frameworks rely on expert knowledge of marginalized groups, such as specific racial groups, and categorical features defining them. Furthermore, most frameworks evaluate bias in models rather than datasets, despite the fact that model bias can often be traced back to dataset shortcomings.\nOur research aims to remedy this gap by capturing dataset flaws in a set of meta-features at the dataset level, and to warn practitioners of bias risk when using such datasets for model training. We neither restrict the feature type nor expect domain knowledge.\nTo this end, we develop methods to synthesize biased datasets and extend current fairness metrics to continuous features in order to quantify dataset-level discrimination risks. Our approach constructs a meta-database of diverse datasets, from which we derive transferable meta-features that capture dataset properties indicative of bias risk.\nOur findings demonstrate that dataset-level characteristics can serve as cost-effective indicators of bias risk, providing a novel method for data auditing that does not rely on expert knowledge. This work lays the foundation for early-warning systems, moving beyond model-focused assessments toward a data-centric approach.","manuscriptTitle":"Assessing the Risk of Discriminatory Bias in Classification Datasets","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-04 03:09:41","doi":"10.21203/rs.3.rs-6370375/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"693b1eff-c4d8-4615-865c-3f10ae741ff7","owner":[],"postedDate":"April 4th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-08-11T15:58:53+00:00","versionOfRecord":{"articleIdentity":"rs-6370375","link":"https://doi.org/10.1007/s10994-025-06843-9","journal":{"identity":"machine-learning","isVorOnly":false,"title":"Machine Learning"},"publishedOn":"2025-08-06 15:56:52","publishedOnDateReadable":"August 6th, 2025"},"versionCreatedAt":"2025-04-04 03:09:41","video":"","vorDoi":"10.1007/s10994-025-06843-9","vorDoiUrl":"https://doi.org/10.1007/s10994-025-06843-9","workflowStages":[]},"version":"v1","identity":"rs-6370375","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6370375","identity":"rs-6370375","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}