Adaptive Dynamic Fusion for Adversarial and Counterfactual Debiasing in Pre-Trained Language Models

Adaptive Dynamic Fusion for Adversarial and Counterfactual Debiasing in Pre-Trained Language Models | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Adaptive Dynamic Fusion for Adversarial and Counterfactual Debiasing in Pre-Trained Language Models Yassine Yazidi, Hamid Garmani, Mohamed Baslam This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6239685/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Pre-trained language models are highly effective in various Natural Language Processing (NLP) tasks; however, they remain vulnerable to stereotypical biases, raising concerns about fairness. This paper presents a framework for mitigating bias in language models through an enhanced version of DeBERTaV3, which incorporates an Adaptive-Dynamic Fusion (ADF) component, referred to as ADFBERT. This model features a dynamically reweighted fusion layer that adapts token interactions based on their positional encoding. We integrate Adversarial Fine-Tuning (AFT) and Counterfactual Data Augmentation (CDA) to improve bias mitigation performance. AFT introduces an adversarial loss that minimizes correlations between learned representations and biased attributes, while CDA generates counterfactual samples to promote invariance across different demographic groups. We evaluate each method using ADFBERT on the StereoSet benchmark dataset using the Idealized Context Association Test (ICAT) score for the assessment. Experimental results show that ADFBERT combined with AFT enhances the ICAT score by 0.78 points compared to XLNet-large. In contrast, ADFBERT with CDA achieves a state-of-the-art ICAT score of 78.90%, surpassing the best baselines such as XLNet-large, RoBERTa, and GPT-3 by 6.90%. These findings highlight that integrating dynamic fusion with adversarial fine-tuning and counterfactual data augmentation significantly improves bias mitigation and fairness in pre-trained language models. Neural Bias Mitigation Adaptive Fusion Transformers Dynamic Contextual Encoding Position-Aware Token Interaction Counterfactual Data Augmentation Fairness in NLP Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6239685","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":439043480,"identity":"a5e9a766-a780-48e7-a205-bdab3f209cce","order_by":0,"name":"Yassine Yazidi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABAklEQVRIiWNgGAWjYLCCBCh9gMfAhnQtaaRax8NwmLAig+PHH394uMMGxHh44E3B+cT5044/YPhRwxDN34BDy5kcM4nEM2lARkLCwTkGtxM33M4xYOw5xpA74wAOLQdy2BgS2w4DGQkHDvOAtEjnMDDwNjDkNuDScv754w+Jbf+BjIcNQC3nEufPTn/A+BeoZT4uLTcSDCQS2w4AGckMQC0HEhtuJxgwg2zZgEOL5I03QL+0JfNI3njGAPRLsjHIL4dljknkbsShhe98+uOPP9vs5ECMD2/+2MkCHfbw4Zsam9x5OLQoQMV5FJAVANkS2NUDgXwDOmMUjIJRMApGAToAAJuyafq3L3o9AAAAAElFTkSuQmCC","orcid":"","institution":"Université Sultan Moulay Slimane","correspondingAuthor":true,"prefix":"","firstName":"Yassine","middleName":"","lastName":"Yazidi","suffix":""},{"id":439043481,"identity":"b0376a32-9b77-4bb4-b196-79bd9514cc7c","order_by":1,"name":"Hamid Garmani","email":"","orcid":"","institution":"Université Sultan Moulay Slimane","correspondingAuthor":false,"prefix":"","firstName":"Hamid","middleName":"","lastName":"Garmani","suffix":""},{"id":439043482,"identity":"6d163dee-509c-4ce1-bcd2-52c052c7763f","order_by":2,"name":"Mohamed Baslam","email":"","orcid":"","institution":"Université Sultan Moulay Slimane","correspondingAuthor":false,"prefix":"","firstName":"Mohamed","middleName":"","lastName":"Baslam","suffix":""}],"badges":[],"createdAt":"2025-03-16 22:53:09","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6239685/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6239685/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":89822988,"identity":"50cc9c67-5d15-414f-97d6-116530817d00","added_by":"auto","created_at":"2025-08-25 12:02:09","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":417116,"visible":true,"origin":"","legend":"","description":"","filename":"ADFBERTCDAAFTDEBIASING.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6239685/v1_covered_39abd48d-c410-429c-b1fe-eb563ea0f081.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Adaptive Dynamic Fusion for Adversarial and Counterfactual Debiasing in Pre-Trained Language Models","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Neural Bias Mitigation, Adaptive Fusion Transformers, Dynamic Contextual Encoding, Position-Aware Token Interaction, Counterfactual Data Augmentation, Fairness in NLP","lastPublishedDoi":"10.21203/rs.3.rs-6239685/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6239685/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Pre-trained language models are highly effective in various Natural Language Processing (NLP) tasks; however, they remain vulnerable to stereotypical biases, raising concerns about fairness. This paper presents a framework for mitigating bias in language models through an enhanced version of DeBERTaV3, which incorporates an Adaptive-Dynamic Fusion (ADF) component, referred to as ADFBERT. This model features a dynamically reweighted fusion layer that adapts token interactions based on their positional encoding. We integrate Adversarial Fine-Tuning (AFT) and Counterfactual Data Augmentation (CDA) to improve bias mitigation performance. AFT introduces an adversarial loss that minimizes correlations between learned representations and biased attributes, while CDA generates counterfactual samples to promote invariance across different demographic groups. We evaluate each method using ADFBERT on the StereoSet benchmark dataset using the Idealized Context Association Test (ICAT) score for the assessment. Experimental results show that ADFBERT combined with AFT enhances the ICAT score by 0.78 points compared to XLNet-large. In contrast, ADFBERT with CDA achieves a state-of-the-art ICAT score of 78.90\\%, surpassing the best baselines such as XLNet-large, RoBERTa, and GPT-3 by 6.90\\%. These findings highlight that integrating dynamic fusion with adversarial fine-tuning and counterfactual data augmentation significantly improves bias mitigation and fairness in pre-trained language models.","manuscriptTitle":"Adaptive Dynamic Fusion for Adversarial and Counterfactual Debiasing in Pre-Trained Language Models","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-14 05:37:10","doi":"10.21203/rs.3.rs-6239685/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e355b507-6c0e-4238-a715-5c7187b7bb91","owner":[],"postedDate":"April 14th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-08-25T11:54:00+00:00","versionOfRecord":[],"versionCreatedAt":"2025-04-14 05:37:10","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6239685","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6239685","identity":"rs-6239685","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}