DA-BAG: A Multi-Model Fusion Text Classification Method Combining BERT and GCN Using Self-Domain Adversarial Training

DA-BAG: A Multi-Model Fusion Text Classification Method Combining BERT and GCN Using Self-Domain Adversarial Training | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article DA-BAG: A Multi-Model Fusion Text Classification Method Combining BERT and GCN Using Self-Domain Adversarial Training Dangguo Shao, Shun Su, Lei Ma, Sanli Yi, Hua Lai This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4731032/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 8 You are reading this latest preprint version Abstract Both pre-training-based methods and GNN-based methods are considered the most advanced techniques in natural language processing tasks, particularly in text classification tasks. However, traditional graph learning methods focus solely on structured information from text to graph, overlooking the hidden local information within the syntactic structure of the text. Conversely, large-scale pre-training model methods tend to overlook global semantic information, potentially introducing new noise and training biases when combined. To tackle these challenges, we introduce DA-BAG, a novel approach that co-trains BERT and graph convolution models. Utilizing a self-domain adversarial training method on a single dataset, DA-BAG extracts multi-domain distribution features across multiple models, enabling self-adversarial domain adaptation training without the need for additional data, thereby enhancing model generalization and robustness. Furthermore, by incorporating an attention mechanism in multiple models, DA-BAG effectively combines the structural semantics of the graph with the token-level semantics of the pre-trained model, leveraging hidden information within the text's syntactic structure. Additionally, a sequential multi-layer graph convolutional neural(GCN) connection structure based on a residual pre-activation variant is employed to stabilize the feature distribution of graph data and adjust the graph data structure accordingly. Extensive evaluations on 5 datasets(20NG, R8, R52, Ohsumed, MR) demonstrate that DA-BAG achieves state-of-the-art performance across a diverse range of datasets. Text classification Domain adversarial GCN BERT Model fusion Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 31 Aug, 2024 Reviews received at journal 29 Aug, 2024 Reviewers agreed at journal 23 Jul, 2024 Reviewers agreed at journal 19 Jul, 2024 Reviewers invited by journal 19 Jul, 2024 Editor assigned by journal 18 Jul, 2024 Submission checks completed at journal 18 Jul, 2024 First submitted to journal 12 Jul, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4731032","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":336841215,"identity":"4f41f80e-5007-4b86-8a6c-391181c1bed9","order_by":0,"name":"Dangguo Shao","email":"","orcid":"","institution":"Kunming University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Dangguo","middleName":"","lastName":"Shao","suffix":""},{"id":336841216,"identity":"8f0d623c-7ac8-411d-8499-fb27a71a6ac1","order_by":1,"name":"Shun Su","email":"","orcid":"","institution":"Kunming University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Shun","middleName":"","lastName":"Su","suffix":""},{"id":336841217,"identity":"6ce27983-754c-4cec-9850-ed1037ff49b7","order_by":2,"name":"Lei Ma","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAzklEQVRIiWNgGAWjYDACZgbGAwwMEvX97I2NDz8QqYUBqMWCcWbP4WZjCWItAmqpYNwwI71NgIcY5QbHeQwO/NwhwWwg+bCNQYLBTk63gYAWyWYeg4O9ZyTYzKUT2x4UMCQbmx0goIWfGWgLb5sEj+XsxHYDCYYDidsIaWEDajn4t01CwuDmQaBGYrSAbDkMtMXA4AYjkVokm9kKDsu2SSRI9iQCA9mACL8YnD+88eHbtroEfvbjDx9+qLCTI6gF3QTSlI+CUTAKRsEowAEAeXo/vopcWaMAAAAASUVORK5CYII=","orcid":"","institution":"Kunming University of Science and Technology","correspondingAuthor":true,"prefix":"","firstName":"Lei","middleName":"","lastName":"Ma","suffix":""},{"id":336841218,"identity":"b23641e5-7480-4163-9b6d-df47fbc16c7b","order_by":3,"name":"Sanli Yi","email":"","orcid":"","institution":"Kunming University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Sanli","middleName":"","lastName":"Yi","suffix":""},{"id":336841219,"identity":"7d1de8a1-aaad-48bb-a517-34dcf05d8767","order_by":4,"name":"Hua Lai","email":"","orcid":"","institution":"Kunming University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Hua","middleName":"","lastName":"Lai","suffix":""}],"badges":[],"createdAt":"2024-07-12 14:41:55","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4731032/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4731032/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":62380187,"identity":"0f44d6cd-8e85-4fbf-85ba-9e036ee2659e","added_by":"auto","created_at":"2024-08-13 13:58:07","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":681909,"visible":true,"origin":"","legend":"","description":"","filename":"DABAGAMultiModelFusionTextClassificationMethodCombiningBERTandGCNUsingSelfDomainAdversarialTraining.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4731032/v1_covered_48e6017f-028c-415e-8227-6e5604881247.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"DA-BAG: A Multi-Model Fusion Text Classification Method Combining BERT and GCN Using Self-Domain Adversarial Training","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"journal-of-intelligent-information-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jiis","sideBox":"Learn more about [Journal of Intelligent Information Systems](http://link.springer.com/journal/10844)","snPcode":"10844","submissionUrl":"https://submission.nature.com/new-submission/10844/3","title":"Journal of Intelligent Information Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Text classification, Domain adversarial, GCN, BERT, Model fusion","lastPublishedDoi":"10.21203/rs.3.rs-4731032/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4731032/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBoth pre-training-based methods and GNN-based methods are considered the most advanced techniques in natural language processing tasks, particularly in text classification tasks. However, traditional graph learning methods focus solely on structured information from text to graph, overlooking the hidden local information within the syntactic structure of the text. Conversely, large-scale pre-training model methods tend to overlook global semantic information, potentially introducing new noise and training biases when combined. To tackle these challenges, we introduce DA-BAG, a novel approach that co-trains BERT and graph convolution models. Utilizing a self-domain adversarial training method on a single dataset, DA-BAG extracts multi-domain distribution features across multiple models, enabling self-adversarial domain adaptation training without the need for additional data, thereby enhancing model generalization and robustness. Furthermore, by incorporating an attention mechanism in multiple models, DA-BAG effectively combines the structural semantics of the graph with the token-level semantics of the pre-trained model, leveraging hidden information within the text's syntactic structure. Additionally, a sequential multi-layer graph convolutional neural(GCN) connection structure based on a residual pre-activation variant is employed to stabilize the feature distribution of graph data and adjust the graph data structure accordingly. Extensive evaluations on 5 datasets(20NG, R8, R52, Ohsumed, MR) demonstrate that DA-BAG achieves state-of-the-art performance across a diverse range of datasets.\u003c/p\u003e","manuscriptTitle":"DA-BAG: A Multi-Model Fusion Text Classification Method Combining BERT and GCN Using Self-Domain Adversarial Training","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-13 13:49:59","doi":"10.21203/rs.3.rs-4731032/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-09-01T01:08:34+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-08-30T02:32:36+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"2479581649100934724714143510227943122","date":"2024-07-23T17:59:15+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"214494675881087548011032233277507119274","date":"2024-07-20T01:50:34+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-07-19T17:43:50+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-07-18T15:20:57+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-07-18T05:59:03+00:00","index":"","fulltext":""},{"type":"submitted","content":"Journal of Intelligent Information Systems","date":"2024-07-12T14:40:39+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"journal-of-intelligent-information-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jiis","sideBox":"Learn more about [Journal of Intelligent Information Systems](http://link.springer.com/journal/10844)","snPcode":"10844","submissionUrl":"https://submission.nature.com/new-submission/10844/3","title":"Journal of Intelligent Information Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"293b31cb-090c-4a5c-b0d7-ca1140386eaa","owner":[],"postedDate":"August 13th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2024-09-10T10:06:28+00:00","versionOfRecord":[],"versionCreatedAt":"2024-08-13 13:49:59","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4731032","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4731032","identity":"rs-4731032","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}