MS-NET-v2: Modular Selective Network Optimized by Systematic Generation of Expert Modules

MS-NET-v2: Modular Selective Network Optimized by Systematic Generation of Expert Modules | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article MS-NET-v2: Modular Selective Network Optimized by Systematic Generation of Expert Modules Md Intisar Chowdhury, Qiangfu Zhao This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4909936/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 20 Feb, 2025 Read the published version in International Journal of Machine Learning and Cybernetics → Version 1 posted 13 You are reading this latest preprint version Abstract Modular architectures enhance Deep Neural Networks (DNNs) by reducing error rates, enabling uncertainty estimation, and increasing inference efficiency through selective module execution. While modular ensemble methods like bagging, boosting, and stacking improve DNNs performance, they are often computationally intensive. Adaptive inference strategies, such as the Modular Selective Network (MS-NET), address these issues by using independent router and expert modules, allowing parallel training and selective inference. In this study, we introduce MS-NET-v2, an optimized version of MS-NET, which enhances the construction of subsets and the expert training methodology. First, we introduce a cut-off variable, O, which systematically limits the sampling of binary class pairs from the Inter-Class Correlation (ICC) matrix. Next, we propose a subset merging algorithm that generates multi-class subsets from these binary subsets. This algorithm creates subsets that encode coarse concepts, in contrast to the fine-grained concepts represented by binary subsets. Based on loss landscape theory and to exploit the non-convexity of DNNs, we train these experts from scratch on multi-class subsets. This approach enhances diversity by covering several unique local minima , resulting in improved collective accuracy. We conduct extensive empirical studies with MS-NET-v2 on the CIFAR-10, CIFAR-100, Tiny-ImageNet, and Pets datasets. To verify the enhanced diversity of the expert networks, we perform function and weight space analyses on MS-NET-v2 experts. These studies demonstrate that MS-NET-v2 significantly improves collective accuracy, expert networks' diversity, and parameter efficiency compared to its predecessor. Additionally, MS-NET-v2 outperforms heavier ensemble methods in both single and multi expert settings. modular neural networks expert and router network system adaptive inference image classification ensemble system Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 20 Feb, 2025 Read the published version in International Journal of Machine Learning and Cybernetics → Version 1 posted Editorial decision: Revision requested 09 Oct, 2024 Reviews received at journal 04 Oct, 2024 Reviews received at journal 16 Sep, 2024 Reviews received at journal 13 Sep, 2024 Reviews received at journal 13 Sep, 2024 Reviewers agreed at journal 07 Sep, 2024 Reviewers agreed at journal 06 Sep, 2024 Reviewers agreed at journal 04 Sep, 2024 Reviewers agreed at journal 30 Aug, 2024 Reviewers invited by journal 23 Aug, 2024 Editor assigned by journal 17 Aug, 2024 Submission checks completed at journal 17 Aug, 2024 First submitted to journal 13 Aug, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4909936","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":350951344,"identity":"580e8d37-fe51-48c4-962b-d61febd9cd18","order_by":0,"name":"Md Intisar Chowdhury","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABD0lEQVRIiWNgGAWjYBACNiA+AGZJADFvA5BgBxIPwEIHcGhhRtfCA+Qn4NHCANTCgKpFIgGmBTvgk+4/ePBHzT0G/tnNxyTe7jicb3Dz8TOgLjs5BsazWK1hkznMcJjnWDGDxJ1jaZJzzxy23HA7zQyoJdmYgeEcVsvYJJIZDjOwJTAYSOSY3eZtO2xgcDvB7EYCw4HEBoYzBri0HPzxD1nLzePfCGo5wNuGrOUGD0FbDA7z9iXwSNxIS/85ty3dQPJMTvmPBINkYzYcfpGfkfj4449vCXL8M5IPG7xtszbgO358s8GHCjs5fgnsIQYDPHCWAlidAcgBZ/DpQLa3Acbi7yFSyygYBaNgFAxzAAAFi2PzJqV+OQAAAABJRU5ErkJggg==","orcid":"","institution":"AWL, Inc. Tokyo","correspondingAuthor":true,"prefix":"","firstName":"Md","middleName":"Intisar","lastName":"Chowdhury","suffix":""},{"id":350951345,"identity":"b35f0db1-33fb-4130-b1e0-f84021eb8774","order_by":1,"name":"Qiangfu Zhao","email":"","orcid":"","institution":"The University of Aizu Aizu-Wakamatsu","correspondingAuthor":false,"prefix":"","firstName":"Qiangfu","middleName":"","lastName":"Zhao","suffix":""}],"badges":[],"createdAt":"2024-08-14 01:36:40","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4909936/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4909936/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s13042-025-02563-7","type":"published","date":"2025-02-20T15:57:34+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":77052622,"identity":"e4e40e39-ad5c-42a5-8481-07f54878bfa1","added_by":"auto","created_at":"2025-02-24 16:18:39","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1655361,"visible":true,"origin":"","legend":"","description":"","filename":"MSNETV2manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4909936/v1_covered_7ae2758c-89af-411a-bc5d-225727197211.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"MS-NET-v2: Modular Selective Network Optimized by Systematic Generation of Expert Modules","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-machine-learning-and-cybernetics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jmlc","sideBox":"Learn more about [International Journal of Machine Learning and Cybernetics](http://actavetscand.biomedcentral.com/)","snPcode":"13042","submissionUrl":"https://submission.nature.com/new-submission/13042/3","title":"International Journal of Machine Learning and Cybernetics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"modular neural networks, expert and router network system, adaptive inference, image classification, ensemble system","lastPublishedDoi":"10.21203/rs.3.rs-4909936/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4909936/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Modular architectures enhance Deep Neural Networks (DNNs) by reducing error rates, enabling uncertainty estimation, and increasing inference efficiency through selective module execution. While modular ensemble methods like bagging, boosting, and stacking improve DNNs performance, they are often computationally intensive. Adaptive inference strategies, such as the Modular Selective Network (MS-NET), address these issues by using independent router and expert modules, allowing parallel training and selective inference. In this study, we introduce MS-NET-v2, an optimized version of MS-NET, which enhances the construction of subsets and the expert training methodology. First, we introduce a cut-off variable, O, which systematically limits the sampling of binary class pairs from the Inter-Class Correlation (ICC) matrix. Next, we propose a subset merging algorithm that generates multi-class subsets from these binary subsets. This algorithm creates subsets that encode coarse concepts, in contrast to the fine-grained concepts represented by binary subsets. Based on loss landscape theory and to exploit the non-convexity of DNNs, we train these experts from scratch on multi-class subsets. This approach enhances diversity by covering several unique local minima , resulting in improved collective accuracy. We conduct extensive empirical studies with MS-NET-v2 on the CIFAR-10, CIFAR-100, Tiny-ImageNet, and Pets datasets. To verify the enhanced diversity of the expert networks, we perform function and weight space analyses on MS-NET-v2 experts. These studies demonstrate that MS-NET-v2 significantly improves collective accuracy, expert networks' diversity, and parameter efficiency compared to its predecessor. Additionally, MS-NET-v2 outperforms heavier ensemble methods in both single and multi expert settings.","manuscriptTitle":"MS-NET-v2: Modular Selective Network Optimized by Systematic Generation of Expert Modules","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-09-13 07:05:04","doi":"10.21203/rs.3.rs-4909936/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-10-09T07:53:34+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-10-04T18:01:57+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-09-16T07:56:16+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-09-13T16:39:20+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-09-13T11:25:36+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"180553686502467064021296811634706323223","date":"2024-09-08T02:03:17+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"25603969223322110742236793821939443248","date":"2024-09-06T04:28:48+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"324651159736414439435505353999186105703","date":"2024-09-04T16:57:08+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"16115798826932866818427035309292246361","date":"2024-08-30T09:06:31+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-08-23T07:17:49+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-08-18T00:32:48+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-08-17T06:10:06+00:00","index":"","fulltext":""},{"type":"submitted","content":"International Journal of Machine Learning and Cybernetics","date":"2024-08-14T01:35:26+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-machine-learning-and-cybernetics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jmlc","sideBox":"Learn more about [International Journal of Machine Learning and Cybernetics](http://actavetscand.biomedcentral.com/)","snPcode":"13042","submissionUrl":"https://submission.nature.com/new-submission/13042/3","title":"International Journal of Machine Learning and Cybernetics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"60bf3e8b-e09a-40b7-9f52-e4807dcb008a","owner":[],"postedDate":"September 13th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-02-24T16:02:20+00:00","versionOfRecord":{"articleIdentity":"rs-4909936","link":"https://doi.org/10.1007/s13042-025-02563-7","journal":{"identity":"international-journal-of-machine-learning-and-cybernetics","isVorOnly":false,"title":"International Journal of Machine Learning and Cybernetics"},"publishedOn":"2025-02-20 15:57:34","publishedOnDateReadable":"February 20th, 2025"},"versionCreatedAt":"2024-09-13 07:05:04","video":"","vorDoi":"10.1007/s13042-025-02563-7","vorDoiUrl":"https://doi.org/10.1007/s13042-025-02563-7","workflowStages":[]},"version":"v1","identity":"rs-4909936","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4909936","identity":"rs-4909936","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}