Optimal Complexity in Lightweight Vision Transformers: A Trade-off Analysis between Representational Power and Optimization Efficiency

Optimal Complexity in Lightweight Vision Transformers: A Trade-off Analysis between Representational Power and Optimization Efficiency | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Optimal Complexity in Lightweight Vision Transformers: A Trade-off Analysis between Representational Power and Optimization Efficiency Yunan Zhang, Jingjing Fan, Jianguang Zhao This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7471450/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The deployment of deep learning models on resource-constrained edge devices necessitates a critical balance between performance and complexity. This study systematically challenges the prevailing assumption that enhancing lightweight vision transformers with sophisticated modules invariably improves performance. By investigating the impact of structural enhancements on the state-of-the-art lightweight Vision Transformer, RepViT-M0.9, our experiments on ImageNet-1K reveal that increasing structural complexity can significantly degrade accuracy and parameter efficiency. Visualizations and feature space analysis suggest that excessive complexity within a lightweight model impairs feature representations and introduces optimization challenges. We propose the Representation-Optimization Trade-off Theory, which models performance as a balance between representational power and optimization cost. Our findings demonstrate that an optimal complexity level exists for lightweight models, beyond which performance deteriorates. This work highlights the importance of structural simplicity and parameter efficiency in developing effective AI solutions for edge devices. The source code and pre-trained models are available at: https://github.com/niyaobuyaochibl/ACR-RepViT with DOI:10.5281/zenodo.16959886. Lightweight Vision Transformers Model Complexity Representation-Optimization Trade-off Parameter Efficiency Computer Vision Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7471450","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":508204412,"identity":"0d367c3e-7a76-466c-bebc-6e3a9edb21d2","order_by":0,"name":"Yunan Zhang","email":"","orcid":"","institution":"Hebei University of Architecture","correspondingAuthor":false,"prefix":"","firstName":"Yunan","middleName":"","lastName":"Zhang","suffix":""},{"id":508204413,"identity":"c5a6a4d5-66e2-49a3-9460-377fbbb7a805","order_by":1,"name":"Jingjing Fan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA9klEQVRIie3RMWsCMRjG8edwuOWOW1Mo9iu8pavoV4kcnIsWRweHG8q53O0ufodzqt3eELBL2tmtTk4n2K0OpQ24FRquW4f8p5Dw4yUJ4PP9w8KcJZ9mokuIwJc96SYR81AtTe/uLwQdHT9kw9oStCMB73Wc69FjWCn+mPeQhGPCeeMgYKlWGz15Kl+lKrcZrsqGgsr8TgaW8NHoSb0bE9txILvoBIV7CseFHtFbQ+oz/8KgDVFxkUnaRWQvxSDRhthHvq3NvdTX2zQS5jBVlYsIk57sV97Q84t+b+b9brJI1/uzg0AY+WOsjR0ASBbuc5/P5/PhG+ZJYjsq2JlQAAAAAElFTkSuQmCC","orcid":"","institution":"Hebei University of Architecture","correspondingAuthor":true,"prefix":"","firstName":"Jingjing","middleName":"","lastName":"Fan","suffix":""},{"id":508204414,"identity":"b049a6d3-b39d-47b6-be2b-c7befb42161f","order_by":2,"name":"Jianguang Zhao","email":"","orcid":"","institution":"Hebei University of Architecture","correspondingAuthor":false,"prefix":"","firstName":"Jianguang","middleName":"","lastName":"Zhao","suffix":""}],"badges":[],"createdAt":"2025-08-27 12:08:16","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7471450/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7471450/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":109429418,"identity":"3248c5b8-0848-4412-b33c-0185e5b60581","added_by":"auto","created_at":"2026-05-18 03:55:50","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3876326,"visible":true,"origin":"","legend":"","description":"","filename":"SUBMISSIONFINAL.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7471450/v1_covered_66db691f-d895-4b61-9508-cb1cfb30a861.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Optimal Complexity in Lightweight Vision Transformers: A Trade-off Analysis between Representational Power and Optimization Efficiency","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Lightweight Vision Transformers, Model Complexity, Representation-Optimization Trade-off, Parameter Efficiency, Computer Vision","lastPublishedDoi":"10.21203/rs.3.rs-7471450/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7471450/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe deployment of deep learning models on resource-constrained edge devices necessitates a critical balance between performance and complexity. This study systematically challenges the prevailing assumption that enhancing lightweight vision transformers with sophisticated modules invariably improves performance. By investigating the impact of structural enhancements on the state-of-the-art lightweight Vision Transformer, RepViT-M0.9, our experiments on ImageNet-1K reveal that increasing structural complexity can significantly degrade accuracy and parameter efficiency. Visualizations and feature space analysis suggest that excessive complexity within a lightweight model impairs feature representations and introduces optimization challenges. We propose the Representation-Optimization Trade-off Theory, which models performance as a balance between representational power and optimization cost. Our findings demonstrate that an optimal complexity level exists for lightweight models, beyond which performance deteriorates. This work highlights the importance of structural simplicity and parameter efficiency in developing effective AI solutions for edge devices. The source code and pre-trained models are available at: https://github.com/niyaobuyaochibl/ACR-RepViT with DOI:10.5281/zenodo.16959886.\u003c/p\u003e","manuscriptTitle":"Optimal Complexity in Lightweight Vision Transformers: A Trade-off Analysis between Representational Power and Optimization Efficiency","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-05 04:15:54","doi":"10.21203/rs.3.rs-7471450/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"c2911ee8-476a-413c-b858-fdfda02f408d","owner":[],"postedDate":"September 5th, 2025","published":true,"recentEditorialEvents":[{"type":"decision","content":"Rejected","date":"2026-05-18T03:37:44+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-09T07:55:58+00:00","index":64,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-05-18T03:54:31+00:00","versionOfRecord":[],"versionCreatedAt":"2025-09-05 04:15:54","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7471450","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7471450","identity":"rs-7471450","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}