What Makes Neural Networks Trainable? Invexity as a Structural Design Principle in AI

What Makes Neural Networks Trainable? Invexity as a Structural Design Principle in AI | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Physical Sciences - Article What Makes Neural Networks Trainable? Invexity as a Structural Design Principle in AI Samuel Pinilla, Ana Sanabria, Jia Bi, Karen Egiazarian This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7215670/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted You are reading this latest preprint version Abstract Despite their non-convex loss landscapes and vast parameter spaces, deep neural networks consistently achieve high performance across domains –from medical diagnostics to natural language processing and computer vision. However, the theoretical basis for their trainability remains unclear. Classical frameworks, such as convex optimization or probabilistic models (e.g., Bayesian optimization), offer only partial explanations and rely on restrictive assumptions that limit architectural expressiveness –such as shallow architectures, non-negative weights, or convex activations. This gap underscores a fundamental question: What Makes Neural Networks Trainable?. Here, we introduce a general framework based on invexity, a property that guarantees all critical points are global minima. We make four key contributions: (i) We demonstrate, for the first time, that the vast majority of commonly used activation functions –over 90% of fifty analyzed– are inherently invex. This reveals that modern architectures are already aligned with this property, even if unintentionally. ii) We show that deep Multilayer Perceptron models can be systematically constructed as invex structures, challenging the adequacy of existing convex and probabilistic optimization paradigms. iii) We prove that widely adopted architectures, such as ResNet, UNet, and Vision Transformer, satisfy invexity, providing a theoretical explanation for their empirical trainability, even at extreme depths, as is the case with ResNet. This provides a guarantee of accessibility to global optima for these well-known networks via standard gradient-based schemes. iv) By reframing trainability as a structural property rather than an empirical coincidence, our results provide a new foundation for understanding and designing neural networks. Physical sciences/Mathematics and computing/Computer science Physical sciences/Mathematics and computing/Scientific data Physical sciences/Mathematics and computing/Applied mathematics Full Text Additional Declarations There is NO Competing Interest. Supplementary Files supplementalmaterial.pdf Supplemental material Cite Share Download PDF Status: Under Review Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7215670","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Physical Sciences - Article","associatedPublications":[],"authors":[{"id":492056647,"identity":"65542015-089f-4dc3-a7cb-52969747c0a6","order_by":0,"name":"Samuel Pinilla","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABC0lEQVRIiWNgGAWjYNCCAgYGNgbGxgMPGGwYGJgPQAQl8GoxAGtpOJDAkMbAw5ZApBYQAGo5TFiLfPvpNIkPBgx5fNLNQFtqztvbszE/YPhRw5A4swGH+Wdyt0nOMGAoZpM5CNRy7HZiDxubAWPPMYbE2TidlLtNmseAIbFNIhGohe12Ao98DwMDbwND4jxcDut/u036D1zLv3P2PGw8DIx/8WhhuAG0hQGmJbHtAGMPUAszyBacDrvxdrNlj4FEMRtYS19yYs8xNoPDMsckjHF5X74/d+ONHxU2efIz0h8++PDNzp69jfnhwzc1NrIzDuByGRhIJKBwDxCKSBBIIKhiFIyCUTAKRi4AAAFOWPY6LNsPAAAAAElFTkSuQmCC","orcid":"","institution":"Rutherford Appleton Laboratory","correspondingAuthor":true,"prefix":"","firstName":"Samuel","middleName":"","lastName":"Pinilla","suffix":""},{"id":492056648,"identity":"847caa28-f366-405e-9ba4-3393c7729013","order_by":1,"name":"Ana Sanabria","email":"","orcid":"","institution":"Universidade de São Paulo","correspondingAuthor":false,"prefix":"","firstName":"Ana","middleName":"","lastName":"Sanabria","suffix":""},{"id":492056649,"identity":"1d482d83-60be-4b85-8a38-5191670656c0","order_by":2,"name":"Jia Bi","email":"","orcid":"","institution":"Rutherford Appleton laboratory","correspondingAuthor":false,"prefix":"","firstName":"Jia","middleName":"","lastName":"Bi","suffix":""},{"id":492056650,"identity":"b18d06ba-6514-4cdc-b5fd-f5cba776b0a2","order_by":3,"name":"Karen Egiazarian","email":"","orcid":"https://orcid.org/0000-0002-8135-1085","institution":"Tampere University","correspondingAuthor":false,"prefix":"","firstName":"Karen","middleName":"","lastName":"Egiazarian","suffix":""}],"badges":[],"createdAt":"2025-07-25 15:35:08","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7215670/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7215670/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":88209891,"identity":"9ab01958-3109-48f5-bff0-b9a6f50c3794","added_by":"auto","created_at":"2025-08-04 04:39:13","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":5542749,"visible":true,"origin":"","legend":"Article File","description":"","filename":"mainvf2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7215670/v1_covered_0a36ea7e-d9e2-4c93-b74f-ad7ae0afbaec.pdf"},{"id":88209519,"identity":"250a89c8-cd8e-494b-ad0e-29acb02fc05e","added_by":"auto","created_at":"2025-08-04 04:31:02","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":4359664,"visible":true,"origin":"","legend":"Supplemental material","description":"","filename":"supplementalmaterial.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7215670/v1/9c2bdb90a4eb72bde0a91b09.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"What Makes Neural Networks Trainable? Invexity as a Structural Design Principle in AI","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-7215670/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7215670/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eDespite their non-convex loss landscapes and vast parameter spaces, deep neural networks consistently achieve high performance across domains –from medical diagnostics to natural language processing and computer vision. However, the theoretical basis for their trainability remains unclear. Classical frameworks, such as convex optimization or probabilistic models (e.g., Bayesian optimization), offer only partial explanations and rely on restrictive assumptions that limit architectural expressiveness –such as shallow architectures, non-negative weights, or convex activations. This gap underscores a fundamental question: What Makes Neural Networks Trainable?. Here, we introduce a general framework based on invexity, a property that guarantees all critical points are global minima. We make four key contributions: (i) We demonstrate, for the first time, that the vast majority of commonly used activation functions –over 90% of fifty analyzed– are inherently invex. This reveals that modern architectures are already aligned with this property, even if unintentionally. ii) We show that deep Multilayer Perceptron models can be systematically constructed as invex structures, challenging the adequacy of existing convex and probabilistic optimization paradigms. iii) We prove that widely adopted architectures, such as ResNet, UNet, and Vision Transformer, satisfy invexity, providing a theoretical explanation for their empirical trainability, even at extreme depths, as is the case with ResNet. This provides a guarantee of accessibility to global optima for these well-known networks via standard gradient-based schemes. iv) By reframing trainability as a structural property rather than an empirical coincidence, our results provide a new foundation for understanding and designing neural networks.\u003c/p\u003e","manuscriptTitle":"What Makes Neural Networks Trainable? Invexity as a Structural Design Principle in AI","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-08-04 04:30:58","doi":"10.21203/rs.3.rs-7215670/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-communications","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"NCOMMS","sideBox":"Learn more about [Nature Communications](http://www.nature.com/ncomms/)","snPcode":"","submissionUrl":"https://mts-ncomms.nature.com/","title":"Nature Communications","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Communications","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"847c6620-c6af-4edc-bc8a-e00987384459","owner":[],"postedDate":"August 4th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":52241911,"name":"Physical sciences/Mathematics and computing/Computer science"},{"id":52241912,"name":"Physical sciences/Mathematics and computing/Scientific data"},{"id":52241913,"name":"Physical sciences/Mathematics and computing/Applied mathematics"}],"tags":[],"updatedAt":"2025-08-04T04:30:58+00:00","versionOfRecord":[],"versionCreatedAt":"2025-08-04 04:30:58","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7215670","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7215670","identity":"rs-7215670","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}