Scaffolded representation learning in deep networks

Scaffolded representation learning in deep networks | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Scaffolded representation learning in deep networks Philipp Stecher, Sandro Radovanović, Vlasta Sikimić, Reinhard Kahle This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9269961/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Deep networks learn coarse structure before fine-grained distinctions, yet whether coarse structure actively scaffolds later differentiation remains untested. Here we show that representations assemble through a load-bearing scaffold. Tracking features at per-sample resolution across 55 runs, three architecture families and two training datasets, we find a reproducible three-phase program: task-general features emerge and dominate first, superclass groupings form next, and class-level distinctions develop last. Selectively corrupting superclass boundaries impairs later differentiation, suggesting that fine-grained learning depends on the coherence of coarser representations. Conversely, a curriculum that pre-builds the scaffold reduces differentiation cost 6.7-fold while nearly preserving accuracy and halving overfitting. These findings connect critical learning periods, neural collapse, progressive differentiation, the lottery ticket hypotheses, and catastrophic forgetting within a single developmental account and provide training diagnostic insights relevant for curriculum design, transfer timing, and mechanistic interpretability. Business and commerce/Information systems and information technology Scientific community and society/Scientific community Full Text Additional Declarations There is NO Competing Interest. Supplementary Files ChecklistScaffoldedRL.pdf Nature MI Checklist for article submissions Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9269961","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":615832919,"identity":"23700c3a-1d93-42ec-837e-dfe05058bd25","order_by":0,"name":"Philipp Stecher","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABAUlEQVRIiWNgGAWjYDACCcbGAwlQNjMQy4EYBx7g19IA0cIG0WIM1pKARweDBFABA5KWxAYQB58Wc+nmhgMPd9yJlp/ffPBzQcW99Plhhx8CbbGT023ArsVyzsGGA4lnnuVuOMaWLD3jTHHuxttpBkAtycZmB7BrMbiRCNTSdjh3AxuPGTNvW0LuxtkJIC0HErcR0jK/jf8bM++/hHTD2ekfiNPScIyHjZm3ISFBXjoHvy2WMxJhfkkzluY5lmC4QTqn4ECCAW6/mEukP3z4c8ed3PnNhx9+5qlJkJefnb75w4cKOzmc3gcRwNhEEjkAFydWi3wDbtWjYBSMglEwMgEAgz9ubUqwZ5MAAAAASUVORK5CYII=","orcid":"https://orcid.org/0009-0006-4149-1741","institution":"University of Tübingen","correspondingAuthor":true,"prefix":"","firstName":"Philipp","middleName":"","lastName":"Stecher","suffix":""},{"id":615832920,"identity":"470a1781-8e26-4da9-8ebd-3a96aba30ab9","order_by":1,"name":"Sandro Radovanović","email":"","orcid":"","institution":"University of Belgrade","correspondingAuthor":false,"prefix":"","firstName":"Sandro","middleName":"","lastName":"Radovanović","suffix":""},{"id":615832921,"identity":"274ac8ce-a40a-4db1-b191-acde2960ec58","order_by":2,"name":"Vlasta Sikimić","email":"","orcid":"","institution":"Philosophy \u0026 Ethics, Department Industrial Engineering \u0026 Innovation Sciences, Eindhoven University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Vlasta","middleName":"","lastName":"Sikimić","suffix":""},{"id":615832922,"identity":"6364cb3c-0416-4a75-84b0-3001855e9153","order_by":3,"name":"Reinhard Kahle","email":"","orcid":"","institution":"University of Tübingen","correspondingAuthor":false,"prefix":"","firstName":"Reinhard","middleName":"","lastName":"Kahle","suffix":""}],"badges":[],"createdAt":"2026-03-30 16:45:57","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9269961/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9269961/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":107483281,"identity":"d27100f3-8572-422f-927d-f60bb38152a2","added_by":"auto","created_at":"2026-04-22 02:27:09","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":5286716,"visible":true,"origin":"","legend":"Article File","description":"","filename":"ScaffoldedDevelopmentvFinal.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9269961/v1_covered_5ce78c12-55c1-4cb7-bba5-67f0ab6501f7.pdf"},{"id":107099451,"identity":"0ada3b09-2d98-4d8c-9673-14bdb272b380","added_by":"auto","created_at":"2026-04-16 18:30:17","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":350014,"visible":true,"origin":"","legend":"Nature MI Checklist for article submissions","description":"","filename":"ChecklistScaffoldedRL.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9269961/v1/5e58d53c3d3c93cc57f803d8.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Scaffolded representation learning in deep networks","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-9269961/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9269961/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Deep networks learn coarse structure before fine-grained distinctions, yet whether coarse structure actively scaffolds later differentiation remains untested. Here we show that representations assemble through a load-bearing scaffold. Tracking features at per-sample resolution across 55 runs, three architecture families and two training datasets, we find a reproducible three-phase program: task-general features emerge and dominate first, superclass groupings form next, and class-level distinctions develop last. Selectively corrupting superclass boundaries impairs later differentiation, suggesting that fine-grained learning depends on the coherence of coarser representations. Conversely, a curriculum that pre-builds the scaffold reduces differentiation cost 6.7-fold while nearly preserving accuracy and halving overfitting. These findings connect critical learning periods, neural collapse, progressive differentiation, the lottery ticket hypotheses, and catastrophic forgetting within a single developmental account and provide training diagnostic insights relevant for curriculum design, transfer timing, and mechanistic interpretability.","manuscriptTitle":"Scaffolded representation learning in deep networks","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-16 18:30:06","doi":"10.21203/rs.3.rs-9269961/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"ee1b9d15-325a-4a28-9bf3-7d579fa34340","owner":[],"postedDate":"April 16th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":65532655,"name":"Business and commerce/Information systems and information technology"},{"id":65532656,"name":"Scientific community and society/Scientific community"}],"tags":[],"updatedAt":"2026-04-16T18:30:06+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-16 18:30:06","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9269961","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9269961","identity":"rs-9269961","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}