Temporal Hierarchy in Spiking Neural Networks

Temporal Hierarchy in Spiking Neural Networks | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Temporal Hierarchy in Spiking Neural Networks Filippo Moro, Pau Vilimelis Aceituno, Laura Kriener, Melika Payvand This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6073810/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Taking inspiration from the brain to perform efficient computation is a fascinating challenge, often hampered by the complexity of the brain itself. This work investigates a key feature observed across the cortices of various mammals: the hierarchy of time scales in cortical areas. Experimental evidence suggests that the intrinsic speed of activity in neuronal populations progressively slows down through the cortical hierarchy, forming what we define as a ``temporal hierarchy''. We explore whether this property provides an advantage in artificial computational systems. To test this hypothesis, we analyze SNNs, neural networks inspired by biological computation and with inherent temporal dynamics, and we endow them with temporal hierarchy. We incorporate the temporal hierarchy in various SNN temporal parameters, including their neuronal dynamics, synaptic delays, and recurrent dynamics. Our study evaluates these temporal hierarchies under two settings, (1) as an inductive bias, where SNNs temporal parameters are initialized with a predefined hierarchy to evaluate performance improvements, and (2) through optimizing the temporal parameters of SNNs. Benchmarking these networks on temporal tasks, the Multi-Timescale-XOR, and keyword spotting, we validate the benefits of hierarchy in various temporal mechanisms under both settings. In the inductive bias setting, we show that under iso-parameters settings, (i) classification accuracy correlates positively with the magnitude of temporal hierarchy on all the benchmarks, and (ii) there is an accuracy gain between 2 to 6% across tasks, when introducing temporal hierarchy, compared to a network without hierarchy. Moreover, under iso-accuracy settings, introducing hierarchy reduces the required number of parameters by up to 5 times. In the emergence from the optimization setting, we show that temporal hierarchy is naturally found as an emergent property through gradient descent. Finally, we conduct a detailed mathematical analysis of the temporal processing capabilities of SNNs, showing that a hierarchical arrangement of time constants enables a logarithmic reduction in the number of layers required to process temporal signals with multiple frequency components. Physical sciences/Mathematics and computing/Computational science Physical sciences/Engineering/Electrical and electronic engineering Physical sciences/Mathematics and computing/Computer science Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6073810","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":437190427,"identity":"fbab9d2e-1c71-4ef9-96e1-d3805ba92f5f","order_by":0,"name":"Filippo Moro","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAw0lEQVRIiWNgGAWjYBAC9mbmhgNw3gcGhgQGBh78WngOMyK0MM4gSssBxgY4h5mHKC3sjI2HC3fYMOi29z58bNtWl8fAfvYAfi3MjA2HZ55JYzA7c9zYOLeNrZiBJy8BrxZ7kBbetsMMZjfS2KRz23gSGyR4DAjbAtdi2SZBqhbGNgMitQD9wmN25hizYc+5hMQ2nhwCWvgPH/4MDDE5s+NtjA9+lNUl9rOfwa8FBIAWIcUFG0H1UC2jYBSMglEwCnADACkZP5vLkE+sAAAAAElFTkSuQmCC","orcid":"","institution":"Institute of Neuroinformatics, University of Zurich and ETH Zurich","correspondingAuthor":true,"prefix":"","firstName":"Filippo","middleName":"","lastName":"Moro","suffix":""},{"id":437190428,"identity":"f70d0caf-8531-43a5-9970-5aa8364b9c1c","order_by":1,"name":"Pau Vilimelis Aceituno","email":"","orcid":"","institution":"Institute of Neuroinformatics, University of Zurich and ETH Zurich","correspondingAuthor":false,"prefix":"","firstName":"Pau","middleName":"Vilimelis","lastName":"Aceituno","suffix":""},{"id":437190429,"identity":"525b5a19-f394-481d-bc76-5a2fa487e5e7","order_by":2,"name":"Laura Kriener","email":"","orcid":"","institution":"Institute of Neuroinformatics, University of Zurich and ETH Zurich","correspondingAuthor":false,"prefix":"","firstName":"Laura","middleName":"","lastName":"Kriener","suffix":""},{"id":437190430,"identity":"1ca31695-98ae-436c-80c4-6b6dd80f7e52","order_by":3,"name":"Melika Payvand","email":"","orcid":"https://orcid.org/0000-0001-5400-067X","institution":"University of Zurich and ETH Zurich","correspondingAuthor":false,"prefix":"","firstName":"Melika","middleName":"","lastName":"Payvand","suffix":""}],"badges":[],"createdAt":"2025-02-20 17:10:17","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6073810/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6073810/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":82185322,"identity":"a5eb7e39-8e2b-48b3-8f34-bfc86a6826ab","added_by":"auto","created_at":"2025-05-07 12:55:10","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1133513,"visible":true,"origin":"","legend":"Article File","description":"","filename":"TemporalHierarchy.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6073810/v1_covered_0b8a0bcd-8df8-41a3-9e44-4c9b24c2dc3c.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Temporal Hierarchy in Spiking Neural Networks","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-6073810/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6073810/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Taking inspiration from the brain to perform efficient computation is a fascinating challenge, often hampered by the complexity of the brain itself. This work investigates a key feature observed across the cortices of various mammals: the hierarchy of time scales in cortical areas. Experimental evidence suggests that the intrinsic speed of activity in neuronal populations progressively slows down through the cortical hierarchy, forming what we define as a ``temporal hierarchy''. We explore whether this property provides an advantage in artificial computational systems. To test this hypothesis, we analyze SNNs, neural networks inspired by biological computation and with inherent temporal dynamics, and we endow them with temporal hierarchy. We incorporate the temporal hierarchy in various SNN temporal parameters, including their neuronal dynamics, synaptic delays, and recurrent dynamics. Our study evaluates these temporal hierarchies under two settings, (1) as an inductive bias, where SNNs temporal parameters are initialized with a predefined hierarchy to evaluate performance improvements, and (2) through optimizing the temporal parameters of SNNs.\r\nBenchmarking these networks on temporal tasks, the Multi-Timescale-XOR, and keyword spotting, we validate the benefits of hierarchy in various temporal mechanisms under both settings.\r\nIn the inductive bias setting, we show that under iso-parameters settings, (i) classification accuracy correlates positively with the magnitude of temporal hierarchy on all the benchmarks, and (ii) there is an accuracy gain between 2 to 6% across tasks, when introducing temporal hierarchy, compared to a network without hierarchy. Moreover, under iso-accuracy settings, introducing hierarchy reduces the required number of parameters by up to 5 times. In the emergence from the optimization setting, we show that temporal hierarchy is naturally found as an emergent property through gradient descent. Finally, we conduct a detailed mathematical analysis of the temporal processing capabilities of SNNs, showing that a hierarchical arrangement of time constants enables a logarithmic reduction in the number of layers required to process temporal signals with multiple frequency components.","manuscriptTitle":"Temporal Hierarchy in Spiking Neural Networks","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-03 12:31:36","doi":"10.21203/rs.3.rs-6073810/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d4bb60e3-0ab6-4fb7-9ffb-aa299dc4f904","owner":[],"postedDate":"April 3rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":46550824,"name":"Physical sciences/Mathematics and computing/Computational science"},{"id":46550825,"name":"Physical sciences/Engineering/Electrical and electronic engineering"},{"id":46550826,"name":"Physical sciences/Mathematics and computing/Computer science"}],"tags":[],"updatedAt":"2025-05-07T12:47:02+00:00","versionOfRecord":[],"versionCreatedAt":"2025-04-03 12:31:36","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6073810","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6073810","identity":"rs-6073810","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}