Removing Temporal Dependencies in Spiking Neural Networks: A Feasibility Study on State Space Reformulation for Parallel-Trainable SNN Language Modeling

Removing Temporal Dependencies in Spiking Neural Networks: A Feasibility Study on State Space Reformulation for Parallel-Trainable SNN Language Modeling | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Removing Temporal Dependencies in Spiking Neural Networks: A Feasibility Study on State Space Reformulation for Parallel-Trainable SNN Language Modeling ChaeWoo Son This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9327191/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Spiking Neural Networks (SNNs) offer potential energy efficiency through event-driven computation, yet their inherent temporal dependencies—analogous to those in RNNs—have prevented them from being trained from scratch at the parameter regimes occupied by modern language models. We propose SNN-SSM, a framework that reformulates Leaky Integrate-and-Fire (LIF) neuron dynamics as a State Space Model (SSM), enabling parallel training via closed-form FFT convolution while preserving sparse spiking activations at inference time. To bridge the gap between the linear convolution used during training and the non-linear reset used during recurrent inference, we introduce a reset annealing schedule that gradually exposes the model to reset dynamics over the course of training. A four-way ablation of the reset annealing coefficient reveals an accuracy–gap trade-off: in controlled experiments on WikiText-2 at 11.1M parameters using a single NVIDIA Tesla T4 GPU, the accuracy-optimized configuration reaches 75.8 best validation perplexity with a 2.2-point train–inference gap, while the gap-optimized default configuration reaches 78.9 perplexity with the train–inference gap reduced to 0.46. Both configurations dramatically outperform a matched-size ANN baseline (152.1 PPL) and trail a matched-size Mamba reference (72.0 PPL) by 4–7 points. We interpret the gap to Mamba as the quantitative cost of committing to binary spike activations; under standard CMOS cost models (Horowitz, 2014), the resulting sparse activation pattern corresponds to an estimated ≈94% energy reduction on idealized neuromorphic hardware. Our contributions are (i) a demonstration that temporal dependencies can be removed from SNNs without collapsing sparsity, (ii) the reset annealing mechanism that makes the train-as-SSM / infer-as-SNN paradigm practically viable, and (iii) a staged empirical study including a faithful pure-PyTorch Mamba reference implementation and a four-way reset annealing ablation. Artificial Intelligence and Machine Learning spiking neural networks state space models LIF neuron language modeling parallel training reset annealing FFT convolution neuromorphic computing Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9327191","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":617827398,"identity":"89f99d9b-cd2f-458f-a118-0b43cbae89fa","order_by":0,"name":"ChaeWoo Son","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA0UlEQVRIiWNgGAWjYFAD9gYgYWBBvAYJBp4DIC0SpGiRSIDQBIF8e+/Bz7w7bOrMJZ9f3fCjQIKBv707Aa8WgzPnkqV5z6RJWM7OKbvZA3SYxJmzG/BrkcgxY85tOyxhcDsn7QYPUIuBRC5+LfIzwFr+SxjcPJN28w8xWhhugLUckDC4wX7sNlG2GJw5Yyz9ty1ZcmdPDtttGQMJHoJ+kW/vMfw4s82O35z9+LObb/7YyPG39xJwGNw6Bh4DEM1DnHKIFvYHxKseBaNgFIyCEQUAD/ND5xUN6LQAAAAASUVORK5CYII=","orcid":"https://orcid.org/0009-0009-6108-2373","institution":"Independent Researcher","correspondingAuthor":true,"prefix":"","firstName":"ChaeWoo","middleName":"","lastName":"Son","suffix":""}],"badges":[],"createdAt":"2026-04-05 15:28:58","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-9327191/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9327191/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":106279606,"identity":"9746570b-cfa4-42cc-a015-e7f96ff43b52","added_by":"auto","created_at":"2026-04-07 05:25:17","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":483135,"visible":true,"origin":"","legend":"","description":"","filename":"SNNSSMPaperFinalv4.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9327191/v1_covered_fc6f986e-fd9c-4661-971e-0bf068c81bd4.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eRemoving Temporal Dependencies in Spiking Neural Networks: A Feasibility Study on State Space Reformulation for Parallel-Trainable SNN Language Modeling\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Independent Researcher","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"spiking neural networks, state space models, LIF neuron, language modeling, parallel training, reset annealing, FFT convolution, neuromorphic computing","lastPublishedDoi":"10.21203/rs.3.rs-9327191/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9327191/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eSpiking Neural Networks (SNNs) offer potential energy efficiency through event-driven computation, yet their inherent temporal dependencies—analogous to those in RNNs—have prevented them from being trained from scratch at the parameter regimes occupied by modern language models. We propose SNN-SSM, a framework that reformulates Leaky Integrate-and-Fire (LIF) neuron dynamics as a State Space Model (SSM), enabling parallel training via closed-form FFT convolution while preserving sparse spiking activations at inference time. To bridge the gap between the linear convolution used during training and the non-linear reset used during recurrent inference, we introduce a reset annealing schedule that gradually exposes the model to reset dynamics over the course of training. A four-way ablation of the reset annealing coefficient reveals an accuracy–gap trade-off: in controlled experiments on WikiText-2 at 11.1M parameters using a single NVIDIA Tesla T4 GPU, the accuracy-optimized configuration reaches 75.8 best validation perplexity with a 2.2-point train–inference gap, while the gap-optimized default configuration reaches 78.9 perplexity with the train–inference gap reduced to 0.46. Both configurations dramatically outperform a matched-size ANN baseline (152.1 PPL) and trail a matched-size Mamba reference (72.0 PPL) by 4–7 points. We interpret the gap to Mamba as the quantitative cost of committing to binary spike activations; under standard CMOS cost models (Horowitz, 2014), the resulting sparse activation pattern corresponds to an estimated ≈94% energy reduction on idealized neuromorphic hardware. Our contributions are (i) a demonstration that temporal dependencies can be removed from SNNs without collapsing sparsity, (ii) the reset annealing mechanism that makes the train-as-SSM / infer-as-SNN paradigm practically viable, and (iii) a staged empirical study including a faithful pure-PyTorch Mamba reference implementation and a four-way reset annealing ablation.\u003c/p\u003e","manuscriptTitle":"Removing Temporal Dependencies in Spiking Neural Networks: A Feasibility Study on State Space Reformulation for Parallel-Trainable SNN Language Modeling","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-07 05:23:07","doi":"10.21203/rs.3.rs-9327191/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d4f302f1-80d7-4e6c-8ded-3cdb4ab065a7","owner":[],"postedDate":"April 7th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":65749441,"name":"Artificial Intelligence and Machine Learning"}],"tags":[],"updatedAt":"2026-04-07T05:23:08+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-07 05:23:07","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9327191","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9327191","identity":"rs-9327191","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}