CTP-Hybrid: From Hybrid Architecture to Native Spiking Foundation — A Two-Phase Report on Consumer-GPU Spiking LLMs

CTP-Hybrid: From Hybrid Architecture to Native Spiking Foundation — A Two-Phase Report on Consumer-GPU Spiking LLMs | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article CTP-Hybrid: From Hybrid Architecture to Native Spiking Foundation — A Two-Phase Report on Consumer-GPU Spiking LLMs Shutong Hou This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9709207/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract We present the complete two-phase development of CTP-Hybrid, a causal-topological pulsing architecture for large language models. Phase 1 established the first reproducible 7B-scale spiking-augmented LLM fine-tuning on an 8 GB consumer GPU (NVIDIA RTX 5060 Laptop), achieving WikiText-2 perplexity of 16.58 with fluent Chinese generation using TD-SCA microcolumns. However, full tri-module joint training (IST+TCP+DSM) failed across seven attempts due to gradient collapse from discrete pulse operations. Phase 2 resolved this fundamentally. We designed three custom autograd.Function implementations—IntervalSpikeEncoder, TemporalCausalMask, and LIFSpikeSurrogate—that provide stable surrogate gradients for previously non-differentiable pulse operations. These functions enabled: (1) the first successful tri-module joint training on a 7B hybrid model with zero NaN, and (2) a fully Transformer-free, 20M-parameter pure spiking baseline trained from scratch on WikiText-2, achieving a perplexity of 13.57. The pure spiking model contains no self-attention, no LayerNorm, and no floating-point FFN, proving that native pulse computation alone can learn natural language. All code, weights, and experiment logs are released. Artificial Intelligence and Machine Learning Large Language Model Causal Inference Consumer GPU Training Pulse Topology Native Spiking Foundation Spiking Neural Networks Full Text Additional Declarations The authors declare no competing interests. Supplementary Files neuroveinv2code.zip Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9709207","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":640025384,"identity":"be689a12-7221-482f-b022-0a30394013e8","order_by":0,"name":"Shutong Hou","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABHUlEQVRIie2RMUvDQBTHLxx4y0Edr4jNVzgIBCEFv8p7FJJFBHFpweFASZeia/wWujlGDtLl3ANd0sW5XSQu4kU6CF4go8P9hlve+/F/7x0hHs8/ZETpegdzwUeUvTa/CrRXGS/zlDRmOhkvVzNJYIAijYmD7V0aSVPGYpBCaogbVBqfakgXn60mYTF7F2SeoGJvpcsICsgkvmh8rLHacNA2t7JxJkPFL8GlUAGlQKPx3qZsSKcwFYsg16gEly7lSNgSdg12wqvWKuEt+xDBV7/CubZWnkbHdn3SDUbWK5ui+hXBcirhcOQTnmbc3vD6DKosyvmFUznXo/22PXzlvp0mk7DInuvdTXL6wIxT+TvpzwvdmoP6PR6Px+PiG1NPZtt+qIt+AAAAAElFTkSuQmCC","orcid":"https://orcid.org/0009-0006-0643-1225","institution":"Shanxi Agricultural University","correspondingAuthor":true,"prefix":"","firstName":"Shutong","middleName":"","lastName":"Hou","suffix":""}],"badges":[],"createdAt":"2026-05-14 03:27:42","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":true,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":true},"doi":"10.21203/rs.3.rs-9709207/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9709207/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":109296602,"identity":"f333597c-c244-49ec-a631-6ed0bac8ec1d","added_by":"auto","created_at":"2026-05-15 08:48:29","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":134269,"visible":true,"origin":"","legend":"","description":"","filename":"main.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9709207/v1_covered_82b33081-26e2-47fa-aed0-999516e4d7b3.pdf"},{"id":109289516,"identity":"73702c68-458b-40af-a3f9-05f1ef706e4f","added_by":"auto","created_at":"2026-05-15 06:27:57","extension":"zip","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":9529776,"visible":true,"origin":"","legend":"","description":"","filename":"neuroveinv2code.zip","url":"https://assets-eu.researchsquare.com/files/rs-9709207/v1/aca799939ce40afbea88f7a3.zip"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eCTP-Hybrid: From Hybrid Architecture to Native Spiking Foundation — A Two-Phase Report on Consumer-GPU Spiking LLMs\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Shanxi Agricultural University","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Large Language Model, Causal Inference, Consumer GPU Training, Pulse Topology, Native Spiking Foundation, Spiking Neural Networks","lastPublishedDoi":"10.21203/rs.3.rs-9709207/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9709207/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eWe present the complete two-phase development of CTP-Hybrid, a causal-topological pulsing architecture for large language models. Phase 1 established the first reproducible 7B-scale spiking-augmented LLM fine-tuning on an 8 GB consumer GPU (NVIDIA RTX 5060 Laptop), achieving WikiText-2 perplexity of 16.58 with fluent Chinese generation using TD-SCA microcolumns. However, full tri-module joint training (IST+TCP+DSM) failed across seven attempts due to gradient collapse from discrete pulse operations.\u003c/p\u003e\n\u003cp\u003ePhase 2 resolved this fundamentally. We designed three custom autograd.Function implementations—IntervalSpikeEncoder, TemporalCausalMask, and LIFSpikeSurrogate—that provide stable surrogate gradients for previously non-differentiable pulse operations. These functions enabled: (1) the first successful tri-module joint training on a 7B hybrid model with zero NaN, and (2) a fully Transformer-free, 20M-parameter pure spiking baseline trained from scratch on WikiText-2, achieving a perplexity of 13.57. The pure spiking model contains no self-attention, no LayerNorm, and no floating-point FFN, proving that native pulse computation alone can learn natural language. All code, weights, and experiment logs are released.\u003c/p\u003e","manuscriptTitle":"CTP-Hybrid: From Hybrid Architecture to Native Spiking Foundation — A Two-Phase Report on Consumer-GPU Spiking LLMs","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-05-15 06:27:51","doi":"10.21203/rs.3.rs-9709207/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"4f03dcd5-d0c3-435d-85f3-6e54633e7db3","owner":[],"postedDate":"May 15th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":68117640,"name":"Artificial Intelligence and Machine Learning"}],"tags":[],"updatedAt":"2026-05-15T06:27:51+00:00","versionOfRecord":[],"versionCreatedAt":"2026-05-15 06:27:51","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9709207","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9709207","identity":"rs-9709207","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}