Improving SpikeProp’s Training Efficiency in Spiking Neural Networks for Large Language Models Through Innovative Weight Initialization

Improving SpikeProp’s Training Efficiency in Spiking Neural Networks for Large Language Models Through Innovative Weight Initialization | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Improving SpikeProp’s Training Efficiency in Spiking Neural Networks for Large Language Models Through Innovative Weight Initialization Falah. Y.H. Ahmed, Muhammad Zakarya, Naveed Khan, Dilovan Asaad Zebari, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6460749/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract Spiking neural networks (SNNs) mimic the functions of biological neurons by leveraging individual temporal spikes for communication and computation. Since SNN was perceived to be complex and analytically challenging, it had long been overlooked. In this study, we explore the enhancement of SpikeProp, a supervised learning model customized for SNNs. Three distinct models are being investigated, including the Proposed Model 1, the Proposed Model 2, and the Proposed Model 3, each providing unique improvements to the SpikeProp algorithm. To accelerate convergence and adapt learning rates, momentum factors are integrated into Proposed Model 1. In Proposed Model 2, a rate dependency is introduced based on Angle Driven Learning. By incorporating particle swarm optimization (PSO), Model 3 combines the strengths of Model 1 and 2. SNNs can be trained and classified more efficiently and accurately using these models. Furthermore, we examine how large language models (LLMs) might inform the design and interpretability of neural architectures and learning methodologies while also enhancing SNN training. Through the use of LLMs, we seek to enhance model transparency and encourage more Responsible AI (RAI) principles. A thorough evaluation and comparison of Proposed Model 1, Proposed Model 2, and Proposed Model 3 with traditional methods confirms that these models consistently outperform them. Consequently, they have a high potential for practical applications in neural network training in real-world settings and LLM-informed development, contributing to the advancement of AI systems. Spiking neural network particle swarm optimization Angel-driven dependency classifi- cation Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 30 May, 2025 Reviews received at journal 29 May, 2025 Reviewers agreed at journal 15 May, 2025 Reviews received at journal 14 May, 2025 Reviewers agreed at journal 14 May, 2025 Reviewers invited by journal 09 May, 2025 Editor assigned by journal 03 May, 2025 Submission checks completed at journal 01 May, 2025 First submitted to journal 16 Apr, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6460749","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":455027738,"identity":"3f2e0d1c-7b3e-4354-957c-1fb6c2dcc180","order_by":0,"name":"Falah. Y.H. Ahmed","email":"","orcid":"","institution":"Sohar University","correspondingAuthor":false,"prefix":"","firstName":"Falah.","middleName":"Y.H.","lastName":"Ahmed","suffix":""},{"id":455027739,"identity":"da46a449-30fb-4549-a23f-4ee3ca7587b4","order_by":1,"name":"Muhammad Zakarya","email":"","orcid":"","institution":"Sohar University","correspondingAuthor":false,"prefix":"","firstName":"Muhammad","middleName":"","lastName":"Zakarya","suffix":""},{"id":455027740,"identity":"4cf4b83c-e417-4851-bfca-0aa6d4380af1","order_by":2,"name":"Naveed Khan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAz0lEQVRIiWNgGAWjYFACNjYgYcPAhyRClJY0oDpm0rQcJkGLOQNb2oOfO84ntvGfP/i4gsFOnkEiLQGvFssGtuOGvWduJ7ZJJDMbnmFINmyQSDuAV4vBAfY2Cd42kBZmNskGBuYEBon0BoJaJP+2nQM67DD7zwaGemK0sB2T5m07kNjGkMzG2MBwGKiFkMMOs6VJy7YlGwP9YizZYHDcsI3nWQJ+LcfbzCTfttnJ9vMffPixoaJanp89zQCvFkRsQEwgIiJHwSgYBaNgFBAGAP6QOpwk3ZvLAAAAAElFTkSuQmCC","orcid":"","institution":"Ulster University","correspondingAuthor":true,"prefix":"","firstName":"Naveed","middleName":"","lastName":"Khan","suffix":""},{"id":455027741,"identity":"ecdb8fa5-0ced-4b36-b3e1-8b8ead44e316","order_by":3,"name":"Dilovan Asaad Zebari","email":"","orcid":"","institution":"Nawroz University","correspondingAuthor":false,"prefix":"","firstName":"Dilovan","middleName":"Asaad","lastName":"Zebari","suffix":""},{"id":455027742,"identity":"e4ae2d6d-a9c6-4ca2-bf48-24a3ecb575c9","order_by":4,"name":"Mahmood Al-Bahri","email":"","orcid":"","institution":"Sohar University","correspondingAuthor":false,"prefix":"","firstName":"Mahmood","middleName":"","lastName":"Al-Bahri","suffix":""},{"id":455027743,"identity":"dbe1af1b-fe9c-4482-9702-f61b1e4836af","order_by":5,"name":"Bwalya Kelvin Joseph","email":"","orcid":"","institution":"Sohar University","correspondingAuthor":false,"prefix":"","firstName":"Bwalya","middleName":"Kelvin","lastName":"Joseph","suffix":""}],"badges":[],"createdAt":"2025-04-16 07:39:13","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6460749/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6460749/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":82729482,"identity":"21dcfcac-6352-43c1-8b25-0738fdb10d02","added_by":"auto","created_at":"2025-05-14 14:35:17","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":640683,"visible":true,"origin":"","legend":"","description":"","filename":"SpikeNetworksUpdated.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6460749/v1_covered_203bf536-02c6-4d9b-bd39-497bd5784ba2.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Improving SpikeProp’s Training Efficiency in Spiking Neural Networks for Large Language Models Through Innovative Weight Initialization","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-computational-intelligence-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [International Journal of Computational Intelligence Systems](https://link.springer.com/journal/44196)","snPcode":"44196","submissionUrl":"https://submission.springernature.com/new-submission/44196/3","title":"International Journal of Computational Intelligence Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Open","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Spiking neural network, particle swarm optimization, Angel-driven dependency, classifi- cation","lastPublishedDoi":"10.21203/rs.3.rs-6460749/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6460749/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Spiking neural networks (SNNs) mimic the functions of biological neurons by leveraging individual temporal spikes for communication and computation. Since SNN was perceived to be complex and analytically challenging, it had long been overlooked. In this study, we explore the enhancement of SpikeProp, a supervised learning model customized for SNNs. Three distinct models are being investigated, including the Proposed Model 1, the Proposed Model 2, and the Proposed Model 3, each providing unique improvements to the SpikeProp algorithm. To accelerate convergence and adapt learning rates, momentum factors are integrated into Proposed Model 1. In Proposed Model 2, a rate dependency is introduced based on Angle Driven Learning. By incorporating particle swarm optimization (PSO), Model 3 combines the strengths of Model 1 and 2. SNNs can be trained and classified more efficiently and accurately using these models. Furthermore, we examine how large language models (LLMs) might inform the design and interpretability of neural architectures and learning methodologies while also enhancing SNN training. Through the use of LLMs, we seek to enhance model transparency and encourage more Responsible AI (RAI) principles. A thorough evaluation and comparison of Proposed Model 1, Proposed Model 2, and Proposed Model 3 with traditional methods confirms that these models consistently outperform them. Consequently, they have a high potential for practical applications in neural network training in real-world settings and LLM-informed development, contributing to the advancement of AI systems.","manuscriptTitle":"Improving SpikeProp’s Training Efficiency in Spiking Neural Networks for Large Language Models Through Innovative Weight Initialization","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-05-14 14:19:11","doi":"10.21203/rs.3.rs-6460749/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-05-30T21:49:29+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-05-29T22:20:00+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"39572928799274361213989083848059959063","date":"2025-05-15T08:20:41+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-05-14T15:42:53+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"269710763654812522477654460397212477951","date":"2025-05-14T14:43:22+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-05-09T09:35:17+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-05-04T03:17:29+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-05-01T05:07:43+00:00","index":"","fulltext":""},{"type":"submitted","content":"International Journal of Computational Intelligence Systems","date":"2025-04-16T07:28:20+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-computational-intelligence-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [International Journal of Computational Intelligence Systems](https://link.springer.com/journal/44196)","snPcode":"44196","submissionUrl":"https://submission.springernature.com/new-submission/44196/3","title":"International Journal of Computational Intelligence Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Open","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e318c913-c635-4cf6-ae40-805ea1048e21","owner":[],"postedDate":"May 14th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2025-08-11T22:53:21+00:00","versionOfRecord":[],"versionCreatedAt":"2025-05-14 14:19:11","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6460749","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6460749","identity":"rs-6460749","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}