Ultra-low power in-sensor neuronal computing with oscillatory retinal neurons for frequency-multiplexed, parallel machine vision

Ultra-low power in-sensor neuronal computing with oscillatory retinal neurons for frequency-multiplexed, parallel machine vision | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Physical Sciences - Article Ultra-low power in-sensor neuronal computing with oscillatory retinal neurons for frequency-multiplexed, parallel machine vision Ragib Ahsan, Hyun Uk Chae, Seyedeh Atiyeh Abbasi Jalal, Jun Tao, and 7 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-2935296/v2 This work is licensed under a CC BY 4.0 License Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Abstract In-sensor and near-sensor computing architectures enable multiply-accumulate operations to be carried out directly at the point of sensing. In-sensor architectures offer dramatic power and speed improvements over traditional von Neumann architectures by eliminating multiple analog-to-digital conversions, data storage, and data movement operations. Current in-sensor processing approaches rely on tunable sensors or additional weighting elements to perform linear functions such as multiply-accumulate operations as the sensor acquires data. We implement in-sensor computing with an oscillatory retinal neuron device that converts incident optical signals into voltage oscillations. We introduce a computing scheme based on the frequency shift of coupled oscillators that enables parallel, frequency multiplexed, non-linear operations on the inputs. Simulation elucidates how this computing occurs. We experimentally implement a 3×3 focal plane array of coupled neurons and show that functions approximating edge detection, thresholding, and segmentation occur in parallel . An example of inference on handwritten digits from the MNIST database is also experimentally demonstrated with a 3×3 array of coupled neurons feeding into a single hidden layer neural network, approximating a liquid state machine. This network demonstrated a reduction in classification error from 2.16% to 1.84% compared to the same neural network with standard photodetector inputs. Finally, the equivalent energy consumption to carry out image processing operations, including peripherals such as the Fourier transform circuits, is projected to be < 20 fJ/OP, possibly reaching as low as 15 aJ/OP. Physical sciences/Engineering/Electrical and electronic engineering Physical sciences/Mathematics and computing/Computational science Full Text Additional Declarations The authors declare no competing interests. Supplementary Files SupplementaryORNnatureelectronicssubmissionunmarked.pdf Supplementary Note Cite Share Download PDF Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-2935296","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Physical Sciences - Article","associatedPublications":[],"authors":[{"id":209682854,"identity":"4d5b089b-9be4-4dda-9efe-5a1072790ac3","order_by":0,"name":"Ragib Ahsan","email":"","orcid":"https://orcid.org/0000-0002-3833-7851","institution":"University of Southern California","correspondingAuthor":false,"prefix":"","firstName":"Ragib","middleName":"","lastName":"Ahsan","suffix":""},{"id":209682855,"identity":"e3b68396-05ba-4f77-82b4-0574b0bbc0e9","order_by":1,"name":"Hyun Uk Chae","email":"","orcid":"","institution":"University of Southern California","correspondingAuthor":false,"prefix":"","firstName":"Hyun","middleName":"Uk","lastName":"Chae","suffix":""},{"id":209682856,"identity":"2d85cce9-2d28-4b30-82bc-c7de548f4912","order_by":2,"name":"Seyedeh Atiyeh Abbasi Jalal","email":"","orcid":"","institution":"University of Southern California","correspondingAuthor":false,"prefix":"","firstName":"Seyedeh","middleName":"Atiyeh Abbasi","lastName":"Jalal","suffix":""},{"id":209682857,"identity":"b856efb6-5918-49d4-b2eb-ab9ce4fb5890","order_by":3,"name":"Jun Tao","email":"","orcid":"https://orcid.org/0000-0003-2156-1731","institution":"University of Southern California","correspondingAuthor":false,"prefix":"","firstName":"Jun","middleName":"","lastName":"Tao","suffix":""},{"id":209682858,"identity":"61d0a8d4-2268-4788-8642-32de0bd2acc2","order_by":4,"name":"Subrata Das","email":"","orcid":"","institution":"University of Southern California","correspondingAuthor":false,"prefix":"","firstName":"Subrata","middleName":"","lastName":"Das","suffix":""},{"id":209682859,"identity":"32e2eedb-0261-440e-bd9f-704534184704","order_by":5,"name":"Hefei Liu","email":"","orcid":"https://orcid.org/0000-0001-6533-7112","institution":"University of Southern California","correspondingAuthor":false,"prefix":"","firstName":"Hefei","middleName":"","lastName":"Liu","suffix":""},{"id":209682860,"identity":"321ae4db-a389-4f83-8416-bbe6fb6f8d81","order_by":6,"name":"Jiang-Bin Wu","email":"","orcid":"https://orcid.org/0000-0002-8751-7082","institution":"University of Southern California","correspondingAuthor":false,"prefix":"","firstName":"Jiang-Bin","middleName":"","lastName":"Wu","suffix":""},{"id":209682861,"identity":"3b5905f8-2116-4287-be13-113802bbfa11","order_by":7,"name":"Stephen Cronin","email":"","orcid":"","institution":"University of Southern California","correspondingAuthor":false,"prefix":"","firstName":"Stephen","middleName":"","lastName":"Cronin","suffix":""},{"id":209682862,"identity":"a3f47547-82ba-49f3-9fdf-2b4a272bc0f3","order_by":8,"name":"Han Wang","email":"","orcid":"","institution":"University of Southern California","correspondingAuthor":false,"prefix":"","firstName":"Han","middleName":"","lastName":"Wang","suffix":""},{"id":209682863,"identity":"eaa556b7-a40b-4ea6-b1b9-e84b14d86d47","order_by":9,"name":"Constantine Sideris","email":"","orcid":"","institution":"University of Southern California","correspondingAuthor":false,"prefix":"","firstName":"Constantine","middleName":"","lastName":"Sideris","suffix":""},{"id":209682864,"identity":"83ce96e8-1573-4f6e-a5d7-7c9a8a9b004e","order_by":10,"name":"Rehan Kapadia","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAv0lEQVRIiWNgGAWjYDACHhDBJsHAD+Eyk6BFsoFELQwMBgeI1WLec/jZgx9lFvLGN9IffmCosE5sIKRF5mybuWHPOQnDbTdyjCUYzqQT1iLBz2AmwdsmwQjUwsbA2HaYGC3s3yT/tknYb56R/oyB8R8xWnh7zKSBtiRukEgwY2BsIEYLz5lyY5lzEskzzrwxlkg4lm5MhJb0bQ/flNXZ9rcDQ+xDjbUsQS0M4EiBgQQilKNpGQWjYBSMglGADQAARE85CUSZzDQAAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0002-7611-0551","institution":"[email protected]","correspondingAuthor":true,"prefix":"","firstName":"Rehan","middleName":"","lastName":"Kapadia","suffix":""}],"badges":[],"createdAt":"2023-05-15 04:40:41","currentVersionCode":2,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-2935296/v2","doiUrl":"https://doi.org/10.21203/rs.3.rs-2935296/v2","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":50380837,"identity":"7305d3bc-1a59-46c6-ac4f-d8c4ca01d147","added_by":"auto","created_at":"2024-01-30 16:58:56","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2303491,"visible":true,"origin":"","legend":"","description":"","filename":"ManuscriptORNsubmissionnatureelectronicsunmarked.pdf","url":"https://assets-eu.researchsquare.com/files/rs-2935296/v2_covered_51de5a5d-dd7a-4b47-adf8-df1d25efad9c.pdf"},{"id":50380247,"identity":"ad29ac8a-4ed2-42f0-b29a-0fced18b943e","added_by":"auto","created_at":"2024-01-30 16:42:50","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":3965488,"visible":true,"origin":"","legend":"\u003cp\u003eSupplementary Note\u003c/p\u003e","description":"","filename":"SupplementaryORNnatureelectronicssubmissionunmarked.pdf","url":"https://assets-eu.researchsquare.com/files/rs-2935296/v2/9a01f82a071b8c64d9bebd57.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"Ultra-low power in-sensor neuronal computing with oscillatory retinal neurons for frequency-multiplexed, parallel machine vision","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-2935296/v2","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-2935296/v2","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eIn-sensor and near-sensor computing architectures enable multiply-accumulate operations to be carried out directly at the point of sensing. In-sensor architectures offer dramatic power and speed improvements over traditional von Neumann architectures by eliminating multiple analog-to-digital conversions, data storage, and data movement operations. Current in-sensor processing approaches rely on tunable sensors or additional weighting elements to perform linear functions such as multiply-accumulate operations as the sensor acquires data. We implement in-sensor computing with an oscillatory retinal neuron device that converts incident optical signals into voltage oscillations. We introduce a computing scheme based on the frequency shift of coupled oscillators that enables parallel, frequency multiplexed, non-linear operations on the inputs. Simulation elucidates how this computing occurs. We experimentally implement a 3×3 focal plane array of coupled neurons and show that functions approximating edge detection, thresholding, and segmentation occur \u003cem\u003ein parallel\u003c/em\u003e. An example of inference on handwritten digits from the MNIST database is also experimentally demonstrated with a 3×3 array of coupled neurons feeding into a single hidden layer neural network, approximating a liquid state machine. This network demonstrated a reduction in classification error from 2.16% to 1.84% compared to the same neural network with standard photodetector inputs.\u003cstrong\u003e \u003c/strong\u003eFinally, the equivalent energy consumption to carry out image processing operations, including peripherals such as the Fourier transform circuits, is projected to be \u0026lt; 20 fJ/OP, possibly reaching as low as 15 aJ/OP.\u003c/p\u003e","manuscriptTitle":"Ultra-low power in-sensor neuronal computing with oscillatory retinal neurons for frequency-multiplexed, parallel machine vision","msid":"","msnumber":"","nonDraftVersions":[{"code":2,"date":"2024-01-30 16:42:45","doi":"10.21203/rs.3.rs-2935296/v2","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}},{"code":1,"date":"2023-06-14 04:08:55","doi":"10.21203/rs.3.rs-2935296/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"907d2cb6-8a17-4907-be64-de5a4e16d3c0","owner":[],"postedDate":"January 30th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":22378990,"name":"Physical sciences/Engineering/Electrical and electronic engineering"},{"id":22378991,"name":"Physical sciences/Mathematics and computing/Computational science"}],"tags":[],"updatedAt":"2024-01-09T10:40:52+00:00","versionOfRecord":[],"versionCreatedAt":"2024-01-30 16:42:45","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v2","identity":"rs-2935296","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-2935296","identity":"rs-2935296","version":["v2"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}