NEOSTI: A Neuromorphic Electronic-Opto Spatial-Temporal Hybrid Image Sensor

NEOSTI: A Neuromorphic Electronic-Opto Spatial-Temporal Hybrid Image Sensor | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article NEOSTI: A Neuromorphic Electronic-Opto Spatial-Temporal Hybrid Image Sensor Milin Zhang, Tianyi Liu, Zheng Huang, Xuecheng Wang, Wanxin Shi, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5770022/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Image sensors in machine vision systems face significant challenges related to energy efficiency and processing capability when storing, transferring, and processing massive amounts of data. In humans, over 80% of information processed by the brain is obtained through the eyes, which are capable of detecting and synchronously processing information with extremely low overall power consumption. Inspired by the biomimetics, here we propose a Neuromorphic Electronic-Opto Spatial Temporal Imager (NEOSTI), the smallest all-in-one eye size fully integrated vision system enabling acquisition and operation in typical indoor/outdoor non-coherent environments, under both natural and artificial lighting conditions, without any extra requirement of the light source, such as laser or coherent light source. NEOSTI combines processing-pre-sensor (PPS) in optical domain, processing-in-sensor (PIS) with non-linear acquisition capability while optical to electronic converting, and processing-near-sensor (PNS) in electronic domain, enabling parallel data computing capabilities while sensing. NEOSTI also integrates a low complexity Binary Neural Network (BNN) on the chip to process image semantic information. It attains near-human performance in five static and dynamic visual processing tasks. Physical sciences/Engineering/Electrical and electronic engineering Physical sciences/Optics and photonics/Optical techniques/Imaging and sensing Neuromorphic Image sensor Processing in sensor Non-coherent image acquisition Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5770022","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":403348000,"identity":"bf15ac74-02dc-4401-b587-caccf8d78db3","order_by":0,"name":"Milin Zhang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABGklEQVRIiWNgGAWjYDACCQgpw8bAfICBwQCMgICNsBYeNga2xAZStDDwAJFhA4hFUIv87OZnD7/mWPDwSfd8f/ij4I7ddonkBwwfyg4z8M9uwKqFcc4xc2PZbUCHyZzd2Mxj8Cx554w0A8YZ5w4zSNw5gFULs0SCmbQkSItE7sZmBoPDyQY3EgyYedsOMxhIJGDVwiaR/g2qJedh4w+wlvQPzH/xaOGRyDGT/AjRwtjAY3DYzuBGjgEzIx4tEhI5ZdKMYC1phrOBWhIMzrwpONhzLp1H4gZ2LfIz0rdJ/txWJyc/I/nBxx9/DtsbHE/f+OBHmbUc/wzsWsBBwIPEAcYnA8MBBnBM4QaMP5A49vhUjoJRMApGwcgEAMgZWjLn4KUaAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0001-6915-0734","institution":"Tsinghua University","correspondingAuthor":true,"prefix":"","firstName":"Milin","middleName":"","lastName":"Zhang","suffix":""},{"id":403348001,"identity":"85222751-80ef-48f6-891d-3d286bcee6fc","order_by":1,"name":"Tianyi Liu","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Tianyi","middleName":"","lastName":"Liu","suffix":""},{"id":403348002,"identity":"7101594a-36f6-49c7-b15b-3c0375e54093","order_by":2,"name":"Zheng Huang","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Zheng","middleName":"","lastName":"Huang","suffix":""},{"id":403348003,"identity":"dda91654-723d-4c19-b157-2a244169e8ea","order_by":3,"name":"Xuecheng Wang","email":"","orcid":"https://orcid.org/0000-0003-1778-2259","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Xuecheng","middleName":"","lastName":"Wang","suffix":""},{"id":403348004,"identity":"4c0cabb0-a2ab-4caf-ac40-cb58aef98003","order_by":4,"name":"Wanxin Shi","email":"","orcid":"","institution":"Tsinghua University, China Mobile Research Institute","correspondingAuthor":false,"prefix":"","firstName":"Wanxin","middleName":"","lastName":"Shi","suffix":""},{"id":403348005,"identity":"67523046-d139-41dc-ad09-a21082b6f8f0","order_by":5,"name":"Hongwei Chen","email":"","orcid":"https://orcid.org/0000-0002-2952-2203","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Hongwei","middleName":"","lastName":"Chen","suffix":""}],"badges":[],"createdAt":"2025-01-06 02:25:22","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5770022/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5770022/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":84086634,"identity":"f9938c8f-4c24-46a6-bebe-6b95d8ff5c22","added_by":"auto","created_at":"2025-06-06 15:10:01","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":8370051,"visible":true,"origin":"","legend":"Article File","description":"","filename":"NEOSTIblind.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5770022/v1_covered_e27f9d70-1f8e-4e9b-8ea2-8a42b0683651.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"NEOSTI: A Neuromorphic Electronic-Opto Spatial-Temporal Hybrid Image Sensor","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Neuromorphic, Image sensor, Processing in sensor, Non-coherent image acquisition","lastPublishedDoi":"10.21203/rs.3.rs-5770022/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5770022/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Image sensors in machine vision systems face significant challenges related to energy efficiency and processing capability when storing, transferring, and processing massive amounts of data. In humans, over 80% of information processed by the brain is obtained through the eyes, which are capable of detecting and synchronously processing information with extremely low overall power consumption. Inspired by the biomimetics, here we propose a Neuromorphic Electronic-Opto Spatial Temporal Imager (NEOSTI), the smallest all-in-one eye size fully integrated vision system enabling acquisition and operation in typical indoor/outdoor non-coherent environments, under both natural and artificial lighting conditions, without any extra requirement of the light source, such as laser or coherent light source. NEOSTI combines processing-pre-sensor (PPS) in optical domain, processing-in-sensor (PIS) with non-linear acquisition capability while optical to electronic converting, and processing-near-sensor (PNS) in electronic domain, enabling parallel data computing capabilities while sensing. NEOSTI also integrates a low complexity Binary Neural Network (BNN) on the chip to process image semantic information. It attains near-human performance in five static and dynamic visual processing tasks.","manuscriptTitle":"NEOSTI: A Neuromorphic Electronic-Opto Spatial-Temporal Hybrid Image Sensor","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-01-20 04:14:06","doi":"10.21203/rs.3.rs-5770022/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"88f999b4-b66f-4615-aecc-0940a834faa6","owner":[],"postedDate":"January 20th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":43004082,"name":"Physical sciences/Engineering/Electrical and electronic engineering"},{"id":43004083,"name":"Physical sciences/Optics and photonics/Optical techniques/Imaging and sensing"}],"tags":[],"updatedAt":"2025-07-25T16:35:50+00:00","versionOfRecord":[],"versionCreatedAt":"2025-01-20 04:14:06","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5770022","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5770022","identity":"rs-5770022","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}