A neuromorphic electronic artist for robotic painting

A neuromorphic electronic artist for robotic painting | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article A neuromorphic electronic artist for robotic painting Lioba Schürmann, Giulia D'Angelo, Giacomo Indiveri, Chiara Bartolozzi, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4528779/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Recent advances in computer vision and deep learning have led to a surge of interest in the field of AI-generated art, including digital image creation and robot-assisted painting. Traditional painting machines rely on static images and offline processing to incorporate visual feedback into their painting process. However, this approach does not consider the dynamic nature of painting and fails to decompose complex overlapping patterns into individual strokes. As an alternative to frame-based RGB cameras, neuromorphic cameras capture changes in light intensity within a scene via asynchronous event streams, promising to overcome some of the inherent limitations of traditional computer vision techniques. In this project, a robotic system for physical painting is presented which utilizes event-based visual input from a Dynamic Vision Sensor (DVS) camera. To take advantage of the camera's ultra-low latency and sparse encoding, the proposed system also employs event-based information processing, implemented with spiking neural networks on the neuromorphic DynapSE-1 processor. The robotic system receives DVS sensory data which represents the trajectory of a brush stroke and computes the required joint velocities to recreate the stroke with a 6-DOF robotic arm in a closed-loop manner. The controller additionally integrates tactile feedback from a force-torque sensor to dynamically adjust the end-effector’s distance towards the canvas depending on the brush’s deformation. Within the scope of the project, it was further demonstrated how speed information about a perceived brush stroke can be extracted from DVS data. The system was tested in a real-world setting and successfully generated a collection of physical brush strokes. The proposed network is a first step towards a fully spiking robotic controller with the ability to seamlessly incorporate event-based sensory feedback, providing ultra-low latency responsiveness. Beyond its utility in robot-assisted painting, the developed network is applicable to any robotic task requiring real-time adaptive control. Scientific community and society/Social sciences/Society Physical sciences/Engineering/Electrical and electronic engineering Physical sciences/Mathematics and computing/Computer science Full Text Additional Declarations Competing interest reported. Authors L.S., G.D., G.I. and L.G. declare no financial or non-financial interests. Author C.B. serves as Associate Editor of this journal and had no role in the peer-review or decision to publish this manuscript. Author C.B. declares no financial competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4528779","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":315279501,"identity":"6b7886b9-0506-4bbd-9ae1-0af84d78cd9f","order_by":0,"name":"Lioba Schürmann","email":"","orcid":"","institution":"University of Zurich","correspondingAuthor":false,"prefix":"","firstName":"Lioba","middleName":"","lastName":"Schürmann","suffix":""},{"id":315279502,"identity":"cd432117-d544-4870-bd0b-19e833bc40ff","order_by":1,"name":"Giulia D'Angelo","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABPUlEQVRIie2RMUvDQBTHXzi4Loc6JrQhX+EkkCno6NdIEJLlBqFLRcGUgi5iV0HQr2CXmw8CrUMWt4M6JHV1aAclLuJdkCo0cRbMb7j7H7wf7+4eQEvLXwSBIQIAI9GHHIBsd4ZDtQPZalbgW1GhZ12mKdUK/qWRgB+KT2UUmfrQpDgHhIr8FOzxzcVjHgyAUMm841e218OAioXcVHZTpQRTcK+fsj4NMiBW9uLNbX6oLoZdl9UoI61gCBPJIjM8V6+eMW9ucaQUgruNygeEd5LFpVZAMK9v8bNGxUFKUZXhvYynoJWdhygyVjxtVCjCRyK8Mt2JZMgMMpPoT+4afEYwqn+LM04nRfnm27cyLpbLgb+vR7l65ycqjIrnui6i2vQgCP0Kar6kWjfLqy7JOnbydTTK+uqWlpaW/8knjz9ndaMilpkAAAAASUVORK5CYII=","orcid":"","institution":"Italian Institute of Technology","correspondingAuthor":true,"prefix":"","firstName":"Giulia","middleName":"","lastName":"D'Angelo","suffix":""},{"id":315279503,"identity":"761f2d73-2e08-4ff0-bce7-e12c479c25ae","order_by":2,"name":"Giacomo Indiveri","email":"","orcid":"","institution":"University of Zurich","correspondingAuthor":false,"prefix":"","firstName":"Giacomo","middleName":"","lastName":"Indiveri","suffix":""},{"id":315279504,"identity":"d3e2e310-609a-4bc1-b6de-e332c0af2027","order_by":3,"name":"Chiara Bartolozzi","email":"","orcid":"","institution":"Italian Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"Chiara","middleName":"","lastName":"Bartolozzi","suffix":""},{"id":315279505,"identity":"e98ecbd3-507c-471b-8f7a-2b0fef6462da","order_by":4,"name":"Liat Grayver","email":"","orcid":"","institution":"Collegium Helveticum","correspondingAuthor":false,"prefix":"","firstName":"Liat","middleName":"","lastName":"Grayver","suffix":""}],"badges":[],"createdAt":"2024-06-04 14:27:51","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4528779/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4528779/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":59425652,"identity":"60b3e8af-1554-4209-8ba6-4b8a09c8a308","added_by":"auto","created_at":"2024-07-01 15:53:30","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":5250077,"visible":true,"origin":"","legend":"","description":"","filename":"npjRoboticsAneuromorphicelectronicartistforroboticpainting.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4528779/v1_covered_a9b38655-7c69-460e-95c8-0cb06d212b8d.pdf"}],"financialInterests":"Competing interest reported. Authors L.S., G.D., G.I. and L.G. declare no financial or non-financial interests. Author C.B. serves as Associate Editor of this journal and had no role in the peer-review or decision to publish this manuscript. Author C.B. declares no financial competing interests.","formattedTitle":"A neuromorphic electronic artist for robotic painting","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-4528779/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4528779/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Recent advances in computer vision and deep learning have led to a surge of interest in the field of AI-generated art, including digital image creation and robot-assisted painting. Traditional painting machines rely on static images and offline processing to incorporate visual feedback into their painting process. However, this approach does not consider the dynamic nature of painting and fails to decompose complex overlapping patterns into individual strokes. As an alternative to frame-based RGB cameras, neuromorphic cameras capture changes in light intensity within a scene via asynchronous event streams, promising to overcome some of the inherent limitations of traditional computer vision techniques. In this project, a robotic system for physical painting is presented which utilizes event-based visual input from a Dynamic Vision Sensor (DVS) camera. To take advantage of the camera's ultra-low latency and sparse encoding, the proposed system also employs event-based information processing, implemented with spiking neural networks on the neuromorphic DynapSE-1 processor. The robotic system receives DVS sensory data which represents the trajectory of a brush stroke and computes the required joint velocities to recreate the stroke with a 6-DOF robotic arm in a closed-loop manner. The controller additionally integrates tactile feedback from a force-torque sensor to dynamically adjust the end-effector’s distance towards the canvas depending on the brush’s deformation. Within the scope of the project, it was further demonstrated how speed information about a perceived brush stroke can be extracted from DVS data. The system was tested in a real-world setting and successfully generated a collection of physical brush strokes. The proposed network is a first step towards a fully spiking robotic controller with the ability to seamlessly incorporate event-based sensory feedback, providing ultra-low latency responsiveness. Beyond its utility in robot-assisted painting, the developed network is applicable to any robotic task requiring real-time adaptive control.","manuscriptTitle":"A neuromorphic electronic artist for robotic painting","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-06-19 02:28:59","doi":"10.21203/rs.3.rs-4528779/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"b1678329-f57f-4cbc-889a-5a501bfd5945","owner":[],"postedDate":"June 19th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":33336893,"name":"Scientific community and society/Social sciences/Society"},{"id":33336894,"name":"Physical sciences/Engineering/Electrical and electronic engineering"},{"id":33336895,"name":"Physical sciences/Mathematics and computing/Computer science"}],"tags":[],"updatedAt":"2024-10-25T03:53:03+00:00","versionOfRecord":[],"versionCreatedAt":"2024-06-19 02:28:59","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4528779","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4528779","identity":"rs-4528779","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}