Spatial-frequency Attention-based Optical and Scene Flow with Cross-Modal Knowledge Distillation

Spatial-frequency Attention-based Optical and Scene Flow with Cross-Modal Knowledge Distillation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Spatial-frequency Attention-based Optical and Scene Flow with Cross-Modal Knowledge Distillation Youjie Zhou, Runyu Jiao, Zhonghan Tao, Xichang Liang, Yi Wan This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4712095/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 09 Oct, 2024 Read the published version in The Visual Computer → Version 1 posted 11 You are reading this latest preprint version Abstract This paper studies the problem of multimodal fusion for optical and scene flow from RGB and depth images, or point clouds. Previous methods fuse multimodal information in “early-fusion” or “late-fusion” strategies, in which an attention mechanism is employed to address the problem of optical and scene flow estimation when RGB information is unreliable. Such attentive approaches either suffer from substantial computational and time complexities or lose the inherent characteristics of features due to downsampling. To address this issue, we propose a novel multimodal fusion approach named SFRAFT, which utilizes Fourier transform to build the spatial-frequency domain transformed self-attention and cross-attention. With the novel attentive mechanism, our approach can extract informative features more efficiently and effectively. We further enhance information exchange between the two modalities by incorporating multi-scale knowledge distillation. Experimental results on Flythings3D and KITTI show that our SFRAFT achieves the best performance with low computational and time complexity. We also prove the strong ability of our approach for flow estimation on our real-world dataset. We release the code and datasets at https://doi.org/10.5281/zenodo.12697968 Optical and scene flow multimodal fusion spatial-frequency domain transform attention knowledge distillation Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 09 Oct, 2024 Read the published version in The Visual Computer → Version 1 posted Editorial decision: Revision requested 22 Aug, 2024 Reviews received at journal 19 Aug, 2024 Reviews received at journal 29 Jul, 2024 Reviewers agreed at journal 23 Jul, 2024 Reviews received at journal 22 Jul, 2024 Reviewers agreed at journal 19 Jul, 2024 Reviewers agreed at journal 18 Jul, 2024 Reviewers invited by journal 18 Jul, 2024 Editor assigned by journal 10 Jul, 2024 Submission checks completed at journal 10 Jul, 2024 First submitted to journal 09 Jul, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4712095","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":333035136,"identity":"e7d36ebb-0a86-42ad-9291-6e047c9959cc","order_by":0,"name":"Youjie Zhou","email":"","orcid":"","institution":"Shandong University","correspondingAuthor":false,"prefix":"","firstName":"Youjie","middleName":"","lastName":"Zhou","suffix":""},{"id":333035137,"identity":"58ff8823-f74e-4179-8326-f97b5c2ab5ea","order_by":1,"name":"Runyu Jiao","email":"","orcid":"","institution":"Shandong University","correspondingAuthor":false,"prefix":"","firstName":"Runyu","middleName":"","lastName":"Jiao","suffix":""},{"id":333035138,"identity":"0e5260c8-9560-4314-80ad-3aea4c95fee2","order_by":2,"name":"Zhonghan Tao","email":"","orcid":"","institution":"Shandong University","correspondingAuthor":false,"prefix":"","firstName":"Zhonghan","middleName":"","lastName":"Tao","suffix":""},{"id":333035139,"identity":"4aaffa43-ef53-4d08-a693-79755d2a6fc3","order_by":3,"name":"Xichang Liang","email":"","orcid":"","institution":"Shandong University","correspondingAuthor":false,"prefix":"","firstName":"Xichang","middleName":"","lastName":"Liang","suffix":""},{"id":333035140,"identity":"40f3f3e5-64a7-4c3e-b44c-a866acb323c7","order_by":4,"name":"Yi Wan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAuElEQVRIiWNgGAWjYLCChAoGxgYgLUGCljMka2FsI0WLwfGzxyQezquT3XCA+eBtHga7PMJazuSlSSRuO2y84QBbsjUPQ3IxQS1mB3LMgFoOJG44wGMmzcNwILGBoJbzb4Ba5tQBtfB/I1LLDZAtDcwgW9iI02J/442xRcKxw8YzD7MZW84xSCasRbI/x/Dmj5o62b7jzQ9vvKmwI6wFCFgg0cEMIgyIUA9S+4E4daNgFIyCUTBiAQDF1D3BXzSB3gAAAABJRU5ErkJggg==","orcid":"","institution":"Shandong University","correspondingAuthor":true,"prefix":"","firstName":"Yi","middleName":"","lastName":"Wan","suffix":""}],"badges":[],"createdAt":"2024-07-09 12:36:17","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4712095/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4712095/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s00371-024-03654-2","type":"published","date":"2024-10-09T15:57:38+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":66597405,"identity":"3f1ab9c0-8236-45bc-8b95-99e697c7e86e","added_by":"auto","created_at":"2024-10-14 16:10:31","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":4145200,"visible":true,"origin":"","legend":"","description":"","filename":"YoujieZhoupaper.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4712095/v1_covered_0ee8f467-34c6-4a9e-a4e7-62dd4c7e7e57.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Spatial-frequency Attention-based Optical and Scene Flow with Cross-Modal Knowledge Distillation","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"the-visual-computer","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"tvcj","sideBox":"Learn more about [The Visual Computer](http://link.springer.com/journal/371)","snPcode":"371","submissionUrl":"https://submission.nature.com/new-submission/371/3","title":"The Visual Computer","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Optical and scene flow, multimodal fusion, spatial-frequency domain transform, attention, knowledge distillation","lastPublishedDoi":"10.21203/rs.3.rs-4712095/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4712095/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"This paper studies the problem of multimodal fusion for optical and scene flow from RGB and depth images, or point clouds. Previous methods fuse multimodal information in “early-fusion” or “late-fusion” strategies, in which an attention mechanism is employed to address the problem of optical and scene flow estimation when RGB information is unreliable. Such attentive approaches either suffer from substantial computational and time complexities or lose the inherent characteristics of features due to downsampling. To address this issue, we propose a novel multimodal fusion approach named SFRAFT, which utilizes Fourier transform to build the spatial-frequency domain transformed self-attention and cross-attention. With the novel attentive mechanism, our approach can extract informative features more efficiently and effectively. We further enhance information exchange between the two modalities by incorporating multi-scale knowledge distillation. Experimental results on Flythings3D and KITTI show that our SFRAFT achieves the best performance with low computational and time complexity. We also prove the strong ability of our approach for flow estimation on our real-world dataset. We release the code and datasets at https://doi.org/10.5281/zenodo.12697968","manuscriptTitle":"Spatial-frequency Attention-based Optical and Scene Flow with Cross-Modal Knowledge Distillation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-05 08:17:05","doi":"10.21203/rs.3.rs-4712095/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-08-22T13:38:34+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-08-19T11:58:52+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-07-29T08:03:33+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"22137280719831493901350520331392986723","date":"2024-07-24T00:53:21+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-07-22T05:56:26+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"6874140039126375144539456476913940731","date":"2024-07-20T00:57:12+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"290044532275901556108007017604824989301","date":"2024-07-19T01:00:49+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-07-18T14:30:10+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-07-10T12:18:35+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-07-10T08:31:49+00:00","index":"","fulltext":""},{"type":"submitted","content":"The Visual Computer","date":"2024-07-09T12:34:54+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"the-visual-computer","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"tvcj","sideBox":"Learn more about [The Visual Computer](http://link.springer.com/journal/371)","snPcode":"371","submissionUrl":"https://submission.nature.com/new-submission/371/3","title":"The Visual Computer","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"e7c69bc8-c588-42ba-8645-24c2450ea233","owner":[],"postedDate":"August 5th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2024-10-14T16:05:47+00:00","versionOfRecord":{"articleIdentity":"rs-4712095","link":"https://doi.org/10.1007/s00371-024-03654-2","journal":{"identity":"the-visual-computer","isVorOnly":false,"title":"The Visual Computer"},"publishedOn":"2024-10-09 15:57:38","publishedOnDateReadable":"October 9th, 2024"},"versionCreatedAt":"2024-08-05 08:17:05","video":"","vorDoi":"10.1007/s00371-024-03654-2","vorDoiUrl":"https://doi.org/10.1007/s00371-024-03654-2","workflowStages":[]},"version":"v1","identity":"rs-4712095","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4712095","identity":"rs-4712095","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}