A multimodal deep learning method of weld defect detection based on 3D point cloud

A multimodal deep learning method of weld defect detection based on 3D point cloud | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A multimodal deep learning method of weld defect detection based on 3D point cloud Kaiyuan Lin, Fang Li, Jiacheng Huang, Chen Shen, Yuelong Zhang, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4855666/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Weld quality inspection is essential in modern manufacturing, requiring the automatic identification, localization, and measurement of defects in industrial environments. Although 2D images and 3D point clouds each have their unique advantages, most current inspection methods focus on only one of these data types. This study proposes a novel system integrating 3D point cloud data with 2D images using PointNet + + and YOLOv5. The 3D point cloud data is mapped into corresponding 2D feature maps and trained separately. Training results show that PointNet + + achieved an accuracy of 98.9% and an IoU of 79.3%, while YOLOv5 achieved an precision of 98.9%, a recall of 97.6%, a [email protected] of 98.8%, and a [email protected] :0.95 of 72.2%. By combining the results of both models, the 2D bounding boxes from YOLOv5 are mapped back into 3D space and integrated with PointNet + + results to create 3D bounding boxes. Reassigning the defect point class weights within each 3D bounding box helps resolve issues where PointNet + + might classify points from a single defect into multiple classes. The proposed method in this study demonstrated an improvement on a test set of 100 samples in mIoU from 60.2–63.0% compared to using PointNet + + alone, resulting in effective identification and measurement of spatter, porosity, and burn-through. Multimodal deep learning Welding quality inspection Object detection 3D point cloud Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4855666","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":339440491,"identity":"a8bc55fe-28fd-4643-9940-1037e4c0d381","order_by":0,"name":"Kaiyuan Lin","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABB0lEQVRIiWNgGAWjYBACCSA+wFAgwcDGzNhwIKGCgRkszENQi4EEAx9788EHD84wMPMQo4WBwYCBQY7nWLLhwzaoanxaJNt7DA8XGFjksUnkmEkkzqtjt5dIYHzwto1B3hyHFmmeMwaHZxhIFEO0bDvMzCORwGw4t43BcGcDdi1yEmkJh3kMJBLbIFoOgLSwSfO2MSQYHMChRf4ZspY5dSAt7L/xaZGWYD4A0QL0vkFiAzPYFmZ8WiR7kqFaQIGccAzolzMPmyXnnJMw3IBDi8Txg82feSrqEuc3MzYc/FFTl8zennzww5syG3lctmCAZAYGxgYGWIQRBeyIVzoKRsEoGAUjBQAA4ydUe5WCKGoAAAAASUVORK5CYII=","orcid":"","institution":"Shanghai Jiao Tong University","correspondingAuthor":true,"prefix":"","firstName":"Kaiyuan","middleName":"","lastName":"Lin","suffix":""},{"id":339440494,"identity":"5f15d234-2a44-430b-a156-6b879cc5f44a","order_by":1,"name":"Fang Li","email":"","orcid":"","institution":"Shanghai Jiao Tong University","correspondingAuthor":false,"prefix":"","firstName":"Fang","middleName":"","lastName":"Li","suffix":""},{"id":339440495,"identity":"b5258aae-8699-47c2-8199-7aedf0641c5f","order_by":2,"name":"Jiacheng Huang","email":"","orcid":"","institution":"Shanghai Jiao Tong University","correspondingAuthor":false,"prefix":"","firstName":"Jiacheng","middleName":"","lastName":"Huang","suffix":""},{"id":339440496,"identity":"81524858-33da-47cf-8509-f2b3faef9307","order_by":3,"name":"Chen Shen","email":"","orcid":"","institution":"Shanghai Jiao Tong University","correspondingAuthor":false,"prefix":"","firstName":"Chen","middleName":"","lastName":"Shen","suffix":""},{"id":339440497,"identity":"1af80301-7045-447d-9e1a-7ceeb90a4c2c","order_by":4,"name":"Yuelong Zhang","email":"","orcid":"","institution":"Shanghai Jiao Tong University","correspondingAuthor":false,"prefix":"","firstName":"Yuelong","middleName":"","lastName":"Zhang","suffix":""},{"id":339440498,"identity":"83c4bf2a-344f-4f36-9dde-77daa5a67631","order_by":5,"name":"Xueming Hua","email":"","orcid":"","institution":"Shanghai Jiao Tong University","correspondingAuthor":false,"prefix":"","firstName":"Xueming","middleName":"","lastName":"Hua","suffix":""}],"badges":[],"createdAt":"2024-08-04 06:53:21","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4855666/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4855666/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":64002956,"identity":"5441beee-800b-435f-ab3c-5e12660d0b62","added_by":"auto","created_at":"2024-09-04 20:41:41","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1938255,"visible":true,"origin":"","legend":"","description":"","filename":"MVAmanuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4855666/v1_covered_8ba46a78-67d6-4673-b2c4-0e994f9b453a.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A multimodal deep learning method of weld defect detection based on 3D point cloud","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Multimodal deep learning, Welding quality inspection, Object detection, 3D point cloud","lastPublishedDoi":"10.21203/rs.3.rs-4855666/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4855666/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eWeld quality inspection is essential in modern manufacturing, requiring the automatic identification, localization, and measurement of defects in industrial environments. Although 2D images and 3D point clouds each have their unique advantages, most current inspection methods focus on only one of these data types. This study proposes a novel system integrating 3D point cloud data with 2D images using PointNet\u0026thinsp;+\u0026thinsp;+\u0026thinsp;and YOLOv5. The 3D point cloud data is mapped into corresponding 2D feature maps and trained separately. Training results show that PointNet\u0026thinsp;+\u0026thinsp;+\u0026thinsp;achieved an accuracy of 98.9% and an IoU of 79.3%, while YOLOv5 achieved an precision of 98.9%, a recall of 97.6%, a [email protected] of 98.8%, and a [email protected]:0.95 of 72.2%. By combining the results of both models, the 2D bounding boxes from YOLOv5 are mapped back into 3D space and integrated with PointNet\u0026thinsp;+\u0026thinsp;+\u0026thinsp;results to create 3D bounding boxes. Reassigning the defect point class weights within each 3D bounding box helps resolve issues where PointNet\u0026thinsp;+\u0026thinsp;+\u0026thinsp;might classify points from a single defect into multiple classes. The proposed method in this study demonstrated an improvement on a test set of 100 samples in mIoU from 60.2\u0026ndash;63.0% compared to using PointNet\u0026thinsp;+\u0026thinsp;+\u0026thinsp;alone, resulting in effective identification and measurement of spatter, porosity, and burn-through.\u003c/p\u003e","manuscriptTitle":"A multimodal deep learning method of weld defect detection based on 3D point cloud","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-30 06:44:06","doi":"10.21203/rs.3.rs-4855666/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"2db92baf-e597-40b0-a982-16153200c6a1","owner":[],"postedDate":"August 30th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-09-04T20:33:24+00:00","versionOfRecord":[],"versionCreatedAt":"2024-08-30 06:44:06","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4855666","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4855666","identity":"rs-4855666","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}