Accelerated Accurate In-line Solder Joint Inspection Technique

Accelerated Accurate In-line Solder Joint Inspection Technique | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Accelerated Accurate In-line Solder Joint Inspection Technique Teng Yeow Ong, Koon Tatt Tan, Ping Chow Teoh, Mohamad Hushnie Haron This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4935106/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 20 Mar, 2025 Read the published version in The International Journal of Advanced Manufacturing Technology → Version 1 posted 5 You are reading this latest preprint version Abstract This paper reviews the entire vision inspection cycle, encompassing image acquisition, image enhancement, Region of Interest (ROI) localization and segmentation, features extraction followed by defect detection and classification. The aim of the study is to identify potential image processing time saving. The investigation innovatively suggests that optimizing image enhancement and ROI localization processing time could significantly accelerate the overall inspection cycle time without negatively impacting inspection accuracy. In Automated Optical Inspection (AOI) machine, camera sensor is mounted on precision X-Y gantries. To acquire images for inspection, the gantries will accurately move the camera to the predetermined coordinate position as stipulated in the inspection program. The vision camera will then capture the desired image using specified Field of View (FOV). Only ROI which is the solder joint position will be extracted out from the FOV image for processing. Meanwhile, the designated solder joint positions (i.e. solder pad coordinates) for all electronic components mounted on the PCB are priory known extracted from the PCB fabrication file. These coordinates can be used directly for ROI localization without employing any algorithm, and yet accuracy is not compromised. Meanwhile, through leveraging the state-of-art vision hardware, namely high-resolution camera and adaptive lighting system, quality images can be acquired and used directly without the need for any enhancement. Comparison analysis based on industrial PCB having 1000 electronics components (with 3000 solder joints of size 140x70 pixels per joint), the processing time utilizing NVIDIA GeForce RTX 2060 series Graphic Processing Unit (GPU) and Template Matching Algorithm for ROI localization needs 2 seconds. whereas when using Multiscale Morphology Algorithm for image enhancement, time required is approximately 3 seconds. Benchmarking of a typical production line with bottleneck cycle time of 25 seconds, indicating that the proposed methodology effectively addresses the challenges faced while implementing real-time machine vision inspection system in the industry, aligned with Industrial 4.0 Smart Manufacturing initiatives. Solder Joint Vision Inspection Region of Interest Localization In-line Application Figures Figure 1 Figure 2 Figure 3 Motivation Developing a fast and accurate online machine vision system for inspecting solder joint integrity in continuous assembly lines remains a challenging task despite the advances in machine vision, sensing technology and learning Neural Network. Thus far, previous research works in this field lack of pragmatic validation of inspection speed capable to cope with real-world industrial ecosystem characterized by high-volume and high-speed complex mass production setting. This paper aims to develop a novel inspection system that having the potential to handle pressing high-volume and high-speed production environment. In order to accomplish this goal, an ingenious technique is proposed that streamline and synergizes the complementary image processing steps to significantly cut down the inspection time while maintaining high level of accuracy. Abbreviations The following abbreviations are used in this manuscript: AI Artificial Intelligence AOI Automated Optical Inspection FOV Field of View GPU Graphic Processing Unit PCB Printed Circuit Board ROI Region of Interest Introduction Market saturation coupled with intense competition in emerging markets have driven manufacturers to streamline their production processes and strategies for high production yield in order to have competitive edges for market leadership. Defective products escalate costs and disrupt production [1, 2]. Timely defect detection enhances quality control, stabilizes production and maintain competitive edge over competition [3]. In the meantime, with technology advancement coupled with process breakthrough, electronic components continue to miniaturize along with the increase in component density. These have resulted in soldering defects occur in microscopic and complex form, demanding advanced and effective computer vision capability. These challenges have been actively instigated by many researchers to develop accurate inspection algorithms [4, 5] Using AOI to detect solder joint defect, of which not visually discernible, is the most common technology in the manufacturing revolutionizing industrial automation environment [6, 7] Recent years, extensive research to enhance vision inspection algorithms with deep learning and reinforced learning techniques were reported, which promise significantly improve the detection and classification accuracy [8, 9, 10, 11, 12]. In the meantime, numerous studies were conducted to develop industrial applicable real-time vision inspection system [13, 14, 15, 16]. Nevertheless, the efforts to enhance the accuracy particularly for inferring minuscule, curved and specular reflective solder defects often lead to slower processing speed, which in turn hinders the practical application in the realm of manufacturing industry. This paper aims to comprehensively analyse the processing and analysing time of the entire image processing workflow including imageacquisition, enhancement, ROI localization and segmentation, feature extraction, defect detection and classification. (See Fig. 1) Potential process synergizing that could accelerate the overall processing speed while maintaining the accuracy of defect detection and classification is identified. A novel vision inspection technique is proposed and the performance of the recommended methodology is assessed based on industrial actual PCB to validate for applicability under real-time settings. (a) Image Acquisition Recent advancements in machine vision technology for image acquisition to enhance the accuracy and efficiency of solder joint inspections, particularly within the industrial manufacturing settings, are cantered around developing cutting-edge, high-performance hardware infrastructure solutions: 1. Advanced Optical Lighting System : Innovations in lighting techniques, such as adaptive lighting schemes and stroboscopic light source are being employed to mitigate shadow and reflection effects that can obscure defect during optical inspections [17, 18, 19, 20]. These systems optimize the lighting conditions according to the surface characteristics of the solder joints. 2. High-Resolution Camera Sensor : The introduction of high-resolution compact sensor has led to substantial improvements in the clarity and detail of images captured during solder joint inspection [21, 22, 23, 24]. This advancement enhances the ability to detect even the smallest defects that could critically affect the reliability of the joints. 3. Accelerated Computing Power: State-of-art high performance processor particularly GPU from leading technology providers have dramatically boosted the computation power by leaps and bounces [25, 26]. All the abovetechnological breakthroughshave greatly reduced the research efforts required to ensure obtaining high quality images, a prerequisite for efficient and accurate inspection results [27, 28]. (b) Image Enhancement Many image enhancing techniques can be used to improve the image quality. Yan et al. demonstrated that wavelet thresholding method is a powerful approach for noise reduction [29]. Fukushima et al. proposed an extension of guided image filtering for smoothing and edge enhancement [30]. Guo and Wan suggested improving images by adjusting the background data [31]. Ma et al. introduced a reflection image enhancement algorithm to make the quality of the images captured better [32]. Gao et al. employed an improved Retinex algorithm to augment the colour images [33]. Al-Hashim et al. used Retinex-Based Multiphase Algorithm for Low-Light Image Enhancement [34]. Roman et al. proposed a contrast enhancement algorithm to boost the colour images [35] while Shi et al. utilized an improved adaptive scene transformation algorithm to intensify the images [36]. However, with cutting-edge commercial machine vision hardware (e.g. high resolution Cognex In-sight 3800 Vision System embedded with robust rule-based algorithm and innovative learning technology), high-quality images can be acquired with reduced noises and enhanced features (see Fig 2) Hence, these excellent images captured can do away with the requirement for noise removal, contrast adjustment, filtering, normalization and image enhancement processes. (c) ROI Localization and Segmentation ROI localization needs to accurately detect the position where each and every solder joint is formed in a given FOV image. With FOV of size 40x40 mm, it can have approximately 30 to 40 solder joints depending on the PCB design. In general, ROI localization is carried out through a series of object detection methods. The ROI localization algorithms can basically group into 3 approaches namely template-matching; image histogram and clustering of pixels [37]. Researches done including Rapid Object Detection investigated by Viola and Jones [38], Histogram of Orientated Gradient plus Support Vector Machine proposed by Shumin et al. [39], Deformable Part Model used by Felzenszwalb et al. [40], Selective Search studied by Uijlings et al. [41], Edge Boxes suggested by Zitnick and Dollar [42] as well as Fast Template Matching researched by Dou et al. [43]. Meanwhile, Alarcon-Herrera et al. proposed to utilize View Point Selection [44], Hao et al. put forward an innovative Adaptive Template [45] and Sibiryakov employed a new template matching methodology [46]. Through leveraging the PCB soldering pad information embedded in Gerber file used for PCB fabrication, ROI localization step can be avoided. During image acquisition, the prior known coordinates and solder pad shapes extracted will be used to superimpose on the FOV to locate the exact solder joint position. (see Fig 3) (Note: a specific software is required to interpret, view and export the required data from the Gerber file) (d) Feature Extraction, Defect Detection and Classification Most of the machine vision inspection studies conducted either using learning algorithm or pattern matching approach for image feature extraction, defect detection and classification. Wu et.al. put forward the ResNet on Improved Yolo v3 to extract the required features for defect classification [47] whereas Zhang et al. classified defects based on improved ResNet mode [48]. Zhang et al. proposed the Convolutional Neural Network-based multi-label classification method [49]. Wu et al. suggested an algorithm with logical shape features [50]. Goodfellow et al. applied Deep Learning methodology for classification [51]. Seul et al. conducted research using template matching operation to extract features from ROI [52]. Wu et al. based on Bayes and Support Vector Machine to classify solder defects [53]. Cai et al. proposed to useVisual Background Extraction Algorithm for automatically inspect the solder joint [54]. Abdelhameed et al. utilized the enhanced threshold-based segmentation method with Discrete Cosine to improve the defect detection capabilities [55]. Wu and Xu applied the transformed “eigensolder” feature to classify the solder joint [56]. Different features and learning algorithms are effective for detecting different types of soldering defects [3, 4, 5, 8, 9, 11, 37, 57], This paper does not conduct a comparative analysis of the accuracy and efficiency performance of these features and learning algorithms. The rationale is that each algorithm possesses distinct analytical strengths tailored to specific defect categories, and any modifications intended to enhance the processing speed could lead to trading-off the accuracy. Processing Time Estimation For comparison study purposes, following assumptions are made (1) An industrial PCB having 1000 electronics components with 3000 solder joints (2) Image size of 140x70 pixels per solder joint (3) NVIDIA GeForce RTX 2060 series GPU (4) Multiscale Morphology Algorithm for image enhancement (5) Template Matching Algorithm for ROI localization Under these conditions, the estimated image enhancement time would be approximately 3 seconds per PCB. Meanwhile, the estimated time required for ROI localization is around 2 seconds per PCB . In summary, proposal to leverage the state-of-art vision hardware to eliminate image enhancement coupled with directly performing ROI localization using extracted PCB solder pads coordination could save the inspection cycle up to 5 seconds per PCB. Conclusion With the demanding industrial setting under high-speed and high-volume environment, vision inspection system capable of providing near real-time inspection result with minimal latency is a must to ensure applicability. This study introduces a novel inspection strategy that employs cutting-edge vision hardware like high resolution cameras, powerful graphic processing units and adaptive intelligence optical lighting system to acquire excellent images, eliminating the need for image enhancement. On the other hand, by using directly the solder pads’ coordinates and shapes information extracted from the PCB Gerber file to locate ROI of solder joints, it is able to totally bypass the need of localization algorithm processing. This synergized inspection approach could significantly cut down the inspection time by 5 seconds, which is about 20% of the bottleneck cycle time of a typical production line. The assumption of using cutting-edge vision hardware like high resolution cameras up to 25 Mega Pixels paired with the advanced Cognex Moritex lighting solutions are capable to acquire high quality images without the need for further image augmentation processes like contrast adjustment, filtering, normalization and noise elimination is not addressed in this study. It is suggested that this assumption be validated in the future research works Meanwhile, with the emergence of Generative Artificial Intelligence (AI), it is now possible to integrate the technology into inspection algorithms, enabling the vision system to generate new images that resemble the existing dataset it has been trained on. This AI technology is potential to revolutionize the machine vision inspection landscape, elevating it to the next level of performance excellence. However, it is crucial to take extra precautions to manage biases in the training data, as these could result in unfair or inappropriate outputs. Declarations Competing interests: The authors declare that no relevant financial or non-financial interests to disclose. Funding: The authors declare that no funds, grants, or other support were received from any organization during the preparation of this manuscript. Authors’ contributions: All authors contributed to the study conception and preparation of this manuscript. References Soto JAC, Tavakolizadeh F, Gyulai D (2019) An online machine learning framework for early detection of product failures in an industry 4.0 context Int. J. Comput. Integr. Manuf., vol. 32, nos. 4–5, pp. 452–465 Yang CM, Kim JS, Kang DW, Eom DS (2024) Vision AI System Development for Improved Productivity in Challenging Industrial Environments: A Sustainable and Efficient Approach. Appl Sci 14(7):2750 Chen J, Wen Y, Nanehkaran YA, Zhang D, Zeb A (2023) Multiscale Attention Networks for Pavement Defect Detection. IEEE Trans Instrum Meas 72:1–12 Czimmermann T, Ciuti G, Milazzo M, Chiurazzi M, Roccella S, Oddo CM, Dario P (2020) Visual-based defect detection and classification approaches for industrial applications - A survey,'' Sensors, vol. 20, no. 5, pp. 1–25 Zhonghe Ren F, Fang N, Yan, Wu Y (2022) State of the Art in Defect Detection Based on Machine Vision. Int J Precision Eng Manufacturing-Green Technol no 9:661–691 Yang J, Li S, Wang Z, Dong H, Wang J, Tang S (2020) Using deep learning to detect defects in manufacturing: A comprehensive survey and current challenges. Mater (Basel) no 13:5755 Chen J, Ran X (2019) Deep learning with edge computing: A review, Proc. IEEE, vol. 107, no. 8, pp. 1655–1674 Huang R, Gu J, Sun X, Hou Y, Uddin S (2019) A rapid recognition method for electronic components based on the improved YOLO-V3 network,'' Electronics. 8(8):825 Muddasar, Naeem, Syed Tahir Hussain Rizvi and, Coronato A (2020) A Gentle Introduction to Reinforcement Learning and Its Application in Different Fields. IEEE Access 8:209320–209344 Ling Q, Isa NAM (2023) Printed Circuit Board Defect Detection Methods Based on Image Processing, Machine Learning and Deep Learning: A Survey, in IEEE Access, vol. 11, pp. 15921–15944 LeCun Y, Bengio Y, Hinton (2015) Deep learning. Nature 521(7553):436–444 JiaYou, Lim (2023) JunYi Lim, Vishnu Monn Baskaran and Xin Wang, A deep context learning based PCB defect detection model with anomalous trend alarming system. Results Eng, 17 Lim HS, Lee YL, Yap KM, Lin MH, Lian TK (2022) Colour-assisted PCB Inspection System with Hardware Support for Real-time Environment, Proceedings of the International Conference on Computer, Information Technology and Intelligent Computing (CITIC) Shreya SR, Priya CS, Rajeshware GS (2016) Design of machine vision system for high speed manufacturing environments, IEEE Annual India Conference (INDICON), Bangalore, India, pp. 1–7, 2016 Li T, Wang S, Luo Y, Wan J, Luo Z, Chen M (2024) 3-D Vision and Intelligent Online Inspection in SMT Microelectronic Packaging: A Review. IEEE J Emerg Sel Top Industrial Electron 5(2):779–789 Caliskan A, Gurkan G (2021) Design and Realization of an Automatic Optical Inspection System for PCB Solder Joints, International Conference on Innovations in Intelligent Systems and Applications (INISTA), Kocaeli, Turkey, pp. 1–6 Kim H, Kim N, Nam K, Nam K-H (2018) Maximum Contrast Imaging Using Chromatic Lighting and Fuzzy Logic: Analyses of Old Handwritings, 2018 18th International Conference on Control, Automation and Systems (ICCAS), PyeongChang, Korea (South), pp. 756–761 Li Y, Wang QL, Li YF, Xu D, Tan M (2018) On-line visual measurement and inspection of weld bead using structured light, IEEE instrumentation and measurement technology conference, pp. 2038–2043 Chen JH, Shen K, Wu X, Zhou X, Han FF, Li JW (2015) Design of stroboscopic light source used in measurement for high-speed motion object, fifth international conference on instrumentation and measurement, computer, communication and control (IMCCC), pp. 1135–1138, IEEE Wang JL, Qu XH, Zhao Y (2009) Design of lighting system in multi vision detection. Electro-Optic Technol Application 24(4):1–5 Lyu C, Li P, Wang D, Yang S, Lai Y, Sui C (2021) High-Speed Optical 3D Measurement Sensor for Industrial Application. IEEE Sens J 21(10):11253–11261 Peng Y, Yan Y, Chen G, Feng B (2022) Automatic compact camera module solder joint inspection method based on machine vision. Meas Sci Technol, 33, 10 Nehir M, Frank C, Asmann S, Achterberg EP (2019) Improving optical measurements: non-linearity compensation of compact charge-coupled device (CCD) spectrometers. Sensors 19(12):2833 Mehta S, Patel A, Mehta J (2015) CCD or CMOS Image sensor for photography, International conference on communications and signal processing (ICCSP), pp. 0291–0294, IEEE Li B, Yang J, Yang Y, Wu N (2014) A high speed vision processor for chip package visual inspection, 2014 12th IEEE International Conference on Solid-State and Integrated Circuit Technology (ICSICT), Guilin, China, pp. 1–3 Gonzalez RC, Woods RE (2007) Digital Image Processing, 3rd edn. Prentice-Hall Inc Liu HJ, Wang YN, Duan F (2003) Image capture Mach Vis Comput Inform Technol 1:18–21 Huang L, Pan S (2022) Analysis and Research of A Defect Detection System Based on Machine Vision for Solder Joints, IEEE 4th Eurasia Conference on IOT, Communication and Engineering (ECICE), Yunlin, Taiwan, pp. 184–187 Yan Z, Xu W, Yang C (2018) A power thresholding function-based wavelet image denoising method. J Imaging Sci Technol 62(1):10506–10501 Fukushima N, Sugimoto K, Kamata S (2018) Guided image filtering with arbitrary window function, International conference on acoustics, speech and signal processing (ICASSP), pp. 1523–1527, IEEE Guo Y, Wan Y (2018) Image Enhancement Algorithm Based on Background Enhancement Coefficient, 10th International Conference on Communications, Circuits and Systems (ICCCAS), Chengdu, China, pp. 413–417 Ma C, Xu C, Zhou P, Zhang Y (2023) Low-Light Aerial Image Enhancement Algorithm Based on Retinex Theory, International Conference on Image Processing, Computer Vision and Machine Learning (ICICML), Chengdu, China, pp. 138–142 Gao Y, Su C, Xu Z (2022) Colour image enhancement algorithm based on improved Retinex algorithm, Asia Conference on Algorithms, Computing and Machine Learning (CACML), Hangzhou, China, pp. 234–238 Al-Hashim M, Abid, Al-Ameen Z (2020) Retinex-Based Multiphase Algorithm for Low-Light Image Enhancement. Traitement du Signal 37.5 Román J, César, Mello (2017) Horacio Legal Ayala, and José Luis Vázquez Noguera. Image color contrast enhancement using multiscale morphology. J Comput Interdisciplinary Sci 8:3 Shi Z, Wu H, Mao W, Wang J, Zhang C (2020) Implementation of An Automatic Image Enhancement Algorithm for Contrast Stretching on FPGA, IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), Chongqing, China, pp. 1890–1896 Dai W, Mujeeb A, Erdt M, Sourin A (2020) Soldering defect detection in automatic optical inspection. Adv Eng Inform, 43 Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features, Proceedings of computer society conference on computer vision and pattern recognition, CVPR, vol. 1, IEEE Shumin D, Zhoufeng L, Chunlei L (2011) AdaBoost learning for fabric defect detection based on HOG and SVM, International conference on multimedia technology, pp. 2903–2906, IEEE Felzenszwalb P, McAllester D, Ramanan D (2008) A discriminatively trained, multiscale, deformable part model, Conference on computer vision and pattern recognition, pp. 1–8, IEEE Uijlings JR, Van De Sande KE, Gevers T, Smeulders AW (2013) Selective search for object recognition. Int J Comput Vision 104(2):154–171 Zitnick CL, Dollar P (2014) Edge boxes: Locating object proposals from edges, European conference on computer vision, pp. 391–405, Springer Dou Y, Huang Y, Li Q, Luo S (2014) A fast template matching-based algorithm for railway bolts detection. Int J Mach Learn Cybernet 5(6):835–844 Alarcon-Herrera JL, Chen X, Zhang X (2014) Viewpoint selection for vision systems in industrial inspection, 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, pp. 4934–4939 Hao X, Li W, Sun Z, Zhu S, Yan S, Zhao Z (2018) Detection of ball grid array solder joints based on adaptive template matching. Int J Heat Technol 36:189–194 Sibiryakov A (2011) Fast and high-performance template matching method. CVPR IEEE, Colorado Springs, USA, pp 1417–1424 Wu W, Li Q (2020) Machine Vision Inspection the improved ResNet on Improved Yolo v3. IEEE Access 8:166184–166196 Zhang Q, Zhang K, Pan K, Huang W (2024) Image defect classification of surface mount technology welding based on the improved ResNet model. J Eng Res 12(2):154–162 Zhang L, Jin Y, Yang X, Li X, Duan X, Sun Y, Liu H (2018) Convolutional neural network-based multi‐label classification of PCB defects. The Journal of Engineering no. 16 (2018): 1612–1616 Wu F, Zhang X, Kuan Y, He Z (2008) An AOI algorithm for PCB based on feature extraction, 7th World Congress on Intelligent Control and Automation, IEEE, pp. 240–247 Goodfellow I, Bengio Y, Courville A, Deep, Learning (2016) Cambridge, MA, USA: MIT Press Seul M, O’Gorman L, Sammon MJ (2000) ‘‘Template matching, Practical Algorithms for Image Analysis: Description, Examples Code, New York, USA, Cambridge University Press Wu H, Zhang X, Xie H, Kuang Y, Gaofei Ouyang (2013) Classification of solder joint using feature selection based on Bayes and support vector machine. IEEE Trans Compon Packag Manuf Technol 3(3):516–522 Cai N, Lin J, Ye Q, Wang H, Weng S, Ling BW-K (2016) A New IC Solder Joint Inspection Method for an Automatic Optical Inspection System Based on an Improved Visual Background Extraction Algorithm, IEEE Transactions on Components. 6(1):161–172Packaging and Manufacturing Technology Abdelhameed MM, Awad MA, Abd El-Aziz HM, A robust methodology for solder joints extraction, 8th International Conference on Computer Engineering, Systems (2013) (ICCES), Cairo, Egypt, pp. 268–273 Wu H, Xu X (2018) Solder joint inspection using eigensolder features. Soldering Surf Mt Technol 30(4):227–232 Ulger F, Yuksel SE, Yilmaz A, Gokcen D (2023) Solder Joint Inspection on Printed Circuit Boards: A Survey and a Dataset. IEEE Trans Instrum Meas 72:1–21 Cite Share Download PDF Status: Published Journal Publication published 20 Mar, 2025 Read the published version in The International Journal of Advanced Manufacturing Technology → Version 1 posted Reviewers agreed at journal 15 Sep, 2024 Reviewers invited by journal 04 Sep, 2024 Editor assigned by journal 02 Sep, 2024 First submitted to journal 01 Sep, 2024 Editorial decision: Major Revisions Needed 23 Aug, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4935106","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":349572698,"identity":"2c5e495c-fc9b-451d-aeaa-718313d7345d","order_by":0,"name":"Teng Yeow Ong","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAtElEQVRIiWNgGAWjYNCCCgglQZRiHjB5hmQtjG2kaLGXSD728Ou8w3IGB5gP3uZh2JbYQNAWibR0Y9lth40NDrAlW/Mw3CZGS46ZtOS224kbDvCYSZOgZQ5IC/834rVIfmwA28JGpJYzz9KkGY79N5Y8zGZsOcfgtjFBLeztycckf9SkyfEdb354403FbVmCWhgEEhiYwZHDDCIMGBwJa+E/wMD4A4lvT1DHKBgFo2AUjDgAAKBEOyaTm5mHAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0009-0001-1960-1792","institution":"Wawasan Open University","correspondingAuthor":true,"prefix":"","firstName":"Teng","middleName":"Yeow","lastName":"Ong","suffix":""},{"id":349572699,"identity":"755e2fca-3674-4296-9419-fd1b2797b79d","order_by":1,"name":"Koon Tatt Tan","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Koon","middleName":"Tatt","lastName":"Tan","suffix":""},{"id":349572700,"identity":"7e87f54c-ea04-485b-b567-e229ec91c2f0","order_by":2,"name":"Ping Chow Teoh","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Ping","middleName":"Chow","lastName":"Teoh","suffix":""},{"id":349572701,"identity":"c42c7415-213f-4b93-8487-93e98449eb68","order_by":3,"name":"Mohamad Hushnie Haron","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Mohamad","middleName":"Hushnie","lastName":"Haron","suffix":""}],"badges":[],"createdAt":"2024-08-19 01:13:56","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4935106/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4935106/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s00170-025-15383-4","type":"published","date":"2025-03-20T15:57:33+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":67119830,"identity":"3262ebee-d304-4f66-b748-751c8a204d81","added_by":"auto","created_at":"2024-10-21 11:06:42","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":215526,"visible":true,"origin":"","legend":"\u003cp\u003eEntire image processing operation\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-4935106/v1/3939eee469adbc625069d3bb.png"},{"id":67121665,"identity":"34149ae0-8ab3-44b8-8652-b4bea5d0ed83","added_by":"auto","created_at":"2024-10-21 11:14:42","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":187708,"visible":true,"origin":"","legend":"\u003cp\u003eIn-sight 3800 Vision System (courtesy of Cognex)\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-4935106/v1/812f3fd3db54307249c1f1df.png"},{"id":67119829,"identity":"3a47373f-aab4-40c7-9222-08c87c748f6d","added_by":"auto","created_at":"2024-10-21 11:06:41","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":246083,"visible":true,"origin":"","legend":"\u003cp\u003eSample of PCB Gerber file and extracted coordinate information\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-4935106/v1/1f3af173f8f1e9a90d3a31f9.png"},{"id":79120722,"identity":"486405fc-0668-49a2-b6af-7a10fd45b5ce","added_by":"auto","created_at":"2025-03-24 16:11:14","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1152728,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4935106/v1/91e460c6-f432-4904-a94a-eb41a3f93497.pdf"}],"financialInterests":"","formattedTitle":"Accelerated Accurate In-line Solder Joint Inspection Technique","fulltext":[{"header":"Motivation","content":"\u003cp\u003eDeveloping a fast and accurate online machine vision system for inspecting solder joint integrity in continuous assembly lines remains a challenging task despite the advances in machine vision, sensing technology and learning Neural Network.\u003c/p\u003e \u003cp\u003eThus far, previous research works in this field lack of pragmatic validation of inspection speed capable to cope with real-world industrial ecosystem characterized by high-volume and high-speed complex mass production setting.\u003c/p\u003e \u003cp\u003eThis paper aims to develop a novel inspection system that having the potential to handle pressing high-volume and high-speed production environment. In order to accomplish this goal, an ingenious technique is proposed that streamline and synergizes the complementary image processing steps to significantly cut down the inspection time while maintaining high level of accuracy.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003eThe following abbreviations are used in this manuscript:\u003c/p\u003e\u003cp\u003eAI Artificial Intelligence\u003c/p\u003e\u003cp\u003eAOI Automated Optical Inspection\u003c/p\u003e\u003cp\u003eFOV Field of View\u003c/p\u003e\u003cp\u003eGPU Graphic Processing Unit\u003c/p\u003e\u003cp\u003ePCB Printed Circuit Board\u003c/p\u003e\u003cp\u003eROI Region of Interest\u003c/p\u003e"},{"header":"Introduction","content":"\u003cp\u003eMarket saturation coupled with intense competition in emerging markets have driven manufacturers to streamline their production processes and strategies for high production yield in order to have competitive edges for market leadership.\u0026nbsp;Defective products escalate costs and disrupt production [1, 2]. Timely defect detection enhances quality control, stabilizes production and maintain competitive edge over competition [3].\u003c/p\u003e\n\u003cp\u003eIn the meantime, with technology advancement coupled with process breakthrough, electronic components continue to miniaturize along with the increase in component density. These have resulted in soldering defects occur in microscopic and complex form, demanding advanced and effective computer vision capability. These challenges have been actively instigated by many researchers to develop accurate inspection algorithms [4, 5]\u003c/p\u003e\n\u003cp\u003eUsing AOI to detect\u0026nbsp;solder joint defect, of which not visually discernible, is the most common technology in the manufacturing revolutionizing industrial automation environment [6, 7]\u003c/p\u003e\n\u003cp\u003eRecent years, extensive research to enhance vision inspection algorithms with deep learning and reinforced learning techniques were reported, which promise significantly improve the detection and classification accuracy [8, 9, 10, 11, 12].\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIn the meantime, numerous studies were conducted to develop industrial applicable real-time vision inspection system [13, 14, 15, 16].\u003c/p\u003e\n\u003cp\u003eNevertheless, the efforts to enhance the accuracy particularly for inferring\u0026nbsp;minuscule, curved and specular reflective solder defects often lead to slower processing speed, which in turn hinders the practical application in the realm of\u0026nbsp;manufacturing industry.\u003c/p\u003e\n\u003cp\u003eThis paper aims to comprehensively analyse the processing and analysing time of the entire image processing workflow including imageacquisition, enhancement, ROI localization and segmentation, feature extraction, defect detection and classification. (See Fig. 1)\u003c/p\u003e\n\u003cp\u003ePotential process synergizing that could accelerate the overall processing speed while maintaining the accuracy of defect detection and classification is identified.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eA novel vision inspection technique is proposed and the performance of the recommended methodology is assessed based on industrial actual PCB to validate for applicability under real-time settings.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e(a) \u0026nbsp;Image Acquisition\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eRecent advancements in machine vision technology for image acquisition to enhance the accuracy and efficiency of solder joint inspections, particularly within the industrial manufacturing settings, are cantered around developing cutting-edge, high-performance hardware infrastructure solutions:\u003c/p\u003e\n\u003cp\u003e1. \u0026nbsp; \u003cstrong\u003eAdvanced Optical Lighting System\u003c/strong\u003e: Innovations in lighting techniques, such as adaptive lighting schemes and\u0026nbsp;stroboscopic light source\u0026nbsp;are being employed to mitigate shadow and reflection effects that can obscure defect during optical inspections [17, 18, 19, 20]. These systems optimize the lighting conditions according to the surface characteristics of the solder joints.\u003c/p\u003e\n\u003cp\u003e2. \u0026nbsp; \u003cstrong\u003eHigh-Resolution Camera Sensor\u003c/strong\u003e: The introduction of high-resolution compact sensor has led to substantial improvements in the clarity and detail of images captured during solder joint inspection [21, 22, 23, 24]. This advancement enhances the ability to detect even the smallest defects that could critically affect the reliability of the joints.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3. \u0026nbsp; \u003cstrong\u003eAccelerated Computing\u0026nbsp;\u003c/strong\u003e\u003c/strong\u003ePower: State-of-art high performance processor particularly\u003cstrong\u003e\u0026nbsp;\u003cstrong\u003eGPU\u003c/strong\u003e\u0026nbsp;\u003c/strong\u003efrom leading technology providers have dramatically boosted the computation power by leaps and bounces [25, 26].\u003c/p\u003e\n\u003cp\u003eAll the abovetechnological breakthroughshave greatly reduced the research efforts\u003c/p\u003e\n\u003cp\u003erequired to\u0026nbsp;ensure obtaining high quality images, a prerequisite for efficient and accurate\u0026nbsp;inspection results [27, 28].\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e(b) \u0026nbsp;Image Enhancement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eMany image enhancing techniques can be used to improve the image quality.\u003c/p\u003e\n\u003cp\u003eYan et al. demonstrated that\u0026nbsp;wavelet thresholding method is a powerful approach for noise reduction [29]. Fukushima et al. proposed an extension of guided image filtering for smoothing and edge enhancement [30].\u0026nbsp;Guo and Wan suggested improving images by\u0026nbsp;adjusting the background data\u0026nbsp;[31].\u0026nbsp;Ma et al. introduced a reflection image enhancement algorithm to make\u0026nbsp;the quality of the images captured better [32].\u0026nbsp;Gao et al. employed an improved Retinex algorithm to augment the colour images [33].\u0026nbsp;Al-Hashim et al. used Retinex-Based Multiphase Algorithm for Low-Light Image Enhancement\u0026nbsp;[34].\u0026nbsp;Roman et al. proposed a contrast enhancement algorithm to boost the colour images [35] while\u0026nbsp;Shi et al. utilized an improved adaptive scene transformation algorithm to intensify the images\u0026nbsp;[36].\u003c/p\u003e\n\u003cp\u003eHowever, with cutting-edge commercial machine vision hardware (e.g. high resolution Cognex In-sight 3800 Vision System embedded with robust rule-based algorithm and innovative learning technology),\u0026nbsp;high-quality images can be acquired with reduced noises and enhanced features (see Fig 2)\u003c/p\u003e\n\u003cp\u003eHence, these excellent images captured can do away with the requirement for noise removal, contrast adjustment, filtering, normalization and image enhancement processes.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e(c) \u0026nbsp;ROI Localization and Segmentation\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eROI localization needs to accurately detect the position where each and every solder joint is formed in a given FOV image. With FOV of size 40x40 mm, it can have approximately 30 to 40 solder joints depending on the PCB design.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIn general, ROI localization is carried out through a series of object detection methods. The\u0026nbsp;ROI localization\u0026nbsp;algorithms\u0026nbsp;can basically group into 3 approaches namely template-matching; image histogram and clustering of pixels [37].\u003c/p\u003e\n\u003cp\u003eResearches done\u0026nbsp;including Rapid Object Detection investigated by Viola and Jones [38],\u0026nbsp;Histogram of Orientated Gradient\u0026nbsp;plus Support Vector Machine proposed by Shumin et al. [39], Deformable Part Model used by Felzenszwalb et al. [40], Selective Search studied by Uijlings et al. [41], Edge Boxes suggested by Zitnick and Dollar [42] as well as Fast Template Matching researched by Dou et al. [43].\u003c/p\u003e\n\u003cp\u003eMeanwhile,\u0026nbsp;Alarcon-Herrera\u0026nbsp;et al. proposed to utilize View Point Selection [44],\u0026nbsp;Hao et al. put forward an innovative Adaptive Template [45] and Sibiryakov employed a new template matching methodology [46].\u003c/p\u003e\n\u003cp\u003eThrough\u0026nbsp;leveraging\u0026nbsp;the PCB soldering pad information embedded in Gerber file used for PCB fabrication,\u0026nbsp;ROI localization step can be avoided.\u0026nbsp;During image acquisition, the prior known coordinates and solder pad shapes extracted will be used to superimpose on the FOV to locate the exact solder joint position. (see Fig 3)\u003c/p\u003e\n\u003cp\u003e(Note: a specific software is required to interpret, view and export the required data from the Gerber file)\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e(d) \u0026nbsp;Feature Extraction, Defect Detection and Classification\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eMost of the machine vision inspection studies conducted either using learning algorithm or pattern matching approach for image feature extraction, defect detection and classification.\u003c/p\u003e\n\u003cp\u003eWu et.al. put forward the ResNet on Improved Yolo v3 to extract the required features for defect classification [47] whereas Zhang et al. classified defects based on improved ResNet mode [48]. Zhang et al. proposed the Convolutional Neural Network-based multi-label classification method [49]. Wu et al. suggested an algorithm with logical shape features [50]. Goodfellow et al. applied Deep Learning methodology for classification [51]. Seul et al. conducted research using template matching operation to extract features from ROI [52]. Wu et al. based on Bayes and Support Vector Machine to classify solder defects [53]. Cai et al. proposed to useVisual Background Extraction Algorithm for automatically inspect the solder joint [54]. Abdelhameed et al. utilized the enhanced threshold-based segmentation method with Discrete Cosine to improve the defect detection capabilities [55]. Wu and Xu applied the transformed \u0026ldquo;eigensolder\u0026rdquo; feature to classify the solder joint [56].\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eDifferent features and learning algorithms are effective for detecting different types of soldering defects [3, 4, 5, 8, 9, 11, 37, 57], This paper does not conduct a comparative analysis of the accuracy and efficiency performance of these features and learning algorithms. The rationale is that each algorithm possesses distinct analytical strengths tailored to specific defect categories, and any modifications intended to enhance the processing speed could lead to trading-off the accuracy.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eProcessing Time Estimation\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFor comparison study purposes,\u0026nbsp;\u003c/strong\u003efollowing assumptions are made\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e(1) \u0026nbsp; An industrial PCB having 1000\u0026nbsp;\u003c/strong\u003eelectronics\u003cstrong\u003e\u0026nbsp;components with 3000 solder joints\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e(2) \u0026nbsp; \u003cstrong\u003eImage size of\u003c/strong\u003e 140x70 pixels per solder joint\u003c/p\u003e\n\u003cp\u003e(3) \u0026nbsp; NVIDIA GeForce RTX 2060 series GPU\u003c/p\u003e\n\u003cp\u003e(4) \u0026nbsp; Multiscale Morphology Algorithm for image enhancement\u003c/p\u003e\n\u003cp\u003e(5) \u0026nbsp; Template Matching Algorithm for ROI localization\u003c/p\u003e\n\u003cp\u003eUnder these conditions, the estimated image enhancement time would be approximately \u003cstrong\u003e3 seconds\u003c/strong\u003eper PCB.\u003c/p\u003e\n\u003cp\u003eMeanwhile, the estimated time required for ROI localization is around \u003cstrong\u003e2 seconds per PCB\u003c/strong\u003e\u003cstrong\u003e.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIn summary, proposal to leverage the state-of-art vision hardware to eliminate image enhancement coupled with directly performing ROI localization using extracted PCB solder pads coordination could save the inspection cycle up to 5 seconds per PCB.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eWith the demanding industrial setting under high-speed and high-volume environment, vision inspection system capable of providing near real-time inspection result with minimal latency is a must to ensure applicability.\u003c/p\u003e\n\u003cp\u003eThis study introduces a novel inspection strategy that employs cutting-edge vision hardware like high resolution cameras, powerful graphic processing units and adaptive intelligence optical lighting system to acquire excellent images, eliminating the need for image enhancement.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eOn the other hand, by using directly the solder pads\u0026rsquo; coordinates and shapes information extracted from the PCB Gerber file to locate ROI of solder joints, it is able to totally bypass the need of localization algorithm processing.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThis synergized inspection approach could significantly cut down the inspection time by 5 seconds, which is about 20% of the bottleneck cycle time of a typical production line.\u003c/p\u003e\n\u003cp\u003eThe assumption of using cutting-edge vision hardware like high resolution cameras up to 25 Mega Pixels paired with the advanced Cognex Moritex lighting solutions are capable to acquire high quality images without the need for further image augmentation processes like contrast adjustment, filtering, normalization and noise elimination is not addressed in this study. It is suggested that this assumption be validated in the future research works\u003c/p\u003e\n\u003cp\u003eMeanwhile, with the emergence of Generative Artificial Intelligence (AI), it is now possible to integrate the technology into inspection algorithms, enabling the vision system to generate new images that resemble the existing dataset it has been trained on. This AI technology is potential to revolutionize the machine vision inspection landscape, elevating it to the next level of performance excellence. However, it is crucial to take extra precautions to manage biases in the training data, as these could result in unfair or inappropriate outputs.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eCompeting interests:\u003c/h2\u003e \u003cp\u003eThe authors declare that no relevant financial or non-financial interests to disclose.\u003c/p\u003e\u003ch2\u003eFunding:\u003c/h2\u003e \u003cp\u003eThe authors declare that no funds, grants, or other support were received from any organization during the preparation of this manuscript.\u003c/p\u003e\u003ch2\u003eAuthors\u0026rsquo; contributions:\u003c/h2\u003e \u003cp\u003eAll authors contributed to the study conception and preparation of this manuscript.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eSoto JAC, Tavakolizadeh F, Gyulai D (2019) An online machine learning framework for early detection of product failures in an industry 4.0 context Int. J. Comput. Integr. Manuf., vol. 32, nos. 4\u0026ndash;5, pp. 452\u0026ndash;465\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang CM, Kim JS, Kang DW, Eom DS (2024) Vision AI System Development for Improved Productivity in Challenging Industrial Environments: A Sustainable and Efficient Approach. Appl Sci 14(7):2750\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen J, Wen Y, Nanehkaran YA, Zhang D, Zeb A (2023) Multiscale Attention Networks for Pavement Defect Detection. IEEE Trans Instrum Meas 72:1\u0026ndash;12\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCzimmermann T, Ciuti G, Milazzo M, Chiurazzi M, Roccella S, Oddo CM, Dario P (2020) Visual-based defect detection and classification approaches for industrial applications - A survey,'' Sensors, vol. 20, no. 5, pp. 1\u0026ndash;25\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhonghe Ren F, Fang N, Yan, Wu Y (2022) State of the Art in Defect Detection Based on Machine Vision. Int J Precision Eng Manufacturing-Green Technol no 9:661\u0026ndash;691\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang J, Li S, Wang Z, Dong H, Wang J, Tang S (2020) Using deep learning to detect defects in manufacturing: A comprehensive survey and current challenges. Mater (Basel) no 13:5755\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen J, Ran X (2019) Deep learning with edge computing: A review, Proc. IEEE, vol. 107, no. 8, pp. 1655\u0026ndash;1674\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang R, Gu J, Sun X, Hou Y, Uddin S (2019) A rapid recognition method for electronic components based on the improved YOLO-V3 network,'' Electronics. 8(8):825\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMuddasar, Naeem, Syed Tahir Hussain Rizvi and, Coronato A (2020) A Gentle Introduction to Reinforcement Learning and Its Application in Different Fields. IEEE Access 8:209320\u0026ndash;209344\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLing Q, Isa NAM (2023) Printed Circuit Board Defect Detection Methods Based on Image Processing, Machine Learning and Deep Learning: A Survey, in IEEE Access, vol. 11, pp. 15921\u0026ndash;15944\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLeCun Y, Bengio Y, Hinton (2015) Deep learning. Nature 521(7553):436\u0026ndash;444\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJiaYou, Lim (2023) JunYi Lim, Vishnu Monn Baskaran and Xin Wang, A deep context learning based PCB defect detection model with anomalous trend alarming system. Results Eng, 17\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLim HS, Lee YL, Yap KM, Lin MH, Lian TK (2022) Colour-assisted PCB Inspection System with Hardware Support for Real-time Environment, Proceedings of the International Conference on Computer, Information Technology and Intelligent Computing (CITIC)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShreya SR, Priya CS, Rajeshware GS (2016) Design of machine vision system for high speed manufacturing environments, IEEE Annual India Conference (INDICON), Bangalore, India, pp. 1\u0026ndash;7, 2016\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi T, Wang S, Luo Y, Wan J, Luo Z, Chen M (2024) 3-D Vision and Intelligent Online Inspection in SMT Microelectronic Packaging: A Review. IEEE J Emerg Sel Top Industrial Electron 5(2):779\u0026ndash;789\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCaliskan A, Gurkan G (2021) Design and Realization of an Automatic Optical Inspection System for PCB Solder Joints, International Conference on Innovations in Intelligent Systems and Applications (INISTA), Kocaeli, Turkey, pp. 1\u0026ndash;6\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKim H, Kim N, Nam K, Nam K-H (2018) Maximum Contrast Imaging Using Chromatic Lighting and Fuzzy Logic: Analyses of Old Handwritings, 2018 18th International Conference on Control, Automation and Systems (ICCAS), PyeongChang, Korea (South), pp. 756\u0026ndash;761\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi Y, Wang QL, Li YF, Xu D, Tan M (2018) On-line visual measurement and inspection of weld bead using structured light, IEEE instrumentation and measurement technology conference, pp. 2038\u0026ndash;2043\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen JH, Shen K, Wu X, Zhou X, Han FF, Li JW (2015) Design of stroboscopic light source used in measurement for high-speed motion object, fifth international conference on instrumentation and measurement, computer, communication and control (IMCCC), pp. 1135\u0026ndash;1138, IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang JL, Qu XH, Zhao Y (2009) Design of lighting system in multi vision detection. Electro-Optic Technol Application 24(4):1\u0026ndash;5\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLyu C, Li P, Wang D, Yang S, Lai Y, Sui C (2021) High-Speed Optical 3D Measurement Sensor for Industrial Application. IEEE Sens J 21(10):11253\u0026ndash;11261\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePeng Y, Yan Y, Chen G, Feng B (2022) Automatic compact camera module solder joint inspection method based on machine vision. Meas Sci Technol, 33, 10\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNehir M, Frank C, Asmann S, Achterberg EP (2019) Improving optical measurements: non-linearity compensation of compact charge-coupled device (CCD) spectrometers. Sensors 19(12):2833\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMehta S, Patel A, Mehta J (2015) CCD or CMOS Image sensor for photography, International conference on communications and signal processing (ICCSP), pp. 0291\u0026ndash;0294, IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi B, Yang J, Yang Y, Wu N (2014) A high speed vision processor for chip package visual inspection, 2014 12th IEEE International Conference on Solid-State and Integrated Circuit Technology (ICSICT), Guilin, China, pp. 1\u0026ndash;3\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGonzalez RC, Woods RE (2007) Digital Image Processing, 3rd edn. Prentice-Hall Inc\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu HJ, Wang YN, Duan F (2003) Image capture Mach Vis Comput Inform Technol 1:18\u0026ndash;21\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang L, Pan S (2022) Analysis and Research of A Defect Detection System Based on Machine Vision for Solder Joints, IEEE 4th Eurasia Conference on IOT, Communication and Engineering (ECICE), Yunlin, Taiwan, pp. 184\u0026ndash;187\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYan Z, Xu W, Yang C (2018) A power thresholding function-based wavelet image denoising method. J Imaging Sci Technol 62(1):10506\u0026ndash;10501\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFukushima N, Sugimoto K, Kamata S (2018) Guided image filtering with arbitrary window function, International conference on acoustics, speech and signal processing (ICASSP), pp. 1523\u0026ndash;1527, IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGuo Y, Wan Y (2018) Image Enhancement Algorithm Based on Background Enhancement Coefficient, 10th International Conference on Communications, Circuits and Systems (ICCCAS), Chengdu, China, pp. 413\u0026ndash;417\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMa C, Xu C, Zhou P, Zhang Y (2023) Low-Light Aerial Image Enhancement Algorithm Based on Retinex Theory, International Conference on Image Processing, Computer Vision and Machine Learning (ICICML), Chengdu, China, pp. 138\u0026ndash;142\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGao Y, Su C, Xu Z (2022) Colour image enhancement algorithm based on improved Retinex algorithm, Asia Conference on Algorithms, Computing and Machine Learning (CACML), Hangzhou, China, pp. 234\u0026ndash;238\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAl-Hashim M, Abid, Al-Ameen Z (2020) Retinex-Based Multiphase Algorithm for Low-Light Image Enhancement. Traitement du Signal 37.5\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRom\u0026aacute;n J, C\u0026eacute;sar, Mello (2017) Horacio Legal Ayala, and Jos\u0026eacute; Luis V\u0026aacute;zquez Noguera. Image color contrast enhancement using multiscale morphology. J Comput Interdisciplinary Sci 8:3\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShi Z, Wu H, Mao W, Wang J, Zhang C (2020) Implementation of An Automatic Image Enhancement Algorithm for Contrast Stretching on FPGA, IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), Chongqing, China, pp. 1890\u0026ndash;1896\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDai W, Mujeeb A, Erdt M, Sourin A (2020) Soldering defect detection in automatic optical inspection. Adv Eng Inform, 43\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eViola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features, Proceedings of computer society conference on computer vision and pattern recognition, CVPR, vol. 1, IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShumin D, Zhoufeng L, Chunlei L (2011) AdaBoost learning for fabric defect detection based on HOG and SVM, International conference on multimedia technology, pp. 2903\u0026ndash;2906, IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFelzenszwalb P, McAllester D, Ramanan D (2008) A discriminatively trained, multiscale, deformable part model, Conference on computer vision and pattern recognition, pp. 1\u0026ndash;8, IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eUijlings JR, Van De Sande KE, Gevers T, Smeulders AW (2013) Selective search for object recognition. Int J Comput Vision 104(2):154\u0026ndash;171\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZitnick CL, Dollar P (2014) Edge boxes: Locating object proposals from edges, European conference on computer vision, pp. 391\u0026ndash;405, Springer\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDou Y, Huang Y, Li Q, Luo S (2014) A fast template matching-based algorithm for railway bolts detection. Int J Mach Learn Cybernet 5(6):835\u0026ndash;844\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlarcon-Herrera JL, Chen X, Zhang X (2014) Viewpoint selection for vision systems in industrial inspection, 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, pp. 4934\u0026ndash;4939\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHao X, Li W, Sun Z, Zhu S, Yan S, Zhao Z (2018) Detection of ball grid array solder joints based on adaptive template matching. Int J Heat Technol 36:189\u0026ndash;194\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSibiryakov A (2011) Fast and high-performance template matching method. CVPR IEEE, Colorado Springs, USA, pp 1417\u0026ndash;1424\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWu W, Li Q (2020) Machine Vision Inspection the improved ResNet on Improved Yolo v3. IEEE Access 8:166184\u0026ndash;166196\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang Q, Zhang K, Pan K, Huang W (2024) Image defect classification of surface mount technology welding based on the improved ResNet model. J Eng Res 12(2):154\u0026ndash;162\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang L, Jin Y, Yang X, Li X, Duan X, Sun Y, Liu H (2018) Convolutional neural network-based multi‐label classification of PCB defects. The Journal of Engineering no. 16 (2018): 1612\u0026ndash;1616\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWu F, Zhang X, Kuan Y, He Z (2008) An AOI algorithm for PCB based on feature extraction, 7th World Congress on Intelligent Control and Automation, IEEE, pp. 240\u0026ndash;247\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGoodfellow I, Bengio Y, Courville A, Deep, Learning (2016) Cambridge, MA, USA: MIT Press\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSeul M, O\u0026rsquo;Gorman L, Sammon MJ (2000) \u0026lsquo;\u0026lsquo;Template matching, Practical Algorithms for Image Analysis: Description, Examples Code, New York, USA, Cambridge University Press\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWu H, Zhang X, Xie H, Kuang Y, Gaofei Ouyang (2013) Classification of solder joint using feature selection based on Bayes and support vector machine. IEEE Trans Compon Packag Manuf Technol 3(3):516\u0026ndash;522\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCai N, Lin J, Ye Q, Wang H, Weng S, Ling BW-K (2016) A New IC Solder Joint Inspection Method for an Automatic Optical Inspection System Based on an Improved Visual Background Extraction Algorithm, IEEE Transactions on Components. 6(1):161\u0026ndash;172Packaging and Manufacturing Technology\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbdelhameed MM, Awad MA, Abd El-Aziz HM, A robust methodology for solder joints extraction, 8th International Conference on Computer Engineering, Systems (2013) (ICCES), Cairo, Egypt, pp. 268\u0026ndash;273\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWu H, Xu X (2018) Solder joint inspection using eigensolder features. Soldering Surf Mt Technol 30(4):227\u0026ndash;232\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eUlger F, Yuksel SE, Yilmaz A, Gokcen D (2023) Solder Joint Inspection on Printed Circuit Boards: A Survey and a Dataset. IEEE Trans Instrum Meas 72:1\u0026ndash;21\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"the-international-journal-of-advanced-manufacturing-technology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jamt","sideBox":"Learn more about [The International Journal of Advanced Manufacturing Technology](https://www.springer.com/journal/170)","snPcode":"170","submissionUrl":"https://submission.nature.com/new-submission/170/3","title":"The International Journal of Advanced Manufacturing Technology","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Solder Joint, Vision Inspection, Region of Interest, Localization, In-line Application","lastPublishedDoi":"10.21203/rs.3.rs-4935106/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4935106/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis paper reviews the entire vision inspection cycle, encompassing image acquisition, image enhancement, Region of Interest (ROI) localization and segmentation, features extraction followed by defect detection and classification.\u003c/p\u003e \u003cp\u003eThe aim of the study is to identify potential image processing time saving.\u003c/p\u003e \u003cp\u003eThe investigation innovatively suggests that optimizing image enhancement and ROI localization processing time could significantly accelerate the overall inspection cycle time without negatively impacting inspection accuracy.\u003c/p\u003e \u003cp\u003eIn Automated Optical Inspection (AOI) machine, camera sensor is mounted on precision X-Y gantries. To acquire images for inspection, the gantries will accurately move the camera to the predetermined coordinate position as stipulated in the inspection program. The vision camera will then capture the desired image using specified Field of View (FOV). Only ROI which is the solder joint position will be extracted out from the FOV image for processing.\u003c/p\u003e \u003cp\u003eMeanwhile, the designated solder joint positions (i.e. solder pad coordinates) for all electronic components mounted on the PCB are priory known extracted from the PCB fabrication file. These coordinates can be used directly for ROI localization without employing any algorithm, and yet accuracy is not compromised.\u003c/p\u003e \u003cp\u003eMeanwhile, through leveraging the state-of-art vision hardware, namely high-resolution camera and adaptive lighting system, quality images can be acquired and used directly without the need for any enhancement.\u003c/p\u003e \u003cp\u003eComparison analysis based on industrial PCB having 1000 electronics components (with 3000 solder joints of size 140x70 pixels per joint), the processing time utilizing NVIDIA GeForce RTX 2060 series Graphic Processing Unit (GPU) and Template Matching Algorithm for ROI localization needs 2 seconds. whereas when using Multiscale Morphology Algorithm for image enhancement, time required is approximately 3 seconds.\u003c/p\u003e \u003cp\u003eBenchmarking of a typical production line with bottleneck cycle time of 25 seconds, indicating that the proposed methodology effectively addresses the challenges faced while implementing real-time machine vision inspection system in the industry, aligned with Industrial 4.0 Smart Manufacturing initiatives.\u003c/p\u003e","manuscriptTitle":"Accelerated Accurate In-line Solder Joint Inspection Technique","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-10-21 11:06:37","doi":"10.21203/rs.3.rs-4935106/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewerAgreed","content":"","date":"2024-09-15T04:30:13+00:00","index":0,"fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-09-04T14:51:51+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-09-03T03:58:05+00:00","index":"","fulltext":""},{"type":"submitted","content":"The International Journal of Advanced Manufacturing Technology","date":"2024-09-01T21:54:13+00:00","index":"","fulltext":""},{"type":"decision","content":"Major Revisions Needed","date":"2024-08-23T11:08:35+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"the-international-journal-of-advanced-manufacturing-technology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jamt","sideBox":"Learn more about [The International Journal of Advanced Manufacturing Technology](https://www.springer.com/journal/170)","snPcode":"170","submissionUrl":"https://submission.nature.com/new-submission/170/3","title":"The International Journal of Advanced Manufacturing Technology","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"25510840-7dd6-49b9-a98e-35c93684f393","owner":[],"postedDate":"October 21st, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-03-24T16:08:17+00:00","versionOfRecord":{"articleIdentity":"rs-4935106","link":"https://doi.org/10.1007/s00170-025-15383-4","journal":{"identity":"the-international-journal-of-advanced-manufacturing-technology","isVorOnly":false,"title":"The International Journal of Advanced Manufacturing Technology"},"publishedOn":"2025-03-20 15:57:33","publishedOnDateReadable":"March 20th, 2025"},"versionCreatedAt":"2024-10-21 11:06:37","video":"","vorDoi":"10.1007/s00170-025-15383-4","vorDoiUrl":"https://doi.org/10.1007/s00170-025-15383-4","workflowStages":[]},"version":"v1","identity":"rs-4935106","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4935106","identity":"rs-4935106","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}