An Augmented Reality Maintenance Assistant with Real-time Quality Inspection on Handheld Mobile Devices

An Augmented Reality Maintenance Assistant with Real-time Quality Inspection on Handheld Mobile Devices | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article An Augmented Reality Maintenance Assistant with Real-time Quality Inspection on Handheld Mobile Devices James T Frandsen, Joe Tenny, Walter Frandsen, Yuri Hovanski This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-1842846/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 06 Feb, 2023 Read the published version in The International Journal of Advanced Manufacturing Technology → Version 1 posted 4 You are reading this latest preprint version Abstract With the advances of industry 4.0, augmented reality (AR) devices are being deployed across the manufacturing sector to enhance worker perception and efficiency. AR is often used to deliver spatially relevant work instructions on mobile devices for maintenance procedures on the factory floor. In these situations, workers use their mobile devices to view instructions in the form of 3D animations and annotations that directly overlay the equipment being maintained. Workers then follow the AR instructions and must ultimately rely on their own judgement and knowledge of the procedure as they progress from step to step. An AR assistant that could validate each stage of the procedure in real time and provide the worker with feedback on any observed errors would ensure that each maintenance procedure is completed successfully. This work presents a mobile, quality inspection system for AR maintenance procedures that is capable of assessing the maintenance task in real time. The system is designed for deployment on handheld mobile devices and can thus manage the challenges inherent to performing quality inspection with a non-fixed vision system. This work enumerates four essential qualities of mobile quality inspection tools and outlines some of the challenges encountered during the development of such a system. In the end, testing established that the system could provide adequate assistance for capturing inspection images, accurately process the captured images using machine vision, and generate detailed feedback from the quality inspection in a timely manner. Mobile augmented reality Quality inspection Computer vision Maintenance procedures Industry 4.0 Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 Figure 12 Figure 13 1. Introduction Industry 4.0 is a modern industrial trend sweeping across the manufacturing sector, and is characterized by a transition to smarter, more connected factories. The primary goal of this transformation is to leverage rapid data exchange and unified connectivity to achieve greater cohesion across the factory as a unit. Often synonymous with Industry 4.0, the term “industrial internet of things” (IIoT) is commonly used to describe the way machines, devices, sensors, and information technology (IT) systems should be connected and work together to create a coordinated and pseudo-living production system, or a “smart” factory. Regardless of the terminology one prefers, it is clear that as factories continue to become smarter it is becoming more important than ever to develop tools that keep the human workers integrated as part of the autonomous manufacturing enterprise. According to Jim Heppelmann, president and CEO of PTC, augmented reality (AR) is a novel human-machine interface that “turns bits and bytes into sounds and sights…” [ 1 ]. In other words, AR can translate the digital conversations taking place between machines and systems throughout the factory into a visual language that people can easily understand. The translated conversations can then be relayed to workers across the factory in a way that is contextually relevant to the tasks they are performing and allows them to stay tuned into the dialog of the smart factory. AR has boundless applications in a factory setting and is currently being deployed across many manufacturing operations. Ho, P. T., et al. (2022), in a recent systematic literature review, surveyed publications around industrial AR and found that AR is most commonly utilized for assembly/disassembly and maintenance operations [ 2 ]. AR is particularly advantageous for assembly/disassembly and maintenance operations because of its ability to deliver work instructions that are both spatially aligned and visually intuitive. It allows instructions to be translated from simplistic pictures in bulky paper manuals into dynamic visual sequences and 3D representations that directly overlay the work area. AR-based instructions have been shown to significantly lower workers’ mental load, which ultimately helps reduce the number of mistakes they make, by up to 82% [ 3 , 4 ]. This is largely because the worker can keep their focus on the workpiece, rather than constantly switching their attention back-and-forth between the workpiece and the paper instruction manual. Research is currently underway to find ways to further mitigate the number of errors that workers may make while being assisted by AR. The simplest of options is to require workers to manually check their workpiece, then indicate to the AR application whether they observed any errors. This type of AR experience has been described in literature for inspection scenarios [ 5 – 14 ], assembly scenarios [ 15 – 18 ], and even in construction settings [ 19 , 20 ]. These AR experiences begin to show the benefits of quality-infused AR. Still though, they necessarily rely on the experience and skill of each worker to be able to manually determine how well each step of the procedure was performed. The next step forward, then, is to develop systems that can make these quality determinations automatically and independent of worker perspective and experience. A system that is capable of visually inspecting the workpiece to find errors is advantageous over the systems described above because it would not be susceptible to operator error. An AR system that could provide objective process validation would be similar to existing automated quality assurance systems, such as an automated optical inspection system (AOI) for PCB boards, except it could make quality assessments at every step of the process and would also be responsible for providing the worker with the instructions necessary to complete the task. Ojer, M., et al. (2020) developed an AR system for PCB assembly that visually guides operators through the component mounting process while simultaneously checking the correctness of each action. This system is designed to display the AR instructions using either a monitor or a projector and can automatically validate each component’s physical characteristics, position, and rotation to detect, in real time, whether the components on the PCB have been mounted correctly [ 21 , 22 ]. This AR-assisted assembly operation takes place at a stationary workstation, so the inspection camera is calibrated for and fixed relative to the work area. Other AR systems demonstrating similar features have also been documented [ 4 , 23 – 33 ]. These AR systems are enabled by machine validation from a fixed camera, which is perfectly suited for tasks that can be completed at a workstation, such as assembly/disassembly. However, a fixed-camera setup is not feasible for tasks that must be completed at various locations across the factory floor (e.g., maintenance on large machinery). It would be much more desirable in these situations to have an AR system that is completely mobile and can be taken directly to the location of the procedure as the maintenance is being performed. AR systems built for mobile devices, such as handheld devices (HHDs) or head-mounted displays (HMDs), are commonly used to display maintenance instructions out on the factory floor because they utilize the camera native to the device to achieve true mobility. Infusing truly mobile AR systems with automated quality inspection has received relatively little attention in the literature because of unique challenges that are introduced when the inspection camera and illumination is no longer fixed relative to the work area. A few mobile AR systems with automatic quality inspection have been reported, but each requires that fiducial markers be placed on all components [ 19 , 34 – 36 ], which would hinder its deployment in an industrial setting. Fiducial markers are commonly used in AR to improve scene recognition and are particularly desirable for industrial scenarios since equipment surfaces usually lack adequate texture to reliably establish tracking using some tracking methods. However, the use of fiducial markers is impractical in industrial environments due to harsh conditions that could damage the markers and because it requires preparation of the workspace in advance [ 2 , 26 , 30 , 37 ]. Once the machine validation has been performed and the results are returned, specific instructions should be given to the worker to assist them with fixing any errors the system detected. These instructions should be clear enough to indicate to the worker exactly what to do to make the necessary repairs before proceeding to the next step. By rendering clear instructions, the validation system relieves any ambiguity related to the detected error and will help all workers move through a procedure more quickly. This level of feedback would transform the system from an AR-enabled error detection system into an AR-enabled maintenance assistant. A few articles have reported incorporating this kind of feedback, but each was deficient in at least one of the characteristics mentioned previously [ 23 , 30 , 31 , 33 , 36 , 38 ]. As has been shown, there are no publications to-date that demonstrate a real-time AR quality inspection system that is suitable for deployment in mobile industrial scenarios. An extensive review of relevant articles has been performed to verify this and the results can be seen in Table 1 . Therefore, this study will seek to understand the limitations of in-process quality inspection as it pertains to mobile AR and determine what specific tools need to be developed to enable a real-time maintenance assistant for mobile AR experiences. This paper is organized as follows: Section 2 gives a detailed description of the environment for which the proposed system was developed, including the tests used to assess the system’s performance. Section 3 presents the outcome of the system development process and discusses obstacles encountered during the development process. Finally, Section 4 draws conclusions based on the data presented and makes suggestions for future work. Table 1 A detailed summary of relevant AR articles showing a hole in the knowledge base. Mobility Display Type Essential Features Relevant Articles HHD HMD Monitor Projector Validation with a Mobile Camera Automatic Machine Vision Validation Markerless Tracking Dynamically Creates Repair Instructions Mobile x x x x [ 24 , 30 , 33 ] x x x [ 38 ] x x x [ 19 ] x x [ 10 , 12 , 39 ] x [ 5 – 9 , 40 – 45 ] x x x x [ 46 ] x x x x [ 31 ] x x x x [ 36 ] x x x [ 38 ] x x x [ 34 , 35 ] x x x [ 13 , 14 ] x x [ 29 , 47 , 48 ] x x [ 18 , 49 , 50 ] x [ 3 , 4 , 15 – 17 , 51 – 54 ] Stationary x x x x [ 31 , 32 ] x x x [ 24 – 28 ] x x [ 21 ] x x [ 20 , 37 , 55 , 56 ] x [ 15 , 57 , 58 ] x x x x [ 23 ] x x [ 22 ] x [ 11 ] 2. Methodology The primary goal of the proposed system is to create a mobile AR assistant that empowers workers in performing complex maintenance tasks. One of the primary barriers to developing such a system is the ability to successfully leverage machine vision to analyze images captured with a mobile device during an AR-guided procedure. To achieve this capability, several different components of the AR system and image processing algorithm must be developed and assessed: 1. The positional accuracy of the underlying AR tracking system to deduce the position of the user’s device within the AR space. This functionality is critical for helping users navigate around the physical equipment to capture an image for inspection. 2. The user’s ability to use the graphical navigational helpers shown in the AR experience to precisely capture a high-quality image for inspection. 3. An image processing algorithm that is robust enough to deal with the image variation inherent to a mobile-based system, while still producing accurate measurements. This program must also be able to analyze images of industrial hardware, which is notoriously texture-less and colorless, and reliably return accurate results. 4. The overall speed of the proposed system to transmit the captured image to the image processing algorithm, perform the automated image analysis, and receive the inspection results back into the AR experience. The environment in which the proposed system was developed will be described hereafter, as well as the testing that was performed to evaluate it. 2.1. Setup 2.1.1. Hardware The use case devised to evaluate the proposed system was based on the Festo Cyber-Physical Lab (CP Lab). The CP Lab is a modular Industry 4.0 learning system that is intended to convey an in-depth understanding of Industry 4.0 concepts. It includes many interchangeable application modules that perform specific functions and are representative of operations that might occur in an actual manufacturing facility. The CP Lab was constructed using industrial hardware to make the learning experience as similar to reality as possible. More detail about the CP Lab’s hardware can be found on Festo’s webpage for the CP Lab [ 59 ]. The development for this project was done with a four-station configuration of the CP Lab (CP Lab 404), and specifically used the iDrilling application module (Fig. 1). For this study, the proposed system was used to check the alignment of the through-beam sensors on the CP Lab’s iDrilling station. There are three sensors that use fiber-optic heads to transmit and receive beams of visible infrared light in order to detect the presence and placement of a workpiece. If these sensors become misaligned, the iDrilling station ceases to function because it cannot determine the state of each workpiece coming through. Realigning the three sensors is a time-consuming process that entails guessing and checking the position of each sensor until a satisfactory alignment is attained. Thus, the proposed system was adapted for this application to help workers ensure the correct alignment of each optical sensor and render specific realignment instructions as necessary. 2.1.2. Software The authoring environment used for the AR component of the proposed system was PTC’s Vuforia Studio. Vuforia Studio is a powerful, user-friendly tool that enables designers, engineers, and technicians to easily create scalable AR experiences without extensive programming knowledge. It integrates the robust computer vision capabilities of Vuforia Engine with real-time data from business systems and IIoT to create AR experiences that are both spatially and contextually relevant. These experiences can be created for both HHDs and HMDs and can be easily accessed across a company through a mobile application. Additionally, Vuforia Studio’s user-friendly interface allowed a focus on the industrial deployment of the proposed system, rather than strictly on its software development. The image processing portion of this project was developed in a Python environment and primarily utilized the OpenCV library for computer vision. Python was selected for the project because it is a versatile programming language that can be used in a variety of applications. Furthermore, the Python OpenCV library is more complete than the OpenCV library in other programming languages which provided a wider range of functions to choose from. The OpenCV library was chosen over other image processing libraries because of its open-source nature, its extensive online support, and its ability to analyze images based solely on a single template image. An alternative approach requires building a training dataset from hundreds or thousands of images. Manually capturing enough images to build a robust classifier is very labor-intensive and time-consuming. It is not feasible in a manufacturing setting when collecting the images could require the machinery being maintained to be shut down and production halted for safety reasons. In such a scenario, it is much more desirable to capture one single image instead of hundreds or thousands. Therefore, the proposed system relied on just three template regions taken from one complete template image. Two of the template regions contained mechanical features separated by a known distance which were used for image calibration, and the third template region contained one of the sensors whose distance would be measured. Each image containing the template region was used to identify key features of the captured inspection images so that the necessary measurements could be taken. It should be noted that the template images were all chosen to be in the same plane in order to minimize the effects of parallax. As noted previously, markerless tracking methods are much better suited for industrial environments than their fiducial-based counterparts. In terms of markerless tracking, Vuforia Studio has the ability to use 3D mesh scans or CAD-based assets to recognize physical objects. Establishing tracking with 3D scans requires a scan of the area/equipment to be taken before the AR experience can recognize the space and perform any rendering. These scans provide robust tracking as features throughout the entire area can be used for reference, but do not perform well if the locale changes (e.g., using one AR experience for multiple pieces of equipment across a factory, the equipment is relocated, or the area around the equipment is significantly modified). Considering these factors and in order to provide a versatile AR experience for maintenance procedures, a CAD-based model target tracking method was selected. This allowed the AR experience to recognize the equipment with no reference to the surrounding region. The overall software architecture for the proposed system consists of a few applications and servers that are used to provide AR-based guidance, transfer/store the captured images, and process the captured image to check for quality. Figure 2 gives a graphical depiction of the relationship between all entities within the proposed system. The relationships will also be described hereafter, and it should be noted that all servers and programs were hosted on-prem at Brigham Young University. Workers used the Vuforia View app to interact with the AR elements of the system. The AR experience was responsible for guiding them through the maintenance task and did so with graphical and text-based instructions at each step. Once workers were ready for their work to be inspected, they triggered the inspection in the system and followed the instructions for capturing a high-quality image. After an image was captured, it was sent to ThingWorx, an IIoT platform, where it was stored until it was retrieved by the Python image processing algorithm. ThingWorx was used as an intermediary between the AR system and the image processing algorithm to facilitate smoother eventual deployment of the proposed system in a real manufacturing facility. The ThingWorx platform has native capabilities to connect to enterprise systems, such as CMMS, MES, or ERP, which could allow the proposed system to be integrated with any company’s existing IT platforms. Once retrieved by the image processing algorithm, the OpenCV algorithms were used to extract the data necessary to perform measurements. Then, the measurements and other pertinent data were passed back to the AR system, via ThingWorx, and instructions were automatically generated to guide the user through repositioning any of the optical sensors that were detected to be out of alignment. 2.1.3. Devices The augmented reality component of the proposed system was designed to be deployed on HHDs, such as a tablet. HHDs were selected for this project because they offer a few unique advantages over other device types. One reason HHDs were selected is that they are a relatively inexpensive and off-the-shelf consumer product, which enables the system to be used by many workers. This makes them very attractive for companies deploying AR solutions since multiple AR-capable devices must be purchased. Another reason is that HHDs have been widely accepted in society so the majority of people are very familiar with them and can already operate them effectively [ 7 ]. There is typically a learning curve when using other AR-capable devices, such as HMDs, due to the unfamiliar and less-intuitive user interface. Additionally, HHDs have both a camera and a display screen which creates a fully functional AR system that can easily be taken anywhere in a factory. Another reason HHDs were selected is that they were the only classification of AR devices approved for enterprise-level use in our industrial sponsor’s manufacturing facilities. The proposed system was evaluated using two different HHDs to demonstrate device agnosticism. Specifically, the two HHDs used for testing were an Apple iPad Pro 12.9" and an Apple iPad 7th Gen. Once the inspection image was captured, the inspection image was transmitted to a laptop computer where the image analysis occurred. A Dell Inspiron 13-7378 laptop with a 6th Generation Intel Dual-Core i5-7200 CPU was used to process the captured images. The image processing was allocated to an external device after preliminary testing revealed the processor on HHDs are typically underpowered for rapid processing of high-resolution images while simultaneously running the AR experience. Running both the AR experience and the image process on the HHD resulted in significant lagging during the AR experience which was undesirable. 2.2. Testing & Evaluation 2.2.1. Test #1: AR Positional Accuracy – Method The purpose of the first test was to determine the AR system’s ability to accurately locate itself relative to the iDrilling station. As stated previously, the tracking for this project was established with a CAD-based model target of the iDrilling station, so all positioning within the AR space happened in relation to the station itself. Once the underlying AR system could recognize the iDrilling station and establish tracking, Vuforia’s Device Pose Observer (DPO) began to compute the position and orientation of the user’s device within the AR space. The DPO analyzed both the live camera stream and data from the device’s sensors to continually estimate the device’s position, even when the reference object for tracking moved out of the camera’s view. The proposed system leveraged the DPO’s positional data to monitor the device’s position relative to the image capture position, which allowed it to automatically capture an inspection image when the device was properly aligned with the system’s navigational helpers. The device’s ability to accurately determine its position was critical to capturing good images for inspection, so testing was performed to quantify the DPO’s position estimation capabilities. An iPad running an AR positioning experience was mounted on a Universal Robots UR3e collaborative robot (Fig. 3), which was programmed to move between four waypoints. The device’s X, Y, Z coordinates, as estimated by the DPO, were recorded at three of the four positions. These three positions were at the image capture position and laterally offset 30mm ± 0.3mm to either side. The fourth position was significantly further away from the other three and was included to simulate the large device movements that typically occur during mobile AR experiences. The iPad repeated this program fifteen times during testing, so a total of 45 data points were collected. The UR3e’s pose repeatability was ± 0.3mm [ 60 ]. 2.2.2. Test #2: Device Alignment Time– Method The second set of tests sought to quantify a user’s ability to position their device using the navigational helpers to capture a repeatable inspection image. As stated previously, the proposed system captured images automatically when it determined that the user had properly aligned their device with the image capture position. A tolerance was defined around the image capture position to create the range of acceptable alignments, which made properly aligning their device more feasible for users. The navigational helpers, as shown in Fig. 4 below, consisted of a feet marker to show users where to stand, a 3D model depicting hands holding a tablet/phone to show users where to position their device, and a dynamic 3D path to lead users to the image capture position from a distance. Each of these components selectively faded as users approached the image capture position to ensure users could see the work area as the inspection image was captured and to increase user-friendliness. The navigational helpers were intended to intuitively guide users through the AR space in order to obtain the viewpoint from which the inspection image should be captured. These navigational AR “helpers” harnessed the power of AR to enhance human cognition and give users a more precise and complete spatial understanding of the work area. By guiding users to a specific point of view, the proposed system ensured that consistent images of the workpiece could be captured, which was requisite for attaining reliable results during image processing. If users could not capture a repeatable image, it would have been impossible for consistent results to be attained, since metrology algorithms depend on high-quality images to produce high-quality results. This component of the AR experience was evaluated by defining the image capture position and an associated tolerance around it, then recording the time a user needed to properly align their device within the defined tolerance, using the navigational helpers. While acknowledging that the alignment process will not always begin from the same position, it is still important to assess the tolerances relative to each other as a means of quantifying their usability. The testing began with an initial tolerance of ± 50mm around the ideal image position and incrementally decreased to ± 1mm. Three practice alignments were performed before data collection began at each tolerance, then ten consecutive alignments were performed and timed. As the alignment tolerance was gradually decremented, it was expected that the time required for users to attain a proper alignment would increase to some degree. Since worker efficiency is paramount in manufacturing settings, it was necessary to determine an optimal balance between the positional precision required from users and the time needed to achieve a proper alignment. 2.2.3. Test #3: Measurement Accuracy – Method The third test aimed to compare the measurement accuracies of two different image processing approaches. Each approach addressed the task of identifying the critical parts of captured images in a different manner. One method was intended to deal with significant variation in the images it would process, both in feature scale and rotation. The other method was better suited for a more controlled scenario in which the captured images could be more consistent. By comparing two methods, it was possible to assess which one was better suited for the proposed system. The comparison was performed using a set of 25 images captured at a resolution of 2732x2048 pixels. The first approach selected relied on extracting key features from the captured image and template images, determining matching keypoints between the images, then using the matched keypoints to take measurements within the captured image. Specifically, the scale-invariant feature transform (SIFT) algorithm [ 61 ] was used to extract keypoints and descriptors from the captured image and each template image. Then, the fast library for approximate nearest neighbors (FLANN) algorithm [ 62 ] compared the keypoints and descriptors from the captured image to the keypoints and descriptors of each template image in order to locate key features within the captured image (e.g., the three optical sensors). Finally, the distances between the key features could be calibrated and measured. A logic diagram illustrating this approach is shown in Fig. 5 . The other image processing approach depended on using each template image in its entirety, rather than individually extracted keypoints, to identify key regions within the captured image. It utilized OpenCV’s template matching algorithm using cross correlation to slide each template image over the captured image and calculate the similarity between the template and the covered area. Once the best matching regions were determined, the measurements across the captured image could be taken. A logic diagram illustrating this approach is shown in Fig. 6 . 2.2.4. Test #4: Total Processing Time – Method This final set of tests was performed to determine the overall image processing time of the proposed system. It was necessary to ensure the automatic inspection was rapid enough as to not encumber the natural flow of the maintenance task or lose the user’s attention. Recent studies have shown that 50% of people will lose interest in and leave a website if it takes more than six seconds to load [ 63 ]. While acknowledging that an AR experience is not necessarily equivalent to a website, this time limit of six seconds was adopted as a standard for the proposed system to ensure users stay engaged with the task at hand. It was anticipated that one of the primary limiting factors for the total processing time would be the resolution of the captured image since this was by far the system’s largest data point. Therefore, to optimize the overall speed of the proposed system, the images for inspection should be captured with lowest possible resolution. However, as the resolution of the captured images was reduced, the images consequently lost pixel information which impacted the system’s ability to measure distances accurately. Therefore, a balance had to be found between using an image with high enough resolution to reliably the inspection yet low enough resolution to be transmitted and processed quickly. For the proposed system, the total processing time could be subdivided into four distinct components, as depicted in Fig. 2 : transmitting the captured image from the AR device to ThingWorx, retrieving the captured image and other vital data from ThingWorx, processing the captured image with the image processing algorithm, and returning the inspection results back to the AR device. Each component was evaluated by transferring or processing the associated data one hundred times at each image resolution and recording the time required for each round. In addition to the time trials, the accuracy of the automatic inspection at each image resolution had to be assessed. To accomplish this, 25 sample images of the iDrilling station were captured with the three optical sensors a set position. Starting with a full-size 2732x2048 image, the images were decimated by 12.5% while the error of the measurements from the sensors’ known position were recorded. Thus, the overall processing times could then be compared to the measurement accuracy at each resolution to determine the optimal image resolution for the proposed system. 3. Results & Discussion 3.1. Test #1: AR Positional Accuracy After performing the test described in 2.2.1, it was determined that the AR system’s DOP could estimate the HHD’s position with an accuracy of ± 1.8mm. This result was obtained by first determining the expected positions for the three test positions. The expected positions were defined by determining the average X, Y, Z coordinates of the center position, then adding/subtracting 30mm to/from the average center x-value to determine the expected right and left positions. Then, the Euclidean distance from each of the 45 measured points to their expected positions was calculated, using Eq. 1. Next, the mean error of all the calculated distances was found. The mean error of the 45 positions was found to be 2.1mm. Therefore, once the positional tolerance of the UR3e was factored into this, the DOP’s positional capabilities can be said to be accurate, on average, to ± 1.8mm. A 3D scatter plot of the 45 measured points can be seen in Fig. 7 . $$d\left(p,q\right)=\sqrt{\sum _{i=1}^{n}{\left({q}_{i}-{p}_{i}\right)}^{2}}$$ Equation 1 where: p , q = two points in Euclidean n-space p i , q i = Euclidean vectors, starting from the origin of the space n = n-space It is important to note that the datapoints displayed in Fig. 7 are not to scale relative to the image of the iDrilling station included on the X-Y plane. The image of the iDrilling station was included to illustrate the coordinate system in which the data was collected. Each of the three axes are equal in length to render an accurate depiction of how the datapoints were distributed. It can be observed that the positional error did not vary significantly between the three test positions. In terms of the proposed quality inspection system, these results indicated that the user’s device could be reliably placed relative to the iDrilling station within a range of ± 1.8mm. This discrepancy, combined with the tolerance around the image capture position determined in Test #2, caused variation in the real-world viewpoint from which inspection images were captured, which increased the difficulty of obtaining valid results from the image processing algorithm. An analysis of the proposed system’s ability to manage this variation was investigated in Test #3. 3.2. Test #2: Device Alignment Time As described in 2.2.2, this test was performed by recording the time a user needed to properly align their device using the navigational helpers. The navigational helpers were intended to provide visual cues to help users quickly align their device so an inspection image could be automatically captured. A set of ten alignments were recorded for each tolerance and the averages of each set were then calculated. The testing revealed that as the tolerances were decreased, their associated alignment times increased exponentially. It should be noted that timing sets were not recorded for the ± 2mm and ± 1mm tolerances because the associated alignments became too difficult to feasibly achieve. Based on these results, an alignment tolerance of ± 10mm about the image capture position was selected for the system because it was the best precision attainable before the average alignment began to increase significantly. Figure 8. A sequence of images depicting how the alignment arrows integrated with the original navigational helpers (subfigure a) to facilitate fine device alignment. As the user approached the navigational helpers, the device/hands model faded away and the alignment arrows appeared (subfigure b). The user then aligned the arrow pinned in front of their device (right arrow in subfigure c) with the arrow at the image capture position (left arrow in subfigure c) until fine alignment was achieved (subfigure d). As the proposed system’s image processing component was continually tested using images captured within this ±10mm threshold, an abundance of feature identification errors were detected. It was discovered that these errors were due to the machine vision algorithm’s inability to adequately deal with the variation created by the alignment threshold. Such a large tolerance created too much variation across captured images, since the size and obliqueness of key measurement features could vary considerably. As it would not be feasible to decrease the alignment tolerance due to alignment time constraints, a pair of alignment arrows were added to the navigational helpers to facilitate fine alignment. These arrows were intended to help users visualize the exact image capture position and the position of their device relative to it. Once the user had used the original navigational helpers to approach the image capture position, an alignment arrow appeared pinned to the center of their screen, while the other arrow remained stationary at the image capture position as a point of reference. Each arrow had specific features intended to help users intuitively monitor their device’s rotational and translational alignment relative to the image capture position, shown as the reference arrow. It was anticipated that these new features would facilitate precise alignment and help users decrease the needed alignment times. The way the alignment arrows were used to achieve fine alignment is shown in in Figure 8. When the tests across the same range of tolerances were repeated, the average alignment times all decreased significantly. As is shown in Fig. 9 , the additional navigational tools helped users align their devices with the image capture positions much more quickly, even when the tolerances were very narrow. From these new results, an alignment tolerance of ± 2mm about the image capture position was implemented for the proposed system. The chosen tolerance provided the optimal balance of alignment speed and precision since it could now be achieved, on average, in just 3.5 seconds. Successful testing of the proposed system’s image processing component with images captured within this new ± 2mm confirmed it as the right choice. It is suspected that the time required to achieve a proper alignment significantly increased for tolerances less than ± 2mm due to the DPO’s positional capabilities. At a tolerance of ± 1mm, the DPO struggled to distinguish whether or not the device was within ± 1mm of the image capture position since it exceeded the DPO’s ability to accurately calculate the position of the user’s device. This caused proper alignments to become more difficult to achieve which produced the lengthier alignment times shown in the data. Thus, the addition of the pair of alignment arrows allowed users to easily position their device with such great precision that the positional capabilities of the underlying AR system became the limiting factor for alignments. With further development of the DPO and underlying AR system, even greater alignment precision could potentially be achieved to produce more consistent inspection images. 3.3. Test #3: Measurement Accuracy Since variation between inspection images had been minimized with the DPO as the limiting factor, it then became appropriate to begin to assess the measurement capabilities of the machine vision system. The measurement accuracies of the two image processing approaches were evaluated using methodology described in 2.2.3. Based on the measurements taken from the set of 25 test images, the average error of measurements by the SIFT + FLANN approach was calculated to be 1.62mm while the average error of the template matching approach was 1.21mm. The entire dataset is shown in Fig. 10 . Testing that was performed previously revealed that the optical sensors themselves could tolerate variations in their positioning up to ± 1.0mm. Because of the hardware’s physical limitations, it became clear that the measurement accuracy of both image processing approaches would be insufficient to help users reliably locate and reposition the optical sensors. Therefore, more work needed to be done to improve the system’s accuracy. An investigation into the captured images themselves revealed that they were being subjected to a degree of distortion from the mobile device’s camera. This distortion from the lens caused the sensors to appear in different positions than they actually were, which compromised the system’s ability to obtain accurate measurements. This issue was initially suspected since the inspection images were being captured by a consumer-grade camera, rather than an industrial-grade vision system. The lower quality camera lenses used in consumer mobile devices produce distorted images due to the relatively short focal lengths of their lenses, as well as other factors. These distortions can be imperceptible to the naked human eye, while still being significant enough to compromise a precise machine vision analysis and measurement. The issue was then confirmed by manually identifying the key features within the captured images and calculating the measurements by hand. Even without using one of the automated measurement approaches, the measurements were still incorrect which confirmed the images were indeed distorted. To overcome the lens-induced distortion, several images of a calibration board were captured so that the intrinsic parameters of the mobile device’s camera could be calculated. These parameters included a distortion matrix and coefficients which described the lens’ optical center and focal length. Using these parameters, the set of 25 test images were corrected and the two image processing approaches were tested again. After analyzing the new results, it became clear that the images had in fact become more distorted since the average errors from both approaches increased. Further consideration of the distortion correction system revealed that the distortion matrix and coefficients being used to correct the captured images could not account for the mobile device’s auto-focus capabilities. The calculated intrinsic parameters could only be used for a single focal length which turned out to be the source of their inaccuracy since the mobile device could constantly change the focal length to best capture each image. The most straightforward solution would have been to fix the camera’s focal length within the AR experience, but this would have caused the AR renderings to be blurred when viewed from any distance other than the one used to capture inspection images. To circumvent this challenge, another set of calibration images were captured, but this time they were captured at the same distance from the calibration board as the inspection images would be from the iDrilling station. This ensured the focal lengths for the new calibration images would be more closely matched to the inspection images. When these tests were repeated a third time using the same set of 25 images, the results improved significantly. As shown in Fig. 10 , the SIFT + FLANN approach had achieved an average measurement error of 1.27mm and the template matching approach could produce measurements with an average error of 0.84mm. In context of the actual measurements being taken, this translates to a 0.7% average error across the three optical sensor measurements. With the new distortion correction measures, the template matching approach was proven to satisfy the positional tolerance of the optical sensors. Thus, the template matching approach was chosen as the preferred image processing engine for the proposed system. In terms of deploying the proposed system to be used across a wide range of mobile devices, it was not sufficient to only use one set of calibration parameters for all devices. The camera on each device could potentially have its own combination of optical center and focal length, so one calibration parameter to rule them all would not be adequate. This project assumed that the image distortion was consistent across all devices of the same type (e.g., all iPad Pro 12.9”), but varied from device type to device type (e.g., iPhone 12 vs iPhone 13). Therefore, it was necessary to build flexibility into the proposed system to use device-specific calibration parameters. To address this challenge, an AR-guided camera calibration tool was developed to help users produce the right parameters for uncalibrated devices. This AR-enabled calibration tool could track users’ position relative to the calibration board, guide them to capture specific images for calibration, and ensure the images were captured at the correct distance from the calibration board. These images were used to calculate the necessary calibration parameters and were then stored in a ThingWorx data table. The individual entries within this data table were later accessed by the image processing algorithm at runtime to allow the inspection images to be corrected according to the device which captured them. It is interesting that the SIFT + FLANN approach, which was better suited to deal with image variation, performed worse than the template matching approach. It was suspected that this was a result of two primary factors: the image repeatability from the AR navigational helpers and the lack of texture on the hardware. When using the navigational helpers to precisely position the AR device, users could ensure images were captured from the same viewpoint relative to the hardware with an average accuracy of 4mm, as established by 3.1 and 3.2. This positional precision eliminated enough variation within the captured images that the scalar and rotational robustness of the SIFT + FLANN approach was no longer advantageous. In addition, the SIFT feature extraction algorithm performs best when used to analyze high-contrast, highly textured images since it relies on these things to detect keypoints. The iDrilling station is mostly comprised of 80/20 aluminum, which has a smooth, monochrome surface. This lack of contrast and texture increased the difficulty of extracting keypoints from the captured images, which ultimately rendered the SIFT + FLANN approach unreliable. 3.4. Rendering Inspection Feedback Once the image processing algorithm completed the inspection, inspection results were returned to the AR experience so that repair instructions could be generated for the user. The realignment feedback took advantage of AR’s ability to visually render work instructions and was designed to help workers understand the detected problem and know exactly how to repair it. Once the inspection results arrived back at the AR experience, a result summary popup appeared to indicate whether any of the three sensors were misaligned. If any alignment errors had been detected, these sensors were flagged as being out of alignment, and the exact distance and direction of the misalignment were displayed in the popup summary. A sample result summary popup is shown in Fig. 11a. Then, AR and text instructions were dynamically generated to help workers repair each detected misalignment step-by-step. The misaligned sensor was highlighted in its current, misaligned position, then an animation depicting the sensor being repositioned to its correct position was shown. A sequence of screenshots, shown in Fig. 11 subfigures b-e, illustrate the realignment animations. Additionally, text instructions accompanied each animation sequence to indicate the displacement distance and direction, as well as the tools required for the repair. Once the worker had realigned each misaligned sensor, the inspection process was repeated, and the sensors’ new positions were verified. This process could be repeated as many times as necessary until proper sensor alignment was achieved. 3.5. Test #4: Total Processing Time Following the methodology outlined in 2.2.4, tests were performed to determine the time required to complete the system’s four processing components. Each component was assessed individually using images at eight different resolutions in order to quantify how image resolution affects the overall processing time of the proposed system. Figure 12 illustrates how each component is affected by the resolution of the image being transferred/processed. It is interesting to note that the time required to return the results JSON from the image processing algorithm back to the AR device was not affected by image resolution. Altogether, the composite processing time of the four components that were assessed ranged from 3.21 seconds to 0.39 seconds. This signified that, regardless of what image resolution was used, the system’s total processing time would always fall below the threshold of six seconds. Still, it was desirable to minimize the total time users spent waiting for the inspection results to be returned. Testing was then performed to determine the system’s ability to take accurate measurements on images with varying resolutions. The template matching approach chosen in test #3 was evaluated again, but this time using images at eight different resolutions. Using the measurement accuracy data alongside the data for processing times, it was determined that the optimal resolution for images captured by the proposed system was 1025x768 pixels. This image resolution offered the best balance between the overall processing time and the accuracy of the measurements that the system took. In fact, the images captured at the lowest resolution, 342x256 pixels, had been reduced so much that the image processing algorithm could not be reliably find the three optical sensors, so the results for this resolution were excluded. As shown in Fig. 13 , the total processing time exhibited a negatively exponential relationship with the changing image resolutions, while the measurement accuracy’s relationship was negatively linear overall. As noted in the analysis for test #3, the alignment tolerance for the iDrilling station’s optical sensors was ± 1mm and has been included in Fig. 13 for reference. Images captured at a resolution of 1025x768 allowed for the fastest processing while also minimizing the average measurement accuracy. 4. Conclusions & Future Work An AR-enabled maintenance assistant has been developed that incorporates the four essential characteristics of a truly mobile in-process quality inspection tool – namely, a mobile inspection camera, automatic work validation via machine vision, markerless tracking, and dynamic creation of repair instructions. During the development process, the primary challenge that was encountered was using a consumer mobile device to capture images that were consistent and distortion-free so accurate inspection results could be reliably computed. As stated previously, if a good image could not be captured for the inspection, it would have been impossible to obtain good results. Therefore, four tests were performed to evaluate the proposed system’s ability to capture a high-quality inspection image, reliably process the image and obtain accurate results, then return the inspection results to users in a timely manner. The AR Positioning test aimed to quantify the DPO’s ability to calculate the device’s real-world position relative to the iDrilling station. The positional precision of this component directly affected how much the inspection images would vary. This test indicated that the DPO’s estimations were accurate to ± 1.8mm on average. The Device Alignment Time test assessed how quickly users could position their device within a set tolerance. The initial results indicated the navigational helpers were not sufficient to help users achieve a tight alignment tolerance, so a pair of alignment arrows were added. With this new addition, all alignment times were reduced due to the newly added spatial clarity and a much tighter alignment tolerance that was then attainable. Based on these new results, the system’s alignment tolerance was set at ± 2mm, which was limited by the DPO’s tested accuracy. The Measurement Accuracy test evaluated two different approaches to image processing and compared the ability of each to produce accurate measurements on captured inspection images. Several rounds of testing uncovered unavoidable challenges that stemmed from using mobile devices for such a precise machine vision task. Lens-induced distortion caused measurement inaccuracies which were remedied with a camera calibration regimen. However, the device’s auto-focus feature warranted that special considerations be addressed during the calibration process. With these issues resolved, the test results indicated that the template matching approach was better suited for the proposed system since sub-millimeter measurement accuracy could be achieved. Finally, the Total Processing Time test sought to minimize the proposed system’s total processing time by gradually reducing the resolution of the inspection images while monitoring the average measurement error. According to the testing results, images captured at a resolution of 1025x768 pixels offered an optimal balance of processing time and measurement accuracy. Using this image resolution, the system was able to return inspection results in less than one second and had an average measurement error of 0.65mm. Ultimately, the testing demonstrated that the proposed system could help users quickly capture acceptable inspection images, reliably take accurate measurements on the captured images, and do so in a timely manner. All these things were designed to be adapted to any consumer mobile device which would be conducive to streamlined deployment in a manufacturing environment. There are a few aspects of this project that could be improved in future versions. In terms of the system’s overall usability, further work could be done to transition the machine vision from inspecting just a single image to processing the live camera stream. This would most likely require the image processing to be relocated to the mobile device for latency’s sake but could potentially open the door to a system that works in real time and is more user-friendly. Regarding the image processing itself, other methods could be explored to try to achieve greater measurement accuracy and robustness as ambient lighting conditions changed and surfaces aged or became soiled. Training data models based on CAD data could allow deep learning techniques to be used, while not requiring hundreds or thousands of actual training images to be captured. Alternatively, as consumer mobile devices are being equipped with more advanced sensors, LiDAR or RGBD cameras could generate 3D point clouds of the work area which could then be used to perform more precise measurements. Declarations Acknowledgements Funding for this research was provided by Northrop Grumman Corporation via contract #4800056662 to BYU. PTC, the parent company to Vuforia, ThingWorx, and Kepware, donated licenses for each of these software to the lab so they could be used in the development of the proposed system. Additionally, Festo Didactic generously supported the integration on their 4-station CP Lab, which was used for this project’s test use case. Statements and Declarations Funding This work was supported by Northrop Grumman Corporation via contract #4800056662. Competing interests The authors have no relevant financial or non-financial interests to disclose. Availability of data and material The authors declare that the data supporting the findings of this study are available within the article and its supplementary information files. Code availability The custom code that supports the findings of this study is available from the corresponding author, James Frandsen, upon reasonable request. Ethics approval The authors confirm that this manuscript is original and has not been published, nor is it currently under consideration for publication elsewhere. Consent to participate Not applicable. Consent for publication Not applicable. Authors’ contributions All authors contributed to the study’s conception, and design, and development. The first draft of the manuscript was written by James Frandsen and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript. References LiveWorx 2021 Episode Three: Digital Transforms Physical . 2021, PTC, Inc. Ho, P.T., et al., Study of Augmented Reality Based Manufacturing for Further Integration of Quality Control 4.0: A Systematic Literature Review. Applied Sciences, 2022. 12 (4): p. 1961. Tang, A., et al. Experimental evaluation of augmented reality in object assembly task . in Proceedings. International Symposium on Mixed and Augmented Reality . 2002. IEEE Comput. Soc. Tang, A., et al. Comparative effectiveness of augmented reality in object assembly . in Proceedings of the conference on Human factors in computing systems - CHI '03 . 2003. ACM Press. Barbieri, L. and E. Marino, An Augmented Reality Tool to Detect Design Discrepancies: A Comparison Test with Traditional Methods , in Lecture Notes in Computer Science . 2019, Springer International Publishing. p. 99-110. Bruno, F., et al., An augmented reality tool to detect and annotate design variations in an Industry 4.0 approach. The International Journal of Advanced Manufacturing Technology, 2019. 105 (1-4): p. 875-887. Holm, M., et al., Adaptive instructions to novice shop-floor operators using Augmented Reality. Journal of Industrial and Production Engineering, 2017. 34 (5): p. 362-374. Marino, E., et al., An Augmented Reality inspection tool to support workers in Industry 4.0 environments. Computers in Industry, 2021. 127 : p. 103412. Yang, X., et al., Edge-based cover recognition and tracking method for an AR-aided aircraft inspection system. The International Journal of Advanced Manufacturing Technology, 2020. 111 (11-12): p. 3505-3518. Polvi, J., et al., Handheld Guides in Inspection Tasks: Augmented Reality versus Picture. IEEE Transactions on Visualization and Computer Graphics, 2018. 24 (7): p. 2118-2128. Zhou, J., et al. Applying spatial augmented reality to facilitate in-situ support for automotive spot welding inspection . in Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry - VRCAI '11 . 2011. ACM Press. Hartl, A.D., et al., Efficient Verification of Holograms Using Mobile Augmented Reality. IEEE Transactions on Visualization and Computer Graphics, 2016. 22 (7): p. 1843-1851. Runji, J.M. and C.-Y. Lin. Automatic Optical Inspection aided Augmented Reality-based PCBA Inspection: A Development . in 2019 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT) . 2019. IEEE. Runji, J.M. and C.-Y. Lin, Markerless cooperative augmented reality-based smart manufacturing double-check system: Case of safe PCBA inspection following automatic optical inspection. Robotics and Computer-Integrated Manufacturing, 2020. 64 : p. 101957. Odenthal, B., et al., A Comparative Study of Head-Mounted and Table-Mounted Augmented Vision Systems for Assembly Error Detection. Human Factors and Ergonomics in Manufacturing & Service Industries, 2014. 24 (1): p. 105-123. Yuan, M.L., S.K. Ong, and A.Y.C. Nee, Augmented reality for assembly guidance using a virtual interactive tool. International Journal of Production Research, 2008. 46 (7): p. 1745-1767. Zauner, J., et al. Authoring of a mixed reality assembly instructor for hierarchical structures . in The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings. 2003. IEEE Comput. Soc. Zhang, J., S.K. Ong, and A.Y.C. Nee, RFID-assisted assembly guidance system in an augmented reality environment. International Journal of Production Research, 2011. 49 (13): p. 3919-3938. Kwon, O.-S., C.-S. Park, and C.-R. Lim, A defect management system for reinforced concrete work utilizing BIM, image-matching and augmented reality. Automation in Construction, 2014. 46 : p. 74-81. Shin, D.H. and P.S. Dunston, Evaluation of Augmented Reality in steel column inspection. Automation in Construction, 2009. 18 (2): p. 118-129. Ojer, M., et al., Real-time automatic optical system to assist operators in the assembling of electronic components. The International Journal of Advanced Manufacturing Technology, 2020. 107 (5-6): p. 2261-2275. Ojer, M., et al., Projection-Based Augmented Reality Assistance for Manual Electronic Component Assembly Processes. Applied Sciences, 2020. 10 (3): p. 796. Alves, J., et al. Comparing Spatial and Mobile Augmented Reality for Guiding Assembling Procedures with Task Validation . in 2019 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC) . 2019. IEEE. Alves, J., et al., Using Augmented Reality and Step by Step Verification in Industrial Quality Control , in Human Systems Engineering and Design III . 2021, Springer International Publishing. p. 350-355. Gupta, A., et al. DuploTrack . in Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST '12 . 2012. ACM Press. Manuri, F., A. Pizzigalli, and A. Sanna, A State Validation System for Augmented Reality Based Maintenance Procedures. Applied Sciences, 2019. 9 (10): p. 2115. Wasenmuller, O., M. Meyer, and D. Stricker. Augmented Reality 3D Discrepancy Check in Industrial Applications . in 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) . 2016. IEEE. Alves, J.B., et al., Using augmented reality for industrial quality assurance: a shop floor user study. The International Journal of Advanced Manufacturing Technology, 2021. 115 (1-2): p. 105-116. Khuong, B.M., et al. The effectiveness of an AR-based context-aware assembly support system in object assembly . in 2014 IEEE Virtual Reality (VR) . 2014. IEEE. Nishihara, A. and J. Okamoto. Object recognition in assembly assisted by augmented reality system . in 2015 SAI Intelligent Systems Conference (IntelliSys) . 2015. IEEE. Pathomaree, N. and S. Charoenseang. Augmented reality for skill transfer in assembly task . in ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005. 2005. IEEE. Wu, L.-C., I.-C. Lin, and M.-H. Tsai. Augmented reality instruction for object assembly based on markerless tracking . in Proceedings of the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games . 2016. ACM. Okamoto, J. and A. Nishihara, Assembly assisted by augmented reality (A 3 R) , in Intelligent Systems and Applications . 2016, Springer. p. 281-300. Feiner, S., B. Macintyre, and D. Seligmann, Knowledge-based augmented reality. Communications of the ACM, 1993. 36 (7): p. 53-62. Liverani, A., G. Amati, and G. Caligiana, A CAD-augmented Reality Integrated Environment for Assembly Sequence Check and Interactive Validation. Concurrent Engineering, 2004. 12 (1): p. 67-77. Westerfield, G., A. Mitrovic, and M. Billinghurst, Intelligent Augmented Reality Training for Motherboard Assembly. International Journal of Artificial Intelligence in Education, 2015. 25 (1): p. 157-172. Alvarez, H., I. Aguinaga, and D. Borro. Providing guidance for maintenance operations using automatic markerless Augmented Reality system . in 2011 10th IEEE International Symposium on Mixed and Augmented Reality . 2011. IEEE. Dalle Mura, M. and G. Dini, An augmented reality approach for supporting panel alignment in car body assembly. Journal of Manufacturing Systems, 2021. 59 : p. 251-260. Lee, H., et al., A Framework for Process Model Based Human-Robot Collaboration System Using Augmented Reality , in Advances in Production Management Systems. Smart Manufacturing for Industry 4.0 . 2018, Springer International Publishing. p. 482-489. Antonelli, D. and S. Astanin, Enhancing the Quality of Manual Spot Welding through Augmented Reality Assisted Guidance. Procedia CIRP, 2015. 33 : p. 556-561. Hakkarainen, M., C. Woodward, and M. Billinghurst. Augmented assembly using a mobile phone . in 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality . 2008. IEEE. Liu, C., et al. Evaluating the benefits of real-time feedback in mobile augmented reality with hand-held devices . in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . 2012. ACM. Mourtzis, D., F. Xanthi, and V. Zogopoulos, An Adaptive Framework for Augmented Reality Instructions Considering Workforce Skill. Procedia CIRP, 2019. 81 : p. 363-368. Schlueter, J.A., Remote Maintenance Assistance Using Real-time Augmented Reality Authoring , in ProQuest Dissertations and Theses . 2018, Iowa State University: Ann Arbor. p. 72. Segovia, D., et al., Augmented Reality as a Tool for Production and Quality Monitoring. Procedia Computer Science, 2015. 75 : p. 291-300. Li, S., P. Zheng, and L. Zheng, An AR-Assisted Deep Learning-Based Approach for Automatic Inspection of Aviation Connectors. IEEE Transactions on Industrial Informatics, 2021. 17 (3): p. 1721-1731. Ferraguti, F., et al., Augmented reality based approach for on-line quality assessment of polished surfaces. Robotics and Computer-Integrated Manufacturing, 2019. 59 : p. 158-167. Mourtzis, D., V. Zogopoulos, and F. Xanthi, Augmented reality application to support the assembly of highly customized products and to adapt to production re-scheduling. The International Journal of Advanced Manufacturing Technology, 2019. 105 (9): p. 3899-3910. Danielsson, O., et al., Operators perspective on augmented reality as a support tool in engine assembly. Procedia CIRP, 2018. 72 : p. 45-50. Lampen, E., et al., Combining Simulation and Augmented Reality Methods for Enhanced Worker Assistance in Manual Assembly. Procedia CIRP, 2019. 81 : p. 588-593. Henderson, S. and S. Feiner, Exploring the Benefits of Augmented Reality Documentation for Maintenance and Repair. IEEE Transactions on Visualization and Computer Graphics, 2011. 17 (10): p. 1355-1368. Qeshmy, D.E., et al., Managing Human Errors: Augmented Reality systems as a tool in the quality journey. Procedia Manufacturing, 2019. 28 : p. 24-30. Schlagowski, R., L. Merkel, and C. Meitinger. Design of an assistant system for industrial maintenance tasks and implementation of a prototype using augmented reality . in 2017 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM) . 2017. IEEE. Zhu, J., S.K. Ong, and A.Y.C. Nee, A context-aware augmented reality system to assist the maintenance operators. International Journal on Interactive Design and Manufacturing (IJIDeM), 2014. 8 (4): p. 293-304. Liu, C., et al., Augmented Reality-assisted Intelligent Window for Cyber-Physical Machine Tools. Journal of Manufacturing Systems, 2017. 44 : p. 280-286. Wang, Y., et al., Mechanical assembly assistance using marker-less augmented reality system. Assembly Automation, 2018. 38 (1): p. 77-87. Loch, F., F. Quint, and I. Brishtel. Comparing Video and Augmented Reality Assistance in Manual Assembly . in 2016 12th International Conference on Intelligent Environments (IE) . 2016. IEEE. Radkowski, R., J. Herrema, and J. Oliver, Augmented Reality-Based Manual Assembly Support With Visual Features for Different Degrees of Difficulty. International Journal of Human-Computer Interaction, 2015. 31 (5): p. 337-349. Festo. CP Lab | Festo USA . 2022; Available from: https://www.festo.com/us/en/e/technical-education/learning-systems/factory-automation-and-industry-4-0/learning-factories/cp-systems-large-scale-industry-4-0-learning-factories/cp-lab-id_36133/. UR3e Technical Details . 2020: @www_universal-robots_com. p. https://www.universal-robots.com/media/1802780/ur3e-32528_ur_technical_details_.pdf Lowe, D.G. Object recognition from local scale-invariant features . in Proceedings of the Seventh IEEE International Conference on Computer Vision . 1999. IEEE. Muja, M. and D.G. Lowe, Fast approximate nearest neighbors with automatic algorithm configuration. VISAPP (1), 2009. 2 (331-340): p. 2. 1 in 2 visitors abandon a website that takes more than 6 seconds to load - Digital.com . 2022; Available from: https://digital.com/1-in-2-visitors-abandon-a-website-that-takes-more-than-6-seconds-to-load/. Supplementary Files LeftRightSensorAlignment.mov LeftSensorAlignmentDoubleCheck.mov TestingData.xlsx Cite Share Download PDF Status: Published Journal Publication published 06 Feb, 2023 Read the published version in The International Journal of Advanced Manufacturing Technology → Version 1 posted Reviewers agreed at journal 19 Jul, 2022 Reviewers invited by journal 19 Jul, 2022 Editor assigned by journal 18 Jul, 2022 First submitted to journal 15 Jul, 2022 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-1842846","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":122319186,"identity":"dc5405d9-1ffe-4f10-90a4-08303d1976d4","order_by":0,"name":"James T Frandsen","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABA0lEQVRIiWNgGAWjYBACxgYwJQciDIDYBiTWeICwlgRjmJY0sBheLRCA0HIYzMerhbn9dOLDnz8M8vlnN2/+8HHHebu17YeBttTYRON0WE/uZmOeBAPLGXeOlUnOPHM7eduZRKCWY2m5DTj9krtNmiHhjwHDjRwzZt6228lmB4BaGBsO49bS/3b7zx8JBgbyN3KMP/9tO5dsdv4hAS0zcrcxAB1mYHAjx0Case2AndkNQrbMeLtZmifNwMDwRlqZZG9bcoLZDaAtCXj8Ytifu/HjDxsDA7kbyZs//Gyzszc7n/7wwYcaG9xa0CUSwQIJOJSDgDy6gD0exaNgFIyCUTBCAQAxaWdojxj9dQAAAABJRU5ErkJggg==","orcid":"","institution":"Brigham Young University","correspondingAuthor":true,"prefix":"","firstName":"James","middleName":"T","lastName":"Frandsen","suffix":""},{"id":122319187,"identity":"79a0f10b-fde4-4fce-aa20-cdfaf24afffe","order_by":1,"name":"Joe Tenny","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Joe","middleName":"","lastName":"Tenny","suffix":""},{"id":122319188,"identity":"6e57d599-4b04-4b67-8091-47eab5d6253b","order_by":2,"name":"Walter Frandsen","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Walter","middleName":"","lastName":"Frandsen","suffix":""},{"id":122319189,"identity":"fd11e6dc-7af4-4690-982d-43a72ebcf39a","order_by":3,"name":"Yuri Hovanski","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Yuri","middleName":"","lastName":"Hovanski","suffix":""}],"badges":[],"createdAt":"2022-07-10 04:44:43","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-1842846/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-1842846/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s00170-023-10978-1","type":"published","date":"2023-02-06T18:43:13+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":24365555,"identity":"cc68d6ec-e5ca-49b6-85df-9ff753d47b0a","added_by":"auto","created_at":"2022-07-26 18:58:52","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":2085380,"visible":true,"origin":"","legend":"\u003cp\u003eThe Festo CP Lab 404 (left) and the iDrilling station application module (right) which will be used to test the proposed system. A close-up view of the three optical sensors is shown (bottom).\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/57b36ce8f2a2f29a375210a3.png"},{"id":24365549,"identity":"530b7970-2daa-4a7a-aa0f-61d344aeef23","added_by":"auto","created_at":"2022-07-26 18:58:52","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":83918,"visible":true,"origin":"","legend":"\u003cp\u003eSoftware architecture of the proposed system showing how data flows between each component\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/34342d61ad66efc7ba43a0b2.png"},{"id":24365552,"identity":"f2b5c378-0449-430b-96bf-4ce4255589a0","added_by":"auto","created_at":"2022-07-26 18:58:52","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":1005563,"visible":true,"origin":"","legend":"\u003cp\u003eAn iPad running an AR positioning experience was mounted on a UR3e to quantify the AR system’s positional accuracy.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/fc7e608cf9f6aba35455a00a.png"},{"id":24368307,"identity":"53e9488d-0cbb-4441-a714-b787ac81ba03","added_by":"auto","created_at":"2022-07-26 19:18:52","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":1583336,"visible":true,"origin":"","legend":"\u003cp\u003eThe navigational helpers, including the feet marker, hand/device model, and 3D path are shown in front of the iDrilling station of the Festo CP Lab.\u0026nbsp;\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/b39c6b1de258ce0401f19a0c.png"},{"id":24365997,"identity":"cbf0814e-1305-45e0-8a0c-207a6348de0e","added_by":"auto","created_at":"2022-07-26 19:03:52","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":69814,"visible":true,"origin":"","legend":"\u003cp\u003eThe general flow of operations that take place for the SIFT + FLANN image processing approach.\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/013bb07e650c69f2d05e0d9f.png"},{"id":24365995,"identity":"4aabe40f-7086-43d1-abb1-a00fec06187f","added_by":"auto","created_at":"2022-07-26 19:03:52","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":57243,"visible":true,"origin":"","legend":"\u003cp\u003eThe general flow of operations that take place for the template matching image processing approach.\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/7005d89b5cfcd7e2377ccca6.png"},{"id":24367004,"identity":"01829d0c-6adf-4356-8f21-95f6437b9448","added_by":"auto","created_at":"2022-07-26 19:08:52","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":468081,"visible":true,"origin":"","legend":"\u003cp\u003eDistribution of the 45 measured points relative to the iDrilling station.\u003c/p\u003e","description":"","filename":"7.png","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/b0d299dd2f10aabc6cc8a057.png"},{"id":24365999,"identity":"eb17259d-6a12-42b8-9ea6-97c664faa285","added_by":"auto","created_at":"2022-07-26 19:03:52","extension":"png","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":1640225,"visible":true,"origin":"","legend":"\u003cp\u003eA sequence of images depicting how the alignment arrows integrated with the original navigational helpers (subfigure a) to facilitate fine device alignment. As the user approached the navigational helpers, the device/hands model faded away and the alignment arrows appeared (subfigure b). The user then aligned the arrow pinned in front of their device (right arrow in subfigure c) with the arrow at the image capture position (left arrow in subfigure c) until fine alignment was achieved (subfigure d).\u003c/p\u003e","description":"","filename":"8.png","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/5de293ce2da295505d4b532a.png"},{"id":24366001,"identity":"c1931eb6-ca2f-40ab-89bf-5c1ffec2155a","added_by":"auto","created_at":"2022-07-26 19:03:52","extension":"png","order_by":9,"title":"Figure 9","display":"","copyAsset":false,"role":"figure","size":83905,"visible":true,"origin":"","legend":"\u003cp\u003eRequired times to achieve proper alignment with the original and improved navigational helpers.\u003c/p\u003e\u003cp\u003e\u003cbr\u003e\u003c/p\u003e","description":"","filename":"9.png","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/742f41c4bace54d38cadbd8f.png"},{"id":24367616,"identity":"f70fc358-743f-48ec-b0a5-9750eb573fd6","added_by":"auto","created_at":"2022-07-26 19:13:52","extension":"png","order_by":10,"title":"Figure 10","display":"","copyAsset":false,"role":"figure","size":64447,"visible":true,"origin":"","legend":"\u003cp\u003eThe measurement error for each image processing approach, with and without distortion correction.\u003c/p\u003e","description":"","filename":"10.png","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/d1a2029bab4d179d245ae8d7.png"},{"id":24365559,"identity":"bff8c0a4-82f6-425e-b357-263d83aef138","added_by":"auto","created_at":"2022-07-26 18:58:52","extension":"png","order_by":11,"title":"Figure 11","display":"","copyAsset":false,"role":"figure","size":2173170,"visible":true,"origin":"","legend":"\u003cp\u003eA sequence of images showing the results summary popup indicating a misaligned sensor (subfigure a), screenshots of the realignment animation sequence (subfigures b-e), and the result summary popup after reinspection without any detected errors (subfigure f).\u003c/p\u003e","description":"","filename":"11.png","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/eab10a99814188d40b4376a5.png"},{"id":24367002,"identity":"2536c87f-6755-400e-8e80-c56c24f8fc5e","added_by":"auto","created_at":"2022-07-26 19:08:52","extension":"png","order_by":12,"title":"Figure 12","display":"","copyAsset":false,"role":"figure","size":121327,"visible":true,"origin":"","legend":"\u003cp\u003eComparison of the duration of the four components of processing at as image resolution is changed\u003c/p\u003e\u003cp\u003e\u003cbr\u003e\u003c/p\u003e","description":"","filename":"12.png","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/34dc8fd4eeb68419ac534985.png"},{"id":24367618,"identity":"ec1b78a6-ad84-46c1-9894-9bb29e868ecc","added_by":"auto","created_at":"2022-07-26 19:13:52","extension":"png","order_by":13,"title":"Figure 13","display":"","copyAsset":false,"role":"figure","size":125577,"visible":true,"origin":"","legend":"\u003cp\u003eComparison of measurement accuracy and overall speed as image resolution is changed\u003c/p\u003e\u003cp\u003e\u003cbr\u003e\u003c/p\u003e","description":"","filename":"13.png","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/f0f5bd8ba79a568fb5b7b943.png"},{"id":44718848,"identity":"4aced770-20ba-459c-af89-318a54db0f56","added_by":"auto","created_at":"2023-10-16 18:51:40","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":11141367,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/107c829a-4460-4254-9577-98ef6d8355fe.pdf"},{"id":24365564,"identity":"5b4f6304-23b0-47c1-95b0-04f7091653bb","added_by":"auto","created_at":"2022-07-26 18:59:05","extension":"mov","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":301431780,"visible":true,"origin":"","legend":"","description":"","filename":"LeftRightSensorAlignment.mov","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/69915c7dbcd7523000b63e67.mov"},{"id":24365572,"identity":"c2af0d1d-62a3-4ab5-ad83-4e0f1faa2e86","added_by":"auto","created_at":"2022-07-26 18:59:06","extension":"mov","order_by":5,"title":"","display":"","copyAsset":false,"role":"supplement","size":324590876,"visible":true,"origin":"","legend":"","description":"","filename":"LeftSensorAlignmentDoubleCheck.mov","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/450c925de411c3d967f9ec3b.mov"},{"id":24367006,"identity":"0339b413-342c-4b32-8192-8088ad7dc7ee","added_by":"auto","created_at":"2022-07-26 19:08:52","extension":"xlsx","order_by":6,"title":"","display":"","copyAsset":false,"role":"supplement","size":138854,"visible":true,"origin":"","legend":"","description":"","filename":"TestingData.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-1842846/v1/22f7126a3c5925417e52f550.xlsx"}],"financialInterests":"","formattedTitle":"An Augmented Reality Maintenance Assistant with Real-time Quality Inspection on Handheld Mobile Devices","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eIndustry 4.0 is a modern industrial trend sweeping across the manufacturing sector, and is characterized by a transition to smarter, more connected factories. The primary goal of this transformation is to leverage rapid data exchange and unified connectivity to achieve greater cohesion across the factory as a unit. Often synonymous with Industry 4.0, the term \u0026ldquo;industrial internet of things\u0026rdquo; (IIoT) is commonly used to describe the way machines, devices, sensors, and information technology (IT) systems should be connected and work together to create a coordinated and pseudo-living production system, or a \u0026ldquo;smart\u0026rdquo; factory. Regardless of the terminology one prefers, it is clear that as factories continue to become smarter it is becoming more important than ever to develop tools that keep the human workers integrated as part of the autonomous manufacturing enterprise.\u003c/p\u003e \u003cp\u003eAccording to Jim Heppelmann, president and CEO of PTC, augmented reality (AR) is a novel human-machine interface that \u0026ldquo;turns bits and bytes into sounds and sights\u0026hellip;\u0026rdquo; [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. In other words, AR can translate the digital conversations taking place between machines and systems throughout the factory into a visual language that people can easily understand. The translated conversations can then be relayed to workers across the factory in a way that is contextually relevant to the tasks they are performing and allows them to stay tuned into the dialog of the smart factory.\u003c/p\u003e \u003cp\u003eAR has boundless applications in a factory setting and is currently being deployed across many manufacturing operations. Ho, P. T., et al. (2022), in a recent systematic literature review, surveyed publications around industrial AR and found that AR is most commonly utilized for assembly/disassembly and maintenance operations [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. AR is particularly advantageous for assembly/disassembly and maintenance operations because of its ability to deliver work instructions that are both spatially aligned and visually intuitive. It allows instructions to be translated from simplistic pictures in bulky paper manuals into dynamic visual sequences and 3D representations that directly overlay the work area. AR-based instructions have been shown to significantly lower workers\u0026rsquo; mental load, which ultimately helps reduce the number of mistakes they make, by up to 82% [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. This is largely because the worker can keep their focus on the workpiece, rather than constantly switching their attention back-and-forth between the workpiece and the paper instruction manual.\u003c/p\u003e \u003cp\u003eResearch is currently underway to find ways to further mitigate the number of errors that workers may make while being assisted by AR. The simplest of options is to require workers to manually check their workpiece, then indicate to the AR application whether they observed any errors. This type of AR experience has been described in literature for inspection scenarios [\u003cspan additionalcitationids=\"CR6 CR7 CR8 CR9 CR10 CR11 CR12 CR13\" citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e], assembly scenarios [\u003cspan additionalcitationids=\"CR16 CR17\" citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e], and even in construction settings [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. These AR experiences begin to show the benefits of quality-infused AR. Still though, they necessarily rely on the experience and skill of each worker to be able to manually determine how well each step of the procedure was performed. The next step forward, then, is to develop systems that can make these quality determinations automatically and independent of worker perspective and experience.\u003c/p\u003e \u003cp\u003eA system that is capable of visually inspecting the workpiece to find errors is advantageous over the systems described above because it would not be susceptible to operator error. An AR system that could provide objective process validation would be similar to existing automated quality assurance systems, such as an automated optical inspection system (AOI) for PCB boards, except it could make quality assessments at every step of the process and would also be responsible for providing the worker with the instructions necessary to complete the task. Ojer, M., et al. (2020) developed an AR system for PCB assembly that visually guides operators through the component mounting process while simultaneously checking the correctness of each action. This system is designed to display the AR instructions using either a monitor or a projector and can automatically validate each component\u0026rsquo;s physical characteristics, position, and rotation to detect, in real time, whether the components on the PCB have been mounted correctly [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. This AR-assisted assembly operation takes place at a stationary workstation, so the inspection camera is calibrated for and fixed relative to the work area. Other AR systems demonstrating similar features have also been documented [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan additionalcitationids=\"CR24 CR25 CR26 CR27 CR28 CR29 CR30 CR31 CR32\" citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e]. These AR systems are enabled by machine validation from a fixed camera, which is perfectly suited for tasks that can be completed at a workstation, such as assembly/disassembly. However, a fixed-camera setup is not feasible for tasks that must be completed at various locations across the factory floor (e.g., maintenance on large machinery). It would be much more desirable in these situations to have an AR system that is completely mobile and can be taken directly to the location of the procedure as the maintenance is being performed.\u003c/p\u003e \u003cp\u003eAR systems built for mobile devices, such as handheld devices (HHDs) or head-mounted displays (HMDs), are commonly used to display maintenance instructions out on the factory floor because they utilize the camera native to the device to achieve true mobility. Infusing truly mobile AR systems with automated quality inspection has received relatively little attention in the literature because of unique challenges that are introduced when the inspection camera and illumination is no longer fixed relative to the work area. A few mobile AR systems with automatic quality inspection have been reported, but each requires that fiducial markers be placed on all components [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e, \u003cspan additionalcitationids=\"CR35\" citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e], which would hinder its deployment in an industrial setting. Fiducial markers are commonly used in AR to improve scene recognition and are particularly desirable for industrial scenarios since equipment surfaces usually lack adequate texture to reliably establish tracking using some tracking methods. However, the use of fiducial markers is impractical in industrial environments due to harsh conditions that could damage the markers and because it requires preparation of the workspace in advance [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e, \u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eOnce the machine validation has been performed and the results are returned, specific instructions should be given to the worker to assist them with fixing any errors the system detected. These instructions should be clear enough to indicate to the worker exactly what to do to make the necessary repairs before proceeding to the next step. By rendering clear instructions, the validation system relieves any ambiguity related to the detected error and will help all workers move through a procedure more quickly. This level of feedback would transform the system from an AR-enabled error detection system into an AR-enabled maintenance assistant. A few articles have reported incorporating this kind of feedback, but each was deficient in at least one of the characteristics mentioned previously [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e, \u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e, \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e, \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eAs has been shown, there are no publications to-date that demonstrate a real-time AR quality inspection system that is suitable for deployment in mobile industrial scenarios. An extensive review of relevant articles has been performed to verify this and the results can be seen in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. Therefore, this study will seek to understand the limitations of in-process quality inspection as it pertains to mobile AR and determine what specific tools need to be developed to enable a real-time maintenance assistant for mobile AR experiences.\u003c/p\u003e \u003cp\u003eThis paper is organized as follows: Section 2 gives a detailed description of the environment for which the proposed system was developed, including the tests used to assess the system\u0026rsquo;s performance. Section 3 presents the outcome of the system development process and discusses obstacles encountered during the development process. Finally, Section 4 draws conclusions based on the data presented and makes suggestions for future work.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eA detailed summary of relevant AR articles showing a hole in the knowledge base.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"10\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eMobility\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"4\" nameend=\"c5\" namest=\"c2\"\u003e \u003cp\u003eDisplay Type\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"4\" nameend=\"c9\" namest=\"c6\"\u003e \u003cp\u003eEssential Features\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eRelevant Articles\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eHHD\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eHMD\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cem\u003eMonitor\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eProjector\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cem\u003eValidation with a Mobile Camera\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cem\u003eAutomatic Machine Vision\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003eValidation\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e\u003cem\u003eMarkerless Tracking\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e\u003cem\u003eDynamically Creates Repair Instructions\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"13\" rowspan=\"14\"\u003e \u003cp\u003eMobile\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e, \u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e, \u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan additionalcitationids=\"CR6 CR7 CR8\" citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan additionalcitationids=\"CR41 CR42 CR43 CR44\" citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e45\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e46\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e, \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e, \u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e47\u003c/span\u003e, \u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e48\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e, \u003cspan citationid=\"CR49\" class=\"CitationRef\"\u003e49\u003c/span\u003e, \u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan additionalcitationids=\"CR16\" citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e, \u003cspan additionalcitationids=\"CR52 CR53\" citationid=\"CR51\" class=\"CitationRef\"\u003e51\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR54\" class=\"CitationRef\"\u003e54\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"7\" rowspan=\"8\"\u003e \u003cp\u003eStationary\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan additionalcitationids=\"CR25 CR26 CR27\" citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e, \u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e, \u003cspan citationid=\"CR55\" class=\"CitationRef\"\u003e55\u003c/span\u003e, \u003cspan citationid=\"CR56\" class=\"CitationRef\"\u003e56\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e, \u003cspan citationid=\"CR57\" class=\"CitationRef\"\u003e57\u003c/span\u003e, \u003cspan citationid=\"CR58\" class=\"CitationRef\"\u003e58\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003ex\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c10\"\u003e \u003cp\u003e[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e"},{"header":"2. Methodology","content":"\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003eThe primary goal of the proposed system is to create a mobile AR assistant that empowers workers in performing complex maintenance tasks. One of the primary barriers to developing such a system is the ability to successfully leverage machine vision to analyze images captured with a mobile device during an AR-guided procedure. To achieve this capability, several different components of the AR system and image processing algorithm must be developed and assessed:\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003e\u003cspan\u003e\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e1. The positional accuracy of the underlying AR tracking system to deduce the position of the user\u0026rsquo;s device within the AR space. This functionality is critical for helping users navigate around the physical equipment to capture an image for inspection.\u003c/p\u003e\u003cspan\u003e\n \u003cp\u003e2. The user\u0026rsquo;s ability to use the graphical navigational helpers shown in the AR experience to precisely capture a high-quality image for inspection.\u003c/p\u003e\n\u003c/span\u003e\u003cspan\u003e\n \u003cp\u003e3. An image processing algorithm that is robust enough to deal with the image variation inherent to a mobile-based system, while still producing accurate measurements. This program must also be able to analyze images of industrial hardware, which is notoriously texture-less and colorless, and reliably return accurate results.\u003c/p\u003e\n\u003c/span\u003e\u003cspan\u003e\n \u003cp\u003e4. The overall speed of the proposed system to transmit the captured image to the image processing algorithm, perform the automated image analysis, and receive the inspection results back into the AR experience.\u003c/p\u003e\n\u003c/span\u003e\n\u003cp\u003e\u003c/p\u003e\n\u003cp\u003eThe environment in which the proposed system was developed will be described hereafter, as well as the testing that was performed to evaluate it.\u003c/p\u003e\n\u003cdiv class=\"Section2\" id=\"Sec3\"\u003e\n \u003ch2\u003e2.1. Setup\u003c/h2\u003e\n \u003cdiv class=\"Section3\" id=\"Sec4\"\u003e\n \u003ch2\u003e2.1.1. Hardware\u003c/h2\u003e\n \u003cp\u003eThe use case devised to evaluate the proposed system was based on the Festo Cyber-Physical Lab (CP Lab). The CP Lab is a modular Industry 4.0 learning system that is intended to convey an in-depth understanding of Industry 4.0 concepts. It includes many interchangeable application modules that perform specific functions and are representative of operations that might occur in an actual manufacturing facility. The CP Lab was constructed using industrial hardware to make the learning experience as similar to reality as possible. More detail about the CP Lab\u0026rsquo;s hardware can be found on Festo\u0026rsquo;s webpage for the CP Lab [\u003cspan class=\"CitationRef\"\u003e59\u003c/span\u003e]. The development for this project was done with a four-station configuration of the CP Lab (CP Lab 404), and specifically used the iDrilling application module (Fig.\u0026nbsp;1).\u003c/p\u003e\n \u003cp\u003eFor this study, the proposed system was used to check the alignment of the through-beam sensors on the CP Lab\u0026rsquo;s iDrilling station. There are three sensors that use fiber-optic heads to transmit and receive beams of visible infrared light in order to detect the presence and placement of a workpiece. If these sensors become misaligned, the iDrilling station ceases to function because it cannot determine the state of each workpiece coming through. Realigning the three sensors is a time-consuming process that entails guessing and checking the position of each sensor until a satisfactory alignment is attained. Thus, the proposed system was adapted for this application to help workers ensure the correct alignment of each optical sensor and render specific realignment instructions as necessary.\u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv class=\"Section3\" id=\"Sec5\"\u003e\n \u003ch2\u003e2.1.2. Software\u003c/h2\u003e\n \u003cp\u003eThe authoring environment used for the AR component of the proposed system was PTC\u0026rsquo;s Vuforia Studio. Vuforia Studio is a powerful, user-friendly tool that enables designers, engineers, and technicians to easily create scalable AR experiences without extensive programming knowledge. It integrates the robust computer vision capabilities of Vuforia Engine with real-time data from business systems and IIoT to create AR experiences that are both spatially and contextually relevant. These experiences can be created for both HHDs and HMDs and can be easily accessed across a company through a mobile application. Additionally, Vuforia Studio\u0026rsquo;s user-friendly interface allowed a focus on the industrial deployment of the proposed system, rather than strictly on its software development.\u003c/p\u003e\n \u003cp\u003eThe image processing portion of this project was developed in a Python environment and primarily utilized the OpenCV library for computer vision. Python was selected for the project because it is a versatile programming language that can be used in a variety of applications. Furthermore, the Python OpenCV library is more complete than the OpenCV library in other programming languages which provided a wider range of functions to choose from. The OpenCV library was chosen over other image processing libraries because of its open-source nature, its extensive online support, and its ability to analyze images based solely on a single template image. An alternative approach requires building a training dataset from hundreds or thousands of images. Manually capturing enough images to build a robust classifier is very labor-intensive and time-consuming. It is not feasible in a manufacturing setting when collecting the images could require the machinery being maintained to be shut down and production halted for safety reasons. In such a scenario, it is much more desirable to capture one single image instead of hundreds or thousands. Therefore, the proposed system relied on just three template regions taken from one complete template image. Two of the template regions contained mechanical features separated by a known distance which were used for image calibration, and the third template region contained one of the sensors whose distance would be measured. Each image containing the template region was used to identify key features of the captured inspection images so that the necessary measurements could be taken. It should be noted that the template images were all chosen to be in the same plane in order to minimize the effects of parallax.\u003c/p\u003e\n \u003cp\u003eAs noted previously, markerless tracking methods are much better suited for industrial environments than their fiducial-based counterparts. In terms of markerless tracking, Vuforia Studio has the ability to use 3D mesh scans or CAD-based assets to recognize physical objects. Establishing tracking with 3D scans requires a scan of the area/equipment to be taken before the AR experience can recognize the space and perform any rendering. These scans provide robust tracking as features throughout the entire area can be used for reference, but do not perform well if the locale changes (e.g., using one AR experience for multiple pieces of equipment across a factory, the equipment is relocated, or the area around the equipment is significantly modified). Considering these factors and in order to provide a versatile AR experience for maintenance procedures, a CAD-based model target tracking method was selected. This allowed the AR experience to recognize the equipment with no reference to the surrounding region.\u003c/p\u003e\n \u003cp\u003eThe overall software architecture for the proposed system consists of a few applications and servers that are used to provide AR-based guidance, transfer/store the captured images, and process the captured image to check for quality. Figure \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e gives a graphical depiction of the relationship between all entities within the proposed system. The relationships will also be described hereafter, and it should be noted that all servers and programs were hosted on-prem at Brigham Young University. Workers used the Vuforia View app to interact with the AR elements of the system. The AR experience was responsible for guiding them through the maintenance task and did so with graphical and text-based instructions at each step. Once workers were ready for their work to be inspected, they triggered the inspection in the system and followed the instructions for capturing a high-quality image. After an image was captured, it was sent to ThingWorx, an IIoT platform, where it was stored until it was retrieved by the Python image processing algorithm. ThingWorx was used as an intermediary between the AR system and the image processing algorithm to facilitate smoother eventual deployment of the proposed system in a real manufacturing facility. The ThingWorx platform has native capabilities to connect to enterprise systems, such as CMMS, MES, or ERP, which could allow the proposed system to be integrated with any company\u0026rsquo;s existing IT platforms. Once retrieved by the image processing algorithm, the OpenCV algorithms were used to extract the data necessary to perform measurements. Then, the measurements and other pertinent data were passed back to the AR system, via ThingWorx, and instructions were automatically generated to guide the user through repositioning any of the optical sensors that were detected to be out of alignment.\u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv class=\"Section3\" id=\"Sec6\"\u003e\n \u003ch2\u003e2.1.3. Devices\u003c/h2\u003e\n \u003cp\u003eThe augmented reality component of the proposed system was designed to be deployed on HHDs, such as a tablet. HHDs were selected for this project because they offer a few unique advantages over other device types. One reason HHDs were selected is that they are a relatively inexpensive and off-the-shelf consumer product, which enables the system to be used by many workers. This makes them very attractive for companies deploying AR solutions since multiple AR-capable devices must be purchased. Another reason is that HHDs have been widely accepted in society so the majority of people are very familiar with them and can already operate them effectively [\u003cspan class=\"CitationRef\"\u003e7\u003c/span\u003e]. There is typically a learning curve when using other AR-capable devices, such as HMDs, due to the unfamiliar and less-intuitive user interface. Additionally, HHDs have both a camera and a display screen which creates a fully functional AR system that can easily be taken anywhere in a factory. Another reason HHDs were selected is that they were the only classification of AR devices approved for enterprise-level use in our industrial sponsor\u0026rsquo;s manufacturing facilities.\u003c/p\u003e\n \u003cp\u003eThe proposed system was evaluated using two different HHDs to demonstrate device agnosticism. Specifically, the two HHDs used for testing were an Apple iPad Pro 12.9\u0026quot; and an Apple iPad 7th Gen. Once the inspection image was captured, the inspection image was transmitted to a laptop computer where the image analysis occurred. A Dell Inspiron 13-7378 laptop with a 6th Generation Intel Dual-Core i5-7200 CPU was used to process the captured images. The image processing was allocated to an external device after preliminary testing revealed the processor on HHDs are typically underpowered for rapid processing of high-resolution images while simultaneously running the AR experience. Running both the AR experience and the image process on the HHD resulted in significant lagging during the AR experience which was undesirable.\u003c/p\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv class=\"Section2\" id=\"Sec7\"\u003e\n \u003ch2\u003e2.2. Testing \u0026amp; Evaluation\u003c/h2\u003e\n \u003cdiv class=\"Section3\" id=\"Sec8\"\u003e\n \u003ch2\u003e2.2.1. Test #1: AR Positional Accuracy \u0026ndash; Method\u003c/h2\u003e\n \u003cp\u003eThe purpose of the first test was to determine the AR system\u0026rsquo;s ability to accurately locate itself relative to the iDrilling station. As stated previously, the tracking for this project was established with a CAD-based model target of the iDrilling station, so all positioning within the AR space happened in relation to the station itself. Once the underlying AR system could recognize the iDrilling station and establish tracking, Vuforia\u0026rsquo;s Device Pose Observer (DPO) began to compute the position and orientation of the user\u0026rsquo;s device within the AR space. The DPO analyzed both the live camera stream and data from the device\u0026rsquo;s sensors to continually estimate the device\u0026rsquo;s position, even when the reference object for tracking moved out of the camera\u0026rsquo;s view.\u003c/p\u003e\n \u003cp\u003eThe proposed system leveraged the DPO\u0026rsquo;s positional data to monitor the device\u0026rsquo;s position relative to the image capture position, which allowed it to automatically capture an inspection image when the device was properly aligned with the system\u0026rsquo;s navigational helpers. The device\u0026rsquo;s ability to accurately determine its position was critical to capturing good images for inspection, so testing was performed to quantify the DPO\u0026rsquo;s position estimation capabilities. An iPad running an AR positioning experience was mounted on a Universal Robots UR3e collaborative robot (Fig.\u0026nbsp;3), which was programmed to move between four waypoints. The device\u0026rsquo;s X, Y, Z coordinates, as estimated by the DPO, were recorded at three of the four positions. These three positions were at the image capture position and laterally offset 30mm\u0026thinsp;\u0026plusmn;\u0026thinsp;0.3mm to either side. The fourth position was significantly further away from the other three and was included to simulate the large device movements that typically occur during mobile AR experiences. The iPad repeated this program fifteen times during testing, so a total of 45 data points were collected. The UR3e\u0026rsquo;s pose repeatability was \u0026plusmn;\u0026thinsp;0.3mm [\u003cspan class=\"CitationRef\"\u003e60\u003c/span\u003e].\u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv class=\"Section3\" id=\"Sec9\"\u003e\n \u003ch2\u003e2.2.2. Test #2: Device Alignment Time\u0026ndash; Method\u003c/h2\u003e\n \u003cp\u003eThe second set of tests sought to quantify a user\u0026rsquo;s ability to position their device using the navigational helpers to capture a repeatable inspection image. As stated previously, the proposed system captured images automatically when it determined that the user had properly aligned their device with the image capture position. A tolerance was defined around the image capture position to create the range of acceptable alignments, which made properly aligning their device more feasible for users.\u003c/p\u003e\n \u003cp\u003eThe navigational helpers, as shown in Fig. \u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e below, consisted of a feet marker to show users where to stand, a 3D model depicting hands holding a tablet/phone to show users where to position their device, and a dynamic 3D path to lead users to the image capture position from a distance. Each of these components selectively faded as users approached the image capture position to ensure users could see the work area as the inspection image was captured and to increase user-friendliness. The navigational helpers were intended to intuitively guide users through the AR space in order to obtain the viewpoint from which the inspection image should be captured. These navigational AR \u0026ldquo;helpers\u0026rdquo; harnessed the power of AR to enhance human cognition and give users a more precise and complete spatial understanding of the work area. By guiding users to a specific point of view, the proposed system ensured that consistent images of the workpiece could be captured, which was requisite for attaining reliable results during image processing. If users could not capture a repeatable image, it would have been impossible for consistent results to be attained, since metrology algorithms depend on high-quality images to produce high-quality results.\u003c/p\u003e\n \u003cp\u003eThis component of the AR experience was evaluated by defining the image capture position and an associated tolerance around it, then recording the time a user needed to properly align their device within the defined tolerance, using the navigational helpers. While acknowledging that the alignment process will not always begin from the same position, it is still important to assess the tolerances relative to each other as a means of quantifying their usability. The testing began with an initial tolerance of \u0026plusmn;\u0026thinsp;50mm around the ideal image position and incrementally decreased to \u0026plusmn;\u0026thinsp;1mm. Three practice alignments were performed before data collection began at each tolerance, then ten consecutive alignments were performed and timed. As the alignment tolerance was gradually decremented, it was expected that the time required for users to attain a proper alignment would increase to some degree. Since worker efficiency is paramount in manufacturing settings, it was necessary to determine an optimal balance between the positional precision required from users and the time needed to achieve a proper alignment.\u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv class=\"Section3\" id=\"Sec10\"\u003e\n \u003ch2\u003e2.2.3. Test #3: Measurement Accuracy \u0026ndash; Method\u003c/h2\u003e\n \u003cp\u003eThe third test aimed to compare the measurement accuracies of two different image processing approaches. Each approach addressed the task of identifying the critical parts of captured images in a different manner. One method was intended to deal with significant variation in the images it would process, both in feature scale and rotation. The other method was better suited for a more controlled scenario in which the captured images could be more consistent. By comparing two methods, it was possible to assess which one was better suited for the proposed system. The comparison was performed using a set of 25 images captured at a resolution of 2732x2048 pixels.\u003c/p\u003e\n \u003cp\u003eThe first approach selected relied on extracting key features from the captured image and template images, determining matching keypoints between the images, then using the matched keypoints to take measurements within the captured image. Specifically, the scale-invariant feature transform (SIFT) algorithm [\u003cspan class=\"CitationRef\"\u003e61\u003c/span\u003e] was used to extract keypoints and descriptors from the captured image and each template image. Then, the fast library for approximate nearest neighbors (FLANN) algorithm [\u003cspan class=\"CitationRef\"\u003e62\u003c/span\u003e] compared the keypoints and descriptors from the captured image to the keypoints and descriptors of each template image in order to locate key features within the captured image (e.g., the three optical sensors). Finally, the distances between the key features could be calibrated and measured. A logic diagram illustrating this approach is shown in Fig. \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e.\u003c/p\u003e\n \u003cp\u003eThe other image processing approach depended on using each template image in its entirety, rather than individually extracted keypoints, to identify key regions within the captured image. It utilized OpenCV\u0026rsquo;s template matching algorithm using cross correlation to slide each template image over the captured image and calculate the similarity between the template and the covered area. Once the best matching regions were determined, the measurements across the captured image could be taken. A logic diagram illustrating this approach is shown in Fig. \u003cspan class=\"InternalRef\"\u003e6\u003c/span\u003e.\u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv class=\"Section3\" id=\"Sec11\"\u003e\n \u003ch2\u003e2.2.4. Test #4: Total Processing Time \u0026ndash; Method\u003c/h2\u003e\n \u003cp\u003eThis final set of tests was performed to determine the overall image processing time of the proposed system. It was necessary to ensure the automatic inspection was rapid enough as to not encumber the natural flow of the maintenance task or lose the user\u0026rsquo;s attention. Recent studies have shown that 50% of people will lose interest in and leave a website if it takes more than six seconds to load [\u003cspan class=\"CitationRef\"\u003e63\u003c/span\u003e]. While acknowledging that an AR experience is not necessarily equivalent to a website, this time limit of six seconds was adopted as a standard for the proposed system to ensure users stay engaged with the task at hand.\u003c/p\u003e\n \u003cp\u003eIt was anticipated that one of the primary limiting factors for the total processing time would be the resolution of the captured image since this was by far the system\u0026rsquo;s largest data point. Therefore, to optimize the overall speed of the proposed system, the images for inspection should be captured with lowest possible resolution. However, as the resolution of the captured images was reduced, the images consequently lost pixel information which impacted the system\u0026rsquo;s ability to measure distances accurately. Therefore, a balance had to be found between using an image with high enough resolution to reliably the inspection yet low enough resolution to be transmitted and processed quickly.\u003c/p\u003e\n \u003cp\u003eFor the proposed system, the total processing time could be subdivided into four distinct components, as depicted in Fig. \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e: transmitting the captured image from the AR device to ThingWorx, retrieving the captured image and other vital data from ThingWorx, processing the captured image with the image processing algorithm, and returning the inspection results back to the AR device. Each component was evaluated by transferring or processing the associated data one hundred times at each image resolution and recording the time required for each round. In addition to the time trials, the accuracy of the automatic inspection at each image resolution had to be assessed. To accomplish this, 25 sample images of the iDrilling station were captured with the three optical sensors a set position. Starting with a full-size 2732x2048 image, the images were decimated by 12.5% while the error of the measurements from the sensors\u0026rsquo; known position were recorded. Thus, the overall processing times could then be compared to the measurement accuracy at each resolution to determine the optimal image resolution for the proposed system.\u003c/p\u003e\n \u003c/div\u003e\n\u003c/div\u003e"},{"header":"3. Results \u0026 Discussion","content":"\u003cdiv class=\"Section2\" id=\"Sec13\"\u003e\n \u003ch2\u003e3.1. Test #1: AR Positional Accuracy\u003c/h2\u003e\n \u003cp\u003eAfter performing the test described in 2.2.1, it was determined that the AR system\u0026rsquo;s DOP could estimate the HHD\u0026rsquo;s position with an accuracy of \u0026plusmn;\u0026thinsp;1.8mm. This result was obtained by first determining the expected positions for the three test positions. The expected positions were defined by determining the average X, Y, Z coordinates of the center position, then adding/subtracting 30mm to/from the average center x-value to determine the expected right and left positions. Then, the Euclidean distance from each of the 45 measured points to their expected positions was calculated, using Eq. 1. Next, the mean error of all the calculated distances was found. The mean error of the 45 positions was found to be 2.1mm. Therefore, once the positional tolerance of the UR3e was factored into this, the DOP\u0026rsquo;s positional capabilities can be said to be accurate, on average, to \u0026plusmn;\u0026thinsp;1.8mm. A 3D scatter plot of the 45 measured points can be seen in Fig. \u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e.\u003c/p\u003e\n \u003cdiv class=\"Equation\" id=\"Equa\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e$$d\\left(p,q\\right)=\\sqrt{\\sum _{i=1}^{n}{\\left({q}_{i}-{p}_{i}\\right)}^{2}}$$\u003c/div\u003e\n \u003c/div\u003e\n \u003cp\u003e\u003cem\u003eEquation 1\u003c/em\u003e\u003c/p\u003e\n \u003cp\u003ewhere:\u003c/p\u003e\n \u003cp\u003e\u003cem\u003ep\u003c/em\u003e, \u003cem\u003eq\u003c/em\u003e\u0026thinsp;=\u0026thinsp;two points in Euclidean n-space\u003c/p\u003e\n \u003cp\u003e\u003cem\u003ep\u003c/em\u003e \u003csub\u003e\u0026nbsp;\u003cem\u003ei\u003c/em\u003e\u0026nbsp;\u003c/sub\u003e, \u003cem\u003eq\u003c/em\u003e\u003csub\u003e\u003cem\u003ei\u003c/em\u003e\u003c/sub\u003e = Euclidean vectors, starting from the origin of the space\u003c/p\u003e\n \u003cp\u003e\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;n-space\u003c/p\u003e\n \u003cp\u003eIt is important to note that the datapoints displayed in Fig. \u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e are not to scale relative to the image of the iDrilling station included on the X-Y plane. The image of the iDrilling station was included to illustrate the coordinate system in which the data was collected. Each of the three axes are equal in length to render an accurate depiction of how the datapoints were distributed. It can be observed that the positional error did not vary significantly between the three test positions. In terms of the proposed quality inspection system, these results indicated that the user\u0026rsquo;s device could be reliably placed relative to the iDrilling station within a range of \u0026plusmn;\u0026thinsp;1.8mm. This discrepancy, combined with the tolerance around the image capture position determined in Test #2, caused variation in the real-world viewpoint from which inspection images were captured, which increased the difficulty of obtaining valid results from the image processing algorithm. An analysis of the proposed system\u0026rsquo;s ability to manage this variation was investigated in Test #3.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"Section2\" id=\"Sec14\"\u003e\n \u003ch2\u003e3.2. Test #2: Device Alignment Time\u003c/h2\u003e\n \u003cp\u003eAs described in 2.2.2, this test was performed by recording the time a user needed to properly align their device using the navigational helpers. The navigational helpers were intended to provide visual cues to help users quickly align their device so an inspection image could be automatically captured. A set of ten alignments were recorded for each tolerance and the averages of each set were then calculated. The testing revealed that as the tolerances were decreased, their associated alignment times increased exponentially. It should be noted that timing sets were not recorded for the \u0026plusmn;\u0026thinsp;2mm and \u0026plusmn;\u0026thinsp;1mm tolerances because the associated alignments became too difficult to feasibly achieve. Based on these results, an alignment tolerance of \u0026plusmn;\u0026thinsp;10mm about the image capture position was selected for the system because it was the best precision attainable before the average alignment began to increase significantly.\u003c/p\u003e\n \u003cp\u003e\u003cem\u003eFigure 8. A sequence of images depicting how the alignment arrows integrated with the original navigational helpers (subfigure a) to facilitate fine device alignment. As the user approached the navigational helpers, the device/hands model faded away and the alignment arrows appeared (subfigure b). The user then aligned the arrow pinned in front of their device (right arrow in subfigure c) with the arrow at the image capture position (left arrow in subfigure c) until fine alignment was achieved (subfigure d).\u003c/em\u003e\u003c/p\u003e\n \u003cp\u003eAs the proposed system\u0026rsquo;s image processing component was continually tested using images captured within this \u0026plusmn;10mm threshold, an abundance of feature identification errors were detected. It was discovered that these errors were due to the machine vision algorithm\u0026rsquo;s inability to adequately deal with the variation created by the alignment threshold. Such a large tolerance created too much variation across captured images, since the size and obliqueness of key measurement features could vary considerably. As it would not be feasible to decrease the alignment tolerance due to alignment time constraints, a pair of alignment arrows were added to the navigational helpers to facilitate fine alignment. These arrows were intended to help users visualize the exact image capture position and the position of their device relative to it. Once the user had used the original navigational helpers to approach the image capture position, an alignment arrow appeared pinned to the center of their screen, while the other arrow remained stationary at the image capture position as a point of reference. Each arrow had specific features intended to help users intuitively monitor their device\u0026rsquo;s rotational and translational alignment relative to the image capture position, shown as the reference arrow. It was anticipated that these new features would facilitate precise alignment and help users decrease the needed alignment times. The way the alignment arrows were used to achieve fine alignment is shown in in Figure 8.\u003c/p\u003e\n \u003cp\u003eWhen the tests across the same range of tolerances were repeated, the average alignment times all decreased significantly. As is shown in Fig. \u003cspan class=\"InternalRef\"\u003e9\u003c/span\u003e, the additional navigational tools helped users align their devices with the image capture positions much more quickly, even when the tolerances were very narrow. From these new results, an alignment tolerance of \u0026plusmn;\u0026thinsp;2mm about the image capture position was implemented for the proposed system. The chosen tolerance provided the optimal balance of alignment speed and precision since it could now be achieved, on average, in just 3.5 seconds. Successful testing of the proposed system\u0026rsquo;s image processing component with images captured within this new\u0026thinsp;\u0026plusmn;\u0026thinsp;2mm confirmed it as the right choice.\u003c/p\u003e\n \u003cp\u003eIt is suspected that the time required to achieve a proper alignment significantly increased for tolerances less than \u0026plusmn;\u0026thinsp;2mm due to the DPO\u0026rsquo;s positional capabilities. At a tolerance of \u0026plusmn;\u0026thinsp;1mm, the DPO struggled to distinguish whether or not the device was within \u0026plusmn;\u0026thinsp;1mm of the image capture position since it exceeded the DPO\u0026rsquo;s ability to accurately calculate the position of the user\u0026rsquo;s device. This caused proper alignments to become more difficult to achieve which produced the lengthier alignment times shown in the data. Thus, the addition of the pair of alignment arrows allowed users to easily position their device with such great precision that the positional capabilities of the underlying AR system became the limiting factor for alignments. With further development of the DPO and underlying AR system, even greater alignment precision could potentially be achieved to produce more consistent inspection images.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"Section2\" id=\"Sec15\"\u003e\n \u003ch2\u003e3.3. Test #3: Measurement Accuracy\u003c/h2\u003e\n \u003cp\u003eSince variation between inspection images had been minimized with the DPO as the limiting factor, it then became appropriate to begin to assess the measurement capabilities of the machine vision system. The measurement accuracies of the two image processing approaches were evaluated using methodology described in 2.2.3. Based on the measurements taken from the set of 25 test images, the average error of measurements by the SIFT\u0026thinsp;+\u0026thinsp;FLANN approach was calculated to be 1.62mm while the average error of the template matching approach was 1.21mm. The entire dataset is shown in Fig. \u003cspan class=\"InternalRef\"\u003e10\u003c/span\u003e. Testing that was performed previously revealed that the optical sensors themselves could tolerate variations in their positioning up to \u0026plusmn;\u0026thinsp;1.0mm. Because of the hardware\u0026rsquo;s physical limitations, it became clear that the measurement accuracy of both image processing approaches would be insufficient to help users reliably locate and reposition the optical sensors. Therefore, more work needed to be done to improve the system\u0026rsquo;s accuracy.\u003c/p\u003e\n \u003cp\u003eAn investigation into the captured images themselves revealed that they were being subjected to a degree of distortion from the mobile device\u0026rsquo;s camera. This distortion from the lens caused the sensors to appear in different positions than they actually were, which compromised the system\u0026rsquo;s ability to obtain accurate measurements. This issue was initially suspected since the inspection images were being captured by a consumer-grade camera, rather than an industrial-grade vision system. The lower quality camera lenses used in consumer mobile devices produce distorted images due to the relatively short focal lengths of their lenses, as well as other factors. These distortions can be imperceptible to the naked human eye, while still being significant enough to compromise a precise machine vision analysis and measurement. The issue was then confirmed by manually identifying the key features within the captured images and calculating the measurements by hand. Even without using one of the automated measurement approaches, the measurements were still incorrect which confirmed the images were indeed distorted.\u003c/p\u003e\n \u003cp\u003eTo overcome the lens-induced distortion, several images of a calibration board were captured so that the intrinsic parameters of the mobile device\u0026rsquo;s camera could be calculated. These parameters included a distortion matrix and coefficients which described the lens\u0026rsquo; optical center and focal length. Using these parameters, the set of 25 test images were corrected and the two image processing approaches were tested again. After analyzing the new results, it became clear that the images had in fact become more distorted since the average errors from both approaches increased.\u003c/p\u003e\n \u003cp\u003eFurther consideration of the distortion correction system revealed that the distortion matrix and coefficients being used to correct the captured images could not account for the mobile device\u0026rsquo;s auto-focus capabilities. The calculated intrinsic parameters could only be used for a single focal length which turned out to be the source of their inaccuracy since the mobile device could constantly change the focal length to best capture each image. The most straightforward solution would have been to fix the camera\u0026rsquo;s focal length within the AR experience, but this would have caused the AR renderings to be blurred when viewed from any distance other than the one used to capture inspection images. To circumvent this challenge, another set of calibration images were captured, but this time they were captured at the same distance from the calibration board as the inspection images would be from the iDrilling station. This ensured the focal lengths for the new calibration images would be more closely matched to the inspection images.\u003c/p\u003e\n \u003cp\u003eWhen these tests were repeated a third time using the same set of 25 images, the results improved significantly. As shown in Fig. \u003cspan class=\"InternalRef\"\u003e10\u003c/span\u003e, the SIFT\u0026thinsp;+\u0026thinsp;FLANN approach had achieved an average measurement error of 1.27mm and the template matching approach could produce measurements with an average error of 0.84mm. In context of the actual measurements being taken, this translates to a 0.7% average error across the three optical sensor measurements. With the new distortion correction measures, the template matching approach was proven to satisfy the positional tolerance of the optical sensors. Thus, the template matching approach was chosen as the preferred image processing engine for the proposed system.\u003c/p\u003e\n \u003cp\u003eIn terms of deploying the proposed system to be used across a wide range of mobile devices, it was not sufficient to only use one set of calibration parameters for all devices. The camera on each device could potentially have its own combination of optical center and focal length, so one calibration parameter to rule them all would not be adequate. This project assumed that the image distortion was consistent across all devices of the same type (e.g., all iPad Pro 12.9\u0026rdquo;), but varied from device type to device type (e.g., iPhone 12 vs iPhone 13). Therefore, it was necessary to build flexibility into the proposed system to use device-specific calibration parameters. To address this challenge, an AR-guided camera calibration tool was developed to help users produce the right parameters for uncalibrated devices. This AR-enabled calibration tool could track users\u0026rsquo; position relative to the calibration board, guide them to capture specific images for calibration, and ensure the images were captured at the correct distance from the calibration board. These images were used to calculate the necessary calibration parameters and were then stored in a ThingWorx data table. The individual entries within this data table were later accessed by the image processing algorithm at runtime to allow the inspection images to be corrected according to the device which captured them.\u003c/p\u003e\n \u003cp\u003eIt is interesting that the SIFT\u0026thinsp;+\u0026thinsp;FLANN approach, which was better suited to deal with image variation, performed worse than the template matching approach. It was suspected that this was a result of two primary factors: the image repeatability from the AR navigational helpers and the lack of texture on the hardware. When using the navigational helpers to precisely position the AR device, users could ensure images were captured from the same viewpoint relative to the hardware with an average accuracy of 4mm, as established by 3.1 and 3.2. This positional precision eliminated enough variation within the captured images that the scalar and rotational robustness of the SIFT\u0026thinsp;+\u0026thinsp;FLANN approach was no longer advantageous. In addition, the SIFT feature extraction algorithm performs best when used to analyze high-contrast, highly textured images since it relies on these things to detect keypoints. The iDrilling station is mostly comprised of 80/20 aluminum, which has a smooth, monochrome surface. This lack of contrast and texture increased the difficulty of extracting keypoints from the captured images, which ultimately rendered the SIFT\u0026thinsp;+\u0026thinsp;FLANN approach unreliable.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"Section2\" id=\"Sec16\"\u003e\n \u003ch2\u003e3.4. Rendering Inspection Feedback\u003c/h2\u003e\n \u003cp\u003eOnce the image processing algorithm completed the inspection, inspection results were returned to the AR experience so that repair instructions could be generated for the user. The realignment feedback took advantage of AR\u0026rsquo;s ability to visually render work instructions and was designed to help workers understand the detected problem and know exactly how to repair it. Once the inspection results arrived back at the AR experience, a result summary popup appeared to indicate whether any of the three sensors were misaligned. If any alignment errors had been detected, these sensors were flagged as being out of alignment, and the exact distance and direction of the misalignment were displayed in the popup summary. A sample result summary popup is shown in Fig.\u0026nbsp;11a.\u003c/p\u003e\n \u003cp\u003eThen, AR and text instructions were dynamically generated to help workers repair each detected misalignment step-by-step. The misaligned sensor was highlighted in its current, misaligned position, then an animation depicting the sensor being repositioned to its correct position was shown. A sequence of screenshots, shown in Fig.\u0026nbsp;11 subfigures b-e, illustrate the realignment animations. Additionally, text instructions accompanied each animation sequence to indicate the displacement distance and direction, as well as the tools required for the repair. Once the worker had realigned each misaligned sensor, the inspection process was repeated, and the sensors\u0026rsquo; new positions were verified. This process could be repeated as many times as necessary until proper sensor alignment was achieved.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"Section2\" id=\"Sec17\"\u003e\n \u003ch2\u003e3.5. Test #4: Total Processing Time\u003c/h2\u003e\n \u003cp\u003eFollowing the methodology outlined in 2.2.4, tests were performed to determine the time required to complete the system\u0026rsquo;s four processing components. Each component was assessed individually using images at eight different resolutions in order to quantify how image resolution affects the overall processing time of the proposed system. Figure \u003cspan class=\"InternalRef\"\u003e12\u003c/span\u003e illustrates how each component is affected by the resolution of the image being transferred/processed. It is interesting to note that the time required to return the results JSON from the image processing algorithm back to the AR device was not affected by image resolution. Altogether, the composite processing time of the four components that were assessed ranged from 3.21 seconds to 0.39 seconds. This signified that, regardless of what image resolution was used, the system\u0026rsquo;s total processing time would always fall below the threshold of six seconds. Still, it was desirable to minimize the total time users spent waiting for the inspection results to be returned.\u003c/p\u003e\n \u003cp\u003eTesting was then performed to determine the system\u0026rsquo;s ability to take accurate measurements on images with varying resolutions. The template matching approach chosen in test #3 was evaluated again, but this time using images at eight different resolutions. Using the measurement accuracy data alongside the data for processing times, it was determined that the optimal resolution for images captured by the proposed system was 1025x768 pixels. This image resolution offered the best balance between the overall processing time and the accuracy of the measurements that the system took. In fact, the images captured at the lowest resolution, 342x256 pixels, had been reduced so much that the image processing algorithm could not be reliably find the three optical sensors, so the results for this resolution were excluded. As shown in Fig. \u003cspan class=\"InternalRef\"\u003e13\u003c/span\u003e, the total processing time exhibited a negatively exponential relationship with the changing image resolutions, while the measurement accuracy\u0026rsquo;s relationship was negatively linear overall. As noted in the analysis for test #3, the alignment tolerance for the iDrilling station\u0026rsquo;s optical sensors was \u0026plusmn;\u0026thinsp;1mm and has been included in Fig. \u003cspan class=\"InternalRef\"\u003e13\u003c/span\u003e for reference. Images captured at a resolution of 1025x768 allowed for the fastest processing while also minimizing the average measurement accuracy.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"4. Conclusions \u0026 Future Work","content":"\u003cp\u003eAn AR-enabled maintenance assistant has been developed that incorporates the four essential characteristics of a truly mobile in-process quality inspection tool \u0026ndash; namely, a mobile inspection camera, automatic work validation via machine vision, markerless tracking, and dynamic creation of repair instructions. During the development process, the primary challenge that was encountered was using a consumer mobile device to capture images that were consistent and distortion-free so accurate inspection results could be reliably computed. As stated previously, if a good image could not be captured for the inspection, it would have been impossible to obtain good results.\u003c/p\u003e \u003cp\u003eTherefore, four tests were performed to evaluate the proposed system\u0026rsquo;s ability to capture a high-quality inspection image, reliably process the image and obtain accurate results, then return the inspection results to users in a timely manner.\u003c/p\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eThe AR Positioning test aimed to quantify the DPO\u0026rsquo;s ability to calculate the device\u0026rsquo;s real-world position relative to the iDrilling station. The positional precision of this component directly affected how much the inspection images would vary. This test indicated that the DPO\u0026rsquo;s estimations were accurate to \u0026plusmn;\u0026thinsp;1.8mm on average.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eThe Device Alignment Time test assessed how quickly users could position their device within a set tolerance. The initial results indicated the navigational helpers were not sufficient to help users achieve a tight alignment tolerance, so a pair of alignment arrows were added. With this new addition, all alignment times were reduced due to the newly added spatial clarity and a much tighter alignment tolerance that was then attainable. Based on these new results, the system\u0026rsquo;s alignment tolerance was set at \u0026plusmn;\u0026thinsp;2mm, which was limited by the DPO\u0026rsquo;s tested accuracy.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eThe Measurement Accuracy test evaluated two different approaches to image processing and compared the ability of each to produce accurate measurements on captured inspection images. Several rounds of testing uncovered unavoidable challenges that stemmed from using mobile devices for such a precise machine vision task. Lens-induced distortion caused measurement inaccuracies which were remedied with a camera calibration regimen. However, the device\u0026rsquo;s auto-focus feature warranted that special considerations be addressed during the calibration process. With these issues resolved, the test results indicated that the template matching approach was better suited for the proposed system since sub-millimeter measurement accuracy could be achieved.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eFinally, the Total Processing Time test sought to minimize the proposed system\u0026rsquo;s total processing time by gradually reducing the resolution of the inspection images while monitoring the average measurement error. According to the testing results, images captured at a resolution of 1025x768 pixels offered an optimal balance of processing time and measurement accuracy. Using this image resolution, the system was able to return inspection results in less than one second and had an average measurement error of 0.65mm.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003cp\u003eUltimately, the testing demonstrated that the proposed system could help users quickly capture acceptable inspection images, reliably take accurate measurements on the captured images, and do so in a timely manner. All these things were designed to be adapted to any consumer mobile device which would be conducive to streamlined deployment in a manufacturing environment.\u003c/p\u003e \u003cp\u003eThere are a few aspects of this project that could be improved in future versions. In terms of the system\u0026rsquo;s overall usability, further work could be done to transition the machine vision from inspecting just a single image to processing the live camera stream. This would most likely require the image processing to be relocated to the mobile device for latency\u0026rsquo;s sake but could potentially open the door to a system that works in real time and is more user-friendly. Regarding the image processing itself, other methods could be explored to try to achieve greater measurement accuracy and robustness as ambient lighting conditions changed and surfaces aged or became soiled. Training data models based on CAD data could allow deep learning techniques to be used, while not requiring hundreds or thousands of actual training images to be captured. Alternatively, as consumer mobile devices are being equipped with more advanced sensors, LiDAR or RGBD cameras could generate 3D point clouds of the work area which could then be used to perform more precise measurements.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAcknowledgements\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eFunding for this research was provided by Northrop Grumman Corporation via contract #4800056662 to BYU. PTC, the parent company to Vuforia, ThingWorx, and Kepware, donated licenses for each of these software to the lab so they could be used in the development of the proposed system. Additionally, Festo Didactic generously supported the integration on their 4-station CP Lab, which was used for this project\u0026rsquo;s test use case.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eStatements and Declarations\u0026nbsp;\u003c/h2\u003e\n\u003ch2\u003eFunding\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eThis work was supported by Northrop Grumman Corporation via contract #4800056662.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eCompeting interests\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eThe authors have no relevant financial or non-financial interests to disclose.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eAvailability of data and material\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eThe authors declare that the data supporting the findings of this study are available within the article and its supplementary information files.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eCode availability\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eThe custom code that supports the findings of this study is available from the corresponding author, James Frandsen, upon reasonable request.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eEthics approval\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eThe authors confirm that this manuscript is original and has not been published, nor is it currently under consideration for publication elsewhere.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eConsent to participate\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eNot applicable.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eConsent for publication\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eNot applicable.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eAuthors\u0026rsquo; contributions\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eAll authors contributed to the study\u0026rsquo;s conception, and design, and development. The first draft of the manuscript was written by James Frandsen and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003e\u003cem\u003eLiveWorx 2021 Episode Three: Digital Transforms Physical\u003c/em\u003e. 2021, PTC, Inc.\u003c/li\u003e\n\u003cli\u003eHo, P.T., et al., \u003cem\u003eStudy of Augmented Reality Based Manufacturing for Further Integration of Quality Control 4.0: A Systematic Literature Review.\u003c/em\u003e Applied Sciences, 2022. \u003cstrong\u003e12\u003c/strong\u003e(4): p. 1961.\u003c/li\u003e\n\u003cli\u003eTang, A., et al. \u003cem\u003eExperimental evaluation of augmented reality in object assembly task\u003c/em\u003e. in \u003cem\u003eProceedings. International Symposium on Mixed and Augmented Reality\u003c/em\u003e. 2002. IEEE Comput. Soc.\u003c/li\u003e\n\u003cli\u003eTang, A., et al. \u003cem\u003eComparative effectiveness of augmented reality in object assembly\u003c/em\u003e. in \u003cem\u003eProceedings of the conference on Human factors in computing systems - CHI \u0026apos;03\u003c/em\u003e. 2003. ACM Press.\u003c/li\u003e\n\u003cli\u003eBarbieri, L. and E. Marino, \u003cem\u003eAn Augmented Reality Tool to Detect Design Discrepancies: A Comparison Test with Traditional Methods\u003c/em\u003e, in \u003cem\u003eLecture Notes in Computer Science\u003c/em\u003e. 2019, Springer International Publishing. p. 99-110.\u003c/li\u003e\n\u003cli\u003eBruno, F., et al., \u003cem\u003eAn augmented reality tool to detect and annotate design variations in an Industry 4.0 approach.\u003c/em\u003e The International Journal of Advanced Manufacturing Technology, 2019. \u003cstrong\u003e105\u003c/strong\u003e(1-4): p. 875-887.\u003c/li\u003e\n\u003cli\u003eHolm, M., et al., \u003cem\u003eAdaptive instructions to novice shop-floor operators using Augmented Reality.\u003c/em\u003e Journal of Industrial and Production Engineering, 2017. \u003cstrong\u003e34\u003c/strong\u003e(5): p. 362-374.\u003c/li\u003e\n\u003cli\u003eMarino, E., et al., \u003cem\u003eAn Augmented Reality inspection tool to support workers in Industry 4.0 environments.\u003c/em\u003e Computers in Industry, 2021. \u003cstrong\u003e127\u003c/strong\u003e: p. 103412.\u003c/li\u003e\n\u003cli\u003eYang, X., et al., \u003cem\u003eEdge-based cover recognition and tracking method for an AR-aided aircraft inspection system.\u003c/em\u003e The International Journal of Advanced Manufacturing Technology, 2020. \u003cstrong\u003e111\u003c/strong\u003e(11-12): p. 3505-3518.\u003c/li\u003e\n\u003cli\u003ePolvi, J., et al., \u003cem\u003eHandheld Guides in Inspection Tasks: Augmented Reality versus Picture.\u003c/em\u003e IEEE Transactions on Visualization and Computer Graphics, 2018. \u003cstrong\u003e24\u003c/strong\u003e(7): p. 2118-2128.\u003c/li\u003e\n\u003cli\u003eZhou, J., et al. \u003cem\u003eApplying spatial augmented reality to facilitate in-situ support for automotive spot welding inspection\u003c/em\u003e. in \u003cem\u003eProceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry - VRCAI \u0026apos;11\u003c/em\u003e. 2011. ACM Press.\u003c/li\u003e\n\u003cli\u003eHartl, A.D., et al., \u003cem\u003eEfficient Verification of Holograms Using Mobile Augmented Reality.\u003c/em\u003e IEEE Transactions on Visualization and Computer Graphics, 2016. \u003cstrong\u003e22\u003c/strong\u003e(7): p. 1843-1851.\u003c/li\u003e\n\u003cli\u003eRunji, J.M. and C.-Y. Lin. \u003cem\u003eAutomatic Optical Inspection aided Augmented Reality-based PCBA Inspection: A Development\u003c/em\u003e. in \u003cem\u003e2019 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT)\u003c/em\u003e. 2019. IEEE.\u003c/li\u003e\n\u003cli\u003eRunji, J.M. and C.-Y. Lin, \u003cem\u003eMarkerless cooperative augmented reality-based smart manufacturing double-check system: Case of safe PCBA inspection following automatic optical inspection.\u003c/em\u003e Robotics and Computer-Integrated Manufacturing, 2020. \u003cstrong\u003e64\u003c/strong\u003e: p. 101957.\u003c/li\u003e\n\u003cli\u003eOdenthal, B., et al., \u003cem\u003eA Comparative Study of Head-Mounted and Table-Mounted Augmented Vision Systems for Assembly Error Detection.\u003c/em\u003e Human Factors and Ergonomics in Manufacturing \u0026amp; Service Industries, 2014. \u003cstrong\u003e24\u003c/strong\u003e(1): p. 105-123.\u003c/li\u003e\n\u003cli\u003eYuan, M.L., S.K. Ong, and A.Y.C. Nee, \u003cem\u003eAugmented reality for assembly guidance using a virtual interactive tool.\u003c/em\u003e International Journal of Production Research, 2008. \u003cstrong\u003e46\u003c/strong\u003e(7): p. 1745-1767.\u003c/li\u003e\n\u003cli\u003eZauner, J., et al. \u003cem\u003eAuthoring of a mixed reality assembly instructor for hierarchical structures\u003c/em\u003e. in \u003cem\u003eThe Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings.\u003c/em\u003e 2003. IEEE Comput. Soc.\u003c/li\u003e\n\u003cli\u003eZhang, J., S.K. Ong, and A.Y.C. Nee, \u003cem\u003eRFID-assisted assembly guidance system in an augmented reality environment.\u003c/em\u003e International Journal of Production Research, 2011. \u003cstrong\u003e49\u003c/strong\u003e(13): p. 3919-3938.\u003c/li\u003e\n\u003cli\u003eKwon, O.-S., C.-S. Park, and C.-R. Lim, \u003cem\u003eA defect management system for reinforced concrete work utilizing BIM, image-matching and augmented reality.\u003c/em\u003e Automation in Construction, 2014. \u003cstrong\u003e46\u003c/strong\u003e: p. 74-81.\u003c/li\u003e\n\u003cli\u003eShin, D.H. and P.S. Dunston, \u003cem\u003eEvaluation of Augmented Reality in steel column inspection.\u003c/em\u003e Automation in Construction, 2009. \u003cstrong\u003e18\u003c/strong\u003e(2): p. 118-129.\u003c/li\u003e\n\u003cli\u003eOjer, M., et al., \u003cem\u003eReal-time automatic optical system to assist operators in the assembling of electronic components.\u003c/em\u003e The International Journal of Advanced Manufacturing Technology, 2020. \u003cstrong\u003e107\u003c/strong\u003e(5-6): p. 2261-2275.\u003c/li\u003e\n\u003cli\u003eOjer, M., et al., \u003cem\u003eProjection-Based Augmented Reality Assistance for Manual Electronic Component Assembly Processes.\u003c/em\u003e Applied Sciences, 2020. \u003cstrong\u003e10\u003c/strong\u003e(3): p. 796.\u003c/li\u003e\n\u003cli\u003eAlves, J., et al. \u003cem\u003eComparing Spatial and Mobile Augmented Reality for Guiding Assembling Procedures with Task Validation\u003c/em\u003e. in \u003cem\u003e2019 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC)\u003c/em\u003e. 2019. IEEE.\u003c/li\u003e\n\u003cli\u003eAlves, J., et al., \u003cem\u003eUsing Augmented Reality and Step by Step Verification in Industrial Quality Control\u003c/em\u003e, in \u003cem\u003eHuman Systems Engineering and Design III\u003c/em\u003e. 2021, Springer International Publishing. p. 350-355.\u003c/li\u003e\n\u003cli\u003eGupta, A., et al. \u003cem\u003eDuploTrack\u003c/em\u003e. in \u003cem\u003eProceedings of the 25th annual ACM symposium on User interface software and technology - UIST \u0026apos;12\u003c/em\u003e. 2012. ACM Press.\u003c/li\u003e\n\u003cli\u003eManuri, F., A. Pizzigalli, and A. Sanna, \u003cem\u003eA State Validation System for Augmented Reality Based Maintenance Procedures.\u003c/em\u003e Applied Sciences, 2019. \u003cstrong\u003e9\u003c/strong\u003e(10): p. 2115.\u003c/li\u003e\n\u003cli\u003eWasenmuller, O., M. Meyer, and D. Stricker. \u003cem\u003eAugmented Reality 3D Discrepancy Check in Industrial Applications\u003c/em\u003e. in \u003cem\u003e2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)\u003c/em\u003e. 2016. IEEE.\u003c/li\u003e\n\u003cli\u003eAlves, J.B., et al., \u003cem\u003eUsing augmented reality for industrial quality assurance: a shop floor user study.\u003c/em\u003e The International Journal of Advanced Manufacturing Technology, 2021. \u003cstrong\u003e115\u003c/strong\u003e(1-2): p. 105-116.\u003c/li\u003e\n\u003cli\u003eKhuong, B.M., et al. \u003cem\u003eThe effectiveness of an AR-based context-aware assembly support system in object assembly\u003c/em\u003e. in \u003cem\u003e2014 IEEE Virtual Reality (VR)\u003c/em\u003e. 2014. IEEE.\u003c/li\u003e\n\u003cli\u003eNishihara, A. and J. Okamoto. \u003cem\u003eObject recognition in assembly assisted by augmented reality system\u003c/em\u003e. in \u003cem\u003e2015 SAI Intelligent Systems Conference (IntelliSys)\u003c/em\u003e. 2015. IEEE.\u003c/li\u003e\n\u003cli\u003ePathomaree, N. and S. Charoenseang. \u003cem\u003eAugmented reality for skill transfer in assembly task\u003c/em\u003e. in \u003cem\u003eROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005.\u003c/em\u003e 2005. IEEE.\u003c/li\u003e\n\u003cli\u003eWu, L.-C., I.-C. Lin, and M.-H. Tsai. \u003cem\u003eAugmented reality instruction for object assembly based on markerless tracking\u003c/em\u003e. in \u003cem\u003eProceedings of the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games\u003c/em\u003e. 2016. ACM.\u003c/li\u003e\n\u003cli\u003eOkamoto, J. and A. Nishihara, \u003cem\u003eAssembly assisted by augmented reality (A 3 R)\u003c/em\u003e, in \u003cem\u003eIntelligent Systems and Applications\u003c/em\u003e. 2016, Springer. p. 281-300.\u003c/li\u003e\n\u003cli\u003eFeiner, S., B. Macintyre, and D. Seligmann, \u003cem\u003eKnowledge-based augmented reality.\u003c/em\u003e Communications of the ACM, 1993. \u003cstrong\u003e36\u003c/strong\u003e(7): p. 53-62.\u003c/li\u003e\n\u003cli\u003eLiverani, A., G. Amati, and G. Caligiana, \u003cem\u003eA CAD-augmented Reality Integrated Environment for Assembly Sequence Check and Interactive Validation.\u003c/em\u003e Concurrent Engineering, 2004. \u003cstrong\u003e12\u003c/strong\u003e(1): p. 67-77.\u003c/li\u003e\n\u003cli\u003eWesterfield, G., A. Mitrovic, and M. Billinghurst, \u003cem\u003eIntelligent Augmented Reality Training for Motherboard Assembly.\u003c/em\u003e International Journal of Artificial Intelligence in Education, 2015. \u003cstrong\u003e25\u003c/strong\u003e(1): p. 157-172.\u003c/li\u003e\n\u003cli\u003eAlvarez, H., I. Aguinaga, and D. Borro. \u003cem\u003eProviding guidance for maintenance operations using automatic markerless Augmented Reality system\u003c/em\u003e. in \u003cem\u003e2011 10th IEEE International Symposium on Mixed and Augmented Reality\u003c/em\u003e. 2011. IEEE.\u003c/li\u003e\n\u003cli\u003eDalle Mura, M. and G. Dini, \u003cem\u003eAn augmented reality approach for supporting panel alignment in car body assembly.\u003c/em\u003e Journal of Manufacturing Systems, 2021. \u003cstrong\u003e59\u003c/strong\u003e: p. 251-260.\u003c/li\u003e\n\u003cli\u003eLee, H., et al., \u003cem\u003eA Framework for Process Model Based Human-Robot Collaboration System Using Augmented Reality\u003c/em\u003e, in \u003cem\u003eAdvances in Production Management Systems. Smart Manufacturing for Industry 4.0\u003c/em\u003e. 2018, Springer International Publishing. p. 482-489.\u003c/li\u003e\n\u003cli\u003eAntonelli, D. and S. Astanin, \u003cem\u003eEnhancing the Quality of Manual Spot Welding through Augmented Reality Assisted Guidance.\u003c/em\u003e Procedia CIRP, 2015. \u003cstrong\u003e33\u003c/strong\u003e: p. 556-561.\u003c/li\u003e\n\u003cli\u003eHakkarainen, M., C. Woodward, and M. Billinghurst. \u003cem\u003eAugmented assembly using a mobile phone\u003c/em\u003e. in \u003cem\u003e2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality\u003c/em\u003e. 2008. IEEE.\u003c/li\u003e\n\u003cli\u003eLiu, C., et al. \u003cem\u003eEvaluating the benefits of real-time feedback in mobile augmented reality with hand-held devices\u003c/em\u003e. in \u003cem\u003eProceedings of the SIGCHI Conference on Human Factors in Computing Systems\u003c/em\u003e. 2012. ACM.\u003c/li\u003e\n\u003cli\u003eMourtzis, D., F. Xanthi, and V. Zogopoulos, \u003cem\u003eAn Adaptive Framework for Augmented Reality Instructions Considering Workforce Skill.\u003c/em\u003e Procedia CIRP, 2019. \u003cstrong\u003e81\u003c/strong\u003e: p. 363-368.\u003c/li\u003e\n\u003cli\u003eSchlueter, J.A., \u003cem\u003eRemote Maintenance Assistance Using Real-time Augmented Reality Authoring\u003c/em\u003e, in \u003cem\u003eProQuest Dissertations and Theses\u003c/em\u003e. 2018, Iowa State University: Ann Arbor. p. 72.\u003c/li\u003e\n\u003cli\u003eSegovia, D., et al., \u003cem\u003eAugmented Reality as a Tool for Production and Quality Monitoring.\u003c/em\u003e Procedia Computer Science, 2015. \u003cstrong\u003e75\u003c/strong\u003e: p. 291-300.\u003c/li\u003e\n\u003cli\u003eLi, S., P. Zheng, and L. Zheng, \u003cem\u003eAn AR-Assisted Deep Learning-Based Approach for Automatic Inspection of Aviation Connectors.\u003c/em\u003e IEEE Transactions on Industrial Informatics, 2021. \u003cstrong\u003e17\u003c/strong\u003e(3): p. 1721-1731.\u003c/li\u003e\n\u003cli\u003eFerraguti, F., et al., \u003cem\u003eAugmented reality based approach for on-line quality assessment of polished surfaces.\u003c/em\u003e Robotics and Computer-Integrated Manufacturing, 2019. \u003cstrong\u003e59\u003c/strong\u003e: p. 158-167.\u003c/li\u003e\n\u003cli\u003eMourtzis, D., V. Zogopoulos, and F. Xanthi, \u003cem\u003eAugmented reality application to support the assembly of highly customized products and to adapt to production re-scheduling.\u003c/em\u003e The International Journal of Advanced Manufacturing Technology, 2019. \u003cstrong\u003e105\u003c/strong\u003e(9): p. 3899-3910.\u003c/li\u003e\n\u003cli\u003eDanielsson, O., et al., \u003cem\u003eOperators perspective on augmented reality as a support tool in engine assembly.\u003c/em\u003e Procedia CIRP, 2018. \u003cstrong\u003e72\u003c/strong\u003e: p. 45-50.\u003c/li\u003e\n\u003cli\u003eLampen, E., et al., \u003cem\u003eCombining Simulation and Augmented Reality Methods for Enhanced Worker Assistance in Manual Assembly.\u003c/em\u003e Procedia CIRP, 2019. \u003cstrong\u003e81\u003c/strong\u003e: p. 588-593.\u003c/li\u003e\n\u003cli\u003eHenderson, S. and S. Feiner, \u003cem\u003eExploring the Benefits of Augmented Reality Documentation for Maintenance and Repair.\u003c/em\u003e IEEE Transactions on Visualization and Computer Graphics, 2011. \u003cstrong\u003e17\u003c/strong\u003e(10): p. 1355-1368.\u003c/li\u003e\n\u003cli\u003eQeshmy, D.E., et al., \u003cem\u003eManaging Human Errors: Augmented Reality systems as a tool in the quality journey.\u003c/em\u003e Procedia Manufacturing, 2019. \u003cstrong\u003e28\u003c/strong\u003e: p. 24-30.\u003c/li\u003e\n\u003cli\u003eSchlagowski, R., L. Merkel, and C. Meitinger. \u003cem\u003eDesign of an assistant system for industrial maintenance tasks and implementation of a prototype using augmented reality\u003c/em\u003e. in \u003cem\u003e2017 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM)\u003c/em\u003e. 2017. IEEE.\u003c/li\u003e\n\u003cli\u003eZhu, J., S.K. Ong, and A.Y.C. Nee, \u003cem\u003eA context-aware augmented reality system to assist the maintenance operators.\u003c/em\u003e International Journal on Interactive Design and Manufacturing (IJIDeM), 2014. \u003cstrong\u003e8\u003c/strong\u003e(4): p. 293-304.\u003c/li\u003e\n\u003cli\u003eLiu, C., et al., \u003cem\u003eAugmented Reality-assisted Intelligent Window for Cyber-Physical Machine Tools.\u003c/em\u003e Journal of Manufacturing Systems, 2017. \u003cstrong\u003e44\u003c/strong\u003e: p. 280-286.\u003c/li\u003e\n\u003cli\u003eWang, Y., et al., \u003cem\u003eMechanical assembly assistance using marker-less augmented reality system.\u003c/em\u003e Assembly Automation, 2018. \u003cstrong\u003e38\u003c/strong\u003e(1): p. 77-87.\u003c/li\u003e\n\u003cli\u003eLoch, F., F. Quint, and I. Brishtel. \u003cem\u003eComparing Video and Augmented Reality Assistance in Manual Assembly\u003c/em\u003e. in \u003cem\u003e2016 12th International Conference on Intelligent Environments (IE)\u003c/em\u003e. 2016. IEEE.\u003c/li\u003e\n\u003cli\u003eRadkowski, R., J. Herrema, and J. Oliver, \u003cem\u003eAugmented Reality-Based Manual Assembly Support With Visual Features for Different Degrees of Difficulty.\u003c/em\u003e International Journal of Human-Computer Interaction, 2015. \u003cstrong\u003e31\u003c/strong\u003e(5): p. 337-349.\u003c/li\u003e\n\u003cli\u003eFesto. \u003cem\u003eCP Lab | Festo USA\u003c/em\u003e. 2022; Available from: https://www.festo.com/us/en/e/technical-education/learning-systems/factory-automation-and-industry-4-0/learning-factories/cp-systems-large-scale-industry-4-0-learning-factories/cp-lab-id_36133/.\u003c/li\u003e\n\u003cli\u003e\u003cem\u003eUR3e Technical Details\u003c/em\u003e. 2020: @www_universal-robots_com. p. https://www.universal-robots.com/media/1802780/ur3e-32528_ur_technical_details_.pdf\u003c/li\u003e\n\u003cli\u003eLowe, D.G. \u003cem\u003eObject recognition from local scale-invariant features\u003c/em\u003e. in \u003cem\u003eProceedings of the Seventh IEEE International Conference on Computer Vision\u003c/em\u003e. 1999. IEEE.\u003c/li\u003e\n\u003cli\u003eMuja, M. and D.G. Lowe, \u003cem\u003eFast approximate nearest neighbors with automatic algorithm configuration.\u003c/em\u003e VISAPP (1), 2009. \u003cstrong\u003e2\u003c/strong\u003e(331-340): p. 2.\u003c/li\u003e\n\u003cli\u003e\u003cem\u003e1 in 2 visitors abandon a website that takes more than 6 seconds to load - Digital.com\u003c/em\u003e. 2022; Available from: https://digital.com/1-in-2-visitors-abandon-a-website-that-takes-more-than-6-seconds-to-load/.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"the-international-journal-of-advanced-manufacturing-technology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jamt","sideBox":"Learn more about [The International Journal of Advanced Manufacturing Technology](https://www.springer.com/journal/170)","snPcode":"170","submissionUrl":"https://submission.nature.com/new-submission/170/3","title":"The International Journal of Advanced Manufacturing Technology","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Mobile augmented reality, Quality inspection, Computer vision, Maintenance procedures, Industry 4.0","lastPublishedDoi":"10.21203/rs.3.rs-1842846/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-1842846/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eWith the advances of industry 4.0, augmented reality (AR) devices are being deployed across the manufacturing sector to enhance worker perception and efficiency. AR is often used to deliver spatially relevant work instructions on mobile devices for maintenance procedures on the factory floor. In these situations, workers use their mobile devices to view instructions in the form of 3D animations and annotations that directly overlay the equipment being maintained. Workers then follow the AR instructions and must ultimately rely on their own judgement and knowledge of the procedure as they progress from step to step. An AR assistant that could validate each stage of the procedure in real time and provide the worker with feedback on any observed errors would ensure that each maintenance procedure is completed successfully. This work presents a mobile, quality inspection system for AR maintenance procedures that is capable of assessing the maintenance task in real time. The system is designed for deployment on handheld mobile devices and can thus manage the challenges inherent to performing quality inspection with a non-fixed vision system. This work enumerates four essential qualities of mobile quality inspection tools and outlines some of the challenges encountered during the development of such a system. In the end, testing established that the system could provide adequate assistance for capturing inspection images, accurately process the captured images using machine vision, and generate detailed feedback from the quality inspection in a timely manner.\u003c/p\u003e","manuscriptTitle":"An Augmented Reality Maintenance Assistant with Real-time Quality Inspection on Handheld Mobile Devices","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2022-07-26 18:58:49","doi":"10.21203/rs.3.rs-1842846/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewerAgreed","content":"","date":"2022-07-19T13:41:49+00:00","index":0,"fulltext":""},{"type":"reviewersInvited","content":"","date":"2022-07-19T13:40:56+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2022-07-19T01:04:35+00:00","index":"","fulltext":""},{"type":"submitted","content":"The International Journal of Advanced Manufacturing Technology","date":"2022-07-15T17:11:33+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"the-international-journal-of-advanced-manufacturing-technology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jamt","sideBox":"Learn more about [The International Journal of Advanced Manufacturing Technology](https://www.springer.com/journal/170)","snPcode":"170","submissionUrl":"https://submission.nature.com/new-submission/170/3","title":"The International Journal of Advanced Manufacturing Technology","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"160ad175-4f62-4ac1-9ebb-58e12e688649","owner":[],"postedDate":"July 26th, 2022","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2023-10-16T18:48:01+00:00","versionOfRecord":{"articleIdentity":"rs-1842846","link":"https://doi.org/10.1007/s00170-023-10978-1","journal":{"identity":"the-international-journal-of-advanced-manufacturing-technology","isVorOnly":false,"title":"The International Journal of Advanced Manufacturing Technology"},"publishedOn":"2023-02-06 18:43:13","publishedOnDateReadable":"February 6th, 2023"},"versionCreatedAt":"2022-07-26 18:58:49","video":"","vorDoi":"10.1007/s00170-023-10978-1","vorDoiUrl":"https://doi.org/10.1007/s00170-023-10978-1","workflowStages":[]},"version":"v1","identity":"rs-1842846","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-1842846","identity":"rs-1842846","version":["v1"]},"buildId":"_2-kVJe1T_tPrBINL-cwx","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}