Development of Multimodal Augmented Reality Application on the Flight Deck for Single-Pilot Operations: System Usability and User Interface Satisfaction

Development of Multimodal Augmented Reality Application on the Flight Deck for Single-Pilot Operations: System Usability and User Interface Satisfaction | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Development of Multimodal Augmented Reality Application on the Flight Deck for Single-Pilot Operations: System Usability and User Interface Satisfaction Wen-Chin Li, Jingyi Zhang, James Blundell, Samel Court, Dujuan Sevillian This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4844391/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 18 Jun, 2026 Read the published version in Virtual Reality → Version 1 posted You are reading this latest preprint version Abstract Implementing augmented reality (AR) technologies has become a popular method of increasing operators' situation awareness by adding virtual information to the physical environment. In the current commercial two-pilot flight deck, the pilot-flying (PF) is responsible for flying the aircraft to an approved flight plan, and the pilot-monitoring (PM) focuses on communicating and monitoring PF’s operational behaviours. The driving factors behind single-pilot operations (SPO) are the foreseen pilot shortage and desire to reduce operating costs. Whilst SPO is expected to be enabled - in part - by advanced flight deck technologies. Forty participants attended simulator trials that involved interacting with multimodal AR apps (Hololens) which included voice and gesture command functionalities. Results revealed voice command scored higher than the gesture command on both the System Usability Scale (SUS) and the Questionnaire for User Interface Satisfaction (QUIS). AR visualisation that blends the physical operation environment and a virtual holographic checklist with guiding cues can improve pilots’ monitoring performance and procedure compliance during the instrument landing trials. Furthermore, the AR voice command permits multiple sensory processing and response by integrating visual, auditory, and tactile inputs simultaneously, which provides the pilot with greater flexibility to meet task requirements. AR gesture command was regarded as an unnecessary burden to the pilot’s cognitive resources and limited hand movement while executing complicated operating procedures. Future research shall explore the implementation of AR voice command with artificial intelligence (AI) on the flight deck to support a single pilot performing both flying and monitoring tasks. This research paradigm needs to be assessed thoroughly for human-centred design to ensure broad acceptance by end users, manufacturers, airlines and regulators. Augmented Reality Multimodal Interaction Single Pilot Operations System Usability User Interface Experience Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 1. Introduction The increasing worldwide demand for commercial air travel over the next 20 years will be dramatically increasing by an expected shortfall of 500,000 qualified pilots (IATA, 2023 ). To address this operational challenge, a concept being pursued by the aviation industry is to reduce the current two-pilot flight deck configuration to a single pilot – known as of Single Pilot Operations (SPO). Multiple interrelated obstacles to the realisation of SPO exist including ‘buy-in’ from relevant key stakeholders, such as regulators, the aviation professional community and the public, and the maintenance of operational safety tolerances related to pilots’ workload and fatigue (Harris, 2023 ; Schmid & Stanton, 2020 ). The use of augmented reality (AR) technology has been proposed as a technological enabler of SPO to facilitate the new forms of system interaction that could be expressed in the SPO environment, while simultaneously mitigating increases in pilot workload (Tran et al., 2018 ). In the current commercial two-pilot flight deck, the pilot-flying (PF) is responsible for flying the aircraft to an approved flight plan, and the pilot-monitoring (PM) focuses on navigating, communicating, and monitoring PF’s operational behaviours. The PM may also support the PF by assisting with high-workload situations and taking control in emergency situations (Lim et al., 2017 ). To meet the expected growth in air transportation and to further reduce operational costs, one active interest for airlines is the introduction of Single Pilot Operations (SPO), which would apply innovative technologies on the flight deck to facilitate minimum pilot intervention, allowing sufficient resting and management of workload to permit one pilot to be rostered to the flight (ICAO, 2022 ). Airlines and manufacturers have been forced to advance innovative technologies to provide sufficient pilots to enhance the cost-efficiency of training and flight operations (ICAO, 2019 ). It is also important to consider the decision-making aids that can be incorporated into the future flight deck to assist one pilot in managing all unexpected scenarios. Therefore, there is potential for the implementation of augmented reality (AR) technology to enhance the pilot’s monitoring performance, allowing a ‘head up, eyes out’ mentality to be achieved. To this end - the current research explores the system usability and user satisfaction of interacting with AR devices within a simulated single pilot operating environment, whilst simultaneously investigating the influence of interaction modality type – verbal versus gesture. 2. Related Work The move to single-pilot operations in commercial aviation is a significant step towards greater efficiency but comes with notable safety challenges. The advanced technologies of augmented reality (AR) with artificial intelligence (AI) have been widely used to increase operators’ situation awareness by combining virtual objects/information in the physical environment, which in turn allows the operators to execute multiple tasks simultaneously (Gregoriades & Sutcliffe, 2018 ; Woodward & Ruiz, 2022 ). There is the potential for future flight deck design applying AR technology to minimise the space required to house critical information in the effective field of view for the pilot and allow real-time processing of flight data to better improve situational awareness throughout the flight (Li et al., 2022 ; Meister et al., 2022 ). 2.1. The Role of AR in Single-Pilot Operations The trend in flight deck design has been on the approach of ‘de-crewing’ over the past 70 years, with crew numbers from five (captain, co-pilot, flight engineer, navigator and radio operator) to two pilots (PF and PM) only. Aircraft manufacturers and avionics systems suppliers are developing advanced technology for such aircraft, centred upon the development of Intelligent Knowledge-Based Systems and adaptive automation (Harris, 2023 ). Aviation human factors focus on human-centric design for the implementation of innovative technology on the flight deck (Boy, 2014 ). For information presentation, the visual saliency of task-critical objects is an overarching factor that should be considered to expand pilots’ visual attention and enhance situation awareness (Dill & Young, 2015 ). The proximity compatibility principle (PCP) suggests integrating different sources of information in close spatial vicinity to facilitate operators using one gaze to catch all relevant information (Wickens & Carswell, 1995 ; Wickens & Ward, 2017 ). Pilots must monitor constantly changing parameters and select appropriate control modes by cross-checking various displays (Burian, 2006 ). Augmented visualisation design with salient mode transition on the optimised PFD can reduce pilots’ cognitive efforts and perceived workload by eliminating the requirement to read text messages compared to a traditional PFD (Li et al., 2020 ). An interactive interface of datalink that provides information of air-ground communication displayed in a salient position can decrease pilots’ cognitive efforts and task completion time (Jorna, 1997 ). Human-centred flight deck design can also facilitate the usage of high-level automation systems and support the implementation of single pilot operations to significantly improve cost efficiency (Harris, 2007 ). Advanced technologies in flight deck design and artificial intelligence are great sources of driving SPO and improving aviation safety and efficiency (Chan & Li, 2021 ). These technological advancements are among several compelling reasons for the consideration of SPO in commercial air transport. To be feasible for SPO in commercial air transport, a higher level of automation than what currently exists needs to be developed and amalgamated with the models of the regulator’s proposed expectations (Myers & Starr, 2021 ). Levels of automation would be inversely proportional to the human operator complement in the system in which one pilot would fly the aircraft, with the functions of the co-pilot being assigned to an augmented/automated system on the flight deck. Moreover, the human-centric flight deck design with innovative interactive modes can improve the pilot’s situation awareness and provide decision support (Carroll & Dahlstrom, 2021 ). The head-mounted display (HMD) and see-through function of AR allows virtual objects to be integrated into the operational environment to assist a pilot with retrieving flight parameters, checking standard operational procedures (SOPs) and controlling the aircraft flight path. Holographic AR glasses can offer an optimal solution to reduce operators’ workload and stress levels by presenting critical information in a timely manner (Jetter et al., 2018 ; Tran et al., 2018 ). The flight deck checklist has evolved over the years with various types and interactive modes. AR can add salient audio-visual pointers and feedback to distinguish between accomplished and non-accomplished procedures. A previous study demonstrated that the application of the voice-command AR checklist could improve pilots’ situation awareness and reduce cognitive workload since augmented visualization cues do not require pilots to divert attention resources away from tasks in hand (Li et al., 2022 ). 2.2 Multimodal Interaction with AR With advances in software and hardware development, multimodal AR that incorporates various modes of interaction is becoming more prevalent to everyday life and workplaces (Aromaa et al., 2020 ; Brown et al., 2021 ). Two common interaction modes include gesture and voice control (He et al., 2017 ; Korkiakoski et al., 2024 ). The advantages of AR gesture-based interactions, compared to traditional touchscreen gesture interactions, is that they are not bound to a specific physical location and, if well designed and consist of a small command set, offer a natural interaction modality (Riener et al., 2017 ). Similarly, speech control provides a naturalistic interaction medium as it is our primary mode of communication (Detjen et al., 2021 ). In terms of task performance with the two interaction modalities, Lee et al., ( 2013 ) demonstrated that using voice commands compared to gesture commands were superior in an abstract, self-paced, shape manipulation task. A recent structured review of 42 input modality studies between 2013 and 2020 by Spittle et al. ( 2023 ) highlighted the task enhancement that voice commands can provide but stressed that this was largely task dependent. For instance, voice commands will be more advantageous for menu-selection based interactions whilst gesture interactions would be more useful for tasks that require spatial manipulation of virtual objects. The performance benefit of voice commands likely stems from offering a ‘hand-free’ advantage during tasks which already entail a high degree of manual input – a capability which was deemed important by the pilots who evaluated SPO concepts in the series of studies conducted by NASA (Bailey et al., 2017 ). The advantage of voice-command is also in keeping with cognitive resource theory - Wicken’s multiple resource theory (Wickens, 2002 ) stipulates that task performance will be superior when the attentional demands of a task are distributed across a wider range of sensory modalities. Thus, reducing the likelihood of encountering a “cognitive bottleneck”. Despite the numerous studies that have demonstrated the performance benefits of voice command applications, there remain mixed results in terms of user preference. In some cases, users still prefer to use gestures instead of voice for certain tasks even when the performance is worse (Sadri et al., 2019 )., More specifically to aviation, there exists a general lack of usability research concerning the implementation of AR in flight operations (Blundell & Harris, 2023 ). The majority of AR usability research has been predominantly based on self-paced tasks across a range of application areas, such as education, entertainment, maintenance and medicine (Dey et al., 2018 ). This is important in the context of using AR to support SPO, as the task environment where AR is likely to be of most benefit will be during an externally paced flight environment – such as during the completion of pre-landing and emergency procedures (Li et al., 2022 ; Tran et al., 2018 ). The use of AR technology to provide pilots with emergency procedures by both providing instructional cues and overlapping the critical displays can improve pilots’ response times and reduce the cognitive loads of performing tasks during high-workload scenarios. In the aviation industry, Bagassi et al. ( 2020 ) proposed an innovative concept of air traffic control based on AR technology. It provides a heads-up view of the air traffic in low visibility conditions, which can reduce controllers' perceived workload, increase task performance and ensure operational safety. The multimodal AR-supported training for procedures compliance with Visual Flight Rules (VFR) demonstrated significant benefits in improving pilots’ performance on visual detection, as well as training engagement and immersion (Moesl et al., 2023 ). The see-through AR device provides critical information in the dynamic environment to assist the pilot in processing mass information and making aeronautical decisions (ADM) in time-limited situations. AR technology has the potential to improve SOP compliance and cross-check between PF and PM. Multimodal AR proposed several innovative interaction techniques, such as sensor-based, audio-based, and visual-based modes, which can facilitate effective human-computer interaction and enhance user experience (Nizam et al., 2018 ). 2.3 Motivation and Hypotheses To successfully implement AR technology in flight operations, the system usability and user interface of AR should be assessed at the early stages of design. The SUS is distinguished into two factors of usability and learnability, which are significantly correlated with each other (Brooke, 1996 ); and have been applied in the human-computer interaction domain (Meister et al., 2022 ). The other assessment tool is Questionnaires for User Interface Satisfaction (QUIS), designed to assess users’ subjective satisfaction with specific aspects of the human-computer interface. The results of QUIS could help the system developer enhance the products’ reliability and validity by means of satisfaction measurements (Chin et al., 1988 ). There are six sub-dimensions involved: overall user reactions, screen visibility, system information, learning, system capabilities, and usability and user interface. QUIS has been widely used to evaluate the interface satisfaction and user experience effectively of an innovative system (Helin et al., 2018 ). The application of AR technology on the flight deck requires pilots’ feedback on performing tasks related to visual perception, attention distribution, and information processing. The aims of this study were to investigate both the system's usability and users’ experiences while interacting with multimodal AR checklists (gesture-command vs voice-command). There are two hypotheses were proposed as follows: H 1 : There is a significant difference in system usability (SUS) between voice-command and gesture-command with AR applications during instrument landing. H 2 : There is a significant difference in user interface satisfaction (QUIS) between voice-command and gesture-command with AR applications during instrument landing. 3. Methods 3.1. Participants Forty participants, including seven females (17.5%) and 33 males (82.5%), were recruited from aviation research institutes with varied flight hours (M = 122.70, SD = 468.29) and familiar with operating the flight simulator used in the experiment. Their ages ranged between 21 and 55 years old (M = 27.90, SD = 8.09). A research proposal was approved by the research ethics committee before conducting the experiment, as this research involved collecting data from human participants. All participants signed a consent form which provided them with detailed information related to the experiment and were guaranteed the right to withdraw from the experiment at any stage. Data collected has been used and stored in accordance with Cranfield’s Research Ethics Policy and Management of Research Data Policy. 3.2. Augmented Reality Device The AR device used in the experiment is a Microsoft HoloLens headset. These glasses comprise see-through holographic waveguides, two HD 16:9 light engines and built-in processors that can display holograms with a resolution of 1280 x 720 pixels per eye, a field of view of 30° \(\:\times\:\) 17.5° and a refresh rate of 60 Hz. The symbols, fonts, colours, and text used in the UI design are optimised to improve pilots’ attention and situation awareness (Fig. 1). The HoloLens comes with built-in sensors: an Inertial Measurement Unit (IMU), four environment understanding cameras, one depth camera, one HD video camera, four microphones and one ambient light sensor. Its audio output consists of two speakers located near the user’s ears that can emit spatial sound. The depth camera is used to carry out the user’s hand gesture recognition and spatial mapping of the surrounding environment. The user can extend or retract the headband and can slide the visor forward or backwards in order to wear the headset more comfortably (Microsoft, 2021). 3.3. AR Applications for SPO The applications of multimodal AR pre-landing checklist interaction modes consisting of voice commands and gesture commands have been developed for this trial. There are only five procedures for both gesture and voice commands on the HoloLens, including (1). Flap 30-Degree; (2). Landing Gear-DOWN; (3) Speedbrakes-ARM, (4). Landing Lights-ON, and (5) Back to Menu (Fig. 2a). Participants followed the standard operating procedures to control airspeed, altitude, descent rate and touchdown distance for safe operation. The AR checklist apps were developed with the Unity game engine and a combination of Microsoft Mixed Reality Toolkit (MRTK) and Vuforia Software Development Kit (SDK). Animated yellow arrow symbols provided visual guidance to participants regarding the position of the current checklist item. The animated yellow symbols always point towards the physical flight deck element of the relevant SOP and disappear when the user gets closer to the check item (top, Fig. 2b). The yellow circular target animations are check-item indicators that are overlaid at the position of the lever/switch to direct the user’s attention to confirm the completion of the check item (bottom, Fig. 2b). With gesture control, participants move the central cursor with their head and select a menu/checklist item (such as Landing Light On) by holding the cursor gaze and by performing an ‘air-tap’ (Fig. 2c). This ‘air-tap’ gesture represents the mouse click commonly found in a graphical user interface (GUI). The physical flight deck had been mapped using the HoloLens depth camera and was registered and aligned with the Unity software to obtain a specific visual orientation on the operational environment (Fig. 2d). Each checklist highlight (yellow circles and spatial text) was precisely positioned according to the spatial mapping registration alignment with Vuforia. With the voice-command interaction, the user reads out the menu/checklist title to activate it and validates each checklist item by saying ‘check’. The B747 cockpit was first mapped via the depth camera to obtain a rough 3D model of the physical scene. Those cues use the spatial mapping technology of Hololens, and each highlight (yellow circles and spatial text shown in Fig. 1b) is then precisely positioned according to this model with the Unity editor (Fig. 3b). The AR checklist needs to be calibrated to position the highlights precisely to the physical objects (Flap, Landing Gear, Speedbreak and Landing Light Button). Calibration is achieved thanks to the Vuforia Engine: the user must scan a QR code located near the throttle levers to begin using the app (Fig. 3c). 3.4. Research Design The experiment was run on the flight simulator with a representative model of the Boeing 747 − 200 flight deck. It was comprised of functioning flight controls, stick-shaker stall warning, over-speed alerts, primary flight and navigation displays, and landing gear levers, among other standard controls. The simplified overhead panel was comprised of light switches, engine fire emergency levers and engine ignition switches (Fig. 3a). The scenario was based on an Instrument Landing System (ILS) final approach from 2,000 ft and eight nautical miles (NM) from the airfield. Participants executed an ILS pre-landing checklist by interacting with the AR device during the final approach and landing. The procedures for all participants were as follows: (1) complete the demographic data including age, gender, qualifications, type hours and total flight hours (five minutes); (2) complete a briefing regarding the purpose of the study and how to use a HoloLens AR device for human-computer interactions (10 minutes); (3) participant seated in the B-747 simulator to calibrate the AR device by placing visual gaze cursor on the QR code with a blue indicator confirming calibration completed (five minutes); (4) familiarisation with the pre-landing checklists on flight simulator (ten minutes); (5) briefing on the AR checklist software, with a detailed explanation of how the item can be highlighted by gesture control and voice control (ten minutes); (6) performing a landing using HoloLens AR gesture-command (five minutes) and AR voice-command (five minutes) - order counterbalanced between participants; (7) post-trial completion of SUS and QUIS based on AP applications and provision of additional descriptive open-end feedbacks (20 minutes). Each participant took approximately 70 minutes to complete the experiment. 3.5. Data Analysis To investigate system usability and user interface satisfaction, respective post-trial SUS and QUIS scores were analysed using separate repeated measures general linear models (GLM). The SUS (Brooke ,1996) subjective measure that is widely applied in HCI evaluation (Lewis, 2018; Rudi et al., 2020) that measures system usability on 10 positively and negatively worded items requiring a five-point Likert scale response (strongly disagree (1) to strongly agree (5). To aid interpretation, and comparison to published normative datasets (Bangor et al., 2008; Sauro & Lewis, 2016), SUS dimension scores are transformed into percentile scores add allocated so-called usability gradings (A:>89; B: 80–89; C: 70–79; D: 60–69; and F < 60). QUIS subscale Cronbach’s Alphas were: Overall (.931), Screen (.781), System Information (.749), Learning (.725), System Capabilities (.802) and Usability and UI (.807). Both analyses included within-subject factors for AR interaction modality (2 levels: voice-command / gesture-command) and for subscale dimension (3-levels for SUS and 6-levels for QUIS). Bonferroni corrected post-hoc comparisons were used to investigate significant main effects and interactions that included the AR interaction modality term. The assumption for normally distributed data was checked by graphically using Q-Q plots. Greenhouse-Geisser corrections were required to address violations of sphericity. 4. Results 4.1. Sample Characteristics Participants evaluated the system usability and user interface satisfaction of the AR device in flight operations using SUS and QUIS, shown in Table 1 . The dependent variables of SUS are divided into Usability and Learnability; QUIS comprises overall user reactions, screen visibility, system information, learning factors, system capabilities, usability, and user interface. There was no outlier observed by boxplots. Based on the visual examination of the Q-Q plots, the SUS and QUIS data on two AR interactive modes were distributed normality. Table 1 The participants’ responses to AR interaction modes between Gesture-command and Voice-command using SUS and QUIS Interaction mode N M SD Usability Gesture-command 40 40.70 20.13 Voice-command 40 72.11 18.02 Learnability Gesture-command 40 45.00 27.42 Voice-command 40 58.13 26.79 SUS Total score Gesture-command 40 41.56 18.85 Voice-command 40 69.31 17.78 Overall reaction Gesture-command 40 4.07 1.68 Voice-command 40 6.90 1.42 Screen visibility Gesture-command 40 6.92 1.34 Voice-command 40 7.61 1.07 Voice-command 40 7.78 1.07 System information Gesture-command 40 6.91 1.26 Voice-command 40 7.57 0.96 Learning Gesture-command 40 6.09 1.55 Voice-command 40 7.45 0.89 System capabilities Gesture-command 40 4.87 1.90 Voice-command 40 6.33 1.81 Usability and user interface Gesture-command 40 6.49 1.83 Voice-command 40 7.35 1.62 QUIS Total score Gesture-command 40 5.89 1.03 Voice-command 40 7.20 0.84 4.2. AR Interactive Modes Impacted on System Usability SUS post-trial scores are presented in Fig. 4 alongside the normative grading bands for usability based on the work by Bangor et al. ( 2008 ) to aid interpretation. Notably, gesture-command interaction usability scores were deemed to be far below average (68%) on all 3 SUS dimensions according to SUS normative data (Bangor et al., 2008 ) – Usability (mean: 40.70, 95CI: 34.26–47.14, grade: “F”), Learnability (mean: 45.00, 95CI: 36.23–53.77, grade: “F”) and SUS total (mean: 41.56, 95CI: 35.53–47.59, grade: “F”). In contrast, whilst the usability of voice-commands interactions was rated below average for Learnability (mean: 58.13, 95CI: 49.56–66.69, grade: “F”), the scores were above average for Usability (mean: 72.11, 95CI: 66.35–77.87, grade “C”) and SUS total (mean: 69.31, 95CI: 63.63–75.00. grade: “D”). The GLM analysis revealed both a main effect for AR interaction modality ( F (1, 39) = 27.429.653, p < .001, η 2 = .413) and an AR interaction modality by dimension interaction ( F (2, 78) = 132.811, p < .001, η 2 = .457). The post-hoc tests showed SUS scores were higher for the voice-command modality condition compared to the gesture-command condition across SUS dimensions. In descending order of mean difference size: usability (mean diff: 31.406, p < .001, 95%CI: 20.961–41.851), total (mean diff: 27.750, p < .001, 95%CI: 17.943–37.557) and learnability (mean diff: 13.125, p = .005, 95%CI: 4.304–21.946). Therefore, the “H 1 : there is a significant difference in system usability between voice-command and gesture-command with AR applications during Instrument Landing” was supported. 4.3. AR Interactive Modes Impacted User Interface Satisfaction QUIS post-trial scores are presented in Fig. 5 . Across all user satisfaction dimensions visual-command interaction was rated higher than gesture interaction. The GLM analysis revealed both a main effect for AR interaction modality ( F (1, 39) = 68.885, p < .001, η 2 = .639) and an AR interaction modality by dimension interaction ( F (5, 71.409) = 16.916, p < .001, η 2 = .303). The post-hoc tests showed QUIS scores were higher for the voice-command modality condition compared to the gesture-command condition across all dimensions. In descending order of mean difference size: Overall Reaction (mean diff: 2.829, p < .001, 95%CI: 2.124–3.534), System Capabilities (mean diff: 1.467, p < .001, 95%CI: 0.898–2.035), Learning (mean diff: 1.350, p < .001, 95%CI: 0.899–1.801), Usability (mean diff: 0.862, p < .001, 95%CI: 0.443–1.282), System Information (mean diff: 0.658, p < .001, 95%CI: 0.310–1.007) and Screen Visibility (mean diff: 0.525, p = .005, 95%CI: 0.168 – .882). Therefore, the “H 2 : there was a significant difference in user interface satisfaction (QUIS) between voice-command and gesture-command with AR applications during Instrument Landing” was supported. 4.4. Qualitative Feedback of AR Application All participants had never used an AR device in the flight simulator previously. The results of current research demonstrate that participants preferred AR voice command over gesture command due to the intuitive design of vocal interaction, low perceived workload and less distraction while flight operations engaging two hands, but it might have potential risks to communicating with ATCOs and distraction while audio alerts activated. The use of the air-tap gesture with cursor control can be held responsible for the high physical and mental demands, as participants needed to precisely move their heads and hands simultaneously to get their hand gestures recognized by the cameras of HoloLens. It is very interesting to find participants’ subjective descriptive feedback on strengths and weaknesses in both voice-command and gesture-command (Table 2 ). Participants felt a level of frustration with the “air-tap”, which increased their physical loads while interacting with the AR device. There is a rising argument about the primary task in the flight deck implementing the AR system. Is interacting with AR the primary task, or is interacting with the flight deck instrument the primary task? Professional pilots mentioned that AR applications could be a great tool for training. They speculated cadet pilots could learn flow-checks and scanning patterns of checklist execution more easily by augmented visualization and audio cues superimposed in the flight deck environment. Table 2 The Summaries of Participants’ Descriptive Feedback on AR Application between Voice-command and Gesture-command for SPO Strengths Voice-command Gesture-command 1. Allow pilots hand-free for multiple tasks using intuitive design of trigger actions by speech 2. Enhance pilots’ operational capacity to switch between the primary tasks and secondary tasks 3. Increases the accessibility of control nuts/buttons and reduces physical workload 4. Provide immediate feedback and guidance under time-pressure scenarios 5. Enable more human-centric design of AR/MR interactions using natural language 1. Not affected by noise environment while multiple alerts activated by system failures 2. Not relying on speech recognition without the accent, language and voice clarity issues 3. Gesture commands are more discreet than voice commands 4.No disruption in communication with ATCOs Weakness 1. Voice interaction mode can be less effective in noisy environments while multiple audio alerting activated simultaneously 2. AR might have difficulties understanding participants' unique accents under time-pressure scenarios 3. The available voice-commend are very limited and restrict the functions of AR applications 4 Pilot might need to learn and adjust his/her normal speech pattern to be recognized by the AR system 5. There was a potential risk of disrupting the communication with ATCO for critical flight information 1. Use gesture commands can lead to arm and hand physical fatigue 2. Hololens with narrow field-of-view created difficulties on triggering the actions 3. Pilot lack of additional inputs (hands) to perform multiple task during take-off/landing and emergency system failures 4. The effectiveness of air-tap can be diminished in poor illuminating conditions of flight deck such as night flight 5. Some pilots struggle to learn the air-tap to command the AR device 6. Must fixate on the gesture, which shared more cognitive resources inducing workload 5. Discussion The application of augmented reality in training is increasingly widespread. These innovative technologies allow users to interact with the real world and the digital world, requiring minimum action between the user and the device. Using voice commands or very simple hand gestures, users can view the necessary information while continuing other tasks that require the use of their hands (Escalada-Hernandez et al., 2024 ). It is well known that monitoring and cross-checking issues of PF and PM still occur in the contemporary two-pilot flight deck configuration (Harris, 2007 ). Pilots' checklist non-compliance often results from the complex human-machine interfaces and human-computer interaction under multiple tasks in the modern flight deck. Therefore, in implementing an AR device in flight operations, system designers must ensure that the system is developed with consideration of human-centric interactive modes to improve pilots’ monitoring and operational performance. The results from the data analysis demonstrated that the AR voice-command mode was rated significantly higher in system usability and user interface applicability than the AR gesture-command mode (Table 1 ). These findings can provide scientific evidence to advance AR design to support pilot-monitoring tasks for future single-pilot operations. An intuitive and human-centric design AR system with multimodal inputs to achieve the goal of system control and information processing is used based on the growth of accuracy and efficiency of gesture recognition and voice synthesis technologies (Bilius & Vatavu, 2020 ). 5.1 Semantic AR Interacting Modes Consistent with Human-Centric Design The system control methods of gesture command and voice input have been widely used in the real-world interaction context of AR experiences. The current research illustrated that AR gesture-command interaction of ‘air-tap’ is rated significantly lower than voice-command in usability, learnability, and SUS scores (Table 1 & Fig. 4 ). To successfully commence an ‘air-tap’ gesture, participants must position their thumb and index finger in front of the HoloLens camera and then pinch the thumb and index finger together. However, due to the limited field of view of the HoloLens and technical issues of motion recognition, the activation of each procedure item could take a longer time to hinder the interaction process and downgrade participants’ rating of system usability. The gesture-command required participants to focus on SOPs and initiate an ‘air-tap’ precisely on time, and the cursor was positioned well to execute the specific action of each checklist. Adding these tasks alongside the high mental and physical task loads of operating a complex aircraft may increase the overload of the working memory. Currently, the primary input modality in the flight deck is the unimodal hand haptic inputs on the control inceptors. The voice-command interaction provided a multimodal interface permitting the integration of multiple sensory channels (sight, hearing, and touch) with higher flexibility and reliability to meet the task needs of different flight scenarios. Turk ( 2014 ) indicated that operators would prefer multimodal interaction over unimodal alternatives as it significantly enhances task efficiency and cognitive information processes. Besides the naturalness and intuition characteristics, the voice-command interactive mode has substantial advantages in memorability and learnability. The semantic interaction of AR voice-command mode will be activated by simple words presented in the interface, such as ‘check’ and ‘next’, to perform each procedure item in the pre-landing checklist. Compared to the ‘air-tap’ gesture, which is difficult to perform precisely, the voice-command mode is learnable and applicable in dynamic flight scenarios without the distractions of pilots’ manual flight controlling and visual monitoring. The voice-command AR interactive mode requires little effort to achieve simple, accurate, and safe operations in flight scenarios. Furthermore, familiarity is a critical factor affecting learnability in the human-computer interaction process to provide self-explanatory cues for understanding the functions that the system offers (Green & Eklundh, 2003 ). Participants’ familiarity with voice-command mode had a significant influence on their preferences and learnability ratings, as voice-command is intuitive and learnable when compared with gesture-command. User experience can be seen as the holistic extension of system usability measures, which covers additional details about human-computer interaction from hedonic and emotional aspects (Hassan & Galal-Edeen, 2017 ). In the current study, the QUIS assessments can reflect the interactive experience and user satisfaction with the utilitarian functions and experiential attributes of the AR gesture- and voice-command interfaces (Chung & Sahari, 2015 ). The suggested that the QUIS average score and six sub-dimension ratings on the AR voice-command mode are significantly higher than the gesture-command. Previous research demonstrated that, compared to the traditional point-and-click style interface, the voice-command dialogue system could enhance the emotional effectiveness of the VR interactive narratives to improve the user experience and interface satisfaction (Osking & Doucette, 2019 ). Furthermore, in the current study, voice input especially has the advantage of interface traversals that can help participants cut through multiple steps through the multimodal process to decrease the task completion time. This point can be confirmed by two rating items of the learning and system capacities sub-dimensions in the QUIS related to the logical steps and response to complete: the AR voice-command mode allowed participants to follow each step to finish the pre-landing checklist operations with a better logical sequence and faster reaction time compared to the ‘air-tap’ interaction (Table 1 & Fig. 5 ). The fluent interaction process and significant time saving also provide a positive emotional effect on the interactive experience (Microsoft, 2022 ). In the sub-dimensions of the overall user reactions, participants showed a significant preference for the voice-command AR interface over the gesture-command mode due to the higher system functionality and accessibility, as well as better interaction experience with voice input. Also, the higher ratings in the learning and system capacities revealed the advantages of the voice-command mode in a straightforward and understandable control manner with emotional effectiveness and operational efficiency. The hand-gesture-command interaction during the approach can cause distraction and frustration while the participant’s attention is entirely focused on flight operations plus executing the ‘air-tap’ itself, downgrading the user experience. Furthermore, there were three dimensions related to visual display and information presentation designs: screen visibility, system information, and usability and user interface. These three dimensions are supposed to provide the same information with the same level of visual presentation and have theoretically no difference between the voice-command and ‘air-tap’ interactive modes. However, the results still demonstrated significant differences between the two interactive modes in these sub-dimensions. One possible explanation might be that the participants are biased on the operational experience of the gesture-command interaction, and they tend to mark lower on the user interface satisfaction compared to the voice-command mode. Such subjective bias can be concluded as an emotional response to influence participants’ operational experience and engagement (Shepherd & Cardon, 2009 ). 5.2 Voice Command Increasing Pilot’s Operational Capacity for Multiple Tasks Participants rated higher on AR voice-command mode generally also appreciated the augmented visualisation information and procedural adherence instructions through an overlayed virtual display based on their qualitative feedback (Table 2 ). Previous research has demonstrated that directional cues in AR may have a beneficial effect on performance. This has primarily been established in the context of tasks requiring navigation or spatial orientation (Peereboom et al., 2024 ). The challenge with flight operations is that various tasks and SOPs must be executed precisely in a specific time frame. In the current study, the AR interface not only provided relevant checklist items required during the instrument landing scenarios but also allowed augmented visualisation cues of visually blended virtual checklists within the physical environment of the flight deck (Fig. 6). These augmented visualisation instructions directed pilots’ attention and action to perform each required procedure in a simple and intuitive workflow through the voice-command interaction to improve the checklist completion accuracy and speed for single-pilot operations. With the support of the AR voice-command checklist, the pilot-flying can hold the control column by the left hand, simultaneously operate the flap to 30 degrees (Fig. 6a), place the landing gear down (Fig. 6b), move the speed-brakes lever arm (Fig. 6c), switch landing lights on (Fig. 6d) as well as any additional interactions required with the flight deck instruments by the right hand following the AR instructions. In this process, the pilot can simply say ‘check’ to go through every item in the pre-landing checklist without additional hand movements or haptic interactions. Furthermore, via the AR voice-command mode while performing SOPs, the pilot’s monitoring performance, situation awareness, and working memory capacity will be enhanced. Therefore, the AR voice-command interactive system can be further developed and generalised in flight procedures training of various segments in flight operations to improve trainees’ working memory, in-flight decision, and operational skills (Squires, 2017 ). The AR interactive technology has the potential to provide adequate integrated information and intuitive human-computer interaction without attentional distraction due to head-down observations of instruments (Kim & Grabbard, 2019; Lu et al., 2020 ). However, there is a significant negative association of the SUS total scores between the voice-command mode and ‘air-tap’, revealing an exact opposite preference in system usability of different interactive modes of the in-flight AR systems. The user interface and human-computer interaction design of AR applications are critical factors in influencing system performance and human cognitive performance; if the system design does not follow human-centred principles, undesirable difficulties and overloaded cognitive demands will be added for the end-users (Pane et al., 2002 ). Based on the collected data analysis, it can be concluded that the in-flight AR technology can provide sufficient information with an intuitive multimodal interface. Future research shall investigate the integration of the Large Language Model (LLM) with AR for semantic interaction of voice command, which may provide a more intuitive human-centric design for pilots to execute multiple tasks precisely. 6. Conclusion The semantic voice-command interaction on an AR device has high system usability and user interface satisfaction, which can increase pilots’ monitoring performance and reduce physical task loads. AR technology blends virtual checklists with the real operational environment and provides guidance cues to direct pilots’ attention using intuitive voice-command. The use of the voice as an input for a multimodal AR device to execute intended actions can facilitate human-centred design concepts on the flight deck. There are some limitations related to voice recognition, such as a limited display field of view and obstacles from augmented vision superimposed in the flight deck that must be improved in the future. However, the benefits of applied AR voice command in flight operations and training still present promising opportunities for further research and development. AR applications in flight operations need to be assessed carefully so that all stakeholders, pilots, avionic engineers, system designers, manufacturers, and regulators can be satisfied with the potential benefits and associated risks. The current research has demonstrated that flight deck automation can be integrated with voice-command AR applications, which could enhance the pilots’ procedural compliance and monitoring performance in the human-centric flight deck. However, there are still several limitations of an AR application in flight operations which may require further research and development. The first issue is the efficiency of voice recognition, as some participants experienced various degrees of frustration due to a lack of response to some unique accents of spoken English. The second issue was that ‘air-tap’ recognition required more accuracy for the identification of the users’ intended gesture. Some participants were frustrated while trying to make commands using an ‘air-tap’ that was not recognised by the camera due to the limited field of view on the HoloLens. The third issue is the readability of virtual objects (text) on the HoloLens display, which may potentially create blind spots when targets are located outside the field of view or text superimposed by other visual elements. These issues may be improved with the new generation of VR/AR/MR technologies, which have integrated real-time eye tracking with two-handed, fully articulated gesture control technology and a wider field of view display. The eye-tracking and hand-tracking functions will have the potential to facilitate AR gaze control and hand control as additional input for single-pilot operations. In future studies, objective measurements of the pilot’s operational performance will be conducted to maximise AR applications to support single-pilot flight deck design. Declarations Declaration of interests ☒ The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Author Contribution Wen-Chin Li 's contributions involved the design of research, data collection, and writing of the paper.Jingyi Zhang's contributions included the literature review, data analysis, writing of the result section, and check the citations and references.James Blundel's contributions included revised the literature, double check the data analysis, revised the result section and provided relevant references for the literature review and discussion.Samuel Court's contributions involved in the AR system design, Applications of Flight Deck HCI for Voice-command and gesture-command, 3-D scenarios building in Hololens.Dujuan Sevillian's contributions included writing SPO and HMI design for AR application in flight operations, literature review and discussion revision. Acknowledgement This article is based on work supported by the UK EPSRC Grant: “Digital Toolkit for Optimisation of Operators and Technology in manufacturing partnerships” (Grant number EP/R032718/1) and the Higher Education Innovation Funding (HEIF 2018–2019). The authors would like to express our appreciation to Tim Bord, Bikram Thapa, Zepu Yan and all participants for their contributions to this project. Data Availability Data will be Published in Cranfield University CORD Data Repository References Aromaa, S., Väätänen, A., Aaltonen, I., Goriachev, V., Helin, K., & Karjalainen, J. (2020). Awareness of the real-world environment when using augmented reality head-mounted display. Applied ergonomics, 88, 103145. https://doi.org/10.1016/j.apergo.2020.103145 Bagassi, S., De Crescenzio, F., Piastra, S., Persiani, C. A., Ellejmi, M., Groskreutz, A. R., & Higuera, J. (2020). Human-in-the-loop evaluation of an augmented reality based interface for the airport control tower. Computers in Industry, 123, 103291. https://doi.org/10.1016/j.compind.2020.103291 Bailey, R. E., Kramer, L. J., Kennedy, K. D., Stephens, C. L., & Etherington Rockwell-Collins, T. J. (2017). An Assessment of Reduced Crew and Single Pilot Operations in Commercial Transport Aircraft Operations. IEEE/AIAA 36th Digital Avionics Systems Conference (DASC), 2017. https://ntrs.nasa.gov/search.jsp?R=20170009542 Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. Intl. Journal of Human–Computer Interaction, 24(6), 574-594. https://doi.org/10.1080/10447310802205776 Bilius, L. B., & Vatavu, R. D. (2020). A multistudy investigation of drivers and passengers’ gesture and voice input preferences for in-vehicle interactions. Journal of Intelligent Transportation Systems, 25(2), 197-220. https://doi.org/10.1080/15472450.2020.1846127 Blundell, J., & Harris, D. (2023). Designing augmented reality for future commercial aviation: a user-requirement analysis with commercial aviation pilots. Virtual Reality. https://doi.org/10.1007/s10055-023-00798-9 Boy, G. A. (2014). Requirements for Single Pilot Operations in Commercial Aviation: A First High-Level Cognitive Function Analysis. Requirements for Single Pilot Operations in Commercial Aviation: A First High-Level Cognitive Function Analysis. Complex Systems Design and Management International Conference, Paris, France. Brooke, J. (1996). SUS-A quick and dirty usability scale. In: Jordan P.W., Weerdmeester B., Thomas A., McLelland I.L. (Eds.), Usability Evaluation in Industry (pp. 189-194). CRC Press. https://doi.org/10.1201/9781498710411-35 Brown, C., Hicks, J., Rinaudo, C. H., & Burch, R. (2021). The use of augmented reality and virtual reality in ergonomic applications for education, aviation, and maintenance. Ergonomics in Design, 1-9. https://doi.org/10.1177/10648046211003469 Burian, B. K. (2006). Design guidance for emergency and abnormal checklists in aviation. In Proceedings of 2006 Human Factors and Ergonomics Society Annual Meeting, SAGE Publications (pp. 106-110). Los Angeles. https://doi.org/10.1177/154193120605000123 Carroll, M., & Dahlstrom, N. (2021). Human computer interaction on the modern flight deck. International Journal of Human–Computer Interaction, 37(7), 585-587. https://doi.org/10.1080/10447318.2021.1890495 Chan, W. T. K., & Li, W. C. (2021). Consideration of cultural differences in future workplace design for single pilot operations. Contemporary Ergonomics & Human Factors, 64-71. Chin, J. P., Diehl, V. A., & Norman, K. L. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 213-218). https://doi.org/10.1145/57167.57203 Chung, T. K., & Sahari, N. (2015). Utilitarian or experiential? An analysis of usability questionnaires. International Journal of Computer Theory and Engineering, 7(2), 167-171. https://doi.org/10.7763/IJCTE.2015.V7.950 Detjen, H., Faltaous, S., Pfleging, B., Geisler, S., & Schneegass, S. (2021). How to Increase Automated Vehicles’ Acceptance through In-Vehicle Interaction Design: A Review. International Journal of Human-Computer Interaction, 37(4), 308–330. https://doi.org/10.1080/10447318.2020.1860517 Dey, A., Billinghurst, M., Lindeman, R. W., & Swan, J. E. (2018). A systematic review of 10 Years of Augmented Reality usability studies: 2005 to 2014. Frontiers Robotics AI, 5(APR), 329739. https://doi.org/10.3389/FROBT.2018.00037/BIBTEX Dill, E. T., & Young, S. D. (2015). Analysis of Eye-Tracking Data with Regards to the Complexity of Flight Deck Information Automation and Management-Inattentional Blindness, System State Awareness, and EFB Usage. In Proceedings of 15th AIAA Aviation Technology, Integration, and Operations Conference (pp. 1-26). AIAA. https://doi.org/10.2514/6.2015-2901 Escalada-Hernandez, P., Soto-Ruiz, N., Ballesteros-Egüés, T. et al. (2024). Usability and user expectations of a HoloLens-based augmented reality application for learning clinical technical skills. Virtual Reality 28, 102 (2024). https://doi.org/10.1007/s10055-024-00984-3 Green, A., & Eklundh, K. S. (2003). Designing for learnability in human-robot communication. IEEE Transactions on industrial electronics, 50(4), 644-650. https://doi.org/10.1109/TIE.2003.814763 Gregoriades, A., & Sutcliffe, A. (2018). Simulation-based evaluation of an in-vehicle smart situation awareness enhancement system. Ergonomics, 61(7), 947-965. https://doi.org/10.1080/00140139.2018.1427803 Harris, D. (2007). A human‐centred design agenda for the development of single crew operated commercial aircraft. Aircraft Engineering and Aerospace Technology, 79(5), 518-526. https://doi.org/10.1108/00022660710780650 Harris, D. (2023). Single-pilot airline operations: Designing the aircraft may be the easy part. The Aeronautical Journal, 127(1313), 1171-1191. https://doi.org/10.1017/aer.2022.110 Hassan, H. M., & Galal-Edeen, G. H. (2017). From usability to user experience. In 2017 International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS) (pp. 216-222). IEEE. https://doi.org/10.1109/ICIIBMS.2017.8279761 He, Z., Chang, T., Lu, S., Ai, H., Wang, D., & Zhou, Q. (2017). Research on human-computer interaction technology of wearable devices such as augmented reality supporting grid work. Procedia Computer Science, 107, 170-175. https://doi.org/10.1016/j.procs.2017.03.074 Helin, K., Kuula, T., Vizzi, C., Karjalainen, J., & Vovk, A. (2018). User experience of augmented reality system for astronaut's manual work support. Frontiers in Robotics and AI, 5, 1-10. https://doi.org/10.3389/frobt.2018.00106 IATA. (2023). Global Outlook for Air Transport: High Resilient, Less Robust. International Air Transport, Montreal, Quebec, Canada. https://www.iata.org/en/iata-repository/publications/economic-reports/global-outlook-for-air-transport----june-2023/ ICAO (2019). Addressing the shortage of pilots and aviation professionals: a global approach to a global problem. ICAO Working Paper, A40-WP/239. International Civil Aviation Organization, Montreal, Quebec, Canada. https://www.icao.int/Meetings/a40/Documents/WP/wp_239_en.pdf ICAO. (2022). An approach to new operational concepts involving extended minimum crew operations and single-pilot operations. ICAO Working Paper, A41-WP/101. Technical Commission, International Civil Aviation Organization, Montreal, Quebec, Canada. https://www.icao.int/Meetings/a41/Documents/WP/wp_101_en.pdf Jorna, P. G. A. M. (1997). Human Machine interfaces for ATM: objective and subjective measurements on human interactions with future Flight deck and Air Traffic Control systems. In FAA/Eurocontrol ATM R&D Seminar, Paris, France. Jetter, J., Eimecke, J., & Rese, A. (2018). Augmented reality tools for industrial applications: What are potential key performance indicators and who benefits? Computers in Human Behavior, 87, 18–33. https://doi.org/10.1016/j.chb.2018.04.054 Kim, H., & Gabbard, J. L. (2019). Assessing Distraction Potential of Augmented Reality Head-Up Displays for Vehicle Drivers. Human Factors, 2019, 1-14. https://doi.org/10.1177/0018720819844845 Korkiakoski, M., Alavesa, P., & Kostakos, P. (2024). Preference in Voice Commands and Gesture Controls With Hands-Free Augmented Reality With Novel Users. IEEE Pervasive Computing, 23(1), 18–26. https://doi.org/10.1109/MPRV.2024.3364541 Lee, M., Billinghurst, M., Baek, W., Green, R., & Woo, W. (2013). A usability study of multimodal input in an augmented reality environment. Virtual Reality, 17, 293–305. https://doi.org/10.1007/s10055-013-0230-0 Lewis, J. R. (2018). The system usability scale: past, present, and future. International Journal of Human–Computer Interaction, 34, 577-590. https://doi.org/10.1080/10447318.2018.1455307 Li, W.-C., Horn, A., Sun, Z., Zhang, J., & Braithwaite, G. (2020). Augmented visualisation cues on primary flight display facilitating pilot's monitoring performance. International Journal of Human-Computer Studies, 135, 1-10. https://doi.org/10.1016/j.ijhcs.2019.102377 Li, W. C., Zhang, J., Court, S., Kearney, P., & Braithwaite, G. (2022). The influence of augmented reality interaction design on Pilot's perceived workload and situation awareness. International Journal of Industrial Ergonomics, 92, 103382. https://doi.org/10.1016/j.ergon.2022.103382 Lim, Y., Bassien-Capsa, V., Ramasamy, S., Liu, J., & Sabatini, R. (2017). Commercial airline single-pilot operations: System design and pathways to certification. IEEE Aerospace and Electronic Systems Magazine, 32(7), 4-21. https://doi.org/10.1109/MAES.2017.160175 Lu, S., Sanchez Perdomo, Y. P., Jiang, X., & Zheng, B. (2020). Integrating Eye-Tracking to Augmented Reality System for Surgical Training. Journal of Medical Systems, 44(11). https://doi.org/10.1007/s10916-020-01656-w Meister, P., Miller, J., Wang, K., Dorneich, M. C., Winer, E., Brown, L. J., & Whitehurst, G. (2022). Designing Three-Dimensional Augmented Reality Weather Visualisations to Enhance General Aviation Weather Education. IEEE Transactions on Professional Communication, 65(2), 321-336. https://doi.org/10.1109/TPC.2022.3155920 Microsoft. (2021). HoloLens (1st gen) Hardware. https://learn.microsoft.com/en-us/hololens/hololens1-hardware#device-specifications Microsoft. (2022). Microsoft HoloLens: Voice Input. https://docs.microsoft.com/en-us/windows/mixed-reality/design/voice-input Moesl, B., Schaffernak, H., Vorraber, W., Braunstingl, R., & Koglbauer, I. V. (2023). Multimodal Augmented Reality Applications for Training of Traffic Procedures in Aviation. Multimodal Technologies and Interaction, 7(1), 1-17. https://doi.org/10.3390/mti7010003 Myers, P. L., & Starr, A. W. (2021). Single pilot operations in commercial cockpits: background, challenges, and options. Journal of Intelligent & Robotic Systems, 102(1), 19. https://doi.org/10.1007/s10846-021-01371-9 Nizam, S. M., Abidin, R. Z., Hashim, N. C., Lam, M. C., Arshad, H., & Majid, N. A. A. (2018). A review of multimodal interaction technique in augmented reality environment. International Journal of Advanced Science Engineering Information Technology, 8(4-2), 1460-1469. Osking, H., & Doucette, J. A. (2019). Enhancing emotional effectiveness of virtual-reality experiences with voice control interfaces. In Proceedings of Immersive Learning Research Network: 5th International Conference (pp. 199-209). Springer, Cham. https://doi.org/10.1007/978-3-030-23089-0_15 Pane, J. F., Myers, B. A., & Miller, L. B. (2002). Using HCI techniques to design a more usable programming system. In Proceedings of IEEE 2002 Symposia on Human Centric Computing Languages and Environments (pp. 198-206). IEEE. https://doi.org/10.1109/HCC.2002.1046372 Peereboom, J., Tabone, W., Dodou, D. et al. Head-locked, world-locked, or conformal diminished-reality? An examination of different AR solutions for pedestrian safety in occluded scenarios. Virtual Reality 28, 119 (2024). https://doi.org/10.1007/s10055-024-01017-9 Riener, A., Jeon, M., Alvarez, I., & Frison, A. K. (2017). Driver in the Loop: Best Practices in Automotive Sensing and Feedback Mechanisms. 295–323. https://doi.org/10.1007/978-3-319-49448-7_11 Rudi, D., Kiefer, P., & Raubal, M. (2020). The instructor assistant system (iASSYST)-utilising eye tracking for commercial aviation training purposes. Ergonomics, 63(1), 61-79. https://doi.org/10.1080/00140139.2019.1685132 Sadri, S., Kohen, S., Elvezio, C., Sun, S., Grinshpoon, A., Loeb, G., Basu, N., & Feiner, S. (2019). Manipulating 3D anatomic models in augmented reality: Comparing a hands-free approach and a manual approach. Proceedings - 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2019, 93–102. https://doi.org/10.1109/ISMAR.2019.00-21 Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research. Morgan Kaufmann, Elsevier, Cambridge, MA, US. Schmid, D., & Stanton, N. A. (2020). Progressing Toward Airliners’ Reduced-Crew Operations: A Systematic Literature Review. In International Journal of Aerospace Psychology 30 (1–2), 1–24. Taylor and Francis Inc. https://doi.org/10.1080/24721840.2019.1696196 Shepherd, D. A., & Cardon, M. S. (2009). Negative emotional reactions to project failure and the self‐compassion to learn from the experience. Journal of Management Studies, 46(6), 923-949. https://doi.org/10.1111/j.1467-6486.2009.00821.x Spittle, B., Frutos-Pascual, M., Creed, C., & Williams, I. (2023). A Review of Interaction Techniques for Immersive Environments. IEEE Transactions on Visualization and Computer Graphics, 29(9), 3900–3921. https://doi.org/10.1109/TVCG.2022.3174805 Squires, D. R. (2017). Working Memory & Augmented Reality's Trajectory: A Literature Review of AR in Education, Online Learning, Workforce Training, and Working Memory Research. Journal of Educational Technology, 14(3), 55-63. https://doi.org/10.26634/jet.14.3.13860 Tran, T. H., Behrend, F., Fünning, N., & Arango, A. (2018). Single pilot operations with AR-glasses using Microsoft HoloLens. In Proceedings of 2018 IEEE/AIAA 37th Digital Avionics Systems Conference (pp. 1-7). IEEE. https://doi.org/10.1109/DASC.2018.8569261 Turk, M. (2014). Multimodal interaction: A review. Pattern recognition letters, 36, 189-195. https://doi.org/10.1016/j.patrec.2013.07.003 Wickens, C. D. (2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3(2), 159–177. https://doi.org/10.1080/14639220210123806 Wickens, C. D., & Carswell, C. M. (1995). The proximity compatibility principle: Its psychological foundation and relevance to display design. Human Factors, 37, 473–494. https://doi.org/10.1518/001872095779049408 Wickens, C. D., & Ward, J. (2017). Cockpit Displays of Traffic and Weather Information: Effects of 3D Perspective Versus 2D Coplanar Rendering and Database Integration. The International Journal of Aerospace Psychology, 27(1-2), 44-56. https://doi.org/10.1080/10508414.2017.1366270 Woodward, J., & Ruiz, J. (2022). Analytic review of using augmented reality for situational awareness. IEEE Transactions on Visualization and Computer Graphics, 29(4), 2166-2183. https://doi.org/10.1109/TVCG.2022.3141585 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 18 Jun, 2026 Read the published version in Virtual Reality → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4844391","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":338494005,"identity":"51215a34-1ff5-45dd-a87e-7eae3b7a30f9","order_by":0,"name":"Wen-Chin Li","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABDklEQVRIie3RsWrDMBCA4ROCy3K1V5lA8goqAZMpfRUbrxkydghFJuBMzdz3KHS2EaSL2q4eAiWLpw4NhkwlVA5th+LEHTvoR4sEH5wkAJfrn8ZBAnic583GR+BKHM+pgyDHqNkEGbC/EBsCySNRXUQ+3hb1bLYB7NGuvp5vhNdL0xLmE5AmbyfmOenfycoOdnEfmHUlkIrFGNYJyCfVSsJyGnKSuiEPTKG+QRFnAjAH+dI+WPj69k2oYuqgBQ63lhzOkJJ+CLI0s0SwzK785GBXZjrqW0L2kUdBurKE4sU4XiUUnLh+sDSXNX3oge/r7U7ttRgudVG+7ycDz0Ttk3316w+isx/pcrlcrq4+AVbrUqj6TKviAAAAAElFTkSuQmCC","orcid":"","institution":"Cranfield University","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Wen-Chin","middleName":"","lastName":"Li","suffix":""},{"id":338494007,"identity":"cacdbf9d-249f-426d-9de6-e93d72a1a8cf","order_by":1,"name":"Jingyi Zhang","email":"","orcid":"","institution":"Cranfield University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Jingyi","middleName":"","lastName":"Zhang","suffix":""},{"id":338494009,"identity":"54435535-8b9b-4223-896f-431171eedd84","order_by":2,"name":"James Blundell","email":"","orcid":"","institution":"Cranfield University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"James","middleName":"","lastName":"Blundell","suffix":""},{"id":338494011,"identity":"470ac705-42da-4573-b10b-3f7255fb0609","order_by":3,"name":"Samel Court","email":"","orcid":"","institution":"Cranfield University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Samel","middleName":"","lastName":"Court","suffix":""},{"id":338494012,"identity":"1d2f437d-1409-4212-9bb6-978651010de3","order_by":4,"name":"Dujuan Sevillian","email":"","orcid":"","institution":"National Transportation Safety Board","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Dujuan","middleName":"","lastName":"Sevillian","suffix":""}],"badges":[],"createdAt":"2024-08-01 20:38:19","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4844391/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4844391/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s10055-026-01411-5","type":"published","date":"2026-06-18T16:01:40+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":64568460,"identity":"6c7c3655-cd26-4823-8a29-78bc8ba69fa7","added_by":"auto","created_at":"2024-09-16 00:40:51","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":214176,"visible":true,"origin":"","legend":"\u003cp\u003eParticipant wearing a HoloLens (shown in the top left-hand inset) and the field of view while operating the Landing Gear (DOWN \u0026amp; GREEN) augmented checklist following UI instruction\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-4844391/v1/4da3417c30f0ee18aeb1ef2a.png"},{"id":64568461,"identity":"5cb21052-8a17-46fc-832a-7d62125a6598","added_by":"auto","created_at":"2024-09-16 00:40:51","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":323610,"visible":true,"origin":"","legend":"\u003cp\u003eThe SOP menus that use different colours to indicate the status of operations, green indicating executed, red for the current operational step, and grey for standby (2a); visual cues of guiding arrows and position indicator (2b); ‘Air-tap’ gesture used for AR interactive mode using yellow arrows for directing pilot’s attention to check Landing Light On (2c); pilot conducting gesture-command wearing HoloLens while performing ‘Speedbrakes On’ (2d).\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-4844391/v1/bd91dc67ed4abcda1d4a7bdb.png"},{"id":64568463,"identity":"9df385db-813c-445d-abd0-ebda980bce9d","added_by":"auto","created_at":"2024-09-16 00:40:51","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":333011,"visible":true,"origin":"","legend":"\u003cp\u003eThe flight deck display systems of a flight simulator for the experiment of instrument landing (3a); the 3D model of the cockpit with a positioned item highlighted in the Unity editor (3b); the user has to fixate a QR code located near the throttle to use the AR app for Instrument Landing on SPO (3c)\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-4844391/v1/089ba4303a1c5fe66a087409.png"},{"id":64569322,"identity":"99404714-4783-4f3e-8484-94f849fe7d35","added_by":"auto","created_at":"2024-09-16 00:48:51","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":39440,"visible":true,"origin":"","legend":"\u003cp\u003eEstimated marginal means for SUS subscale (usability and learnability) and total scores grouped by interaction modality. SUS normative grading bands from Bangor et al. (2008) are included. The dashed horizontal line represents the average SUS score - 68% - from the same normative datasets. Errors bars represent 95% confidence intervals.\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-4844391/v1/e4cd612c45a32278026cd007.png"},{"id":64568459,"identity":"b56ee003-7c40-4c4c-b967-883a3708ae4e","added_by":"auto","created_at":"2024-09-16 00:40:51","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":42632,"visible":true,"origin":"","legend":"\u003cp\u003eEstimated marginal means for QUIS subscale grouped by interaction modality. Errors bars represent 95% confidence intervals\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-4844391/v1/c4520a50ae4b40a000122862.png"},{"id":64569912,"identity":"aa36bdd0-94e8-4e2b-ae0f-898badf67f74","added_by":"auto","created_at":"2024-09-16 00:56:51","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":207348,"visible":true,"origin":"","legend":"\u003cp\u003eThe augmented visualisation cues to guide pilots in executing required actions to perform instrument landing procedures consisted of the following actions: move Flap to 30 Degree (figure 6a); put landing-gear down (figure 6b); move speed-brakes to ARM (figure 6c); switch landing light on (figure 6d)\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-4844391/v1/2acdb8282a8c6a1086fa514e.png"},{"id":112747459,"identity":"d7aab72d-0acb-47e8-9028-a5b4db87f6d7","added_by":"auto","created_at":"2026-06-22 16:25:43","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1636565,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4844391/v1/acf788b2-7d26-4d1f-8505-e9f6188a02e1.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Development of Multimodal Augmented Reality Application on the Flight Deck for Single-Pilot Operations: System Usability and User Interface Satisfaction","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eThe increasing worldwide demand for commercial air travel over the next 20 years will be dramatically increasing by an expected shortfall of 500,000 qualified pilots (IATA, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). To address this operational challenge, a concept being pursued by the aviation industry is to reduce the current two-pilot flight deck configuration to a single pilot \u0026ndash; known as of Single Pilot Operations (SPO). Multiple interrelated obstacles to the realisation of SPO exist including \u0026lsquo;buy-in\u0026rsquo; from relevant key stakeholders, such as regulators, the aviation professional community and the public, and the maintenance of operational safety tolerances related to pilots\u0026rsquo; workload and fatigue (Harris, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Schmid \u0026amp; Stanton, \u003cspan citationid=\"CR52\" class=\"CitationRef\"\u003e2020\u003c/span\u003e). The use of augmented reality (AR) technology has been proposed as a technological enabler of SPO to facilitate the new forms of system interaction that could be expressed in the SPO environment, while simultaneously mitigating increases in pilot workload (Tran et al., \u003cspan citationid=\"CR56\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). In the current commercial two-pilot flight deck, the pilot-flying (PF) is responsible for flying the aircraft to an approved flight plan, and the pilot-monitoring (PM) focuses on navigating, communicating, and monitoring PF\u0026rsquo;s operational behaviours. The PM may also support the PF by assisting with high-workload situations and taking control in emergency situations (Lim et al., \u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e2017\u003c/span\u003e). To meet the expected growth in air transportation and to further reduce operational costs, one active interest for airlines is the introduction of Single Pilot Operations (SPO), which would apply innovative technologies on the flight deck to facilitate minimum pilot intervention, allowing sufficient resting and management of workload to permit one pilot to be rostered to the flight (ICAO, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). Airlines and manufacturers have been forced to advance innovative technologies to provide sufficient pilots to enhance the cost-efficiency of training and flight operations (ICAO, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). It is also important to consider the decision-making aids that can be incorporated into the future flight deck to assist one pilot in managing all unexpected scenarios. Therefore, there is potential for the implementation of augmented reality (AR) technology to enhance the pilot\u0026rsquo;s monitoring performance, allowing a \u0026lsquo;head up, eyes out\u0026rsquo; mentality to be achieved. To this end - the current research explores the system usability and user satisfaction of interacting with AR devices within a simulated single pilot operating environment, whilst simultaneously investigating the influence of interaction modality type \u0026ndash; verbal versus gesture.\u003c/p\u003e"},{"header":"2. Related Work","content":"\u003cp\u003eThe move to single-pilot operations in commercial aviation is a significant step towards greater efficiency but comes with notable safety challenges. The advanced technologies of augmented reality (AR) with artificial intelligence (AI) have been widely used to increase operators\u0026rsquo; situation awareness by combining virtual objects/information in the physical environment, which in turn allows the operators to execute multiple tasks simultaneously (Gregoriades \u0026amp; Sutcliffe, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2018\u003c/span\u003e; Woodward \u0026amp; Ruiz, \u003cspan citationid=\"CR61\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). There is the potential for future flight deck design applying AR technology to minimise the space required to house critical information in the effective field of view for the pilot and allow real-time processing of flight data to better improve situational awareness throughout the flight (Li et al., \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Meister et al., \u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e2022\u003c/span\u003e).\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1. The Role of AR in Single-Pilot Operations\u003c/h2\u003e \u003cp\u003eThe trend in flight deck design has been on the approach of \u0026lsquo;de-crewing\u0026rsquo; over the past 70 years, with crew numbers from five (captain, co-pilot, flight engineer, navigator and radio operator) to two pilots (PF and PM) only. Aircraft manufacturers and avionics systems suppliers are developing advanced technology for such aircraft, centred upon the development of Intelligent Knowledge-Based Systems and adaptive automation (Harris, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Aviation human factors focus on human-centric design for the implementation of innovative technology on the flight deck (Boy, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2014\u003c/span\u003e). For information presentation, the visual saliency of task-critical objects is an overarching factor that should be considered to expand pilots\u0026rsquo; visual attention and enhance situation awareness (Dill \u0026amp; Young, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2015\u003c/span\u003e). The proximity compatibility principle (PCP) suggests integrating different sources of information in close spatial vicinity to facilitate operators using one gaze to catch all relevant information (Wickens \u0026amp; Carswell, \u003cspan citationid=\"CR59\" class=\"CitationRef\"\u003e1995\u003c/span\u003e; Wickens \u0026amp; Ward, \u003cspan citationid=\"CR60\" class=\"CitationRef\"\u003e2017\u003c/span\u003e). Pilots must monitor constantly changing parameters and select appropriate control modes by cross-checking various displays (Burian, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2006\u003c/span\u003e). Augmented visualisation design with salient mode transition on the optimised PFD can reduce pilots\u0026rsquo; cognitive efforts and perceived workload by eliminating the requirement to read text messages compared to a traditional PFD (Li et al., \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e2020\u003c/span\u003e). An interactive interface of datalink that provides information of air-ground communication displayed in a salient position can decrease pilots\u0026rsquo; cognitive efforts and task completion time (Jorna, \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e1997\u003c/span\u003e). Human-centred flight deck design can also facilitate the usage of high-level automation systems and support the implementation of single pilot operations to significantly improve cost efficiency (Harris, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2007\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eAdvanced technologies in flight deck design and artificial intelligence are great sources of driving SPO and improving aviation safety and efficiency (Chan \u0026amp; Li, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). These technological advancements are among several compelling reasons for the consideration of SPO in commercial air transport. To be feasible for SPO in commercial air transport, a higher level of automation than what currently exists needs to be developed and amalgamated with the models of the regulator\u0026rsquo;s proposed expectations (Myers \u0026amp; Starr, \u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Levels of automation would be inversely proportional to the human operator complement in the system in which one pilot would fly the aircraft, with the functions of the co-pilot being assigned to an augmented/automated system on the flight deck. Moreover, the human-centric flight deck design with innovative interactive modes can improve the pilot\u0026rsquo;s situation awareness and provide decision support (Carroll \u0026amp; Dahlstrom, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). The head-mounted display (HMD) and see-through function of AR allows virtual objects to be integrated into the operational environment to assist a pilot with retrieving flight parameters, checking standard operational procedures (SOPs) and controlling the aircraft flight path. Holographic AR glasses can offer an optimal solution to reduce operators\u0026rsquo; workload and stress levels by presenting critical information in a timely manner (Jetter et al., \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2018\u003c/span\u003e; Tran et al., \u003cspan citationid=\"CR56\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). The flight deck checklist has evolved over the years with various types and interactive modes. AR can add salient audio-visual pointers and feedback to distinguish between accomplished and non-accomplished procedures. A previous study demonstrated that the application of the voice-command AR checklist could improve pilots\u0026rsquo; situation awareness and reduce cognitive workload since augmented visualization cues do not require pilots to divert attention resources away from tasks in hand (Li et al., \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e2022\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Multimodal Interaction with AR\u003c/h2\u003e \u003cp\u003eWith advances in software and hardware development, multimodal AR that incorporates various modes of interaction is becoming more prevalent to everyday life and workplaces (Aromaa et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2020\u003c/span\u003e; Brown et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Two common interaction modes include gesture and voice control (He et al., \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2017\u003c/span\u003e; Korkiakoski et al., \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). The advantages of AR gesture-based interactions, compared to traditional touchscreen gesture interactions, is that they are not bound to a specific physical location and, if well designed and consist of a small command set, offer a natural interaction modality (Riener et al., \u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e2017\u003c/span\u003e). Similarly, speech control provides a naturalistic interaction medium as it is our primary mode of communication (Detjen et al., \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). In terms of task performance with the two interaction modalities, Lee et al., (\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e2013\u003c/span\u003e) demonstrated that using voice commands compared to gesture commands were superior in an abstract, self-paced, shape manipulation task. A recent structured review of 42 input modality studies between 2013 and 2020 by Spittle et al. (\u003cspan citationid=\"CR54\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) highlighted the task enhancement that voice commands can provide but stressed that this was largely task dependent. For instance, voice commands will be more advantageous for menu-selection based interactions whilst gesture interactions would be more useful for tasks that require spatial manipulation of virtual objects. The performance benefit of voice commands likely stems from offering a \u0026lsquo;hand-free\u0026rsquo; advantage during tasks which already entail a high degree of manual input \u0026ndash; a capability which was deemed important by the pilots who evaluated SPO concepts in the series of studies conducted by NASA (Bailey et al., \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2017\u003c/span\u003e). The advantage of voice-command is also in keeping with cognitive resource theory - Wicken\u0026rsquo;s multiple resource theory (Wickens, \u003cspan citationid=\"CR58\" class=\"CitationRef\"\u003e2002\u003c/span\u003e) stipulates that task performance will be superior when the attentional demands of a task are distributed across a wider range of sensory modalities. Thus, reducing the likelihood of encountering a \u0026ldquo;cognitive bottleneck\u0026rdquo;.\u003c/p\u003e \u003cp\u003eDespite the numerous studies that have demonstrated the performance benefits of voice command applications, there remain mixed results in terms of user preference. In some cases, users still prefer to use gestures instead of voice for certain tasks even when the performance is worse (Sadri et al., \u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e2019\u003c/span\u003e)., More specifically to aviation, there exists a general lack of usability research concerning the implementation of AR in flight operations (Blundell \u0026amp; Harris, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). The majority of AR usability research has been predominantly based on self-paced tasks across a range of application areas, such as education, entertainment, maintenance and medicine (Dey et al., \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). This is important in the context of using AR to support SPO, as the task environment where AR is likely to be of most benefit will be during an externally paced flight environment \u0026ndash; such as during the completion of pre-landing and emergency procedures (Li et al., \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Tran et al., \u003cspan citationid=\"CR56\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). The use of AR technology to provide pilots with emergency procedures by both providing instructional cues and overlapping the critical displays can improve pilots\u0026rsquo; response times and reduce the cognitive loads of performing tasks during high-workload scenarios. In the aviation industry, Bagassi et al. (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2020\u003c/span\u003e) proposed an innovative concept of air traffic control based on AR technology. It provides a heads-up view of the air traffic in low visibility conditions, which can reduce controllers' perceived workload, increase task performance and ensure operational safety. The multimodal AR-supported training for procedures compliance with Visual Flight Rules (VFR) demonstrated significant benefits in improving pilots\u0026rsquo; performance on visual detection, as well as training engagement and immersion (Moesl et al., \u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). The see-through AR device provides critical information in the dynamic environment to assist the pilot in processing mass information and making aeronautical decisions (ADM) in time-limited situations. AR technology has the potential to improve SOP compliance and cross-check between PF and PM. Multimodal AR proposed several innovative interaction techniques, such as sensor-based, audio-based, and visual-based modes, which can facilitate effective human-computer interaction and enhance user experience (Nizam et al., \u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e2018\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Motivation and Hypotheses\u003c/h2\u003e \u003cp\u003eTo successfully implement AR technology in flight operations, the system usability and user interface of AR should be assessed at the early stages of design. The SUS is distinguished into two factors of usability and learnability, which are significantly correlated with each other (Brooke, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e1996\u003c/span\u003e); and have been applied in the human-computer interaction domain (Meister et al., \u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). The other assessment tool is Questionnaires for User Interface Satisfaction (QUIS), designed to assess users\u0026rsquo; subjective satisfaction with specific aspects of the human-computer interface. The results of QUIS could help the system developer enhance the products\u0026rsquo; reliability and validity by means of satisfaction measurements (Chin et al., \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e1988\u003c/span\u003e). There are six sub-dimensions involved: overall user reactions, screen visibility, system information, learning, system capabilities, and usability and user interface. QUIS has been widely used to evaluate the interface satisfaction and user experience effectively of an innovative system (Helin et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). The application of AR technology on the flight deck requires pilots\u0026rsquo; feedback on performing tasks related to visual perception, attention distribution, and information processing. The aims of this study were to investigate both the system's usability and users\u0026rsquo; experiences while interacting with multimodal AR checklists (gesture-command vs voice-command). There are two hypotheses were proposed as follows:\u003c/p\u003e \u003cp\u003eH\u003csub\u003e1\u003c/sub\u003e: There is a significant difference in system usability (SUS) between voice-command and gesture-command with AR applications during instrument landing.\u003c/p\u003e \u003cp\u003eH\u003csub\u003e2\u003c/sub\u003e: There is a significant difference in user interface satisfaction (QUIS) between voice-command and gesture-command with AR applications during instrument landing.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. Methods","content":"\u003cdiv id=\"Sec7\"\u003e\n \u003ch2\u003e3.1. Participants\u003c/h2\u003e\n \u003cp\u003eForty participants, including seven females (17.5%) and 33 males (82.5%), were recruited from aviation research institutes with varied flight hours (M\u0026thinsp;=\u0026thinsp;122.70, SD\u0026thinsp;=\u0026thinsp;468.29) and familiar with operating the flight simulator used in the experiment. Their ages ranged between 21 and 55 years old (M\u0026thinsp;=\u0026thinsp;27.90, SD\u0026thinsp;=\u0026thinsp;8.09). A research proposal was approved by the research ethics committee before conducting the experiment, as this research involved collecting data from human participants. All participants signed a consent form which provided them with detailed information related to the experiment and were guaranteed the right to withdraw from the experiment at any stage. Data collected has been used and stored in accordance with Cranfield\u0026rsquo;s Research Ethics Policy and Management of Research Data Policy.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec8\"\u003e\n \u003ch2\u003e3.2. Augmented Reality Device\u003c/h2\u003e\n \u003cp\u003eThe AR device used in the experiment is a Microsoft HoloLens headset. These glasses comprise see-through holographic waveguides, two HD 16:9 light engines and built-in processors that can display holograms with a resolution of 1280 x 720 pixels per eye, a field of view of 30\u0026deg; \\(\\:\\times\\:\\) 17.5\u0026deg; and a refresh rate of 60 Hz. The symbols, fonts, colours, and text used in the UI design are optimised to improve pilots\u0026rsquo; attention and situation awareness (Fig.\u0026nbsp;1). The HoloLens comes with built-in sensors: an Inertial Measurement Unit (IMU), four environment understanding cameras, one depth camera, one HD video camera, four microphones and one ambient light sensor. Its audio output consists of two speakers located near the user\u0026rsquo;s ears that can emit spatial sound. The depth camera is used to carry out the user\u0026rsquo;s hand gesture recognition and spatial mapping of the surrounding environment. The user can extend or retract the headband and can slide the visor forward or backwards in order to wear the headset more comfortably (Microsoft, 2021).\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec9\"\u003e\n \u003ch2\u003e3.3. AR Applications for SPO\u003c/h2\u003e\n \u003cp\u003eThe applications of multimodal AR pre-landing checklist interaction modes consisting of voice commands and gesture commands have been developed for this trial. There are only five procedures for both gesture and voice commands on the HoloLens, including (1). Flap 30-Degree; (2). Landing Gear-DOWN; (3) Speedbrakes-ARM, (4). Landing Lights-ON, and (5) Back to Menu (Fig.\u0026nbsp;2a). Participants followed the standard operating procedures to control airspeed, altitude, descent rate and touchdown distance for safe operation. The AR checklist apps were developed with the Unity game engine and a combination of Microsoft Mixed Reality Toolkit (MRTK) and Vuforia Software Development Kit (SDK). Animated yellow arrow symbols provided visual guidance to participants regarding the position of the current checklist item. The animated yellow symbols always point towards the physical flight deck element of the relevant SOP and disappear when the user gets closer to the check item (top, Fig.\u0026nbsp;2b). The yellow circular target animations are check-item indicators that are overlaid at the position of the lever/switch to direct the user\u0026rsquo;s attention to confirm the completion of the check item (bottom, Fig.\u0026nbsp;2b). With gesture control, participants move the central cursor with their head and select a menu/checklist item (such as Landing Light On) by holding the cursor gaze and by performing an \u0026lsquo;air-tap\u0026rsquo; (Fig.\u0026nbsp;2c). This \u0026lsquo;air-tap\u0026rsquo; gesture represents the mouse click commonly found in a graphical user interface (GUI). The physical flight deck had been mapped using the HoloLens depth camera and was registered and aligned with the Unity software to obtain a specific visual orientation on the operational environment (Fig.\u0026nbsp;2d). Each checklist highlight (yellow circles and spatial text) was precisely positioned according to the spatial mapping registration alignment with Vuforia. With the voice-command interaction, the user reads out the menu/checklist title to activate it and validates each checklist item by saying \u0026lsquo;check\u0026rsquo;.\u003c/p\u003e\n \u003cp\u003eThe B747 cockpit was first mapped via the depth camera to obtain a rough 3D model of the physical scene. Those cues use the spatial mapping technology of Hololens, and each highlight (yellow circles and spatial text shown in Fig.\u0026nbsp;1b) is then precisely positioned according to this model with the Unity editor (Fig.\u0026nbsp;3b). The AR checklist needs to be calibrated to position the highlights precisely to the physical objects (Flap, Landing Gear, Speedbreak and Landing Light Button). Calibration is achieved thanks to the Vuforia Engine: the user must scan a QR code located near the throttle levers to begin using the app (Fig.\u0026nbsp;3c).\u003c/p\u003e\n \u003cdiv id=\"Sec10\"\u003e\n \u003ch2\u003e3.4. Research Design\u003c/h2\u003e\n \u003cp\u003eThe experiment was run on the flight simulator with a representative model of the Boeing 747\u0026thinsp;\u0026minus;\u0026thinsp;200 flight deck. It was comprised of functioning flight controls, stick-shaker stall warning, over-speed alerts, primary flight and navigation displays, and landing gear levers, among other standard controls. The simplified overhead panel was comprised of light switches, engine fire emergency levers and engine ignition switches (Fig.\u0026nbsp;3a). The scenario was based on an Instrument Landing System (ILS) final approach from 2,000 ft and eight nautical miles (NM) from the airfield. Participants executed an ILS pre-landing checklist by interacting with the AR device during the final approach and landing.\u003c/p\u003e\n \u003cp\u003eThe procedures for all participants were as follows: (1) complete the demographic data including age, gender, qualifications, type hours and total flight hours (five minutes); (2) complete a briefing regarding the purpose of the study and how to use a HoloLens AR device for human-computer interactions (10 minutes); (3) participant seated in the B-747 simulator to calibrate the AR device by placing visual gaze cursor on the QR code with a blue indicator confirming calibration completed (five minutes); (4) familiarisation with the pre-landing checklists on flight simulator (ten minutes); (5) briefing on the AR checklist software, with a detailed explanation of how the item can be highlighted by gesture control and voice control (ten minutes); (6) performing a landing using HoloLens AR gesture-command (five minutes) and AR voice-command (five minutes) - order counterbalanced between participants; (7) post-trial completion of SUS and QUIS based on AP applications and provision of additional descriptive open-end feedbacks (20 minutes). Each participant took approximately 70 minutes to complete the experiment.\u003c/p\u003e\n \u003c/div\u003e\n \u003cdiv id=\"Sec11\"\u003e\n \u003ch2\u003e3.5. Data Analysis\u003c/h2\u003e\n \u003cp\u003eTo investigate system usability and user interface satisfaction, respective post-trial SUS and QUIS scores were analysed using separate repeated measures general linear models (GLM). The SUS (Brooke ,1996) subjective measure that is widely applied in HCI evaluation (Lewis, 2018; Rudi et al., 2020) that measures system usability on 10 positively and negatively worded items requiring a five-point Likert scale response (strongly disagree (1) to strongly agree (5). To aid interpretation, and comparison to published normative datasets (Bangor et al., 2008; Sauro \u0026amp; Lewis, 2016), SUS dimension scores are transformed into percentile scores add allocated so-called usability gradings (A:\u0026gt;89; B: 80\u0026ndash;89; C: 70\u0026ndash;79; D: 60\u0026ndash;69; and F\u0026thinsp;\u0026lt;\u0026thinsp;60). QUIS subscale Cronbach\u0026rsquo;s Alphas were: Overall (.931), Screen (.781), System Information (.749), Learning (.725), System Capabilities (.802) and Usability and UI (.807). Both analyses included within-subject factors for \u003cem\u003eAR interaction modality\u003c/em\u003e (2 levels: voice-command / gesture-command) and for \u003cem\u003esubscale dimension\u003c/em\u003e (3-levels for SUS and 6-levels for QUIS). Bonferroni corrected post-hoc comparisons were used to investigate significant main effects and interactions that included the \u003cem\u003eAR interaction modality\u003c/em\u003e term. The assumption for normally distributed data was checked by graphically using Q-Q plots. Greenhouse-Geisser corrections were required to address violations of sphericity.\u003c/p\u003e\n \u003c/div\u003e\n\u003c/div\u003e"},{"header":"4. Results","content":"\u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e4.1. Sample Characteristics\u003c/h2\u003e \u003cp\u003eParticipants evaluated the system usability and user interface satisfaction of the AR device in flight operations using SUS and QUIS, shown in Table \u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. The dependent variables of SUS are divided into Usability and Learnability; QUIS comprises overall user reactions, screen visibility, system information, learning factors, system capabilities, usability, and user interface. There was no outlier observed by boxplots. Based on the visual examination of the Q-Q plots, the SUS and QUIS data on two AR interactive modes were distributed normality.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThe participants\u0026rsquo; responses to AR interaction modes between Gesture-command and Voice-command using SUS and QUIS\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eInteraction mode\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eN\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eM\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSD\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003eUsability\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGesture-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e40.70\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e20.13\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVoice-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e72.11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e18.02\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003eLearnability\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGesture-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e45.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e27.42\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVoice-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e58.13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e26.79\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003eSUS Total score\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGesture-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e41.56\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e18.85\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVoice-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e69.31\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e17.78\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003eOverall reaction\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGesture-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e4.07\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.68\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVoice-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e6.90\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.42\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e\u003cb\u003eScreen visibility\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGesture-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e6.92\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.34\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVoice-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.61\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.07\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVoice-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.78\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.07\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003eSystem information\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGesture-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e6.91\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.26\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVoice-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.57\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.96\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003eLearning\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGesture-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e6.09\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.55\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVoice-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.45\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.89\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003eSystem capabilities\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGesture-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e4.87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.90\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVoice-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e6.33\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.81\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003eUsability and user interface\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGesture-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e6.49\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.83\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVoice-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.62\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e\u003cb\u003eQUIS Total score\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGesture-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e5.89\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1.03\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVoice-command\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e7.20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.84\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003e4.2. AR Interactive Modes Impacted on System Usability\u003c/h2\u003e \u003cp\u003eSUS post-trial scores are presented in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e4\u003c/span\u003e alongside the normative grading bands for usability based on the work by Bangor et al. (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2008\u003c/span\u003e) to aid interpretation. Notably, gesture-command interaction usability scores were deemed to be far below average (68%) on all 3 SUS dimensions according to SUS normative data (Bangor et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2008\u003c/span\u003e) \u0026ndash; Usability (mean: 40.70, 95CI: 34.26\u0026ndash;47.14, grade: \u0026ldquo;F\u0026rdquo;), Learnability (mean: 45.00, 95CI: 36.23\u0026ndash;53.77, grade: \u0026ldquo;F\u0026rdquo;) and SUS total (mean: 41.56, 95CI: 35.53\u0026ndash;47.59, grade: \u0026ldquo;F\u0026rdquo;). In contrast, whilst the usability of voice-commands interactions was rated below average for Learnability (mean: 58.13, 95CI: 49.56\u0026ndash;66.69, grade: \u0026ldquo;F\u0026rdquo;), the scores were above average for Usability (mean: 72.11, 95CI: 66.35\u0026ndash;77.87, grade \u0026ldquo;C\u0026rdquo;) and SUS total (mean: 69.31, 95CI: 63.63\u0026ndash;75.00. grade: \u0026ldquo;D\u0026rdquo;).\u003c/p\u003e \u003cp\u003eThe GLM analysis revealed both a main effect for \u003cem\u003eAR interaction modality\u003c/em\u003e (\u003cem\u003eF\u003c/em\u003e (1, 39)\u0026thinsp;=\u0026thinsp;27.429.653, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, η\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.413) and an \u003cem\u003eAR interaction modality\u003c/em\u003e by \u003cem\u003edimension\u003c/em\u003e interaction (\u003cem\u003eF\u003c/em\u003e (2, 78)\u0026thinsp;=\u0026thinsp;132.811, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, η\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.457). The post-hoc tests showed SUS scores were higher for the voice-command modality condition compared to the gesture-command condition across SUS dimensions. In descending order of mean difference size: usability (mean diff: 31.406, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, 95%CI: 20.961\u0026ndash;41.851), total (mean diff: 27.750, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, 95%CI: 17.943\u0026ndash;37.557) and learnability (mean diff: 13.125, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.005, 95%CI: 4.304\u0026ndash;21.946). Therefore, the \u0026ldquo;H\u003csub\u003e1\u003c/sub\u003e: there is a significant difference in system usability between voice-command and gesture-command with AR applications during Instrument Landing\u0026rdquo; was supported.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e4.3. AR Interactive Modes Impacted User Interface Satisfaction\u003c/h2\u003e \u003cp\u003eQUIS post-trial scores are presented in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e5\u003c/span\u003e. Across all user satisfaction dimensions visual-command interaction was rated higher than gesture interaction. The GLM analysis revealed both a main effect for \u003cem\u003eAR interaction modality\u003c/em\u003e (\u003cem\u003eF\u003c/em\u003e (1, 39)\u0026thinsp;=\u0026thinsp;68.885, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, η\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.639) and an \u003cem\u003eAR interaction modality\u003c/em\u003e by \u003cem\u003edimension\u003c/em\u003e interaction (\u003cem\u003eF\u003c/em\u003e (5, 71.409)\u0026thinsp;=\u0026thinsp;16.916, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, η\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;.303). The post-hoc tests showed QUIS scores were higher for the voice-command modality condition compared to the gesture-command condition across all dimensions. In descending order of mean difference size: Overall Reaction (mean diff: 2.829, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, 95%CI: 2.124\u0026ndash;3.534), System Capabilities (mean diff: 1.467, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, 95%CI: 0.898\u0026ndash;2.035), Learning (mean diff: 1.350, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, 95%CI: 0.899\u0026ndash;1.801), Usability (mean diff: 0.862, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, 95%CI: 0.443\u0026ndash;1.282), System Information (mean diff: 0.658, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001, 95%CI: 0.310\u0026ndash;1.007) and Screen Visibility (mean diff: 0.525, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.005, 95%CI: 0.168 \u0026ndash; .882). Therefore, the \u0026ldquo;H\u003csub\u003e2\u003c/sub\u003e: there was a significant difference in user interface satisfaction (QUIS) between voice-command and gesture-command with AR applications during Instrument Landing\u0026rdquo; was supported.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003e4.4. Qualitative Feedback of AR Application\u003c/h2\u003e \u003cp\u003eAll participants had never used an AR device in the flight simulator previously. The results of current research demonstrate that participants preferred AR voice command over gesture command due to the intuitive design of vocal interaction, low perceived workload and less distraction while flight operations engaging two hands, but it might have potential risks to communicating with ATCOs and distraction while audio alerts activated. The use of the air-tap gesture with cursor control can be held responsible for the high physical and mental demands, as participants needed to precisely move their heads and hands simultaneously to get their hand gestures recognized by the cameras of HoloLens. It is very interesting to find participants\u0026rsquo; subjective descriptive feedback on strengths and weaknesses in both voice-command and gesture-command (Table \u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). Participants felt a level of frustration with the \u0026ldquo;air-tap\u0026rdquo;, which increased their physical loads while interacting with the AR device. There is a rising argument about the primary task in the flight deck implementing the AR system. Is interacting with AR the primary task, or is interacting with the flight deck instrument the primary task? Professional pilots mentioned that AR applications could be a great tool for training. They speculated cadet pilots could learn flow-checks and scanning patterns of checklist execution more easily by augmented visualization and audio cues superimposed in the flight deck environment.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThe Summaries of Participants\u0026rsquo; Descriptive Feedback on AR Application between Voice-command and Gesture-command for SPO\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eStrengths\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVoice-command\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGesture-command\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1. Allow pilots hand-free for multiple tasks using intuitive design of trigger actions by speech\u003c/p\u003e \u003cp\u003e2. Enhance pilots\u0026rsquo; operational capacity to switch between the primary tasks and secondary tasks\u003c/p\u003e \u003cp\u003e3. Increases the accessibility of control nuts/buttons and reduces physical workload\u003c/p\u003e \u003cp\u003e4. Provide immediate feedback and guidance under time-pressure scenarios\u003c/p\u003e \u003cp\u003e5. Enable more human-centric design of AR/MR interactions using natural language\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1. Not affected by noise environment while multiple alerts activated by system failures\u003c/p\u003e \u003cp\u003e2. Not relying on speech recognition without the accent, language and voice clarity issues\u003c/p\u003e \u003cp\u003e3. Gesture commands are more discreet than voice commands\u003c/p\u003e \u003cp\u003e4.No disruption in communication with ATCOs\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eWeakness\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1. Voice interaction mode can be less effective in noisy environments while multiple audio alerting activated simultaneously\u003c/p\u003e \u003cp\u003e2. AR might have difficulties understanding participants' unique accents under time-pressure scenarios\u003c/p\u003e \u003cp\u003e3. The available voice-commend are very limited and restrict the functions of AR applications\u003c/p\u003e \u003cp\u003e4 Pilot might need to learn and adjust his/her normal speech pattern to be recognized by the AR system\u003c/p\u003e \u003cp\u003e5. There was a potential risk of disrupting the communication with ATCO for critical flight information\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1. Use gesture commands can lead to arm and hand physical fatigue\u003c/p\u003e \u003cp\u003e2. Hololens with narrow field-of-view created difficulties on triggering the actions\u003c/p\u003e \u003cp\u003e3. Pilot lack of additional inputs (hands) to perform multiple task during take-off/landing and emergency system failures\u003c/p\u003e \u003cp\u003e4. The effectiveness of air-tap can be diminished in poor illuminating conditions of flight deck such as night flight\u003c/p\u003e \u003cp\u003e5. Some pilots struggle to learn the air-tap to command the AR device\u003c/p\u003e \u003cp\u003e6. Must fixate on the gesture, which shared more cognitive resources inducing workload\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"5. Discussion","content":"\u003cp\u003eThe application of augmented reality in training is increasingly widespread. These innovative technologies allow users to interact with the real world and the digital world, requiring minimum action between the user and the device. Using voice commands or very simple hand gestures, users can view the necessary information while continuing other tasks that require the use of their hands (Escalada-Hernandez et al., \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e). It is well known that monitoring and cross-checking issues of PF and PM still occur in the contemporary two-pilot flight deck configuration (Harris, \u003cspan class=\"CitationRef\"\u003e2007\u003c/span\u003e). Pilots\u0026apos; checklist non-compliance often results from the complex human-machine interfaces and human-computer interaction under multiple tasks in the modern flight deck. Therefore, in implementing an AR device in flight operations, system designers must ensure that the system is developed with consideration of human-centric interactive modes to improve pilots\u0026rsquo; monitoring and operational performance. The results from the data analysis demonstrated that the AR voice-command mode was rated significantly higher in system usability and user interface applicability than the AR gesture-command mode (Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e). These findings can provide scientific evidence to advance AR design to support pilot-monitoring tasks for future single-pilot operations. An intuitive and human-centric design AR system with multimodal inputs to achieve the goal of system control and information processing is used based on the growth of accuracy and efficiency of gesture recognition and voice synthesis technologies (Bilius \u0026amp; Vatavu, \u003cspan class=\"CitationRef\"\u003e2020\u003c/span\u003e).\u003c/p\u003e\n\u003cdiv id=\"Sec18\" class=\"Section2\"\u003e\n \u003ch2\u003e5.1 Semantic AR Interacting Modes Consistent with Human-Centric Design\u003c/h2\u003e\n \u003cp\u003eThe system control methods of gesture command and voice input have been widely used in the real-world interaction context of AR experiences. The current research illustrated that AR gesture-command interaction of \u0026lsquo;air-tap\u0026rsquo; is rated significantly lower than voice-command in usability, learnability, and SUS scores (Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e \u0026amp; Fig. \u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e). To successfully commence an \u0026lsquo;air-tap\u0026rsquo; gesture, participants must position their thumb and index finger in front of the HoloLens camera and then pinch the thumb and index finger together. However, due to the limited field of view of the HoloLens and technical issues of motion recognition, the activation of each procedure item could take a longer time to hinder the interaction process and downgrade participants\u0026rsquo; rating of system usability. The gesture-command required participants to focus on SOPs and initiate an \u0026lsquo;air-tap\u0026rsquo; precisely on time, and the cursor was positioned well to execute the specific action of each checklist. Adding these tasks alongside the high mental and physical task loads of operating a complex aircraft may increase the overload of the working memory. Currently, the primary input modality in the flight deck is the unimodal hand haptic inputs on the control inceptors. The voice-command interaction provided a multimodal interface permitting the integration of multiple sensory channels (sight, hearing, and touch) with higher flexibility and reliability to meet the task needs of different flight scenarios. Turk (\u003cspan class=\"CitationRef\"\u003e2014\u003c/span\u003e) indicated that operators would prefer multimodal interaction over unimodal alternatives as it significantly enhances task efficiency and cognitive information processes. Besides the naturalness and intuition characteristics, the voice-command interactive mode has substantial advantages in memorability and learnability. The semantic interaction of AR voice-command mode will be activated by simple words presented in the interface, such as \u0026lsquo;check\u0026rsquo; and \u0026lsquo;next\u0026rsquo;, to perform each procedure item in the pre-landing checklist. Compared to the \u0026lsquo;air-tap\u0026rsquo; gesture, which is difficult to perform precisely, the voice-command mode is learnable and applicable in dynamic flight scenarios without the distractions of pilots\u0026rsquo; manual flight controlling and visual monitoring. The voice-command AR interactive mode requires little effort to achieve simple, accurate, and safe operations in flight scenarios. Furthermore, familiarity is a critical factor affecting learnability in the human-computer interaction process to provide self-explanatory cues for understanding the functions that the system offers (Green \u0026amp; Eklundh, \u003cspan class=\"CitationRef\"\u003e2003\u003c/span\u003e). Participants\u0026rsquo; familiarity with voice-command mode had a significant influence on their preferences and learnability ratings, as voice-command is intuitive and learnable when compared with gesture-command.\u003c/p\u003e\n \u003cp\u003eUser experience can be seen as the holistic extension of system usability measures, which covers additional details about human-computer interaction from hedonic and emotional aspects (Hassan \u0026amp; Galal-Edeen, \u003cspan class=\"CitationRef\"\u003e2017\u003c/span\u003e). In the current study, the QUIS assessments can reflect the interactive experience and user satisfaction with the utilitarian functions and experiential attributes of the AR gesture- and voice-command interfaces (Chung \u0026amp; Sahari, \u003cspan class=\"CitationRef\"\u003e2015\u003c/span\u003e). The suggested that the QUIS average score and six sub-dimension ratings on the AR voice-command mode are significantly higher than the gesture-command. Previous research demonstrated that, compared to the traditional point-and-click style interface, the voice-command dialogue system could enhance the emotional effectiveness of the VR interactive narratives to improve the user experience and interface satisfaction (Osking \u0026amp; Doucette, \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e). Furthermore, in the current study, voice input especially has the advantage of interface traversals that can help participants cut through multiple steps through the multimodal process to decrease the task completion time. This point can be confirmed by two rating items of the learning and system capacities sub-dimensions in the QUIS related to the logical steps and response to complete: the AR voice-command mode allowed participants to follow each step to finish the pre-landing checklist operations with a better logical sequence and faster reaction time compared to the \u0026lsquo;air-tap\u0026rsquo; interaction (Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e \u0026amp; Fig. \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e). The fluent interaction process and significant time saving also provide a positive emotional effect on the interactive experience (Microsoft, \u003cspan class=\"CitationRef\"\u003e2022\u003c/span\u003e).\u003c/p\u003e\n \u003cp\u003eIn the sub-dimensions of the overall user reactions, participants showed a significant preference for the voice-command AR interface over the gesture-command mode due to the higher system functionality and accessibility, as well as better interaction experience with voice input. Also, the higher ratings in the learning and system capacities revealed the advantages of the voice-command mode in a straightforward and understandable control manner with emotional effectiveness and operational efficiency. The hand-gesture-command interaction during the approach can cause distraction and frustration while the participant\u0026rsquo;s attention is entirely focused on flight operations plus executing the \u0026lsquo;air-tap\u0026rsquo; itself, downgrading the user experience. Furthermore, there were three dimensions related to visual display and information presentation designs: screen visibility, system information, and usability and user interface. These three dimensions are supposed to provide the same information with the same level of visual presentation and have theoretically no difference between the voice-command and \u0026lsquo;air-tap\u0026rsquo; interactive modes. However, the results still demonstrated significant differences between the two interactive modes in these sub-dimensions. One possible explanation might be that the participants are biased on the operational experience of the gesture-command interaction, and they tend to mark lower on the user interface satisfaction compared to the voice-command mode. Such subjective bias can be concluded as an emotional response to influence participants\u0026rsquo; operational experience and engagement (Shepherd \u0026amp; Cardon, \u003cspan class=\"CitationRef\"\u003e2009\u003c/span\u003e).\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec19\" class=\"Section2\"\u003e\n \u003ch2\u003e5.2 Voice Command Increasing Pilot\u0026rsquo;s Operational Capacity for Multiple Tasks\u003c/h2\u003e\n \u003cp\u003eParticipants rated higher on AR voice-command mode generally also appreciated the augmented visualisation information and procedural adherence instructions through an overlayed virtual display based on their qualitative feedback (Table \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e). Previous research has demonstrated that directional cues in AR may have a beneficial effect on performance. This has primarily been established in the context of tasks requiring navigation or spatial orientation (Peereboom et al., \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e). The challenge with flight operations is that various tasks and SOPs must be executed precisely in a specific time frame. In the current study, the AR interface not only provided relevant checklist items required during the instrument landing scenarios but also allowed augmented visualisation cues of visually blended virtual checklists within the physical environment of the flight deck (Fig. 6). These augmented visualisation instructions directed pilots\u0026rsquo; attention and action to perform each required procedure in a simple and intuitive workflow through the voice-command interaction to improve the checklist completion accuracy and speed for single-pilot operations. With the support of the AR voice-command checklist, the pilot-flying can hold the control column by the left hand, simultaneously operate the flap to 30 degrees (Fig. 6a), place the landing gear down (Fig. 6b), move the speed-brakes lever arm (Fig. 6c), switch landing lights on (Fig. 6d) as well as any additional interactions required with the flight deck instruments by the right hand following the AR instructions. In this process, the pilot can simply say \u0026lsquo;check\u0026rsquo; to go through every item in the pre-landing checklist without additional hand movements or haptic interactions. Furthermore, via the AR voice-command mode while performing SOPs, the pilot\u0026rsquo;s monitoring performance, situation awareness, and working memory capacity will be enhanced. Therefore, the AR voice-command interactive system can be further developed and generalised in flight procedures training of various segments in flight operations to improve trainees\u0026rsquo; working memory, in-flight decision, and operational skills (Squires, \u003cspan class=\"CitationRef\"\u003e2017\u003c/span\u003e).\u003c/p\u003e\n \u003cp\u003eThe AR interactive technology has the potential to provide adequate integrated information and intuitive human-computer interaction without attentional distraction due to head-down observations of instruments (Kim \u0026amp; Grabbard, 2019; Lu et al., \u003cspan class=\"CitationRef\"\u003e2020\u003c/span\u003e). However, there is a significant negative association of the SUS total scores between the voice-command mode and \u0026lsquo;air-tap\u0026rsquo;, revealing an exact opposite preference in system usability of different interactive modes of the in-flight AR systems. The user interface and human-computer interaction design of AR applications are critical factors in influencing system performance and human cognitive performance; if the system design does not follow human-centred principles, undesirable difficulties and overloaded cognitive demands will be added for the end-users (Pane et al., \u003cspan class=\"CitationRef\"\u003e2002\u003c/span\u003e). Based on the collected data analysis, it can be concluded that the in-flight AR technology can provide sufficient information with an intuitive multimodal interface. Future research shall investigate the integration of the Large Language Model (LLM) with AR for semantic interaction of voice command, which may provide a more intuitive human-centric design for pilots to execute multiple tasks precisely.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"6. Conclusion","content":"\u003cp\u003eThe semantic voice-command interaction on an AR device has high system usability and user interface satisfaction, which can increase pilots\u0026rsquo; monitoring performance and reduce physical task loads. AR technology blends virtual checklists with the real operational environment and provides guidance cues to direct pilots\u0026rsquo; attention using intuitive voice-command. The use of the voice as an input for a multimodal AR device to execute intended actions can facilitate human-centred design concepts on the flight deck. There are some limitations related to voice recognition, such as a limited display field of view and obstacles from augmented vision superimposed in the flight deck that must be improved in the future. However, the benefits of applied AR voice command in flight operations and training still present promising opportunities for further research and development. AR applications in flight operations need to be assessed carefully so that all stakeholders, pilots, avionic engineers, system designers, manufacturers, and regulators can be satisfied with the potential benefits and associated risks. The current research has demonstrated that flight deck automation can be integrated with voice-command AR applications, which could enhance the pilots\u0026rsquo; procedural compliance and monitoring performance in the human-centric flight deck. However, there are still several limitations of an AR application in flight operations which may require further research and development. The first issue is the efficiency of voice recognition, as some participants experienced various degrees of frustration due to a lack of response to some unique accents of spoken English. The second issue was that \u0026lsquo;air-tap\u0026rsquo; recognition required more accuracy for the identification of the users\u0026rsquo; intended gesture. Some participants were frustrated while trying to make commands using an \u0026lsquo;air-tap\u0026rsquo; that was not recognised by the camera due to the limited field of view on the HoloLens. The third issue is the readability of virtual objects (text) on the HoloLens display, which may potentially create blind spots when targets are located outside the field of view or text superimposed by other visual elements. These issues may be improved with the new generation of VR/AR/MR technologies, which have integrated real-time eye tracking with two-handed, fully articulated gesture control technology and a wider field of view display. The eye-tracking and hand-tracking functions will have the potential to facilitate AR gaze control and hand control as additional input for single-pilot operations. In future studies, objective measurements of the pilot\u0026rsquo;s operational performance will be conducted to maximise AR applications to support single-pilot flight deck design.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eDeclaration of interests\u003c/p\u003e\n\u003cp\u003e☒ The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eWen-Chin Li 's contributions involved the design of research, data collection, and writing of the paper.Jingyi Zhang's contributions included the literature review, data analysis, writing of the result section, and check the citations and references.James Blundel's contributions included revised the literature, double check the data analysis, revised the result section and provided relevant references for the literature review and discussion.Samuel Court's contributions involved in the AR system design, Applications of Flight Deck HCI for Voice-command and gesture-command, 3-D scenarios building in Hololens.Dujuan Sevillian's contributions included writing SPO and HMI design for AR application in flight operations, literature review and discussion revision.\u003c/p\u003e\u003ch2\u003eAcknowledgement\u003c/h2\u003e \u003cp\u003eThis article is based on work supported by the UK EPSRC Grant: \u0026ldquo;Digital Toolkit for Optimisation of Operators and Technology in manufacturing partnerships\u0026rdquo; (Grant number EP/R032718/1) and the Higher Education Innovation Funding (HEIF 2018\u0026ndash;2019). The authors would like to express our appreciation to Tim Bord, Bikram Thapa, Zepu Yan and all participants for their contributions to this project.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eData will be Published in Cranfield University CORD Data Repository\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAromaa, S., V\u0026auml;\u0026auml;t\u0026auml;nen, A., Aaltonen, I., Goriachev, V., Helin, K., \u0026amp; Karjalainen, J. (2020). Awareness of the real-world environment when using augmented reality head-mounted display. Applied ergonomics, 88, 103145. https://doi.org/10.1016/j.apergo.2020.103145\u003c/li\u003e\n\u003cli\u003eBagassi, S., De Crescenzio, F., Piastra, S., Persiani, C. A., Ellejmi, M., Groskreutz, A. R., \u0026amp; Higuera, J. (2020). Human-in-the-loop evaluation of an augmented reality based interface for the airport control tower. Computers in Industry, 123, 103291. https://doi.org/10.1016/j.compind.2020.103291\u003c/li\u003e\n\u003cli\u003eBailey, R. E., Kramer, L. J., Kennedy, K. D., Stephens, C. L., \u0026amp; Etherington Rockwell-Collins, T. J. (2017). An Assessment of Reduced Crew and Single Pilot Operations in Commercial Transport Aircraft Operations. IEEE/AIAA 36th Digital Avionics Systems Conference (DASC), 2017. https://ntrs.nasa.gov/search.jsp?R=20170009542\u003c/li\u003e\n\u003cli\u003eBangor, A., Kortum, P. T., \u0026amp; Miller, J. T. (2008). An empirical evaluation of the system usability scale. Intl. Journal of Human\u0026ndash;Computer Interaction, 24(6), 574-594. https://doi.org/10.1080/10447310802205776\u003c/li\u003e\n\u003cli\u003eBilius, L. B., \u0026amp; Vatavu, R. D. (2020). A multistudy investigation of drivers and passengers\u0026rsquo; gesture and voice input preferences for in-vehicle interactions. Journal of Intelligent Transportation Systems, 25(2), 197-220. https://doi.org/10.1080/15472450.2020.1846127\u003c/li\u003e\n\u003cli\u003eBlundell, J., \u0026amp; Harris, D. (2023). Designing augmented reality for future commercial aviation: a user-requirement analysis with commercial aviation pilots. Virtual Reality. https://doi.org/10.1007/s10055-023-00798-9\u003c/li\u003e\n\u003cli\u003eBoy, G. A. (2014). Requirements for Single Pilot Operations in Commercial Aviation: A First High-Level Cognitive Function Analysis. Requirements for Single Pilot Operations in Commercial Aviation: A First High-Level Cognitive Function Analysis. Complex Systems Design and Management International Conference, Paris, France. \u003c/li\u003e\n\u003cli\u003eBrooke, J. (1996). SUS-A quick and dirty usability scale. In: Jordan P.W., Weerdmeester B., Thomas A., McLelland I.L. (Eds.), Usability Evaluation in Industry (pp. 189-194). CRC Press. https://doi.org/10.1201/9781498710411-35\u003c/li\u003e\n\u003cli\u003eBrown, C., Hicks, J., Rinaudo, C. H., \u0026amp; Burch, R. (2021). The use of augmented reality and virtual reality in ergonomic applications for education, aviation, and maintenance. Ergonomics in Design, 1-9. https://doi.org/10.1177/10648046211003469\u003c/li\u003e\n\u003cli\u003eBurian, B. K. (2006). Design guidance for emergency and abnormal checklists in aviation. In Proceedings of 2006 Human Factors and Ergonomics Society Annual Meeting, SAGE Publications (pp. 106-110). Los Angeles. https://doi.org/10.1177/154193120605000123\u003c/li\u003e\n\u003cli\u003eCarroll, M., \u0026amp; Dahlstrom, N. (2021). Human computer interaction on the modern flight deck. International Journal of Human\u0026ndash;Computer Interaction, 37(7), 585-587. https://doi.org/10.1080/10447318.2021.1890495\u003c/li\u003e\n\u003cli\u003eChan, W. T. K., \u0026amp; Li, W. C. (2021). Consideration of cultural differences in future workplace design for single pilot operations. Contemporary Ergonomics \u0026amp; Human Factors, 64-71.\u003c/li\u003e\n\u003cli\u003eChin, J. P., Diehl, V. A., \u0026amp; Norman, K. L. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 213-218). https://doi.org/10.1145/57167.57203\u003c/li\u003e\n\u003cli\u003eChung, T. K., \u0026amp; Sahari, N. (2015). Utilitarian or experiential? An analysis of usability questionnaires. International Journal of Computer Theory and Engineering, 7(2), 167-171. https://doi.org/10.7763/IJCTE.2015.V7.950\u003c/li\u003e\n\u003cli\u003eDetjen, H., Faltaous, S., Pfleging, B., Geisler, S., \u0026amp; Schneegass, S. (2021). How to Increase Automated Vehicles\u0026rsquo; Acceptance through In-Vehicle Interaction Design: A Review. International Journal of Human-Computer Interaction, 37(4), 308\u0026ndash;330. https://doi.org/10.1080/10447318.2020.1860517\u003c/li\u003e\n\u003cli\u003eDey, A., Billinghurst, M., Lindeman, R. W., \u0026amp; Swan, J. E. (2018). A systematic review of 10 Years of Augmented Reality usability studies: 2005 to 2014. Frontiers Robotics AI, 5(APR), 329739. https://doi.org/10.3389/FROBT.2018.00037/BIBTEX\u003c/li\u003e\n\u003cli\u003eDill, E. T., \u0026amp; Young, S. D. (2015). Analysis of Eye-Tracking Data with Regards to the Complexity of Flight Deck Information Automation and Management-Inattentional Blindness, System State Awareness, and EFB Usage. In Proceedings of 15th AIAA Aviation Technology, Integration, and Operations Conference (pp. 1-26). AIAA. https://doi.org/10.2514/6.2015-2901\u003c/li\u003e\n\u003cli\u003eEscalada-Hernandez, P., Soto-Ruiz, N., Ballesteros-Eg\u0026uuml;\u0026eacute;s, T. et al. (2024). Usability and user expectations of a HoloLens-based augmented reality application for learning clinical technical skills. Virtual Reality 28, 102 (2024). https://doi.org/10.1007/s10055-024-00984-3\u003c/li\u003e\n\u003cli\u003eGreen, A., \u0026amp; Eklundh, K. S. (2003). Designing for learnability in human-robot communication. IEEE Transactions on industrial electronics, 50(4), 644-650. https://doi.org/10.1109/TIE.2003.814763\u003c/li\u003e\n\u003cli\u003eGregoriades, A., \u0026amp; Sutcliffe, A. (2018). Simulation-based evaluation of an in-vehicle smart situation awareness enhancement system. Ergonomics, 61(7), 947-965. https://doi.org/10.1080/00140139.2018.1427803\u003c/li\u003e\n\u003cli\u003eHarris, D. (2007). A human‐centred design agenda for the development of single crew operated commercial aircraft. Aircraft Engineering and Aerospace Technology, 79(5), 518-526. https://doi.org/10.1108/00022660710780650\u003c/li\u003e\n\u003cli\u003eHarris, D. (2023). Single-pilot airline operations: Designing the aircraft may be the easy part. The Aeronautical Journal, 127(1313), 1171-1191. https://doi.org/10.1017/aer.2022.110\u003c/li\u003e\n\u003cli\u003eHassan, H. M., \u0026amp; Galal-Edeen, G. H. (2017). From usability to user experience. In 2017 International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS) (pp. 216-222). IEEE. https://doi.org/10.1109/ICIIBMS.2017.8279761\u003c/li\u003e\n\u003cli\u003eHe, Z., Chang, T., Lu, S., Ai, H., Wang, D., \u0026amp; Zhou, Q. (2017). Research on human-computer interaction technology of wearable devices such as augmented reality supporting grid work. Procedia Computer Science, 107, 170-175. https://doi.org/10.1016/j.procs.2017.03.074\u003c/li\u003e\n\u003cli\u003eHelin, K., Kuula, T., Vizzi, C., Karjalainen, J., \u0026amp; Vovk, A. (2018). User experience of augmented reality system for astronaut\u0026apos;s manual work support. Frontiers in Robotics and AI, 5, 1-10. https://doi.org/10.3389/frobt.2018.00106\u003c/li\u003e\n\u003cli\u003eIATA. (2023). Global Outlook for Air Transport: High Resilient, Less Robust. International Air Transport, Montreal, Quebec, Canada. https://www.iata.org/en/iata-repository/publications/economic-reports/global-outlook-for-air-transport----june-2023/\u003c/li\u003e\n\u003cli\u003eICAO (2019). Addressing the shortage of pilots and aviation professionals: a global approach to a global problem. ICAO Working Paper, A40-WP/239. International Civil Aviation Organization, Montreal, Quebec, Canada. https://www.icao.int/Meetings/a40/Documents/WP/wp_239_en.pdf\u003c/li\u003e\n\u003cli\u003eICAO. (2022). An approach to new operational concepts involving extended minimum crew operations and single-pilot operations. ICAO Working Paper, A41-WP/101. Technical Commission, International Civil Aviation Organization, Montreal, Quebec, Canada. https://www.icao.int/Meetings/a41/Documents/WP/wp_101_en.pdf\u003c/li\u003e\n\u003cli\u003eJorna, P. G. A. M. (1997). Human Machine interfaces for ATM: objective and subjective measurements on human interactions with future Flight deck and Air Traffic Control systems. In FAA/Eurocontrol ATM R\u0026amp;D Seminar, Paris, France. \u003c/li\u003e\n\u003cli\u003eJetter, J., Eimecke, J., \u0026amp; Rese, A. (2018). Augmented reality tools for industrial applications: What are potential key performance indicators and who benefits? Computers in Human Behavior, 87, 18\u0026ndash;33. https://doi.org/10.1016/j.chb.2018.04.054\u003c/li\u003e\n\u003cli\u003eKim, H., \u0026amp; Gabbard, J. L. (2019). Assessing Distraction Potential of Augmented Reality Head-Up Displays for Vehicle Drivers. Human Factors, 2019, 1-14. https://doi.org/10.1177/0018720819844845\u003c/li\u003e\n\u003cli\u003eKorkiakoski, M., Alavesa, P., \u0026amp; Kostakos, P. (2024). Preference in Voice Commands and Gesture Controls With Hands-Free Augmented Reality With Novel Users. IEEE Pervasive Computing, 23(1), 18\u0026ndash;26. https://doi.org/10.1109/MPRV.2024.3364541\u003c/li\u003e\n\u003cli\u003eLee, M., Billinghurst, M., Baek, W., Green, R., \u0026amp; Woo, W. (2013). A usability study of multimodal input in an augmented reality environment. Virtual Reality, 17, 293\u0026ndash;305. https://doi.org/10.1007/s10055-013-0230-0\u003c/li\u003e\n\u003cli\u003eLewis, J. R. (2018). The system usability scale: past, present, and future. International Journal of Human\u0026ndash;Computer Interaction, 34, 577-590. https://doi.org/10.1080/10447318.2018.1455307\u003c/li\u003e\n\u003cli\u003eLi, W.-C., Horn, A., Sun, Z., Zhang, J., \u0026amp; Braithwaite, G. (2020). Augmented visualisation cues on primary flight display facilitating pilot\u0026apos;s monitoring performance. International Journal of Human-Computer Studies, 135, 1-10. https://doi.org/10.1016/j.ijhcs.2019.102377\u003c/li\u003e\n\u003cli\u003eLi, W. C., Zhang, J., Court, S., Kearney, P., \u0026amp; Braithwaite, G. (2022). The influence of augmented reality interaction design on Pilot\u0026apos;s perceived workload and situation awareness. International Journal of Industrial Ergonomics, 92, 103382. https://doi.org/10.1016/j.ergon.2022.103382\u003c/li\u003e\n\u003cli\u003eLim, Y., Bassien-Capsa, V., Ramasamy, S., Liu, J., \u0026amp; Sabatini, R. (2017). Commercial airline single-pilot operations: System design and pathways to certification. IEEE Aerospace and Electronic Systems Magazine, 32(7), 4-21. https://doi.org/10.1109/MAES.2017.160175\u003c/li\u003e\n\u003cli\u003eLu, S., Sanchez Perdomo, Y. P., Jiang, X., \u0026amp; Zheng, B. (2020). Integrating Eye-Tracking to Augmented Reality System for Surgical Training. Journal of Medical Systems, 44(11). https://doi.org/10.1007/s10916-020-01656-w\u003c/li\u003e\n\u003cli\u003eMeister, P., Miller, J., Wang, K., Dorneich, M. C., Winer, E., Brown, L. J., \u0026amp; Whitehurst, G. (2022). Designing Three-Dimensional Augmented Reality Weather Visualisations to Enhance General Aviation Weather Education. IEEE Transactions on Professional Communication, 65(2), 321-336. https://doi.org/10.1109/TPC.2022.3155920\u003c/li\u003e\n\u003cli\u003eMicrosoft. (2021). HoloLens (1st gen) Hardware. https://learn.microsoft.com/en-us/hololens/hololens1-hardware#device-specifications\u003c/li\u003e\n\u003cli\u003eMicrosoft. (2022). Microsoft HoloLens: Voice Input. https://docs.microsoft.com/en-us/windows/mixed-reality/design/voice-input \u003c/li\u003e\n\u003cli\u003eMoesl, B., Schaffernak, H., Vorraber, W., Braunstingl, R., \u0026amp; Koglbauer, I. V. (2023). Multimodal Augmented Reality Applications for Training of Traffic Procedures in Aviation. Multimodal Technologies and Interaction, 7(1), 1-17. https://doi.org/10.3390/mti7010003\u003c/li\u003e\n\u003cli\u003eMyers, P. L., \u0026amp; Starr, A. W. (2021). Single pilot operations in commercial cockpits: background, challenges, and options. Journal of Intelligent \u0026amp; Robotic Systems, 102(1), 19. https://doi.org/10.1007/s10846-021-01371-9\u003c/li\u003e\n\u003cli\u003eNizam, S. M., Abidin, R. Z., Hashim, N. C., Lam, M. C., Arshad, H., \u0026amp; Majid, N. A. A. (2018). A review of multimodal interaction technique in augmented reality environment. International Journal of Advanced Science Engineering Information Technology, 8(4-2), 1460-1469.\u003c/li\u003e\n\u003cli\u003eOsking, H., \u0026amp; Doucette, J. A. (2019). Enhancing emotional effectiveness of virtual-reality experiences with voice control interfaces. In Proceedings of Immersive Learning Research Network: 5th International Conference (pp. 199-209). Springer, Cham. https://doi.org/10.1007/978-3-030-23089-0_15\u003c/li\u003e\n\u003cli\u003ePane, J. F., Myers, B. A., \u0026amp; Miller, L. B. (2002). Using HCI techniques to design a more usable programming system. In Proceedings of IEEE 2002 Symposia on Human Centric Computing Languages and Environments (pp. 198-206). IEEE. https://doi.org/10.1109/HCC.2002.1046372\u003c/li\u003e\n\u003cli\u003ePeereboom, J., Tabone, W., Dodou, D. et al. Head-locked, world-locked, or conformal diminished-reality? An examination of different AR solutions for pedestrian safety in occluded scenarios. Virtual Reality 28, 119 (2024). https://doi.org/10.1007/s10055-024-01017-9\u003c/li\u003e\n\u003cli\u003eRiener, A., Jeon, M., Alvarez, I., \u0026amp; Frison, A. K. (2017). Driver in the Loop: Best Practices in Automotive Sensing and Feedback Mechanisms. 295\u0026ndash;323. https://doi.org/10.1007/978-3-319-49448-7_11\u003c/li\u003e\n\u003cli\u003eRudi, D., Kiefer, P., \u0026amp; Raubal, M. (2020). The instructor assistant system (iASSYST)-utilising eye tracking for commercial aviation training purposes. Ergonomics, 63(1), 61-79. https://doi.org/10.1080/00140139.2019.1685132\u003c/li\u003e\n\u003cli\u003eSadri, S., Kohen, S., Elvezio, C., Sun, S., Grinshpoon, A., Loeb, G., Basu, N., \u0026amp; Feiner, S. (2019). Manipulating 3D anatomic models in augmented reality: Comparing a hands-free approach and a manual approach. Proceedings - 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2019, 93\u0026ndash;102. https://doi.org/10.1109/ISMAR.2019.00-21\u003c/li\u003e\n\u003cli\u003eSauro, J., \u0026amp; Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research. Morgan Kaufmann, Elsevier, Cambridge, MA, US. \u003c/li\u003e\n\u003cli\u003eSchmid, D., \u0026amp; Stanton, N. A. (2020). Progressing Toward Airliners\u0026rsquo; Reduced-Crew Operations: A Systematic Literature Review. In International Journal of Aerospace Psychology 30 (1\u0026ndash;2), 1\u0026ndash;24. Taylor and Francis Inc. https://doi.org/10.1080/24721840.2019.1696196\u003c/li\u003e\n\u003cli\u003eShepherd, D. A., \u0026amp; Cardon, M. S. (2009). Negative emotional reactions to project failure and the self‐compassion to learn from the experience. Journal of Management Studies, 46(6), 923-949. https://doi.org/10.1111/j.1467-6486.2009.00821.x\u003c/li\u003e\n\u003cli\u003eSpittle, B., Frutos-Pascual, M., Creed, C., \u0026amp; Williams, I. (2023). A Review of Interaction Techniques for Immersive Environments. IEEE Transactions on Visualization and Computer Graphics, 29(9), 3900\u0026ndash;3921. https://doi.org/10.1109/TVCG.2022.3174805\u003c/li\u003e\n\u003cli\u003eSquires, D. R. (2017). Working Memory \u0026amp; Augmented Reality\u0026apos;s Trajectory: A Literature Review of AR in Education, Online Learning, Workforce Training, and Working Memory Research. Journal of Educational Technology, 14(3), 55-63. https://doi.org/10.26634/jet.14.3.13860\u003c/li\u003e\n\u003cli\u003eTran, T. H., Behrend, F., F\u0026uuml;nning, N., \u0026amp; Arango, A. (2018). Single pilot operations with AR-glasses using Microsoft HoloLens. In Proceedings of 2018 IEEE/AIAA 37th Digital Avionics Systems Conference (pp. 1-7). IEEE. https://doi.org/10.1109/DASC.2018.8569261\u003c/li\u003e\n\u003cli\u003eTurk, M. (2014). Multimodal interaction: A review. Pattern recognition letters, 36, 189-195. https://doi.org/10.1016/j.patrec.2013.07.003\u003c/li\u003e\n\u003cli\u003eWickens, C. D. (2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3(2), 159\u0026ndash;177. https://doi.org/10.1080/14639220210123806\u003c/li\u003e\n\u003cli\u003eWickens, C. D., \u0026amp; Carswell, C. M. (1995). The proximity compatibility principle: Its psychological foundation and relevance to display design. Human Factors, 37, 473\u0026ndash;494. https://doi.org/10.1518/001872095779049408\u003c/li\u003e\n\u003cli\u003eWickens, C. D., \u0026amp; Ward, J. (2017). Cockpit Displays of Traffic and Weather Information: Effects of 3D Perspective Versus 2D Coplanar Rendering and Database Integration. The International Journal of Aerospace Psychology, 27(1-2), 44-56. https://doi.org/10.1080/10508414.2017.1366270\u003c/li\u003e\n\u003cli\u003eWoodward, J., \u0026amp; Ruiz, J. (2022). Analytic review of using augmented reality for situational awareness. IEEE Transactions on Visualization and Computer Graphics, 29(4), 2166-2183. https://doi.org/10.1109/TVCG.2022.3141585 \u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Augmented Reality, Multimodal Interaction, Single Pilot Operations, System Usability, User Interface Experience ","lastPublishedDoi":"10.21203/rs.3.rs-4844391/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4844391/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Implementing augmented reality (AR) technologies has become a popular method of increasing operators' situation awareness by adding virtual information to the physical environment. In the current commercial two-pilot flight deck, the pilot-flying (PF) is responsible for flying the aircraft to an approved flight plan, and the pilot-monitoring (PM) focuses on communicating and monitoring PF’s operational behaviours. The driving factors behind single-pilot operations (SPO) are the foreseen pilot shortage and desire to reduce operating costs. Whilst SPO is expected to be enabled - in part - by advanced flight deck technologies. Forty participants attended simulator trials that involved interacting with multimodal AR apps (Hololens) which included voice and gesture command functionalities. Results revealed voice command scored higher than the gesture command on both the System Usability Scale (SUS) and the Questionnaire for User Interface Satisfaction (QUIS). AR visualisation that blends the physical operation environment and a virtual holographic checklist with guiding cues can improve pilots’ monitoring performance and procedure compliance during the instrument landing trials. Furthermore, the AR voice command permits multiple sensory processing and response by integrating visual, auditory, and tactile inputs simultaneously, which provides the pilot with greater flexibility to meet task requirements. AR gesture command was regarded as an unnecessary burden to the pilot’s cognitive resources and limited hand movement while executing complicated operating procedures. Future research shall explore the implementation of AR voice command with artificial intelligence (AI) on the flight deck to support a single pilot performing both flying and monitoring tasks. This research paradigm needs to be assessed thoroughly for human-centred design to ensure broad acceptance by end users, manufacturers, airlines and regulators.","manuscriptTitle":"Development of Multimodal Augmented Reality Application on the Flight Deck for Single-Pilot Operations: System Usability and User Interface Satisfaction","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-09-16 00:40:46","doi":"10.21203/rs.3.rs-4844391/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"24dc751e-870b-4bce-8bd3-a7124bb2e9fd","owner":[],"postedDate":"September 16th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-06-22T16:20:15+00:00","versionOfRecord":{"articleIdentity":"rs-4844391","link":"https://doi.org/10.1007/s10055-026-01411-5","journal":{"identity":"virtual-reality","isVorOnly":false,"title":"Virtual Reality"},"publishedOn":"2026-06-18 16:01:40","publishedOnDateReadable":"June 18th, 2026"},"versionCreatedAt":"2024-09-16 00:40:46","video":"","vorDoi":"10.1007/s10055-026-01411-5","vorDoiUrl":"https://doi.org/10.1007/s10055-026-01411-5","workflowStages":[]},"version":"v1","identity":"rs-4844391","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4844391","identity":"rs-4844391","version":["v1"]},"buildId":"cBFmMYwuxLRRLfASyISRj","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}