Static Gesture Recognition Based on Thermal Imaging Sensors

Static Gesture Recognition Based on Thermal Imaging Sensors | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Static Gesture Recognition Based on Thermal Imaging Sensors ZHANG Zhi-yuan, REN Hao, LI Hao, YUAN Kang-hui, Chu-feng ZHU This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4689884/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 16 You are reading this latest preprint version Abstract Gesture recognition technology has demonstrated its applicability in various scenarios within the field of human-computer interaction, owing to its flexibility and non-contact nature. In particular, thermal imaging technology which is not limited by lighting conditions and effectively reduces the risk of privacy disclosure by capture only thermal radiation rather than detailed features. In this study, a low resolution 32 x 24 pixels end-to-end embedded infrared thermal imaging camera gesture recognition system is developed. A thermal imaging gesture dataset of 4,500 images is constructed to train and evaluate the system. The effects on improve gesture recognition accuracy by research the spatial transform network attention mechanism (STN) is discussed. Thus, a new method combines lightweight Convolutional Neural Networks (CNN) and STN is propose. Additionally, the proposed method achieves 98.5% recognition accuracy and inference time of only 59 milliseconds per frame on embedded devices when tested on self-made infrared thermal imaging sign language gesture dataset, outperforming mainstream lightweight models. Thermal imaging Gesture recognition Convolutional neural networks Embedded systems Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 1 Introduction Gestures serve as a silent yet potent form of communication, finding utility in diverse fields such as automotive interfaces, healthcare systems, assistive technologies, entertainment and human-computer interaction [ 1 – 2 ]. This versatility enables effective and contactless communication across various contexts, which not only enables immersive virtual reality (VR) and provides touchless control over smart devices but also assists individuals with hear and speech impairments. It plays a crucial role in advanced domains such as robotic control technology and medical diagnosis [ 3 – 10 ]. The advent of deep learning has propelled advancements in digital gesture recognition, with methodologies predominantly based on Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). CNN is widely recognized for their superior ability in image classification and feature extraction, whereas RNN excel in handling sequential data. These attributes allow for the nuanced recognition of complex gestures. V.M., et al. used a low-resolution thermal imaging camera for gesture prediction and introduced a novel detection technique that integrates 2D CNN with Temporal Convolutional Networks (TCN). This approach attained a classification accuracy of 95.9% and a mean Average Precision (mAP) [ 11 ]. Wu J., et al. developed a dynamic gesture recognition model that employs data gloves, combine a CNN for local feature capture and Bi-directional Long Short-Term Memory (BiLSTM) networks for temporal feature extraction, culminating in a 95.05% accuracy [ 12 ]. Lin Z., et al. proposed a dynamic gesture recognition technique used a CNN with a 3D receptive field, achieves 97.5% accuracy [ 13 ]. Zhang X.J., et al. leveraged the AlexNet-based CNN model for gesture recognition, achieves an average accuracy of 98% [ 14 ]. Tsironi, et al. tried to use CNN and Long Short-Term Memory (LSTM) for dynamic gesture recognition, with a comprehensive accuracy of 80.10% [ 15 ]. Bao P., et al. advocated for a direct classification of seven gesture types via deep CNN, bypassing the segmentation or detection phases, and secured a 97.1% accuracy in simple background settings [ 16 ]. Oyedotun, et al. performed static gesture recognition based on deep learning and proposed CNN and stacked denoising autoencoder (SDAE), achieves an accuracy of 91.33% [ 17 ]. Molina J., et al. employed Deep Neural Networks (DNN) for classify seven gesture actions via Time-of-Flight (TOF) cameras, with a success rate of 94% [ 18 ]. Most of the above recognition models comprise several convolutional layers, max pooling layers, a variety of regularization layers and Transformer architectures. Therefore these models will have larger parameters, require more memory and longer model inference times. Most of studies employ deep learning models that utilize either RGB cameras to capture visible light or depth cameras to collect depth information that are suitable for a wide range of scenarios. However, gesture recognition accuracy may be affected by the quality of RGB images under different lighting conditions. Additionally, the high cost associated with depth cameras restricts their widespread adoption. [ 19 – 21 ]. Miniature infrared sensors employed thermal imaging technology effectively capture gesture information via temperature differentials in environment where insufficient light. This significantly enhances the adaptability and robustness of gesture recognition systems. By acquire thermal rather than color images, these sensors ensure user privacy at the hardware level, safeguard against the inadvertent disclosure of personal identity information. Thermal imaging cameras are distinguished by their ability to operate independently of ambient light conditions and their cost-efficiency compared to TOF cameras. This kind of sensor accurately capture thermal fluctuations induced by gestures through differentiating between the ambient background and the warmer human body temperature. After conducted a thorough review, this method has selected the MLX90640 infrared sensor due to its compact size, affordability, and high precision, which is characterized by a 32x24 pixel resolution and an adjustable sampling frequency. This study introduces a novel gesture recognition approach that utilizes a lightweight CNN architecture integrate a Spatial Transformer Network (STN) module, thereby enhancing the model's ability to handle variations in imagery. The effectiveness of this lightweight CNN framework was evaluated against the lightweight classification model FastViT [ 22 ], renowned for its superior performance on the ImageNet-1K dataset. The successful deployment of this optimized CNN model on the Raspberry Pi which low-energy computational platform, affirms its practical applicability and operational effectiveness in real-world scenarios. 2 Sign Language Gesture Collection 2.1 Collection Method The Melexis MLX90640 matrix infrared temperature sensor was utilized to collect thermal imaging data for gesture-based sign language. The sensor is equipped with an Inter-IC (I2C) interface for seamless integration with Raspberry Pi embedded system. A custom expansion board is essential to establish a reliable connection between the matrix infrared temperature sensor and the Raspberry Pi. This expansion board securely mounts the matrix infrared temperature sensor and ensures the smooth operation of the thermal imaging camera. It also facilitates accurate signal transmission through the implementation of pull-up resistors to effectively prevent potential voltage fluctuations or signal interference. One end of the pull-up resistor connects to the power supply, while the other end connects to the data and clock lines. The custom expansion board is thoughtfully designed with appropriate pins to facilitate secure attachment of the connection cables to the Raspberry Pi. Data from the matrix infrared temperature sensor is collected via the I2C interface of the Raspberry Pi. The connection method between the I2C interface of the Raspberry Pi and the expansion board is shown in the Raspberry Pi and the sensor are visualized in Figure 1. Table 1 Raspberry Pi connected to sensor Raspberry Pi MLX90640 3.3V VIN GND GND SDA(Pin3) SDA SCL(Pin5) SCL A comprehensive thermal imaging dataset of sign language gestures was created by connecting an MLX90640 matrix infrared temperature sensor to a Raspberry Pi to capture and classify various types of images. An organized folder structure is established, with image folders labeled from 0 to 9 to store the corresponding sign language gesture images. The MLX90640 matrix infrared temperature sensor operates by measure half of the 768 (32x24) pixels at a time, with each measurement referred to as a sub-page. A complete data frame consists of two sub-pages. The merged of these sub-pages is crucial for form the complete set of 768 pixels. Consequently, the two read arrays are normalized and merged into a complete data array after the sensor data is read twice consecutively. As a final step, the merged arrays are converted into grayscale images to forming a single measurement data frame. A Python-based thermal imaging collection program was developed for seamlessly capture thermal images via the Raspberry Pi embedded system to ensure efficient collection of thermal imaging data for sign language gestures. Real-time images are processed used bilinear interpolation and pseudo-color mapping techniques to enhance recognition of sign language gesture types, as opposed to display the original grayscale images directly. The intricate step-by-step process encompassing the entire collection program is depicted in Fig. 2. 2.2 Dataset Construction A dataset contains 4,500 thermal imaging pictures of sign language gestures was constructed following the steps shown in the Fig. 3 to provide necessary train data for deep learning. These gestures were collected from individuals of various ages, result in a diverse dataset due to variations in the angle and distance of the hand relative to the camera during image capture. Calculations and empirical test of the MLX90640 matrix infrared temperature sensor's Field of View (FoV: 55°x35°) established that gestures should be placed within a range of 12cm to 25cm from the sensor to ensure optimal image collection. Notably, the MLX90640 sensor has a resolution of merely 32×24 pixels. Consequently, the feature information necessary for traditional machine learning tasks is reduced and thereby increasing the difficulty of classification due to the captured images possess low resolution. The entire dataset of 4,500 images occupies a mere 3.4MB of disk space, which is even smaller than that occupied by a single 1080P image (1080x1920 pixels, 8-bit depth, RGB color space). As a result, the dataset's reduced storage requirements allow for flexibility when training various neural network models. The visual quality of each sign language gesture image varies depending on the hand's position, posture, and external factors unrelated to the thermal imaging sensor itself. The sensor primarily detects relative temperature differences between the background and foreground objects, which is effectively reflect the temperature differential of the human hand in relation to its background environment. Figure 3 exemplifies sign language gesture images from the same classes captured by the same individual. Specifically, Fig. 3 (a) represents an image of high quality: clear visibility of all fingers and the entire palm, with the center of the sign language gesture closely aligned with the center of the image. Such quality is desirable for effective deep learning training processes. It is essential to include images like those in Fig. 3 (b), where the posture of the gesture obscures the fingers to enhance the model's generalization ability. Figures 3 (c) and 3(d) respectively depict images of poor quality resulting from offset angles of gesture pitch and rotation. Machine learning algorithms require classification of gestures with varying temperatures and postures. Every sign language gesture is contributed by different individuals, wherein hand temperatures, distances, and postures can significantly vary. The dataset exhibits substantial diversification even within the same classes. Consequently, a vast amount of profoundly diverse thermal imaging gesture data were collected for this study, which enabling machine learning algorithms to acquire abundant features and successfully differentiate between distinct sign language gestures. Standard image classification tasks utilize high-resolution and high-quality images for training and learning purposes. These high-quality train images contribute to stability, reduce variations in image quality, and provide the advantage of faster image acquisition speed. However, when capture images of consistent quality used thermal imaging technology becomes challenge due to temperature differences among individuals within the same background environment. The thermal images employed in this research possess a significantly low resolution, with the representation of temperature difference between the hand and background limited to two color changes, without additional gradients or detail layers. As a result, classification becomes particularly challenging. Figure 4 showcases the 10 types of thermal imaging sign language gestures captured used the MLX90640 matrix infrared temperature sensor. Due to the sensor's low-resolution nature, higher temperature differences are required to distinguish objects from the background. The areas that appear gray in the thermal image shown in Fig. 4 are due to heat leakage where the camera captures heat from the surrounding air and walls. The low resolution also leads to visual similarities between certain images within the dataset, such as images 4(a) and 4(j), as well as images 4(d) and 4(h). While these similarities pose little difficulty for experienced experts in gesture classification, accurate classification by computers necessitates the employment of effective algorithms. Including a set of high-quality images substantially improves the success rate of accurate gesture detection for both humans and machines. Following the collection of 4,500 images (450 per classes), the dataset was partitioned into training, validation, and test sets according to the proportions specified in Table 2. Table 2 Data set proportion Class Train Validation Test 0 320 65 65 1 320 65 65 2 320 65 65 3 320 65 65 4 320 65 65 5 320 65 65 6 320 65 65 7 320 65 65 8 320 65 65 9 320 65 65 Summary 3200 650 650 3 Lightweight Gesture Recognition Model 3.1 Attention Mechanism The Spatial Transformer Network (STN) utilizes an attention mechanism to parameterize transformations through a localization network, allowing for direct prediction of transformation parameters \(\:\theta\:\) from the input data. This process comprises three key components: a localization network that predicts the transformation parameters, a grid generator that facilitates the creation of a sampling grid, and a sampler that applies the transformation to produce the transformed output feature map. The transformation can be mathematically represented as Eq. 1 : $$V_{i}^{c}=\sum\limits_{n}^{H} {\sum\limits_{m}^{W} {U_{{nm}}^{c}} } \cdot k(x_{i}^{s} - m;{\Phi _x}) \cdot k(y_{i}^{s} - n;{\Phi _y})$$ 1 Where \(\:{V}_{i}^{c}\) is the output pixel value at position \(\:i\) in channel \(\:c\) , \(\:{U}_{nm}^{c}\) ​ is the input pixel value at position \(\:(n,m)\) in channel \(\:c\) , \(\:k\) represents the sampling kernel parameterized by \(\:{{\Phi\:}}_{x}\) and \(\:{{\Phi\:}}_{y}\) ​, and \(\:({x}_{i}^{s},{y}_{i}^{s})\) represents the source coordinates in the input feature map. The STN module enables the network to output feature maps based on the input feature maps by spatial transformation. This is particularly critical for dealing with changes in gesture appearance due to scale, rotation, and pose changes in gesture recognition tasks. It can effectively reduce the difficulty of low-resolution infrared image classification because of the STN module actively transforms and magnifies the relevant features of gestures. When the STN attention mechanism is added to the proposed lightweight CNN model, the detection accuracy is increased by 1.5%. Figure 5 (a) is the image before the space transformation and Fig. 5 (b) is the image after the space transformation. 3.2 Depthwise Separable Convolution The structure for lightweighting the model is primarily inspired by the principle of depthwise separable convolution, which is found in the renowned MobileNet architecture[ 23 ]. Traditional CNN model often demand significant computational resources and memory space when process large-scale data, it is posing challenges in resource-constrained environments such as mobile devices or embedded systems. Therefore, there is a need to design more lightweight model structures that can adapt to such resource-limited scenarios. As an effective operation, the depthwise separable convolution reduces the number of parameters and computational cost. It achieves this by initially increasing dimensions through a 1x1 convolution, followed by utilizing depthwise convolution with higher input and output channels, which allows for the extraction of more information. The shortcut connection module, assists in address the issues of vanishing or exploding gradients during the train process of deep neural networks. By incorporating skip connections that directly pass the input to the output layer, the shortcut connection module minimizes information loss and accelerates the training process. Figure 6 illustrates the bottleneck structure, while the parameters for each layer are presented in Table 3 . Table 3 Bottleneck layer parameters LayerType Output Size Input (None, M, H, W) 1x1 Conv (None, MZ, H, W) 3x3 Depthwise Conv (None, MZ, H/S, W/S) 1x1 Conv (None, Z, H/S, W/S) Add (None, Z, H/S, W/S) Output (None, Z, H/S, W/S) 3.3 Improved Gesture Recognition Network The traditional deep learning models exhibit insufficient robustness when applied to classification tasks involving low-quality thermal images. In order to solve the above problems, the present study introduces a deep learning model that learns image features exclusively from the training dataset. The proposed model is characterized by its lightweight and efficient nature, feature 1,690,138 parameters and a weight size of 6.5 megabytes. Inspired by depthwise separable convolutional networks, the model is composed of multiple convolutional layers and two inverted residual structures. Moreover, the proposed model leverages 3x3 convolutional kernels which employed batch normalization and the ReLU activation function following each convolution, except for the initial convolutional layer. The overall structure of the model proposed in this study is shown in Fig. 7 while the parameters for each layer are presented in Table 4 . The input of the model is 32x24 pixel gray image collected by the thermal imaging sensor. The input size of the model is (None, 32, 32, 1). Where None represents the value of the number of parameters passed to the program for training, 32x32 represents the resolution of the training image, and 1 represents the input image channel as 1, which also signifies it is a grayscale image. The STN attention mechanism performs spatial transformation based on the input image's intrinsic features, resulting in the output of transformed feature maps. Since the spatial transformation does not alter the size and channel count of the image, the output feature maps maintain the same dimensions of 32x32 and a channel count of 1 as the input. In the inverted residual part, the input channel number is expanded through a 1x1 expansion layer used an expansion ratio (expand_ratio) to enhance the feature dimensions. This inverted residual block within an input channel number of 32 and an expansion ratio of 6, and the output channel number becomes 192. Following the inverted residual block, a 3x3 depthwise separable convolution layer (Depthwise Separable Convolution) performs individual convolution operations on each input channel. Finally, a 1x1 compression layer compresses the expanded channel number back to the target output channel number, followed by batch normalization. In the fully connected layer, the model first flattens the feature map of the last layer into a one-dimensional vector. Next, through a fully connected layer, these features are mapped to the target classes or values. The fully connected layer maps the flattened feature vector to corresponding 10 outputs. The thermal gesture images are transformed via the STN attention mechanism, afterwards convolution and a stack of inverted residual blocks, to obtain high-level features. These features are further processed to generate gesture recognition probabilities, which are then converted into probability distributions through the softmax activation function. Table 4 Lightweight CNN model parameters Layer Type Output Size Input (None, 32, 32, 1) STN (None, 32, 32, 1) Conv (None, 16, 16, 64) Conv (None, 8, 8, 128) Bottleneck (None, 8, 8, 128) Conv (None, 4, 4, 256) Bottleneck (None, 4, 4, 256) Conv (None, 2, 2, 128) Flatten (None, 512) Dense (None, 10) 4 Experiments and Results Analysis 4.1 Experimental Setup The model was tested in a 64-bit Windows 10 environment with Python 3.7 and PyTorch, on a Lenovo laptop with the following hardware specifications: CPU Intel i5-9300H, GPU GTX1650 with 4GB memory, and CUDAv10.2. The dataset used for the experiment is a custom low-resolution infrared sign language gesture dataset, as shown in Table 2. This custom infrared dataset is divided into training set, validation set and test set with 3,200, 650 and 650 images respectively. Each images being with being 32 pixels in height and 24 pixels in width, totaled 768 pixels. During training, the original image size was scaled to 32 pixels in both height and width, with the batch size set to 32. This experiment falls under the classes of image classification tasks, thus the Softmax function was adopted. 4.2 Model Comparison This research compares the proposed lightweight gesture recognition model with the FastViT model for the task of low-resolution infrared sign language gesture recognition. The FastViT model is a more advanced model for the ImageNet dataset classification task, it is outperforming other competitive architectures in various tasks such as image classification, detection, and segmentation. Moreover, it significantly reduces latency on mobile devices and desktop GPUs. The FastViT model comes in various sizes, each with a different architectural size. For this gesture recognition comparison, only utilizes the lightweight variant of the FastViT model. Table 5 presents a detailed comparison between the proposed lightweight gesture recognition model and the FastViT model. Table 5 Model comparison Model Accuracy Model size (MB) parameters Proposed Model 98.50% 6.53 1690138 FastViT-T8 92.27% 13.08 3427720 FastViT-T12 95.91% 25.46 6675304 FastViT-S12 99.77 % 40.78 10689768 FastViT-SA12 97.95% 40.78 10689768 The proposed lightweight CNN gesture recognition model achieved an accuracy of 98.50% on the test dataset. The confusion matrix obtained from testing on the test set is shown in Fig. 8 . Figure 8 is observed that the digits "0," "3," "5," "6," "8," and "9" could be correctly identified with high accuracy for their respective classes. The digits '4' and '7' managed to correctly identify their respective classes samples with 98% accuracy. 96% of digit '4' classes samples identified correctly, and 92% of digit '7' classes samples identified correctly. The average inference time for a test sample was 59ms on the Raspberry Pi edge computing device, less than the 62.5ms each frame of thermal imaging is captured. Comparing the performance of the proposed lightweight gesture recognition model with other gesture recognition models (shown in Table 5 ), it is evident that the proposed model is more lightweight. The model weight file and the number of parameters are less than half of those in the lightest version of the compared models, while achieves higher accuracy. Given the prevalence of thermal imaging sensors as standard peripherals in embedded devices, it is imperative to develop suitable models as assistive technologies for their widespread application in embedded devices and smartphones. The lightweight CNN model possesses a clear advantage in terms of seamless deployment in embedded devices and integration into mobile applications for real-time inference on smartphones. The notable enhancement in accuracy when compared to conventional models, can be attributed to the strategic integration of the STN attention mechanism and depth-wise separable convolution in the proposed model. 5 Conclusion This study presents a complete end-to-end embedded infrared gesture recognition system which is based on a low-resolution thermal imaging camera with a resolution of 32×24 pixels for gesture recognition. A thermal imaging dataset containing 4,500 images with 450 thermal images per gesture category was created to support this study. The study explores the impact of the spatial transformation network attention mechanism on gesture recognition accuracy, result in an improvement of 1.5% in accuracy for gesture recognition tasks. A gesture recognition method is proposed which combines lightweight CNN with the spatial transformation network attention mechanism, which achieves an accuracy of 98.5% on the test dataset. This approach significantly enhances the real-time performance and accuracy of thermal imaging gesture detection on embedded devices. The inference time for each frame image on the embedded device is only 59 milliseconds, which is less than the time to collect one frame of infrared image. It was considered an ideal surveillance technology for nighttime or low-light environments because the gesture recognition method based on thermal imaging technology is not influenced by background lighting conditions. Moreover, the utilization of thermal imaging technology solely captures thermal radiation instead of detailed personality features, thereby mitigate the risk of privacy leakage at the hardware level and provide substantial privacy protection advantages for users. Declarations Acknowledgments The authors would like to thank for the support by the National Natural Science Foundation, China (No. 52175256) and the Henan Provincial Science and Technology R&D Joint Fund (Key Program) , China (No. 225200810029). Author contributions All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by REN Hao, LI Hao, YUAN Kang-hui and ZHU Chu-feng. Supervision of the whole process by ZHANG Zhi-yuan. The first draft of the manuscript was written by ZHANG Zhi-yuan and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript. Funding Partial financial support was received from Project supported by the National Natural Science Foundation, China (No. 52175256) and the Henan Provincial Science and Technology R&D Joint Fund (Key Program) , China (No. 225200810029). Conflict of Interest The authors have no competing interests to declare that are relevant to the content of this article. Data Availability The thermal imaging gesture dataset supporting Table. 2 and Figs. 3–5, are published in the ScienceDB, as part of this record: https://doi.org/10.57760/sciencedb.08344. Thermal image data were captured from the volunteers at Zhengzhou University of Light Industry stored in .png format on ScienceDB as collection thermal imaging gesture dataset. All thermal image data were obtained from the Thermal Imaging Gesture Dataset (TIGD) are collect from the volunteers. Compliance with Ethical Standards Research involving Human Participants In the study, an infrared camera was used to collect human sign language gesture data. Gestures and postures are placed with the consent of the volunteers. The entire data collection process does not involve contact between experimental equipment and the human body. Informed consent All data collected in the study were obtained with the consent of the volunteers. The processing and using of data are carried out with the consent of the volunteers. References Ahmed S, Khan F, Ghaffar A, Hussain F, Cho SH. (2019) Finger-counting-based gesture recognition within cars using impulse radar with convolutional neural network. Sensors, 19(6):1429. https://doi.org/10.3390/s19061429 Nogales RE, Benalcázar ME (2021) Hand gesture recognition using machine learning and infrared information: a systematic literature review. Int. J. Mach. Learn. Cyber 12(10):2859-86. https://doi.org/10.1007/s13042-021-01372-y Quesada L, López G, Guerrero L (2017) Automatic recognition of the American sign language fingerspelling alphabet to assist people living with speech or hearing impairments. J AMB INTEL HUM COMP 8:625-35. https://doi.org/10.1007/s12652-017-0475-7 Tateno, Shigeyuki, Hongbin Liu, and Junhong Ou (2020) Development of sign language motion recognition system for hearing-impaired people using electromyography signal. Sensors 20(20):5807. https://doi.org/10.3390/s20205807 Obermeier, Christian, Thomas Dolk, Thomas C Gunter (2012) The benefit of gestures during communication: Evidence from hearing and hearing-impaired individuals. CORTEX 48(7):857-870. https://doi.org/10.1016/j.cortex.2011.02.007 Saad, M, Bleakley, CJ, Nigram, V, & Kettle, P (2018) Ultrasonic hand gesture recognition for mobile devices. J MULTIMODAL USER IN, 12:31-39. https://doi.org/10.1007/s12193-017-0257-8 Sagayam, K M, Hemanth, D J (2017) Hand posture and gesture recognition techniques for virtual reality applications: a survey. VIRTUAL REAL-LONDON 21 pp 91-107. https://doi.org/10.1007/s10055-016-0301-0 Stančić I, Musić J, Grujić T (2017) Gesture recognition system for real-time mobile robot control based on inertial sensors and motion strings. ENG APPL ARTIF INTEL 66:33-48. https://doi.org/10.1016/j.engappai.2017.08.013 Hu, Bin, Jiacun Wang (2020) Deep learning based hand gesture recognition and UAV flight controls. INT J AUTOM COMPUT 17(1): 17-29. https://doi.org/10.1007/s11633-019-1194-7 Mahmoud N M, Fouad H, Soliman A M (2021) Smart healthcare solutions using the internet of medical things for hand gesture recognition system. COMPLEX INTELL SYST 7:1253-1264. https://doi.org/10.1007/s40747-020-00194-9 Vandersteegen M, Reusen W, Van Beeck K, Goedemé T (2020) Low-latency hand gesture recognition with a low-resolution thermal imager. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . pp: 98-99. Wu J, Ren P, Song B, Zhang R, Zhao C, Zhang X (2023) Data glove-based gesture recognition using CNN-BiLSTM model with attention mechanism. PLOS ONE 18(11): e0294174. https://doi.org/10.1371/journal.pone.0294174 Lin Zhiwei, Zhu Wenzhang, Chen Hao (2021) Dynamic Gesture Recognition Based on Spatio-Temporal Feature Extraction of Neural Networks. Computer and Modernization 6: 41-47. Zhang Xingjian, Zhang Jianxin (2020) Gesture Recognition Control System Based on Convolutional Neural Networks. Computer Applications and Software 37(10): 220-224. Tsironi E, Barros P, Weber C, Wermter S (2017) An analysis of convolutional long short-term memory recurrent neural networks for gesture recognition. NEUROCOMPUTING 268: 76-86. https://doi.org/10.1016/j.neucom.2016.12.088 Bao P, Maqueda AI, del-Blanco CR, García N (2017) Tiny hand gesture recognition without localization via a deep convolutional network. IEEE T CONSUM ELECTR 63(3): 251-257. Oyedotun, Oyebade K., and Adnan Khashman (2017) Deep learning in vision-based static hand gesture recognition. NEURAL COMPUT APPL 28(12): 3941-3951. https://doi.org/10.1007/s00521-016-2294-8 Molina J, Pajuelo JA, Martínez JM (2017) Real-time motion-based hand gestures recognition from time-of-flight video. J SIGNAL PROCESS SYS 86:17-25. https://doi.org/10.1007/s11265-015-1090-5 Roh MC, Christmas B, Kittler J, Lee SW (2008) Gesture spotting for low-resolution sports video annotation. PATTERN RECOGN 41(3):1124-1137. https://doi.org/10.1016/j.patcog.2007.07.013 Pisharady PK, Saerbeck M (2015) Recent methods and databases in vision-based hand gesture recognition: A review. COMPUT VIS IMAGE UND 141: 152-165. https://doi.org/10.1016/j.cviu.2015.08.004 Vishwakarma DK, Grover V (2017) Hand gesture recognition in low-intensity environment using depth images. International Conference on Intelligent Sustainable Systems (ICISS) pp 429-433. https://doi.org/10.1109/ISS1.2017.8389446 Vasu PK, Gabriel J, Zhu J, Tuzel O, Ranjan A (2023) FastViT: A fast hybrid vision transformer using structural reparameterization. In the Proceedings of the IEEE/CVF International Conference on Computer Vision pp 5785-5795. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp 4510-4520. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 17 Sep, 2024 Reviews received at journal 08 Sep, 2024 Reviews received at journal 06 Sep, 2024 Reviews received at journal 06 Sep, 2024 Reviews received at journal 28 Aug, 2024 Reviewers agreed at journal 26 Aug, 2024 Reviewers agreed at journal 19 Aug, 2024 Reviewers agreed at journal 19 Aug, 2024 Reviewers agreed at journal 19 Aug, 2024 Reviewers agreed at journal 18 Aug, 2024 Reviewers agreed at journal 17 Aug, 2024 Reviewers agreed at journal 17 Aug, 2024 Reviewers invited by journal 17 Aug, 2024 Editor assigned by journal 09 Jul, 2024 Submission checks completed at journal 09 Jul, 2024 First submitted to journal 05 Jul, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4689884","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":328378285,"identity":"698a8855-b057-4678-abad-4b55c9c5e108","order_by":0,"name":"ZHANG Zhi-yuan","email":"","orcid":"","institution":"Zhengzhou University of Light Industry","correspondingAuthor":false,"prefix":"","firstName":"ZHANG","middleName":"","lastName":"Zhi-yuan","suffix":""},{"id":328378286,"identity":"3f147f8f-fc5e-4d69-b704-69fc9343aa3b","order_by":1,"name":"REN Hao","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABRUlEQVRIie2RsWrDMBBATxXYi6hWGRf8CzIGk1KX/IqNwVM6FUKGQBUM7lLImk79hXbJ7CDwFDoX1CFTpw7O1HQJlUynxIGOHfSGE9zpcacTgMXyDwloudqkE4YdTLebVmcctysMiA68TwkXTc7bdeJSF6JwYRTcFdhJBepR7D1WBfXmEPvmGvwqcEpBouY+caTPJRR+MlkG5xjw9rtiF+CWSwbT90PFxSKNCJGRVuTVaK3CSnfhXqUHI82YQfPR06XOCZM5l2imbiqFjJKGRmGjmCEhj9+SCUm4vHuWGPzLvRoapc6MEnyeUHI0W6QF9krnzEdCZVpBYtV1Ib2KXjKGtk4wxQSHD43KK4xKJF4ZcUhxO0ibIyWg869dttdfSddos5uq66f7Um5342RIXfny1k6PB+tdfHc4JqR/ESwWi8VyyA8JYWeCnVRh8AAAAABJRU5ErkJggg==","orcid":"","institution":"Zhengzhou University of Light Industry","correspondingAuthor":true,"prefix":"","firstName":"REN","middleName":"","lastName":"Hao","suffix":""},{"id":328378287,"identity":"0c48c8a9-7572-4be7-b411-091687ba8917","order_by":2,"name":"LI Hao","email":"","orcid":"","institution":"Zhengzhou University of Light Industry","correspondingAuthor":false,"prefix":"","firstName":"LI","middleName":"","lastName":"Hao","suffix":""},{"id":328378288,"identity":"2ee3b0fa-1e1d-4b73-a97a-8aa7cd1546f8","order_by":3,"name":"YUAN Kang-hui","email":"","orcid":"","institution":"Zhengzhou University of Light Industry","correspondingAuthor":false,"prefix":"","firstName":"YUAN","middleName":"","lastName":"Kang-hui","suffix":""},{"id":328378289,"identity":"04d4274e-8a47-4b92-86c7-afa7e73c1c1d","order_by":4,"name":"Chu-feng ZHU","email":"","orcid":"","institution":"Zhengzhou University of Light Industry","correspondingAuthor":false,"prefix":"","firstName":"Chu-feng","middleName":"","lastName":"ZHU","suffix":""}],"badges":[],"createdAt":"2024-07-05 05:51:58","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4689884/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4689884/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":62184283,"identity":"2ca0abe1-3acc-4832-af55-fb0ec2aa4dd7","added_by":"auto","created_at":"2024-08-10 11:40:58","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":214323,"visible":true,"origin":"","legend":"\u003cp\u003eRaspberry Pi connected to sensor\u003c/p\u003e","description":"","filename":"Figure1.png","url":"https://assets-eu.researchsquare.com/files/rs-4689884/v1/650d1e364943c9011bddf09e.png"},{"id":62183071,"identity":"639326a2-f588-4178-ace3-9bef8d00f083","added_by":"auto","created_at":"2024-08-10 11:32:58","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":26262,"visible":true,"origin":"","legend":"\u003cp\u003eThermal imaging gesture collection steps\u003c/p\u003e","description":"","filename":"Figure2.png","url":"https://assets-eu.researchsquare.com/files/rs-4689884/v1/bd1a149e17aac22c60ce53a6.png"},{"id":62184281,"identity":"e105b840-3ca5-4282-8168-bce544a0da51","added_by":"auto","created_at":"2024-08-10 11:40:58","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":78727,"visible":true,"origin":"","legend":"\u003cp\u003eGesture images of different qualities\u003c/p\u003e","description":"","filename":"Figure3.png","url":"https://assets-eu.researchsquare.com/files/rs-4689884/v1/64b9d4de9e10c684badd844c.png"},{"id":62184280,"identity":"af3b6e4b-6a1e-4cf2-9664-fba42bcd10c0","added_by":"auto","created_at":"2024-08-10 11:40:58","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":104442,"visible":true,"origin":"","legend":"\u003cp\u003eThermal sign language gestures all categories\u003c/p\u003e","description":"","filename":"Figure4.png","url":"https://assets-eu.researchsquare.com/files/rs-4689884/v1/bb2b156915d2b4eea5fd8e0a.png"},{"id":62183068,"identity":"7bdb947f-a367-49f0-8a15-126dae9a075f","added_by":"auto","created_at":"2024-08-10 11:32:58","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":12613,"visible":true,"origin":"","legend":"\u003cp\u003eComparison after spatial transformation\u003c/p\u003e","description":"","filename":"Figure5.png","url":"https://assets-eu.researchsquare.com/files/rs-4689884/v1/55fd58fa440af8a01683ae3e.png"},{"id":62184979,"identity":"ccb4aba5-6175-4479-8275-04afa4df71f3","added_by":"auto","created_at":"2024-08-10 11:48:58","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":10676,"visible":true,"origin":"","legend":"\u003cp\u003eBottleneck structure\u003c/p\u003e","description":"","filename":"Figure6.png","url":"https://assets-eu.researchsquare.com/files/rs-4689884/v1/29718689eb5eeab96c33e61e.png"},{"id":62183075,"identity":"b99cd14d-00a3-4043-8850-72f9d703c29f","added_by":"auto","created_at":"2024-08-10 11:32:58","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":15403,"visible":true,"origin":"","legend":"\u003cp\u003eLightweight CNN model structure\u003c/p\u003e","description":"","filename":"Figure7.png","url":"https://assets-eu.researchsquare.com/files/rs-4689884/v1/62cec50452aa1a80b62ce3e7.png"},{"id":62183073,"identity":"997fd729-d3b6-44fd-90db-93326b2c464f","added_by":"auto","created_at":"2024-08-10 11:32:58","extension":"png","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":27618,"visible":true,"origin":"","legend":"\u003cp\u003eConfusion matrix\u003c/p\u003e","description":"","filename":"Figure8.png","url":"https://assets-eu.researchsquare.com/files/rs-4689884/v1/d1d3a873f1b6bf90a9c56185.png"},{"id":62185752,"identity":"722d488a-9deb-4cd7-bba7-5349a117c9f5","added_by":"auto","created_at":"2024-08-10 11:56:59","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":442647,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4689884/v1/af3ab0e0-c557-41c6-adb9-a440039a3549.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Static Gesture Recognition Based on Thermal Imaging Sensors","fulltext":[{"header":"1 Introduction","content":"\u003cp\u003eGestures serve as a silent yet potent form of communication, finding utility in diverse fields such as automotive interfaces, healthcare systems, assistive technologies, entertainment and human-computer interaction [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. This versatility enables effective and contactless communication across various contexts, which not only enables immersive virtual reality (VR) and provides touchless control over smart devices but also assists individuals with hear and speech impairments. It plays a crucial role in advanced domains such as robotic control technology and medical diagnosis [\u003cspan additionalcitationids=\"CR4 CR5 CR6 CR7 CR8 CR9\" citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe advent of deep learning has propelled advancements in digital gesture recognition, with methodologies predominantly based on Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). CNN is widely recognized for their superior ability in image classification and feature extraction, whereas RNN excel in handling sequential data. These attributes allow for the nuanced recognition of complex gestures. V.M., et al. used a low-resolution thermal imaging camera for gesture prediction and introduced a novel detection technique that integrates 2D CNN with Temporal Convolutional Networks (TCN). This approach attained a classification accuracy of 95.9% and a mean Average Precision (mAP) [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Wu J., et al. developed a dynamic gesture recognition model that employs data gloves, combine a CNN for local feature capture and Bi-directional Long Short-Term Memory (BiLSTM) networks for temporal feature extraction, culminating in a 95.05% accuracy [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. Lin Z., et al. proposed a dynamic gesture recognition technique used a CNN with a 3D receptive field, achieves 97.5% accuracy [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. Zhang X.J., et al. leveraged the AlexNet-based CNN model for gesture recognition, achieves an average accuracy of 98% [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. Tsironi, et al. tried to use CNN and Long Short-Term Memory (LSTM) for dynamic gesture recognition, with a comprehensive accuracy of 80.10% [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Bao P., et al. advocated for a direct classification of seven gesture types via deep CNN, bypassing the segmentation or detection phases, and secured a 97.1% accuracy in simple background settings [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. Oyedotun, et al. performed static gesture recognition based on deep learning and proposed CNN and stacked denoising autoencoder (SDAE), achieves an accuracy of 91.33% [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Molina J., et al. employed Deep Neural Networks (DNN) for classify seven gesture actions via Time-of-Flight (TOF) cameras, with a success rate of 94% [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. Most of the above recognition models comprise several convolutional layers, max pooling layers, a variety of regularization layers and Transformer architectures. Therefore these models will have larger parameters, require more memory and longer model inference times.\u003c/p\u003e \u003cp\u003eMost of studies employ deep learning models that utilize either RGB cameras to capture visible light or depth cameras to collect depth information that are suitable for a wide range of scenarios. However, gesture recognition accuracy may be affected by the quality of RGB images under different lighting conditions. Additionally, the high cost associated with depth cameras restricts their widespread adoption. [\u003cspan additionalcitationids=\"CR20\" citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eMiniature infrared sensors employed thermal imaging technology effectively capture gesture information via temperature differentials in environment where insufficient light. This significantly enhances the adaptability and robustness of gesture recognition systems. By acquire thermal rather than color images, these sensors ensure user privacy at the hardware level, safeguard against the inadvertent disclosure of personal identity information.\u003c/p\u003e \u003cp\u003eThermal imaging cameras are distinguished by their ability to operate independently of ambient light conditions and their cost-efficiency compared to TOF cameras. This kind of sensor accurately capture thermal fluctuations induced by gestures through differentiating between the ambient background and the warmer human body temperature. After conducted a thorough review, this method has selected the MLX90640 infrared sensor due to its compact size, affordability, and high precision, which is characterized by a 32x24 pixel resolution and an adjustable sampling frequency. This study introduces a novel gesture recognition approach that utilizes a lightweight CNN architecture integrate a Spatial Transformer Network (STN) module, thereby enhancing the model's ability to handle variations in imagery. The effectiveness of this lightweight CNN framework was evaluated against the lightweight classification model FastViT [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e], renowned for its superior performance on the ImageNet-1K dataset. The successful deployment of this optimized CNN model on the Raspberry Pi which low-energy computational platform, affirms its practical applicability and operational effectiveness in real-world scenarios.\u003c/p\u003e"},{"header":"2 Sign Language Gesture Collection","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Collection Method\u003c/h2\u003e \u003cp\u003eThe Melexis MLX90640 matrix infrared temperature sensor was utilized to collect thermal imaging data for gesture-based sign language. The sensor is equipped with an Inter-IC (I2C) interface for seamless integration with Raspberry Pi embedded system. A custom expansion board is essential to establish a reliable connection between the matrix infrared temperature sensor and the Raspberry Pi. This expansion board securely mounts the matrix infrared temperature sensor and ensures the smooth operation of the thermal imaging camera. It also facilitates accurate signal transmission through the implementation of pull-up resistors to effectively prevent potential voltage fluctuations or signal interference. One end of the pull-up resistor connects to the power supply, while the other end connects to the data and clock lines. The custom expansion board is thoughtfully designed with appropriate pins to facilitate secure attachment of the connection cables to the Raspberry Pi. Data from the matrix infrared temperature sensor is collected via the I2C interface of the Raspberry Pi. The connection method between the I2C interface of the Raspberry Pi and the expansion board is shown in the Raspberry Pi and the sensor are visualized in Figure 1.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eRaspberry Pi connected to sensor\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRaspberry Pi\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMLX90640\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e3.3V\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVIN\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGND\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGND\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSDA(Pin3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSDA\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSCL(Pin5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSCL\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eA comprehensive thermal imaging dataset of sign language gestures was created by connecting an MLX90640 matrix infrared temperature sensor to a Raspberry Pi to capture and classify various types of images. An organized folder structure is established, with image folders labeled from 0 to 9 to store the corresponding sign language gesture images. The MLX90640 matrix infrared temperature sensor operates by measure half of the 768 (32x24) pixels at a time, with each measurement referred to as a sub-page. A complete data frame consists of two sub-pages. The merged of these sub-pages is crucial for form the complete set of 768 pixels. Consequently, the two read arrays are normalized and merged into a complete data array after the sensor data is read twice consecutively. As a final step, the merged arrays are converted into grayscale images to forming a single measurement data frame. A Python-based thermal imaging collection program was developed for seamlessly capture thermal images via the Raspberry Pi embedded system to ensure efficient collection of thermal imaging data for sign language gestures. Real-time images are processed used bilinear interpolation and pseudo-color mapping techniques to enhance recognition of sign language gesture types, as opposed to display the original grayscale images directly. The intricate step-by-step process encompassing the entire collection program is depicted in Fig.\u0026nbsp;2.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Dataset Construction\u003c/h2\u003e \u003cp\u003eA dataset contains 4,500 thermal imaging pictures of sign language gestures was constructed following the steps shown in the Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e3\u003c/span\u003e to provide necessary train data for deep learning. These gestures were collected from individuals of various ages, result in a diverse dataset due to variations in the angle and distance of the hand relative to the camera during image capture. Calculations and empirical test of the MLX90640 matrix infrared temperature sensor's Field of View (FoV: 55\u0026deg;x35\u0026deg;) established that gestures should be placed within a range of 12cm to 25cm from the sensor to ensure optimal image collection.\u003c/p\u003e \u003cp\u003eNotably, the MLX90640 sensor has a resolution of merely 32\u0026times;24 pixels. Consequently, the feature information necessary for traditional machine learning tasks is reduced and thereby increasing the difficulty of classification due to the captured images possess low resolution. The entire dataset of 4,500 images occupies a mere 3.4MB of disk space, which is even smaller than that occupied by a single 1080P image (1080x1920 pixels, 8-bit depth, RGB color space). As a result, the dataset's reduced storage requirements allow for flexibility when training various neural network models.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe visual quality of each sign language gesture image varies depending on the hand's position, posture, and external factors unrelated to the thermal imaging sensor itself. The sensor primarily detects relative temperature differences between the background and foreground objects, which is effectively reflect the temperature differential of the human hand in relation to its background environment. Figure\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e3\u003c/span\u003e exemplifies sign language gesture images from the same classes captured by the same individual. Specifically, Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e3\u003c/span\u003e(a) represents an image of high quality: clear visibility of all fingers and the entire palm, with the center of the sign language gesture closely aligned with the center of the image. Such quality is desirable for effective deep learning training processes. It is essential to include images like those in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e3\u003c/span\u003e(b), where the posture of the gesture obscures the fingers to enhance the model's generalization ability. Figures\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e3\u003c/span\u003e(c) and 3(d) respectively depict images of poor quality resulting from offset angles of gesture pitch and rotation. Machine learning algorithms require classification of gestures with varying temperatures and postures. Every sign language gesture is contributed by different individuals, wherein hand temperatures, distances, and postures can significantly vary. The dataset exhibits substantial diversification even within the same classes. Consequently, a vast amount of profoundly diverse thermal imaging gesture data were collected for this study, which enabling machine learning algorithms to acquire abundant features and successfully differentiate between distinct sign language gestures.\u003c/p\u003e \u003cp\u003eStandard image classification tasks utilize high-resolution and high-quality images for training and learning purposes. These high-quality train images contribute to stability, reduce variations in image quality, and provide the advantage of faster image acquisition speed. However, when capture images of consistent quality used thermal imaging technology becomes challenge due to temperature differences among individuals within the same background environment. The thermal images employed in this research possess a significantly low resolution, with the representation of temperature difference between the hand and background limited to two color changes, without additional gradients or detail layers. As a result, classification becomes particularly challenging. Figure\u0026nbsp;4 showcases the 10 types of thermal imaging sign language gestures captured used the MLX90640 matrix infrared temperature sensor. Due to the sensor's low-resolution nature, higher temperature differences are required to distinguish objects from the background. The areas that appear gray in the thermal image shown in Fig.\u0026nbsp;4 are due to heat leakage where the camera captures heat from the surrounding air and walls. The low resolution also leads to visual similarities between certain images within the dataset, such as images 4(a) and 4(j), as well as images 4(d) and 4(h). While these similarities pose little difficulty for experienced experts in gesture classification, accurate classification by computers necessitates the employment of effective algorithms. Including a set of high-quality images substantially improves the success rate of accurate gesture detection for both humans and machines.\u003c/p\u003e \u003cp\u003e Following the collection of 4,500 images (450 per classes), the dataset was partitioned into training, validation, and test sets according to the proportions specified in Table\u0026nbsp;2.\u003c/p\u003e \u003cp\u003e Table 2 Data set proportion\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Taba\" border=\"1\"\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eClass\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTrain\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eValidation\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTest\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e320\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e320\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e320\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e320\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e320\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e320\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e320\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e320\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e320\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e320\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSummary\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e3200\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e650\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e650\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"3 Lightweight Gesture Recognition Model","content":"\u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Attention Mechanism\u003c/h2\u003e \u003cp\u003eThe Spatial Transformer Network (STN) utilizes an attention mechanism to parameterize transformations through a localization network, allowing for direct prediction of transformation parameters \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\theta\\:\\)\u003c/span\u003e\u003c/span\u003e from the input data. This process comprises three key components: a localization network that predicts the transformation parameters, a grid generator that facilitates the creation of a sampling grid, and a sampler that applies the transformation to produce the transformed output feature map. The transformation can be mathematically represented as Eq.\u0026nbsp;\u003cspan refid=\"Equ1\" class=\"InternalRef\"\u003e1\u003c/span\u003e:\u003cdiv id=\"Equ1\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ1\" name=\"EquationSource\"\u003e\n$$V_{i}^{c}=\\sum\\limits_{n}^{H} {\\sum\\limits_{m}^{W} {U_{{nm}}^{c}} } \\cdot k(x_{i}^{s} - m;{\\Phi _x}) \\cdot k(y_{i}^{s} - n;{\\Phi _y})$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e1\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eWhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{V}_{i}^{c}\\)\u003c/span\u003e\u003c/span\u003e is the output pixel value at position \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:i\\)\u003c/span\u003e\u003c/span\u003e in channel \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:c\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{U}_{nm}^{c}\\)\u003c/span\u003e\u003c/span\u003e​ is the input pixel value at position \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:(n,m)\\)\u003c/span\u003e\u003c/span\u003e in channel \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:c\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:k\\)\u003c/span\u003e\u003c/span\u003e represents the sampling kernel parameterized by \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{{\\Phi\\:}}_{x}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{{\\Phi\\:}}_{y}\\)\u003c/span\u003e\u003c/span\u003e​, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:({x}_{i}^{s},{y}_{i}^{s})\\)\u003c/span\u003e\u003c/span\u003e represents the source coordinates in the input feature map. The STN module enables the network to output feature maps based on the input feature maps by spatial transformation. This is particularly critical for dealing with changes in gesture appearance due to scale, rotation, and pose changes in gesture recognition tasks. It can effectively reduce the difficulty of low-resolution infrared image classification because of the STN module actively transforms and magnifies the relevant features of gestures. When the STN attention mechanism is added to the proposed lightweight CNN model, the detection accuracy is increased by 1.5%. Figure\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e5\u003c/span\u003e(a) is the image before the space transformation and Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e5\u003c/span\u003e(b) is the image after the space transformation.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Depthwise Separable Convolution\u003c/h2\u003e \u003cp\u003eThe structure for lightweighting the model is primarily inspired by the principle of depthwise separable convolution, which is found in the renowned MobileNet architecture[\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. Traditional CNN model often demand significant computational resources and memory space when process large-scale data, it is posing challenges in resource-constrained environments such as mobile devices or embedded systems. Therefore, there is a need to design more lightweight model structures that can adapt to such resource-limited scenarios. As an effective operation, the depthwise separable convolution reduces the number of parameters and computational cost. It achieves this by initially increasing dimensions through a 1x1 convolution, followed by utilizing depthwise convolution with higher input and output channels, which allows for the extraction of more information. The shortcut connection module, assists in address the issues of vanishing or exploding gradients during the train process of deep neural networks. By incorporating skip connections that directly pass the input to the output layer, the shortcut connection module minimizes information loss and accelerates the training process. Figure\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e6\u003c/span\u003e illustrates the bottleneck structure, while the parameters for each layer are presented in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e3\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eTable 3 Bottleneck layer parameters\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Tabb\" border=\"1\"\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003eLayerType\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eOutput Size\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003eInput\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(None, M, H, W)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003e1x1 Conv\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(None, MZ, H, W)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003e3x3 Depthwise Conv\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(None, MZ, H/S, W/S)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003e1x1 Conv\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(None, Z, H/S, W/S)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" colname=\"c1\"\u003e \u003cp\u003eAdd\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, Z, H/S, W/S)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003eOutput\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(None, Z, H/S, W/S)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Improved Gesture Recognition Network\u003c/h2\u003e \u003cp\u003eThe traditional deep learning models exhibit insufficient robustness when applied to classification tasks involving low-quality thermal images. In order to solve the above problems, the present study introduces a deep learning model that learns image features exclusively from the training dataset. The proposed model is characterized by its lightweight and efficient nature, feature 1,690,138 parameters and a weight size of 6.5 megabytes. Inspired by depthwise separable convolutional networks, the model is composed of multiple convolutional layers and two inverted residual structures. Moreover, the proposed model leverages 3x3 convolutional kernels which employed batch normalization and the ReLU activation function following each convolution, except for the initial convolutional layer.\u003c/p\u003e \u003cp\u003eThe overall structure of the model proposed in this study is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e7\u003c/span\u003e while the parameters for each layer are presented in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e4\u003c/span\u003e. The input of the model is 32x24 pixel gray image collected by the thermal imaging sensor. The input size of the model is (None, 32, 32, 1). Where None represents the value of the number of parameters passed to the program for training, 32x32 represents the resolution of the training image, and 1 represents the input image channel as 1, which also signifies it is a grayscale image. The STN attention mechanism performs spatial transformation based on the input image's intrinsic features, resulting in the output of transformed feature maps. Since the spatial transformation does not alter the size and channel count of the image, the output feature maps maintain the same dimensions of 32x32 and a channel count of 1 as the input. In the inverted residual part, the input channel number is expanded through a 1x1 expansion layer used an expansion ratio (expand_ratio) to enhance the feature dimensions. This inverted residual block within an input channel number of 32 and an expansion ratio of 6, and the output channel number becomes 192. Following the inverted residual block, a 3x3 depthwise separable convolution layer (Depthwise Separable Convolution) performs individual convolution operations on each input channel. Finally, a 1x1 compression layer compresses the expanded channel number back to the target output channel number, followed by batch normalization. In the fully connected layer, the model first flattens the feature map of the last layer into a one-dimensional vector. Next, through a fully connected layer, these features are mapped to the target classes or values. The fully connected layer maps the flattened feature vector to corresponding 10 outputs. The thermal gesture images are transformed via the STN attention mechanism, afterwards convolution and a stack of inverted residual blocks, to obtain high-level features. These features are further processed to generate gesture recognition probabilities, which are then converted into probability distributions through the softmax activation function.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eLightweight CNN model parameters\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLayer Type\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOutput Size\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInput\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 32, 32, 1)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSTN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 32, 32, 1)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eConv\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 16, 16, 64)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eConv\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 8, 8, 128)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBottleneck\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 8, 8, 128)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eConv\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 4, 4, 256)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBottleneck\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 4, 4, 256)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eConv\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 2, 2, 128)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFlatten\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 512)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDense\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 10)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"4 Experiments and Results Analysis","content":"\u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Experimental Setup\u003c/h2\u003e \u003cp\u003eThe model was tested in a 64-bit Windows 10 environment with Python 3.7 and PyTorch, on a Lenovo laptop with the following hardware specifications: CPU Intel i5-9300H, GPU GTX1650 with 4GB memory, and CUDAv10.2. The dataset used for the experiment is a custom low-resolution infrared sign language gesture dataset, as shown in Table\u0026nbsp;2. This custom infrared dataset is divided into training set, validation set and test set with 3,200, 650 and 650 images respectively. Each images being with being 32 pixels in height and 24 pixels in width, totaled 768 pixels. During training, the original image size was scaled to 32 pixels in both height and width, with the batch size set to 32. This experiment falls under the classes of image classification tasks, thus the Softmax function was adopted.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Model Comparison\u003c/h2\u003e \u003cp\u003eThis research compares the proposed lightweight gesture recognition model with the FastViT model for the task of low-resolution infrared sign language gesture recognition. The FastViT model is a more advanced model for the ImageNet dataset classification task, it is outperforming other competitive architectures in various tasks such as image classification, detection, and segmentation. Moreover, it significantly reduces latency on mobile devices and desktop GPUs. The FastViT model comes in various sizes, each with a different architectural size. For this gesture recognition comparison, only utilizes the lightweight variant of the FastViT model. Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e5\u003c/span\u003e presents a detailed comparison between the proposed lightweight gesture recognition model and the FastViT model.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eModel comparison\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eModel size (MB)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eparameters\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eProposed Model\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e98.50%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e6.53\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e1690138\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFastViT-T8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e92.27%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e13.08\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3427720\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFastViT-T12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e95.91%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e25.46\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e6675304\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFastViT-S12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e99.77\u003c/b\u003e%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e40.78\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e10689768\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFastViT-SA12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e97.95%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e40.78\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e10689768\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe proposed lightweight CNN gesture recognition model achieved an accuracy of 98.50% on the test dataset. The confusion matrix obtained from testing on the test set is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e8\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eFigure\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e8\u003c/span\u003e is observed that the digits \"0,\" \"3,\" \"5,\" \"6,\" \"8,\" and \"9\" could be correctly identified with high accuracy for their respective classes. The digits '4' and '7' managed to correctly identify their respective classes samples with 98% accuracy. 96% of digit '4' classes samples identified correctly, and 92% of digit '7' classes samples identified correctly. The average inference time for a test sample was 59ms on the Raspberry Pi edge computing device, less than the 62.5ms each frame of thermal imaging is captured. Comparing the performance of the proposed lightweight gesture recognition model with other gesture recognition models (shown in Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e5\u003c/span\u003e), it is evident that the proposed model is more lightweight. The model weight file and the number of parameters are less than half of those in the lightest version of the compared models, while achieves higher accuracy.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eGiven the prevalence of thermal imaging sensors as standard peripherals in embedded devices, it is imperative to develop suitable models as assistive technologies for their widespread application in embedded devices and smartphones. The lightweight CNN model possesses a clear advantage in terms of seamless deployment in embedded devices and integration into mobile applications for real-time inference on smartphones. The notable enhancement in accuracy when compared to conventional models, can be attributed to the strategic integration of the STN attention mechanism and depth-wise separable convolution in the proposed model.\u003c/p\u003e \u003c/div\u003e"},{"header":"5 Conclusion","content":"\u003cp\u003eThis study presents a complete end-to-end embedded infrared gesture recognition system which is based on a low-resolution thermal imaging camera with a resolution of 32\u0026times;24 pixels for gesture recognition. A thermal imaging dataset containing 4,500 images with 450 thermal images per gesture category was created to support this study. The study explores the impact of the spatial transformation network attention mechanism on gesture recognition accuracy, result in an improvement of 1.5% in accuracy for gesture recognition tasks. A gesture recognition method is proposed which combines lightweight CNN with the spatial transformation network attention mechanism, which achieves an accuracy of 98.5% on the test dataset. This approach significantly enhances the real-time performance and accuracy of thermal imaging gesture detection on embedded devices. The inference time for each frame image on the embedded device is only 59 milliseconds, which is less than the time to collect one frame of infrared image. It was considered an ideal surveillance technology for nighttime or low-light environments because the gesture recognition method based on thermal imaging technology is not influenced by background lighting conditions. Moreover, the utilization of thermal imaging technology solely captures thermal radiation instead of detailed personality features, thereby mitigate the risk of privacy leakage at the hardware level and provide substantial privacy protection advantages for users.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAcknowledgments\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors would like to thank for the support by the National Natural Science Foundation, China (No. 52175256) and the Henan Provincial Science and Technology R\u0026amp;D Joint Fund (Key Program) , China (No. 225200810029).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor contributions\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by REN Hao, LI Hao, YUAN Kang-hui and ZHU Chu-feng. Supervision of the whole process by\u0026nbsp;ZHANG Zhi-yuan. The first draft of the manuscript was written by\u0026nbsp;ZHANG Zhi-yuan\u0026nbsp;and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.\u003c/p\u003e\n\u003cp\u003eFunding\u003c/p\u003e\n\u003cp\u003ePartial financial support was received from Project supported by the National Natural Science Foundation, China (No. 52175256) and the Henan Provincial Science and Technology R\u0026amp;D Joint Fund (Key Program) , China (No. 225200810029).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConflict of Interest\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors have no competing interests to declare that are relevant to the content of this article.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData Availability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe thermal imaging gesture dataset supporting Table. 2 and Figs. 3\u0026ndash;5, are published in the ScienceDB, as part of this record: https://doi.org/10.57760/sciencedb.08344.\u003c/p\u003e\n\u003cp\u003eThermal image data were captured from the volunteers at Zhengzhou University of Light Industry stored in .png format on ScienceDB as collection thermal imaging gesture dataset. All thermal image data were obtained from the Thermal Imaging Gesture Dataset (TIGD) are collect from the volunteers.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompliance with Ethical Standards\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eResearch involving Human Participants\u003c/p\u003e\n\u003cp\u003eIn the study, an infrared camera was used to collect human sign language gesture data. Gestures and postures are placed with the consent of the volunteers. The entire data collection process does not involve contact between experimental equipment and the human body.\u003c/p\u003e\n\u003cp\u003eInformed consent \u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAll data collected in the study were obtained with the consent of the volunteers. The processing and using of data are carried out with the consent of the volunteers.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAhmed S, Khan F, Ghaffar A, Hussain F, Cho SH. (2019) Finger-counting-based gesture recognition within cars using impulse radar with convolutional neural network. Sensors, 19(6):1429. https://doi.org/10.3390/s19061429\u003c/li\u003e\n\u003cli\u003eNogales RE, Benalc\u0026aacute;zar ME (2021) Hand gesture recognition using machine learning and infrared information: a systematic literature review. Int. J. Mach. Learn. Cyber 12(10):2859-86. https://doi.org/10.1007/s13042-021-01372-y\u003c/li\u003e\n\u003cli\u003eQuesada L, L\u0026oacute;pez G, Guerrero L (2017) Automatic recognition of the American sign language fingerspelling alphabet to assist people living with speech or hearing impairments. J AMB INTEL HUM COMP 8:625-35. https://doi.org/10.1007/s12652-017-0475-7\u003c/li\u003e\n\u003cli\u003eTateno, Shigeyuki, Hongbin Liu, and Junhong Ou (2020) Development of sign language motion recognition system for hearing-impaired people using electromyography signal. Sensors 20(20):5807. https://doi.org/10.3390/s20205807\u003c/li\u003e\n\u003cli\u003eObermeier, Christian, Thomas Dolk, Thomas C Gunter (2012) The benefit of gestures during communication: Evidence from hearing and hearing-impaired individuals. CORTEX 48(7):857-870. https://doi.org/10.1016/j.cortex.2011.02.007\u003c/li\u003e\n\u003cli\u003eSaad, M, Bleakley, CJ, Nigram, V, \u0026amp; Kettle, P (2018) Ultrasonic hand gesture recognition for mobile devices. J MULTIMODAL USER IN, 12:31-39. https://doi.org/10.1007/s12193-017-0257-8\u003c/li\u003e\n\u003cli\u003eSagayam, K M, Hemanth, D J (2017) Hand posture and gesture recognition techniques for virtual reality applications: a survey. VIRTUAL REAL-LONDON 21 pp 91-107. https://doi.org/10.1007/s10055-016-0301-0\u003c/li\u003e\n\u003cli\u003eStančić I, Musić J, Grujić T (2017) Gesture recognition system for real-time mobile robot control based on inertial sensors and motion strings. ENG APPL ARTIF INTEL 66:33-48. https://doi.org/10.1016/j.engappai.2017.08.013\u003c/li\u003e\n\u003cli\u003eHu, Bin, Jiacun Wang (2020) Deep learning based hand gesture recognition and UAV flight controls. INT J AUTOM COMPUT 17(1): 17-29. https://doi.org/10.1007/s11633-019-1194-7\u003c/li\u003e\n\u003cli\u003eMahmoud N M, Fouad H, Soliman A M (2021) Smart healthcare solutions using the internet of medical things for hand gesture recognition system. COMPLEX INTELL SYST 7:1253-1264. https://doi.org/10.1007/s40747-020-00194-9\u003c/li\u003e\n\u003cli\u003eVandersteegen M, Reusen W, Van Beeck K, Goedem\u0026eacute; T (2020) Low-latency hand gesture recognition with a low-resolution thermal imager. In \u003cem\u003eProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops\u003c/em\u003e . pp: 98-99.\u003c/li\u003e\n\u003cli\u003eWu J, Ren P, Song B, Zhang R, Zhao C, Zhang X (2023) Data glove-based gesture recognition using CNN-BiLSTM model with attention mechanism. PLOS ONE 18(11): e0294174. https://doi.org/10.1371/journal.pone.0294174\u003c/li\u003e\n\u003cli\u003eLin Zhiwei, Zhu Wenzhang, Chen Hao (2021) Dynamic Gesture Recognition Based on Spatio-Temporal Feature Extraction of Neural Networks. Computer and Modernization 6: 41-47.\u003c/li\u003e\n\u003cli\u003eZhang Xingjian, Zhang Jianxin (2020) Gesture Recognition Control System Based on Convolutional Neural Networks. Computer Applications and Software 37(10): 220-224.\u003c/li\u003e\n\u003cli\u003eTsironi E, Barros P, Weber C, Wermter S (2017) An analysis of convolutional long short-term memory recurrent neural networks for gesture recognition. NEUROCOMPUTING 268: 76-86. https://doi.org/10.1016/j.neucom.2016.12.088\u003c/li\u003e\n\u003cli\u003eBao P, Maqueda AI, del-Blanco CR, Garc\u0026iacute;a N (2017) Tiny hand gesture recognition without localization via a deep convolutional network. IEEE T CONSUM ELECTR 63(3): 251-257.\u003c/li\u003e\n\u003cli\u003eOyedotun, Oyebade K., and Adnan Khashman (2017) Deep learning in vision-based static hand gesture recognition. NEURAL COMPUT APPL 28(12): 3941-3951. https://doi.org/10.1007/s00521-016-2294-8\u003c/li\u003e\n\u003cli\u003eMolina J, Pajuelo JA, Mart\u0026iacute;nez JM (2017) Real-time motion-based hand gestures recognition from time-of-flight video. J SIGNAL PROCESS SYS 86:17-25. https://doi.org/10.1007/s11265-015-1090-5\u003c/li\u003e\n\u003cli\u003eRoh MC, Christmas B, Kittler J, Lee SW (2008) Gesture spotting for low-resolution sports video annotation. PATTERN RECOGN 41(3):1124-1137. https://doi.org/10.1016/j.patcog.2007.07.013\u003c/li\u003e\n\u003cli\u003ePisharady PK, Saerbeck M (2015) Recent methods and databases in vision-based hand gesture recognition: A review. COMPUT VIS IMAGE UND 141: 152-165. https://doi.org/10.1016/j.cviu.2015.08.004\u003c/li\u003e\n\u003cli\u003eVishwakarma DK, Grover V (2017) Hand gesture recognition in low-intensity environment using depth images. International Conference on Intelligent Sustainable Systems (ICISS) pp 429-433. https://doi.org/10.1109/ISS1.2017.8389446\u003c/li\u003e\n\u003cli\u003eVasu PK, Gabriel J, Zhu J, Tuzel O, Ranjan A (2023) FastViT: A fast hybrid vision transformer using structural reparameterization. In the Proceedings of the IEEE/CVF International Conference on Computer Vision pp 5785-5795.\u003c/li\u003e\n\u003cli\u003eSandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp 4510-4520.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"the-journal-of-supercomputing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [The Journal of Supercomputing](https://www.springer.com/journal/11227)","snPcode":"11227","submissionUrl":"https://submission.nature.com/new-submission/11227/3","title":"The Journal of Supercomputing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Thermal imaging, Gesture recognition, Convolutional neural networks, Embedded systems","lastPublishedDoi":"10.21203/rs.3.rs-4689884/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4689884/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eGesture recognition technology has demonstrated its applicability in various scenarios within the field of human-computer interaction, owing to its flexibility and non-contact nature. In particular, thermal imaging technology which is not limited by lighting conditions and effectively reduces the risk of privacy disclosure by capture only thermal radiation rather than detailed features. In this study, a low resolution 32 x 24 pixels end-to-end embedded infrared thermal imaging camera gesture recognition system is developed. A thermal imaging gesture dataset of 4,500 images is constructed to train and evaluate the system. The effects on improve gesture recognition accuracy by research the spatial transform network attention mechanism (STN) is discussed. Thus, a new method combines lightweight Convolutional Neural Networks (CNN) and STN is propose. Additionally, the proposed method achieves 98.5% recognition accuracy and inference time of only 59 milliseconds per frame on embedded devices when tested on self-made infrared thermal imaging sign language gesture dataset, outperforming mainstream lightweight models.\u003c/p\u003e","manuscriptTitle":"Static Gesture Recognition Based on Thermal Imaging Sensors","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-10 11:32:53","doi":"10.21203/rs.3.rs-4689884/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-09-17T14:50:56+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-09-08T17:53:37+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-09-06T11:43:44+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-09-06T04:30:37+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-08-28T15:15:21+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"99668811610604516164316514661725094453","date":"2024-08-26T08:30:58+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"307607008941326966354632877038486971927","date":"2024-08-19T07:33:58+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"90605758005417404628950244901165162321","date":"2024-08-19T04:24:44+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"287985648642758568412999383675261919598","date":"2024-08-19T04:17:14+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"255985732854354131955475161152867156202","date":"2024-08-19T02:44:21+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"66819047664342645623492161895466166019","date":"2024-08-18T03:03:19+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"52228733930523048160891269210483290704","date":"2024-08-17T05:16:27+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-08-17T04:14:40+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-07-09T18:51:15+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-07-09T18:16:17+00:00","index":"","fulltext":""},{"type":"submitted","content":"The Journal of Supercomputing","date":"2024-07-05T05:50:29+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"the-journal-of-supercomputing","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [The Journal of Supercomputing](https://www.springer.com/journal/11227)","snPcode":"11227","submissionUrl":"https://submission.nature.com/new-submission/11227/3","title":"The Journal of Supercomputing","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"dc179fb2-0bab-47ae-8552-fd67c81c7484","owner":[],"postedDate":"August 10th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2025-03-01T12:53:33+00:00","versionOfRecord":[],"versionCreatedAt":"2024-08-10 11:32:53","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4689884","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4689884","identity":"rs-4689884","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}