Infant gesture detection algorithm Based on StarNet Multi-channel Cartesian product network

Infant gesture detection algorithm Based on StarNet Multi-channel Cartesian product network | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Infant gesture detection algorithm Based on StarNet Multi-channel Cartesian product network Xiong Zou, zikang ling, hanyu Zhang, Yutong Chen, Junming Zhang, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6217770/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Infants communicate with others through nonverbal gestures in their early years, expressing their needs and emotions through gestural signs. But it is very challenging for adults to understand babies who have no language skills. In order to better understand these infant gestures, the author first constructed a dataset (babyhand_gesture), which contains four gestures, such as: "happy", "nervous", "sleep" and "sleepy". On this basis, this paper proposes the S-MCCP (Based on StarNet Multi-channel Cartesian product) network framework for infant gesture detection. This method not only utilizes the model expression capability of the high-dimensional feature space of the StarNet structure, but also utilizes the Cartesian product operation to enhance the number of feature images, thereby improving the model accuracy and generalization ability. Experimental results show that the algorithm successfully achieved a recognition accuracy of 96.4% on the dataset, which is nearly one percentage point higher than other algorithms. When tested on three other datasets, the proposed algorithm also had the highest accuracy. Physical sciences/Mathematics and computing/Computer science Physical sciences/Mathematics and computing/Information technology Infant gestures StarNet Convolutional neural networks Computer vision Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Introduction Communication is essential for humans, especially interactions between infants and adults [ 1 , 2 ] . There are two forms of human communication: verbal and nonverbal. Verbal communication involves talking to others; however, nonverbal communication encompasses facial expressions, gestures, posture, and hand movements. It is well known that infants begin to interact with their parents and caregivers through verbal expression from an early age. For example, crying is the main way infants communicate [ 3 , 4 ] , and all infants use different cries and sounds to express their needs and discomfort [ 5 ] . Crying can effectively elicit various responses from caregivers, quickly conveying that the baby wants to express. However, another type of nonverbal communication is not easily understood. According to Melissa J. Cesafsky [ 6 ] , baby sign language is a way for you and your baby (or toddler) to communicate through the use of specific signs, gestures, and movements [ 7 ] . A recent study conducted by Professor Gwyneth Doherty-Sneddon from the University of Stirling in the UK confirmed [ 8 ] that baby sign language not only enhances children’s vocabulary and intellectual development but also reduces tantrums and improves relationships with parents, especially since communication is fundamental to a child’s growth. Parents use gestures to enhance communication with their infants, who are typically aged between 6 months and 6–8 years. Connecting with infants reduces parental frustration, accelerates infant learning [ 9 , 10 ] , enhances the parent-infant relationship, and enable infants to convey crucial information, like when they are hurt or hungry. Besides the psychological benefits, infants who learn sign language tend to gain more certainty and confidence [ 11 ] . For example, if they are trained in sign language, the sense of frustration resulting from communication difficulties is likely to be reduced. In addition, for a child who is too excited to express themselves clearly, the ability to use sign language may be of great help. Research status Although deep learning has made some progress in the field of infant gesture recognition in recent years, there are still significant technical bottlenecks and limitations. For example, the CNN-LSTM hybrid model proposed by Sulochana Nadgeri et al. [ 12 ] can capture the spatiotemporal characteristics of dynamic gestures, but it relies on large-scale labeled data (tens of thousands of video frames need to be manually labeled) and has 120 million model parameters, which results in high training costs (requiring 4 V100 GPUs to run for 48 hours) and is difficult to deploy in ordinary hardware environments. In addition, although the transfer learning method of MobileNet V1 [ 13 ] verified the optimizer performance in the classification of 311 gestures, its generalization ability across age groups is insufficient: when the test set contains data of 6–12 month old infants who did not participate in training, the accuracy rate decreases, exposing the sensitivity of feature extraction to changes in limb proportions. Existing technologies also face practical obstacles in multimodal fusion. Although the RGB-D image fusion algorithm proposed by Duan H’s team [ 14 ] can improve recognition accuracy, it relies on the collaboration of depth sensors and high-resolution cameras and is not robust enough to lighting changes in ordinary home environments, such as low-light conditions at night. While the armband [ 15 ] developed by Mongardi’s team can achieve camera-free recognition, its electrodes need to be in close contact with the baby’s skin, which can easily cause skin sensitivity in infants and young children, making it challenging to meet the demands of round-the-clock monitoring. The three major trends in current research still face challenges: lightweight models, such as the improved Inception-v3 [ 16 ] reduce the number of parameters through channel compression, but their channel pruning strategy leads to the loss of gesture edge features, affecting the accuracy of distinguishing small movements (such as finger bending); although GAN-based data augmentation technology [ 17 ] can generate synthetic images, the generated gestures often have non-physiological joint distortions, which are difficult to be visually verified by pediatricians; the DTW algorithm [ 18 ] in the field of unsupervised learning does not require a predefined gesture library, but cannot distinguish between semantically similar "grasping" and "patting" gestures. These limitations highlight the shortcomings of existing methods in infant gesture recognition scenarios, and provide an improvement direction for the Multi-channel Cartesian product network model proposed in this study: through nonlinear expansion and lightweight design of the feature space, the representation ability of tiny features is enhanced while reducing hardware dependence. In the current research work, the authors conducted a literature review of various existing sign languages. Although there are numerous recognition systems for sign languages used by the deaf or hearing-impaired [ 19 , 20 ] , there is still a lack of recognition systems for early infant gestures. To fill this gap, the authors in the current study explored the recognition of early infant gestures through a literature review. Image recognition and classification technologies have matured significantly, but there are still some areas for improvement. For example, the traditional convolutional network obtains fewer features. In order to increase the feature image and improve the generalization ability of the model, Xu Ma et al. proposed the StarNet structure [ 21 ] , which maps the input to an extremely high-dimensional nonlinear feature space, thereby enhancing the model's expressiveness. However, the model can also obtain more feature images, that is, the dual-branch Cartesian product network model proposed in this paper, which obtains a much higher number of feature images than the former by performing Cartesian product operation on the dual-branch feature image after input convolution extraction, thereby improving the model's expressiveness. After realizing that there was no dataset available for infant gestures, the authors created a static dataset for four categories of early infant gestures, all images were resized to a uniform resolution of 256 * 256 pixels. They were then fed into a dual-branch Cartesian product network model for training. Unlike the traditional convolutional network that extracts single-channel feature images and fixedly multiplies the StarNet feature images, we perform a Cartesian product operation on the extracted dual-branch feature images to obtain a much higher number of feature images than the former, thereby improving the model's accuracy and generalization ability. Through comparative experiments, it is verified that the proposed Multi-channel Cartesian product (S-MCCP) network model performs better than the traditional model in the task of infant gesture recognition. Therefore, a infant gesture recognition system is developed for adults to understand the meaning of infant gestures, help adults better understand infants' nonverbal communication, promote interaction between infants and adults, and improve infants' emotional expression and communication abilities. Dataset 2.1 Dataset Creation Since there is no relevant dataset in the existing public datasets that fully matches the objectives of this study, we chose to create our own dataset "babyhand_gesture" [ 22 ] . Infants' gestures are primary characterized by the shape of their fingers, which convey a variety of feelings and needs.. They are different from common letter or number sign language symbols and are relatively simple. By collecting open source images on the Internet, we have curated a dataset comprising four categories of gesture images, namely "happy", "nervous", "sleep" and "sleepy". Specific gestures include: opening the hands with fingers extended forward to indicate an invitation to play [ 23 , 24 ] ; clenching the fist to indicate nervousness or fear [ 25 ] ; loosely clenching the fist to indicate sleep [ 26 ] ; and slightly opening the hands with fingers bent and soft to indicate a desire to sleep [ 27 ] . They were preprocessed to meet the needs of the subsequent deep learning model, as shown in Fig. 1 . 2.2 Dataset Preprocessing Data preprocessing is an integral part of the workflow, as it directly affects the performance and generalization of the model. For the infant gesture recognition task, we implemented the following preprocessing steps: Unified image size: Since the images obtained through the network have inconsistent dimensions, in order to reduce computational complexity and memory consumption, we resize all images to a fixed size of 256 * 256 pixels. This process not only standardizes the data, but also provides convenience for subsequent model training. Data enhancement: Relying solely on raw image data is insufficient for ensuring the generalization ability of the model. To this end, we have performed various data enhancements, including rotation, flipping, and saturation adjustment. These enhancements simulate gesture changes under different conditions and help the model learn more robust feature expressions during training. Enhanced details: 13 different variants are generated for each image, including different axis, angle and color changes. This extensive augmentation significantly expand the data size while ensuring that the enhanced samples cover a variety of poses and environmental variations. Data Split: The augmented dataset was divided into training and testing sets in proportion, with the training set accounting for 80% and the testing for 20%. This split ensures that the model thoroughly trained and its generalization performance is evaluated during the testing phase. Pixel Normalization and One-hot Encoding: To improve the training efficiency of the model, we statistically normalize the image pixel values. The normalization operation not only accelerates model convergence, but also optimizes memory usage. At the same time, in order to adapt to the classification task, the category label is converted to the One-hot encoding format. This processing method eliminates the order correlation between categories and improve the performance of the classification task. Through the above data preprocessing methods, we provide high-quality input data for the deep learning model, enhance the generalization ability and performance of the model, and lay a solid foundation for subsequent training and testing. The detailed information of the dataset is shown in Table 1 . Table 1 Sample size Gesture meaning Number of original gesture types Number of images after rotation Number of images after color change Total number of RGB images happy 59 413 1099 3913 nervous 78 546 1610 sheep 35 245 805 sheepy 21 147 399 Method 3.1 Improved Multi-channel Cartesian product network structure Traditional convolutional neural networks [ 28 ] only have a single-channel feature image and cannot obtain more features, as shown in Fig. 2 . The StarNet structure [ 21 ] , as shown in Fig. 3 , performs a dual-branch convolution, following the traditional convolution layer. By performing element-wise multiplication between the feature channel maps of the activated Branch 1 (Conv1) and Branch 2 (Conv2), the model can capture more complex feature interactions, thereby enhancing the model's representation ability. However, the number of channels has not changed, and the model cannot obtain more complex features. Different from the dual-branch multiplication of StarNet, the improved algorithm based on StarNet proposed in this paper performs Cartesian product on the feature channels of the dual-branch feature map after activation. Subsequently, redundant features are eliminated through dual-branch average pooling and maximum pooling, which greatly increases the number of feature channels and the complexity of features, and does not increase the model parameters like the fully connected network, resulting in increased training costs. The formula for calculating the number of characteristic channels of the StarNet structure is as follows: StarNet_channels = x1 \(\:\text{⊙}\) x2 = x1 × x2 (1) Among them, x1 and x2 represent the feature channel graph after dual-branch convolution, "⊙" represents the calculation of the number of feature channels of the StarNet structure, and StarNet_channels is the number of feature channels of the network structure. The number of feature channels will not increase after calculation. The number of feature channels of the improved algorithm is calculated using the Cartesian product formula: Mutily_channels = x1 \(\:\otimes\:\) x2 = {(a,b)|a∈x1 and b∈x2} (2) Mutily_channels is the number of feature channels. The number of feature channels is the product of the number of channels in the two branches, which increases in a square, as shown in Fig. 4 . Working procedure The model implementation process includes five stages, as shown in Fig. 5 , from left to right: first, the Conv2D convolution operation is performed on the input data set to generate a feature map to achieve local feature extraction and spatial dimension compression; then, the feature maps x1 and x2 are generated respectively through the dual-branch convolution structure to enhance the model's feature expression ability; then the Cartesian product operation is performed on the dual-branch features, combining the feature vector elements of each branch in pairs to generate a multi-channel feature map. For instance, two branches with M and N eigenvalues can form M×N combinations. This feature crossover method significantly improves the model's ability to capture complex feature relationships by explicitly constructing new feature dimensions; Subsequently, the multi-channel feature map is reduced in dimension through the parallel operations of mean pooling and maximum pooling. Mean pooling suppresses the neighborhood error variance to retain background information, while maximum pooling eliminates parameter error offset to enhance texture features. The synergistic effect of these operations effectively removes redundant information and retains core features. Finally, the pooled feature map is flattened into a one-dimensional vector, and high-level feature combination and classification decisions are realized through the fully connected layer. The flattening operation reshapes the multi-dimensional features into a single-dimensional vector, and the fully connected layer completes the nonlinear mapping via the weight matrix, and finally outputs the four-category probability distribution. 3.2 S-MCCP network structure construction Based on the model flow, we constructed the S-MCCP network, as shown in Fig. 6, from left to right: 3 layers of DepthwiseConv2D convolution layer, 64 layers of Conv2D convolution layer, 32 layers of Conv2D two-branch convolution, 1024 layers of Cartesian product, 32 layers of DepthwiseConv2D, 1 layer of mean pooling and 1 layer of maximum pooling of dual channels, followed by 2 layers of Concat splicing layer, and finally 32 layers of pooling layer and flattening layer. The following is a detailed description of the process. In the S-MCCP model in this article, a 128 * 128 DepthwiseConv2D is first performed to reduce the number of parameters, and then a 64-layer 128 * 128 Conv2D convolution is performed to obtain a feature map. Then, a 32-layer 128 * 128 Conv2D two-branch convolution is performed on the feature map, and then a Cartesian product is performed on the convolved two-branch feature channel map. The detailed steps for performing Cartesian product are to split the 32-layer feature channel maps x1 and x2 into 32 128 * 128 * 1 feature maps respectively, then calculate from x1_channel1 * x2_channel1 to x1_channel32 * x2_channel32, a total of 1024 layers of feature maps, and then splice them into a feature map of 128 * 128 * 1024 shape. Then, DepthwiseConv2D is performed again to reduce the number of parameters, and then 128 * 128 * 1 mean pooling and 128 * 128 * 1 maximum pooling are performed to remove redundant information. The pooling layer is then concatenated into a 128 * 128 * 2 feature map. Finally, it is flattened into a one-dimensional vector and connected to the fully connected layer, which outputs four categories. The model greatly increases the number of feature channels, thereby improving the generalization ability of the model. Detailed model structure information is shown in Table 2 , and the specific schematic diagram of the Multi-channel Cartesian product is shown in Fig. 7 . Table 2 Detailed structure diagram of the model Layer (type) Output Shape Param # Connected to Input_1 (InputLayer) [(None, 128, 128, 3)] 0 [ ] depthwise_conv2d (DepthwiseConv2D) (None, 128, 128, 3) 30 ['input_1[0][0]'] conv2d (Conv2D) (None, 128, 128, 64) 256 ['depthwise_conv2d[0][0]'] conv2d_1 (Conv2D) (None, 128, 128, 32) 18464 ['conv2d[0][0]'] conv2d_2 (Conv2D) (None, 128, 128, 32) 18464 ['conv2d[0][0]'] tf.split (split x1) [(None, 128, 128, 1), 0 ['conv2d_1[0][0]'] (None, 128, 128, 1), ['conv2d_1[0][1]'] \(\:\times\:32\) \(\:\times\:32\) (None, 128, 128, 1), ['conv2d_1[0][30]'] (None, 128, 128, 1)] ['conv2d_1[0][31]'] tf.split_1 (split x2) [(None, 128, 128, 1), 0 ['conv2d_2[0][0]'] (None, 128, 128, 1), ['conv2d_2[0][1]'] \(\:\times\:32\) \(\:\times\:32\) (None, 128, 128, 1), ['conv2d_2[0][30]'] (None, 128, 128, 1)] ['conv2d_2[0][31]'] tf.math.multiply (x1channel1*x2channel1) (None, 128, 128, 1) 0 ['tf.split[0][0]','tf.split_1[0][0]'] tf.math.multiply_1 (x1channel1*x2channel2) (None, 128, 128, 1) 0 ['tf.split[0][0]’,'tf.split_1[0][1]'] \(\:\times\:1024\) \(\:\times\:1024\) tf.math.multiply_1022 (x1channel32*x2channel31) (None, 128, 128, 1) 0 ['tf.split[0][31]','tf.split_1[0][30]'] tf.math.multiply_1023 (x1channel32*x2channel32) (None, 128, 128, 1) 0 ['tf.split[0][31]','tf.split_1[0][31]’] tf.concat (concat all channels) (None, 128, 128, 1024) 0 ['tf.math.multiply[0][0]’, depthwise_conv2d_1 (DepthwiseConv2D) (None, 128, 128, 1024) 10240 ['tf.concat[0][0]'] conv2d_3 (Conv2D) (None, 128, 128, 32) 32800 ['depthwise_conv2d_1[0][0]'] lambda (avg_pool) (None, 128, 128, 1) 0 ['conv2d_3[0][0]'] lambda_1 (max_pool) (None, 128, 128, 1) 0 ['conv2d_3[0][0]'] concatenate (Concatenate) (None, 128, 128, 2) 0 ['lambda[0][0]','lambda_1[0][0]'] conv2d_4 (Conv2D) (None, 128, 128, 1) 99 ['concatenate[0][0]'] multiply (Multiply) (None, 128, 128, 32) 0 ['conv2d_3[0][0]','conv2d_4[0][0]'] max_pooling2d (MaxPooling2) (None, 64, 64, 32) 0 ['multiply[0][0]'] flatten (Flatten) (None, 131072) 0 ['max_pooling2d[0][0]'] dense (Dense) (None, 64) 8388672 ['flatten[0][0]'] dense_1 (Dense) (None, 64) 4160 ['dense[0][0]'] dense_2 (Dense) (None, 32) 2080 ['dense_1[0][0]'] dense_3 (Dense) (None, 4) 132 ['dense_2[0][0]'] Total params: 8475397 (32.33 MB) Trainable params: 8475397 (32.33 MB) Non-trainable params: 0 (0.00 Byte) Experiment The model is mainly divided into two stages. The first stage is data set preprocessing, and the second stage includes model construction and training. In order to facilitate the subsequent model training, we carried out simple preprocessing on the dataset and fed it into the S-MCCP model. The model first extracts feature maps from gesture images through depthwise separable convolution, and then produces two branches x1 and x2 through Multi-channel Cartesian product. The feature channels x1 and x2 are split into single 128 * 128 * 1 feature channels through split, and each separate 128 * 128 * 1 feature channel of x1 and x2 is calculated through Cartesian product, and concatenated to obtain a feature map with a shape of 128 * 128 * 1024. Then depthwise_conv and conv are performed on it, and then it is flattened into a one-dimensional vector by Flatten, and then output through 64, 64, and 32 fully connected layers to form 4 categories. The dataset is divided into three subsets: 3131 images are used as training set, 800 images are used as validation set, and 782 images are used as test set. The same segmentation ratio is used in each category to ensure the rationality of the training results. The model training loss uses the cross entropy loss function to calculate the loss, uses the RMSprop optimizer, and sets the learning rate to x: \(\:{10}^{-3}\ast\:{0.99}^{x}\) to dynamically decrease. Five different random seeds of 8, 9, 10, 11, and 12 are used for training, and the average of the test results is taken as the training result to reduce the experimental error. The model training uses the early stopping strategy and trains for 50 epochs. The performance of the model on the test set under different random seeds is shown in Table 3 . Table 3 Loss and accuracy under different seeds 8 9 10 11 12 Mean Training accuracy 0.9895 0.9930 0.9911 0.9907 0.9946 0.991780 Verification accuracy 0.9706 0.9642 0.9565 0.9565 0.9744 0.964440 Training loss 0.0293 0.0271 0.0210 0.0219 0.0187 0.023600 Validation loss 0.2944 0.2719 0.4140 0.2510 0.1514 0.276540 F1-score 0.970529 0.964143 0.956300 0.956408 0.974366 0.964349 In order to perform a comprehensive evaluation of the model, we calculated the precision, recall, and F1-score of the model on the test set. To display the prediction results on the test set more intuitively and to facilitate the computation of these assessment metrics, we also generated the confusion matrix.The confusion matrix comprises four key metrics: TP (True Positives), TN (True Negatives), FP (False Positives), and FN (False Negatives). TP represents correctly identifying positive samples as positive; TN indicates correctly classifying negative samples as negative; FP refers to incorrectly labeling negative samples as positive; and FN denotes mistakenly classifying positive samples as negative. The confusion matrix is shown in Fig. 8 and the three assessment metrics are shown in Table 4 . The accuracy and loss change curves of the model during training are shown in Fig. 9 and the mathematical formulas for calculating the three assessment metrics are shown below: $$\:\text{P}\text{r}\text{e}\text{c}\text{i}\text{s}\text{i}\text{o}\text{n}=\frac{\text{T}\text{P}}{\left(\text{T}\text{P}+\text{F}\text{P}\right)}$$ 3 $$\:\text{R}\text{e}\text{c}\text{a}\text{l}\text{l}=\frac{\text{T}\text{P}}{\left(\text{T}\text{P}+\text{F}\text{N}\right)}$$ 4 The F1-score, which is the reconciled mean of precision and recall, is calculated as follows: $$\:\text{F}1=\frac{2}{\frac{1}{\text{P}}+\frac{1}{\text{R}}}=\frac{2\times\:\text{P}\times\:\text{R}}{\text{P}+\text{R}}$$ 5 Table 4 Three assessment metrics Category Precision Recall F1-score Actual number of categories happy 0.978261 0.991189 0.984683 227 nervous 0.987988 0.967647 0.977712 340 sheep 0.936709 0.961039 0.948718 154 sheepy 0.950820 0.950820 0.950820 31 Micro average 0.971867 0.971867 0.971867 - Macro average 0.963444 0.967674 0.965554 - Weighted average 0.972167 0.971867 0.972017 - Results and Discussion For infant gesture recognition, a variety of methods can be used [21,28,29,30]. For example, the traditional convolutional neural networks [28] are limited to single-channel feature image and cannot capture more features. The StarNet dual-branch architecture can only obtain the same number of feature images through fixed multiplication. The improved Cartesian product of the x1 and x2 branches proposed in this study can greatly increase the number of feature images, thereby improving the model accuracy. In order to further verify the improvement of the generalization ability of the Multi-channel feature channel Cartesian product compared with the previous network structure. A comparative experiment was designed to compare the traditional convolutional network structure, the StarNet structure, and the StarNet Multi-channel Cartesian product network structure. The same dataset, five random seeds 8, 9, 10, 11, 12, learning rate, optimizer, etc. were used. The experimental conditions were exactly the same except for the model network structure. The final average test accuracy is shown in Table 5. It can be clearly seen from the table that our model has a better performance on the test set, which further verifies that the Multi-channel Cartesian product can improve the model capability. Table 5 shows that our model can obtain better generalization ability compared with similar static datasets. Some models have different reference standards for accuracy due to different datasets, but overall the S-MCCP model still has some improvements. Table 5. Some past research results on static gesture recognition Methodology Dataset Accuracy Centroid and area of edge+Euclidean distance [26] Real-time recognition of 26 ASL alphabets. 90.19% Networked human motion capture+ DNN [27] 26 ASL alphabets. 98.12% Histogram of Oriented Gradient (HOG)+ K-NN [28] 26 ASL alphabets. 94.23% ORB K-means clustering Bag of words (BoW)+ K-NN [29] ASL Finger Spelling Dataset 95.81% S-MCCP Network Model（Ours） Real-time recognition of 26 ASL alphabets. 94.84% 26 ASL alphabets. 95.16% ASL Finger Spelling Dataset 96.31% Self-built dataset 96.44% By observing Table 6, we can see that this method has good evaluation results under different indicators. Compared with the traditional convolutional network structure [23,24,25], the improved StarNet structure proposed in this paper has a better average test accuracy of 0.964440 on the same dataset. Table 6. Comparison between adding Cartesian product and not adding it Model structure Features Average loss Average accuracy Traditional Conv Single channel convolution 0.3231 0.9540 StarNet Dual-branch fixed multiplication 0.2363 0.9578 S-MCCP Network Model（Ours） Multi-channel Cartesian product 0.276540 0.964440 Conclusion The branch feature channel Cartesian product network model proposed in this study extracts infant gestures from RGB images for recognition. To better understand infant gestures, this paper first established a dataset named "babyhand_gesture". On this basis, based on StarNet Multi-channel Cartesian product is proposed. The model is inspired by the StarNet structure. The previous feature extraction directly uses single-channel convolution, while StarNet uses dual-branch fixed channel multiplication. However, the acquired feature images are limited. The network structure we proposed can significantly increase the number of feature images, thereby enhancing the model's generalization ability. Experimental results show that the recognition accuracy of the algorithm on the dataset is higher than that of other algorithms. In future work, we will start from more perspectives. Multimodal expansion and dynamic gesture analysis will further enhance practicality, and the release of open source resources will promote the collaborative development of academia and industry. Declarations Data availability The dataset used in this article is the author's self-built dataset "babyhand_gesture", the code and dataset have been made public through GitHub. The address is: https://github.com/Joyrides/babyhand_gesture Acknowledgements This work was supported in part by the 2023 Henan Province Graduate Education Reform and Quality Improvement Project (YJS2023AL092), in part by the Henan Provincial Natural Science Foundation under Grant 242300420189, in part by the Postgraduate Joint Training Base Project of Henan Province under grant YJS2022JD45, and in part by the Key Science and Technology Research of Henan Province under grant Nos. 232102211038, 232102210076 and 232102210129. Author Contributions X.Z. and Z.L. write the main manuscript text, H.Z. and Y.C. run the code, J.Z. revise the manuscript, and J.G. review the manuscript. Competing interests The authors declare no competing interests. Additional information Correspondence and requests for materials should be addressed to J.G. References Zhou, Xiaodi. (2023). Language and the mind: How language shapes our thinking. Journal of World Languages. 10.1515/jwl-2023-0018. Altarriba, J., & Basnight-Brown, D. (2022). The Psychology of Communication: The Interplay Between Language and Culture Through Time. Journal of Cross-Cultural Psychology, 53(7-8), 860-874. Lockhart-Bouron, M., Anikin, A., Pisanski, K. et al. Infant cries convey both stable and dynamic information about age and identity. Commun Psychol 1, 26 (2023). Capirci O, Bonsignori C, Di Renzo A. Signed languages: A triangular semiotic dimension[J]. Frontiers in Psychology, 2022, 12: 802911. George A S H, George A S. Decoding Infant Communication: Understanding the Meaning Behind Baby's Cries[J]. Partners Universal Innovative Research Cesafsky M J. Baby sign language: hindering or enhancing communication in infants and toddlers?[J]. 2009. Gurrola A. The Effects of Using Baby Sign Training on the Interactions Between Mothers and Infants[D]. The University of Texas at El Paso, 2023. s Doherty-Sneddon G. The great baby signing debate: Academia meets public interest[J]. The Psychologist, 2008, 21(4): 300-303. Pontecorvo E, Higgins M, Mora J, et al. Learning a sign language does not hinder acquisition of a spoken language[J]. Journal of Speech, Language, and Hearing Research, 2023, 66(4): 1291-1308. Fay N, Walker B, Ellison T M, et al. Gesture is the primary modality for language creation[J]. Proceedings of the Royal Society B, 2022, 289(1970): 20220066. Congdon E L, Goldin-Meadow S. Mechanisms of embodied learning through gestures and actions: Lessons from development[J]. Handbook of embodied psychology: Thinking, feeling, and acting, 2021: 527-546. Nadgeri, Sulochana & Kumar, Arun. (2022). DEEP LEARNING BASED FRAMEWORK FOR DYNAMIC BABY SIGN LANGUAGE RECOGNITION SYSTEM. Indian Journal of Computer Science and Engineering. 13. 550-563. 10.21817/indjcse/2022/v13i2/221302127. Howard A G, Zhu M, Chen B, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications[J]. arXiv preprint arXiv:1704.04861, 2017. Duan H, Sun Y, Cheng W, et al. Gesture recognition based on multi‐modal feature weight[J]. Concurrency and Computation: Practice and Experience, 2021, 33(5): e5991. Mongardi A, Rossi F, Prestia A, et al. Hand gestures recognition for human-machine interfaces: A low-power bio-inspired armband[J]. IEEE Transactions on Biomedical Circuits and Systems, 2022, 16(6): 1348-1365. Karsh B, Laskar R H, Karsh R K. mIV3Net: modified inception V3 network for hand gesture recognition[J]. Multimedia Tools and Applications, 2024, 83(4): 10587-10613. Chen J, Chen S, Wee L, et al. Deep learning based unpaired image-to-image translation applications for medical physics: a systematic review[J]. Physics in Medicine & Biology, 2023, 68(5): 05TR01. Tchane Djogdom G V, Otis M J D, Meziane R. Dynamic time warping–based feature selection method for foot gesture cobot operation mode selection[J]. The International Journal of Advanced Manufacturing Technology, 2023, 126(9): 4521-4541. Adeyanju I A, Bello O O, Adegboye M A. Machine learning methods for sign language recognition: A critical review and analysis[J]. Intelligent Systems with Applications, 2021, 12: 200056. Kasapbaşi A, Elbushra A E A, Omar A L H, et al. DeepASLR: A CNN based human computer interface for American Sign Language recognition for hearing-impaired individuals[J]. Computer methods and programs in biomedicine update, 2022, 2: 100048. Ma X, Dai X, Bai Y, et al. Rewrite the Stars[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 5694-5703. https://github.com/Joyrides/babyhand_gesture Eysenbach G, CONSORT-EHEALTH Group CONSORT-EHEALTH: Improving and Standardizing Evaluation Reports of Web-based and Mobile Health Interventions J Med Internet Res 2011;13(4):e126 https://psychology.fandom.com/wiki/Posture_(psychology) https://www.improvementtower.com/bodylanguage/body_langage_signs_of_nervousness_tension.html https://test.scienceabc.com/humans/why-do-our-fingers-curl-when-sleeping.html Cobb, K., Goodwin, R., & Saelens, E. (1966). Spontaneous Hand Positions of Newborn Infants. The Journal of Genetic Psychology, 108(2), 225–237. LeCun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324. Krizhevsky A , Sutskever I , Hinton G . ImageNet Classification with Deep Convolutional Neural Networks[J]. Advances in neural information processing systems, 2012, 25(2). Simonyan K , Zisserman A . Very Deep Convolutional Networks for Large-Scale Image Recognition[J]. Computer Science, 2014. Pansare, J R, Dhumal, H., Babar, S., and International, K.S. (2015). Real time static hand gesture recognition system in complex background that uses number system of indian sign language static hand gesture recognition for sign language alphabets using edge oriented histogram and multi class SVM real ­ Time hand gesture R. 2(3), 2014–2015. J. Li, Z.-L. Wang, H. Zhao, R. Gravina, G. Fortino, Y. Jiang, et al., "Networked human motion capture system based on quaternion navigation", Proc. 11th EAI Int. Conf. Body Area Netw., vol. 5, pp. 38-44, 2016. I. Mahmud, T. Tabassum, M.P. Uddin, E. Ali, A.M. Nitu, M.I. Afjal Efficient noise reduction and HOG feature extraction for sign language recognition Proceedings of the international conference on advancement in electrical and electronic engineering, ICAEEE 2018 (2019), pp. A. Sharma, A. Mittal, S. Singh, V Awatramani Hand gesture recognition using image processing and feature extraction techniques Procedia Computer Science, 173 (2019) (2020), pp. 181-190,10.1016/j.procs.2020.06.022. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6217770","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":442870762,"identity":"cab0ac6f-5712-4df7-84c2-d9f8ac03a326","order_by":0,"name":"Xiong Zou","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABEklEQVRIiWNgGAWjYLCCBCDmlwAzDzAwMAMpHmK0SM4gSQsIGNyAaWEgoEW+vcdM4kHFHbvNt3sMPxf8upPY387A+OBtG4O8OQ4tjD1njA0SzjxL3nbnjLH0zL5niTMOMzAbzm1jMNzZgF0Ls0SO4YPEtsPJZjdyDKR5ew4nNhxmYJPmbWNIMDiAXQubRI7BAZAW4xk5xr9BWuYfZmD/jU8LD9QWOwOJHDNpnh+HEzcAbWHGp0WC51gx0C+HEyRupJVZ8zYcNt54mLFZcs45CcMNOLTItzdvk/xRcdief0by5ts8fw7Lzjt/+OCHN2U28rhsgYHEBhDJ2AYmQWwJ/OqBwB5C/SGocBSMglEwCkYgAAAsXF/KYfp4dgAAAABJRU5ErkJggg==","orcid":"","institution":"Huanghuai University","correspondingAuthor":true,"prefix":"","firstName":"Xiong","middleName":"","lastName":"Zou","suffix":""},{"id":442870763,"identity":"188419d9-6ee0-4410-afbd-b4d8abe91656","order_by":1,"name":"zikang ling","email":"","orcid":"","institution":"Huanghuai University","correspondingAuthor":false,"prefix":"","firstName":"zikang","middleName":"","lastName":"ling","suffix":""},{"id":442870764,"identity":"c1b6339b-f7f2-480f-ab32-fdf1aff80408","order_by":2,"name":"hanyu Zhang","email":"","orcid":"","institution":"Huanghuai University","correspondingAuthor":false,"prefix":"","firstName":"hanyu","middleName":"","lastName":"Zhang","suffix":""},{"id":442870765,"identity":"d8281868-76cf-49a2-94b0-502c5e38ecf2","order_by":3,"name":"Yutong Chen","email":"","orcid":"","institution":"Huanghuai University","correspondingAuthor":false,"prefix":"","firstName":"Yutong","middleName":"","lastName":"Chen","suffix":""},{"id":442870766,"identity":"49bfcdae-1d2f-48aa-b165-1f8dc552838e","order_by":4,"name":"Junming Zhang","email":"","orcid":"","institution":"Huanghuai University","correspondingAuthor":false,"prefix":"","firstName":"Junming","middleName":"","lastName":"Zhang","suffix":""},{"id":442870767,"identity":"cb883f8e-60cd-4dd9-bfd0-393ad03fd73c","order_by":5,"name":"Jinfeng Gao","email":"","orcid":"","institution":"Huanghuai University","correspondingAuthor":false,"prefix":"","firstName":"Jinfeng","middleName":"","lastName":"Gao","suffix":""}],"badges":[],"createdAt":"2025-03-13 08:08:23","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6217770/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6217770/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":81022635,"identity":"1ffea434-962c-40d3-b1d8-40a2372fd0d8","added_by":"auto","created_at":"2025-04-21 10:02:10","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":82830,"visible":true,"origin":"","legend":"\u003cp\u003eSample data\u003c/p\u003e","description":"","filename":"1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6217770/v1/9795bee8cd09cd3486e1f1a8.jpg"},{"id":81021982,"identity":"a4ce90f1-73a1-4d50-9f18-534e60c0af1d","added_by":"auto","created_at":"2025-04-21 09:54:10","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":12232,"visible":true,"origin":"","legend":"\u003cp\u003eTraditional convolutional network structure\u003c/p\u003e","description":"","filename":"2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6217770/v1/70f644df32ec6d22be2d849d.jpg"},{"id":81022636,"identity":"1df1ab65-4e52-4a3c-8be2-6587f35b0dc3","added_by":"auto","created_at":"2025-04-21 10:02:10","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":41071,"visible":true,"origin":"","legend":"\u003cp\u003eStarNet structure\u003c/p\u003e","description":"","filename":"3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6217770/v1/647dfb6c63c387cd85dbed85.jpg"},{"id":81021986,"identity":"098738c9-2989-486a-873e-ba9f45720586","added_by":"auto","created_at":"2025-04-21 09:54:10","extension":"jpg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":45378,"visible":true,"origin":"","legend":"\u003cp\u003eBased on StarNet Multi-channel Cartesian product network structure (S-MCCP)\u003c/p\u003e","description":"","filename":"4.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6217770/v1/c0adcb81acc883da44aea30d.jpg"},{"id":81021990,"identity":"a141abf1-a423-429c-87b0-9f8e224d9c2c","added_by":"auto","created_at":"2025-04-21 09:54:10","extension":"jpg","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":23252,"visible":true,"origin":"","legend":"\u003cp\u003eModel implementation flow chart\u003c/p\u003e","description":"","filename":"5.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6217770/v1/90bab1a7d37f5ae406576c3d.jpg"},{"id":81022638,"identity":"6f883b4f-ebfe-4bb1-8d61-41654b8240da","added_by":"auto","created_at":"2025-04-21 10:02:10","extension":"jpg","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":98866,"visible":true,"origin":"","legend":"\u003cp\u003eDetailed structure of the model\u003c/p\u003e","description":"","filename":"7.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6217770/v1/0b2143177a7938c4e4de91d4.jpg"},{"id":81023962,"identity":"210d4a72-0961-40dc-a5e6-8b087f27b288","added_by":"auto","created_at":"2025-04-21 10:10:10","extension":"jpg","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":35908,"visible":true,"origin":"","legend":"\u003cp\u003eSchematic diagram of Multi-channel Cartesian product\u003c/p\u003e","description":"","filename":"8.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6217770/v1/2c785dc981b534dce39e8344.jpg"},{"id":81021988,"identity":"b8fe6172-fadc-412e-a980-349502e91fa0","added_by":"auto","created_at":"2025-04-21 09:54:10","extension":"jpg","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":26751,"visible":true,"origin":"","legend":"\u003cp\u003eModel training confusion matrix\u003c/p\u003e","description":"","filename":"8a.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6217770/v1/5ccaf0f6f38e393e3ad6d5ea.jpg"},{"id":81021994,"identity":"03955f0c-f235-4e9e-9094-ca2e54ba6c31","added_by":"auto","created_at":"2025-04-21 09:54:10","extension":"jpg","order_by":9,"title":"Figure 9","display":"","copyAsset":false,"role":"figure","size":40242,"visible":true,"origin":"","legend":"\u003cp\u003eModel training accuracy and loss curve\u003c/p\u003e","description":"","filename":"9.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6217770/v1/89df4b2a72c2e9aeada7ad64.jpg"},{"id":82777273,"identity":"90dc3d2d-7bbe-42c4-9455-7efbef993469","added_by":"auto","created_at":"2025-05-15 07:32:10","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1217741,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6217770/v1/9a5ea93f-d372-420e-acbf-6e117b431538.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Infant gesture detection algorithm Based on StarNet Multi-channel Cartesian product network","fulltext":[{"header":"Introduction","content":"\u003cp\u003eCommunication is essential for humans, especially interactions between infants and adults\u003csup\u003e[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]\u003c/sup\u003e. There are two forms of human communication: verbal and nonverbal. Verbal communication involves talking to others; however, nonverbal communication encompasses facial expressions, gestures, posture, and hand movements. It is well known that infants begin to interact with their parents and caregivers through verbal expression from an early age. For example, crying is the main way infants communicate\u003csup\u003e[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]\u003c/sup\u003e, and all infants use different cries and sounds to express their needs and discomfort\u003csup\u003e[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]\u003c/sup\u003e. Crying can effectively elicit various responses from caregivers, quickly conveying that the baby wants to express. However, another type of nonverbal communication is not easily understood.\u003c/p\u003e \u003cp\u003eAccording to Melissa J. Cesafsky\u003csup\u003e[\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]\u003c/sup\u003e, baby sign language is a way for you and your baby (or toddler) to communicate through the use of specific signs, gestures, and movements\u003csup\u003e[\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]\u003c/sup\u003e. A recent study conducted by Professor Gwyneth Doherty-Sneddon from the University of Stirling in the UK confirmed\u003csup\u003e[\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]\u003c/sup\u003e that baby sign language not only enhances children\u0026rsquo;s vocabulary and intellectual development but also reduces tantrums and improves relationships with parents, especially since communication is fundamental to a child\u0026rsquo;s growth. Parents use gestures to enhance communication with their infants, who are typically aged between 6 months and 6\u0026ndash;8 years. Connecting with infants reduces parental frustration, accelerates infant learning\u003csup\u003e[\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]\u003c/sup\u003e, enhances the parent-infant relationship, and enable infants to convey crucial information, like when they are hurt or hungry. Besides the psychological benefits, infants who learn sign language tend to gain more certainty and confidence\u003csup\u003e[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]\u003c/sup\u003e. For example, if they are trained in sign language, the sense of frustration resulting from communication difficulties is likely to be reduced. In addition, for a child who is too excited to express themselves clearly, the ability to use sign language may be of great help.\u003c/p\u003e\n\u003ch3\u003eResearch status\u003c/h3\u003e\n\u003cp\u003eAlthough deep learning has made some progress in the field of infant gesture recognition in recent years, there are still significant technical bottlenecks and limitations. For example, the CNN-LSTM hybrid model proposed by Sulochana Nadgeri et al.\u003csup\u003e[\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]\u003c/sup\u003e can capture the spatiotemporal characteristics of dynamic gestures, but it relies on large-scale labeled data (tens of thousands of video frames need to be manually labeled) and has 120\u0026nbsp;million model parameters, which results in high training costs (requiring 4 V100 GPUs to run for 48 hours) and is difficult to deploy in ordinary hardware environments. In addition, although the transfer learning method of MobileNet V1\u003csup\u003e[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]\u003c/sup\u003e verified the optimizer performance in the classification of 311 gestures, its generalization ability across age groups is insufficient: when the test set contains data of 6\u0026ndash;12 month old infants who did not participate in training, the accuracy rate decreases, exposing the sensitivity of feature extraction to changes in limb proportions. Existing technologies also face practical obstacles in multimodal fusion. Although the RGB-D image fusion algorithm proposed by Duan H\u0026rsquo;s team\u003csup\u003e[\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]\u003c/sup\u003e can improve recognition accuracy, it relies on the collaboration of depth sensors and high-resolution cameras and is not robust enough to lighting changes in ordinary home environments, such as low-light conditions at night. While the armband\u003csup\u003e[\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]\u003c/sup\u003e developed by Mongardi\u0026rsquo;s team can achieve camera-free recognition, its electrodes need to be in close contact with the baby\u0026rsquo;s skin, which can easily cause skin sensitivity in infants and young children, making it challenging to meet the demands of round-the-clock monitoring.\u003c/p\u003e \u003cp\u003eThe three major trends in current research still face challenges: lightweight models, such as the improved Inception-v3\u003csup\u003e[\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]\u003c/sup\u003e reduce the number of parameters through channel compression, but their channel pruning strategy leads to the loss of gesture edge features, affecting the accuracy of distinguishing small movements (such as finger bending); although GAN-based data augmentation technology\u003csup\u003e[\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]\u003c/sup\u003e can generate synthetic images, the generated gestures often have non-physiological joint distortions, which are difficult to be visually verified by pediatricians; the DTW algorithm\u003csup\u003e[\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]\u003c/sup\u003e in the field of unsupervised learning does not require a predefined gesture library, but cannot distinguish between semantically similar \"grasping\" and \"patting\" gestures. These limitations highlight the shortcomings of existing methods in infant gesture recognition scenarios, and provide an improvement direction for the Multi-channel Cartesian product network model proposed in this study: through nonlinear expansion and lightweight design of the feature space, the representation ability of tiny features is enhanced while reducing hardware dependence.\u003c/p\u003e \u003cp\u003eIn the current research work, the authors conducted a literature review of various existing sign languages. Although there are numerous recognition systems for sign languages used by the deaf or hearing-impaired\u003csup\u003e[\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]\u003c/sup\u003e, there is still a lack of recognition systems for early infant gestures. To fill this gap, the authors in the current study explored the recognition of early infant gestures through a literature review. Image recognition and classification technologies have matured significantly, but there are still some areas for improvement. For example, the traditional convolutional network obtains fewer features. In order to increase the feature image and improve the generalization ability of the model, Xu Ma et al. proposed the StarNet structure\u003csup\u003e[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]\u003c/sup\u003e, which maps the input to an extremely high-dimensional nonlinear feature space, thereby enhancing the model's expressiveness. However, the model can also obtain more feature images, that is, the dual-branch Cartesian product network model proposed in this paper, which obtains a much higher number of feature images than the former by performing Cartesian product operation on the dual-branch feature image after input convolution extraction, thereby improving the model's expressiveness. After realizing that there was no dataset available for infant gestures, the authors created a static dataset for four categories of early infant gestures, all images were resized to a uniform resolution of 256 * 256 pixels. They were then fed into a dual-branch Cartesian product network model for training. Unlike the traditional convolutional network that extracts single-channel feature images and fixedly multiplies the StarNet feature images, we perform a Cartesian product operation on the extracted dual-branch feature images to obtain a much higher number of feature images than the former, thereby improving the model's accuracy and generalization ability. Through comparative experiments, it is verified that the proposed Multi-channel Cartesian product (S-MCCP) network model performs better than the traditional model in the task of infant gesture recognition. Therefore, a infant gesture recognition system is developed for adults to understand the meaning of infant gestures, help adults better understand infants' nonverbal communication, promote interaction between infants and adults, and improve infants' emotional expression and communication abilities.\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eDataset\u003c/h2\u003e \u003cdiv id=\"Sec4\" class=\"Section3\"\u003e \u003ch2\u003e2.1 Dataset Creation\u003c/h2\u003e \u003cp\u003eSince there is no relevant dataset in the existing public datasets that fully matches the objectives of this study, we chose to create our own dataset \"babyhand_gesture\" \u003csup\u003e[\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]\u003c/sup\u003e. Infants' gestures are primary characterized by the shape of their fingers, which convey a variety of feelings and needs.. They are different from common letter or number sign language symbols and are relatively simple. By collecting open source images on the Internet, we have curated a dataset comprising four categories of gesture images, namely \"happy\", \"nervous\", \"sleep\" and \"sleepy\". Specific gestures include: opening the hands with fingers extended forward to indicate an invitation to play\u003csup\u003e[\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e]\u003c/sup\u003e; clenching the fist to indicate nervousness or fear\u003csup\u003e[\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]\u003c/sup\u003e; loosely clenching the fist to indicate sleep\u003csup\u003e[\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]\u003c/sup\u003e; and slightly opening the hands with fingers bent and soft to indicate a desire to sleep\u003csup\u003e[\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]\u003c/sup\u003e. They were preprocessed to meet the needs of the subsequent deep learning model, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003c/div\u003e\n\u003ch3\u003e2.2 Dataset Preprocessing\u003c/h3\u003e\n\u003cp\u003eData preprocessing is an integral part of the workflow, as it directly affects the performance and generalization of the model. For the infant gesture recognition task, we implemented the following preprocessing steps:\u003c/p\u003e \u003cp\u003eUnified image size: Since the images obtained through the network have inconsistent dimensions, in order to reduce computational complexity and memory consumption, we resize all images to a fixed size of 256 * 256 pixels. This process not only standardizes the data, but also provides convenience for subsequent model training.\u003c/p\u003e \u003cp\u003eData enhancement: Relying solely on raw image data is insufficient for ensuring the generalization ability of the model. To this end, we have performed various data enhancements, including rotation, flipping, and saturation adjustment. These enhancements simulate gesture changes under different conditions and help the model learn more robust feature expressions during training.\u003c/p\u003e \u003cp\u003eEnhanced details: 13 different variants are generated for each image, including different axis, angle and color changes. This extensive augmentation significantly expand the data size while ensuring that the enhanced samples cover a variety of poses and environmental variations.\u003c/p\u003e \u003cp\u003eData Split: The augmented dataset was divided into training and testing sets in proportion, with the training set accounting for 80% and the testing for 20%. This split ensures that the model thoroughly trained and its generalization performance is evaluated during the testing phase.\u003c/p\u003e \u003cp\u003ePixel Normalization and One-hot Encoding: To improve the training efficiency of the model, we statistically normalize the image pixel values. The normalization operation not only accelerates model convergence, but also optimizes memory usage. At the same time, in order to adapt to the classification task, the category label is converted to the One-hot encoding format. This processing method eliminates the order correlation between categories and improve the performance of the classification task.\u003c/p\u003e \u003cp\u003eThrough the above data preprocessing methods, we provide high-quality input data for the deep learning model, enhance the generalization ability and performance of the model, and lay a solid foundation for subsequent training and testing. The detailed information of the dataset is shown in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSample size\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGesture meaning\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNumber of original gesture types\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNumber of images after rotation\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eNumber of images after color change\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eTotal number of RGB images\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ehappy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e59\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e413\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1099\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\" morerows=\"3\" rowspan=\"4\"\u003e \u003cp\u003e3913\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003enervous\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e78\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e546\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1610\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003esheep\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e245\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e805\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003esheepy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e147\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e399\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e"},{"header":"Method","content":"\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Improved Multi-channel Cartesian product network structure\u003c/h2\u003e \u003cp\u003eTraditional convolutional neural networks\u003csup\u003e[\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]\u003c/sup\u003e only have a single-channel feature image and cannot obtain more features, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe StarNet structure\u003csup\u003e[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]\u003c/sup\u003e, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, performs a dual-branch convolution, following the traditional convolution layer. By performing element-wise multiplication between the feature channel maps of the activated Branch 1 (Conv1) and Branch 2 (Conv2), the model can capture more complex feature interactions, thereby enhancing the model's representation ability. However, the number of channels has not changed, and the model cannot obtain more complex features.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eDifferent from the dual-branch multiplication of StarNet, the improved algorithm based on StarNet proposed in this paper performs Cartesian product on the feature channels of the dual-branch feature map after activation. Subsequently, redundant features are eliminated through dual-branch average pooling and maximum pooling, which greatly increases the number of feature channels and the complexity of features, and does not increase the model parameters like the fully connected network, resulting in increased training costs.\u003c/p\u003e \u003cp\u003eThe formula for calculating the number of characteristic channels of the StarNet structure is as follows:\u003c/p\u003e \u003cp\u003eStarNet_channels\u0026thinsp;=\u0026thinsp;x1\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\text{⊙}\\)\u003c/span\u003e\u003c/span\u003ex2 = x1 \u0026times; x2 (1)\u003c/p\u003e \u003cp\u003eAmong them, x1 and x2 represent the feature channel graph after dual-branch convolution, \"⊙\" represents the calculation of the number of feature channels of the StarNet structure, and StarNet_channels is the number of feature channels of the network structure. The number of feature channels will not increase after calculation.\u003c/p\u003e \u003cp\u003eThe number of feature channels of the improved algorithm is calculated using the Cartesian product formula:\u003c/p\u003e \u003cp\u003eMutily_channels\u0026thinsp;=\u0026thinsp;x1\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\otimes\\:\\)\u003c/span\u003e\u003c/span\u003ex2 = {(a,b)|a\u0026isin;x1 and b\u0026isin;x2} (2)\u003c/p\u003e \u003cp\u003eMutily_channels is the number of feature channels. The number of feature channels is the product of the number of channels in the two branches, which increases in a square, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eWorking procedure\u003c/h2\u003e \u003cp\u003eThe model implementation process includes five stages, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e, from left to right: first, the Conv2D convolution operation is performed on the input data set to generate a feature map to achieve local feature extraction and spatial dimension compression; then, the feature maps x1 and x2 are generated respectively through the dual-branch convolution structure to enhance the model's feature expression ability; then the Cartesian product operation is performed on the dual-branch features, combining the feature vector elements of each branch in pairs to generate a multi-channel feature map. For instance, two branches with M and N eigenvalues can form M\u0026times;N combinations. This feature crossover method significantly improves the model's ability to capture complex feature relationships by explicitly constructing new feature dimensions; Subsequently, the multi-channel feature map is reduced in dimension through the parallel operations of mean pooling and maximum pooling. Mean pooling suppresses the neighborhood error variance to retain background information, while maximum pooling eliminates parameter error offset to enhance texture features. The synergistic effect of these operations effectively removes redundant information and retains core features. Finally, the pooled feature map is flattened into a one-dimensional vector, and high-level feature combination and classification decisions are realized through the fully connected layer. The flattening operation reshapes the multi-dimensional features into a single-dimensional vector, and the fully connected layer completes the nonlinear mapping via the weight matrix, and finally outputs the four-category probability distribution.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003e3.2 S-MCCP network structure construction\u003c/h3\u003e\n\u003cp\u003eBased on the model flow, we constructed the S-MCCP network, as shown in Fig.\u0026nbsp;6, from left to right: 3 layers of DepthwiseConv2D convolution layer, 64 layers of Conv2D convolution layer, 32 layers of Conv2D two-branch convolution, 1024 layers of Cartesian product, 32 layers of DepthwiseConv2D, 1 layer of mean pooling and 1 layer of maximum pooling of dual channels, followed by 2 layers of Concat splicing layer, and finally 32 layers of pooling layer and flattening layer. The following is a detailed description of the process.\u003c/p\u003e \u003cp\u003eIn the S-MCCP model in this article, a 128 * 128 DepthwiseConv2D is first performed to reduce the number of parameters, and then a 64-layer 128 * 128 Conv2D convolution is performed to obtain a feature map. Then, a 32-layer 128 * 128 Conv2D two-branch convolution is performed on the feature map, and then a Cartesian product is performed on the convolved two-branch feature channel map. The detailed steps for performing Cartesian product are to split the 32-layer feature channel maps x1 and x2 into 32 128 * 128 * 1 feature maps respectively, then calculate from x1_channel1 * x2_channel1 to x1_channel32 * x2_channel32, a total of 1024 layers of feature maps, and then splice them into a feature map of 128 * 128 * 1024 shape. Then, DepthwiseConv2D is performed again to reduce the number of parameters, and then 128 * 128 * 1 mean pooling and 128 * 128 * 1 maximum pooling are performed to remove redundant information. The pooling layer is then concatenated into a 128 * 128 * 2 feature map. Finally, it is flattened into a one-dimensional vector and connected to the fully connected layer, which outputs four categories. The model greatly increases the number of feature channels, thereby improving the generalization ability of the model. Detailed model structure information is shown in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, and the specific schematic diagram of the Multi-channel Cartesian product is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e7\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eDetailed structure diagram of the model\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLayer (type)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOutput Shape\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eParam #\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eConnected to\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInput_1 (InputLayer)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e[(None, 128, 128, 3)]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e[ ]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003edepthwise_conv2d (DepthwiseConv2D)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e30\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['input_1[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003econv2d (Conv2D)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 64)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e256\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['depthwise_conv2d[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003econv2d_1 (Conv2D)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 32)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18464\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['conv2d[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003econv2d_2 (Conv2D)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 32)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18464\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['conv2d[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"4\" rowspan=\"5\"\u003e \u003cp\u003etf.split (split x1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e[(None, 128, 128, 1),\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\" morerows=\"4\" rowspan=\"5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['conv2d_1[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1),\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['conv2d_1[0][1]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\times\\:32\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\times\\:32\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1),\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['conv2d_1[0][30]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1)]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['conv2d_1[0][31]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"4\" rowspan=\"5\"\u003e \u003cp\u003etf.split_1 (split x2)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e[(None, 128, 128, 1),\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\" morerows=\"4\" rowspan=\"5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['conv2d_2[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1),\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['conv2d_2[0][1]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\times\\:32\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\times\\:32\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1),\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['conv2d_2[0][30]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1)]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['conv2d_2[0][31]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003etf.math.multiply (x1channel1*x2channel1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['tf.split[0][0]','tf.split_1[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003etf.math.multiply_1 (x1channel1*x2channel2)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['tf.split[0][0]\u0026rsquo;,'tf.split_1[0][1]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\times\\:1024\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\times\\:1024\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003etf.math.multiply_1022 (x1channel32*x2channel31)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['tf.split[0][31]','tf.split_1[0][30]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003etf.math.multiply_1023 (x1channel32*x2channel32)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['tf.split[0][31]','tf.split_1[0][31]\u0026rsquo;]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003etf.concat (concat all channels)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1024)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['tf.math.multiply[0][0]\u0026rsquo;,\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003edepthwise_conv2d_1 (DepthwiseConv2D)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1024)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e10240\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['tf.concat[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003econv2d_3 (Conv2D)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 32)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e32800\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['depthwise_conv2d_1[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003elambda (avg_pool)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['conv2d_3[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003elambda_1 (max_pool)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['conv2d_3[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003econcatenate (Concatenate)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 2)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['lambda[0][0]','lambda_1[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003econv2d_4 (Conv2D)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e99\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['concatenate[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003emultiply (Multiply)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 128, 128, 32)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['conv2d_3[0][0]','conv2d_4[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003emax_pooling2d (MaxPooling2)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 64, 64, 32)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['multiply[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eflatten (Flatten)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 131072)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['max_pooling2d[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003edense (Dense)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 64)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e8388672\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['flatten[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003edense_1 (Dense)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 64)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e4160\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['dense[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003edense_2 (Dense)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 32)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2080\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['dense_1[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003edense_3 (Dense)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None, 4)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e132\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e['dense_2[0][0]']\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"4\" nameend=\"c4\" namest=\"c1\"\u003e \u003cp\u003eTotal params: 8475397 (32.33 MB)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"4\" nameend=\"c4\" namest=\"c1\"\u003e \u003cp\u003eTrainable params: 8475397 (32.33 MB)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"4\" nameend=\"c4\" namest=\"c1\"\u003e \u003cp\u003eNon-trainable params: 0 (0.00 Byte)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \n\u003ch3\u003eExperiment\u003c/h3\u003e\n\u003cp\u003eThe model is mainly divided into two stages. The first stage is data set preprocessing, and the second stage includes model construction and training. In order to facilitate the subsequent model training, we carried out simple preprocessing on the dataset and fed it into the S-MCCP model. The model first extracts feature maps from gesture images through depthwise separable convolution, and then produces two branches x1 and x2 through Multi-channel Cartesian product. The feature channels x1 and x2 are split into single 128 * 128 * 1 feature channels through split, and each separate 128 * 128 * 1 feature channel of x1 and x2 is calculated through Cartesian product, and concatenated to obtain a feature map with a shape of 128 * 128 * 1024. Then depthwise_conv and conv are performed on it, and then it is flattened into a one-dimensional vector by Flatten, and then output through 64, 64, and 32 fully connected layers to form 4 categories.\u003c/p\u003e \u003cp\u003eThe dataset is divided into three subsets: 3131 images are used as training set, 800 images are used as validation set, and 782 images are used as test set. The same segmentation ratio is used in each category to ensure the rationality of the training results. The model training loss uses the cross entropy loss function to calculate the loss, uses the RMSprop optimizer, and sets the learning rate to x: \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{10}^{-3}\\ast\\:{0.99}^{x}\\)\u003c/span\u003e\u003c/span\u003e to dynamically decrease. Five different random seeds of 8, 9, 10, 11, and 12 are used for training, and the average of the test results is taken as the training result to reduce the experimental error. The model training uses the early stopping strategy and trains for 50 epochs. The performance of the model on the test set under different random seeds is shown in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eLoss and accuracy under different seeds\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eMean\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTraining accuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.9895\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.9930\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.9911\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.9907\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.9946\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.991780\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eVerification accuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.9706\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.9642\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.9565\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.9565\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.9744\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.964440\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTraining loss\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.0293\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.0271\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.0210\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.0219\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.0187\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.023600\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eValidation loss\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.2944\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.2719\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.4140\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.2510\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.1514\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.276540\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eF1-score\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.970529\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.964143\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.956300\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.956408\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.974366\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.964349\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eIn order to perform a comprehensive evaluation of the model, we calculated the precision, recall, and F1-score of the model on the test set. To display the prediction results on the test set more intuitively and to facilitate the computation of these assessment metrics, we also generated the confusion matrix.The confusion matrix comprises four key metrics: TP (True Positives), TN (True Negatives), FP (False Positives), and FN (False Negatives). TP represents correctly identifying positive samples as positive; TN indicates correctly classifying negative samples as negative; FP refers to incorrectly labeling negative samples as positive; and FN denotes mistakenly classifying positive samples as negative. The confusion matrix is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e8\u003c/span\u003e and the three assessment metrics are shown in Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e. The accuracy and loss change curves of the model during training are shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig8\" class=\"InternalRef\"\u003e9\u003c/span\u003e and the mathematical formulas for calculating the three assessment metrics are shown below:\u003cdiv id=\"Equ1\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ1\" name=\"EquationSource\"\u003e\n$$\\:\\text{P}\\text{r}\\text{e}\\text{c}\\text{i}\\text{s}\\text{i}\\text{o}\\text{n}=\\frac{\\text{T}\\text{P}}{\\left(\\text{T}\\text{P}+\\text{F}\\text{P}\\right)}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e3\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equ2\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ2\" name=\"EquationSource\"\u003e\n$$\\:\\text{R}\\text{e}\\text{c}\\text{a}\\text{l}\\text{l}=\\frac{\\text{T}\\text{P}}{\\left(\\text{T}\\text{P}+\\text{F}\\text{N}\\right)}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e4\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eThe F1-score, which is the reconciled mean of precision and recall, is calculated as follows:\u003cdiv id=\"Equ3\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ3\" name=\"EquationSource\"\u003e\n$$\\:\\text{F}1=\\frac{2}{\\frac{1}{\\text{P}}+\\frac{1}{\\text{R}}}=\\frac{2\\times\\:\\text{P}\\times\\:\\text{R}}{\\text{P}+\\text{R}}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e5\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThree assessment metrics\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCategory\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePrecision\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eF1-score\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eActual number of categories\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ehappy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.978261\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.991189\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.984683\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e227\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003enervous\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.987988\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.967647\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.977712\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e340\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003esheep\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.936709\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.961039\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.948718\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e154\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003esheepy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.950820\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.950820\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.950820\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e31\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMicro average\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.971867\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.971867\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.971867\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMacro average\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.963444\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.967674\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.965554\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eWeighted average\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.972167\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.971867\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.972017\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e"},{"header":"Results and Discussion","content":"\u003cp\u003eFor infant gesture recognition, a variety of methods can be used [21,28,29,30]. For example, the traditional convolutional neural networks\u003csup\u003e[28]\u003c/sup\u003e are limited to single-channel feature image and cannot capture more features. The StarNet dual-branch architecture can only obtain the same number of feature images through fixed multiplication. The improved Cartesian product of the x1 and x2 branches proposed in this study can greatly increase the number of feature images, thereby improving the model accuracy.\u003c/p\u003e\n\u003cp\u003eIn order to further verify the improvement of the generalization ability of the Multi-channel feature channel Cartesian product compared with the previous network structure. A comparative experiment was designed to compare the traditional convolutional network structure, the StarNet structure, and the StarNet Multi-channel Cartesian product network structure. The same dataset, five random seeds 8, 9, 10, 11, 12, learning rate, optimizer, etc. were used. The experimental conditions were exactly the same except for the model network structure. The final average test accuracy is shown in Table 5. It can be clearly seen from the table that our model has a better performance on the test set, which further verifies that the Multi-channel Cartesian product can improve the model capability.\u003c/p\u003e\n\u003cp\u003eTable 5 shows that our model can obtain better generalization ability compared with similar static datasets. Some models have different reference standards for accuracy due to different datasets, but overall the S-MCCP model still has some improvements.\u003c/p\u003e\nTable 5. Some past research results on static gesture recognition\n \u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"551\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 293px;\"\u003e\n \u003cp\u003eMethodology\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\n \u003cp\u003eDataset\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003eAccuracy\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 293px;\"\u003e\n \u003cp\u003eCentroid and area of edge+Euclidean distance\u003csup\u003e[26]\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\n \u003cp\u003eReal-time recognition of 26 ASL alphabets.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003e90.19%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 293px;\"\u003e\n \u003cp\u003eNetworked human motion capture+ DNN\u003csup\u003e[27]\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\n \u003cp\u003e26 ASL alphabets.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003e98.12%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 293px;\"\u003e\n \u003cp\u003eHistogram of Oriented Gradient (HOG)+ K-NN\u003csup\u003e[28]\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\n \u003cp\u003e26 ASL alphabets.\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003e94.23%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 293px;\"\u003e\n \u003cp\u003eORB K-means clustering Bag of words (BoW)+ K-NN\u003csup\u003e\u0026nbsp;\u003c/sup\u003e\u003csup\u003e[29]\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\n \u003cp\u003eASL Finger Spelling Dataset\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003e95.81%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"4\" valign=\"top\" style=\"width: 293px;\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eS-MCCP Network Model（Ours）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\n \u003cp\u003eReal-time recognition of 26 ASL alphabets.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003e94.84%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\n \u003cp\u003e26 ASL alphabets.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003e95.16%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\n \u003cp\u003eASL Finger Spelling Dataset\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003e96.31%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 156px;\"\u003e\n \u003cp\u003eSelf-built dataset\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003e96.44%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003eBy observing Table 6, we can see that this method has good evaluation results under different indicators. Compared with the traditional convolutional network structure [23,24,25], the improved StarNet structure proposed in this paper has a better average test accuracy of 0.964440 on the same dataset.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 6.\u0026nbsp;\u003c/strong\u003eComparison between adding Cartesian product and not adding it\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"553\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 126px;\"\u003e\n \u003cp\u003eModel structure\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 173px;\"\u003e\n \u003cp\u003eFeatures\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 113px;\"\u003e\n \u003cp\u003eAverage loss\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 141px;\"\u003e\n \u003cp\u003eAverage accuracy\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 126px;\"\u003e\n \u003cp\u003eTraditional Conv\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 173px;\"\u003e\n \u003cp\u003eSingle channel convolution\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 113px;\"\u003e\n \u003cp\u003e0.3231\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 141px;\"\u003e\n \u003cp\u003e0.9540\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 126px;\"\u003e\n \u003cp\u003eStarNet\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 173px;\"\u003e\n \u003cp\u003eDual-branch fixed multiplication\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 113px;\"\u003e\n \u003cp\u003e0.2363\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 141px;\"\u003e\n \u003cp\u003e0.9578\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 126px;\"\u003e\n \u003cp\u003eS-MCCP Network Model（Ours）\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 173px;\"\u003e\n \u003cp\u003eMulti-channel Cartesian product\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 113px;\"\u003e\n \u003cp\u003e0.276540\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 141px;\"\u003e\n \u003cp\u003e0.964440\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eThe branch feature channel Cartesian product network model proposed in this study extracts infant gestures from RGB images for recognition. To better understand infant gestures, this paper first established a dataset named \"babyhand_gesture\". On this basis, based on StarNet Multi-channel Cartesian product is proposed. The model is inspired by the StarNet structure. The previous feature extraction directly uses single-channel convolution, while StarNet uses dual-branch fixed channel multiplication. However, the acquired feature images are limited. The network structure we proposed can significantly increase the number of feature images, thereby enhancing the model's generalization ability. Experimental results show that the recognition accuracy of the algorithm on the dataset is higher than that of other algorithms. In future work, we will start from more perspectives. Multimodal expansion and dynamic gesture analysis will further enhance practicality, and the release of open source resources will promote the collaborative development of academia and industry.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eData availability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe dataset used in this article is the author\u0026apos;s self-built dataset \u0026quot;babyhand_gesture\u0026quot;, the code and dataset have been made public through GitHub.\u003c/p\u003e\n\u003cp\u003eThe address is: https://github.com/Joyrides/babyhand_gesture\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis work was supported in part by the 2023 Henan Province Graduate Education Reform and Quality Improvement Project (YJS2023AL092), in part by the Henan Provincial Natural Science Foundation under Grant 242300420189, in part by the Postgraduate Joint Training Base Project of Henan Province under grant YJS2022JD45, and in part by the Key Science and Technology Research of Henan Province under grant Nos. 232102211038, 232102210076 and 232102210129.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor Contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eX.Z. and Z.L. write the main manuscript text, H.Z. and Y.C. run the code, J.Z. revise the manuscript, and J.G. review the manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare no competing interests.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAdditional information\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eCorrespondence and requests for materials should be addressed to J.G.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eZhou, Xiaodi. (2023). Language and the mind: How language shapes our thinking. Journal of World Languages. 10.1515/jwl-2023-0018.\u003c/li\u003e\n\u003cli\u003eAltarriba, J., \u0026amp; Basnight-Brown, D. (2022). The Psychology of Communication: The Interplay Between Language and Culture Through Time. Journal of Cross-Cultural Psychology, 53(7-8), 860-874. \u003c/li\u003e\n\u003cli\u003eLockhart-Bouron, M., Anikin, A., Pisanski, K. et al. Infant cries convey both stable and dynamic information about age and identity. Commun Psychol 1, 26 (2023).\u003c/li\u003e\n\u003cli\u003eCapirci O, Bonsignori C, Di Renzo A. Signed languages: A triangular semiotic dimension[J]. Frontiers in Psychology, 2022, 12: 802911. \u003c/li\u003e\n\u003cli\u003eGeorge A S H, George A S. Decoding Infant Communication: Understanding the Meaning Behind Baby\u0026apos;s Cries[J]. Partners Universal Innovative Research\u003c/li\u003e\n\u003cli\u003eCesafsky M J. Baby sign language: hindering or enhancing communication in infants and toddlers?[J]. 2009.\u003c/li\u003e\n\u003cli\u003eGurrola A. The Effects of Using Baby Sign Training on the Interactions Between Mothers and Infants[D]. The University of Texas at El Paso, 2023. s\u003c/li\u003e\n\u003cli\u003eDoherty-Sneddon G. The great baby signing debate: Academia meets public interest[J]. The Psychologist, 2008, 21(4): 300-303.\u003c/li\u003e\n\u003cli\u003ePontecorvo E, Higgins M, Mora J, et al. Learning a sign language does not hinder acquisition of a spoken language[J]. Journal of Speech, Language, and Hearing Research, 2023, 66(4): 1291-1308.\u003c/li\u003e\n\u003cli\u003eFay N, Walker B, Ellison T M, et al. Gesture is the primary modality for language creation[J]. Proceedings of the Royal Society B, 2022, 289(1970): 20220066.\u003c/li\u003e\n\u003cli\u003eCongdon E L, Goldin-Meadow S. Mechanisms of embodied learning through gestures and actions: Lessons from development[J]. Handbook of embodied psychology: Thinking, feeling, and acting, 2021: 527-546.\u003c/li\u003e\n\u003cli\u003eNadgeri, Sulochana \u0026amp; Kumar, Arun. (2022). DEEP LEARNING BASED FRAMEWORK FOR DYNAMIC BABY SIGN LANGUAGE RECOGNITION SYSTEM. Indian Journal of Computer Science and Engineering. 13. 550-563. 10.21817/indjcse/2022/v13i2/221302127.\u003c/li\u003e\n\u003cli\u003eHoward A G, Zhu M, Chen B, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications[J]. arXiv preprint arXiv:1704.04861, 2017.\u003c/li\u003e\n\u003cli\u003eDuan H, Sun Y, Cheng W, et al. Gesture recognition based on multi‐modal feature weight[J]. Concurrency and Computation: Practice and Experience, 2021, 33(5): e5991.\u003c/li\u003e\n\u003cli\u003eMongardi A, Rossi F, Prestia A, et al. Hand gestures recognition for human-machine interfaces: A low-power bio-inspired armband[J]. IEEE Transactions on Biomedical Circuits and Systems, 2022, 16(6): 1348-1365.\u003c/li\u003e\n\u003cli\u003eKarsh B, Laskar R H, Karsh R K. mIV3Net: modified inception V3 network for hand gesture recognition[J]. Multimedia Tools and Applications, 2024, 83(4): 10587-10613.\u003c/li\u003e\n\u003cli\u003eChen J, Chen S, Wee L, et al. Deep learning based unpaired image-to-image translation applications for medical physics: a systematic review[J]. Physics in Medicine \u0026amp; Biology, 2023, 68(5): 05TR01.\u003c/li\u003e\n\u003cli\u003eTchane Djogdom G V, Otis M J D, Meziane R. Dynamic time warping\u0026ndash;based feature selection method for foot gesture cobot operation mode selection[J]. The International Journal of Advanced Manufacturing Technology, 2023, 126(9): 4521-4541.\u003c/li\u003e\n\u003cli\u003eAdeyanju I A, Bello O O, Adegboye M A. Machine learning methods for sign language recognition: A critical review and analysis[J]. Intelligent Systems with Applications, 2021, 12: 200056.\u003c/li\u003e\n\u003cli\u003eKasapbaşi A, Elbushra A E A, Omar A L H, et al. DeepASLR: A CNN based human computer interface for American Sign Language recognition for hearing-impaired individuals[J]. Computer methods and programs in biomedicine update, 2022, 2: 100048.\u003c/li\u003e\n\u003cli\u003eMa X, Dai X, Bai Y, et al. Rewrite the Stars[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 5694-5703.\u003c/li\u003e\n\u003cli\u003ehttps://github.com/Joyrides/babyhand_gesture\u003c/li\u003e\n\u003cli\u003eEysenbach G, CONSORT-EHEALTH Group CONSORT-EHEALTH: Improving and Standardizing Evaluation Reports of Web-based and Mobile Health Interventions J Med Internet Res 2011;13(4):e126\u003c/li\u003e\n\u003cli\u003ehttps://psychology.fandom.com/wiki/Posture_(psychology)\u003c/li\u003e\n\u003cli\u003ehttps://www.improvementtower.com/bodylanguage/body_langage_signs_of_nervousness_tension.html\u003c/li\u003e\n\u003cli\u003ehttps://test.scienceabc.com/humans/why-do-our-fingers-curl-when-sleeping.html\u003c/li\u003e\n\u003cli\u003eCobb, K., Goodwin, R., \u0026amp; Saelens, E. (1966). Spontaneous Hand Positions of Newborn Infants. The Journal of Genetic Psychology, 108(2), 225\u0026ndash;237. \u003c/li\u003e\n\u003cli\u003eLeCun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.\u003c/li\u003e\n\u003cli\u003eKrizhevsky A , Sutskever I , Hinton G . ImageNet Classification with Deep Convolutional Neural Networks[J]. Advances in neural information processing systems, 2012, 25(2).\u003c/li\u003e\n\u003cli\u003eSimonyan K , Zisserman A . Very Deep Convolutional Networks for Large-Scale Image Recognition[J]. Computer Science, 2014.\u003c/li\u003e\n\u003cli\u003ePansare, J R, Dhumal, H., Babar, S., and International, K.S. (2015). Real time static hand gesture recognition system in complex background that uses number system of indian sign language static hand gesture recognition for sign language alphabets using edge oriented histogram and multi class SVM real \u0026shy; Time hand gesture R. 2(3), 2014\u0026ndash;2015.\u003c/li\u003e\n\u003cli\u003eJ. Li, Z.-L. Wang, H. Zhao, R. Gravina, G. Fortino, Y. Jiang, et al., \u0026quot;Networked human motion capture system based on quaternion navigation\u0026quot;, Proc. 11th EAI Int. Conf. Body Area Netw., vol. 5, pp. 38-44, 2016.\u003c/li\u003e\n\u003cli\u003eI. Mahmud, T. Tabassum, M.P. Uddin, E. Ali, A.M. Nitu, M.I. Afjal Efficient noise reduction and HOG feature extraction for sign language recognition Proceedings of the international conference on advancement in electrical and electronic engineering, ICAEEE 2018 (2019), pp.\u003c/li\u003e\n\u003cli\u003eA. Sharma, A. Mittal, S. Singh, V Awatramani Hand gesture recognition using image processing and feature extraction techniques Procedia Computer Science, 173 (2019) (2020), pp. 181-190,10.1016/j.procs.2020.06.022.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Infant gestures, StarNet, Convolutional neural networks, Computer vision","lastPublishedDoi":"10.21203/rs.3.rs-6217770/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6217770/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eInfants communicate with others through nonverbal gestures in their early years, expressing their needs and emotions through gestural signs. But it is very challenging for adults to understand babies who have no language skills. In order to better understand these infant gestures, the author first constructed a dataset (babyhand_gesture), which contains four gestures, such as: \"happy\", \"nervous\", \"sleep\" and \"sleepy\". On this basis, this paper proposes the S-MCCP (Based on StarNet Multi-channel Cartesian product) network framework for infant gesture detection. This method not only utilizes the model expression capability of the high-dimensional feature space of the StarNet structure, but also utilizes the Cartesian product operation to enhance the number of feature images, thereby improving the model accuracy and generalization ability. Experimental results show that the algorithm successfully achieved a recognition accuracy of 96.4% on the dataset, which is nearly one percentage point higher than other algorithms. When tested on three other datasets, the proposed algorithm also had the highest accuracy.\u003c/p\u003e","manuscriptTitle":"Infant gesture detection algorithm Based on StarNet Multi-channel Cartesian product network","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-21 09:54:05","doi":"10.21203/rs.3.rs-6217770/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"783e64ca-c8c9-4ae4-b9b7-e986cb6fe08b","owner":[],"postedDate":"April 21st, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":47149866,"name":"Physical sciences/Mathematics and computing/Computer science"},{"id":47149867,"name":"Physical sciences/Mathematics and computing/Information technology"}],"tags":[],"updatedAt":"2025-05-15T07:23:55+00:00","versionOfRecord":[],"versionCreatedAt":"2025-04-21 09:54:05","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6217770","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6217770","identity":"rs-6217770","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}