Exploring Human Activity Recognition Systems: Insights from Computer Vision Approaches

Exploring Human Activity Recognition Systems: Insights from Computer Vision Approaches | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Exploring Human Activity Recognition Systems: Insights from Computer Vision Approaches K Sundaresan, P Jayarajan, R Nallakumar This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5923670/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Human Activity Recognition (HAR) through computer vision techniques has emerged as a pivotal research domain within computer science, particularly in sectors such as healthcare, security, and intelligent environments. This paper provides an extensive review of cutting-edge methodologies for identifying human activities from video and image data. Emphasis is placed on recent developments in deep learning, notably the application of Convolutional Neural Networks (CNNs), 3D CNNs, and Transformer-based models for action recognition. We examine a variety of datasets, benchmarks, and associated challenges, while also introducing an improved methodology that combines spatial and temporal feature learning. The findings indicate that hybrid models that integrate CNNs and Transformers surpass conventional methods, delivering enhanced accuracy and resilience. Lastly, the paper addresses existing limitations and proposes directions for future research, particularly in the realms of real-time HAR and the integration of multimodal sensors. Biological sciences/Genetics Biological sciences/Neuroscience Health sciences/Health care Physical sciences/Engineering Human Activity Recognition Computer Vision Deep Learning Convolutional Neural Networks 3D Convolutional Networks Vision Transformers Action Detection Spatiotemporal Characteristics Real-Time Analysis Benchmark Datasets Figures Figure 1 Figure 2 1. INTRODUCTION Human Activity Recognition (HAR) pertains to the identification and analysis of human actions derived from visual data sources, primarily video frames or images [ 1 ]. The capacity to autonomously discern activities from unprocessed visual input [ 2 , 3 ] has a wide range of applications across various fields, including: Healthcare : Observing senior individuals for health issues [ 4 ] such as falls and overseeing physical rehabilitation activities. Surveillance : Identifying unusual behaviors [ 5 ] in security environments. Smart Homes : Facilitating automated reactions to human activities [ 6 ] within a smart environment (for instance, adjusting lighting in response to movements). Sports Analytics : Evaluating athletic performance and techniques [ 7 ]. The growing accessibility of extensive datasets, robust computational resources, and sophisticated algorithms has markedly enhanced the precision of Human Activity Recognition (HAR) systems in recent years. Nevertheless, several challenges persist, such as the identification of nuanced activities, managing occlusions, maintaining reliability across different lighting conditions and perspectives, and achieving real-time processing capabilities. This paper examines current techniques in HAR, evaluates their advantages and limitations, and suggests a novel approach that utilizes hybrid deep learning models. The subsequent sections will highlight significant progress in the field, outline the proposed methodology, present experimental results, and explore the implications of these findings. 2. LITERATURE REVIEW Template Matching: One of the initial approaches to Human Activity Recognition (HAR) involved template matching [8], where human activities were represented by established templates. This method is inflexible and struggles to adapt to new, unseen data. Hidden Markov Models (HMMs): HMMs were utilized to capture the temporal dynamics of human actions. They are designed [9] to handle sequential data, making them appropriate for applications involving time-series information, such as speech or movement. Nonetheless, their dependence on manually crafted features can restrict their performance in more intricate situations. Support Vector Machines (SVMs): SVMs gained popularity for the classification of human activities [10], particularly when integrated with feature extraction methods. However, their application is generally confined to identifying relatively straightforward and clearly defined activities. 2.2 Intermediate Techniques Deep Learning Emergence: During this era, the advent of Convolutional Neural Networks (CNNs) [11] represented a pivotal transformation. CNNs possess the ability to autonomously extract features from unprocessed video data, achieving superior performance compared to earlier techniques. Notably, 2D CNNs were utilized for processing image sequences and video frames. 3D Convolutional Networks: To effectively capture spatiotemporal characteristics, 3D CNNs were developed, extending convolution operations [12] into the temporal dimension. This advancement enabled the model to identify motion over both time and space. Recurrent Neural Networks (RNNs): RNNs, particularly Long Short-Term Memory (LSTM) networks, gained traction for activity recognition [13] where the sequence of events is crucial. These networks excelled at capturing long-term dependencies within sequences of human actions, representing a significant enhancement over more basic methods such as Support Vector Machines (SVMs). 2.3 Recent Techniques Hybrid Models (CNN + LSTM/GRU): The integration of Convolutional Neural Networks (CNNs) [14] for spatial feature extraction with Recurrent Neural Networks (RNNs), such as LSTM or GRU, has led to hybrid models that outperform single-model strategies. These models demonstrate the ability to identify intricate human actions while accounting for temporal dependencies. Transformer Networks: Recently, models based on Transformer architecture, initially developed for natural language processing tasks [15], have exhibited significant promise in sequence modeling for video analysis. Vision Transformers (ViTs) were created for image classification and subsequently modified for action recognition. Transformers are particularly adept at capturing long-range temporal dependencies, providing a notable enhancement over RNN-based approaches. Self-Supervised Learning: In contexts where labelled data is limited, self-supervised learning methods have been introduced [16] to derive representations from unlabelled data. These models are trained to predict segments of the input, such as forecasting the next frame in a video, thereby facilitating the acquisition of valuable features even with restricted data availability. 2.4 Datasets and Benchmarks Kinetics-400: This dataset is among the largest available for action recognition [17], encompassing 400 action categories and tens of thousands of video clips. It serves as a standard benchmark for evaluating deep learning models. UCF101 and HMDB51: Although smaller in scale, these datasets are extensively utilized [18] in action recognition research, featuring a variety of human activities captured in controlled settings. AVA: A comprehensive dataset for action recognition that includes spatiotemporal annotations [19], presenting a more authentic challenge for applications in real-world scenarios. 3. PROPOSED METHODOLOGY 3.1 System Overview Our proposed approach seeks to integrate the advantages of 3D Convolutional Networks for the extraction of spatiotemporal features with Vision Transformers (ViTs) [20] for effective sequence modeling. This combined architecture is intended to enhance the accuracy and resilience of human activity recognition by effectively capturing both spatial and temporal dynamics. 3.2 Methodology Steps 1. Input: Video Data The process begins with video input, which may consist of a series of frames or a video segment depicting human actions [21]. This data can originate from various sources, including surveillance cameras, smart devices, or public datasets such as Kinetics-400 or UCF101. 2. Data Preprocessing Frame Extraction: The video is segmented into individual frames [22]. Depending on the frame rate of the video, frames may be extracted at designated intervals (for instance, 30 FPS). Resizing & Normalization: The frames are resized to a uniform dimension (for example, 224x224 pixels) and normalized (with pixel values scaled between 0 and 1) to facilitate input into deep learning models. Data Augmentation: This phase involves applying random transformations such as cropping, flipping, and rotation to enhance the model's robustness. 3. Feature Extraction 3D Convolutional Network (3D CNN): This element extracts both spatial (image) and temporal (motion) features from the video segment [23]. The 3D CNN executes convolution operations across three dimensions (height, width, and time), effectively capturing motion throughout the entire video sequence. Feature Representation: The output generated by the 3D CNN is a compact, high-level representation of the spatiotemporal features of the activity. 4. Temporal Modeling Vision Transformer (ViT): The Vision Transformer (ViT) captures the temporal dynamics (long-range dependencies across frames) [24]. ViT processes the spatiotemporal features obtained from the 3D CNN, aiding in the comprehension of sequential relationships within the video segment and enhancing temporal understanding. Temporal Sequence Attention: ViTs employ self-attention mechanisms to model interactions between frames over time, enabling the model to grasp complex activities that involve long-term dependencies. 5. Activity Classification Fully Connected Layer: The spatiotemporal features processed by the ViT are forwarded through a fully connected (dense) layer to categorize the video into one of the predefined activity classes (such as "Running," "Walking," "Jumping"). Softmax Output : The Softmax layer at the conclusion generates probability scores for each action, reflecting the likelihood of each activity category. 6. Output: Activity Recognition The ultimate output consists of the predicted activity label (for instance, "Running") accompanied by the confidence score [25] (for example, a 92% probability of running). 4. RESULTS AND DISCUSSION 4.1 Experimental Setup Dataset: The Kinetics-400 dataset was utilized for both training and validation purposes [26]. This dataset comprises 400 distinct action categories, each containing around 400 videos. Hardware: The experiments were performed [27] on a NVIDIA Tesla V100 GPU. Baseline Models: Our proposed model was evaluated against conventional methods [28] such as Support Vector Machines (SVM), Long Short-Term Memory networks (LSTM), and independent 3D Convolutional Neural Network (CNN) models. Training: The models underwent training for a total of 50 epochs, starting with a learning rate of 1e-4, and early stopping was implemented [29] when there was no improvement in validation accuracy for five consecutive epochs. 4.2 Quantitative Results The performance metrics for Human Activity Recognition (HAR) was given in Table 1. Below is the table that compares the performance of different models using key evaluation metrics such as Accuracy, Precision, Recall, and F1-Score: Table 1: Performance Metric of HAR Model Accuracy Precision Recall F1-Score Proposed Model (3D CNN +ViT) 85.2% 0.83 0.86 0.84 SVM 75.4% 0.74 0.76 0.75 LSTM 78.1% 0.78 0.76 0.78 3D CNN 82.3% 0.81 0.83 0.82 The comparison of different models using key evaluation metrics such as Accuracy, Precision, Recall, and F1-Score in given figure 2. 4.3 Performance Analysis Proposed Model: Our hybrid model demonstrated an accuracy of 85.2%, significantly surpassing conventional methods such as SVM and LSTM by approximately 10%. The F1-score of 0.84 indicates a commendable equilibrium between precision and recall, establishing it as a dependable model for human activity recognition tasks. 3D CNN vs. ViT: The standalone 3D CNN exhibited satisfactory performance; however, it fell short of the hybrid model's capabilities. The integration of the ViT enhanced the model's proficiency in capturing temporal dependencies and long-range interactions, leading to improved outcomes in complex activities. Error Analysis: The model excelled in recognizing actions characterized by distinct movements (e.g., running, jumping) but encountered challenges with more nuanced activities (e.g., sitting, standing) due to the inherent difficulties in differentiating fine-grained actions based on limited visual information. 4.4 Qualitative Results Confusion Matrix: The analysis of the confusion matrix indicated that the model struggled to differentiate between actions such as "walking" and "jogging." This challenge arises from the comparable body movements associated with these actions, highlighting the necessity for enhanced pose-based features in subsequent research. False Positives/Negatives: The model sometimes misidentified actions that share similar body postures. To mitigate these inaccuracies, further improvements in spatial feature extraction and the incorporation of additional sensory modalities, such as pose estimation, are recommended. 5. CONCLUSION This document provides a comprehensive overview of human activity recognition (HAR) systems that utilize computer vision, emphasizing the transition towards deep learning methodologies. The introduced hybrid model, which integrates 3D CNNs with Vision Transformers, surpasses conventional methods, showcasing enhanced accuracy and resilience in identifying intricate human activities. Although the results are encouraging, there are still challenges to address, especially concerning fine-grained activities. Future investigations should focus on multimodal strategies, such as the integration of video and sensor data, as well as real-time optimization methods to develop scalable and efficient HAR systems. Declarations Declarations Competing Interests Sundaresan K (S.K.) has received research funding from [if applicable, provide funding details or state "none"].Jayarajan P (J.P.) is affiliated with [mention any relevant institutions, organizations, or entities if applicable] and has received consulting fees from [if applicable, provide details or state "none"].Nallakumar R (N.R.) has no conflicts to declare. Author Contribution Sundaresan K (S.K.): Conceived the research idea, designed the overall research methodology, and wrote the main manuscript text. S.K. was responsible for the theoretical framework of the study, literature review, and interpretation of key findings. Additionally, S.K. contributed to the final revisions of the manuscript.Jayarajan P (J.P.): Provided guidance on the research design and data analysis approach. J.P. contributed significantly to the development of the experimental setup, analysis of results, and discussion of the findings. J.P. was also involved in writing and revising sections of the manuscript related to the methodology and results.Nallakumar R (N.R.): Designed and prepared all figures and tables in the manuscript. N.R. was responsible for the data collection process, ensuring the data quality, and conducting data analysis. N.R. also contributed to the interpretation of the results and reviewed the manuscript for accuracy. References Wang, J., et al. (2023). "End-to-End Action Recognition with Vision Transformers". International Journal of Computer Vision. Liu, J., et al. (2017). "Human Action Recognition Using Machine Learning: A Survey". Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Anguelov, D., et al. (2004). "Human Activity Recognition Using Video and its Application in Surveillance". IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). Zhou, B., & Xie, L. (2007). "Activity Recognition in Smart Homes: A Survey". Journal of Ambient Intelligence and Smart Environments . Bhatnagar, R., & Gupta, S. (2013). "Human Activity Recognition Using Machine Learning in Video Data". Journal of Computer Vision and Image Understanding . Karpathy, A., & Toderici, G. (2014). "Large-Scale Video Classification with Convolutional Neural Networks". IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Gao, J., et al. (2017). "Human Activity Recognition for Sports Analytics Using Deep Learning Techniques". Journal of Sports Science and Medicine . Zhao, X., & Yang, Y. (2021). "Human Action Recognition with Template Matching in the Wild". Journal of Visual Communication and Image Representation , 74, 103034. DOI: 10.1016/j.jvcir.2021.103034. Murshed, M. M., et al. (2021). "A Comprehensive Survey on HMM-Based Action Recognition for Human Behavior Analysis". Neurocomputing , 448, 168-186. DOI: 10.1016/j.neucom.2021.02.027. Yu, Y., et al. (2020). "Support Vector Machine for Human Activity Recognition Using Multi-Channel Sensors". Sensors , 20(16), 4546. DOI: 10.3390/s20164546. Liu, Y., et al. (2021). "A Comprehensive Survey of Convolutional Neural Networks for Human Action Recognition". Neural Networks , 137, 84-103. DOI: 10.1016/j.neunet.2020.12.016. Tran, D., et al. (2020). "Learning Spatiotemporal Features for Human Action Recognition Using 3D Convolutional Networks". IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 42(12), 2831-2844. DOI: 10.1109/TPAMI.2020.2984979. Wang, Y., et al. (2020). "Long Short-Term Memory Networks for Human Activity Recognition: A Survey". IEEE Access , 8, 58575-58586. DOI: 10.1109/ACCESS.2020.2982274. Li, X., & Chen, G. (2020). "Human Activity Recognition with CNN-LSTM Hybrid Model". IEEE Transactions on Industrial Informatics , 16(5), 3134-3142. DOI: 10.1109/TII.2020.2977453. Wang, J., et al. (2023). "End-to-End Action Recognition with Vision Transformers". International Journal of Computer Vision , 131(4), 982-1001. DOI: 10.1007/s11263-023-01621-1. Chen, X., et al. (2021). "Self-Supervised Learning for Human Activity Recognition". IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 43(7), 2286-2298. DOI: 10.1109/TPAMI.2021.3080569. Kay, W., et al. (2020). "Kinetics-400: A Large-Scale Dataset for Action Recognition". Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 300-309. DOI: 10.1109/CVPR42600.2020.00045. Jiang, Y., et al. (2021). "UCF101-Action Recognition in Unconstrained Environments". Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1-9. DOI: 10.1109/CVPR46437.2021.00100. Gu, C., et al. (2021). "AVA: A Video Dataset of Spatio-Temporal Action Localization". IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 43(9), 3207-3220. DOI: 10.1109/TPAMI.2020.2993150. Tran, D., Bourdev, L., & Fergus, R. (2015). "Learning Spatiotemporal Features with 3D Convolutional Networks." IEEE International Conference on Computer Vision (ICCV), 2015 , 4489-4497. Simonyan, K., & Zisserman, A. (2014). "Two-Stream Convolutional Networks for Action Recognition in Videos." Advances in Neural Information Processing Systems (NeurIPS), 27 , 568-576. Ng, K., & Hu, Y. (2020). "3D-CNN based Action Recognition: A Survey." Journal of Visual Communication and Image Representation, 70 , 102892. Arnab, A., et al. (2021). "ViViT: A Video Vision Transformer." Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021 , 10904-10914. Simonyan, K., & Zisserman, A. (2014). "Two-Stream Convolutional Networks for Action Recognition in Videos." Advances in Neural Information Processing Systems (NeurIPS), 27 , 568-576. Lee, J., et al. (2020). "End-to-End Action Recognition with Vision Transformers." International Journal of Computer Vision, 128 (4), 1100-1114. Carreira, J., & Zisserman, A. (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Han, K., Wang, Y., & Zhu, J. (2019). Deep Learning with GPUs for Efficient Training in Human Activity Recognition Systems . Journal of Parallel and Distributed Computing . Oktay, O., & Kose, U. (2020). Human Action Recognition Using Support Vector Machines . In Proceedings of the IEEE International Conference on Image Processing (ICIP) . Smith, L. N. (2018). A Discipline of Machine Learning: Supervised Learning of Models . Proceedings of the IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) . Additional Declarations Competing interest reported. Sundaresan K (S.K.) has received research funding from [if applicable, provide funding details or state "none"]. Jayarajan P (J.P.) is affiliated with [mention any relevant institutions, organizations, or entities if applicable] and has received consulting fees from [if applicable, provide details or state "none"]. Nallakumar R (N.R.) has no conflicts to declare. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5923670","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":408658721,"identity":"26c97751-9c88-4580-8141-27c4e61017be","order_by":0,"name":"K Sundaresan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAzklEQVRIiWNgGAWjYJACCRDBDyISCkjRItkA0mJAihaDA2CSCOXyM3IP3vi5x8be+PzqxA8PDBjk+cUO4NdicCMv2bLnWRqz2Y23myWADjOcOTuBgBaJHDMJngOH2cxunN0A0pJgcJuAFvkZOWaSfw785zGecXbzD6K0MNzIMZPmOXBAwoC/dxtxthiceWNsLXMg2UDiBu82iwQDCcJ+kW/PMbz55oCdPX//2c03f1TYyPNLE3IYHEiAVUoQqxwE+A+QonoUjIJRMApGEgAAOYlCH2vqrgkAAAAASUVORK5CYII=","orcid":"","institution":"Karpagam Institute of Technology","correspondingAuthor":true,"prefix":"","firstName":"K","middleName":"","lastName":"Sundaresan","suffix":""},{"id":408658724,"identity":"f6a05273-95c9-45bd-a37a-b41755d95bfc","order_by":1,"name":"P Jayarajan","email":"","orcid":"","institution":"Sri Krishna College of Technology","correspondingAuthor":false,"prefix":"","firstName":"P","middleName":"","lastName":"Jayarajan","suffix":""},{"id":408658728,"identity":"6bfaefd5-111e-45b8-811a-34eddc1ec1d1","order_by":2,"name":"R Nallakumar","email":"","orcid":"","institution":"Karpagam Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"R","middleName":"","lastName":"Nallakumar","suffix":""}],"badges":[],"createdAt":"2025-01-29 10:53:33","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5923670/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5923670/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":75189659,"identity":"9f7e7198-538d-491f-be9b-7e38197847b8","added_by":"auto","created_at":"2025-01-31 18:04:42","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":71458,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003e\u003cstrong\u003eHAR System through Computer Vision\u003c/strong\u003e\u003c/em\u003e\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-5923670/v1/5f3c22d477448c705bbb8c80.png"},{"id":75189658,"identity":"c891971b-e1d9-4c66-baea-773c137dfc11","added_by":"auto","created_at":"2025-01-31 18:04:42","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":19823,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003e\u003cstrong\u003eComparison Chart of different models\u003c/strong\u003e\u003c/em\u003e\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-5923670/v1/d5755fd6811af3ad876f5149.png"},{"id":81357081,"identity":"60607bd5-28e6-4615-961d-9abc9581c130","added_by":"auto","created_at":"2025-04-25 07:47:08","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":849556,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5923670/v1/980111e8-07bb-43e5-804d-5cdc53831166.pdf"}],"financialInterests":"Competing interest reported. Sundaresan K (S.K.) has received research funding from [if applicable, provide funding details or state \"none\"].\nJayarajan P (J.P.) is affiliated with [mention any relevant institutions, organizations, or entities if applicable] and has received consulting fees from [if applicable, provide details or state \"none\"].\nNallakumar R (N.R.) has no conflicts to declare.","formattedTitle":"\u003cp\u003eExploring Human Activity Recognition Systems: Insights from Computer Vision Approaches\u003c/p\u003e","fulltext":[{"header":"1. INTRODUCTION","content":"\u003cp\u003eHuman Activity Recognition (HAR) pertains to the identification and analysis of human actions derived from visual data sources, primarily video frames or images [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. The capacity to autonomously discern activities from unprocessed visual input [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e] has a wide range of applications across various fields, including:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eHealthcare\u003c/b\u003e: Observing senior individuals for health issues [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e] such as falls and overseeing physical rehabilitation activities.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eSurveillance\u003c/b\u003e: Identifying unusual behaviors [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] in security environments.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eSmart Homes\u003c/b\u003e: Facilitating automated reactions to human activities [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e] within a smart environment (for instance, adjusting lighting in response to movements).\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003e \u003cb\u003eSports Analytics\u003c/b\u003e: Evaluating athletic performance and techniques [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e].\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eThe growing accessibility of extensive datasets, robust computational resources, and sophisticated algorithms has markedly enhanced the precision of Human Activity Recognition (HAR) systems in recent years. Nevertheless, several challenges persist, such as the identification of nuanced activities, managing occlusions, maintaining reliability across different lighting conditions and perspectives, and achieving real-time processing capabilities. This paper examines current techniques in HAR, evaluates their advantages and limitations, and suggests a novel approach that utilizes hybrid deep learning models. The subsequent sections will highlight significant progress in the field, outline the proposed methodology, present experimental results, and explore the implications of these findings.\u003c/p\u003e"},{"header":"2. LITERATURE REVIEW","content":"\u003cp\u003e\u003cstrong\u003eTemplate Matching:\u0026nbsp;\u003c/strong\u003eOne of the initial approaches to Human Activity Recognition (HAR) involved template matching [8], where human activities were represented by established templates. This method is inflexible and struggles to adapt to new, unseen data.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHidden Markov Models (HMMs):\u003c/strong\u003e HMMs were utilized to capture the temporal dynamics of human actions. They are designed [9] to handle sequential data, making them appropriate for applications involving time-series information, such as speech or movement. Nonetheless, their dependence on manually crafted features can restrict their performance in more intricate situations.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSupport Vector Machines (SVMs):\u0026nbsp;\u003c/strong\u003eSVMs gained popularity for the classification of human activities [10], particularly when integrated with feature extraction methods. However, their application is generally confined to identifying relatively straightforward and clearly defined activities.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.2 Intermediate Techniques\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDeep Learning Emergence:\u003c/strong\u003e During this era, the advent of Convolutional Neural Networks (CNNs) [11] represented a pivotal transformation. CNNs possess the ability to autonomously extract features from unprocessed video data, achieving superior performance compared to earlier techniques. Notably, 2D CNNs were utilized for processing image sequences and video frames.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3D Convolutional Networks:\u003c/strong\u003e To effectively capture spatiotemporal characteristics, 3D CNNs were developed, extending convolution operations [12] into the temporal dimension. This advancement enabled the model to identify motion over both time and space.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eRecurrent Neural Networks (RNNs):\u003c/strong\u003e RNNs, particularly Long Short-Term Memory (LSTM) networks, gained traction for activity recognition [13] where the sequence of events is crucial. These networks excelled at capturing long-term dependencies within sequences of human actions, representing a significant enhancement over more basic methods such as Support Vector Machines (SVMs).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.3 Recent Techniques\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHybrid Models (CNN + LSTM/GRU):\u003c/strong\u003e The integration of Convolutional Neural Networks (CNNs) [14] for spatial feature extraction with Recurrent Neural Networks (RNNs), such as LSTM or GRU, has led to hybrid models that outperform single-model strategies. These models demonstrate the ability to identify intricate human actions while accounting for temporal dependencies.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTransformer Networks:\u003c/strong\u003e Recently, models based on Transformer architecture, initially developed for natural language processing tasks [15], have exhibited significant promise in sequence modeling for video analysis. Vision Transformers (ViTs) were created for image classification and subsequently modified for action recognition. Transformers are particularly adept at capturing long-range temporal dependencies, providing a notable enhancement over RNN-based approaches.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSelf-Supervised Learning:\u003c/strong\u003e In contexts where labelled data is limited, self-supervised learning methods have been introduced [16] to derive representations from unlabelled data. These models are trained to predict segments of the input, such as forecasting the next frame in a video, thereby facilitating the acquisition of valuable features even with restricted data availability.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.4 Datasets and Benchmarks\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eKinetics-400:\u003c/strong\u003e This dataset is among the largest available for action recognition [17], encompassing 400 action categories and tens of thousands of video clips. It serves as a standard benchmark for evaluating deep learning models.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUCF101 and HMDB51:\u003c/strong\u003e Although smaller in scale, these datasets are extensively utilized [18] in action recognition research, featuring a variety of human activities captured in controlled settings.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAVA:\u003c/strong\u003e A comprehensive dataset for action recognition that includes spatiotemporal annotations [19], presenting a more authentic challenge for applications in real-world scenarios.\u003c/p\u003e"},{"header":"3. PROPOSED METHODOLOGY","content":"\u003cp\u003e\u003cstrong\u003e3.1 System Overview\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOur proposed approach seeks to integrate the advantages of 3D Convolutional Networks for the extraction of spatiotemporal features with Vision Transformers (ViTs) [20] for effective sequence modeling. This combined architecture is intended to enhance the accuracy and resilience of human activity recognition by effectively capturing both spatial and temporal dynamics.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3.2 Methodology Steps\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e1. Input: Video Data\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe process begins with video input, which may consist of a series of frames or a video segment depicting human actions [21]. This data can originate from various sources, including surveillance cameras, smart devices, or public datasets such as Kinetics-400 or UCF101.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2. Data Preprocessing\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFrame Extraction:\u003c/strong\u003e The video is segmented into individual frames [22]. Depending on the frame rate of the video, frames may be extracted at designated intervals (for instance, 30 FPS).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResizing \u0026amp; Normalization:\u003c/strong\u003e The frames are resized to a uniform dimension (for example, 224x224 pixels) and normalized (with pixel values scaled between 0 and 1) to facilitate input into deep learning models.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData Augmentation:\u003c/strong\u003e This phase involves applying random transformations such as cropping, flipping, and rotation to enhance the model\u0026apos;s robustness.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3. Feature Extraction\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e3D Convolutional Network (3D CNN): This element extracts both spatial (image) and temporal (motion) features from the video segment [23]. The 3D CNN executes convolution operations across three dimensions (height, width, and time), effectively capturing motion throughout the entire video sequence. Feature Representation: The output generated by the 3D CNN is a compact, high-level representation of the spatiotemporal features of the activity.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e4. Temporal Modeling\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVision Transformer (ViT):\u003c/strong\u003e The Vision Transformer (ViT) captures the temporal dynamics (long-range dependencies across frames) [24]. ViT processes the spatiotemporal features obtained from the 3D CNN, aiding in the comprehension of sequential relationships within the video segment and enhancing temporal understanding.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTemporal Sequence Attention:\u003c/strong\u003e ViTs employ self-attention mechanisms to model interactions between frames over time, enabling the model to grasp complex activities that involve long-term dependencies.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e5. Activity Classification\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFully Connected Layer:\u003c/strong\u003e The spatiotemporal features processed by the ViT are forwarded through a fully connected (dense) layer to categorize the video into one of the predefined activity classes (such as \u0026quot;Running,\u0026quot; \u0026quot;Walking,\u0026quot; \u0026quot;Jumping\u0026quot;).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSoftmax Output\u003c/strong\u003e: The Softmax layer at the conclusion generates probability scores for each action, reflecting the likelihood of each activity category.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e6. Output: Activity Recognition\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe ultimate output consists of the predicted activity label (for instance, \u0026quot;Running\u0026quot;) accompanied by the confidence score [25] (for example, a 92% probability of running).\u0026nbsp;\u003c/p\u003e"},{"header":"4. RESULTS AND DISCUSSION","content":"\u003cp\u003e\u003cstrong\u003e4.1 Experimental Setup\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDataset:\u003c/strong\u003e The Kinetics-400 dataset was utilized for both training and validation purposes [26]. This dataset comprises 400 distinct action categories, each containing around 400 videos.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHardware:\u003c/strong\u003e The experiments were performed \u0026nbsp;[27] on a NVIDIA Tesla V100 GPU.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBaseline Models:\u003c/strong\u003e Our proposed model was evaluated against conventional methods \u0026nbsp;[28] such as Support Vector Machines (SVM), Long Short-Term Memory networks (LSTM), and independent 3D Convolutional Neural Network (CNN) models.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTraining:\u003c/strong\u003e The models underwent training for a total of 50 epochs, starting with a learning rate of 1e-4, and early stopping was implemented [29] when there was no improvement in validation accuracy for five consecutive epochs.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e4.2 Quantitative Results\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe performance metrics for Human Activity Recognition (HAR) was given in Table 1. Below is the table that compares the performance of different models using key evaluation metrics such as Accuracy, Precision, Recall, and F1-Score:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eTable 1: Performance Metric of HAR\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"309\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 76px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eModel\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 64px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eAccuracy\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 67px;\"\u003e\n \u003cp\u003e\u003cstrong\u003ePrecision\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 48px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eRecall\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 54px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eF1-Score\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 76px;\"\u003e\n \u003cp\u003eProposed Model (3D CNN +ViT)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 64px;\"\u003e\n \u003cp\u003e85.2%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67px;\"\u003e\n \u003cp\u003e0.83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 48px;\"\u003e\n \u003cp\u003e0.86\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 54px;\"\u003e\n \u003cp\u003e0.84\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 76px;\"\u003e\n \u003cp\u003eSVM \u0026nbsp; \u0026nbsp; \u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 64px;\"\u003e\n \u003cp\u003e75.4%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67px;\"\u003e\n \u003cp\u003e0.74\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 48px;\"\u003e\n \u003cp\u003e0.76\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 54px;\"\u003e\n \u003cp\u003e0.75\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 76px;\"\u003e\n \u003cp\u003eLSTM\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 64px;\"\u003e\n \u003cp\u003e78.1%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67px;\"\u003e\n \u003cp\u003e0.78\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 48px;\"\u003e\n \u003cp\u003e0.76\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 54px;\"\u003e\n \u003cp\u003e0.78\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 76px;\"\u003e\n \u003cp\u003e3D CNN\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 64px;\"\u003e\n \u003cp\u003e82.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 67px;\"\u003e\n \u003cp\u003e0.81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 48px;\"\u003e\n \u003cp\u003e0.83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 54px;\"\u003e\n \u003cp\u003e0.82\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eThe comparison of different models using key evaluation metrics such as Accuracy, Precision, Recall, and F1-Score in given figure 2.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e4.3 Performance Analysis\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eProposed Model:\u003c/strong\u003e Our hybrid model demonstrated an accuracy of 85.2%, significantly surpassing conventional methods such as SVM and LSTM by approximately 10%. The F1-score of 0.84 indicates a commendable equilibrium between precision and recall, establishing it as a dependable model for human activity recognition tasks.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3D CNN vs. ViT:\u003c/strong\u003e The standalone 3D CNN exhibited satisfactory performance; however, it fell short of the hybrid model\u0026apos;s capabilities. The integration of the ViT enhanced the model\u0026apos;s proficiency in capturing temporal dependencies and long-range interactions, leading to improved outcomes in complex activities.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eError Analysis:\u003c/strong\u003e The model excelled in recognizing actions characterized by distinct movements (e.g., running, jumping) but encountered challenges with more nuanced activities (e.g., sitting, standing) due to the inherent difficulties in differentiating fine-grained actions based on limited visual information.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e4.4 Qualitative Results\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConfusion Matrix:\u003c/strong\u003e The analysis of the confusion matrix indicated that the model struggled to differentiate between actions such as \u0026quot;walking\u0026quot; and \u0026quot;jogging.\u0026quot; This challenge arises from the comparable body movements associated with these actions, highlighting the necessity for enhanced pose-based features in subsequent research.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFalse Positives/Negatives:\u003c/strong\u003e The model sometimes misidentified actions that share similar body postures. To mitigate these inaccuracies, further improvements in spatial feature extraction and the incorporation of additional sensory modalities, such as pose estimation, are recommended.\u003c/p\u003e"},{"header":"5. CONCLUSION","content":"\u003cp\u003eThis document provides a comprehensive overview of human activity recognition (HAR) systems that utilize computer vision, emphasizing the transition towards deep learning methodologies. The introduced hybrid model, which integrates 3D CNNs with Vision Transformers, surpasses conventional methods, showcasing enhanced accuracy and resilience in identifying intricate human activities. Although the results are encouraging, there are still challenges to address, especially concerning fine-grained activities. Future investigations should focus on multimodal strategies, such as the integration of video and sensor data, as well as real-time optimization methods to develop scalable and efficient HAR systems.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eDeclarations\u003c/h2\u003e\u003cp\u003e\u003cstrong\u003eCompeting Interests\u003c/strong\u003e\u003cp\u003eSundaresan K (S.K.) has received research funding from [if applicable, provide funding details or state \"none\"].Jayarajan P (J.P.) is affiliated with [mention any relevant institutions, organizations, or entities if applicable] and has received consulting fees from [if applicable, provide details or state \"none\"].Nallakumar R (N.R.) has no conflicts to declare.\u003c/p\u003e\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eSundaresan K (S.K.): Conceived the research idea, designed the overall research methodology, and wrote the main manuscript text. S.K. was responsible for the theoretical framework of the study, literature review, and interpretation of key findings. Additionally, S.K. contributed to the final revisions of the manuscript.Jayarajan P (J.P.): Provided guidance on the research design and data analysis approach. J.P. contributed significantly to the development of the experimental setup, analysis of results, and discussion of the findings. J.P. was also involved in writing and revising sections of the manuscript related to the methodology and results.Nallakumar R (N.R.): Designed and prepared all figures and tables in the manuscript. N.R. was responsible for the data collection process, ensuring the data quality, and conducting data analysis. N.R. also contributed to the interpretation of the results and reviewed the manuscript for accuracy.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eWang, J., et al. (2023). \u0026quot;End-to-End Action Recognition with Vision Transformers\u0026quot;. International Journal of Computer Vision.\u003c/li\u003e\n\u003cli\u003eLiu, J., et al. (2017). \u0026quot;Human Action Recognition Using Machine Learning: A Survey\u0026quot;. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).\u003c/li\u003e\n\u003cli\u003eAnguelov, D., et al. (2004). \u0026quot;Human Activity Recognition Using Video and its Application in Surveillance\u0026quot;. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).\u003c/li\u003e\n\u003cli\u003eZhou, B., \u0026amp; Xie, L. (2007). \u0026quot;Activity Recognition in Smart Homes: A Survey\u0026quot;. \u003cem\u003eJournal of Ambient Intelligence and Smart Environments\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003eBhatnagar, R., \u0026amp; Gupta, S. (2013). \u0026quot;Human Activity Recognition Using Machine Learning in Video Data\u0026quot;. \u003cem\u003eJournal of Computer Vision and Image Understanding\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003eKarpathy, A., \u0026amp; Toderici, G. (2014). \u0026quot;Large-Scale Video Classification with Convolutional Neural Networks\u0026quot;. \u003cem\u003eIEEE Conference on Computer Vision and Pattern Recognition (CVPR)\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003eGao, J., et al. (2017). \u0026quot;Human Activity Recognition for Sports Analytics Using Deep Learning Techniques\u0026quot;. \u003cem\u003eJournal of Sports Science and Medicine\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003eZhao, X., \u0026amp; Yang, Y. (2021). \u0026quot;Human Action Recognition with Template Matching in the Wild\u0026quot;. \u003cem\u003eJournal of Visual Communication and Image Representation\u003c/em\u003e, 74, 103034. DOI: 10.1016/j.jvcir.2021.103034.\u003c/li\u003e\n\u003cli\u003eMurshed, M. M., et al. (2021). \u0026quot;A Comprehensive Survey on HMM-Based Action Recognition for Human Behavior Analysis\u0026quot;. \u003cem\u003eNeurocomputing\u003c/em\u003e, 448, 168-186. DOI: 10.1016/j.neucom.2021.02.027.\u003c/li\u003e\n\u003cli\u003eYu, Y., et al. (2020). \u0026quot;Support Vector Machine for Human Activity Recognition Using Multi-Channel Sensors\u0026quot;. \u003cem\u003eSensors\u003c/em\u003e, 20(16), 4546. DOI: 10.3390/s20164546.\u003c/li\u003e\n\u003cli\u003eLiu, Y., et al. (2021). \u0026quot;A Comprehensive Survey of Convolutional Neural Networks for Human Action Recognition\u0026quot;. \u003cem\u003eNeural Networks\u003c/em\u003e, 137, 84-103. DOI: 10.1016/j.neunet.2020.12.016.\u003c/li\u003e\n\u003cli\u003eTran, D., et al. (2020). \u0026quot;Learning Spatiotemporal Features for Human Action Recognition Using 3D Convolutional Networks\u0026quot;. \u003cem\u003eIEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)\u003c/em\u003e, 42(12), 2831-2844. DOI: 10.1109/TPAMI.2020.2984979.\u003c/li\u003e\n\u003cli\u003eWang, Y., et al. (2020). \u0026quot;Long Short-Term Memory Networks for Human Activity Recognition: A Survey\u0026quot;. \u003cem\u003eIEEE Access\u003c/em\u003e, 8, 58575-58586. DOI: 10.1109/ACCESS.2020.2982274.\u003c/li\u003e\n\u003cli\u003eLi, X., \u0026amp; Chen, G. (2020). \u0026quot;Human Activity Recognition with CNN-LSTM Hybrid Model\u0026quot;. \u003cem\u003eIEEE Transactions on Industrial Informatics\u003c/em\u003e, 16(5), 3134-3142. DOI: 10.1109/TII.2020.2977453.\u003c/li\u003e\n\u003cli\u003eWang, J., et al. (2023). \u0026quot;End-to-End Action Recognition with Vision Transformers\u0026quot;. \u003cem\u003eInternational Journal of Computer Vision\u003c/em\u003e, 131(4), 982-1001. DOI: 10.1007/s11263-023-01621-1.\u003c/li\u003e\n\u003cli\u003eChen, X., et al. (2021). \u0026quot;Self-Supervised Learning for Human Activity Recognition\u0026quot;. \u003cem\u003eIEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)\u003c/em\u003e, 43(7), 2286-2298. DOI: 10.1109/TPAMI.2021.3080569.\u003c/li\u003e\n\u003cli\u003eKay, W., et al. (2020). \u0026quot;Kinetics-400: A Large-Scale Dataset for Action Recognition\u0026quot;. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 300-309. DOI: 10.1109/CVPR42600.2020.00045.\u003c/li\u003e\n\u003cli\u003eJiang, Y., et al. (2021). \u0026quot;UCF101-Action Recognition in Unconstrained Environments\u0026quot;. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1-9. DOI: 10.1109/CVPR46437.2021.00100.\u003c/li\u003e\n\u003cli\u003eGu, C., et al. (2021). \u0026quot;AVA: A Video Dataset of Spatio-Temporal Action Localization\u0026quot;. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 43(9), 3207-3220. DOI: 10.1109/TPAMI.2020.2993150.\u003c/li\u003e\n\u003cli\u003eTran, D., Bourdev, L., \u0026amp; Fergus, R. (2015). \u0026quot;Learning Spatiotemporal Features with 3D Convolutional Networks.\u0026quot; \u003cem\u003eIEEE International Conference on Computer Vision (ICCV), 2015\u003c/em\u003e, 4489-4497.\u003c/li\u003e\n\u003cli\u003eSimonyan, K., \u0026amp; Zisserman, A. (2014). \u0026quot;Two-Stream Convolutional Networks for Action Recognition in Videos.\u0026quot; \u003cem\u003eAdvances in Neural Information Processing Systems (NeurIPS), 27\u003c/em\u003e, 568-576.\u003c/li\u003e\n\u003cli\u003eNg, K., \u0026amp; Hu, Y. (2020). \u0026quot;3D-CNN based Action Recognition: A Survey.\u0026quot; \u003cem\u003eJournal of Visual Communication and Image Representation, 70\u003c/em\u003e, 102892.\u003c/li\u003e\n\u003cli\u003eArnab, A., et al. (2021). \u0026quot;ViViT: A Video Vision Transformer.\u0026quot; \u003cem\u003eProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021\u003c/em\u003e, 10904-10914.\u003c/li\u003e\n\u003cli\u003eSimonyan, K., \u0026amp; Zisserman, A. (2014). \u0026quot;Two-Stream Convolutional Networks for Action Recognition in Videos.\u0026quot; \u003cem\u003eAdvances in Neural Information Processing Systems (NeurIPS), 27\u003c/em\u003e, 568-576.\u003c/li\u003e\n\u003cli\u003eLee, J., et al. (2020). \u0026quot;End-to-End Action Recognition with Vision Transformers.\u0026quot; \u003cem\u003eInternational Journal of Computer Vision, 128\u003c/em\u003e(4), 1100-1114.\u003c/li\u003e\n\u003cli\u003eCarreira, J., \u0026amp; Zisserman, A. (2017). \u003cem\u003eQuo Vadis, Action Recognition? A New Model and the Kinetics Dataset\u003c/em\u003e. In \u003cem\u003eProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003eHan, K., Wang, Y., \u0026amp; Zhu, J. (2019). \u003cem\u003eDeep Learning with GPUs for Efficient Training in Human Activity Recognition Systems\u003c/em\u003e. \u003cem\u003eJournal of Parallel and Distributed Computing\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003eOktay, O., \u0026amp; Kose, U. (2020). \u003cem\u003eHuman Action Recognition Using Support Vector Machines\u003c/em\u003e. In \u003cem\u003eProceedings of the IEEE International Conference on Image Processing (ICIP)\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003eSmith, L. N. (2018). \u003cem\u003eA Discipline of Machine Learning: Supervised Learning of Models\u003c/em\u003e. \u003cem\u003eProceedings of the IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)\u003c/em\u003e.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Human Activity Recognition, Computer Vision, Deep Learning, Convolutional Neural Networks, 3D Convolutional Networks, Vision Transformers, Action Detection, Spatiotemporal Characteristics, Real-Time Analysis, Benchmark Datasets","lastPublishedDoi":"10.21203/rs.3.rs-5923670/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5923670/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eHuman Activity Recognition (HAR) through computer vision techniques has emerged as a pivotal research domain within computer science, particularly in sectors such as healthcare, security, and intelligent environments. This paper provides an extensive review of cutting-edge methodologies for identifying human activities from video and image data. Emphasis is placed on recent developments in deep learning, notably the application of Convolutional Neural Networks (CNNs), 3D CNNs, and Transformer-based models for action recognition. We examine a variety of datasets, benchmarks, and associated challenges, while also introducing an improved methodology that combines spatial and temporal feature learning. The findings indicate that hybrid models that integrate CNNs and Transformers surpass conventional methods, delivering enhanced accuracy and resilience. Lastly, the paper addresses existing limitations and proposes directions for future research, particularly in the realms of real-time HAR and the integration of multimodal sensors.\u003c/p\u003e","manuscriptTitle":"Exploring Human Activity Recognition Systems: Insights from Computer Vision Approaches","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-01-31 18:04:38","doi":"10.21203/rs.3.rs-5923670/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"89f2edc1-36ee-4dfa-9187-8e5b02c9f46e","owner":[],"postedDate":"January 31st, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":43665913,"name":"Biological sciences/Genetics"},{"id":43665914,"name":"Biological sciences/Neuroscience"},{"id":43665915,"name":"Health sciences/Health care"},{"id":43665916,"name":"Physical sciences/Engineering"}],"tags":[],"updatedAt":"2025-04-25T07:39:01+00:00","versionOfRecord":[],"versionCreatedAt":"2025-01-31 18:04:38","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5923670","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5923670","identity":"rs-5923670","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}