Enhancing American Sign Language Alphabet Recognition: A Fusion of Media Pipe and LSTM Technologies

Enhancing American Sign Language Alphabet Recognition: A Fusion of Media Pipe and LSTM Technologies | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Enhancing American Sign Language Alphabet Recognition: A Fusion of Media Pipe and LSTM Technologies Pentamaraju Abhinav, Harshlata Vishwakarma This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4323367/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 3 You are reading this latest preprint version Abstract With the advancement of today’s technologies in artificial intelligence, humans tend to use hand gestures in their communication to convey their ideas. Gesture recognition is an active area of research in the human-computer interface (HCI). Gesture recognition is important for communication between deaf-mute people, HCI, robot control, home automation, and medical applications. In this article, a simple and efficient vision-based approach for American Sign Language (ASL) alphabets recognition has been discussed to recognize both static and dynamic gestures. Media pipe introduced by Google had been used to get hand landmarks and a custom data set has been created and used for the experimental study. Hand gesture recognition has been done by using Long short-term memory (LSTM). The proposed system has been investigated with 26 alphabets and an accuracy of 99% has been achieved. This work can be used to convert hand gestures into text. Hand Gesture Recognition American Sign Language (ASL) Media pipe Long Short-Term Memory (LSTM) Human-Computer Interface Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 1. INTRODUCTION The advancement in the development of various technologies for smartphones, laptops, notebooks, and various handheld devices like game applications flourishes research in the gesture recognition domain. a wearable hand gloves-based sensor was used for gesture recognition system. the production cost is high Now camera vision-based sensors are used because it is Userfriendly and simple. Gestures are any patterns or motions of the hands, face, or body used to create certain expressions in sign language. Additionally, static hand positions and configurations without any movement can be used to express gestures. A single hand position can represent a distance. The proposed system uses ASL for alphabet recognition. ASL is one of the most widely used languages in the world. There are around 250,000 to 500,000 people using sign language on a daily basis around the world. It is very difficult for people to understand sign language if they are not familiar with it. Gesture recognition can be used in these situations to translate gestures into natural language text for communication. The aim of the study is to help people recognize alphabets that are communicated using ASL. The proposed system can be used to translate the sign language into text and HCI. The novelty of the proposed system is that it uses Google’s Media Pipe for hand landmark detection which is more accurate and faster when compared to the conventional methods that use geometric, shape, and edge features. LSTM model was found to be very effective for sequence data modeling and has been used for gesture recognition. Sign language is a form of communication that relies heavily on hand kinematics and facial emotions. Hearing-impaired persons use it frequently to communicate with one another, but it is rarely utilized by non-hearing-impaired people. As a result, they exclusively deal directly with hearing-impaired people, drastically limiting social interactions. Real-time translation with interpreters is an option, able it. As a result, an automatic translation system would be quite beneficial. In this subject, many novel strategies have lately been created. We implement a code to translate sign language into OpenCV in this project. It describes a method for recognizing and translating Custom Sign Language 2. LITERATURE REVIEW Many authors have proposed various image processing and Machine Learning (ML) techniques for sign language recognition. Some of the relevant works have been discussed here in the middle of the 1970s, Myron W. Krueger originally suggested gesture recognition as a brand-new method of communication between people and computers. With the rapid advancement of computer hardware and vision systems over the last few years, it has emerged as a very significant study topic. It discussed how to use grayscale images and edge detection techniques to detect hand gestures. The drawback of this technique is that there are some limitations when it comes to grayscale images as they are only 2D data and it is difficult to extract the key features of the hand by Sammon Babu [ 1 ] He introduced a technique to recognize hand gestures using a leap motion controller which is quite expensive and requires additional hardware by Sundar Baglamas T [ 2 ] It is a technique that uses gloves and several hand sensors to recognize different hand gestures. The sensors have motion detectors that are costly, and it makes the hand hard to move because of the weight by Sai Bharath Padigala Gogineni Hrushikesh MadhavSaranu Kishore Kumar [ 3 ] It is discussed a technique using deep learning to recognize the ASL gestures with the help of a public dataset available on Kaggle. It employs several ML algorithms in the dataset and gives a classified report on the results for each ML algorithm by Sakshi Mankar Kanishka Mohapatra Mansi Talavadekar [ 4 ]. It is a technique that uses color features and contour extraction to determine rather than the different hand gestures by Mallikarjun Rao, Cheguri SowmyaP.A. Sujasri [ 5 ] It is an approach for Realtime hand gesture recognition for human computer interaction using CNN. The drawback here is that it does not detect any sign language gestures rather it detects some hand signs that are used for Human computer Interface rather than sign language by Pei Xu [ 6 ]. It is technique Recognition of Dynamic Hand Gestures from 3D Motion Data using LSTM and CNN architectures. Published in 2017 by Chinmaya R. Naguri, Razvan C. Bunescu.[ 7 ]. It is technique Hand Gesture Tracking and Recognition based Human Computer Interaction System and Its Applications published in 2018 by Chinmaya R. Naguri, Razvan C. Bunescu [ 8 ]. A Method for Stochastic Optimization. Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015 by Diederik P. Kingma, Jimmy Ba. Adam [ 9 ]. The Linguistics of American Sign Language, American National Standard for Information Sciences.by Clayton Valli, Ceil Lucas [ 10 ]. It is Hand Gesture Recognition for Human Computer Interaction. 7th International Conference on Advances in Computing Communications, ICACC- 2017, 22–24 August 2017, Cochin, India by Aashni Hariaa, Archanasri Subramaniana, Nivedhitha Asokkumara, Shristi Poddara, Jyothi S Nayaka,[ 11 ] It is Hand Gesture Detection for American Sign Language using K- Nearest Neighbor with Media pipe by Arsheldy Alvin1, Nabila Husna Shabrina2, Aurelius Ryo3, Edgar Christian4 Fakultas Teknik dan Informatika, Universitas Multimedia Nusan- Tara, Teknik Komputer Tangerang [ 12 ] A Real- time Hand Gesture Recognition and Human- Computer Interaction System by Pei Xu Department of Electrical and Computer Engineering, University of Minnesota, Twin Cities [ 13 ] A Google Re- search 1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA by Fan Zhang ,Valentin Bazarevsky ,Andrey Vakunov ,Andrei Tkachenka ,George Sung, Chuo Ling Chang,Matthias Grundmann [14] Hand gesture recognition using machine learning algorithms. Computer Science and Information Technologies by Abhishek B, Kanya Krishi, Meghana M, Mohammed Daaniyaal, Anupama HS, BMS Institute of Technology, Bangalore, India.[ 15 ] The Hand gesture recognition is based on convolution neural network. part of Springer Nature 2017 by Gongfa Li, · Heng Tang, · Ying Sun, · Jianyi Kong, · Guozhang Jiang, · D u Jiang · Bo Tao, ·Shuang Xu, · Honghai Liu.[ 16 ]. The Dynamic Hand Gesture Recognition Using 3DCNN and LSTM with FSM Context-Aware Model by Noorkholis Luthfil Hakim, Timothy K. Shih, Sandeli Priyanwada Kasthuri Arachchi ,Wisnu Aditya, Yi-Cheng Chen and Chih-Yang Lin [ 17 ]. 3. Proposed System The proposed system for the ASL alphabet recognition is shown in Fig. 5 . The proposed system recognizes the American sign language of alphabets. It recognizes gestures with motion and gestures without motion. The first step is to collect the dataset for the system which will be discussed in the results and discussion section. Media Pipe hands is a high- fidelity hand and finger tracking framework used in dataset preparation. It makes use of a number of ML models to infer the 21 three dimensional landmarks of a hand in real time from just one frame. Media pipe Media pipe hand key point detection works with a machine learning pipeline. A palm detection model that works on the entire image is one of the models in the Media Pipe machine learning pipeline. It receives the entire image and outputs a hand-bounding box that is orientated. a hand landmark model that uses the palm.Detector cropped image region and outputs highly accurate 3D hand key points [ 15 ]. Additionally, the hand is cropped by the ML pipeline using the hand landmarks found in the previous frame, and palm detection is only used to localize the hand when the land- mark model is unable to do so. Figure 2 shows the 21hand landmarks that can be tracked from the Media pipe's hand landmark detector. Arsheldy Alvin, etal [ 13 ] have used Media pipe and K nearest neighbors algorithm to determine the hand gestures which helps us in understanding how to use media pipes for hand gesture recognition The difference between tensor flow and media pipe is that tensor flow is a free and open-source software library for machine learning and artificial intelligence. It can be used across a range of tasks but has a particular focus on training and inference of deep neural networks. Whereas Media pipe offers open-source cross-platform, customizable ML solutions for live and streaming media. In this paper we have used media pipe for capturing the hand key points and tensor flow for training and detecting the ML algorithm. Media pipe runs with both on CPU and GPU. No additional processing power is required to run Media pipe. Sign Language Recognition Using LSTM The next step is to train the LSTM model using the dataset. LSTM is one of the feed- forward neural networks. Since, in American alphabet hand gestures there are several hand gestures that have hand motion in them instead of a Single static pose. So, it's better to use LSTM using which we can recognize the continuous motion. The advantage of using LSTM is that it allows not only single data points i.e. images, but it also allows the entire sequences of data [ 19 ]. An artificial neural network is a network of interconnected neurons, each of which is connected to one another and is modelled after the brain’s organic neural networks. We can execute sophisticated operations on the data by combining a number of different algorithms, each of which is capable of completing a different task [ 20 ]. A class of neural networks called Recurring Neural Networks, or RNNs, is designed specifically to handle temporal data. The input is processed using the internal state of the RNN neurons, which is accomplished with the aid of loops built into the neural network. RNN neurons have a cell state or memory. From the training dataset, Spatiotemporal features are extracted, and the LSTM model classification model is derived. The generated ML model is stored for prediction of gestures in real time. All the 26 alphabet gestures are collected one after the other and are preprocessed into NumPy arrays and stored. The dataset collected is then passed onto the LSTM model. The LSTM model will have all the 26 alphabet classes for each of the alphabets. The algorithm is made to run on 2000 epochs until the accuracy becomes closer to 1 and training loss goes to nearly 0. The algorithm runs with the above specified features and generates the model which is stored for future use. The model can now be used for tracking the live hand gestures and for further operations. The system uses RELU activation. The rectified linear activation function, also known as Re LU, is a piecewise linear function that, if the input is positive, outputs the input directly; if not, it outputs zero. Because a model that utilizes it is simpler to train and frequently performs better, it has evolved into the default activation function for many types of neutral networks [ 20 ]. The system uses Adam Optimizer technique. In place of the conventional stochastic gradient descent method, Adam is an optimization technique that may be used to repeatedly update network weights depending on training data [ 10 ]. For all weight updates, stochastic gradient descent maintains a constant learning rate (referred to alpha),which does not change throughout training. As learning progresses, the learning rate is maintained and independently adjusted for each network weight (parameter). The main reason for using the LSTM model is that we can detect motion from hand gestures, and it can be used to detect hand gestures that have motion in them. In Fig. 3 , we can see the diagram of LSTM cell. The LSTM cell has three steps namely the forget gate, input gate and output gate. The forget gate establishes how much the prior data should be forgotten. The amount of information to be printed on the internal cell is determined by the input gate. The output gate decides what output to produce based on the internal cell state at the moment. Electric energy consumption modelling using LSTM has been studied in. LSTM based natural language processing model has been used for Malayalam language analysis and paraphrase identification. Table 1 Summary of the sequential model Layer Output Shape Param Conv1d(Conv1D) (None,None,60) 360 lstm(LSTM) (None,None,60) 29040 lstm1(Lstm) (None,60) 29040 dense(Dense) (None,30) 1830 dense_`1(Dense) (None,10) 310 dense_2(Dense) (None,1) 11 lamda(Lambda) (None,1) 0 4. EXPERIMENTAL RESULTS Dataset S. The dataset required for the experimental study of the proposed algorithm is created using Media Pipe which is an opensource library. It captures the hand gestures for 30 frames and stores them as NumPy array for training the model. The dataset has 30 different permutations of a single alphabet. The same process is repeated with four different demographs of humans to get a diverse dataset. The four different humans are two male hands, one female hand, and one kid’s hand. This makes a total of 93,600 NumPy array, these are indexed and stored. The dataset has different parameters that are x ,y and z axes of the hand joints. The dataset is divided into 90% for training and10% fortesting.. 5. CONCLUSION The proposed vision based system can be used to recognize the ASL alphabet and convert them into text. Media Pipe hand landmark features and LSTM model was found to be effective for gesture recognition. Accuracy, Micro Average and Weighted Average of the alphabets are 0.99 for the custom dataset, this corroborates that the proposed system can be effectively used for HCI. The system can be used in HCI like interacting with the computer based on the gesture performed. Furthermore, an example could be to use the alphabets detected on a word document to type the alphabets instead of a keyboard or for typing on the search engine. This work can be further extended work with real time applications and multiple letters, or word recognition can be done to control or interact with the applications. Declarations Declarations The authors declare no conflict of interest and affirm that this work has not been published elsewhere. Author Contribution Identifying the research objectives and designing the study approachDeveloping the algorithms, models, and experimental setup for sign language recognition.Gathering sign language datasets and ensuring their quality and relevance.Writing the code and software necessary for sign language recognition.Testing and validating the accuracy and effectiveness of the sign language recognition system Data Availability Statment: The dataset comprising ASL hand gesture images encompasses 2425 samples captured from 5 individuals. Notably, their study exhibits a limitation wherein the ASL images undergo rotation and processing through open-source tools like ImageMagick. References A. A. Abdulhussein and F. A. Raheem, “Hand gesture recognition of static letters American sign language (ASL) using deep Learning,” Engineering and Technology Journal, Vol. 38, No.06 pp. 926–937, 2020. Teak Wei Chong and BoonGiin Lee, American Sign Language Recognition Using Leap Motion Controller with Machine Learning Approach, Department of Electronic Engineering, Keimyung University, Daegu 42601, Korea, October 2018. Matteo Rinalduzzi, Alessio De Angelis, Francesco Santoni, Emanuele Buchicchio, Antonio Moschitta, Paolo Carbone, Paolo Bellitti, Mauro Serpelloni, Gesture Recognition of Sign Language Alphabet Using a Magnetic Positioning System, June 2021. Rajesh George Rajan, Dr.M.Judith Leo, American Sign Language Alphabets Recognition using Hand Crafted and Deep Learning Features IEEE Xplore Part Number: CFP20F70-ART published on 2020. Sourav Bhowmick, Sushant Kumar and Anurag Kumar, Hand Gesture Recognition of English Alphabets using Artificial Neural Network published on 2015. Ross E. Mitchell, Gallaudet Research Institute, Draft manuscript accepted for publication in Sign Language Studies, Volume 6, Number 3, 2006. Onamon Pinsanoh, Yuttana Kitjaidure, Ariya Thongtawee. A Novel Feature Extraction for American Sign Language Recognition Using Web- cam. Published in 2018. Chinmaya R. Naguri, Razvan C. Bunescu.Recognition of Dynamic Hand Gestures from 3D Motion Data using LSTM and CNN architectures. Published in 2017. Kai Li, Qieshi Zhang, Jun Cheng, Jianming Liu. Hand Gesture Tracking and Recognition based Human Computer Interaction System and Its Applications published on 2018. Diederik P. Kingma, Jimmy Ba. Adam: A Method for Stochastic Optimization. Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015. Clayton Valli, Ceil Lucas, Linguistics Of American Sign Language, American National Standard for Information Sciences. Aashni Hariaa, Archanasri Subramaniana, Nivedhitha Asokkumara, Shristi Poddara, Jyothi S Nayaka, Hand Gesture Recognition for Human Computer Interaction. 7th International Conference on Advances in Computing Communications, ICACC- 2017, 22–24 August 2017, Cochin, India. Arsheldy Alvin1, Nabila Husna Shabrina2, Aurelius Ryo3, Edgar Christian4 Fakultas Teknik dan Informatika, Universitas Multimedia Nusan- Tara, Teknik Komputer Tangerang, Indonesia. Hand Gesture Detection for American Sign Language using K- Nearest Neighbor with Media pipe .[14]. Pei Xu, Department of Electrical and Computer Engineering, University of Minnesota, Twin Cities. A Real- time Hand Gesture Recognition and Human- Computer Interaction System. Fan Zhang,Valentin Bazarevsky ,Andrey Vakunov ,Andrei Tkachenka ,George Sung, Chuo Ling Chang,Matthias Grundmann. Google Re- search 1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA. Abhishek B, Kanya Krishi, Meghana M, Mohammed Daaniyaal, Anupama H S, BMS Institute of Technology, Bangalore, India. Hand gesture recognition using machine learning algorithms. Computer Science and Information Technologies. Gongfa Li, · Heng Tang, · Ying Sun, · Jianyi Kong, · Guozhang Jiang, · D u Jiang · Bo Tao, ·Shuang Xu, · Honghai Liu. Hand gesture recognition based on convolution neural network. part of Springer Nature 2017. Noorkholis Luthfil Hakim, Timothy K. Shih, Sandeli Priyanwada Kasthuri Arachchi ,Wisnu Aditya, Yi-Cheng Chen and Chih-Yang Lin. Dynamic Hand Gesture Recognition Using 3DCNN and LSTM with FSM Context-Aware Model. Sepp Hochreiter,Fakultat fur Informatik, Technische Universitat Munchen, 80290 Munchen, Germany. LONG SHORT-TERM MEMORY, Neural Computation 9(8):1735–1780, 1997. Deep Learning in Neural Networks: An Overview.Jurgen Schmidhuber,The Swiss AI Lab IDSIA,Istituto Dalle Molle di Studi sull’Intelligenza Artificiale,University of Lugano SUPSI,Galleria 2, 6928 Manno-Lugano,Switzerland,8 October 2014. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editor assigned by journal 21 Jun, 2024 Submission checks completed at journal 20 Jun, 2024 First submitted to journal 25 Apr, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4323367","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":317244898,"identity":"26e34d5f-a2d7-4e0f-80d6-317959f731aa","order_by":0,"name":"Pentamaraju Abhinav","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABLklEQVRIiWNgGAWjYHCDAwzMjA0SQAbzASAhIYNHKWMDmha2BJAWHiK0MIC0gCgeAzCJS705e+/xBz9z7PL5G88Yfi7cYZHHPyPn86sbNRY8DOyHj27AosWy51xiY++2ZMsZB84YS888I1EscSN3m3XOMaDDeNLSbmDRYnAjx7CBdxuzAcOBMwbSvG0SiQ1nzm4zzmEDapHgMcOq5f4bw8a/2+oN5IG2/AZpmX/mzDPjnH94tNzgMWzm3XbYwODAGTOwLRuO9zA/zm3DrcWyJ8dwtuy24waGB46VWc8Eatl4vM2MObdPgocNh1/M2c8YfHy7rdpA7sbhzbcL2+oS5x1mfvw551udHD/74WNYHQZnSRyAM9kkwCQW5aha+BvgTOYPOFSPglEwCkbByAQADTtqN4q0B0MAAAAASUVORK5CYII=","orcid":"","institution":"Vit Bhopal University","correspondingAuthor":true,"prefix":"","firstName":"Pentamaraju","middleName":"","lastName":"Abhinav","suffix":""},{"id":317244899,"identity":"3ae31be9-aa2d-4fee-8c8e-ee6835f3405d","order_by":1,"name":"Harshlata Vishwakarma","email":"","orcid":"","institution":"Vit Bhopal University","correspondingAuthor":false,"prefix":"","firstName":"Harshlata","middleName":"","lastName":"Vishwakarma","suffix":""}],"badges":[],"createdAt":"2024-04-25 10:11:36","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4323367/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4323367/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":59950685,"identity":"dc93995e-9be8-40e8-be23-5710ad882dec","added_by":"auto","created_at":"2024-07-09 17:34:34","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":124977,"visible":true,"origin":"","legend":"\u003cp\u003eHand Landmarks representation used in Media Pipe.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-4323367/v1/080526ceafd3239bb1605f4f.png"},{"id":59950682,"identity":"48d1f91c-5912-4a36-af07-8d72878cd081","added_by":"auto","created_at":"2024-07-09 17:34:34","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":124185,"visible":true,"origin":"","legend":"\u003cp\u003eLSTM Architecture. From\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-4323367/v1/bbb6f70f240b578377a32c0e.png"},{"id":59951092,"identity":"40999ac3-7ee1-43e7-9616-032de65fb556","added_by":"auto","created_at":"2024-07-09 17:42:34","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":34034,"visible":true,"origin":"","legend":"\u003cp\u003eGestures for Each Alphabets from ASL\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-4323367/v1/b00abfcaad0a38854e18eb77.png"},{"id":59951093,"identity":"1cd0244e-88cc-412e-893b-d29fa9011e28","added_by":"auto","created_at":"2024-07-09 17:42:34","extension":"jpeg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":802123,"visible":true,"origin":"","legend":"\u003cp\u003eScreenshots of the Results\u003c/p\u003e","description":"","filename":"floatimage4.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-4323367/v1/2e93128c71d8956692633e31.jpeg"},{"id":59950683,"identity":"e5ab0834-1069-422b-acf0-1e8327bd2ecf","added_by":"auto","created_at":"2024-07-09 17:34:34","extension":"jpeg","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":155973,"visible":true,"origin":"","legend":"\u003cp\u003eModel accuracy \u0026amp; loss graphs\u003c/p\u003e","description":"","filename":"floatimage5.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-4323367/v1/8333a5223365cb53d393d2ed.jpeg"},{"id":59950687,"identity":"2e4cdb1a-558a-4c3a-9ca6-8246e7906cfb","added_by":"auto","created_at":"2024-07-09 17:34:34","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":197821,"visible":true,"origin":"","legend":"\u003cp\u003eConfusion matrix\u003c/p\u003e","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-4323367/v1/5dd10322285960b6d8df8928.png"},{"id":59951097,"identity":"fb6b397c-c00c-4455-9e45-ba765a7ff48d","added_by":"auto","created_at":"2024-07-09 17:42:43","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1742697,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4323367/v1/6b07f15c-6e3c-4974-bcc3-6ea4b39b80d9.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Enhancing American Sign Language Alphabet Recognition: A Fusion of Media Pipe and LSTM Technologies","fulltext":[{"header":"1. INTRODUCTION","content":"\u003cp\u003eThe advancement in the development of various technologies for smartphones, laptops, notebooks, and various handheld devices like game applications flourishes research in the gesture recognition domain. a wearable hand gloves-based sensor was used for gesture recognition system. the production cost is high Now camera vision-based sensors are used because it is Userfriendly and simple. Gestures are any patterns or motions of the hands, face, or body used to create certain expressions in sign language. Additionally, static hand positions and configurations without any movement can be used to express gestures. A single hand position can represent a distance.\u003c/p\u003e \u003cp\u003eThe proposed system uses ASL for alphabet recognition. ASL is one of the most widely used languages in the world. There are around 250,000 to 500,000 people using sign language on a daily basis around the world. It is very difficult for people to understand sign language if they are not familiar with it. Gesture recognition can be used in these situations to translate gestures into natural language text for communication.\u003c/p\u003e \u003cp\u003eThe aim of the study is to help people recognize alphabets that are communicated using ASL. The proposed system can be used to translate the sign language into text and HCI. The novelty of the proposed system is that it uses Google\u0026rsquo;s Media Pipe for hand landmark detection which is more accurate and faster when compared to the conventional methods that use geometric, shape, and edge features. LSTM model was found to be very effective for sequence data modeling and has been used for gesture recognition.\u003c/p\u003e \u003cp\u003eSign language is a form of communication that relies heavily on hand kinematics and facial emotions. Hearing-impaired persons use it frequently to communicate with one another, but it is rarely utilized by non-hearing-impaired people. As a result, they exclusively deal directly with hearing-impaired people, drastically limiting social interactions. Real-time translation with interpreters is an option, able it. As a result, an automatic translation system would be quite beneficial. In this subject, many novel strategies have lately been created. We implement a code to translate sign language into OpenCV in this project. It describes a method for recognizing and translating Custom Sign Language\u003c/p\u003e"},{"header":"2. LITERATURE REVIEW","content":"\u003cp\u003eMany authors have proposed various image processing and Machine Learning (ML) techniques for sign language recognition. Some of the relevant works have been discussed here in the middle of the 1970s, Myron W. Krueger originally suggested gesture recognition as a brand-new method of communication between people and computers. With the rapid advancement of computer hardware and vision systems over the last few years, it has emerged as a very significant study topic.\u003c/p\u003e \u003cp\u003eIt discussed how to use grayscale images and edge detection techniques to detect hand gestures. The drawback of this technique is that there are some limitations when it comes to grayscale images as they are only 2D data and it is difficult to extract the key features of the hand by Sammon Babu [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]\u003c/p\u003e \u003cp\u003eHe introduced a technique to recognize hand gestures using a leap motion controller which is quite expensive and requires additional hardware by Sundar Baglamas T [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]\u003c/p\u003e \u003cp\u003eIt is a technique that uses gloves and several hand sensors to recognize different hand gestures. The sensors have motion detectors that are costly, and it makes the hand hard to move because of the weight by Sai Bharath Padigala Gogineni Hrushikesh MadhavSaranu Kishore Kumar [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]\u003c/p\u003e \u003cp\u003eIt is discussed a technique using deep learning to recognize the ASL gestures with the help of a public dataset available on Kaggle. It employs several ML algorithms in the dataset and gives a classified report on the results for each ML algorithm by Sakshi Mankar Kanishka Mohapatra Mansi Talavadekar [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIt is a technique that uses color features and contour extraction to determine rather than the different hand gestures by Mallikarjun Rao, Cheguri SowmyaP.A. Sujasri [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]\u003c/p\u003e \u003cp\u003eIt is an approach for Realtime hand gesture recognition for human computer interaction using CNN. The drawback here is that it does not detect any sign language gestures rather it detects some hand signs that are used for Human computer Interface rather than sign language by Pei Xu [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIt is technique Recognition of Dynamic Hand Gestures from 3D Motion Data using LSTM and CNN architectures. Published in 2017 by Chinmaya R. Naguri, Razvan C. Bunescu.[\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIt is technique Hand Gesture Tracking and Recognition based Human Computer Interaction System and Its Applications published in 2018 by Chinmaya R. Naguri, Razvan C. Bunescu [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eA Method for Stochastic Optimization. Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015 by Diederik P. Kingma, Jimmy Ba. Adam [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe Linguistics of American Sign Language, American National Standard for Information Sciences.by Clayton Valli, Ceil Lucas [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIt is Hand Gesture Recognition for Human Computer Interaction. 7th International Conference on Advances in Computing Communications, ICACC- 2017, 22\u0026ndash;24 August 2017, Cochin, India by Aashni Hariaa, Archanasri Subramaniana, Nivedhitha Asokkumara, Shristi Poddara, Jyothi S Nayaka,[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]\u003c/p\u003e \u003cp\u003eIt is Hand Gesture Detection for American Sign Language using K- Nearest Neighbor with Media pipe by Arsheldy Alvin1, Nabila Husna Shabrina2, Aurelius Ryo3, Edgar Christian4 Fakultas Teknik dan Informatika, Universitas Multimedia Nusan- Tara, Teknik Komputer Tangerang [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]\u003c/p\u003e \u003cp\u003eA Real- time Hand Gesture Recognition and Human- Computer Interaction System by Pei Xu Department of Electrical and Computer Engineering, University of Minnesota, Twin Cities [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]\u003c/p\u003e \u003cp\u003eA Google Re- search 1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA by Fan Zhang ,Valentin Bazarevsky ,Andrey Vakunov ,Andrei Tkachenka ,George Sung, Chuo Ling Chang,Matthias Grundmann [14]\u003c/p\u003e \u003cp\u003eHand gesture recognition using machine learning algorithms. Computer Science and Information Technologies by Abhishek B, Kanya Krishi, Meghana M, Mohammed Daaniyaal, Anupama HS, BMS Institute of Technology, Bangalore, India.[\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e15\u003c/span\u003e]\u003c/p\u003e \u003cp\u003eThe Hand gesture recognition is based on convolution neural network. part of Springer Nature 2017 by Gongfa Li, \u0026middot; Heng Tang, \u0026middot; Ying Sun, \u0026middot; Jianyi Kong, \u0026middot; Guozhang Jiang, \u0026middot; D u Jiang \u0026middot; Bo Tao, \u0026middot;Shuang Xu, \u0026middot; Honghai Liu.[\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e16\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe Dynamic Hand Gesture Recognition Using 3DCNN and LSTM with FSM Context-Aware Model by Noorkholis Luthfil Hakim, Timothy K. Shih, Sandeli Priyanwada Kasthuri Arachchi ,Wisnu Aditya, Yi-Cheng Chen and Chih-Yang Lin [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e17\u003c/span\u003e].\u003c/p\u003e"},{"header":"3. Proposed System","content":"\u003cp\u003eThe proposed system for the ASL alphabet recognition is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e. The proposed system recognizes the American sign language of alphabets. It recognizes gestures with motion and gestures without motion. The first step is to collect the dataset for the system which will be discussed in the results and discussion section. Media Pipe hands is a high- fidelity hand and finger tracking framework used in dataset preparation. It makes use of a number of ML models to infer the 21 three dimensional landmarks of a hand in real time from just one frame.\u003c/p\u003e \u003cp\u003eMedia pipe\u003c/p\u003e \u003cp\u003eMedia pipe hand key point detection works with a machine learning pipeline. A palm detection model that works on the entire image is one of the models in the Media Pipe machine learning pipeline. It receives the entire image and outputs a hand-bounding box that is orientated. a hand landmark model that uses the palm.Detector cropped image region and outputs highly accurate 3D hand key points [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Additionally, the hand is cropped by the ML pipeline using the hand landmarks found in the previous frame, and palm detection is only used to localize the hand when the land- mark model is unable to do so. Figure\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e shows the 21hand landmarks that can be tracked from the Media pipe's hand landmark detector. Arsheldy Alvin, etal [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e] have used Media pipe and K nearest neighbors algorithm to determine the hand gestures which helps us in understanding how to use media pipes for hand gesture recognition\u003c/p\u003e \u003cp\u003eThe difference between tensor flow and media pipe is that tensor flow is a free and open-source software library for machine learning and artificial intelligence. It can be used across a range of tasks but has a particular focus on training and inference of deep neural networks. Whereas Media pipe offers open-source cross-platform, customizable ML solutions for live and streaming media. In this paper we have used media pipe for capturing the hand key points and tensor flow for training and detecting the ML algorithm. Media pipe runs with both on CPU and GPU. No additional processing power is required to run Media pipe.\u003c/p\u003e \u003cp\u003eSign Language Recognition Using LSTM\u003c/p\u003e \u003cp\u003eThe next step is to train the LSTM model using the dataset. LSTM is one of the feed- forward neural networks. Since, in American alphabet hand gestures there are several hand gestures that have hand motion in them instead of a\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eSingle static pose. So, it's better to use LSTM using which we can recognize the continuous motion. The advantage of using LSTM is that it allows not only single data points i.e. images, but it also allows the entire sequences of data [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. An artificial neural network is a network of interconnected neurons, each of which is connected to one another and is modelled after the brain\u0026rsquo;s organic neural networks. We can execute sophisticated operations on the data by combining a number of different algorithms, each of which is capable of completing a different task [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. A class of neural networks called Recurring Neural Networks, or RNNs, is designed specifically to handle temporal data. The input is processed using the internal state of the RNN neurons, which is accomplished with the aid of loops built into the neural network. RNN neurons have a cell state or memory.\u003c/p\u003e \u003cp\u003eFrom the training dataset, Spatiotemporal features are extracted, and the LSTM model classification model is derived. The generated ML model is stored for prediction of gestures in real time. All the 26 alphabet gestures are collected one after the other and are preprocessed into NumPy arrays and stored. The dataset collected is then passed onto the LSTM model. The LSTM model will have all the 26 alphabet classes for each of the alphabets. The algorithm is made to run on 2000 epochs until the accuracy becomes closer to 1 and training loss goes to nearly 0.\u003c/p\u003e \u003cp\u003eThe algorithm runs with the above specified features and generates the model which is stored for future use. The model can now be used for tracking the live hand gestures and for further operations. The system uses RELU activation. The rectified linear activation function, also known as Re LU, is a piecewise linear function that, if the input is positive, outputs the input directly; if not, it outputs zero. Because a model that utilizes it is simpler to train and frequently performs better, it has evolved into the default activation function for many types of neutral networks [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e20\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe system uses Adam Optimizer technique. In place of the conventional stochastic gradient descent method, Adam is an optimization technique that may be used to repeatedly update network weights depending on training data [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. For all weight updates, stochastic gradient descent maintains a constant learning rate (referred to alpha),which does not change throughout training. As learning progresses, the learning rate is maintained and independently adjusted for each network weight (parameter). The main reason for using the LSTM model is that we can detect motion from hand gestures, and it can be used to detect hand gestures that have motion in them.\u003c/p\u003e \u003cp\u003eIn Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, we can see the diagram of LSTM cell. The LSTM cell has three steps namely the forget gate, input gate and output gate. The forget gate establishes how much the prior data should be forgotten. The amount of information to be printed on the internal cell is determined by the input gate. The output gate decides what output to produce based on the internal cell state at the moment. Electric energy consumption modelling using LSTM has been studied in. LSTM based natural language processing model has been used for Malayalam language analysis and paraphrase identification.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSummary of the sequential model\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLayer\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOutput Shape\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eParam\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eConv1d(Conv1D)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None,None,60)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e360\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003elstm(LSTM)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None,None,60)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e29040\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003elstm1(Lstm)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None,60)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e29040\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003edense(Dense)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None,30)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e1830\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003edense_`1(Dense)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None,10)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e310\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003edense_2(Dense)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None,1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003elamda(Lambda)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(None,1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e"},{"header":"4. EXPERIMENTAL RESULTS","content":"\u003cp\u003e \u003cem\u003eDataset\u003c/em\u003e \u003c/p\u003e \u003cp\u003eS. The dataset required for the experimental study of the proposed algorithm is created using Media Pipe which is an opensource library. It captures the hand gestures for 30 frames and stores them as NumPy array for training the model. The dataset has 30 different permutations of a single alphabet. The same process is repeated with four different demographs of humans to get a diverse dataset. The four different humans are two male hands, one female hand, and one kid\u0026rsquo;s hand. This makes a total of 93,600 NumPy array, these are indexed and stored. The dataset has different parameters that are x ,y and z axes of the hand joints. The dataset is divided into 90% for training and10% fortesting..\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e"},{"header":"5. CONCLUSION","content":"\u003cp\u003eThe proposed vision based system can be used to recognize the ASL alphabet and convert them into text. Media Pipe hand landmark features and LSTM model was found to be effective for gesture recognition. Accuracy, Micro Average and Weighted Average of the alphabets are 0.99 for the custom dataset, this corroborates that the proposed system can be effectively used for HCI. The system can be used in HCI like interacting with the computer based on the gesture performed. Furthermore, an example could be to use the alphabets detected on a word document to type the alphabets instead of a keyboard or for typing on the search engine. This work can be further extended work with real time applications and multiple letters, or word recognition can be done to control or interact with the applications.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eDeclarations\u003c/h2\u003e \u003cp\u003eThe authors declare no conflict of interest and affirm that this work has not been published elsewhere.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eIdentifying the research objectives and designing the study approachDeveloping the algorithms, models, and experimental setup for sign language recognition.Gathering sign language datasets and ensuring their quality and relevance.Writing the code and software necessary for sign language recognition.Testing and validating the accuracy and effectiveness of the sign language recognition system\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eData Availability Statment:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe dataset comprising ASL hand gesture images encompasses 2425 samples captured from 5 individuals. Notably, their study exhibits a limitation wherein the ASL images undergo rotation and processing through open-source tools like ImageMagick.\u0026nbsp;\u0026nbsp;\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eA. A. Abdulhussein and F. A. Raheem, \u0026ldquo;Hand gesture recognition of static letters American sign language (ASL) using deep Learning,\u0026rdquo; Engineering and Technology Journal, Vol. 38, No.06 pp. 926\u0026ndash;937, 2020.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTeak Wei Chong and BoonGiin Lee, American Sign Language Recognition Using Leap Motion Controller with Machine Learning Approach, Department of Electronic Engineering, Keimyung University, Daegu 42601, Korea, October 2018.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMatteo Rinalduzzi, Alessio De Angelis, Francesco Santoni, Emanuele Buchicchio, Antonio Moschitta, Paolo Carbone, Paolo Bellitti, Mauro Serpelloni, Gesture Recognition of Sign Language Alphabet Using a Magnetic Positioning System, June 2021.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRajesh George Rajan, Dr.M.Judith Leo, American Sign Language Alphabets Recognition using Hand Crafted and Deep Learning Features IEEE Xplore Part Number: CFP20F70-ART published on 2020.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSourav Bhowmick, Sushant Kumar and Anurag Kumar, Hand Gesture Recognition of English Alphabets using Artificial Neural Network published on 2015.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRoss E. Mitchell, Gallaudet Research Institute, Draft manuscript accepted for publication in Sign Language Studies, Volume 6, Number 3, 2006.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOnamon Pinsanoh, Yuttana Kitjaidure, Ariya Thongtawee. A Novel Feature Extraction for American Sign Language Recognition Using Web- cam. Published in 2018.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChinmaya R. Naguri, Razvan C. Bunescu.Recognition of Dynamic Hand Gestures from 3D Motion Data using LSTM and CNN architectures. Published in 2017.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKai Li, Qieshi Zhang, Jun Cheng, Jianming Liu. Hand Gesture Tracking and Recognition based Human Computer Interaction System and Its Applications published on 2018.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDiederik P. Kingma, Jimmy Ba. Adam: A Method for Stochastic Optimization. Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eClayton Valli, Ceil Lucas, Linguistics Of American Sign Language, American National Standard for Information Sciences.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAashni Hariaa, Archanasri Subramaniana, Nivedhitha Asokkumara, Shristi Poddara, Jyothi S Nayaka, Hand Gesture Recognition for Human Computer Interaction. 7th International Conference on Advances in Computing Communications, ICACC- 2017, 22\u0026ndash;24 August 2017, Cochin, India.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eArsheldy Alvin1, Nabila Husna Shabrina2, Aurelius Ryo3, Edgar Christian4 Fakultas Teknik dan Informatika, Universitas Multimedia Nusan- Tara, Teknik Komputer Tangerang, Indonesia. Hand Gesture Detection for American Sign Language using K- Nearest Neighbor with Media pipe .[14]. Pei Xu, Department of Electrical and Computer Engineering, University of Minnesota, Twin Cities. A Real- time Hand Gesture Recognition and Human- Computer Interaction System.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFan Zhang,Valentin Bazarevsky ,Andrey Vakunov ,Andrei Tkachenka ,George Sung, Chuo Ling Chang,Matthias Grundmann. Google Re- search 1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbhishek B, Kanya Krishi, Meghana M, Mohammed Daaniyaal, Anupama H S, BMS Institute of Technology, Bangalore, India. Hand gesture recognition using machine learning algorithms. Computer Science and Information Technologies.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGongfa Li, \u0026middot; Heng Tang, \u0026middot; Ying Sun, \u0026middot; Jianyi Kong, \u0026middot; Guozhang Jiang, \u0026middot; D u Jiang \u0026middot; Bo Tao, \u0026middot;Shuang Xu, \u0026middot; Honghai Liu. Hand gesture recognition based on convolution neural network. part of Springer Nature 2017.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNoorkholis Luthfil Hakim, Timothy K. Shih, Sandeli Priyanwada Kasthuri Arachchi ,Wisnu Aditya, Yi-Cheng Chen and Chih-Yang Lin. Dynamic Hand Gesture Recognition Using 3DCNN and LSTM with FSM Context-Aware Model.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSepp Hochreiter,Fakultat fur Informatik, Technische Universitat Munchen, 80290 Munchen, Germany. LONG SHORT-TERM MEMORY, Neural Computation 9(8):1735\u0026ndash;1780, 1997.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDeep Learning in Neural Networks: An Overview.Jurgen Schmidhuber,The Swiss AI Lab IDSIA,Istituto Dalle Molle di Studi sull\u0026rsquo;Intelligenza Artificiale,University of Lugano SUPSI,Galleria 2, 6928 Manno-Lugano,Switzerland,8 October 2014.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"discover-artificial-intelligence","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"diai","sideBox":"Learn more about [Discover Artificial Intelligence](https://www.springer.com/44163)","snPcode":"","submissionUrl":"","title":"Discover Artificial Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Hand Gesture Recognition, American Sign Language (ASL), Media pipe, Long Short-Term Memory (LSTM), Human-Computer Interface","lastPublishedDoi":"10.21203/rs.3.rs-4323367/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4323367/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eWith the advancement of today\u0026rsquo;s technologies in artificial intelligence, humans tend to use hand gestures in their communication to convey their ideas. Gesture recognition is an active area of research in the human-computer interface (HCI). Gesture recognition is important for communication between deaf-mute people, HCI, robot control, home automation, and medical applications. In this article, a simple and efficient vision-based approach for American Sign Language (ASL) alphabets recognition has been discussed to recognize both static and dynamic gestures. Media pipe introduced by Google had been used to get hand landmarks and a custom data set has been created and used for the experimental study. Hand gesture recognition has been done by using Long short-term memory (LSTM). The proposed system has been investigated with 26 alphabets and an accuracy of 99% has been achieved. This work can be used to convert hand gestures into text.\u003c/p\u003e","manuscriptTitle":"Enhancing American Sign Language Alphabet Recognition: A Fusion of Media Pipe and LSTM Technologies","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-07-09 17:34:30","doi":"10.21203/rs.3.rs-4323367/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorAssigned","content":"","date":"2024-06-21T09:31:30+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-06-20T06:47:46+00:00","index":"","fulltext":""},{"type":"submitted","content":"Discover Artificial Intelligence","date":"2024-04-25T10:03:02+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"discover-artificial-intelligence","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"diai","sideBox":"Learn more about [Discover Artificial Intelligence](https://www.springer.com/44163)","snPcode":"","submissionUrl":"","title":"Discover Artificial Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"637523b6-c6a0-4416-8af8-614e2e964464","owner":[],"postedDate":"July 9th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2024-07-09T17:34:30+00:00","versionOfRecord":[],"versionCreatedAt":"2024-07-09 17:34:30","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4323367","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4323367","identity":"rs-4323367","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}