Occlusion Robust Sign Language Recognition System for Indian Sign Language Using CNN and Pose Features

Occlusion Robust Sign Language Recognition System for Indian Sign Language Using CNN and Pose Features | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Occlusion Robust Sign Language Recognition System for Indian Sign Language Using CNN and Pose Features SOUMEN DAS, Saroj kr. Biswas, Biswajit Purkayastha This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-2801772/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The Sign Language Recognition System (SLRS) is a cutting-edge technology that aims to enhance communication accessibility for the deaf community in India by replacing the traditional approach of using human interpreters. However, the existing SLRS for Indian Sign Language (ISL) do not focus on some major problems including occlusion, similar hand gesture, multi viewing angle problem and inefficiency due to extracting features from a large sequence of frame that contains redundant and unnecessary information. Therefore, in this research paper an occlusion robust SLRS named Multi Featured Deep Network (MF-DNet) is proposed for recognizing ISL words. The suggested MF-DNet uses a histogram difference based keyframe selection technique to remove redundant frames. To resolve occlusion, similar hand gesture, and multi viewing angle problem the suggested MF-DNet incorporates pose features with Convolution Neural Network (CNN) features. For classification the proposed system uses Bi Directional Long Shor Term Memory (BiLSTM) network, which is compared with different classifier such as LSTM, ConvLSTM and stacked LSTM networks. The proposed SLRS achieved an average classification accuracy of 96.88% on the ISL dataset and 99.06% on the benchmark LSA64 dataset. The results obtained from the MF-DNet is compared with some of the existing SLRS where the proposed method outperformed the existing methods. Sign Language Recognition System Indian Sign Language Pose estimation Deep Learning Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-2801772","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":193007959,"identity":"097073f0-8eba-44ed-9553-a5dc6c82843e","order_by":0,"name":"SOUMEN DAS","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA8UlEQVRIiWNgGAWjYFAD9gYwxQPlShChhecAhOIhXotEAgOKNTiBfHuP6YaPOxjy5Wc+fvzhA8MdGXv+A4wffjBY5OHSYnDmjNnNmWcYLDfcTjOTnMHwjIdHIoFZsodBohinFokcs9u8bQwGBtI5bMw8DIeBWhgYpIHuTGzA5bAZQC1/gVrkZ55h/vwHpIX/APNvfFoYbgC1MAK1MNzgARkO1MKQwIbXFoMzx8pu9oIcdgbolx4DoJYbiW2WPQZ4HNbevO3GT5DD2g8//vCj4rA9e//hwzd+VNThdhgE/IdZCiIYG6CMUTAKRsEoGAXkAgChcEz9JvG6SQAAAABJRU5ErkJggg==","orcid":"","institution":"National Institute Of Technology Silchar","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"SOUMEN","middleName":"","lastName":"DAS","suffix":""},{"id":193007960,"identity":"80ce1366-9ec5-432a-8028-4c45f41f7370","order_by":1,"name":"Saroj kr. Biswas","email":"","orcid":"","institution":"National Institute Of Technology Silchar","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Saroj","middleName":"kr.","lastName":"Biswas","suffix":""},{"id":193007961,"identity":"223500ae-71dd-47ec-9065-edd7319b59c6","order_by":2,"name":"Biswajit Purkayastha","email":"","orcid":"","institution":"National Institute Of Technology Silchar","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Biswajit","middleName":"","lastName":"Purkayastha","suffix":""}],"badges":[],"createdAt":"2023-04-11 14:29:18","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-2801772/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-2801772/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":36530091,"identity":"54690b4e-d64d-4d79-996a-d2ca6d05301a","added_by":"auto","created_at":"2023-05-02 13:59:33","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":403371,"visible":true,"origin":"","legend":"","description":"","filename":"poseSLRS.pdf","url":"https://assets-eu.researchsquare.com/files/rs-2801772/v1_covered_c1a0b699-451a-4489-a0cb-baffe066ed81.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Occlusion Robust Sign Language Recognition System for Indian Sign Language Using CNN and Pose Features","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Sign Language Recognition System, Indian Sign Language, Pose estimation, Deep Learning","lastPublishedDoi":"10.21203/rs.3.rs-2801772/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-2801772/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe Sign Language Recognition System (SLRS) is a cutting-edge technology that aims to enhance communication accessibility for the deaf community in India by replacing the traditional approach of using human interpreters. However, the existing SLRS for Indian Sign Language (ISL) do not focus on some major problems including occlusion, similar hand gesture, multi viewing angle problem and inefficiency due to extracting features from a large sequence of frame that contains redundant and unnecessary information. Therefore, in this research paper an occlusion robust SLRS named Multi Featured Deep Network (MF-DNet) is proposed for recognizing ISL words. The suggested MF-DNet uses a histogram difference based keyframe selection technique to remove redundant frames. To resolve occlusion, similar hand gesture, and multi viewing angle problem the suggested MF-DNet incorporates pose features with Convolution Neural Network (CNN) features. For classification the proposed system uses Bi Directional Long Shor Term Memory (BiLSTM) network, which is compared with different classifier such as LSTM, ConvLSTM and stacked LSTM networks. The proposed SLRS achieved an average classification accuracy of 96.88% on the ISL dataset and 99.06% on the benchmark LSA64 dataset. The results obtained from the MF-DNet is compared with some of the existing SLRS where the proposed method outperformed the existing methods.\u003c/p\u003e","manuscriptTitle":"Occlusion Robust Sign Language Recognition System for Indian Sign Language Using CNN and Pose Features","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2023-04-20 09:51:58","doi":"10.21203/rs.3.rs-2801772/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"b192a1eb-d860-4d20-9b04-45ae7788355c","owner":[],"postedDate":"April 20th, 2023","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2023-05-02T13:59:17+00:00","versionOfRecord":[],"versionCreatedAt":"2023-04-20 09:51:58","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-2801772","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-2801772","identity":"rs-2801772","version":["v1"]},"buildId":"FbvkV6FR0MCFSLy54lSbu","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}