A Transparent Transformer Framework for Mid-Resource English-Hindi Neural Machine Translation: Tokenization Efficiency, Interpretability, and Uncertainty Analysis

A Transparent Transformer Framework for Mid-Resource English-Hindi Neural Machine Translation: Tokenization Efficiency, Interpretability, and Uncertainty Analysis | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Transparent Transformer Framework for Mid-Resource English-Hindi Neural Machine Translation: Tokenization Efficiency, Interpretability, and Uncertainty Analysis Yash Mishra, Nishaant Singh, Arun Kumar Rai, Saurabh Kumar, Aditya Kumar, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9327858/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The Neural Machine Translation (NMT) systems used for mid-resource languages suffer from unique algorithmic and linguistic challenges which are usually hidden because of the opaque nature of pre-trained Large Language Models (LLMs). Though the generally used multilingual frameworks possess strong translation capabilities, at the same time their ``black-box'' architecture does not allow flexible translations across different language pairs. Hence, this research presents an in-depth study of a transparent, from-scratch Transformer framework designed for English-Hindi translation. It is based on the concepts of tokenization and sequence management protocols by focusing on morphological differences between English and Hindi. Instead of using the legacy WordLevel tokenization, we transitioned to a Byte Pair Encoding (BPE) vocabulary of 5,000 tokens. This transition has helped to resolve the Out-of-Vocabulary (OOV) failures and decrease the statistical class imbalances within the autoregressive decoder's SoftMax classifier. Training is performed on 1.6 million sentence IIT Bombay parallel corpus along with sequence-length filtering and serverless GPU infrastructure for computational stability. The main contribution of this research is the detailed analysis of the model which includes tokenization efficiency, sequence-length degradation and prediction entropy. The results are the proof that BPE Tokenization is a better tokenization technique (41.7 tokens per sentence) than WordLevel Tokenization (27.2 tokens per sentence). It also improves the morphological reconstruction and semantic preservation. Also, the layer-wise attention interpretability shows that the semantic consolidation improves diagonally as we move towards the network's depth. Neural Machine Translation Transformer Architecture Byte Pair Encoding Attention Interpretability Prediction Entropy Mid-Resource Languages English-Hindi Translation Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9327858","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":620283798,"identity":"f55ead94-f015-42d4-888d-b7f9da55d20c","order_by":0,"name":"Yash Mishra","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABDElEQVRIiWNgGAWjYHACNgYGA4YEBgbGBoOEChugAGPjASK1MDcUfDiTBtLSQIQWBpAW9oaPM9sOg4XwajE4fvzZoxsFdXnm7QcbN/OcOW+3tv0w0JYam2icWs7kmBvnGBwuljmT2GzMU3E7eduZRKCWY2m5Dbi0HMhhk84xOJA4gyGxzZjnzO1kswNALYwNh3FrOf/8GVBLXeIM/oftv3nbziWbnX9IQMuNBDOgFubEGRKJDYYz2w7Ymd0gYIvkjTcgLYeBWh42GHw4k5xgdgNoSwIev/CdTwc67A/IYekPgFFpZ292Pv3hgw81Nji1KBxAE0gEq0zAoRwE5NHNssejeBSMglEwCkYoAACccm0mo3ENsAAAAABJRU5ErkJggg==","orcid":"","institution":"Dr. A.P.J. Abdul Kalam Technical University","correspondingAuthor":true,"prefix":"","firstName":"Yash","middleName":"","lastName":"Mishra","suffix":""},{"id":620283799,"identity":"344f97d5-0537-4951-adad-86eee772e0fa","order_by":1,"name":"Nishaant Singh","email":"","orcid":"","institution":"Dr. A.P.J. Abdul Kalam Technical University","correspondingAuthor":false,"prefix":"","firstName":"Nishaant","middleName":"","lastName":"Singh","suffix":""},{"id":620283800,"identity":"95fd67fd-2372-4377-ac95-034dc0ca2f7d","order_by":2,"name":"Arun Kumar Rai","email":"","orcid":"","institution":"Dr. A.P.J. Abdul Kalam Technical University","correspondingAuthor":false,"prefix":"","firstName":"Arun","middleName":"Kumar","lastName":"Rai","suffix":""},{"id":620283801,"identity":"57f5e644-2960-4e5f-801a-b3dab74f2c11","order_by":3,"name":"Saurabh Kumar","email":"","orcid":"","institution":"Dr. A.P.J. Abdul Kalam Technical University","correspondingAuthor":false,"prefix":"","firstName":"Saurabh","middleName":"","lastName":"Kumar","suffix":""},{"id":620283802,"identity":"844456e0-51c8-4d3b-b51b-f9765a6a6b26","order_by":4,"name":"Aditya Kumar","email":"","orcid":"","institution":"Dr. A.P.J. Abdul Kalam Technical University","correspondingAuthor":false,"prefix":"","firstName":"Aditya","middleName":"","lastName":"Kumar","suffix":""},{"id":620283803,"identity":"a83bed74-8b0c-41a8-8c4c-d6956e6dfb32","order_by":5,"name":"Amrish Dubey","email":"","orcid":"","institution":"Dr. A.P.J. Abdul Kalam Technical University","correspondingAuthor":false,"prefix":"","firstName":"Amrish","middleName":"","lastName":"Dubey","suffix":""}],"badges":[],"createdAt":"2026-04-05 17:53:07","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9327858/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9327858/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":107706685,"identity":"65ce487f-b8a3-4f91-9399-b26f8ada61c7","added_by":"auto","created_at":"2026-04-24 09:18:31","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":15367390,"visible":true,"origin":"","legend":"","description":"","filename":"ATransparentTransformerFrameworkforMidResourceEnglishHindiNeuralMachineTranslation.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9327858/v1_covered_1a7ec091-b949-4bc1-8344-f8bdf556c073.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A Transparent Transformer Framework for Mid-Resource English-Hindi Neural Machine Translation: Tokenization Efficiency, Interpretability, and Uncertainty Analysis","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Neural Machine Translation, Transformer Architecture, Byte Pair Encoding, Attention Interpretability, Prediction Entropy, Mid-Resource Languages, English-Hindi Translation","lastPublishedDoi":"10.21203/rs.3.rs-9327858/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9327858/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe Neural Machine Translation (NMT) systems used for mid-resource languages suffer from unique algorithmic and linguistic challenges which are usually hidden because of the opaque nature of pre-trained Large Language Models (LLMs). Though the generally used multilingual frameworks possess strong translation capabilities, at the same time their ``black-box'' architecture does not allow flexible translations across different language pairs. Hence, this research presents an in-depth study of a transparent, from-scratch Transformer framework designed for English-Hindi translation. It is based on the concepts of tokenization and sequence management protocols by focusing on morphological differences between English and Hindi. Instead of using the legacy WordLevel tokenization, we transitioned to a Byte Pair Encoding (BPE) vocabulary of 5,000 tokens. This transition has helped to resolve the Out-of-Vocabulary (OOV) failures and decrease the statistical class imbalances within the autoregressive decoder's SoftMax classifier. Training is performed on 1.6 million sentence IIT Bombay parallel corpus along with sequence-length filtering and serverless GPU infrastructure for computational stability. The main contribution of this research is the detailed analysis of the model which includes tokenization efficiency, sequence-length degradation and prediction entropy. The results are the proof that BPE Tokenization is a better tokenization technique (41.7 tokens per sentence) than WordLevel Tokenization (27.2 tokens per sentence). It also improves the morphological reconstruction and semantic preservation. Also, the layer-wise attention interpretability shows that the semantic consolidation improves diagonally as we move towards the network's depth.\u003c/p\u003e","manuscriptTitle":"A Transparent Transformer Framework for Mid-Resource English-Hindi Neural Machine Translation: Tokenization Efficiency, Interpretability, and Uncertainty Analysis","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-24 05:03:06","doi":"10.21203/rs.3.rs-9327858/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"f7ddc21d-b60d-4426-ab25-5dda99d9cbd7","owner":[],"postedDate":"April 24th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-04-24T05:03:06+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-24 05:03:06","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9327858","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9327858","identity":"rs-9327858","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}