Tokens with Meaning: A Hybrid Tokenization Approach for NLP

Tokens with Meaning: A Hybrid Tokenization Approach for NLP | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Tokens with Meaning: A Hybrid Tokenization Approach for NLP M. Ali Bayram, Ali Arda Fincan, Ahmet Semih Gümüş, Sercan Karakaş, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6513777/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Tokenization plays a pivotal role in natural language processing (NLP), shaping how textual data is segmented, interpreted, and processed by language models. Despite the success of subword-based tokenization techniques such as Byte Pair Encoding (BPE) and WordPiece, these methods often fall short in morphologically rich and agglutinative languages due to their reliance on statistical frequency rather than linguistic structure. This paper introduces a linguistically informed hybrid tokenization framework that integrates rule-based morphological analysis with statistical subword segmentation to address these limitations. The proposed approach leverages phonological normalization, root-affix dictionaries, and a novel tokenization algorithm that balances morpheme preservation with vocabulary efficiency. It assigns shared identifiers to phonologically variant affixes (e.g., \textit{-ler} and \textit{-lar}) and phonologically altered root forms (e.g., \textit{kitap} vs.\textit{kitabı}), significantly reducing redundancy while maintaining semantic integrity. The framework also incorporates special tokens for whitespace and orthographic case, including an \texttt{} token to prevent vocabulary inflation from capitalization. Byte Pair Encoding is integrated to support out-of-vocabulary coverage without compromising morphological coherence. Evaluation on the TR-MMLU benchmark—a large-scale, Turkish-specific NLP benchmark—demonstrates that the proposed tokenizer achieves the highest Turkish Token Percentage (90.29%) and Pure Token Percentage (85.8%) among all tested models. Comparative analysis against widely used tokenizers from models such as LLaMA, Gemma, and OpenAI's GPT reveals that the proposed method yields more linguistically meaningful and semantically coherent tokens. A qualitative case study further illustrates improved morpheme segmentation and interpretability in complex Turkish sentences. Although the implementation focuses on Turkish, the underlying methodology is language-independent and adaptable to other languages. This work contributes to ongoing efforts to improve tokenizer design through linguistic alignment, offering a practical and extensible solution for enhancing both interpretability and performance in multilingual NLP systems. Tokenization Morphologically Rich Languages Morphological Segmentation Byte Pair Encoding Turkish NLP Linguistic Integrity Low-Resource Languages Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6513777","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":496027109,"identity":"5511644e-036e-436c-98f1-aaddadacf1a6","order_by":0,"name":"M. Ali Bayram","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA90lEQVRIiWNgGAWjYFAC5obDUMaBAx+AFBs7QS2MMC1siQ9ngChmIrRA1fAYG/OAbSOgQbf9YOPhgprD+fzTDphJ2/zaJs/HzMD44WMObi1mZxIbDs84dthyxu2ENOncvtuGbcwMzJIzt+HRcgCohYftsAHD7YRj0rk9txmBWtiYefFpOf8QqOXfYQP524lt0pY9t+0Ja7kBtIW37bCBwe1kZmOGH0CNhLUAbZnZl25geDuN8WFvw+3kNmbGZvx+OZ98+HPBN2sDudv5Hw78+HPbdn5788EPH/FogYJmCMXYBiYbCKoHgjoo/YcYxaNgFIyCUTDSAACnHln8bfrtrAAAAABJRU5ErkJggg==","orcid":"","institution":"Yıldız Technical University","correspondingAuthor":true,"prefix":"","firstName":"M.","middleName":"Ali","lastName":"Bayram","suffix":""},{"id":496027110,"identity":"c72bf861-8efe-42d8-8619-39a1e992dce0","order_by":1,"name":"Ali Arda Fincan","email":"","orcid":"","institution":"Yeditepe University","correspondingAuthor":false,"prefix":"","firstName":"Ali","middleName":"Arda","lastName":"Fincan","suffix":""},{"id":496027111,"identity":"3e2ec106-44b8-490d-9bdf-fc6865a9fa89","order_by":2,"name":"Ahmet Semih Gümüş","email":"","orcid":"","institution":"Yeditepe University","correspondingAuthor":false,"prefix":"","firstName":"Ahmet","middleName":"Semih","lastName":"Gümüş","suffix":""},{"id":496027112,"identity":"965d7150-4dbe-44e4-8fd2-6d1ddd1db1c0","order_by":3,"name":"Sercan Karakaş","email":"","orcid":"","institution":"University of Chicago","correspondingAuthor":false,"prefix":"","firstName":"Sercan","middleName":"","lastName":"Karakaş","suffix":""},{"id":496027113,"identity":"579a75c2-3a71-4e92-b2a3-787cf8728812","order_by":4,"name":"Banu Diri","email":"","orcid":"","institution":"Yıldız Technical University","correspondingAuthor":false,"prefix":"","firstName":"Banu","middleName":"","lastName":"Diri","suffix":""},{"id":496027114,"identity":"dc59ffe4-a1dc-45da-8d4a-755efb0bbbd7","order_by":5,"name":"Savaş Yıldırım","email":"","orcid":"","institution":"Istanbul Bilgi University","correspondingAuthor":false,"prefix":"","firstName":"Savaş","middleName":"","lastName":"Yıldırım","suffix":""},{"id":496027115,"identity":"5c847cdb-a62e-4b27-b20e-10731d3ce27a","order_by":6,"name":"Demircan Çelik","email":"","orcid":"","institution":"Yeditepe University","correspondingAuthor":false,"prefix":"","firstName":"Demircan","middleName":"","lastName":"Çelik","suffix":""}],"badges":[],"createdAt":"2025-04-23 14:53:23","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6513777/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6513777/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":88391124,"identity":"b3aac76e-cd9d-4b6f-8cd6-fdec4e2670ad","added_by":"auto","created_at":"2025-08-06 04:28:50","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":365482,"visible":true,"origin":"","legend":"","description":"","filename":"TokenswithMeaningAHybridTokenizationApproachforNLP1.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6513777/v1_covered_dc40cb4d-adea-4f01-b3f5-49e41ef60d7c.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Tokens with Meaning: A Hybrid Tokenization Approach for NLP","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Tokenization, Morphologically Rich Languages, Morphological Segmentation, Byte Pair Encoding, Turkish NLP, Linguistic Integrity, Low-Resource Languages","lastPublishedDoi":"10.21203/rs.3.rs-6513777/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6513777/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Tokenization plays a pivotal role in natural language processing (NLP), shaping how textual data is segmented, interpreted, and processed by language models. Despite the success of subword-based tokenization techniques such as Byte Pair Encoding (BPE) and WordPiece, these methods often fall short in morphologically rich and agglutinative languages due to their reliance on statistical frequency rather than linguistic structure. This paper introduces a linguistically informed hybrid tokenization framework that integrates rule-based morphological analysis with statistical subword segmentation to address these limitations. The proposed approach leverages phonological normalization, root-affix dictionaries, and a novel tokenization algorithm that balances morpheme preservation with vocabulary efficiency. It assigns shared identifiers to phonologically variant affixes (e.g., \\textit{-ler} and \\textit{-lar}) and phonologically altered root forms (e.g., \\textit{kitap} vs.\\\\textit{kitabı}), significantly reducing redundancy while maintaining semantic integrity. The framework also incorporates special tokens for whitespace and orthographic case, including an \\texttt{} token to prevent vocabulary inflation from capitalization. Byte Pair Encoding is integrated to support out-of-vocabulary coverage without compromising morphological coherence. Evaluation on the TR-MMLU benchmark—a large-scale, Turkish-specific NLP benchmark—demonstrates that the proposed tokenizer achieves the highest Turkish Token Percentage (90.29\\%) and Pure Token Percentage (85.8\\%) among all tested models. Comparative analysis against widely used tokenizers from models such as LLaMA, Gemma, and OpenAI's GPT reveals that the proposed method yields more linguistically meaningful and semantically coherent tokens. A qualitative case study further illustrates improved morpheme segmentation and interpretability in complex Turkish sentences. Although the implementation focuses on Turkish, the underlying methodology is language-independent and adaptable to other languages. This work contributes to ongoing efforts to improve tokenizer design through linguistic alignment, offering a practical and extensible solution for enhancing both interpretability and performance in multilingual NLP systems.","manuscriptTitle":"Tokens with Meaning: A Hybrid Tokenization Approach for NLP","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-08-06 04:20:45","doi":"10.21203/rs.3.rs-6513777/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"01866fcc-ec6d-48d5-bbdc-54f83820a7e6","owner":[],"postedDate":"August 6th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-08-06T04:20:45+00:00","versionOfRecord":[],"versionCreatedAt":"2025-08-06 04:20:45","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6513777","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6513777","identity":"rs-6513777","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}