A Novel Multi-Layer Semantic Chunking and Embedding Dimension Transformation Techniques for Enhanced Retrieval Augmented Generation

A Novel Multi-Layer Semantic Chunking and Embedding Dimension Transformation Techniques for Enhanced Retrieval Augmented Generation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Novel Multi-Layer Semantic Chunking and Embedding Dimension Transformation Techniques for Enhanced Retrieval Augmented Generation Luyen Nguyen Tien, Binh Hoang Tieu This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7562846/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Retrieval-Augmented Generation (RAG) systems combine information retrieval with generative models to provide accurate responses, but existing frameworks face limitations in semantic chunking flexibility and embedding compatibility across models. This paper introduces MERCED RAG, a novel framework addressing these challenges through twokey innovations: (1) multi-layer semantic chunking with HDBSCAN/Agglomerative clustering algorithms and intelligent outlier handling strategies, and (2) comprehensive embedding dimension transformation techniques including DCT-based upsampling, orthogonal projection, weighted redistribution, and PCA reduction for cross-model interoperability. Our segmentation approach integrates semantic clustering with advanced algorithms and performs comparisons with traditional token-based methods, achieving 98.9% faithfulness and 100% answer relevancy. Our dimension transformation suite enables seamless integration of diverse embedding models while preserving 87.4% semantic similarity. Experimental evaluation on Vietnamese Undergraduate Training Regulations (VUTR) and Narrative Comprehension(NarrativeQA) datasets demonstrates significant improvements: overall scores of 1.027 on VUTR (+3.0% over semantic baseline, +5.2% over traditional baseline) and 0.760 on NarrativeQA (+2.6% over semantic baseline, +4.7% over traditional baseline). The NDCG score improvesfrom 2.889 to 2.948 (+2.0%), establishing new benchmarks for RAG systems and validating ourintegrated approach to semantic chunking andcross-model compatibility. Retrieval-Augmented Generation Semantic Chunking Embedding Dimension Transformation Cross-Model Compatibility Information Retrieval Natural Language Processing Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7562846","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":513886552,"identity":"14c25b5f-e82f-4117-875a-6c3a7b6b9a60","order_by":0,"name":"Luyen Nguyen Tien","email":"","orcid":"","institution":"CMC University","correspondingAuthor":false,"prefix":"","firstName":"Luyen","middleName":"Nguyen","lastName":"Tien","suffix":""},{"id":513886554,"identity":"93805cb2-78c9-48e1-ba13-95112de46969","order_by":1,"name":"Binh Hoang Tieu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAuUlEQVRIiWNgGAWjYBACAwYehgOMDTZQLhsxWtjAWtJI1MLA2HCYBC3m8r0HDxfuOC+v237GgOFD2WEGg9sN+LVYtvElHJ555rbhtjM5BowzzgG13DlAwGHHeAwO87bdTjC7wWPAzNt2mEFyRgJRWs5BtPwlQcsBiBZGoBZ+CYJa8hKAWpKBfkkrONhzLp2HsJbDZw9/5m2zkzc7fnjjgx9l1nJshLSggANAzEOC+lEwCkbBKBgFuAAAk6hEDL//WZ8AAAAASUVORK5CYII=","orcid":"","institution":"CMC University","correspondingAuthor":true,"prefix":"","firstName":"Binh","middleName":"Hoang","lastName":"Tieu","suffix":""}],"badges":[],"createdAt":"2025-09-08 10:08:11","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7562846/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7562846/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":104362371,"identity":"1ad34d8b-c8ca-4d14-b0c4-3fadf70e9de4","added_by":"auto","created_at":"2026-03-11 01:55:23","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":820207,"visible":true,"origin":"","legend":"","description":"","filename":"PAPERJournal.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7562846/v1_covered_8d0f0ff1-65e5-4a32-ba7b-5193d8b4738a.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A Novel Multi-Layer Semantic Chunking and Embedding Dimension Transformation Techniques for Enhanced Retrieval Augmented Generation","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Retrieval-Augmented Generation, Semantic Chunking, Embedding Dimension Transformation, Cross-Model Compatibility, Information Retrieval, Natural Language Processing","lastPublishedDoi":"10.21203/rs.3.rs-7562846/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7562846/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Retrieval-Augmented Generation (RAG) systems combine information retrieval with generative models to provide accurate responses, but existing frameworks face limitations in semantic chunking flexibility and embedding compatibility across models. This paper introduces MERCED RAG, a novel framework addressing these challenges through twokey innovations: (1) multi-layer semantic chunking with HDBSCAN/Agglomerative clustering algorithms and intelligent outlier handling strategies, and (2) comprehensive embedding dimension transformation techniques including DCT-based upsampling, orthogonal projection, weighted redistribution, and PCA reduction for cross-model interoperability. Our segmentation approach integrates semantic clustering with advanced algorithms and performs comparisons with traditional token-based methods, achieving 98.9% faithfulness and 100% answer relevancy. Our dimension transformation suite enables seamless integration of diverse embedding models while preserving 87.4% semantic similarity. Experimental evaluation on Vietnamese Undergraduate Training Regulations (VUTR) and Narrative Comprehension(NarrativeQA) datasets demonstrates significant improvements: overall scores of 1.027 on VUTR (+3.0% over semantic baseline, +5.2% over traditional baseline) and 0.760 on NarrativeQA (+2.6% over semantic baseline, +4.7% over traditional baseline). The NDCG score improvesfrom 2.889 to 2.948 (+2.0%), establishing new benchmarks for RAG systems and validating ourintegrated approach to semantic chunking andcross-model compatibility.","manuscriptTitle":"A Novel Multi-Layer Semantic Chunking and Embedding Dimension Transformation Techniques for Enhanced Retrieval Augmented Generation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-17 00:04:54","doi":"10.21203/rs.3.rs-7562846/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"cfc69266-95c4-476b-941e-c76494e1973d","owner":[],"postedDate":"September 17th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-03-11T01:54:18+00:00","versionOfRecord":[],"versionCreatedAt":"2025-09-17 00:04:54","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7562846","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7562846","identity":"rs-7562846","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}