reEtym: A Natively Feature-Disentangled Transformer for Interpretability

reEtym: A Natively Feature-Disentangled Transformer for Interpretability | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article reEtym: A Natively Feature-Disentangled Transformer for Interpretability Hongyu Shi This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9416412/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Based on the hypothesis that "human language is composed of fundamental semantic atoms," this paper proposes reEtym, a feature-disentangled architecture that modifies only the embedding layer. By factorizing the embedding matrix into a "recipe" matrix W_recipe and an "etymological basis" matrix W_basis, the model is guided to maintain a continuous set of semantic etymological bases in the latent space. At 0.5B parameters and 50k pretraining steps, reEtym achieves near-lossless equivalence with conventional architectures on zero-shot benchmarks (fluctuations within ±2.4%), while improving topic coherence by 28.4% and reducing extreme failure cases by 98.6%. Concurrently, interpretable structures spontaneously emerge in the etymological space: semantic algebra (6/6 hits, including linguistic and arithmetic analogies), natural sparsity (11-13% activation rate), and signal-level causal traceability (ablating a single signal reduces prediction from 8.31% to 0.03%), revealing new avenues for exploration. Unlike post-hoc reconstruction methods, the etymological space in reEtym is directly defined by the architecture and constitutes a native component of the model's computation. This enables audit findings to be directly translated into model modifications—adjusting recipes or bases can achieve behavioral steering such as sentiment manipulation and topic coherence enhancement, without retraining. Since modifications are confined to the embedding layer, this mechanism naturally extends to non-Transformer architectures such as Mamba and RWKV. The complete source code, model weights, training logs, and an online interpretability platform are publicly available under the MIT license at: https://github.com/reEtym/reEtym Theoretical Computer Science Artificial Intelligence and Machine Learning Information Theory reEtym Mechanistic Interpretability Disentangled Representation Learning Embedding Layer Factorization Semantic Etymology Large Language Models Model Steering Causal Traceability Natural Language Processing Mamba RWKV Zero-shot Learning Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9416412","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":623027457,"identity":"f42b286c-702d-4f77-beff-b9e2a3b56774","order_by":0,"name":"Hongyu Shi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA20lEQVRIiWNgGAWjYPACCRDB+ABI8PARpyMBrIXZAKSFjUgtYJINrJGgFvP25oePC39Y5BkcP/ys8muOnQwbA/PDRzfwaJE5c8zYeEaCRLHBmTSz27LbkoEOYzM2zsGjRUIih02aJ0EiccOBHLbbktuYgVp42KTxapF/A9Vy/g1bseS2eiK0SPBAtdzIYWP8uO0wEVp40oyNedIkEmfeeGYszbjtOA8bMyG/sB9++JjHpi6x73zyw48/t1Xb87MDwxCfFhTAzAMmiVUOAow/SFE9CkbBKBgFIwYAAFZgPsZs3rYdAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0009-0006-0778-8613","institution":"Independent Researcher","correspondingAuthor":true,"prefix":"","firstName":"Hongyu","middleName":"","lastName":"Shi","suffix":""}],"badges":[],"createdAt":"2026-04-14 13:55:47","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-9416412/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9416412/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":106973853,"identity":"c620469e-2df0-4b2b-a59b-bbc92571f0c7","added_by":"auto","created_at":"2026-04-15 10:29:52","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":282729,"visible":true,"origin":"","legend":"","description":"","filename":"reetymiclr2026enarxiv.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9416412/v1_covered_20d77092-860d-46bc-951b-511c12422cea.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003ereEtym: A Natively Feature-Disentangled Transformer for Interpretability\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"reEtym, Mechanistic Interpretability, Disentangled Representation Learning, Embedding Layer Factorization, Semantic Etymology, Large Language Models, Model Steering, Causal Traceability, Natural Language Processing, Mamba, RWKV, Zero-shot Learning","lastPublishedDoi":"10.21203/rs.3.rs-9416412/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9416412/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBased on the hypothesis that \"human language is composed of fundamental semantic atoms,\" this paper proposes reEtym, a feature-disentangled architecture that modifies only the embedding layer. By factorizing the embedding matrix into a \"recipe\" matrix W_recipe and an \"etymological basis\" matrix W_basis, the model is guided to maintain a continuous set of semantic etymological bases in the latent space.\u003c/p\u003e\n\u003cp\u003eAt 0.5B parameters and 50k pretraining steps, reEtym achieves near-lossless equivalence with conventional architectures on zero-shot benchmarks (fluctuations within ±2.4%), while improving topic coherence by 28.4% and reducing extreme failure cases by 98.6%. Concurrently, interpretable structures spontaneously emerge in the etymological space: semantic algebra (6/6 hits, including linguistic and arithmetic analogies), natural sparsity (11-13% activation rate), and signal-level causal traceability (ablating a single signal reduces prediction from 8.31% to 0.03%), revealing new avenues for exploration.\u003c/p\u003e\n\u003cp\u003eUnlike post-hoc reconstruction methods, the etymological space in reEtym is directly defined by the architecture and constitutes a native component of the model's computation. This enables audit findings to be directly translated into model modifications—adjusting recipes or bases can achieve behavioral steering such as sentiment manipulation and topic coherence enhancement, without retraining. Since modifications are confined to the embedding layer, this mechanism naturally extends to non-Transformer architectures such as Mamba and RWKV.\u003c/p\u003e\n\u003cp\u003eThe complete source code, model weights, training logs, and an online interpretability platform are publicly available under the MIT license at: https://github.com/reEtym/reEtym\u003c/p\u003e","manuscriptTitle":"reEtym: A Natively Feature-Disentangled Transformer for Interpretability","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-15 10:20:49","doi":"10.21203/rs.3.rs-9416412/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6849470c-ef4e-4606-b552-8149917e7e61","owner":[],"postedDate":"April 15th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":66302427,"name":"Theoretical Computer Science"},{"id":66302428,"name":"Artificial Intelligence and Machine Learning"},{"id":66302429,"name":"Information Theory"}],"tags":[],"updatedAt":"2026-04-15T10:20:49+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-15 10:20:49","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9416412","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9416412","identity":"rs-9416412","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}