Document embeddings for long texts from Transformers and Autoencoders

Document embeddings for long texts from Transformers and Autoencoders | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Document embeddings for long texts from Transformers and Autoencoders Lenos Christou, Agorakis Bompotas, Christos Makris This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5459822/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract In document categorization, the escalating volume of digital data underscores the critical need for advanced techniques in generating semantically rich document embedding, particularly for topic modelling applications. This article presents a novel methodology that leverages the capabilities of Sentence-BERT (SBERT) for sentence embedding, aiming to address the challenges associated with embedding longer texts. The process begins by decomposing documents into individual sentences that can be processed using the SBERT model which generates an embedding for each sentence. Subsequently, a clustering algorithm is employed to select representative sentence embeddings for each document. These embeddings serve as the foundation for constructing comprehensive document embeddings through an autoencoder. This research exclusively explores the autoencoder approach, discarding alternative methods for brevity and focus. The autoencoder strategy showed promising results, occasionally outperforming the base model, Doc2Vec, in specific scenarios. This highlights the method's potential effectiveness in document embedding tasks where understanding long context is important. Moreover, when the Transformer Autoencoder was fine-tuned on test data, it achieved performance comparable to that of Doc2Vec, underscoring the viability of this approach. The findings suggest considerable room for improvement in various aspects of the methodology, including the clustering process and the encoder architecture. This article underscores the potential of autoencoders in advancing the state-of-the-art in document embedding technologies, setting the stage for further refinement and exploration. Document Embeddings Transformers Autoencoders Large Texts Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5459822","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":390169459,"identity":"c757aad7-ad84-4a6e-8134-67d6fc637265","order_by":0,"name":"Lenos Christou","email":"","orcid":"","institution":"University of Patras","correspondingAuthor":false,"prefix":"","firstName":"Lenos","middleName":"","lastName":"Christou","suffix":""},{"id":390169460,"identity":"f08ff992-0fd0-497b-8400-0514966e681e","order_by":1,"name":"Agorakis Bompotas","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABAklEQVRIiWNgGAWjYFAD9gYGhgQ24tUbMDDwHIBpYSZWi0QCkCZGi+7s5qObCyr+yMvPfPzwwYMyhsTt7OePffjxh8FuewN2LWZ3jqXdnnHGwHDD7TRjg4RzDIk7e5KZZ/a2MSTPOYBDy40cs9u8bQaMG6Rz2CQS2xgSNxxIZmbgbWBIlsDhMJgW+/kzz7D/AGs5/5iZ8c8fwloSG27wsDGAtdxIZmYGsu1waoH4xTh5w5k0Y4mEcxLGO2c8NmaWbZNIwKnldvOx2wUVcrbz2w8//PijzEZ2O3/iY8Y3f2zscWlhkECNBwlQFIEZiQ1EamGAaWGwx6VjFIyCUTAKRhwAACsrXFFatoFkAAAAAElFTkSuQmCC","orcid":"","institution":"University of Patras","correspondingAuthor":true,"prefix":"","firstName":"Agorakis","middleName":"","lastName":"Bompotas","suffix":""},{"id":390169461,"identity":"3678e9b1-2581-473a-a010-93e1a6e4841f","order_by":2,"name":"Christos Makris","email":"","orcid":"","institution":"University of Patras","correspondingAuthor":false,"prefix":"","firstName":"Christos","middleName":"","lastName":"Makris","suffix":""}],"badges":[],"createdAt":"2024-11-15 10:38:17","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5459822/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5459822/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":76512857,"identity":"056d1723-3cdd-4c89-9b8d-9319f0f37806","added_by":"auto","created_at":"2025-02-18 02:32:11","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":534401,"visible":true,"origin":"","legend":"","description":"","filename":"DocumentembeddingsforlongtextsfromTransformersandAutoencodersKAIS.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5459822/v1_covered_5d5c1f4d-49ad-4660-bd6a-cd63a47d3bf1.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Document embeddings for long texts from Transformers and Autoencoders","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Document Embeddings, Transformers, Autoencoders, Large Texts","lastPublishedDoi":"10.21203/rs.3.rs-5459822/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5459822/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eIn document categorization, the escalating volume of digital data underscores the critical need for advanced techniques in generating semantically rich document embedding, particularly for topic modelling applications. This article presents a novel methodology that leverages the capabilities of Sentence-BERT (SBERT) for sentence embedding, aiming to address the challenges associated with embedding longer texts. The process begins by decomposing documents into individual sentences that can be processed using the SBERT model which generates an embedding for each sentence. Subsequently, a clustering algorithm is employed to select representative sentence embeddings for each document. These embeddings serve as the foundation for constructing comprehensive document embeddings through an autoencoder. This research exclusively explores the autoencoder approach, discarding alternative methods for brevity and focus. The autoencoder strategy showed promising results, occasionally outperforming the base model, Doc2Vec, in specific scenarios. This highlights the method's potential effectiveness in document embedding tasks where understanding long context is important. Moreover, when the Transformer Autoencoder was fine-tuned on test data, it achieved performance comparable to that of Doc2Vec, underscoring the viability of this approach. The findings suggest considerable room for improvement in various aspects of the methodology, including the clustering process and the encoder architecture. This article underscores the potential of autoencoders in advancing the state-of-the-art in document embedding technologies, setting the stage for further refinement and exploration.\u003c/p\u003e","manuscriptTitle":"Document embeddings for long texts from Transformers and Autoencoders","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-12-16 04:35:03","doi":"10.21203/rs.3.rs-5459822/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"a71ea5b4-59ea-49eb-adfb-436df2c806f3","owner":[],"postedDate":"December 16th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-02-18T02:23:48+00:00","versionOfRecord":[],"versionCreatedAt":"2024-12-16 04:35:03","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5459822","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5459822","identity":"rs-5459822","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}