Benchmarking MeSH-Augmented Embeddings for Biomedical Document Similarity

Benchmarking MeSH-Augmented Embeddings for Biomedical Document Similarity | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Benchmarking MeSH-Augmented Embeddings for Biomedical Document Similarity Rohitha Ravinder, Lukas Geist, Nelson Quiñones, Suhasini Venkatesh, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9116846/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 9 You are reading this latest preprint version Abstract Background : The extensive volume of biomedical scientific literature requires efficient methods for retrieving relevant documents based on semantic technologies and biomedical concepts. While embedding-based methods have shown improvements over traditional keyword-based methods, the integration of domain-specific terminologies like Medical Subject Headings (MeSH) into these models remain underexplored. Methods : This study compares three hybrid methods that integrate MeSH-based annotations with document embeddings (called ''pre-annotation", ''post-annotation" and''post-reduction"). We benchmark these hybrid methods against the following traditional methods: TF-IDF, standard neural embeddings (Word2Vec, fastText, Doc2Vec) and publicly available transformer-based models (BioBERT, SciBERT, SPECTER), using cosine similarity and Word Mover’s Distance (WMD) as evaluation metrics. The benchmark experiments are based on the RELISH corpus, a manually curated dataset of PubMed articles where experts have labeled pairs of documents with regards to their relevance to each other, providing a 2-class (relevant vs. non-relevant) as well as a 3-class (relevant, partially relevant, non-relevant) judgment. Results : Transformer-based models, particularly fine-tuned BioBERT and SciBERT align best with the expert judgements after fine-tuning. Among non-transformer methods, Doc2Vec and MeSH-based hybrid methods perform also well, demonstrating the benefits from combining structured biomedical vocabularies with embedding methods. Our experiments deliver extensive results showing that the baseline performance of 76-78~\% precision at position 5 can be achieved through almost all approaches, improvements of 2-4~\% with MeSH concepts can be achieved, but performances up to 90~\% is left to the fine-tuned large-scale public models. Conclusion : The performance gains from the integration of concepts may be underwhelming, however the benefits lie in the successful integration and benchmarking of structured vocabularies with embedding methods, the applicability of these techniques to aligning literature with other data sources via a controlled vocabulary and the potential for stronger performance on tasks and corpora where concept-based resources are better suited. All experiments have been conserved as a Dockerized pipeline, making the full benchmarking workflow reproducible and supporting future research in biomedical document retrieval. biomedical document similarity biomedical document retrieval document embeddings semantic similarity recommendation systems Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Revision requested 03 May, 2026 Reviews received at journal 03 May, 2026 Reviewers agreed at journal 23 Apr, 2026 Reviews received at journal 17 Apr, 2026 Reviewers agreed at journal 08 Apr, 2026 Reviewers invited by journal 07 Apr, 2026 Editor assigned by journal 31 Mar, 2026 Submission checks completed at journal 18 Mar, 2026 First submitted to journal 13 Mar, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9116846","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":619453523,"identity":"e8b3c5c6-5344-4d3d-b765-15f4d0feca21","order_by":0,"name":"Rohitha Ravinder","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAzElEQVRIiWNgGAWjYJCCAwxsDAz8DDxAJhsDYwPRWiQbSNECUslgcIBYLebtZwwPF5TZ5BufP3t0w4cyBtl+QlpkzuQYHJ5xLs1y24FzaTdnnGMwnknIGgmGtITDvG2HDcwO9pjd5m1jSNxwgJAW/mcgLf8NjJt5zG7/BWrZT1CLRPIBoJYDBgZsQC2MIFsI+UVC4vGBwzznkg0kzvCl3ew5J2E8g7DDEps/85TZGfD3nz1240eZjWx/AyFr0I0gUf0oGAWjYBSMAqwAALFWQqtNMZ8gAAAAAElFTkSuQmCC","orcid":"","institution":"ZB MED Information Centre for Life Sciences","correspondingAuthor":true,"prefix":"","firstName":"Rohitha","middleName":"","lastName":"Ravinder","suffix":""},{"id":619453524,"identity":"c7b3d863-f771-42ec-8b6e-604b65a03ba8","order_by":1,"name":"Lukas Geist","email":"","orcid":"","institution":"ZB MED Information Centre for Life Sciences","correspondingAuthor":false,"prefix":"","firstName":"Lukas","middleName":"","lastName":"Geist","suffix":""},{"id":619453525,"identity":"bdf1cc3d-1c11-41d4-a7b8-d5a0cf1c44dc","order_by":2,"name":"Nelson Quiñones","email":"","orcid":"","institution":"ZB MED Information Centre for Life Sciences","correspondingAuthor":false,"prefix":"","firstName":"Nelson","middleName":"","lastName":"Quiñones","suffix":""},{"id":619453526,"identity":"32e3cb16-bf5f-4448-98c9-3ee45a73e2d8","order_by":3,"name":"Suhasini Venkatesh","email":"","orcid":"","institution":"ZB MED Information Centre for Life Sciences","correspondingAuthor":false,"prefix":"","firstName":"Suhasini","middleName":"","lastName":"Venkatesh","suffix":""},{"id":619453527,"identity":"47e8f951-943f-4b4a-a2b8-a9a2343de608","order_by":4,"name":"Leyla Jael Castro","email":"","orcid":"","institution":"ZB MED Information Centre for Life Sciences","correspondingAuthor":false,"prefix":"","firstName":"Leyla","middleName":"Jael","lastName":"Castro","suffix":""},{"id":619453529,"identity":"cb0a63fa-25e8-4338-8722-13680e1be32d","order_by":5,"name":"Dietrich Rebholz-Schuhmann","email":"","orcid":"","institution":"ZB MED Information Centre for Life Sciences","correspondingAuthor":false,"prefix":"","firstName":"Dietrich","middleName":"","lastName":"Rebholz-Schuhmann","suffix":""}],"badges":[],"createdAt":"2026-03-13 16:53:06","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9116846/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9116846/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":106744947,"identity":"0c50944f-2aee-4ef7-966c-31f05158f649","added_by":"auto","created_at":"2026-04-13 05:19:06","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1396835,"visible":true,"origin":"","legend":"","description":"","filename":"revisedmanuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9116846/v1_covered_f25d5b18-60c8-414d-8015-1d6d67d4c71c.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Benchmarking MeSH-Augmented Embeddings for Biomedical Document Similarity","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"journal-of-biomedical-semantics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jbsm","sideBox":"Learn more about [Journal of Biomedical Semantics](http://jbiomedsem.biomedcentral.com/)","snPcode":"13326","submissionUrl":"https://submission.nature.com/new-submission/13326/3","title":"Journal of Biomedical Semantics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"biomedical document similarity, biomedical document retrieval, document embeddings, semantic similarity, recommendation systems","lastPublishedDoi":"10.21203/rs.3.rs-9116846/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9116846/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e: The extensive volume of biomedical scientific literature requires efficient methods for retrieving relevant documents based on semantic technologies and biomedical concepts. While embedding-based methods have shown improvements over traditional keyword-based methods, the integration of domain-specific terminologies like Medical Subject Headings (MeSH) into these models remain underexplored.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods\u003c/strong\u003e: This study compares three hybrid methods that integrate MeSH-based annotations with document embeddings (called ''pre-annotation\", ''post-annotation\" and''post-reduction\"). We benchmark these hybrid methods against the following traditional methods: TF-IDF, standard neural embeddings (Word2Vec, fastText, Doc2Vec) and publicly available transformer-based models (BioBERT, SciBERT, SPECTER), using cosine similarity and Word Mover’s Distance (WMD) as evaluation metrics. The benchmark experiments are based on the RELISH corpus, a manually curated dataset of PubMed articles where experts have labeled pairs of documents with regards to their relevance to each other, providing a 2-class (relevant vs. non-relevant) as well as a 3-class (relevant, partially relevant, non-relevant) judgment.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e: Transformer-based models, particularly fine-tuned BioBERT and SciBERT align best with the expert judgements after fine-tuning. Among non-transformer methods, Doc2Vec and MeSH-based hybrid methods perform also well, demonstrating the benefits from combining structured biomedical vocabularies with embedding methods. Our experiments deliver extensive results showing that the baseline performance of 76-78~\\% precision at position 5 can be achieved through almost all approaches, improvements of 2-4~\\% with MeSH concepts can be achieved, but performances up to 90~\\% is left to the fine-tuned large-scale public models.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion\u003c/strong\u003e:\u0026nbsp;The performance gains from the integration of concepts may be underwhelming, however the benefits lie in the successful integration and benchmarking of structured vocabularies with embedding methods, the applicability of these techniques to aligning literature with other data sources via a controlled vocabulary and the potential for stronger performance on tasks and corpora where concept-based resources are better suited. All experiments have been conserved as a Dockerized pipeline, making the full benchmarking workflow reproducible and supporting future research in biomedical document retrieval.\u003c/p\u003e","manuscriptTitle":"Benchmarking MeSH-Augmented Embeddings for Biomedical Document Similarity","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-13 05:18:46","doi":"10.21203/rs.3.rs-9116846/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-05-03T11:27:54+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-03T09:28:33+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"208790491698658387962091675438797151843","date":"2026-04-23T07:43:24+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-17T12:00:33+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"301931295044741712547309624952821599884","date":"2026-04-08T07:47:14+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-07T08:04:00+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-03-31T11:35:45+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-03-18T12:11:58+00:00","index":"","fulltext":""},{"type":"submitted","content":"Journal of Biomedical Semantics","date":"2026-03-13T16:36:35+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"journal-of-biomedical-semantics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jbsm","sideBox":"Learn more about [Journal of Biomedical Semantics](http://jbiomedsem.biomedcentral.com/)","snPcode":"13326","submissionUrl":"https://submission.nature.com/new-submission/13326/3","title":"Journal of Biomedical Semantics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"8048bde6-e3a1-4d4b-8340-975c72fa4929","owner":[],"postedDate":"April 13th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Revision requested","date":"2026-05-03T11:27:54+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-03T09:28:33+00:00","index":26,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[],"tags":[],"updatedAt":"2026-05-03T11:39:00+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-13 05:18:46","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9116846","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9116846","identity":"rs-9116846","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}