A Version-aware Data Layout for Heterogeneous Workloads in In-Memory Database Systems

A Version-aware Data Layout for Heterogeneous Workloads in In-Memory Database Systems | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Version-aware Data Layout for Heterogeneous Workloads in In-Memory Database Systems First Qian Zhang, Second Jianhao Wei, Third Shichen Zhang, Fourth Hao Luan, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4105094/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Most modern in-memory database systems rely on multi-version concurrency control (MVCC) to support real-time data analysis without interfering with concurrent writes. While MVCC elegantly enables lightweight isolation of readers and writers, it also generates many short-lived versions and results in long version chains. Existing version storage designs employ traditional data layouts lacking version information. Consequently, as time elapses, the long-running query latency increases due to increased chain lengths. In this paper, we introduce a novel version-aware data layout for in-memory MVCC databases. This layout improves search performance for heterogeneous workloads. It differs from previous approaches by organizing data in a version-centric manner, which supports index-only searches and reduces costly version traversals. Specifically, the layout utilizes a version B + -tree to organize the latest tuple versions of a table, making the table index-organized. Additionally, it employs range partitioning and epoch partitioning techniques to group older tuple versions efficiently. This design allows for fast retrieval of the correct versions visible to transactions, minimizing redundant memory accesses. Compared to existing version storage schemes, the version-aware data layout considers workload access patterns and the opportunity to skip unnecessary version accesses. The experimental evaluation indicates that the version-aware data layout can outperform the state-of-the-art designs by 1.15× ∼ 2×. In-memory databases MVCC heterogeneous workloads data layout Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4105094","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":279871995,"identity":"d0cb4c89-d0da-4791-a0db-539145270726","order_by":0,"name":"First Qian Zhang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABGElEQVRIie3PvWrDMBDAcRmBvBxoPZF+vIKKIW2wwa9iU7CXEOiWQAcZg7IEMvdNOtYIPDl0TcniTF2bLYVQ4ngqxQaPHfQHaTn94ESIzfYPC5dZVosTAqO0qH8NaC+RYHI5UcEVd9mjJNEQgqnGhUoCsYYxDiL3JM7lx6sBach48X00hLubEsncj5W7eesiE1Vk+5eqJckOIkPEapYgqdJYwSzqIqTIcg9ZS8odaYjcNhs62sQKQXYS4+jRqSWOfjo2JGzJTz+RpaNR6AREThm5LCbxQlQ/ESsnl1gFwCmjI0hSwGrqPURl6mmYdhJ++7mvcY4h4++HwzHwb/iyutt+PfvXa7fq/svfoL2j5rBB7202m83W1RkS3lvSHhG+hgAAAABJRU5ErkJggg==","orcid":"","institution":"East China Normal University","correspondingAuthor":true,"prefix":"","firstName":"First","middleName":"Qian","lastName":"Zhang","suffix":""},{"id":279871996,"identity":"867ee793-343e-493f-b742-36211cde1259","order_by":1,"name":"Second Jianhao Wei","email":"","orcid":"","institution":"East China Normal University","correspondingAuthor":false,"prefix":"","firstName":"Second","middleName":"Jianhao","lastName":"Wei","suffix":""},{"id":279871997,"identity":"496da280-f2f5-46c8-a161-962459c922f4","order_by":2,"name":"Third Shichen Zhang","email":"","orcid":"","institution":"East China Normal University","correspondingAuthor":false,"prefix":"","firstName":"Third","middleName":"Shichen","lastName":"Zhang","suffix":""},{"id":279871998,"identity":"8c5d4a58-5bff-434a-9148-76f8b5a070b8","order_by":3,"name":"Fourth Hao Luan","email":"","orcid":"","institution":"East China Normal University","correspondingAuthor":false,"prefix":"","firstName":"Fourth","middleName":"Hao","lastName":"Luan","suffix":""},{"id":279871999,"identity":"26afb0c1-646c-415b-b53d-9b6fdf0e059e","order_by":4,"name":"Fifth Xueqing Gong","email":"","orcid":"","institution":"East China Normal University","correspondingAuthor":false,"prefix":"","firstName":"Fifth","middleName":"Xueqing","lastName":"Gong","suffix":""}],"badges":[],"createdAt":"2024-03-15 05:44:36","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4105094/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4105094/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":53612503,"identity":"6a3ebe7d-6deb-4568-a76a-48d2327b48e9","added_by":"auto","created_at":"2024-03-28 05:37:41","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1831961,"visible":true,"origin":"","legend":"","description":"","filename":"aversionawaredatalayoutforinmemorydatabases.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4105094/v1_covered_cd54b494-1a7d-4e7d-86f8-cde2a6c17356.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A Version-aware Data Layout for Heterogeneous Workloads in In-Memory Database Systems","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"In-memory databases, MVCC, heterogeneous workloads, data layout","lastPublishedDoi":"10.21203/rs.3.rs-4105094/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4105094/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eMost modern in-memory database systems rely on multi-version concurrency control (MVCC) to support real-time data analysis without interfering with concurrent writes. While MVCC elegantly enables lightweight isolation of readers and writers, it also generates many short-lived versions and results in long version chains. Existing version storage designs employ traditional data layouts lacking version information. Consequently, as time elapses, the long-running query latency increases due to increased chain lengths. In this paper, we introduce a novel version-aware data layout for in-memory MVCC databases. This layout improves search performance for heterogeneous workloads. It differs from previous approaches by organizing data in a version-centric manner, which supports index-only searches and reduces costly version traversals. Specifically, the layout utilizes a version B\u003csup\u003e+\u003c/sup\u003e-tree to organize the latest tuple versions of a table, making the table index-organized. Additionally, it employs range partitioning and epoch partitioning techniques to group older tuple versions efficiently. This design allows for fast retrieval of the correct versions visible to transactions, minimizing redundant memory accesses. Compared to existing version storage schemes, the version-aware data layout considers workload access patterns and the opportunity to skip unnecessary version accesses. The experimental evaluation indicates that the version-aware data layout can outperform the state-of-the-art designs by 1.15× ∼ 2×.\u003c/p\u003e","manuscriptTitle":"A Version-aware Data Layout for Heterogeneous Workloads in In-Memory Database Systems","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-03-19 03:10:30","doi":"10.21203/rs.3.rs-4105094/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d636542a-dac6-495d-a4cb-44c1f3eeb270","owner":[],"postedDate":"March 19th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-03-28T05:29:30+00:00","versionOfRecord":[],"versionCreatedAt":"2024-03-19 03:10:30","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4105094","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4105094","identity":"rs-4105094","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}