An Innovative Comparative Performance Analysis on Document Store Non- Relational Databases

An Innovative Comparative Performance Analysis on Document Store Non- Relational Databases | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Short Report An Innovative Comparative Performance Analysis on Document Store Non- Relational Databases Shah Miah, Kiran Fahd, Sitalakshmi Venkatraman, Sazia Parvin This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4478249/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The crucial role of competent software architecture is essential in managing the challenging of big data processing for both relational and nonrelational databases. Relational databases are designed to structure data for facilitating vertical scalability, while non-relational databases excel in handling vast volumes of unstructured data for enhancing horizontal scalability. Choosing the right database paradigm is determined by the needs of the organization, yet selecting the best option may often be a challenging task. Large number of applications still use relational databases due to its benefits of reliability, flexibility, robustness, and scalability. However, with the rapid growth in web and mobile applications as well as huge amounts of complex unstructured data generated via online and offline platforms, nonrelational databases are compensating for the inefficiency of relational databases. Since selecting the right nonrelational database method for high performing applications from a plethora of possibilities is a challenging task, existing studies are still at emergent stage to compare the performance of different popular nonrelational databases. This paper introduces a novel benchmarking approach for tailoring the comparative study of nonrelational databases. To illustrate our approach, we compare two leading non-relational databases, Aerospike and MongoDB, focusing on their average transaction times to evaluate the database performance. Our comprehensive analysis reveals the strengths of each database in read and write operations for single record and bulk record batch transactions. NoSQL Non-relational database Performance comparison Document Store Key Value Store MongoDB Aerospike Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 1. Introduction The database landscape has become complicated tasks as choosing the right database from a range of options is crucial for a successful software development. Therefore existing literature recognised that determining the optimum database solution for a software application within a business context is a multifaceted challenge for practitioners and researchers (Jones, 2022). With a myriad of existing databases, the selection process of an optimum database solution becomes significantly complex due to the requirement to explore options from an ever-growing set of diverse database types. Big data researchers need to consider several parameters when deciding which database to choose. Some of the key properties to consider are data Model, CAP support, multi data center support, capacity, performance, query API, reliability, and data persistence. Currently, there are hundreds of options that require evaluation of a number of database characteristics, and the recent focus is on considering sustainable success factors such as flexible data structure, performance, and scalability. In this paper, we confine our study to the evaluation of performance measures of databases. Among the plethora of available database types, the main two types that developers/researchers explore are: i) traditional relational databases that guarantee the consistency and integrity of the structured data (e.g. Postgres and Oracle), and ii) dynamic/modern non-relational databases (aka NoSQL database) that emphasize schema flexibility and efficiency in data retrieval (e.g. DynamoDB and MongoDB). Non-relational database relies on the BASE (Basically Available, Soft state, Eventual consistency) framework, where eventual consistency means the data across all the nodes are to be eventually consistent instead of immediate and instantaneous consistent status which is the case with traditional databases. Modern technologies have be transformed with the advent of social media, mobile applications for enhancing their capacities. Internet of Things (IoT) and their applications are emerging rapidly in contemporary industry with notable initiatives such as smart agriculture, smart transportation, and smart cities, generate massive and complex data. These technologies leverage non-relational databases due to flexible data models and scalability to store structured, semi-structured and unstructured data. Big data term is often used to describe data generated from these modern technological platforms. Five innate characteristics of Big data, known as 5Vs, are Velocity – the speed at which data is generated; Volume – a large scale of the data; Variety – different forms and sources of the data; Veracity – reliability, quality and accuracy of the data and Value – usefulness of the data (Balusamy et al., 2021 ; Sivarajah et al., 2017 ). Big data systems must have functionalities for managing massive volume of data at any scale. One of the main requirements to deal with big data is having the ability to scale the servers, and once again, scaling up relational database servers is very expensive and scaling it out is difficult. This is the central interests of many big data researchers. However, non-relational databases were designed to handle the big data and because of that methods were implemented to improve performance when retrieving and storing data (Mohamed et al., 2014 ). For big data processing, a variety of frameworks are employed, e.g. Hadoop/MapReduce, Kafka, and Spark. Due the vast options of non-relational databases, the selection become even more nuanced. For example, key value store such as Aerospike offers rapid and direct access to data, whereas document stores such as MongoDB are best suited for scenarios where unparalleled flexibility in managing semi-structured data is of prime importance. To meet the increasing demand for diverse data models, it is critical to select appropriate type of database. This paper investigates the benchmarking process of comparing database performance with a focus on transaction speed and responsiveness across two different non-relational databases. Through a tailored benchmarking and analysis approach, the paper aims to provide value insights to guide the decision-making process for selecting the right type of non-relational database. There are several disadvantages of using standard tools such as YCSB and they have been reported with an inherent throughput issue (Gembalczyk et al., 2017 ). Due to their inability to provide flexibility, scalability and extensibility, custom benchmark tools are being developed. Our contribution uniquely differentiates from other comparative studies found in the literature as we propose a comprehensive tailored approach to benchmarking for arriving at an optimal database for an application. The remainder of this paper is organized as follows. First, we present a systematic review of related work to provide the research background and the gaps in literature related to studies conducted on the selected key value store and document store databases. Next, we outline the research methodology adopted detailing the research design and the experimental setup, with a proposed benchmark workflow model of the experiment. Following this, the paper reports the experimental results and findings utilizing visual aids to enhance data presentation. Finally, we provide concluding remarks on this research study and briefly discuss future research directions. 2. Existing relevant works The study recognises the significance of background studies in previous literature of performance comparison of databases, specifically related to non-relational databases. Different non-relational databases methods offer different storage, data model, schema, and features. In general, non-relational databases can be categorised into four major groups: Document store, Key Value store, Column-wide stores, and Graph stores. Table 1 shows the difference between the key attributes offered by Document store and Key Value store categories (Venkatraman et al., 2016 ). Non-relational databases utilise a diverse approach to data modelling, data storage and access in comparison to relational databases especially such as availability and horizontal scalability. Table 1 Features comparison of Document store and Key Value store Data Model Performance Scalability Flexibility Complexity Advantage Key-Value Stores (e.g. Aerospike, Redis, Raik) Collection of key-value pairs High High High None Fast search Document Stores (e.g. MongoDB, CouchDB, DocumentDB, DynamoDB) Key-value where value is structured document object High Variable (High) High Low Flexible schema Key value store Aerospike test, uses data-in-memory storage, storing indices in the random-access memory (RAM) of the server and serves read and write operations to the disk. Aerospike, based on its name, is designed to rapidly scale in size and is best used as a cache replacement, info store, and more (Srinivasan et al., 2023 ). Document store MongoDB is not a data-in-memory storage. However, it makes use of caches storing data in the RAM for fast access. This caching might show slow start for the first benchmark but would accelerate results for the subsequent runs. MongoDB is also advertised to be scalable and flexible. It is recommended to be used for catalogues, payment ledgers, and so on so forth (Ouyang et al., 2021 ). Table 2 summarises the existing studies related to performance analysis of non-relational databases with a focus on document store databases and key value store databases. Table 2 Summary of existing studies about comparison of non-relational databases Study Study Aim (Landuyt et al., 2023 ) This study compared query execution time, the two multi-model non-relational databases, ArangoDB and OrientDB, with a polyglot database setup, and query execution time. The study lacks an analysis of scalability and database performance under varying workloads. (Khan et al., 2023) This study compared the structures, scalability, and performance of relational and non-relational databases in handling large data application. The study did not compare two non-relational databases. (Alyasiri et al., 2022 ) This study compares the characteristics, challenges, and practical query processing of two different databases, one non-relational database MongoDB and other relational database MySQL. It lacks an exploration of performance comparison between two non-relational databases. (Li & Gai, 2021 ) This study provides comparative analysis of several non-relational database of read and write performances of MongoDB and Redis across different usage scenarios. The study highlighted the lack of detailed comparative analysis focusing on the performance aspects, especially comparison on read and write performance, of non-relational databases. (Cui & Chen, 2021 ) This study compared the read and write performance of two column-wide store databases i.e. HBase and Cassandra with experimental results obtained from the Yahoo! Cloud Serving Benchmark (YCSB). The study compared two non-relational databases from same category of non-relational databases i.e. column-family stores or wide-column stores, not from two different types of NoSQL categories like key-value or document store. Our summary of literature review shown in Table 2 illustrates significant gaps that we bring into researchers attention in this paper. This is to assist big data oriented application developers/researchers in selecting a suitable database method among document store and key value store databases based on their comparative performance criteria. We justify the requirements for performance comparison between two widely used non-relational databases with a narrowed focus to two specific categories of non-relational databases: document store and key-value store by providing insights that are highly relevant to applications that may prefer one database over the other based on their operational requirements. There is a need for more tailored approach by employing customized benchmarking scripts that offers flexibility and customization in the benchmarking process, allowing for tailored scenarios that are outside the scope of standard benchmarking tools. Previous studies have reported drawbacks of using standard benchmarking tools such as YCSB have throughput issues (Gembalczyk et al., 2017 ). Some studies have attempted for a custom benchmark tool to cater to flexibility, extensibility and scalability in comparing relational and non-relational databases. However, there is lack of studies that address these issues for comparing contemporary non-relational databases of the types, key-value store and document store. In brief, it is imperative to address the gap in the target body of literature by creating a benchmarking approach that can be customised to specifically focus on the performance comparison of key-value store and document store non-relational databases. Our study offers a focused comparison of document store and key-value non-relational databases through tailored scripts, specifically exploring transaction read and write times for real world scenarios. It precisely addresses the need for performance comparison insights in scenarios most relevant to developers/researchers, thereby filling an important gap as the right database selection for application optimization is critical for businesses. 3. Research Methodology We adopt a research methodology with a theoretical approach starting with a literature survey with an analysis of relevant scholarly articles as detailed in previous section, followed by an experimental study to serve as a benchmark for evaluating the performance of non-relational databases. The purpose of this approach is to develop a custom benchmarking that would aid data researchers/application developers in choosing the optimal one for an application. It’s important to offer a benchmark workflow model to address the lack of support for customisation of the experimental study in standard benchmark tools such as YCSB. The aim of this study is to apply our proposed workflow model to measure and compare the performance of two non-relational databases as a proof-of-concept. Figure 1 gives a pictorial representation of our benchmark workflow model of the experimental study adopted in this research work. In this experimental study, two representative non-relational databases are selected, each from of queries into one benchmark tool, which was not possible with YCSB and other standard benchmark tools. Hence, using our proposed benchmark workflow model, we compare two contemporary non-relational databases, namely key value store and document store due their popularity in big data applications. We have selected Aerospike as representative of key value store and MongoDB for document store as explained in previous section to examine the performance of typical database operations, write (create, update, delete) and read on different numbers of records for quantitative comparison. Numerous factors like security, flexibility, scalability, and performance can affect the database selection (Győrödi et al., 2020 ; Khan et al., 2019 ). For example, the performance is influenced by number of the users with write access and number of records modified during write operations (Čerešňák & Kvet, 2019 ). It is important to identify the method used to measure and compare the database performance and database efficiency. In this study, mean transaction time, execution time of database operation for different numbers of records, in milliseconds per operation is identified as database performance measurement methods as it is considered as one of the foremost methods according to the existing literature (Kolonko, 2018 ). An ultimate experiment is required to be executed to measure numerical average transaction time for different number of records for different database operations and their relationship is presented numerically. The member database stores information about member name and membership number data. In MongoDB, a member record for a member stored as a document within a collection with fields for each attribute, such as {"memberName": "John Smith", "membershipNumber": "MVIC1234"}. For Aerospike, a member record would consist of a set identified by a unique key within a namespace with bins, where each bin corresponds to an attribute, e.g. {"memberName": "John Smith", "membershipNumber": "MVIC1234"}. In this experiment, a python script is executed for Aerospike and MongoDB to get execution time measures for different numbers of records i.e. a single record and a bulk batch of records. This provides a broad understanding into how database performs ass the number of records increases for each database operation. A python script is executed to create a record for write or read transactions and bulk records for bulk read and write transactions. Another the python script establishes a connection with the database on server. It executes two operations conducted iteratively over 100 cycles. In first phase, it performs read and write operation for a single record and calculate the mean transaction time of read and write for single record. Then in second phase, it executes read and write transactions for mass records as bulk batch and calculates the mean transaction time for bulk batch. The script concludes by clean-up operation and closing the database connection. Time for the single server connection and clean-up operation is also calculated. The results of both scenarios are compared in the next section. 4. Experimental Results and Findings As mentioned earlier, we conducted experiments for executing read and write operations using a dual-machine setup comprising a client and a server over the intranet. The client operates on Arch Linux 6.1.8 and on the server side, services are deployed within Docker containers, which are hosted on the Microsoft Windows 11, version 22H2, leveraging the Windows Subsystem for Linux 2 (WSL2). The docker setup ensure compatibility and ease of deployment across different environments. Python libraries 4.3.2 and Aerospike 7.1.1 are used for client whilst MongoDB 6.0.2 and Aerospike 6.1.0.3 are used for server. Intranet provided a seamless and controlled communication channel for the execution of experiments. Therefore, external variables are minimized to maintain uniformity in database settings of the experiment to compare performance of two non-relational database. This experiment script is executed three (3) times to enhancing the reliability of the results for each database. The performance of a key value store database i.e. Aerospike and document store database i.e. MongoDB, is compared over three runs based on the average transaction time (in milliseconds(msec)) it takes to complete standard database operations specifically Create, Read, Update, and Delete (CRUD), across three consecutive runs. In this experiment, these operations are categorised into read operations and write operations; where ‘read transactions’ includes only the Read operation, and write operations include Create, Delete and Update operations. A lower average transaction time signifies better performing database. This information is crucial when considering which database to select based on the specific needs of the applications, whether it is read heavy or a write heavy application. Following subsections outlines and analyse the results of execution time of create, update, delete, and read operations. 4.1 Create Operation The outcome of three iterations of the write create operation for one record is depicted in Fig. 2.1 . This figure illustrates the time, in msec, required to add one record using MongoDB and Aerospike. In this operation, a key is automatically generated, and the member’s name and membership number data are inserted as a single record. The comparison outcome shows that while Aerospike constantly required less time to execute create operation for a single record across all three runs, MongoDB exhibited improvement from the first to the second run, signifying a potential improvement in successive operations. The result of the three iterations of the create operation shows the performance of Aerospike and MongoDB when inserting bulk records. The average execution time of the three iterations of the create operation is presented in Fig. 2.2 , where the time is measured in msec. The results shows that MongoDB outperformed Aerospike for bulk create operation with shorter execution times across all three runs, which highlights higher efficiency of MongoDB in handling insert operations for large datasets. Overall, the results of the create operation presents the average transaction time of MongoDB and Aerospike. Figures 2.1 and 2.2 illustrate that the average transaction time for MongoDB is consistently lower than that for Aerospike except for the first iteration for single record creation. The higher transaction time for the first iteration for single record in MongoDB could be attributed to its flexibility in schema that could require more initial setup overhead that gets averaged out with bulk records. 4.2 Update Operation This study used two types of update operations (i) Single update operation (ii) Bulk update operation. In both update operations, the time of average transaction time of the update operations is measured in seconds. The outcome of three iterations of the write update operation for one record is shown in Fig. 2.3 . Single update operation finds a record based on random key and update the associated member name and membership number information. The result of the three iterations of the bulk write update operation shows the performance of Aerospike and MongoDB when updating bulk records as depicted in Fig. 2.4 . Bulk update operation mass updates the values of member name and membership number for all records in the non-relational databases. The result shows that MongoDB demonstrated a comparatively consistent performance for bulk update operation highlighting its stability in handling updates of bulk records. Overall, Aerospike not only consistently maintains better average transaction times for single update operation but also displays more uniform performance across all iterations for bulk update operation in comparison to MongoDB. 4.3 Delete Operation The outcome of the delete operations are shown in Fig. 2.5 and Fig. 2.6 . In this study, two delete operations are used to compare the write performance of the two non-relational databases with average transaction time. Single delete operation deletes the latest record or latest key. The average transaction time in msec of single delete operation is presented in Fig. 2.5 . The data indicates that Aerospike consistently executed delete operations for a single record more rapidly than MongoDB across all three runs, highlighting the efficient single deletion operation for the latest records based on keys. Second bulk update operation to compare non-relational databases write performance based on deleting random set of mass records from the database. The average transaction time in msec of the bulk delete operation is shown in Fig. 2.6 . The comparative outcome demonstrates that Aerospike constantly executed delete operation for bulk records in less time than MongoDB across all three runs, which shows its efficiency in handling bulk records delete operation. 4.4 Read Operation The read performance of both non-relational databases is presented in msec in Fig. 2.7 and 2.8. Single read operation read a record with random key. Figure 2.7 shows that MongoDB maintaining a significantly lower average transaction time for read operation for single record across all three iterations. The average transaction time of Aerospike is substantially higher and showed an increasing trend, remarkably noticeable in the third run. Bulk read operation retrieves a substantial portion of the records stored in the database, if not all. The bulk read performance for MongoDB is better with lower transaction time when compared to Aerospike as shown in Fig. 2.8 . 5. Overall Analysis Table 3 briefly summarizes the overall performance of Aerospike and MongoDB across the different operations, highlighting which database excels on the average transaction time based on our experimental study in each CRUD operation for a single record and a bulk batch of records. ​ However, we believe our detailed microanalysis given below would serve as valuable insights in making the right decision of choosing the preferred database to suit the business context. Table 3 Summary of CRUD operation performance comparison between MongoDB and Aerospike CRUD Operations A Single Record Bulk Records C – Create MongoDB excels Aerospike excels R – Read Aerospike excels Aerospike excels U – Update MongoDB excels MongoDB excels D – Delete MongoDB excels MongoDB excels The comparison of average read and write transactions time for single record and bulk record operations to evaluate the performance disparities between Aerospike and MongoDB is illustrated in Fig. 2.1 to Fig. 2.8 . For single read operation, MongoDB is the unequivocal lead outperformed Aerospike, with average times consistently under 40 msec, which is a fraction of the time taken by Aerospike, averaging well over 4,000 msec. This gap highlights that the MongoDB is highly read efficient, potentially due to robust indexing system and in-memory storage engine that allow for rapid data retrieval. For single write operations (create, update, delete), the average transaction times across Create, Update, and Delete operations demonstrates that Aerospike maintains a significant advantage. The transaction time of Aerospike is significantly lower and showed less inconsistency between runs compared to MongoDB. Whereas MongoDB initial average write transaction time is considerably higher but improved in later. This signifies a potential for write performance optimization for MongoDB over time. But, with this improved time, still MongoDB does not consistently match the lower average write transaction time of Aerospike. For bulk read operations for Aerospike and MongoDB performance, the difference between the two average read transaction time is stark. The average read transaction time for MongoDB is exceedingly low compared to the time of Aerospike and remained almost consistent across the runs with only a marginal increase which highlights MongoDB highly efficient read operations. However, in case of bulk write operations, MongoDB on average outpaced Aerospike in contrast to single write operations. Aerospike demonstrates a gradual increase in average write transaction times with each run, specifically in the Create and Update operations, which might be the relative performance degradation under extended load. According to the results, MongoDB, consistently demonstrating read times that are significantly lower than the Aerospike in both scenarios, is the superior choice for read intensive workload. The average bulk read transaction time in MongoDB demonstrate an enhanced performance, exceedingly even the average time observed in its single record read transaction in first scenario. It provides improved performance for read-intensive applications whether handle one record at a time or large volumes of data. It also demonstrates that the performance profile for write transaction depends upon the single record operation or bulk batch operation. The result suggests that Aerospike may be better suited for write-heavy application which handles single record transaction, whereas MongoDB stands out in write operation for write-heavy application which process multiple records in a single operation i.e. bulk batch. This shows that architecture and optimization strategies of Aerospike does not scale as effectively as of MongoDB when handling bulk batch i.e. large volume of data. The primary focus of this study is to identify sustainable success factor through tailored benchmarking process that compares the performance of two non-relational databases from two different categories, with an emphasis of comparing performance using transaction time. In this study, experiments are performed under meticulously controlled environment and eliminates as many variables as possible to maintain uniformity in database settings of the experiment to ensure a fair and accurate comparison. However, certain intrinsic variables, such as database cache sizes and memory storage capacities remain unavoidable due to the characteristics and operational mechanisms of the non-relational databases. Furthermore, the benchmark is executed utilizing comparatively small dataset, but it allowed us to apply our benchmark workflow model to compare the performance of each non-relational database to provide valuable insights. 6. Conclusion The selection of an optimal database from high preforming application from the vast array of options poses a significant challenge. Numerous factors including performance, flexibility, and scalability impacts the selections. The disadvantages faced in standard benchmark tools have motivated us to propose a benchmark workflow model that could be customised for performing a comparative study to choose an optimal database suitable to meet a requirement. This paper has provided an experimental study by applying the model to conduct a comparison of performance of two prominent non-relational databases, Aerospike and MongoDB, with average transaction time addressing the critical need. Our findings highlighted the advantages of MongoDB in reads involving single record and bulk batch transactions and Aerospike in write operations, particularly in the scenarios of single record transactions. The experiment conducted in this study, focusing on transactional speed with constrained datasets, should not be the sole factor in evaluating Aerospike and MongoDB overall performance. When selecting a database, considerations must extend beyond mere average transaction time, considering how the features of the database fit to application requirements. The most optimum database selection hinges on its performance and capabilities being closely matched to the application unique demands, functional and non-functional requirements. In this study, we applied our benchmark workflow model to conduct our experimental study with certain limitations such as using a relatively small dataset and avoiding too many variables. However, our purpose was to provide valuable insights on the measures of performance and these limitations necessitate further research in this direction. In future, we could explore a broader range of datasets, different non-relational databases, and more diverse scenarios to further explore the capabilities and optimal applications of different non-relational databases and to refine our benchmarking model in various problem domains, such as in addressing data issues in higher education (Fahd et al. 2021 ; Fahd, Miah and Ahmad, 2021; Muhammad, Isa, Samsudin, and Miah, 2020 ), healthcare or clinical support (Aljarboa and Miah, 2023 ; Miah, Hasan, and Gammack, 2019 ; Miah, Blake, and Kerr, 2020 ) and other businesses such as SMEs, agricultural or government decision support (Shee et al. 2020 ; Almtiri, Miah, and Noman, 2022 ; Miah, Vu and Gammack, 2019 ; Miah, Debuse and Kerr, 2021 ). We believe this paper encourages further ongoing big data research in this direction that can lead to the betterment of benchmarking for enhancing the insights on the suitability of each non-relational database for specific application requirements and business settings. Declarations Ethical Approval - Not applicable Funding – Not applicable Availability of data and materials - Not applicable Conflict of Interests - None Author Contribution All authors contribute as follows:Kiran Fahd - 60% effort Sitalakshmi Venkatraman - 20% effortSazia Parvin - 20% effortShah J Miah - 20% effort References Aljarboa S, Miah SJ. Acceptance of clinical decision support systems in Saudi healthcare organisations. Inform Dev. 2023;39(1):86–106. Almtiri ZHA, Miah SJ, Noman N. (2022). Application of E-Commerce Technologies in accelerating the Success of SME Operation, In the Proceedings of the Seventh International Congress on Information and Communication Technology, 463–470, accessed form https://link.springer.com/chapter/ 10.1007/978-981-19-1610-6_40 . Alyasiri B, Sahi B, AL-Khafaji N. Will it be an alternative to a relational database? MySQL vs MongoDB comparison. Sakarya, Turkey: IMDC-IST; 2022. Balusamy B, Abirami NR, Kadry S, Gandomi AH. Introduction to the World of Big Data. Big Data. John Wiley & Sons, Inc.; 2021. Čerešňák R, Kvet M. (2019, 2019/05/29). Comparison of query performance in relational and non-relation databases. Proceedings of the 13th International Scientific Conference on Sustainable, Modern and Safe Transport (TRANSCOM 2019), High Tatras, Novy Smokovec – Grand Hotel Bellevue, Slovak Republic. Cui X, Chen W. (2021). Performance Comparison Test of HBase and Cassandra Based on YCSB . https://doi.org/10.1109/ICIS51600.2021.9516864 . Fahd K, Miah SJ, Ahmed K, Venkatraman S, Miao Y. Integrating design science research and design based research frameworks for developing education support systems. Educ Inform Technol. 2021;26:4027–48. Fahd K, Miah SJ, Ahmed K. Predicting student performance in a blended learning environment using learning management system interaction data. Appl Comput Inf. 2021. https://doi.org/10.1108/ACI-06-2021-0150 . Gembalczyk D, Schuhknecht FM, Dittrich J. (2017). An Experimental Analysis of Different Key-Value Stores and Relational Databases. In Datenbanksysteme für Business, Technologie und Web (BTW 2017) . Győrödi CA, Dumşe-Burescu DV, Zmaranda DR, Győrödi RŞ, Gabor GA, Pecherle GD. Performance Analysis of NoSQL and Relational Databases with CouchDB and MySQL for Application’s Data Storage. Appl Sci. 2020;10:8524. https://doi.org/10.3390/app10238524 . Khan W, Ahmad W, Luo B, Ahmed E. (2019). SQL Database with physical database tuning technique and NoSQL graph database comparisons. 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). Khan, W., Kumar, T., Zhang, C., Raj, K., Roy, A. M., & Luo, B. (2023). SQL and NoSQL Database Software Architecture Performance Analysis and Assessments—A Systematic Literature Review. Big Data and Cognitive Computing, 7 (2). Kolonko K. Performance comparison of the most popular relational and non-relational database management systems. Blekinge Institute of Technology]; 2018. Landuyt DV, Benaouda J, Reniers V, Rafique A, Joosen W. (2023). A Comparative Performance Evaluation of Multi-Model NoSQL Databases and Polyglot Persistence Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing, Tallinn, Estonia. https://doi.org/10.1145/3555776.3577645 . Li J, Gai R. (2021). Survey of Performance Comparison Based on Non Relational Database . https://doi.org/10.1109/HPCC-DSS-SmartCity-DependSys53884.2021.00342 . Miah SJ, Blake J, Kerr D. Meta-design Knowledge for Clinical Decision Support Systems. Australasian J Inform Syst. 2020;24:1–21. Miah SJ, Debuse J, Kerr D. A development-oriented DSS evaluation approach: A case demonstration for conceptual assessment. Australasian J Inform Syst. 2021;17(2):43–55. Miah SJ, Vu HQ, Gammack JG. A Big-Data Analytics Method for capturing visitor activities and flows: the case of an Island Country. Inf Technol Manage. 2019;20(4):203–21. Miah SJ, Hasan N, Gammack JG. A Methodological Requirement for designing Healthcare Analytics Solution: A Literature Analysis. Health Inf J. 2019;26(4):2300–14. Mohamed M, Altrafi O, Ismail M. 05/01). Relational Vs. NoSQL databases: A survey. Int J Comput Inform Technol (IJCIT). 2014;03:598. Muhammad JS, Isa AM, Samsudin AZH, Miah SJ. Critical factors for implementing effective information governance in Nigerian universities: A case study investigation. Educ Info Technol. 2020;25:5565–80. 10.1007/s10639-020-10206-3 . Ouyang H, Wei H, Huang Y. (2021). Checking Causal Consistency of MongoDB Proceedings of the 12th Asia-Pacific Symposium on Internetware, Singapore, Singapore. https://doi.org/10.1145/3457913.3457928 . Shee H, Miah SJ, Taboada I, De Vass T. (2020). Smart city–smart logistics amalgamation, In the proceedings of the 2020 IEEE European Technology and Engineering Management Summit (E-TEMS), 1–4, 10.1109/E-TEMS46250.2020.9111852 . Sivarajah U, Kamal MM, Irani Z, Weerakkody V. (2017). Critical analysis of Big Data challenges and analytical methods. J Bus Reserach. Srinivasan V, Gooding A, Sayyaparaju S, Lopatic T, Porter K, Shinde A, Narendran B. (2023). Techniques and Efficiencies from Building a Real-Time DBMS. Proc. VLDB Endow., 16 (12), 3676–3688. https://doi.org/10.14778/3611540.3611556 . Venkatraman S, Fahd K, Kaspi S, Venkatraman R. SQL Versus NoSQL Movement with Big Data Analytics. Int J Inform Technol Comput Sci. 2016;8:59–66. https://doi.org/10.5815/ijitcs.2016.12.07 . Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4478249","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Short Report","associatedPublications":[],"authors":[{"id":307133531,"identity":"129d0c98-3d2c-4cff-aaef-b40caacdafd9","order_by":0,"name":"Shah Miah","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABFUlEQVRIie2OMUvDUBDH73GQLCddnxSar/BCIRFE81USMnRJHLtYNFDQJR8g0C+RKeCWEkiW0DmiQ0RwE9pFBBVMqps+zOjwfnA33N2P+wMoFP8RDtq+Cz36nuh904YolH8NCIcpAIK7AxVjtXxqWXBk2IfP60danDoOYg7beQGjxP1VEfelLVjGzZvVmT+l0vdi1FyWbArgjUThrsU7haV3gTUOI3QJSeDBVQEgUYxk9tIrTnpb229hdOkQjrb40SmGRIEm2H/x0oYsDKOCxUiArFOELFgTzLmXcT+tg+n4vay8uNDEOt7MyKxbWbCM77KLk7SqzV2yOHf06+VD+zo/nkwqSbCeH6u8K5LfKxQKheIvPgGdYFiowH+kwAAAAABJRU5ErkJggg==","orcid":"","institution":"University of Newcastle Australia","correspondingAuthor":true,"prefix":"","firstName":"Shah","middleName":"","lastName":"Miah","suffix":""},{"id":307133532,"identity":"2d07a21e-33f3-41a4-8f00-ad244a28dc23","order_by":1,"name":"Kiran Fahd","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Kiran","middleName":"","lastName":"Fahd","suffix":""},{"id":307133533,"identity":"8b48726b-1327-46ca-a31e-5f6c0161bd19","order_by":2,"name":"Sitalakshmi Venkatraman","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Sitalakshmi","middleName":"","lastName":"Venkatraman","suffix":""},{"id":307133534,"identity":"b0fd1f3b-1bef-4bcf-8544-32366cb8bcad","order_by":3,"name":"Sazia Parvin","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Sazia","middleName":"","lastName":"Parvin","suffix":""}],"badges":[],"createdAt":"2024-05-26 00:23:22","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4478249/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4478249/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":57913480,"identity":"7792864c-a7a8-46e1-b155-6029a9328748","added_by":"auto","created_at":"2024-06-07 11:24:52","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":43229,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eBenchmark Workflow Model of the experiment\u003c/em\u003e\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-4478249/v1/9ade0eedbd1983776774e7fd.png"},{"id":57913486,"identity":"c4eb52e5-54e5-4edc-a315-59f78bce1093","added_by":"auto","created_at":"2024-06-07 11:24:53","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":26488,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eFigure 2.1: Create operation average transaction time in milliseconds for single record\u003c/em\u003e\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-4478249/v1/96386450d6a800b7016d6764.png"},{"id":57913141,"identity":"a6e95966-c10f-4300-898f-81e16494e809","added_by":"auto","created_at":"2024-06-07 11:16:52","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":24796,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eFigure 2.2: Create operation average transaction time in milliseconds for bulk records\u003c/em\u003e\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-4478249/v1/e436072e6598f9104be66f49.png"},{"id":57913146,"identity":"f7550f12-317b-46d4-bd8e-edeedadedd43","added_by":"auto","created_at":"2024-06-07 11:16:52","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":23843,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eFigure 2.3: Single update operation average transaction time in milliseconds\u003c/em\u003e\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-4478249/v1/3194cdf2d6c2f4d73b4a9b0e.png"},{"id":57914055,"identity":"16a3ce01-dcf3-4c4b-9c80-2cda57d4b34c","added_by":"auto","created_at":"2024-06-07 11:32:52","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":21946,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eFigure 2.4: Bulk update operation average transaction time in milliseconds\u003c/em\u003e\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-4478249/v1/e2f17f67ec103d1ab7dc3fc4.png"},{"id":57913483,"identity":"98aa633c-48ff-459d-93ea-98332549ee27","added_by":"auto","created_at":"2024-06-07 11:24:52","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":23065,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eFigure 2.5: Single delete operation average transaction time in milliseconds\u003c/em\u003e\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-4478249/v1/03aa2fa0c5ff51bfabec480b.png"},{"id":57914054,"identity":"31ac255e-3404-44d2-8aeb-fb758962c51a","added_by":"auto","created_at":"2024-06-07 11:32:52","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":22046,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eFigure 2.6: Bulk delete operation average transaction time in milliseconds\u003c/em\u003e\u003c/p\u003e","description":"","filename":"7.png","url":"https://assets-eu.researchsquare.com/files/rs-4478249/v1/8a614713f5fea2d518a4cc32.png"},{"id":57913147,"identity":"d2a49da9-a484-4b17-acf0-2da2866c5c97","added_by":"auto","created_at":"2024-06-07 11:16:55","extension":"png","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":28475,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eFigure 2.7: Read operation performance in milliseconds for a single record\u003c/em\u003e\u003c/p\u003e","description":"","filename":"8.png","url":"https://assets-eu.researchsquare.com/files/rs-4478249/v1/d720f61885afbb4bcccdfed4.png"},{"id":57914057,"identity":"4bc8beee-2cc7-4806-be01-c5bd6361b0c8","added_by":"auto","created_at":"2024-06-07 11:32:52","extension":"png","order_by":9,"title":"Figure 9","display":"","copyAsset":false,"role":"figure","size":31219,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eFigure 2.8: Read operation performance in milliseconds for bulk records\u003c/em\u003e\u003c/p\u003e","description":"","filename":"9.png","url":"https://assets-eu.researchsquare.com/files/rs-4478249/v1/6b9102c81164791145fd9499.png"},{"id":60231052,"identity":"c4cc0e84-5779-4654-a411-950f3090821a","added_by":"auto","created_at":"2024-07-13 19:37:22","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":645082,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4478249/v1/9e857079-554c-4d5c-b07d-50c1e78a8069.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"An Innovative Comparative Performance Analysis on Document Store Non- Relational Databases","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eThe database landscape has become complicated tasks as choosing the right database from a range of options is crucial for a successful software development. Therefore existing literature recognised that determining the optimum database solution for a software application within a business context is a multifaceted challenge for practitioners and researchers (Jones, 2022). With a myriad of existing databases, the selection process of an optimum database solution becomes significantly complex due to the requirement to explore options from an ever-growing set of diverse database types. Big data researchers need to consider several parameters when deciding which database to choose. Some of the key properties to consider are data Model, CAP support, multi data center support, capacity, performance, query API, reliability, and data persistence. Currently, there are hundreds of options that require evaluation of a number of database characteristics, and the recent focus is on considering sustainable success factors such as flexible data structure, performance, and scalability. In this paper, we confine our study to the evaluation of performance measures of databases.\u003cdiv class=\"BlockQuote\"\u003e\u003cp\u003eAmong the plethora of available database types, the main two types that developers/researchers explore are: i) traditional relational databases that guarantee the consistency and integrity of the structured data (e.g. Postgres and Oracle), and ii) dynamic/modern non-relational databases (aka NoSQL database) that emphasize schema flexibility and efficiency in data retrieval (e.g. DynamoDB and MongoDB). Non-relational database relies on the BASE (Basically Available, Soft state, Eventual consistency) framework, where eventual consistency means the data across all the nodes are to be eventually consistent instead of immediate and instantaneous consistent status which is the case with traditional databases.\u003c/p\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eModern technologies have be transformed with the advent of social media, mobile applications for enhancing their capacities. Internet of Things (IoT) and their applications are emerging rapidly in contemporary industry with notable initiatives such as smart agriculture, smart transportation, and smart cities, generate massive and complex data. These technologies leverage non-relational databases due to flexible data models and scalability to store structured, semi-structured and unstructured data. Big data term is often used to describe data generated from these modern technological platforms. Five innate characteristics of Big data, known as 5Vs, are Velocity \u0026ndash; the speed at which data is generated; Volume \u0026ndash; a large scale of the data; Variety \u0026ndash; different forms and sources of the data; Veracity \u0026ndash; reliability, quality and accuracy of the data and Value \u0026ndash; usefulness of the data (Balusamy et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Sivarajah et al., \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2017\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eBig data systems must have functionalities for managing massive volume of data at any scale. One of the main requirements to deal with big data is having the ability to scale the servers, and once again, scaling up relational database servers is very expensive and scaling it out is difficult. This is the central interests of many big data researchers. However, non-relational databases were designed to handle the big data and because of that methods were implemented to improve performance when retrieving and storing data (Mohamed et al., \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2014\u003c/span\u003e). For big data processing, a variety of frameworks are employed, e.g. Hadoop/MapReduce, Kafka, and Spark. Due the vast options of non-relational databases, the selection become even more nuanced. For example, key value store such as Aerospike offers rapid and direct access to data, whereas document stores such as MongoDB are best suited for scenarios where unparalleled flexibility in managing semi-structured data is of prime importance. To meet the increasing demand for diverse data models, it is critical to select appropriate type of database.\u003c/p\u003e \u003cp\u003eThis paper investigates the benchmarking process of comparing database performance with a focus on transaction speed and responsiveness across two different non-relational databases. Through a tailored benchmarking and analysis approach, the paper aims to provide value insights to guide the decision-making process for selecting the right type of non-relational database. There are several disadvantages of using standard tools such as YCSB and they have been reported with an inherent throughput issue (Gembalczyk et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2017\u003c/span\u003e). Due to their inability to provide flexibility, scalability and extensibility, custom benchmark tools are being developed. Our contribution uniquely differentiates from other comparative studies found in the literature as we propose a comprehensive tailored approach to benchmarking for arriving at an optimal database for an application.\u003c/p\u003e \u003cp\u003eThe remainder of this paper is organized as follows. First, we present a systematic review of related work to provide the research background and the gaps in literature related to studies conducted on the selected key value store and document store databases. Next, we outline the research methodology adopted detailing the research design and the experimental setup, with a proposed benchmark workflow model of the experiment. Following this, the paper reports the experimental results and findings utilizing visual aids to enhance data presentation. Finally, we provide concluding remarks on this research study and briefly discuss future research directions.\u003c/p\u003e"},{"header":"2. Existing relevant works","content":"\u003cp\u003eThe study recognises the significance of background studies in previous literature of performance comparison of databases, specifically related to non-relational databases. Different non-relational databases methods offer different storage, data model, schema, and features. In general, non-relational databases can be categorised into four major groups: Document store, Key Value store, Column-wide stores, and Graph stores. Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e shows the difference between the key attributes offered by Document store and Key Value store categories (Venkatraman et al., \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2016\u003c/span\u003e). Non-relational databases utilise a diverse approach to data modelling, data storage and access in comparison to relational databases especially such as availability and horizontal scalability.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eFeatures comparison of Document store and Key Value store\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eData Model\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePerformance\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eScalability\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eFlexibility\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eComplexity\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eAdvantage\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eKey-Value Stores \u003c/b\u003e\u003c/p\u003e \u003cp\u003e(e.g. Aerospike, Redis, Raik)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCollection of key-value pairs\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHigh\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHigh\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eHigh\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eNone\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eFast search\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eDocument Stores\u003c/b\u003e\u003c/p\u003e \u003cp\u003e(e.g. MongoDB, CouchDB, DocumentDB, DynamoDB)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eKey-value where value is structured document object\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHigh\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eVariable (High)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eHigh\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eLow\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eFlexible schema\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eKey value store\u003c/strong\u003e \u003cp\u003eAerospike test, uses data-in-memory storage, storing indices in the random-access memory (RAM) of the server and serves read and write operations to the disk. Aerospike, based on its name, is designed to rapidly scale in size and is best used as a cache replacement, info store, and more (Srinivasan et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e).\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eDocument store\u003c/strong\u003e \u003cp\u003eMongoDB is not a data-in-memory storage. However, it makes use of caches storing data in the RAM for fast access. This caching might show slow start for the first benchmark but would accelerate results for the subsequent runs. MongoDB is also advertised to be scalable and flexible. It is recommended to be used for catalogues, payment ledgers, and so on so forth (Ouyang et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2021\u003c/span\u003e).\u003c/p\u003e \u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e summarises the existing studies related to performance analysis of non-relational databases with a focus on document store databases and key value store databases.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSummary of existing studies about comparison of non-relational databases\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eStudy\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStudy Aim\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e(Landuyt et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2023\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis study compared query execution time, the two multi-model non-relational databases, ArangoDB and OrientDB, with a polyglot database setup, and query execution time.\u003c/p\u003e \u003cp\u003eThe study lacks an analysis of scalability and database performance under varying workloads.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e(Khan et al., 2023)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis study compared the structures, scalability, and performance of relational and non-relational databases in handling large data application.\u003c/p\u003e \u003cp\u003eThe study did not compare two non-relational databases.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e(Alyasiri et al., \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2022\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis study compares the characteristics, challenges, and practical query processing of two different databases, one non-relational database MongoDB and other relational database MySQL.\u003c/p\u003e \u003cp\u003eIt lacks an exploration of performance comparison between two non-relational databases.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e(Li \u0026amp; Gai, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2021\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis study provides comparative analysis of several non-relational database of read and write performances of MongoDB and Redis across different usage scenarios. The study highlighted the lack of detailed comparative analysis focusing on the performance aspects, especially comparison on read and write performance, of non-relational databases.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e(Cui \u0026amp; Chen, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2021\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis study compared the read and write performance of two column-wide store databases i.e. HBase and Cassandra with experimental results obtained from the Yahoo! Cloud Serving Benchmark (YCSB).\u003c/p\u003e \u003cp\u003eThe study compared two non-relational databases from same category of non-relational databases i.e. column-family stores or wide-column stores, not from two different types of NoSQL categories like key-value or document store.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eOur summary of literature review shown in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e illustrates significant gaps that we bring into researchers attention in this paper. This is to assist big data oriented application developers/researchers in selecting a suitable database method among document store and key value store databases based on their comparative performance criteria. We justify the requirements for performance comparison between two widely used non-relational databases with a narrowed focus to two specific categories of non-relational databases: document store and key-value store by providing insights that are highly relevant to applications that may prefer one database over the other based on their operational requirements.\u003c/p\u003e \u003cp\u003eThere is a need for more tailored approach by employing customized benchmarking scripts that offers flexibility and customization in the benchmarking process, allowing for tailored scenarios that are outside the scope of standard benchmarking tools. Previous studies have reported drawbacks of using standard benchmarking tools such as YCSB have throughput issues (Gembalczyk et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2017\u003c/span\u003e). Some studies have attempted for a custom benchmark tool to cater to flexibility, extensibility and scalability in comparing relational and non-relational databases. However, there is lack of studies that address these issues for comparing contemporary non-relational databases of the types, key-value store and document store. In brief, it is imperative to address the gap in the target body of literature by creating a benchmarking approach that can be customised to specifically focus on the performance comparison of key-value store and document store non-relational databases. Our study offers a focused comparison of document store and key-value non-relational databases through tailored scripts, specifically exploring transaction read and write times for real world scenarios. It precisely addresses the need for performance comparison insights in scenarios most relevant to developers/researchers, thereby filling an important gap as the right database selection for application optimization is critical for businesses.\u003c/p\u003e"},{"header":"3. Research Methodology","content":"\u003cp\u003eWe adopt a research methodology with a theoretical approach starting with a literature survey with an analysis of relevant scholarly articles as detailed in previous section, followed by an experimental study to serve as a benchmark for evaluating the performance of non-relational databases. The purpose of this approach is to develop a custom benchmarking that would aid data researchers/application developers in choosing the optimal one for an application. It\u0026rsquo;s important to offer a benchmark workflow model to address the lack of support for customisation of the experimental study in standard benchmark tools such as YCSB. The aim of this study is to apply our proposed workflow model to measure and compare the performance of two non-relational databases as a proof-of-concept. Figure\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e gives a pictorial representation of our benchmark workflow model of the experimental study adopted in this research work.\u003c/p\u003e \u003cp\u003eIn this experimental study, two representative non-relational databases are selected, each from of queries into one benchmark tool, which was not possible with YCSB and other standard benchmark tools. Hence, using our proposed benchmark workflow model, we compare two contemporary non-relational databases, namely key value store and document store due their popularity in big data applications. We have selected Aerospike as representative of key value store and MongoDB for document store as explained in previous section to examine the performance of typical database operations, write (create, update, delete) and read on different numbers of records for quantitative comparison.\u003c/p\u003e \u003cp\u003eNumerous factors like security, flexibility, scalability, and performance can affect the database selection (Győr\u0026ouml;di et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2020\u003c/span\u003e; Khan et al., \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). For example, the performance is influenced by number of the users with write access and number of records modified during write operations (Čerešň\u0026aacute;k \u0026amp; Kvet, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). It is important to identify the method used to measure and compare the database performance and database efficiency. In this study, mean transaction time, execution time of database operation for different numbers of records, in milliseconds per operation is identified as database performance measurement methods as it is considered as one of the foremost methods according to the existing literature (Kolonko, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2018\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eAn ultimate experiment is required to be executed to measure numerical average transaction time for different number of records for different database operations and their relationship is presented numerically. The member database stores information about member name and membership number data. In MongoDB, a member record for a member stored as a document within a collection with fields for each attribute, such as {\"memberName\": \"John Smith\", \"membershipNumber\": \"MVIC1234\"}. For Aerospike, a member record would consist of a set identified by a unique key within a namespace with bins, where each bin corresponds to an attribute, e.g. {\"memberName\": \"John Smith\", \"membershipNumber\": \"MVIC1234\"}.\u003c/p\u003e \u003cp\u003eIn this experiment, a python script is executed for Aerospike and MongoDB to get execution time measures for different numbers of records i.e. a single record and a bulk batch of records. This provides a broad understanding into how database performs ass the number of records increases for each database operation. A python script is executed to create a record for write or read transactions and bulk records for bulk read and write transactions. Another the python script establishes a connection with the database on server. It executes two operations conducted iteratively over 100 cycles. In first phase, it performs read and write operation for a single record and calculate the mean transaction time of read and write for single record. Then in second phase, it executes read and write transactions for mass records as bulk batch and calculates the mean transaction time for bulk batch. The script concludes by clean-up operation and closing the database connection. Time for the single server connection and clean-up operation is also calculated. The results of both scenarios are compared in the next section.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e"},{"header":"4. Experimental Results and Findings","content":"\u003cp\u003eAs mentioned earlier, we conducted experiments for executing read and write operations using a dual-machine setup comprising a client and a server over the intranet. The client operates on Arch Linux 6.1.8 and on the server side, services are deployed within Docker containers, which are hosted on the Microsoft Windows 11, version 22H2, leveraging the Windows Subsystem for Linux 2 (WSL2). The docker setup ensure compatibility and ease of deployment across different environments. Python libraries 4.3.2 and Aerospike 7.1.1 are used for client whilst MongoDB 6.0.2 and Aerospike 6.1.0.3 are used for server. Intranet provided a seamless and controlled communication channel for the execution of experiments. Therefore, external variables are minimized to maintain uniformity in database settings of the experiment to compare performance of two non-relational database.\u003c/p\u003e \u003cp\u003eThis experiment script is executed three (3) times to enhancing the reliability of the results for each database. The performance of a key value store database i.e. Aerospike and document store database i.e. MongoDB, is compared over three runs based on the average transaction time (in milliseconds(msec)) it takes to complete standard database operations specifically Create, Read, Update, and Delete (CRUD), across three consecutive runs. In this experiment, these operations are categorised into read operations and write operations; where \u0026lsquo;read transactions\u0026rsquo; includes only the Read operation, and write operations include Create, Delete and Update operations. A lower average transaction time signifies better performing database. This information is crucial when considering which database to select based on the specific needs of the applications, whether it is read heavy or a write heavy application. Following subsections outlines and analyse the results of execution time of create, update, delete, and read operations.\u003c/p\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Create Operation\u003c/h2\u003e \u003cp\u003eThe outcome of three iterations of the write create operation for one record is depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2.1\u003c/span\u003e. This figure illustrates the time, in msec, required to add one record using MongoDB and Aerospike. In this operation, a key is automatically generated, and the member\u0026rsquo;s name and membership number data are inserted as a single record. The comparison outcome shows that while Aerospike constantly required less time to execute create operation for a single record across all three runs, MongoDB exhibited improvement from the first to the second run, signifying a potential improvement in successive operations.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe result of the three iterations of the create operation shows the performance of Aerospike and MongoDB when inserting bulk records. The average execution time of the three iterations of the create operation is presented in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e2.2\u003c/span\u003e, where the time is measured in msec. The results shows that MongoDB outperformed Aerospike for bulk create operation with shorter execution times across all three runs, which highlights higher efficiency of MongoDB in handling insert operations for large datasets.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eOverall, the results of the create operation presents the average transaction time of MongoDB and Aerospike. Figures\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2.1\u003c/span\u003e and 2.2 illustrate that the average transaction time for MongoDB is consistently lower than that for Aerospike except for the first iteration for single record creation. The higher transaction time for the first iteration for single record in MongoDB could be attributed to its flexibility in schema that could require more initial setup overhead that gets averaged out with bulk records.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Update Operation\u003c/h2\u003e \u003cp\u003eThis study used two types of update operations (i) Single update operation (ii) Bulk update operation. In both update operations, the time of average transaction time of the update operations is measured in seconds. The outcome of three iterations of the write update operation for one record is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e2.3\u003c/span\u003e. Single update operation finds a record based on random key and update the associated member name and membership number information.\u003c/p\u003e \u003cp\u003eThe result of the three iterations of the bulk write update operation shows the performance of Aerospike and MongoDB when updating bulk records as depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e2.4\u003c/span\u003e. Bulk update operation mass updates the values of member name and membership number for all records in the non-relational databases. The result shows that MongoDB demonstrated a comparatively consistent performance for bulk update operation highlighting its stability in handling updates of bulk records.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eOverall, Aerospike not only consistently maintains better average transaction times for single update operation but also displays more uniform performance across all iterations for bulk update operation in comparison to MongoDB.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e4.3 Delete Operation\u003c/h2\u003e \u003cp\u003eThe outcome of the delete operations are shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e2.5\u003c/span\u003e and Fig.\u0026nbsp;\u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e2.6\u003c/span\u003e. In this study, two delete operations are used to compare the write performance of the two non-relational databases with average transaction time. Single delete operation deletes the latest record or latest key. The average transaction time in msec of single delete operation is presented in Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e2.5\u003c/span\u003e. The data indicates that Aerospike consistently executed delete operations for a single record more rapidly than MongoDB across all three runs, highlighting the efficient single deletion operation for the latest records based on keys.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eSecond bulk update operation to compare non-relational databases write performance based on deleting random set of mass records from the database. The average transaction time in msec of the bulk delete operation is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e2.6\u003c/span\u003e. The comparative outcome demonstrates that Aerospike constantly executed delete operation for bulk records in less time than MongoDB across all three runs, which shows its efficiency in handling bulk records delete operation.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e4.4 Read Operation\u003c/h2\u003e \u003cp\u003eThe read performance of both non-relational databases is presented in msec in Fig.\u0026nbsp;\u003cspan refid=\"Fig8\" class=\"InternalRef\"\u003e2.7\u003c/span\u003e and 2.8. Single read operation read a record with random key. Figure\u0026nbsp;\u003cspan refid=\"Fig8\" class=\"InternalRef\"\u003e2.7\u003c/span\u003e shows that MongoDB maintaining a significantly lower average transaction time for read operation for single record across all three iterations. The average transaction time of Aerospike is substantially higher and showed an increasing trend, remarkably noticeable in the third run.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eBulk read operation retrieves a substantial portion of the records stored in the database, if not all. The bulk read performance for MongoDB is better with lower transaction time when compared to Aerospike as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig9\" class=\"InternalRef\"\u003e2.8\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"5. Overall Analysis","content":"\u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e briefly summarizes the overall performance of Aerospike and MongoDB across the different operations, highlighting which database excels on the average transaction time based on our experimental study in each CRUD operation for a single record and a bulk batch of records. ​ However, we believe our detailed microanalysis given below would serve as valuable insights in making the right decision of choosing the preferred database to suit the business context.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSummary of CRUD operation performance comparison between MongoDB and Aerospike\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCRUD Operations\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eA Single Record\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eBulk Records\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eC \u0026ndash; Create\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMongoDB excels\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAerospike excels\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eR \u0026ndash; Read\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAerospike excels\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAerospike excels\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eU \u0026ndash; Update\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMongoDB excels\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMongoDB excels\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eD \u0026ndash; Delete\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMongoDB excels\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMongoDB excels\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe comparison of average read and write transactions time for single record and bulk record operations to evaluate the performance disparities between Aerospike and MongoDB is illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2.1\u003c/span\u003e to Fig.\u0026nbsp;\u003cspan refid=\"Fig9\" class=\"InternalRef\"\u003e2.8\u003c/span\u003e. For single read operation, MongoDB is the unequivocal lead outperformed Aerospike, with average times consistently under 40 msec, which is a fraction of the time taken by Aerospike, averaging well over 4,000 msec. This gap highlights that the MongoDB is highly read efficient, potentially due to robust indexing system and in-memory storage engine that allow for rapid data retrieval. For single write operations (create, update, delete), the average transaction times across Create, Update, and Delete operations demonstrates that Aerospike maintains a significant advantage. The transaction time of Aerospike is significantly lower and showed less inconsistency between runs compared to MongoDB. Whereas MongoDB initial average write transaction time is considerably higher but improved in later. This signifies a potential for write performance optimization for MongoDB over time. But, with this improved time, still MongoDB does not consistently match the lower average write transaction time of Aerospike.\u003c/p\u003e \u003cp\u003eFor bulk read operations for Aerospike and MongoDB performance, the difference between the two average read transaction time is stark. The average read transaction time for MongoDB is exceedingly low compared to the time of Aerospike and remained almost consistent across the runs with only a marginal increase which highlights MongoDB highly efficient read operations. However, in case of bulk write operations, MongoDB on average outpaced Aerospike in contrast to single write operations. Aerospike demonstrates a gradual increase in average write transaction times with each run, specifically in the Create and Update operations, which might be the relative performance degradation under extended load.\u003c/p\u003e \u003cp\u003eAccording to the results, MongoDB, consistently demonstrating read times that are significantly lower than the Aerospike in both scenarios, is the superior choice for read intensive workload. The average bulk read transaction time in MongoDB demonstrate an enhanced performance, exceedingly even the average time observed in its single record read transaction in first scenario. It provides improved performance for read-intensive applications whether handle one record at a time or large volumes of data.\u003c/p\u003e \u003cp\u003eIt also demonstrates that the performance profile for write transaction depends upon the single record operation or bulk batch operation. The result suggests that Aerospike may be better suited for write-heavy application which handles single record transaction, whereas MongoDB stands out in write operation for write-heavy application which process multiple records in a single operation i.e. bulk batch. This shows that architecture and optimization strategies of Aerospike does not scale as effectively as of MongoDB when handling bulk batch i.e. large volume of data.\u003c/p\u003e \u003cp\u003eThe primary focus of this study is to identify sustainable success factor through tailored benchmarking process that compares the performance of two non-relational databases from two different categories, with an emphasis of comparing performance using transaction time. In this study, experiments are performed under meticulously controlled environment and eliminates as many variables as possible to maintain uniformity in database settings of the experiment to ensure a fair and accurate comparison. However, certain intrinsic variables, such as database cache sizes and memory storage capacities remain unavoidable due to the characteristics and operational mechanisms of the non-relational databases. Furthermore, the benchmark is executed utilizing comparatively small dataset, but it allowed us to apply our benchmark workflow model to compare the performance of each non-relational database to provide valuable insights.\u003c/p\u003e"},{"header":"6. Conclusion","content":"\u003cp\u003eThe selection of an optimal database from high preforming application from the vast array of options poses a significant challenge. Numerous factors including performance, flexibility, and scalability impacts the selections. The disadvantages faced in standard benchmark tools have motivated us to propose a benchmark workflow model that could be customised for performing a comparative study to choose an optimal database suitable to meet a requirement. This paper has provided an experimental study by applying the model to conduct a comparison of performance of two prominent non-relational databases, Aerospike and MongoDB, with average transaction time addressing the critical need.\u003c/p\u003e \u003cp\u003eOur findings highlighted the advantages of MongoDB in reads involving single record and bulk batch transactions and Aerospike in write operations, particularly in the scenarios of single record transactions. The experiment conducted in this study, focusing on transactional speed with constrained datasets, should not be the sole factor in evaluating Aerospike and MongoDB overall performance. When selecting a database, considerations must extend beyond mere average transaction time, considering how the features of the database fit to application requirements. The most optimum database selection hinges on its performance and capabilities being closely matched to the application unique demands, functional and non-functional requirements.\u003c/p\u003e \u003cp\u003eIn this study, we applied our benchmark workflow model to conduct our experimental study with certain limitations such as using a relatively small dataset and avoiding too many variables. However, our purpose was to provide valuable insights on the measures of performance and these limitations necessitate further research in this direction. In future, we could explore a broader range of datasets, different non-relational databases, and more diverse scenarios to further explore the capabilities and optimal applications of different non-relational databases and to refine our benchmarking model in various problem domains, such as in addressing data issues in higher education (Fahd et al. \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Fahd, Miah and Ahmad, 2021; Muhammad, Isa, Samsudin, and Miah, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2020\u003c/span\u003e), healthcare or clinical support (Aljarboa and Miah, \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Miah, Hasan, and Gammack, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Miah, Blake, and Kerr, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2020\u003c/span\u003e) and other businesses such as SMEs, agricultural or government decision support (Shee et al. \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e2020\u003c/span\u003e; Almtiri, Miah, and Noman, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Miah, Vu and Gammack, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Miah, Debuse and Kerr, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). We believe this paper encourages further ongoing big data research in this direction that can lead to the betterment of benchmarking for enhancing the insights on the suitability of each non-relational database for specific application requirements and business settings.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthical Approval - Not applicable\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding \u0026ndash; Not applicable\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and materials - Not applicable\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConflict of Interests - None\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor Contribution\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll authors contribute as follows:Kiran Fahd - 60% effort Sitalakshmi Venkatraman - 20% effortSazia Parvin - 20% effortShah J Miah - 20% effort\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eAljarboa S, Miah SJ. Acceptance of clinical decision support systems in Saudi healthcare organisations. Inform Dev. 2023;39(1):86\u0026ndash;106.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlmtiri ZHA, Miah SJ, Noman N. (2022). Application of E-Commerce Technologies in accelerating the Success of SME Operation, In the Proceedings of the Seventh International Congress on Information and Communication Technology, 463\u0026ndash;470, accessed form \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://link.springer.com/chapter/\u003c/span\u003e\u003cspan address=\"https://link.springer.com/chapter/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/978-981-19-1610-6_40\u003c/span\u003e\u003cspan address=\"10.1007/978-981-19-1610-6_40\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlyasiri B, Sahi B, AL-Khafaji N. Will it be an alternative to a relational database? MySQL vs MongoDB comparison. Sakarya, Turkey: IMDC-IST; 2022.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBalusamy B, Abirami NR, Kadry S, Gandomi AH. Introduction to the World of Big Data. Big Data. John Wiley \u0026amp; Sons, Inc.; 2021.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eČerešň\u0026aacute;k R, Kvet M. (2019, 2019/05/29). Comparison of query performance in relational and non-relation databases. Proceedings of the 13th International Scientific Conference on Sustainable, Modern and Safe Transport (TRANSCOM 2019), High Tatras, Novy Smokovec \u0026ndash; Grand Hotel Bellevue, Slovak Republic.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCui X, Chen W. (2021). \u003cem\u003ePerformance Comparison Test of HBase and Cassandra Based on YCSB\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/ICIS51600.2021.9516864\u003c/span\u003e\u003cspan address=\"10.1109/ICIS51600.2021.9516864\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFahd K, Miah SJ, Ahmed K, Venkatraman S, Miao Y. Integrating design science research and design based research frameworks for developing education support systems. Educ Inform Technol. 2021;26:4027\u0026ndash;48.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFahd K, Miah SJ, Ahmed K. Predicting student performance in a blended learning environment using learning management system interaction data. Appl Comput Inf. 2021. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1108/ACI-06-2021-0150\u003c/span\u003e\u003cspan address=\"10.1108/ACI-06-2021-0150\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGembalczyk D, Schuhknecht FM, Dittrich J. (2017). An Experimental Analysis of Different Key-Value Stores and Relational Databases. In \u003cem\u003eDatenbanksysteme f\u0026uuml;r Business, Technologie und Web (BTW 2017)\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGyőr\u0026ouml;di CA, Dumşe-Burescu DV, Zmaranda DR, Győr\u0026ouml;di RŞ, Gabor GA, Pecherle GD. Performance Analysis of NoSQL and Relational Databases with CouchDB and MySQL for Application\u0026rsquo;s Data Storage. Appl Sci. 2020;10:8524. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/app10238524\u003c/span\u003e\u003cspan address=\"10.3390/app10238524\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKhan W, Ahmad W, Luo B, Ahmed E. (2019). SQL Database with physical database tuning technique and NoSQL graph database comparisons. 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKhan, W., Kumar, T., Zhang, C., Raj, K., Roy, A. M., \u0026amp; Luo, B. (2023). SQL and NoSQL Database Software Architecture Performance Analysis and Assessments\u0026mdash;A Systematic Literature Review. \u003cem\u003eBig Data and Cognitive Computing, 7\u003c/em\u003e(2).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKolonko K. Performance comparison of the most popular relational and non-relational database management systems. Blekinge Institute of Technology]; 2018.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLanduyt DV, Benaouda J, Reniers V, Rafique A, Joosen W. (2023). \u003cem\u003eA Comparative Performance Evaluation of Multi-Model NoSQL Databases and Polyglot Persistence\u003c/em\u003e Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing, Tallinn, Estonia. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3555776.3577645\u003c/span\u003e\u003cspan address=\"10.1145/3555776.3577645\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi J, Gai R. (2021). \u003cem\u003eSurvey of Performance Comparison Based on Non Relational Database\u003c/em\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/HPCC-DSS-SmartCity-DependSys53884.2021.00342\u003c/span\u003e\u003cspan address=\"10.1109/HPCC-DSS-SmartCity-DependSys53884.2021.00342\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMiah SJ, Blake J, Kerr D. Meta-design Knowledge for Clinical Decision Support Systems. Australasian J Inform Syst. 2020;24:1\u0026ndash;21.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMiah SJ, Debuse J, Kerr D. A development-oriented DSS evaluation approach: A case demonstration for conceptual assessment. Australasian J Inform Syst. 2021;17(2):43\u0026ndash;55.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMiah SJ, Vu HQ, Gammack JG. A Big-Data Analytics Method for capturing visitor activities and flows: the case of an Island Country. Inf Technol Manage. 2019;20(4):203\u0026ndash;21.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMiah SJ, Hasan N, Gammack JG. A Methodological Requirement for designing Healthcare Analytics Solution: A Literature Analysis. Health Inf J. 2019;26(4):2300\u0026ndash;14.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMohamed M, Altrafi O, Ismail M. 05/01). Relational Vs. NoSQL databases: A survey. Int J Comput Inform Technol (IJCIT). 2014;03:598.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMuhammad JS, Isa AM, Samsudin AZH, Miah SJ. Critical factors for implementing effective information governance in Nigerian universities: A case study investigation. Educ Info Technol. 2020;25:5565\u0026ndash;80. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s10639-020-10206-3\u003c/span\u003e\u003cspan address=\"10.1007/s10639-020-10206-3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOuyang H, Wei H, Huang Y. (2021). \u003cem\u003eChecking Causal Consistency of MongoDB\u003c/em\u003e Proceedings of the 12th Asia-Pacific Symposium on Internetware, Singapore, Singapore. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3457913.3457928\u003c/span\u003e\u003cspan address=\"10.1145/3457913.3457928\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShee H, Miah SJ, Taboada I, De Vass T. (2020). Smart city\u0026ndash;smart logistics amalgamation, In the proceedings of the 2020 IEEE European Technology and Engineering Management Summit (E-TEMS), 1\u0026ndash;4, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/E-TEMS46250.2020.9111852\u003c/span\u003e\u003cspan address=\"10.1109/E-TEMS46250.2020.9111852\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSivarajah U, Kamal MM, Irani Z, Weerakkody V. (2017). Critical analysis of Big Data challenges and analytical methods. J Bus Reserach.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSrinivasan V, Gooding A, Sayyaparaju S, Lopatic T, Porter K, Shinde A, Narendran B. (2023). Techniques and Efficiencies from Building a Real-Time DBMS. \u003cem\u003eProc. VLDB Endow., 16\u003c/em\u003e(12), 3676\u0026ndash;3688. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.14778/3611540.3611556\u003c/span\u003e\u003cspan address=\"10.14778/3611540.3611556\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVenkatraman S, Fahd K, Kaspi S, Venkatraman R. SQL Versus NoSQL Movement with Big Data Analytics. Int J Inform Technol Comput Sci. 2016;8:59\u0026ndash;66. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.5815/ijitcs.2016.12.07\u003c/span\u003e\u003cspan address=\"10.5815/ijitcs.2016.12.07\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"NoSQL, Non-relational database, Performance comparison, Document Store, Key Value Store, MongoDB, Aerospike","lastPublishedDoi":"10.21203/rs.3.rs-4478249/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4478249/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe crucial role of competent software architecture is essential in managing the challenging of big data processing for both relational and nonrelational databases. Relational databases are designed to structure data for facilitating vertical scalability, while non-relational databases excel in handling vast volumes of unstructured data for enhancing horizontal scalability. Choosing the right database paradigm is determined by the needs of the organization, yet selecting the best option may often be a challenging task. Large number of applications still use relational databases due to its benefits of reliability, flexibility, robustness, and scalability. However, with the rapid growth in web and mobile applications as well as huge amounts of complex unstructured data generated via online and offline platforms, nonrelational databases are compensating for the inefficiency of relational databases.\u003c/p\u003e \u003cp\u003eSince selecting the right nonrelational database method for high performing applications from a plethora of possibilities is a challenging task, existing studies are still at emergent stage to compare the performance of different popular nonrelational databases. This paper introduces a novel benchmarking approach for tailoring the comparative study of nonrelational databases. To illustrate our approach, we compare two leading non-relational databases, Aerospike and MongoDB, focusing on their average transaction times to evaluate the database performance. Our comprehensive analysis reveals the strengths of each database in read and write operations for single record and bulk record batch transactions.\u003c/p\u003e","manuscriptTitle":"An Innovative Comparative Performance Analysis on Document Store Non- Relational Databases","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-06-07 11:16:47","doi":"10.21203/rs.3.rs-4478249/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"ef031640-3f59-4955-8427-01522f767549","owner":[],"postedDate":"June 7th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-07-13T19:29:15+00:00","versionOfRecord":[],"versionCreatedAt":"2024-06-07 11:16:47","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4478249","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4478249","identity":"rs-4478249","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}