Understanding the Impact of Dataset Characteristics on RAG based Multi-hop QA Performance

Understanding the Impact of Dataset Characteristics on RAG-based Multi-hop QA Performance | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Understanding the Impact of Dataset Characteristics on RAG-based Multi-hop QA Performance Nimet Aksoy, Zekeriya Anıl Güven, Murat Osman Ünalır This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6968562/v2 This work is licensed under a CC BY 4.0 License Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Abstract Large language models (LLMs) have improved natural language understanding and made QA (question answering) systems more common in everyday tools and platforms. As these systems are used more frequently, their performance becomes more important, and their tolerance for error decreases. One of the biggest problems with LLMs is hallucination; when they cannot infer the answer from the context, they use huge information sources and give answers that are not supported by the text. To alleviate this problem, systems such as Retrieval-Augmented Generation (RAG) have been developed. They combine the power of language models with external information sources. In real-life questions, the answer is often not directly visible and requires reasoning between multiple pieces of information. Multi-hop QA datasets are useful for testing such systems realistically and have a structure that requires inference. However, each dataset has different characteristics that can affect performance and require different architectural requirements. In this study, we test a RAG system on three multi-hop QA datasets: HotpotQA, QASPER, and MultiHopQA. In addition to the overall results, we also conduct in-depth performance analysis by question type, difficulty level, and reasoning complexity to better understand the system behavior. The results indicate that, under the fixed RAG configuration, the system achieved its highest performance on MultiHopQA (Cosine: 0.961, BERT F1: 0.979), while QASPER was more difficult (Cosine: 0.257, BERT F1: 0.624), and HotpotQA had moderate results (Cosine: 0.641, BERT F1: 0.754). BERTScore proved more effective than cosine similarity for measuring semantic alignment in all three datasets. Artificial Intelligence and Machine Learning Information Retrieval and Management large language models retrieval-augmented generation (RAG) multi-hop question answering dataset characteristics semantic evaluation Figures Figure 1 Figure 2 Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6968562","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":479409858,"identity":"9dc6d42f-cfc7-4725-ad58-e26ed849f972","order_by":0,"name":"Nimet Aksoy","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABCklEQVRIiWNgGAWjYDACZghlAMSMDxIqbEB04wGCWg5AtDAbPDiTBtLSgF8LA0ILm+TDtsMwAdzAnJ078fEHBjtj+fYeM4mEM+ft1rYfBtpSYxONS4tlM+9mgwMMyWYGZ84YWyRU3E7ediYRqOVYWm4DDi0Gh3m3SRxgYLYxkMgxvJFw5nay2QGgFsaGw/i0bP9xgKHeRn5GjoFEYtu5ZLPzDwlq2Qb07WEzhhs5RkAtB+zMbhC2ZbPEGYPjxgZnjhUbJJxJTjC7AbQlAZ9fzp/d+KGiotpwfnvzxoc/Kuzszc6nP3zwocYGpxaoRhDBASYTwSoT8CqHA/YHINKeOMWjYBSMglEwkgAAsa5oWX518DcAAAAASUVORK5CYII=","orcid":"","institution":"Ege University","correspondingAuthor":true,"prefix":"","firstName":"Nimet","middleName":"","lastName":"Aksoy","suffix":""},{"id":479409859,"identity":"d503f36d-a04b-4975-8ba1-b775cc9f36ec","order_by":1,"name":"Zekeriya Anıl Güven","email":"","orcid":"","institution":"Bakırçay University","correspondingAuthor":false,"prefix":"","firstName":"Zekeriya","middleName":"Anıl","lastName":"Güven","suffix":""},{"id":479409860,"identity":"4f752a62-ab78-4192-94d3-f965546c9a74","order_by":2,"name":"Murat Osman Ünalır","email":"","orcid":"","institution":"Ege University","correspondingAuthor":false,"prefix":"","firstName":"Murat","middleName":"Osman","lastName":"Ünalır","suffix":""}],"badges":[],"createdAt":"2025-06-24 19:38:11","currentVersionCode":2,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-6968562/v2","doiUrl":"https://doi.org/10.21203/rs.3.rs-6968562/v2","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":98429388,"identity":"bf8c8fc7-d774-4aa9-97a2-50623385ca4f","added_by":"auto","created_at":"2025-12-17 16:43:18","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1021858,"visible":true,"origin":"","legend":"","description":"","filename":"UnderstandingtheImpactofDataset..Springer.docx","url":"https://assets-eu.researchsquare.com/files/rs-6968562/v2/2660b7a0b722745e73f2ac18.docx"},{"id":98100473,"identity":"117e12e1-517a-4c11-8b3d-e68e3bde4b23","added_by":"auto","created_at":"2025-12-12 20:22:57","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5248,"visible":true,"origin":"","legend":"","description":"","filename":"72b975f610b246f891fbe4d5d643df3f.json","url":"https://assets-eu.researchsquare.com/files/rs-6968562/v2/eac1056f863b17257f726e39.json"},{"id":98100478,"identity":"ddd0a7dd-af5e-4fae-8cc3-f0fdd83ee172","added_by":"auto","created_at":"2025-12-12 20:22:57","extension":"xml","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":148721,"visible":true,"origin":"","legend":"","description":"","filename":"72b975f610b246f891fbe4d5d643df3f1enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-6968562/v2/caad9cd050c414d70fc377eb.xml"},{"id":98430105,"identity":"909ba9c8-c15c-4b58-862e-730cc8a6335d","added_by":"auto","created_at":"2025-12-17 16:44:49","extension":"png","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":833119,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-6968562/v2/5e32edcc393c6553d59cdf9d.png"},{"id":98430502,"identity":"cf3f8d9b-b2a4-41b8-90e6-41afb4bba61e","added_by":"auto","created_at":"2025-12-17 16:45:36","extension":"png","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":46100,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-6968562/v2/d6128201c84afba3e7066522.png"},{"id":98430524,"identity":"6cb37ee1-4d43-4a15-87ca-bbf35ff96faa","added_by":"auto","created_at":"2025-12-17 16:45:37","extension":"xml","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":146318,"visible":true,"origin":"","legend":"","description":"","filename":"72b975f610b246f891fbe4d5d643df3f1structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-6968562/v2/7226168bc2b9bc38023c8c90.xml"},{"id":98100481,"identity":"620a494f-a063-429e-b7e4-386d9c3512a9","added_by":"auto","created_at":"2025-12-12 20:22:57","extension":"html","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":166610,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-6968562/v2/969a678cad26b1d9289e0c5e.html"},{"id":98100472,"identity":"1c75cd81-ced4-473d-b96f-459c9c00949c","added_by":"auto","created_at":"2025-12-12 20:22:57","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":42169,"visible":true,"origin":"","legend":"\u003cp\u003eAverage context length across datasets in characters and approximate tokens\u003c/p\u003e","description":"","filename":"chartoken.png","url":"https://assets-eu.researchsquare.com/files/rs-6968562/v2/254fa827e0d84828db8c4ae1.png"},{"id":98100477,"identity":"f92035c6-68b0-4e8f-8a9b-fd1acb68fb92","added_by":"auto","created_at":"2025-12-12 20:22:57","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":609767,"visible":true,"origin":"","legend":"\u003cp\u003eRAG Pipeline Structure Used in the Study\u003c/p\u003e","description":"","filename":"figure2.png","url":"https://assets-eu.researchsquare.com/files/rs-6968562/v2/0106ce67e7ab3a12ed75f591.png"},{"id":98444840,"identity":"fc1e27ec-d277-48b9-9272-489d6c75c23a","added_by":"auto","created_at":"2025-12-17 17:17:47","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":659201,"visible":true,"origin":"","legend":"","description":"","filename":"UnderstandingtheImpactofDataset..Springerafterrev1.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6968562/v2_covered_a6167262-7e26-4272-a48f-d543241e1fed.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eUnderstanding the Impact of Dataset Characteristics on RAG-based Multi-hop QA Performance\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"large language models, retrieval-augmented generation (RAG), multi-hop question answering, dataset characteristics, semantic evaluation","lastPublishedDoi":"10.21203/rs.3.rs-6968562/v2","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6968562/v2","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eLarge language models (LLMs) have improved natural language understanding and made QA (question answering) systems more common in everyday tools and platforms. As these systems are used more frequently, their performance becomes more important, and their tolerance for error decreases. One of the biggest problems with LLMs is hallucination; when they cannot infer the answer from the context, they use huge information sources and give answers that are not supported by the text. To alleviate this problem, systems such as Retrieval-Augmented Generation (RAG) have been developed. They combine the power of language models with external information sources. In real-life questions, the answer is often not directly visible and requires reasoning between multiple pieces of information. Multi-hop QA datasets are useful for testing such systems realistically and have a structure that requires inference. However, each dataset has different characteristics that can affect performance and require different architectural requirements. In this study, we test a RAG system on three multi-hop QA datasets: HotpotQA, QASPER, and MultiHopQA. In addition to the overall results, we also conduct in-depth performance analysis by question type, difficulty level, and reasoning complexity to better understand the system behavior. The results indicate that, under the fixed RAG configuration, the system achieved its highest performance on MultiHopQA (Cosine: 0.961, BERT F1: 0.979), while QASPER was more difficult (Cosine: 0.257, BERT F1: 0.624), and HotpotQA had moderate results (Cosine: 0.641, BERT F1: 0.754). BERTScore proved more effective than cosine similarity for measuring semantic alignment in all three datasets.\u003c/p\u003e","manuscriptTitle":"Understanding the Impact of Dataset Characteristics on RAG-based Multi-hop QA Performance","msid":"","msnumber":"","nonDraftVersions":[{"code":2,"date":"2025-12-12 20:22:52","doi":"10.21203/rs.3.rs-6968562/v2","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}},{"code":1,"date":"2025-07-02 06:18:16","doi":"10.21203/rs.3.rs-6968562/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6e5ecec5-a973-4d89-8761-88d3ba400c5d","owner":[],"postedDate":"December 12th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":59035415,"name":"Artificial Intelligence and Machine Learning"},{"id":59035416,"name":"Information Retrieval and Management"}],"tags":[],"updatedAt":"2025-12-07T13:10:43+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-12 20:22:52","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v2","identity":"rs-6968562","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6968562","identity":"rs-6968562","version":["v2"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}