An Evaluation-Aware Systematic Literature Review of DeepLearning-Based Recommender Systems: Domain Imbalance,Dataset Bias, and Evaluation Practices

An Evaluation-Aware Systematic Literature Review of DeepLearning-Based Recommender Systems: Domain Imbalance,Dataset Bias, and Evaluation Practices | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Short Report An Evaluation-Aware Systematic Literature Review of DeepLearning-Based Recommender Systems: Domain Imbalance,Dataset Bias, and Evaluation Practices Khlood Alrassi, Yazeed Al Moaiad, Mais Alkhateeb, Mosa Alokla, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8669155/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Deep learning-based recommender systems have been widely adopted across various application domains, leading to a rapid growth in domain-specific research. Despite this expansion, researchers often face challenges when selecting appropriate application domains, identifying suitable datasets, and choosing evaluation metrics that accurately reflect real-world objectives. Existing studies primarily focus on performance improvements using a small set of well-established benchmarks, with limited attention to domain imbalance, dataset-driven bias, and gaps in evaluation practices. The findings reveal a strong concentration of research in domains, including education, e-commerce, and social media, whereas application areas, including healthcare and scholarly publication, remain comparatively underexplored. In addition, the review identifies a heavy reliance on a limited set of public benchmark datasets, such as MovieLens and Amazon, alongside a predominant use of accuracy and ranking-oriented evaluation metrics. These patterns indicated that reported model effectiveness is often influenced by dataset availability and evaluation conventions rather than genuine domain-specific requirements. By highlighting underexplored application domains, commonly used datasets, and overlooked evaluation metrics, this review provides decision-oriented and evaluation-aware insights to guide researchers in selecting research directions, choosing appropriate datasets, and designing more balanced evaluation strategies for future deep learning-based recommender systems research. Recommendation Systems Deep Learning Application Domains Datasets Evaluation metrics Systematic Literature Review Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8669155","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Short Report","associatedPublications":[],"authors":[{"id":597363542,"identity":"705f7409-1ed7-463f-8636-1795f4ff559f","order_by":0,"name":"Khlood Alrassi","email":"","orcid":"","institution":"Al-Madinah International University","correspondingAuthor":false,"prefix":"","firstName":"Khlood","middleName":"","lastName":"Alrassi","suffix":""},{"id":597363545,"identity":"a79bae87-8bf5-4488-9a2d-6f3bbfe4caed","order_by":1,"name":"Yazeed Al Moaiad","email":"","orcid":"","institution":"Al-Madinah International University","correspondingAuthor":false,"prefix":"","firstName":"Yazeed","middleName":"Al","lastName":"Moaiad","suffix":""},{"id":597363546,"identity":"2a0d670f-72b5-41db-89fc-d6d7e27355eb","order_by":2,"name":"Mais Alkhateeb","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5klEQVRIiWNgGAWjYFAC5mZmMC3BfPAAkGJsIKyFEaTFAKiFLYFkLTwGxGnRbT/YbFxQ80eOX7rnw2EeBhvZDQeYN37Ap8XsTGJz8oxjBsaSc85uAGpJM95wgK1YAq+WA4nNh3nYDBI33MgFaTmcuOEAjwF+LecfArX8M6jffyPnAVDLf5AW4x94tdwAOoy3zSDBQCKHAajlAEiLGX5bbjxsNubtMzacceeYwcE5BsnGMw+zlVngd1jyYWmeb3Ly/LObHz54U2En23e8efMNfFrQADB+GJjBJImADC2jYBSMglEwnAEAO85Pqu3IiEQAAAAASUVORK5CYII=","orcid":"","institution":"Lusail University","correspondingAuthor":true,"prefix":"","firstName":"Mais","middleName":"","lastName":"Alkhateeb","suffix":""},{"id":597363548,"identity":"dfbe94d3-c4eb-4dc4-be8f-7197a5d637ad","order_by":3,"name":"Mosa Alokla","email":"","orcid":"","institution":"Community College of Qatar.","correspondingAuthor":false,"prefix":"","firstName":"Mosa","middleName":"","lastName":"Alokla","suffix":""},{"id":597363550,"identity":"ab67a5f1-7be5-4883-8cf1-30bbf0b6c623","order_by":4,"name":"Ghalya Alwhishi","email":"","orcid":"","institution":"Concordia University","correspondingAuthor":false,"prefix":"","firstName":"Ghalya","middleName":"","lastName":"Alwhishi","suffix":""}],"badges":[],"createdAt":"2026-01-22 11:38:10","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8669155/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8669155/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":107644772,"identity":"981ecc9a-035b-4103-94a7-2b7cd8cc976d","added_by":"auto","created_at":"2026-04-23 13:56:56","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1604116,"visible":true,"origin":"","legend":"","description":"","filename":"SLRpaper.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8669155/v1_covered_fb478d0f-0396-4b1e-8ac5-994c4d8e3559.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"An Evaluation-Aware Systematic Literature Review of DeepLearning-Based Recommender Systems: Domain Imbalance,Dataset Bias, and Evaluation Practices","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Recommendation Systems, Deep Learning, Application Domains, Datasets, Evaluation metrics, Systematic Literature Review","lastPublishedDoi":"10.21203/rs.3.rs-8669155/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8669155/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Deep learning-based recommender systems have been widely adopted across various application domains, leading to a rapid growth in domain-specific research. Despite this expansion, researchers often face challenges when selecting appropriate application domains, identifying suitable datasets, and choosing evaluation metrics that accurately reflect real-world objectives. Existing studies primarily focus on performance improvements using a small set of well-established benchmarks, with limited attention to domain imbalance, dataset-driven bias, and gaps in evaluation practices. The findings reveal a strong concentration of research in domains, including education, e-commerce, and social media, whereas application areas, including healthcare and scholarly publication, remain comparatively underexplored. In addition, the review identifies a heavy reliance on a limited set of public benchmark datasets, such as MovieLens and Amazon, alongside a predominant use of accuracy and ranking-oriented evaluation metrics. These patterns indicated that reported model effectiveness is often influenced by dataset availability and evaluation conventions rather than genuine domain-specific requirements. By highlighting underexplored application domains, commonly used datasets, and overlooked evaluation metrics, this review provides decision-oriented and evaluation-aware insights to guide researchers in selecting research directions, choosing appropriate datasets, and designing more balanced evaluation strategies for future deep learning-based recommender systems research. ","manuscriptTitle":"An Evaluation-Aware Systematic Literature Review of DeepLearning-Based Recommender Systems: Domain Imbalance,Dataset Bias, and Evaluation Practices","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-02 06:31:44","doi":"10.21203/rs.3.rs-8669155/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"750a22ba-4d0a-4012-86af-5b2da01917ba","owner":[],"postedDate":"March 2nd, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-04-23T13:56:18+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-02 06:31:44","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8669155","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8669155","identity":"rs-8669155","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}