Comparative Analysis of Evaluation Methods for Generative Artificial Intelligence Systems and Development of Selection Algorithm

Comparative Analysis of Evaluation Methods for Generative Artificial Intelligence Systems and Development of Selection Algorithm | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Comparative Analysis of Evaluation Methods for Generative Artificial Intelligence Systems and Development of Selection Algorithm Aleksandr Meshkov This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8658385/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract With the development of generative artificial intelligence and the active implementation of large language models in the ubiquitous field, a very important task arises, which requires an objective evaluation of the quality of such AI systems. Traditional machine learning metrics turn out to be inapplicable, since solution responses of LLM-based solutions demonstrate high variability in wording while maintaining semantic correctness. This paper analyzes existing approaches to evaluate the quality of systems built on the basis of generative AI, such as lexical methods, semantic embeddings, hybrid approaches based on LLM-as-a-Judge and natural language inference (NLI) methods. Particular attention is paid to the development of an algorithm for selecting the optimal evaluation strategy depending on various tasks, including the latency of evaluation, the correctness and interpretability of the results, as well as the stability and reproducibility of the obtained evaluation results. For comparison, the work presents the results of various evaluation methods using the example of analyzing the accuracy and relevance of a response from an AI system on a set of 500 test examples, demonstrating a correlation with expert assessments in the range from 0.67 to 0.92, depending on the chosen approach. The proposed algorithm can be used to build a suitable evaluation process for AI systems in various domains. generative artificial intelligence large language models quality assessment metrics LLM-as-a-Judge test automation AI agents evaluation method selection algorithm Full Text Additional Declarations Acknowledgements. The author would like to thank the reviewers for their valuable feedback and suggestions. Funding. This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Competing Interests. The author declares no competing interests. Ethics Approval. Ethics approval: not applicable. This research did not involve human participants, animal subjects, or any materials requiring ethical approval. Consent to Participate. Consent to participate: not applicable. Consent to Publish. Consent to publish: not applicable. Data Availability. The datasets generated and analyzed during the current study are available in the GitHub repository: https://github.com/meshkovQA/AI research. git Author Contributions. Aleksandr Meshkov is the sole author and is responsible for all aspects of this work, including conceptualization, methodology, data collection, analysis, visualization, and manuscript preparation. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8658385","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":607318863,"identity":"423ac4c0-ca14-43db-a4e2-24db8d486c17","order_by":0,"name":"Aleksandr Meshkov","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA9ElEQVRIiWNgGAWjYDACdgZmBoYCEKOx8QGQ4uEjqIUZpMUApPbwYTDFRrwWibQ0CZAAQS38zcyHDX8YHM4zuJFjVvk1x06GjYH54aMbeLRIHGZLTuYxOFws2fPG7LbstmSgw9iMjXPwWXOYx/gwg8HhxH72HLPbktuYgVp42KTxaZE/zP/5INBhiW0MOWbFktvqCWsxOMzDnMADsoUjLY3x47bDhLUYHga6nMcgPXFmz+HD0ozbjvOwMRPwi9zx5seSPyqsEzccb2z8+HNbtT0/e/PDx3i9DwHNYJKZB0wSVg4CdWCS8QdxqkfBKBgFo2CEAQDGZ0UVrDH8ZgAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0009-0001-4942-4061","institution":"Independent Researcher","correspondingAuthor":true,"prefix":"","firstName":"Aleksandr","middleName":"","lastName":"Meshkov","suffix":""}],"badges":[],"createdAt":"2026-01-21 10:35:52","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8658385/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8658385/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":105028022,"identity":"5a769893-eee2-4d04-9b48-02c042cb6c77","added_by":"auto","created_at":"2026-03-20 05:40:45","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1223787,"visible":true,"origin":"","legend":"","description":"","filename":"ComparativeAnalysisofEvaluationMethodsblinded.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8658385/v1_covered_adb60b8c-c250-4483-a107-ad822855574c.pdf"}],"financialInterests":"\u003cp\u003eAcknowledgements. The author would like to thank the reviewers for their valuable feedback and suggestions.\u003c/p\u003e\n\u003cp\u003eFunding. This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.\u003c/p\u003e\n\u003cp\u003eCompeting Interests. The author declares no competing interests.\u003c/p\u003e\n\u003cp\u003eEthics Approval. Ethics approval: not applicable. This research did not involve human participants, animal subjects, or any materials requiring ethical approval.\u003c/p\u003e\n\u003cp\u003eConsent to Participate. Consent to participate: not applicable.\u003c/p\u003e\n\u003cp\u003eConsent to Publish. Consent to publish: not applicable.\u003c/p\u003e\n\u003cp\u003eData Availability. The datasets generated and analyzed during the current study are available in the GitHub repository: https://github.com/meshkovQA/AI research. git\u003c/p\u003e\n\u003cp\u003eAuthor Contributions. Aleksandr Meshkov is the sole author and is responsible for all aspects of this work, including conceptualization, methodology, data collection, analysis, visualization, and manuscript preparation.\u003c/p\u003e","formattedTitle":"Comparative Analysis of Evaluation Methods for Generative Artificial Intelligence Systems and Development of Selection Algorithm","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":true,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"generative artificial intelligence, large language models, quality assessment, metrics, LLM-as-a-Judge, test automation, AI agents, evaluation method selection algorithm","lastPublishedDoi":"10.21203/rs.3.rs-8658385/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8658385/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"With the development of generative artificial intelligence and the active implementation of large language models in the ubiquitous field, a very important task arises, which requires an objective evaluation of the quality of such AI systems. Traditional machine learning metrics turn out to be inapplicable, since solution responses of LLM-based solutions demonstrate high variability in wording while maintaining semantic correctness. This paper analyzes existing approaches to evaluate the quality of systems built on the basis of generative AI, such as lexical methods, semantic embeddings, hybrid approaches based on LLM-as-a-Judge and natural language inference (NLI) methods. Particular attention is paid to the development of an algorithm for selecting the optimal evaluation strategy depending on various tasks, including the latency of evaluation, the correctness and interpretability of the results, as well as the stability and reproducibility of the obtained evaluation results. For comparison, the work presents the results of various evaluation methods using the example of analyzing the accuracy and relevance of a response from an AI system on a set of 500 test examples, demonstrating a correlation with expert assessments in the range from 0.67 to 0.92, depending on the chosen approach. The proposed algorithm can be used to build a suitable evaluation process for AI systems in various domains.","manuscriptTitle":"Comparative Analysis of Evaluation Methods for Generative Artificial Intelligence Systems and Development of Selection Algorithm","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-20 05:38:56","doi":"10.21203/rs.3.rs-8658385/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6afaa351-ce83-4a8b-8aaf-67405823ef7c","owner":[],"postedDate":"March 20th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Major revision","date":"2026-05-14T16:09:17+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-05-14T20:09:53+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-20 05:38:56","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8658385","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8658385","identity":"rs-8658385","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}