An Optimized Content Retriever from Web Articles using Large Language Models and FAISS Indexing

An Optimized Content Retriever from Web Articles using Large Language Models and FAISS Indexing | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article An Optimized Content Retriever from Web Articles using Large Language Models and FAISS Indexing Veerababu Reddy, Veeranjaneyulu N This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6411861/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Natural Language Processing (NLP) has transformed the way unstructured textual data is processed and analyzed, enabling the extraction of meaningful insights from extensive information sources. Traditional keyword-based retrieval systems often fail to capture the semantic relationships between words, leading to suboptimal search accuracy. The existing retrieval methods explore various NLP techniques, including rule-based and machine learningbased approaches, to analyze skill acquisition, taxonomy creation, and future skill predictions. However, these methods face limitations in handling large-scale datasets, maintaining contextual coherence, and providing semantically relevant responses. The proposed system addresses these challenges by integrating Large Language Models (LLMs), similarity search using FAISS indexing, text segmentation, and optimized embeddings to enhance semantic understanding. The system implements vector search techniques to improve content analysis, intelligent retrieval, and response generation. By implementing an advanced dense retrieval model, the system ensures that retrieved content is not only relevant but also contextually accurate and semantically rich. Performance evaluation is conducted using Precision, Recall, F1-score, Hit@K, BERTScore and Embedding Similarity Scores, demonstrating a 91% accuracy when compared against manually curated ground-truth answers. Artificial Intelligence (AI) Natural Language Processing (NLP) Machine Learning Large Language Models (LLMs) Information Retrieval Summarization and Contextual Analysis Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6411861","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":440627193,"identity":"6ffee761-b8d6-459e-b79d-90b43ff9980f","order_by":0,"name":"Veerababu Reddy","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABHklEQVRIie3RMWuDQBTA8SeCLgZXhWK+womQDKX4VU6Ezo4dMpwIdpHMfoxMLdkMD5JF6OrgYBFculgCKYUQehYCQkylW6H3H94dx/tNByAS/d0yIN1R8amDxHrvY4QSAib7HeEH+WHtu3nqb+qHRTmdG1QxaHB0ncKP3oJFCfpjJmFwSW6Ke9/Ot429TjtCiPdUePFtum3AyClgekkMLZ+ZTEFpVVCZcEJn+SZ2NAUBCgDUhsjLxyc7oXsmrpN05IQwvUbURJHCGD1OpIoTaaWGUT2JEchVEjtmuGz8dVJ1xPHSXRjLkyVqdu6xQSLLr+/sUN49qxSy9mi5eqTWe+2AlrVD3A+QXnp7vikGH3y596djye34jkgkEv2jvgAni21Z0IDXAQAAAABJRU5ErkJggg==","orcid":"","institution":"Vignan's Foundation for Science, Technology \u0026 Research","correspondingAuthor":true,"prefix":"","firstName":"Veerababu","middleName":"","lastName":"Reddy","suffix":""},{"id":440627194,"identity":"a18179fa-d5ba-485f-8d6e-80f1d55cbfbc","order_by":1,"name":"Veeranjaneyulu N","email":"","orcid":"","institution":"Vignan's Foundation for Science, Technology \u0026 Research","correspondingAuthor":false,"prefix":"","firstName":"Veeranjaneyulu","middleName":"","lastName":"N","suffix":""}],"badges":[],"createdAt":"2025-04-09 12:23:21","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6411861/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6411861/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":80543743,"identity":"0913ea75-0b78-4301-9846-017b9d525cf9","added_by":"auto","created_at":"2025-04-14 13:32:02","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1317400,"visible":true,"origin":"","legend":"","description":"","filename":"AnOptimizedContentRetrieverfromWebArcticlesusingLLMandFAISSIndex.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6411861/v1_covered_8ce2da56-ff0f-413f-ab65-60986f4b10bb.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"An Optimized Content Retriever from Web Articles using Large Language Models and FAISS Indexing","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Artificial Intelligence (AI), Natural Language Processing (NLP), Machine Learning, Large Language Models (LLMs), Information Retrieval, Summarization, and Contextual Analysis","lastPublishedDoi":"10.21203/rs.3.rs-6411861/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6411861/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eNatural Language Processing (NLP) has transformed the way unstructured textual data is processed and analyzed, enabling the extraction of meaningful insights from extensive information sources. Traditional keyword-based retrieval systems often fail to capture the semantic relationships between words, leading to suboptimal search accuracy. The existing retrieval methods explore various NLP techniques, including rule-based and machine learningbased approaches, to analyze skill acquisition, taxonomy creation, and future skill predictions. However, these methods face limitations in handling large-scale datasets, maintaining contextual coherence, and providing semantically relevant responses. The proposed system addresses these challenges by integrating Large Language Models (LLMs), similarity search using FAISS indexing, text segmentation, and optimized embeddings to enhance semantic understanding. The system implements vector search techniques to improve content analysis, intelligent retrieval, and response generation. By implementing an advanced dense retrieval model, the system ensures that retrieved content is not only relevant but also contextually accurate and semantically rich. Performance evaluation is conducted using Precision, Recall, F1-score, Hit@K, BERTScore and Embedding Similarity Scores, demonstrating a 91% accuracy when compared against manually curated ground-truth answers.\u003c/p\u003e","manuscriptTitle":"An Optimized Content Retriever from Web Articles using Large Language Models and FAISS Indexing","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-11 07:00:41","doi":"10.21203/rs.3.rs-6411861/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"b3988f64-c7aa-4c86-bd18-1685f20b5173","owner":[],"postedDate":"April 11th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-04-16T07:03:04+00:00","versionOfRecord":[],"versionCreatedAt":"2025-04-11 07:00:41","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6411861","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6411861","identity":"rs-6411861","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}