VisionVerse: Dynamic Video Question Answering Through Retrieval-Augmented Generation

VisionVerse: Dynamic Video Question Answering Through Retrieval-Augmented Generation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article VisionVerse: Dynamic Video Question Answering Through Retrieval-Augmented Generation Abhiram S Sajeev, Adhya Sanil Joseph, Amal Madhav T, Surekha Mariam Varghese, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4372886/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract In the digital age, video content has become a distinguished form for information sharing, entertainment, and education. However, navigating and comprehending lengthy video content can be time-consuming and challenging for users. The project introduces an innovative solution that can sway state-of-the-art language models to transform extensive video content into concise text contents, making it more accessible and user-friendly. By leveraging Retrieval Augmented Generation(RAG), it accurately condenses videos into text form, ensuring that the core message and key details are retained. This process enhances the efficiency of content consumption by providing users with a quick, readable overview of the video's contents. Furthermore, introducing an interactive chatbot that enables users to engage with the video content. Users can ask questions, seek clarifications, or ferret in deeper into specific aspects of the video. The chatbot is powered by a Large Language Model, which enables meaningful and context-aware interactions. The idea not only facilitates better understanding but also encourages active participation and knowledge retention. Also, benefits of interactive chatbot and video summarisation technologies together, offering users a dynamic and engaging means to access and interact with video content. The system employs advanced video-to-text summarisation techniques to automatically extract the most relevant information from videos. This innovation has significant potential applications in education, research, and the digital content landscape, where the efficient dissemination of information is paramount. Video summarization RAG Natural language processing Chatbots Information retrieval Human-computer interaction Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4372886","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":299215170,"identity":"d95a56ff-548e-416b-a95e-55d3d2e04068","order_by":0,"name":"Abhiram S Sajeev","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABDUlEQVRIiWNgGAWjYDACCR6GA0BKjo2ZsQHElwMRBx4QocWYn70ZrMUYrCWBgBYQSJzZcxysJRFM4tPCP7v34OGKmm2MG24AFf/cU5c+P+zwQ6AtdnK6DTgsuXMu4eCZY7eZDYBaGHueHc7deDvNAKgl2djsAA5rbuQYHGxgu80G0sLAc+BA7sbZCSAtBxK34dAiD9by7zYP2JY/B+rSDWenf8CrxQCkpbHttoRkz8EGZp4DzAny0jn4bTG8cwaope+2AT97Y8NhmQOHDTdI5xQcSDDA7Re52z3GHxu+3a5vY2Z/+PDNgTp5+dnpmz98qLCTw+l9ZABWYwAhiVAOB/INpKgeBaNgFIyCkQAAmShvLR+CYMEAAAAASUVORK5CYII=","orcid":"","institution":"","correspondingAuthor":true,"prefix":"","firstName":"Abhiram","middleName":"S","lastName":"Sajeev","suffix":""},{"id":299215176,"identity":"8cfd4efe-1a31-4042-934c-7949b5c24c32","order_by":1,"name":"Adhya Sanil Joseph","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Adhya","middleName":"Sanil","lastName":"Joseph","suffix":""},{"id":299215182,"identity":"a2c77f5b-ba5d-48f2-90ab-5a0012045237","order_by":2,"name":"Amal Madhav T","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Amal","middleName":"Madhav","lastName":"T","suffix":""},{"id":299215188,"identity":"c5a5c11f-b068-4d72-ac12-af93dbd07c8b","order_by":3,"name":"Surekha Mariam Varghese","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Surekha","middleName":"Mariam","lastName":"Varghese","suffix":""},{"id":299215203,"identity":"b477134b-5b3c-47bc-860d-27a45b30965b","order_by":4,"name":"Aby Abahai T","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Aby","middleName":"Abahai","lastName":"T","suffix":""}],"badges":[],"createdAt":"2024-05-05 19:38:32","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4372886/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4372886/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":62724024,"identity":"09bd484f-5a91-48c3-bb53-f8b411b4ccca","added_by":"auto","created_at":"2024-08-18 18:16:36","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2161336,"visible":true,"origin":"","legend":"","description":"","filename":"VisionVerseDynamicVideoQuestionAnsweringThroughRetrievalAugmentedGeneration.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4372886/v1_covered_25840b52-5d2d-4f19-9693-777ac7e7aa0e.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"VisionVerse: Dynamic Video Question Answering Through Retrieval-Augmented Generation","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Video summarization, RAG, Natural language processing, Chatbots, Information retrieval, Human-computer interaction","lastPublishedDoi":"10.21203/rs.3.rs-4372886/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4372886/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"In the digital age, video content has become a distinguished form for information sharing, entertainment, and education. However, navigating and comprehending lengthy video content can be time-consuming and challenging for users. The project introduces an innovative solution that can sway state-of-the-art language models to transform extensive video content into concise text contents, making it more accessible and user-friendly. By leveraging Retrieval Augmented Generation(RAG), it accurately condenses videos into text form, ensuring that the core message and key details are retained. This process enhances the efficiency of content consumption by providing users with a quick, readable overview of the video's contents. Furthermore, introducing an interactive chatbot that enables users to engage with the video content. Users can ask questions, seek clarifications, or ferret in deeper into specific aspects of the video. The chatbot is powered by a Large Language Model, which enables meaningful and context-aware interactions. The idea not only facilitates better understanding but also encourages active participation and knowledge retention. Also, benefits of interactive chatbot and video summarisation technologies together, offering users a dynamic and engaging means to access and interact with video content. The system employs advanced video-to-text summarisation techniques to automatically extract the most relevant information from videos. This innovation has significant potential applications in education, research, and the digital content landscape, where the efficient dissemination of information is paramount.","manuscriptTitle":"VisionVerse: Dynamic Video Question Answering Through Retrieval-Augmented Generation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-05-10 04:55:31","doi":"10.21203/rs.3.rs-4372886/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"55e7b3f0-a0e1-4089-8f3c-7cf34767bc1a","owner":[],"postedDate":"May 10th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-08-18T18:08:24+00:00","versionOfRecord":[],"versionCreatedAt":"2024-05-10 04:55:31","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4372886","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4372886","identity":"rs-4372886","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}