Automating Multilingual SDG Event Extraction from Regional Portals Using Web Scraping and LangChain Frameworks

Automating Multilingual SDG Event Extraction from Regional Portals Using Web Scraping and LangChain Frameworks | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Automating Multilingual SDG Event Extraction from Regional Portals Using Web Scraping and LangChain Frameworks Bhawna Singla, Neha Bansal This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6483420/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract This research presents a novel, scalable, and multilingual data extraction framework designed specifically to collect and structure Sustainable Development Goal (SDG) event information from a wide range of regional portals and language-specific SDG websites. As SDG-related activities are increasingly being organized and reported by diverse stakeholders across the globe—ranging from local governments to international NGOs—event data is often dispersed across decentralized platforms, published in different languages, and presented in unstructured or semi-structured formats. Traditional data collection methods struggle to keep up with the volume, variability, and linguistic diversity of such data sources. To address these challenges, this study leverages a hybrid approach that combines web scraping techniques with the LangChain framework , which allows seamless integration of large language models (LLMs) for downstream natural language understanding tasks. The proposed automated pipeline performs end-to-end data extraction: it first scrapes event content from HTML pages, detects the source language, applies automatic translation (when necessary), and then uses prompt-based LLM reasoning to extract key event attributes (e.g., title, date, location, thematic focus). This approach not only accelerates the process of collecting and curating SDG event data but also ensures cross-lingual scalability and adaptability to region-specific formats. By enabling structured data extraction from multilingual and heterogeneous sources, the framework contributes to creating a more unified and comprehensive dataset of global SDG activities. Ultimately, this work underscores the critical role that AI-enhanced data pipelines can play in supporting evidence-based policy-making, enhancing transparency, and enabling real-time monitoring of progress toward the 2030 Agenda for Sustainable Development. Artificial Intelligence and Machine Learning Full Text Additional Declarations The authors declare potential competing interests as follows: NO Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6483420","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":445072731,"identity":"97c38588-7d84-4a10-8101-9fa749321da2","order_by":0,"name":"Bhawna Singla","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA/ElEQVRIiWNgGAWjYBACAwkog42BgfEAQwVQhBQtDAcYzpCiBQQOMLYRocVcuvmZdEENgz2fdPuFAz/nHZY3Z28+wPCjYhtOLZZzjplJzzjGkNgmc6bgYO+2w4Y7e44lMPacuY3bYTcSzG7zsDEksEnkJBzg3XaYccONHANmxjZ8WtK/3eb5x2AP0nLw75zD9kRoyTG7zdvGwNgmkX7gMG/D4USCWixn5JT/5u0D+kUih+GwzLH05A1njiUcxOcXc4n0zcY83xjs5WekP3z4psbadsPx5oMPflTg1gIF/4GYBxQjzWDuAULqoYD9AZCoI1LxKBgFo2AUjCQAAHciXPoJPstKAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0003-0825-0314","institution":"Geeta University","correspondingAuthor":true,"prefix":"","firstName":"Bhawna","middleName":"","lastName":"Singla","suffix":""},{"id":445072955,"identity":"aba46459-dda2-47d9-9bf8-b0b5a05dd1db","order_by":1,"name":"Neha Bansal","email":"","orcid":"https://orcid.org/0009-0005-4290-5080","institution":"Geeta University","correspondingAuthor":false,"prefix":"","firstName":"Neha","middleName":"","lastName":"Bansal","suffix":""}],"badges":[],"createdAt":"2025-04-19 08:23:02","currentVersionCode":1,"declarations":{"humanSubjects":true,"vertebrateSubjects":false,"conflictsOfInterestStatement":true,"humanSubjectEthicalGuidelines":true,"humanSubjectConsent":true,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-6483420/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6483420/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":81079474,"identity":"07ea11b4-20d0-4138-b71c-88f8894bb3f7","added_by":"auto","created_at":"2025-04-22 04:00:03","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":382249,"visible":true,"origin":"","legend":"","description":"","filename":"AutomatingMultilingualSDGEventExtractionfromRegionalPortalsUsingWebScrapingandLangChainFrameworks.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6483420/v1_covered_90f48609-6be7-45b0-ac62-602854f7e5c1.pdf"}],"financialInterests":"The authors declare potential competing interests as follows: NO","formattedTitle":"\u003cp\u003e\u003cstrong\u003eAutomating Multilingual SDG Event Extraction from Regional Portals Using Web Scraping and LangChain Frameworks\u003c/strong\u003e\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Geeta University","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-6483420/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6483420/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis research presents a novel, scalable, and multilingual data extraction framework designed specifically to collect and structure Sustainable Development Goal (SDG) event information from a wide range of regional portals and language-specific SDG websites. As SDG-related activities are increasingly being organized and reported by diverse stakeholders across the globe\u0026mdash;ranging from local governments to international NGOs\u0026mdash;event data is often dispersed across decentralized platforms, published in different languages, and presented in unstructured or semi-structured formats. Traditional data collection methods struggle to keep up with the volume, variability, and linguistic diversity of such data sources.\u003c/p\u003e \u003cp\u003eTo address these challenges, this study leverages a hybrid approach that combines \u003cb\u003eweb scraping techniques\u003c/b\u003e with the \u003cb\u003eLangChain framework\u003c/b\u003e, which allows seamless integration of \u003cb\u003elarge language models (LLMs)\u003c/b\u003e for downstream natural language understanding tasks. The proposed automated pipeline performs end-to-end data extraction: it first scrapes event content from HTML pages, detects the source language, applies automatic translation (when necessary), and then uses prompt-based LLM reasoning to extract key event attributes (e.g., title, date, location, thematic focus).\u003c/p\u003e \u003cp\u003eThis approach not only accelerates the process of collecting and curating SDG event data but also ensures cross-lingual scalability and adaptability to region-specific formats. By enabling structured data extraction from multilingual and heterogeneous sources, the framework contributes to creating a more unified and comprehensive dataset of global SDG activities. Ultimately, this work underscores the critical role that AI-enhanced data pipelines can play in supporting evidence-based policy-making, enhancing transparency, and enabling real-time monitoring of progress toward the 2030 Agenda for Sustainable Development.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e","manuscriptTitle":"Automating Multilingual SDG Event Extraction from Regional Portals Using Web Scraping and LangChain Frameworks","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-22 03:35:58","doi":"10.21203/rs.3.rs-6483420/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6dad3281-f082-4f01-b128-521e452eb2f6","owner":[],"postedDate":"April 22nd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":47388790,"name":"Artificial Intelligence and Machine Learning"}],"tags":[],"updatedAt":"2025-04-22T03:35:58+00:00","versionOfRecord":[],"versionCreatedAt":"2025-04-22 03:35:58","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6483420","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6483420","identity":"rs-6483420","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}