A Framework for Web-Based News Data Mining Using Crawlers and NLP Techniques

A Framework for Web-Based News Data Mining Using Crawlers and NLP Techniques | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Framework for Web-Based News Data Mining Using Crawlers and NLP Techniques P Sukumar, Robert L This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6484381/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The explosive growth of online news content offers a vast opportunity for data-driven research in web mining. This paper proposes a comprehensive framework for mining news data using web crawlers and Natural Language Processing (NLP) techniques. A customized crawler is developed to extract articles from prominent Indian news websites including India Today, The Hindu, and Indian Express, covering the period from January 2020 to April 2021. The raw data undergoes rigorous preprocessing, including tokenization, normalization, stop-word removal, and lemmatization, to produce a clean, structured corpus. The final dataset is organized into standard formats such as structured corpus files and Document-Term Matrices (DTM), facilitating downstream applications such as classification, clustering, and sentiment analysis. This framework lays the foundation for large-scale, real-time news analytics and can be adapted for multilingual or domain-specific news mining tasks. Data Mining Machine learning Text mining Data Preprocessing NLP Figures Figure 1 Figure 2 Figure 3 Figure 4 1. Introduction News media, often regarded as the Fourth Pillar of Democracy, has transitioned rapidly into the digital realm. With over 624 million internet users in India alone as of January 2021, access to online news content has become ubiquitous. As the volume of news articles published daily continues to surge, there is a growing need to efficiently collect, process, and analyze this information [10]. Web content mining—particularly focused on news data—enables the extraction of valuable insights for applications such as public sentiment tracking, misinformation detection, and event prediction [12]. However, challenges such as unstructured data formats, high noise levels, and inconsistencies in web design hinder effective analysis [6]. This paper presents a robust framework that addresses these challenges by implementing a web-based news crawler coupled with advanced NLP-driven preprocessing [16]. The system systematically collects, cleans, and organizes news data, producing high-quality corpora for downstream machine learning and text mining applications [25]. The study demonstrates the practicality and effectiveness of the framework through large-scale data collection from leading Indian news portals [14]. This paper is composed as follows. Segment II summarizes related work. Segment III presents the overall methodology. Segment IV presents the results and discussion. Finally, Segment V concludes the paper and suggests directions for future work. 2. Related Work Numerous studies have been conducted in the domain of news data extraction and web content mining, with a particular focus on web crawlers, data preprocessing, and corpus construction. Yu Linxuan et al. [26] provided an extensive survey of web crawler technologies, highlighting the limitations of fixed search modes. Wang Jihou et al. [24] designed a news-specific crawler capable of rapid content acquisition. Gunawan et al. [8] developed a distributed focused crawler, enhancing performance through improved bandwidth and storage utilization. In the area of data preprocessing, Babanejad et al. [4] proposed a structured framework leveraging word vector models. Kumar et al. [13] demonstrated a three-stage NLP pipeline for sentiment analysis on Twitter data, including HTML tag removal, tokenization, and stop-word filtering. Similarly, Chaudhary et al. [5] and Anandarajan et al. [1] emphasized the significance of cleaning and normalizing textual data using common NLP techniques. Large-scale corpus development efforts such as the Malay Chat-style-text Corpus [2] and crime-related text collections [3] have shown the practical value of curated text datasets. Lu Mengmeng et al. [15] introduced a configurable news crawler system capable of multi-source content acquisition. Hui-chang Wang et al. [11] improved URL filtering to reduce redundant crawling efforts. These prior studies collectively affirm the importance of web-based data extraction and preprocessing pipelines. Building on this foundation, the present research implements a tailored web crawler and a comprehensive NLP preprocessing framework to create a structured and analyzable corpus of Indian news content. 3. Methodology 3.1 Web Crawling for News Data Collection In this study, a domain-specific web crawler was developed to systematically extract news articles from three leading Indian news portals: India Today, The Hindu, and Indian Express. The crawler was implemented in Python and designed to operate in a multithreaded environment, allowing simultaneous requests to multiple web pages [18]. The crawling process follows three main phases: Fetching, Parsing, and Filtering. Initially, the crawler initiates from a root URL and recursively follows hyperlinks within the domain [22]. During the parsing phase, HTML content is cleaned by removing JavaScript, style sheets, and non-article elements. In the filtering phase, relevant metadata such as title, publication date, and full-text content are extracted and stored in a raw corpus [20]. The pseudocode of the crawling process is outlined in Algorithm 1 . The final output of this stage is a corpus of raw news articles stored in .txt format, which serves as the input for the preprocessing phase. Algorithm 1 Web News Crawler Input: Root URL of the news domain Output: Raw corpus of crawled news article 1. Initialize link lists and visited URLs 2. While link list is not empty and crawl limit not reached: a. Fetch current URL content b. Parse HTML to remove scripts and styles c. Extract article content, title, timestamp d. Store the article in raw corpus e. Extract and queue internal links for further crawling 3. End 3.2 Text Data Preprocessing Raw text data extracted from web pages is often noisy, containing HTML artifacts, special characters, and inconsistent formatting [7]. To ensure suitability for downstream analysis, a robust preprocessing pipeline was implemented using Python’s nltk, re, and spacy libraries [26]. The preprocessing workflow includes the following steps: Contraction Mapping: Expands contractions (e.g., “don’t” → “do not”) Tokenization: Splits sentences into individual tokens (words) Lowercasing: Normalizes all words to lowercase Punctuation and Number Removal: Cleans symbols and digits Stop Word Removal: Filters out high-frequency, low-value words (e.g., "the", "is") Lemmatization: Converts words to their base form (e.g., “running” → “run”) Algorithm 2 Text Preprocessing Pipeline Input: Raw corpus of news articles Output: Preprocessed structured corpus 1. For each .txt article in the corpus: a. Expand contractions b. Tokenize sentences into words c. Convert tokens to lowercase d. Remove punctuation, URLs, and digits e. Remove stop words f. Apply lemmatization 2. Save preprocessed corpus in both .txt and .csv formats 3.3 Data Organization for Analysis To enable statistical text analysis and machine learning applications, the preprocessed data is organized into two commonly used formats: Structured Corpus A collection of cleaned text files, each representing a news article with associated metadata (e.g., source, date) [9]. Document-Term Matrix (DTM) A sparse matrix representation where each row corresponds to a document and each column represents a unique term. Matrix values indicate the frequency of terms within documents. The DTM increases the statistical strength of the corpus and enables various NLP applications, such as clustering, classification, and similarity computation [23]. 4. Results and Discussion 4.1 Web Crawler Statistics The web crawler was deployed to collect articles from three prominent Indian news websites: India Today, The Hindu, and Indian Express, over a span of four months (January 2021 – April 2021). As shown in Fig. 1 , the crawler successfully extracted 170,952 articles in total. India Today contributed the highest number of articles (70,909), followed closely by The Hindu (66,277). The Indian Express provided 33,766 articles. The variation is primarily due to differences in the structure and frequency of news postings across these platforms. This distribution indicates the effectiveness of the crawler in adapting to diverse HTML layouts and content formats. 4.2 Preprocessing Summary The preprocessing phase applied standard NLP techniques to clean and structure the text [21]. These techniques include contraction mapping, tokenization, lowercasing, punctuation removal, stop word removal, and lemmatization. Figure 2 shows the trend of tokenized word counts per month, highlighting a consistent increase over time. The spike observed in March and April 2021 correlates with major events such as the COVID-19 second wave and state elections, which led to increased publishing volume. A total of over 64 million tokens were generated from the raw corpus, with the final preprocessed corpus containing over 37 million clean terms. 4.3 Document-Term Matrix Analysis A Document-Term Matrix (DTM) was constructed from the preprocessed corpus to enable further statistical analysis. The DTM encodes term frequencies across documents and is a foundational structure for tasks such as clustering, classification, and sentiment analysis. Figure 3 presents a sample heatmap of term frequencies across a few selected documents. Terms such as “cbi”, “report”, and “say” occurred frequently, reflecting common themes in political and investigative journalism. The DTM also supports feature selection by identifying high-impact terms, which will be leveraged in future machine learning-based news classification models. 4.4 Word Cloud and Topic Distribution Word clouds were generated to visualize the most frequently occurring terms in each month. These clouds provide an intuitive understanding of the news focus areas over time. For instance, January 2021’s word cloud (shown in Fig. 4 ) prominently features terms such as “government”, “minister”, “assembly”, and “vaccine”, which align with the major national political and health developments of that time. Similar analysis was performed for each month, capturing shifts in dominant narratives, including COVID-19, lockdowns, protests, elections, and law enforcement activities. 5. Conclusion and Future Work This study presents a comprehensive framework for mining web-based news data using a custom-built web crawler combined with advanced Natural Language Processing (NLP) techniques. The crawler was successfully implemented to extract over 170,000 articles from major Indian news sources, covering a critical time window from January 2020 to April 2021. The extracted raw corpus underwent structured preprocessing, including normalization, tokenization, stop word removal, and lemmatization. The cleaned data was subsequently transformed into structured formats such as a Document-Term Matrix (DTM), enhancing its suitability for downstream machine learning applications. Visualizations including word clouds, frequency charts, and heatmaps provided meaningful insights into topical trends, vocabulary shifts, and term prominence across the news corpus. This framework offers a scalable and flexible solution for future text mining tasks such as sentiment analysis, fake news detection, named entity recognition, and real-time topic modeling. However, certain limitations were encountered during the study. The crawler's performance is sensitive to changes in website structures, which may require manual adjustments to parsing rules. Additionally, the preprocessing phase was computationally intensive and time-consuming due to the volume of data. Future work will focus on the following enhancements: Implementing real-time crawling and live news stream ingestion. Integrating multilingual support to process news in regional Indian languages. Deploying machine learning models for automatic topic classification and clustering. Visualizing news trends over time using interactive dashboards. The results of this study lay the groundwork for advanced applications in digital journalism, media analytics, and socio-political event monitoring. Declarations Author Contribution P.S. (P. Sukumar) conceived and designed the study. P.S. developed the web crawler and conducted data collection. P.S. performed data preprocessing and analysis. P.S. prepared the visualizations, including word clouds and heatmaps. P.S. wrote the initial draft of the manuscript. All authors reviewed and approved the final manuscript.​ References Anandarajan, Murugan, Chelsey Hill, and Thomas Nolan. "Text Preprocessing." Practical Text Analytics. Springer, Cham, pp. 45–59, 2019 Arshi Saloot, Mohammad, et al. "Twitter corpus creation: The case of a Malay Chat-style-text Corpus (MCC)." Digital Scholarship in the Humanities Vol.31(2), pp. 227–243, 2016 Arulanandam, Rexy, Bastin Tony Roy Savarimuthu, and Maryam A. Purvis. "Extracting crime information from online newspaper articles." In Proceedings of the second Australasian web conference-volume 155, pp. 31–38. 2014. Babanejad, Nastaran, Ameeta Agrawal, Aijun an, and Manos Papagelis. "A comprehensive analysis of preprocessing for word representation learning in affective tasks." In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5799–5810. 2020. Chaudhary, Jashubhai Rameshbhai, and Joy Paulose. "Opinion mining on newspaper headlines using SVM and NLP." International Journal of Electrical and Computer Engineering, V9(.3) pp. 2152, 2019 Dinucă, Claudia Elena, and Dumitru Ciobanu. "Web content mining." Annals of the University of Petroşani. Economics, 12, pp. 85–92, 2019 García, Salvador, Julián Luengo, and Francisco Herrera. Data preprocessing in data mining. Vol. 72. Cham, Switzerland: Springer International Publishing, 2015 Gunawan, Dani, Amalia Amalia, and Atras Najwan. "Improving data collection on article clustering by using distributed focused crawler." Data Science: Journal of Computing and Applied Informatics Vol. 1(1), pp.1–12, 2017 Hickman, Louis, et al. "Text preprocessing for text mining in organizational research: Review and recommendations." Organizational Research Methods (2020): 1094428120971683. https://datareportal.com/reports/digital-2021-india Hui-chang, Wang, Ruan Shu-hua, and Tang Qi-jie. "The implementation of a web crawler URL filter algorithm based on caching." 2009 Second International Workshop on Computer Science and Engineering. Vol. 2. IEEE, 2009 Johnson, Faustina & Gupta, Santosh. (2012). Web Content Mining Techniques: A Survey. International Journal of Computer Applications. Vol. 47. pp. 44–50. 2012 Kumar, CS Pavan, and LD Dhinesh Babu. "Novel text preprocessing framework for sentiment analysis." Smart Intelligent Computing and Applications. Springer, Singapore, pp. 309–317, 2019 Kuonen, Diego. "Data mining and Statistics: What is the connection?" The Data Administration Newsletter Vol. 30, pp. 1–6, 2004 Lu, Mengmeng, et al. "The design and implementation of configurable news collection system based on web crawler." 2017 3rd IEEE International Conference on Computer and Communications (ICCC). IEEE, pp. 2812–2816, 2017 Nawab, Khalid, Gretchen Ramsey, and Richard Schreiber. "Natural language processing to extract meaningful information from patient experience feedback." Applied clinical informatics Vol.11(02), pp. 242–252, 2020 Patel, Jay M. "Natural Language Processing (NLP) and Text Analytics." Getting Structured Data from the Internet. Apress, Berkeley, CA, pp. 135–223, 2020 Perkins Jacob. “Python text processing with NLTK 2.0 cookbook”,.Packt publishing Ltd, 2010. Sri, Mathangi. "NLP in Customer Service." Practical Natural Language Processing with Python. Apress, Berkeley, CA, pp. 13–63, 2021 Sukumar, P., L. Robert, “A Probe on Crime Data in Various Domains”, International Journal of Recent Technology and Engineering (IJRTE), Blue Eyes Intelligence Engineering & Sciences Publication, Vol.8 (6), pp. 3940–3948, 2020. Sukumar, P., L. Robert, and S. Yuvaraj. "Review on modern Data Preprocessing techniques in Web usage mining (WUM).”International Conference on Computation System and Information Technology for Sustainable Solutions (CSITSS). IEEE, 2016. Thelwall, Mike. "A web crawler design for data mining." Journal of Information Science, Vol. 27(5), pp. 319–325, 2001 Wang, J., et al. "Research on computer application software monitoring data processing technology based on NLP." IOP Conference Series: Materials Science and Engineering. IOP Publishing, Vol. 1043(3), 2021. Wang, Jihou, et al. "News Crawling Based on Python Crawler." Journal of Physics: Conference Series. Vol. 1757. No. 1. IOP Publishing, 2021. Welbers, Kasper, Wouter Van Atteveldt, and Kenneth Benoit. "Text analysis in R." Communication Methods and Measures Vol. 11(4), pp. 245–265, 2019 Yu, Linxuan, et al. "Summary of web crawler technology research." Journal of Physics: Conference Series, IOP Publishing, Vol. 1449(1), pp. 1–7, 2020 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6484381","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":445688809,"identity":"13ba70f0-ff4f-4edf-ad98-602d2e45c05b","order_by":0,"name":"P Sukumar","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA4ElEQVRIiWNgGAWjYDADCQbmAyBKhhQtbAkgiocULTwGIJqwFvkZ6dekbtTcs5ec3fP51Y0aCx4G9sNHN+DTYnAjp0w651hx4myZs9usc44BHcaTlnYDrxaJnDTpHLaEBDmJ3G3GOWxALRI8Zni1yM8AafmXYC8nkfPMOOcfEVoYbqQfk85tS2CcLZHD/Di3jQgtBmfeMFvn9iUkzpxzzIw5t0+Ch42QX+Tb0x/ezvmWYC9xu/nx55xvdXL87IeP4XcYNDrAcSkBotnwKwcB9gcwLcwfCKseBaNgFIyCkQgAPeZEkN2SsL4AAAAASUVORK5CYII=","orcid":"","institution":"Rathinam College of Arts and Science","correspondingAuthor":true,"prefix":"","firstName":"P","middleName":"","lastName":"Sukumar","suffix":""},{"id":445688811,"identity":"721dd5ca-4a35-425a-82cc-a52c7df2546b","order_by":1,"name":"Robert L","email":"","orcid":"","institution":"Government Arts College","correspondingAuthor":false,"prefix":"","firstName":"Robert","middleName":"","lastName":"L","suffix":""}],"badges":[],"createdAt":"2025-04-19 11:23:10","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6484381/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6484381/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":81346924,"identity":"83edd0b1-c121-4039-9f4e-c58c7a1a6d3b","added_by":"auto","created_at":"2025-04-25 05:09:11","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":45764,"visible":true,"origin":"","legend":"\u003cp\u003eNumber of Articles Crawled per News Source\u003c/p\u003e","description":"","filename":"image1.png","url":"https://assets-eu.researchsquare.com/files/rs-6484381/v1/03a1f97bc221bceb684f2c28.png"},{"id":81346334,"identity":"1929ab83-d608-48d6-bd3f-ea147c0c4d20","added_by":"auto","created_at":"2025-04-25 05:01:11","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":29407,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eTotal Tokenized Words per Month\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"image2.png","url":"https://assets-eu.researchsquare.com/files/rs-6484381/v1/defa8a9d5169c7bf2c1c61fe.png"},{"id":81346335,"identity":"80d4fa8f-873f-4f2e-a8e1-4d2324d64ab8","added_by":"auto","created_at":"2025-04-25 05:01:11","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":27136,"visible":true,"origin":"","legend":"\u003cp\u003eSample Heatmap of Document-Term Matrix\u003c/p\u003e","description":"","filename":"image3.png","url":"https://assets-eu.researchsquare.com/files/rs-6484381/v1/5eeff35d076c0c26f4086310.png"},{"id":81346342,"identity":"445cf52e-8da9-4dac-986d-2447e3c96e72","added_by":"auto","created_at":"2025-04-25 05:01:11","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":52549,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eWord Cloud – January 2021\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-6484381/v1/7403c72e32abfb39060c6967.png"},{"id":82004309,"identity":"746afe34-5343-429a-8bd8-1344316cf138","added_by":"auto","created_at":"2025-05-05 21:01:29","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":659868,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6484381/v1/f63c8063-3998-41b7-a763-2a449a3f9883.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A Framework for Web-Based News Data Mining Using Crawlers and NLP Techniques","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eNews media, often regarded as the Fourth Pillar of Democracy, has transitioned rapidly into the digital realm. With over 624\u0026nbsp;million internet users in India alone as of January 2021, access to online news content has become ubiquitous. As the volume of news articles published daily continues to surge, there is a growing need to efficiently collect, process, and analyze this information [10].\u003c/p\u003e \u003cp\u003eWeb content mining\u0026mdash;particularly focused on news data\u0026mdash;enables the extraction of valuable insights for applications such as public sentiment tracking, misinformation detection, and event prediction [12]. However, challenges such as unstructured data formats, high noise levels, and inconsistencies in web design hinder effective analysis [6].\u003c/p\u003e \u003cp\u003eThis paper presents a robust framework that addresses these challenges by implementing a web-based news crawler coupled with advanced NLP-driven preprocessing [16]. The system systematically collects, cleans, and organizes news data, producing high-quality corpora for downstream machine learning and text mining applications [25]. The study demonstrates the practicality and effectiveness of the framework through large-scale data collection from leading Indian news portals [14].\u003c/p\u003e \u003cp\u003eThis paper is composed as follows. Segment II summarizes related work. Segment III presents the overall methodology. Segment IV presents the results and discussion. Finally, Segment V concludes the paper and suggests directions for future work.\u003c/p\u003e"},{"header":"2. Related Work","content":"\u003cp\u003eNumerous studies have been conducted in the domain of news data extraction and web content mining, with a particular focus on web crawlers, data preprocessing, and corpus construction.\u003c/p\u003e \u003cp\u003eYu Linxuan et al. [26] provided an extensive survey of web crawler technologies, highlighting the limitations of fixed search modes. Wang Jihou et al. [24] designed a news-specific crawler capable of rapid content acquisition. Gunawan et al. [8] developed a distributed focused crawler, enhancing performance through improved bandwidth and storage utilization.\u003c/p\u003e \u003cp\u003eIn the area of data preprocessing, Babanejad et al. [4] proposed a structured framework leveraging word vector models. Kumar et al. [13] demonstrated a three-stage NLP pipeline for sentiment analysis on Twitter data, including HTML tag removal, tokenization, and stop-word filtering. Similarly, Chaudhary et al. [5] and Anandarajan et al. [1] emphasized the significance of cleaning and normalizing textual data using common NLP techniques.\u003c/p\u003e \u003cp\u003eLarge-scale corpus development efforts such as the Malay Chat-style-text Corpus [2] and crime-related text collections [3] have shown the practical value of curated text datasets. Lu Mengmeng et al. [15] introduced a configurable news crawler system capable of multi-source content acquisition. Hui-chang Wang et al. [11] improved URL filtering to reduce redundant crawling efforts.\u003c/p\u003e \u003cp\u003eThese prior studies collectively affirm the importance of web-based data extraction and preprocessing pipelines. Building on this foundation, the present research implements a tailored web crawler and a comprehensive NLP preprocessing framework to create a structured and analyzable corpus of Indian news content.\u003c/p\u003e"},{"header":"3. Methodology","content":"\u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Web Crawling for News Data Collection\u003c/h2\u003e \u003cp\u003eIn this study, a domain-specific web crawler was developed to systematically extract news articles from three leading Indian news portals: India Today, The Hindu, and Indian Express. The crawler was implemented in Python and designed to operate in a multithreaded environment, allowing simultaneous requests to multiple web pages [18].\u003c/p\u003e \u003cp\u003eThe crawling process follows three main phases: Fetching, Parsing, and Filtering. Initially, the crawler initiates from a root URL and recursively follows hyperlinks within the domain [22]. During the parsing phase, HTML content is cleaned by removing JavaScript, style sheets, and non-article elements. In the filtering phase, relevant metadata such as title, publication date, and full-text content are extracted and stored in a raw corpus [20].\u003c/p\u003e \u003cp\u003eThe pseudocode of the crawling process is outlined in Algorithm \u003cspan refid=\"FPar1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. The final output of this stage is a corpus of raw news articles stored in .txt format, which serves as the input for the preprocessing phase.\u003c/p\u003e \u003cp\u003e \u003cstrong\u003eAlgorithm 1\u003c/strong\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eWeb News Crawler\u003c/b\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Taba\" border=\"1\"\u003e \u003ccolgroup cols=\"1\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInput: Root URL of the news domain\u003c/p\u003e \u003cp\u003eOutput: Raw corpus of crawled news article\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1. Initialize link lists and visited URLs\u003c/p\u003e \u003cp\u003e2. While link list is not empty and crawl limit not reached:\u003c/p\u003e \u003cp\u003ea. Fetch current URL content\u003c/p\u003e \u003cp\u003eb. Parse HTML to remove scripts and styles\u003c/p\u003e \u003cp\u003ec. Extract article content, title, timestamp\u003c/p\u003e \u003cp\u003ed. Store the article in raw corpus\u003c/p\u003e \u003cp\u003ee. Extract and queue internal links for further crawling\u003c/p\u003e \u003cp\u003e3. End\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Text Data Preprocessing\u003c/h2\u003e \u003cp\u003eRaw text data extracted from web pages is often noisy, containing HTML artifacts, special characters, and inconsistent formatting [7]. To ensure suitability for downstream analysis, a robust preprocessing pipeline was implemented using Python\u0026rsquo;s nltk, re, and spacy libraries [26].\u003c/p\u003e \u003cp\u003eThe preprocessing workflow includes the following steps:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eContraction Mapping: Expands contractions (e.g., \u0026ldquo;don\u0026rsquo;t\u0026rdquo; \u0026rarr; \u0026ldquo;do not\u0026rdquo;)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eTokenization: Splits sentences into individual tokens (words)\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eLowercasing: Normalizes all words to lowercase\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003ePunctuation and Number Removal: Cleans symbols and digits\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eStop Word Removal: Filters out high-frequency, low-value words (e.g., \"the\", \"is\")\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eLemmatization: Converts words to their base form (e.g., \u0026ldquo;running\u0026rdquo; \u0026rarr; \u0026ldquo;run\u0026rdquo;)\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eAlgorithm 2\u003c/strong\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eText Preprocessing Pipeline\u003c/b\u003e \u003c/p\u003e\u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Tabb\" border=\"1\"\u003e \u003ccolgroup cols=\"1\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInput: Raw corpus of news articles\u003c/p\u003e \u003cp\u003eOutput: Preprocessed structured corpus\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1. For each .txt article in the corpus:\u003c/p\u003e \u003cp\u003ea. Expand contractions\u003c/p\u003e \u003cp\u003eb. Tokenize sentences into words\u003c/p\u003e \u003cp\u003ec. Convert tokens to lowercase\u003c/p\u003e \u003cp\u003ed. Remove punctuation, URLs, and digits\u003c/p\u003e \u003cp\u003ee. Remove stop words\u003c/p\u003e \u003cp\u003ef. Apply lemmatization\u003c/p\u003e \u003cp\u003e2. Save preprocessed corpus in both .txt and .csv formats\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Data Organization for Analysis\u003c/h2\u003e \u003cp\u003eTo enable statistical text analysis and machine learning applications, the preprocessed data is organized into two commonly used formats:\u003c/p\u003e \u003cp\u003e \u003cstrong\u003eStructured Corpus\u003c/strong\u003e \u003cp\u003eA collection of cleaned text files, each representing a news article with associated metadata (e.g., source, date) [9].\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eDocument-Term Matrix (DTM)\u003c/strong\u003e\u003c/p\u003e \u003cp\u003eA sparse matrix representation where each row corresponds to a document and each column represents a unique term. Matrix values indicate the frequency of terms within documents. The DTM increases the statistical strength of the corpus and enables various NLP applications, such as clustering, classification, and similarity computation [23].\u003c/p\u003e \u003c/div\u003e"},{"header":"4. Results and Discussion","content":"\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Web Crawler Statistics\u003c/h2\u003e \u003cp\u003eThe web crawler was deployed to collect articles from three prominent Indian news websites: India Today, The Hindu, and Indian Express, over a span of four months (January 2021 \u0026ndash; April 2021).\u003c/p\u003e \u003cp\u003eAs shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, the crawler successfully extracted 170,952 articles in total. India Today contributed the highest number of articles (70,909), followed closely by The Hindu (66,277). The Indian Express provided 33,766 articles. The variation is primarily due to differences in the structure and frequency of news postings across these platforms. This distribution indicates the effectiveness of the crawler in adapting to diverse HTML layouts and content formats.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Preprocessing Summary\u003c/h2\u003e \u003cp\u003eThe preprocessing phase applied standard NLP techniques to clean and structure the text [21]. These techniques include contraction mapping, tokenization, lowercasing, punctuation removal, stop word removal, and lemmatization. Figure\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e shows the trend of tokenized word counts per month, highlighting a consistent increase over time. The spike observed in March and April 2021 correlates with major events such as the COVID-19 second wave and state elections, which led to increased publishing volume. A total of over 64\u0026nbsp;million tokens were generated from the raw corpus, with the final preprocessed corpus containing over 37\u0026nbsp;million clean terms.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e4.3 Document-Term Matrix Analysis\u003c/h2\u003e \u003cp\u003eA Document-Term Matrix (DTM) was constructed from the preprocessed corpus to enable further statistical analysis. The DTM encodes term frequencies across documents and is a foundational structure for tasks such as clustering, classification, and sentiment analysis. Figure\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e presents a sample heatmap of term frequencies across a few selected documents. Terms such as \u0026ldquo;cbi\u0026rdquo;, \u0026ldquo;report\u0026rdquo;, and \u0026ldquo;say\u0026rdquo; occurred frequently, reflecting common themes in political and investigative journalism. The DTM also supports feature selection by identifying high-impact terms, which will be leveraged in future machine learning-based news classification models.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e4.4 Word Cloud and Topic Distribution\u003c/h2\u003e \u003cp\u003eWord clouds were generated to visualize the most frequently occurring terms in each month. These clouds provide an intuitive understanding of the news focus areas over time.\u003c/p\u003e \u003cp\u003eFor instance, January 2021\u0026rsquo;s word cloud (shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e) prominently features terms such as \u0026ldquo;government\u0026rdquo;, \u0026ldquo;minister\u0026rdquo;, \u0026ldquo;assembly\u0026rdquo;, and \u0026ldquo;vaccine\u0026rdquo;, which align with the major national political and health developments of that time.\u003c/p\u003e \u003cp\u003eSimilar analysis was performed for each month, capturing shifts in dominant narratives, including COVID-19, lockdowns, protests, elections, and law enforcement activities.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"5. Conclusion and Future Work","content":"\u003cp\u003eThis study presents a comprehensive framework for mining web-based news data using a custom-built web crawler combined with advanced Natural Language Processing (NLP) techniques. The crawler was successfully implemented to extract over 170,000 articles from major Indian news sources, covering a critical time window from January 2020 to April 2021.\u003c/p\u003e \u003cp\u003eThe extracted raw corpus underwent structured preprocessing, including normalization, tokenization, stop word removal, and lemmatization. The cleaned data was subsequently transformed into structured formats such as a Document-Term Matrix (DTM), enhancing its suitability for downstream machine learning applications. Visualizations including word clouds, frequency charts, and heatmaps provided meaningful insights into topical trends, vocabulary shifts, and term prominence across the news corpus.\u003c/p\u003e \u003cp\u003eThis framework offers a scalable and flexible solution for future text mining tasks such as sentiment analysis, fake news detection, named entity recognition, and real-time topic modeling.\u003c/p\u003e \u003cp\u003eHowever, certain limitations were encountered during the study. The crawler's performance is sensitive to changes in website structures, which may require manual adjustments to parsing rules. Additionally, the preprocessing phase was computationally intensive and time-consuming due to the volume of data.\u003c/p\u003e \u003cp\u003eFuture work will focus on the following enhancements:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eImplementing real-time crawling and live news stream ingestion.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eIntegrating multilingual support to process news in regional Indian languages.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eDeploying machine learning models for automatic topic classification and clustering.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eVisualizing news trends over time using interactive dashboards.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eThe results of this study lay the groundwork for advanced applications in digital journalism, media analytics, and socio-political event monitoring.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eP.S. (P. Sukumar) conceived and designed the study. P.S. developed the web crawler and conducted data collection. P.S. performed data preprocessing and analysis. P.S. prepared the visualizations, including word clouds and heatmaps. P.S. wrote the initial draft of the manuscript. All authors reviewed and approved the final manuscript.​\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eAnandarajan, Murugan, Chelsey Hill, and Thomas Nolan. \"Text Preprocessing.\" Practical Text Analytics. Springer, Cham, pp. 45\u0026ndash;59, 2019\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eArshi Saloot, Mohammad, et al. \"Twitter corpus creation: The case of a Malay Chat-style-text Corpus (MCC).\" Digital Scholarship in the Humanities Vol.31(2), pp. 227\u0026ndash;243, 2016\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eArulanandam, Rexy, Bastin Tony Roy Savarimuthu, and Maryam A. Purvis. \"Extracting crime information from online newspaper articles.\" In Proceedings of the second Australasian web conference-volume 155, pp. 31\u0026ndash;38. 2014.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBabanejad, Nastaran, Ameeta Agrawal, Aijun an, and Manos Papagelis. \"A comprehensive analysis of preprocessing for word representation learning in affective tasks.\" In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5799\u0026ndash;5810. 2020.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChaudhary, Jashubhai Rameshbhai, and Joy Paulose. \"Opinion mining on newspaper headlines using SVM and NLP.\" International Journal of Electrical and Computer Engineering, V9(.3) pp. 2152, 2019\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDinucă, Claudia Elena, and Dumitru Ciobanu. \"Web content mining.\" Annals of the University of Petroşani. Economics, 12, pp. 85\u0026ndash;92, 2019\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGarc\u0026iacute;a, Salvador, Juli\u0026aacute;n Luengo, and Francisco Herrera. Data preprocessing in data mining. Vol. 72. Cham, Switzerland: Springer International Publishing, 2015\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGunawan, Dani, Amalia Amalia, and Atras Najwan. \"Improving data collection on article clustering by using distributed focused crawler.\" Data Science: Journal of Computing and Applied Informatics Vol. 1(1), pp.1\u0026ndash;12, 2017\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHickman, Louis, et al. \"Text preprocessing for text mining in organizational research: Review and recommendations.\" Organizational Research Methods (2020): 1094428120971683.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://datareportal.com/reports/digital-2021-india\u003c/span\u003e\u003cspan address=\"https://datareportal.com/reports/digital-2021-india\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHui-chang, Wang, Ruan Shu-hua, and Tang Qi-jie. \"The implementation of a web crawler URL filter algorithm based on caching.\" 2009 Second International Workshop on Computer Science and Engineering. Vol. 2. IEEE, 2009\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJohnson, Faustina \u0026amp; Gupta, Santosh. (2012). Web Content Mining Techniques: A Survey. International Journal of Computer Applications. Vol. 47. pp. 44\u0026ndash;50. 2012\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKumar, CS Pavan, and LD Dhinesh Babu. \"Novel text preprocessing framework for sentiment analysis.\" Smart Intelligent Computing and Applications. Springer, Singapore, pp. 309\u0026ndash;317, 2019\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKuonen, Diego. \"Data mining and Statistics: What is the connection?\" The Data Administration Newsletter Vol. 30, pp. 1\u0026ndash;6, 2004\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLu, Mengmeng, et al. \"The design and implementation of configurable news collection system based on web crawler.\" 2017 3rd IEEE International Conference on Computer and Communications (ICCC). IEEE, pp. 2812\u0026ndash;2816, 2017\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNawab, Khalid, Gretchen Ramsey, and Richard Schreiber. \"Natural language processing to extract meaningful information from patient experience feedback.\" Applied clinical informatics Vol.11(02), pp. 242\u0026ndash;252, 2020\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePatel, Jay M. \"Natural Language Processing (NLP) and Text Analytics.\" Getting Structured Data from the Internet. Apress, Berkeley, CA, pp. 135\u0026ndash;223, 2020\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePerkins Jacob. \u0026ldquo;Python text processing with NLTK 2.0 cookbook\u0026rdquo;,.Packt publishing Ltd, 2010.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSri, Mathangi. \"NLP in Customer Service.\" Practical Natural Language Processing with Python. Apress, Berkeley, CA, pp. 13\u0026ndash;63, 2021\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSukumar, P., L. Robert, \u0026ldquo;A Probe on Crime Data in Various Domains\u0026rdquo;, International Journal of Recent Technology and Engineering (IJRTE), Blue Eyes Intelligence Engineering \u0026amp; Sciences Publication, Vol.8 (6), pp. 3940\u0026ndash;3948, 2020.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSukumar, P., L. Robert, and S. Yuvaraj. \"Review on modern Data Preprocessing techniques in Web usage mining (WUM).\u0026rdquo;International Conference on Computation System and Information Technology for Sustainable Solutions (CSITSS). IEEE, 2016.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eThelwall, Mike. \"A web crawler design for data mining.\" Journal of Information Science, Vol. 27(5), pp. 319\u0026ndash;325, 2001\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang, J., et al. \"Research on computer application software monitoring data processing technology based on NLP.\" IOP Conference Series: Materials Science and Engineering. IOP Publishing, Vol. 1043(3), 2021.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang, Jihou, et al. \"News Crawling Based on Python Crawler.\" Journal of Physics: Conference Series. Vol. 1757. No. 1. IOP Publishing, 2021.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWelbers, Kasper, Wouter Van Atteveldt, and Kenneth Benoit. \"Text analysis in R.\" Communication Methods and Measures Vol. 11(4), pp. 245\u0026ndash;265, 2019\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYu, Linxuan, et al. \"Summary of web crawler technology research.\" Journal of Physics: Conference Series, IOP Publishing, Vol. 1449(1), pp. 1\u0026ndash;7, 2020\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Data Mining, Machine learning, Text mining, Data Preprocessing, NLP","lastPublishedDoi":"10.21203/rs.3.rs-6484381/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6484381/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe explosive growth of online news content offers a vast opportunity for data-driven research in web mining. This paper proposes a comprehensive framework for mining news data using web crawlers and Natural Language Processing (NLP) techniques. A customized crawler is developed to extract articles from prominent Indian news websites including India Today, The Hindu, and Indian Express, covering the period from January 2020 to April 2021. The raw data undergoes rigorous preprocessing, including tokenization, normalization, stop-word removal, and lemmatization, to produce a clean, structured corpus. The final dataset is organized into standard formats such as structured corpus files and Document-Term Matrices (DTM), facilitating downstream applications such as classification, clustering, and sentiment analysis. This framework lays the foundation for large-scale, real-time news analytics and can be adapted for multilingual or domain-specific news mining tasks.\u003c/p\u003e","manuscriptTitle":"A Framework for Web-Based News Data Mining Using Crawlers and NLP Techniques","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-25 05:01:06","doi":"10.21203/rs.3.rs-6484381/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"44b08e77-21fc-4632-8415-8653422733dd","owner":[],"postedDate":"April 25th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-05-05T20:53:22+00:00","versionOfRecord":[],"versionCreatedAt":"2025-04-25 05:01:06","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6484381","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6484381","identity":"rs-6484381","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}