Web Scraping as a Data Collection Strategy: The Perils and Pitfalls

The methodology of discourse analysis within the field of social work is often conducted manually - from data collection to analysis. However, many opportunities lie at the nexus of qualitative methods and technology. This article discusses the use of the artificial intelligence (AI) technique of web scraping in research through the case of a study regarding the blogging discourse on Black women’s mental health in the context of the dual pandemics of COVID 19 and anti-Black racism. The research explores blogs about Black women’s mental health between the timeframe of March 2020 to December 2021. Web scraping is broadly defined as a method of automatically pulling information from an online source. This study used Python coding to extract blog posts from the platform Medium.com. The researchers offer a discussion of obstacles, resolutions and key recommendations for future studies exploring the methods of web scraping. This article provides common barriers in the process, such as expertise and technology resources, as well as key considerations, such as strategies for conducting methods in a virtual work environment and navigating both hardware and software demands. After many obstacles, the team ultimately accomplished the method aim, finding that web scraping is a highly effective and efficient research method that demands in-depth planning, preparedness for obstacles, and resource considerations.