Mapping Sustainable Development Goals to Citizen Science projects - a comparative evaluation of automatic classifiers

Mapping Sustainable Development Goals to Citizen Science projects - a comparative evaluation of automatic classifiers | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Mapping Sustainable Development Goals to Citizen Science projects - a comparative evaluation of automatic classifiers Patricia Santos, Ishari Amarashinghe, Miriam Calvera-Isabal, Cleo Shulten, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4781489/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 15 Dec, 2024 Read the published version in International Journal of Data Science and Analytics → Version 1 posted 11 You are reading this latest preprint version Abstract Traditional data sources provide insufficient knowledge for measuring the United Nations Sustainable Development Goals (SDGs). Data related to SDGs are sourced primarily from global databases maintained by international organizations, national statistical offices and other government agencies. Recent studies show the value of using data from Citizen Science (CS) for assessing the SDGs. There is an important online presence of CS programs, professional networks for CS and online communities of citizen scientists, leading to the generation of several CS platforms. In this context, the role of computational data science is key. This paper explores and exemplifies opportunities for combining web-data mining techniques and automatic classifiers to enhance the understanding of the inter-relation between CS and the SDGs. An analysis of different automatic classifiers is presented by comparing the results obtained from their application in a sample of 208 CS project descriptions. The results of this study indicate the benefits and limitations of these techniques (nCoder, ESA, OSDG and BERT), but also provides a discussion of the potential benefits of using data from CS projects to map the 17 SDGs. Information Technology and Systems Data mining Web mining Text analysis Machine learning Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 1 Introduction The seventeen Sustainable Development Goals (SDGs) formulated and set up by the United Nations General Assembly (UN-GA) in 2015 provide a framework of sustainability goals and targets that is universally accepted. The SDGs summarize priority action areas to help society achieve justice, prosperity, and environmental security (see: https://sdgs.un.org/goals). Improving the understanding of how the SDGs are implemented around the world is a priority task of the UN 2030 Agenda [1]. The potential of Information and Communication Technologies to achieve the SDGs have been explored by some authors [2],[3]. But one of the main problems is related to having access to data associated to SDGs. Data related to the SDGs is sourced primarily from global databases maintained by international organizations, national statistical offices, and other government agencies. This data is costly to obtain and still incomplete in detail and coverage. Fritz et al. [4] state that traditional data sources are not sufficient for measuring the achievement of SDGs so that it is necessary to identify new and additional information sources. In this line, recent studies show the value of using data from Citizen Science for improving the reporting of SDGs [5]. Citizen Science (CS) is a broad term that is interpreted differently depending on specific science cultures, research orientations, disciplines, or the types of citizen activities. As stated by Eitzel et al. [6] no single term is appropriate for all contexts and practices associated with the term CS. In this paper, we refer to the definition provided by the European Commission in the Science with and for Society (SwafS) work programme 2018-2020 as a baseline: “Citizen Science should be understood broadly, covering a range of different levels of participation, from raising public knowledge of science, encouraging citizens to participate in the scientific process by observing, gathering and processing data, right up to setting scientific agenda and co-designing and implementing science-related policies. It could also involve publication of results and teaching science”. According to Heigl et al [7], public participation in scientific projects is increasing around the world as part of projects and activities labeled as CS. Since 2010, CS has been explicitly placed in different European science policy frameworks aligned with the objectives of the Europe 2020 strategy and related to more specific areas such as the Digital Agenda, Science 2.0, Responsible Research and Innovation, Open Science, and the SDGs [8]. For this reason, Fritz et al in [4] pointed out the importance of considering the transformative role of CS as an engine for monitoring SDGs. CS is a growing and flourishing practice of CS in Europe and across the world [9]. A strategic, scalable and pluralistic approach is needed to understand how CS is reported. Nowadays, multiple CS platforms, websites and social media accounts exist, and therefore the ensuing knowledge is dispersed on the Internet [10]. Therefore, although CS has the potential to support a better understanding of the status of SDGs, it is necessary to explore technical approaches to solve the problem of scattered data sources on the Internet and to support the automated analysis of large amounts of textual information. Currently, most of the studies or projects on inter-relating CS and the SDGs deal with limited samples of initiatives focused mainly on one SDG [11],[12],[13],[14]. Others are based on reporting how traditional (human-based) analytical methods can be applied to understand the connection between CS and SDGs [5]. So far, most of these studies are focused on topics related to environmental research/impact, but there is still a lack of knowledge of how CS contributions can be mapped to the total of the 17 SDGs. In this paper, we explore and exemplify opportunities for combining web-data mining techniques and automatic classifiers to enhance the understanding of the inter-relation between CS and the SDGs. This work is based on a cooperation and partnership in a SwafS-H2020 research project called: ‘CS Track: Expanding our knowledge on Citizen Science through analytics and analysis’ (see: cstrack.eu). CS Track relies on a combination of web analytics techniques and classical social studies methods with the aim of broadening the knowledge about CS. The work done by Roldan-Álvarez et al [15] is a first example showing the benefits of applying social networks analysis (on Twitter) to connect data from CS and SDGs. Yet, as stated by these authors, Twitter blogosphere is a limited source of information related to CS activities, which are more often and, in more detail, represented in project websites and on specific CS online platforms. For this reason, one main goal of CS Track has been to build a central database aiming to compile a comprehensive collection of CS projects, essentially relying on available web data. Building such a DB is the first step to solve the problem of having the data from CS projects dispersed in multiple online sites. Once this is solved, the next step involves exploring how the textual data from CS projects reported online can be connected with the SDGs and therefore be analyzed. In this paper, we show the benefits of applying automatic classifiers to understand the connection between the data from CS projects with the SDGs. In particular, we compare the application of nCoder, ESA and BERT (more detail below). The goal of this article is to present opportunities, achievements, and future challenges in using computational analytics to better understand the connection between CS and the SDGs. The work in its status does not fully cover SDGs in CS, but it evaluates and shows the potential of the text-classification techniques for identifying SDGs in CS project descriptions and for assessing trends in connection of CS and SDGs based on available data. Accordingly, our main research question is: How can a data analytics approach based on web-based data mining and automatic classifiers contribute to the reporting of SDGs related to CS activities and projects? The rest of this article is organized as follows: Section 2 reviews the state of the art. Section 3 analyzes emerging computational analytics techniques that can be applied for the purpose of this research. Section 4 presents preliminary results to show the potential of the techniques described in section 3. In section 5 the limitations and challenges faced during the study. The final section identifies future research lines to be further explored and formulates conclusions based on the work so far. 2 Background 2.1 Traditional data sources to report SDGs Different reports from the United Nations, claim that a data revolution for sustainable development is needed. In general the data used to understand the SDGs comes from official statistics for governance, the central one is the UN Statistical Commission but also national and regional official institutions. The global indicator framework for Sustainable Development Goals was developed by the Inter-Agency and Expert Group on SDG Indicators (IAEG-SDGs) and agreed upon at the 48th session of the United Nations Statistical Commission held in March 2017 [1] . This framework is refined annually and reviewed by the Statistical Commission. The global indicator framework is complemented by indicators at the regional and national levels, which will be developed by Member States. This includes a list of quantitative indicators aligned with the 17 SDGs. As stated by Riegner [16] one common problem in this process is that often it is difficult to have updated data, sometimes it is inaccurate, unavailable, or based on unreliable estimates. On the other hand, statistical capacities in many developing countries are insufficient. The author states that international standards need to be improved to be able to enhance these problems. In this line, “The data revolution report” [2] provides a reference point for efforts to implement the data revolution under the SDG framework. The report makes emphasis on the importance of agreeing on basic principles related to data revolution and adopting standards for openness and exchange of data and for the protection of human rights. Part of the recommendations to progress with the data revolution include the use of new data sources and big data analytics and the diffusion of information technologies. In the last lustrum, some authors have demonstrated the value of using data from CS for the SDGs [4],[5] among others already cited in the introduction. Fraisl et al [5] propose a methodology, to conduct a systematic review of the SDG indicators for CS, that has been applied as part of the manual coding process described in this paper. In this paper we propose an extension of this methodology by including the application of text-classification techniques. In this context there is no previous research proposing a solution in relation to the diffusion issue suggested by the data revolution report in the context of CS, and how the application of text-classification techniques have the potential to classify big amounts of CS data to SDGs. 2.2 Citizen Science data The term Citizen Science was coined in the 1990s and has gained popularity since then [9]. One recommendation for future development of CS is to anchor the citizens in research and development, to understand better the positioning of CS with the global agenda of the SDGs. In the last decade, the number of citizen science programs, professional networks for CS stakeholders and online communities of citizen scientists have strongly increased, leading to the generation of several CS platforms. Some examples of CS Platforms are for instance: Zooniverse (see: https://www.zooniverse.org/), eu-citizen science (see: https://eu-citizen.science/) or Scistarter (see: https://scistarter.org/). According to Liu et al [10] CS platforms are: “web-based infrastructures with one single entrance point that contain one or several of the following functionalities: (1) present active citizen science projects and activities; (2) display citizen science data and information; (3) provide overall guidelines and tools that can be used to support citizen science projects and activities in general; (4) present good practice examples and lessons learned; and (5) offer relevant scientific outcomes for people who are involved or interested in citizen science”. At the same time these authors identify the following types of CS platforms: commercial; for specific projects; for specific scientific topics; national or continental platforms. Other sources of information in CS are individual websites of projects, and (if any) associated social media accounts (e.g., Twitter, Facebook) In this paper, our attention is especially focused on CS platforms as the main source to find information summarizing the main goals of CS projects (i.e., project descriptions). We also discuss the potential of using data from Twitter to establish connections between CS and SDGs. According to Sturm et al. [17], CS platforms can be developed as a technical framework designed to store data and information from multiple types of applications. In this context, the collaborative success for citizen science and public participation in scientific research (PPSR) led to the development of the PPSR_CORE metadata standard [18], [19]. The PPSR is proposed as the CS metadata standard with the main aim to provide a common consensus on how to structure the data from CS projects in technical platforms. Unfortunately, the PPSR_Core standard is not yet established as the common metadata standard in the CS community, therefore only few CS platforms now implement the standard at the level of the project descriptions, something that makes the automatic analysis of the content reported on existing platforms a challenge. In addition to this, the current version of the proposed metadata standard does not include any attribute related to the SDGs. As a consequence, there is no official standardized method at the moment to report data from CS contributions connected with the SDG quantitative framework. One of the most popular CS platforms, Scistarter, integrates a system called OSDG as a way to classify text data from projects’ descriptions to SDGs. OSDG (see: https://osdg.ai/) is a citizen science initiative to (manually and automatically) classify textual data to SDGs. This initiative is not focused on classifying data from CS projects exclusively. In this paper, we have selected this system, among other text-classification techniques to understand the potential and limitations of these types of classifiers. Our plan is to apply text-classification techniques to obtain first insights of the relationship between CS and SDGs. For this reason, before mapping specific quantitative contributions from CS with the SDGs indicators, we propose that it is necessary to first understand how existing CS projects are covering topics related to the SDGs. Opportunities to achieve this first step is what is covered in this paper. Once we are able to classify specific CS projects associated with specific SDGs, then the next step is to analyze the association of the derived contributions of the projects and the SDGs quantitative indicators. [1] https://unstats.un.org/sdgs/indicators/indicators-list/ [2] https://www.undatarevolution.org/report/ 3 Dataset Preparation 3.1 Building a source of knowledge: the CS Track Database In order to build our source of knowledge, with data from CS projects to be analyzed, a Database (DB) has been created by manually searching for existing CS platforms (e.g. https://eu-citizen.science/; https://scistarter.org/) containing CS projects (e.g. FLOODUP or COVID near you). First, we manually selected CS platforms from Europe and platforms containing projects conducted fully online. So far, a total of 56 platforms have been used to extract project descriptions, the list of CS platforms and projects is consistently updated for the duration of the project (2019-2022). As a first exercise to understand the potential of the techniques explored in this study, 16 CS platforms have been considered to extract and analyze CS project descriptions (more info about the dataset used below). The CS Track DB opens a new perspective on CS knowledge by observing and characterizing initiatives through a quantitative approach that relies on web-based data mining and network analytics. A crawler has been developed taking into account the different web structures of the selected CS platforms. The detailed process followed is out of the scope of this paper, but as a summary the main steps are as follows: (1) checking if a robots exclusion protocol allows data extraction (2) open URL of the websites of the platforms selected; (3) retrieve URL for specific CS projects listed on the selected platform, (4) open the URL (5) select, extract and classify the information from the website according to its HTML structure; (6) data cleaning and classification (following the Project Metadata Model (PMM) from PPRS_Core metadata standard) and (7) store the data into a single database organized by typology of data. To ensure GDPR compliance, a data anonymization process has also been applied. So far, the DB contains 4737 CS projects and the additional information that has been gathered in the DB is an information resource that can be analyzed on its own as well as serve as a useful starting point for other research, as the one proposed in this manuscript. The collection of CS projects in our DB reveals a complex picture (e.g., the unstructured reporting by CS platforms, the lack of information for some attribute, the projects that are too diverse) that challenges the analysis of specific topics of interest (i.e., SDGs covered by CS projects). In this research, the unit of analysis is the description of individual CS projects extracted from CS platforms. For this reason, we propose to explore the potential benefits (and limitations) of different techniques based on text-classification. In the following subsections we describe the process needed to apply the techniques selected, to proceed with this first exercise a subset of the total number of projects in the DB has been selected. 3.2 Manual classification: hand-coding of the dataset We decided to start with an initial subset of the data from our DB (see Figure 1). A total of 208 projects from 16 CS platforms were randomly selected with the following criteria: project descriptions should be in english; platforms should contain a list of projects situated in Europe or should be projects conducted online. Following the initial steps defined by Fraisl et al. [5], we extracted and reviewed SDGs targets and indicators metadata from the official websites of the United Nations (i.e., https://unstats.un.org/sdgs/metadata/). This review was done by 3 researchers to identify a list of keywords to be applied for SDG classification purposes. For this study we used the list published by Monash University and Australia S.D.S.N. [20]. Two authors of this paper and two research assistants (n=4) were involved in manual coding of the 208 project descriptions into relevant SDGs. In addition to the keywords selected in the previous step, new ones emerge from the manual coding process. The initial manual coding provided a ground truth against which the performance of the three methods (i.e nCoder, ESA and OSDF) could be evaluated. Results obtained via manual coding are presented in Figure 3. 4 A comparison of different text classification techniques In this study three different text classification techniques are compared: First, nCoder has been selected because this technique is the closest one to the process followed for manual classification. nCoder automates the process of coding, allows researchers to discover concepts in data flexibly, so in this case researchers provided expert knowledge. Second, Explicit Semantic Analysis (ESA) because for this an external source of knowledge (in this case Wikipedia articles about each SDG) is used to to calculate semantic similarities between words and texts. Third, we compare our results with the existing tool ‘OSDG’ also based on text-classification from multiple sources. Additionally, we discuss the results obtained with these three techniques with the results from a previous study [14] where BERT was used to classify CS data from Twitter to SDGs. More detail about the techniques compared and discussed in this manuscript continue as follows: 4.1 nCoder nCoder is a tool that enables to train classifiers that automatically label text according to some set of classification rules [21]. The tool is freely available online and an R package. The process of training text classifiers in nCoder consists of several steps. First, it is required to define the keywords and/or regular expressions to describe language patterns that we are looking for in the input datasets. nCoder uses a simple regular expression matching algorithm to automatically label the text data for a targeted concept. In this study, we used keywords that were derived from related sources (see section above) and also used a TF-IDF (term frequency-inverse document frequency) approach to understand relevant concepts. During the next step, nCoder prompts the human rater a randomly selected sample from the input dataset that consists of 80 samples (training set). The human rater is required to go through the 80 samples and to indicate the presence of the targeted concept, i.e., a particular SDG, by indicating yes/no. After manually coding 80 samples the tool provides statistics based on kappa and rho measures to indicate the validity of the automatic classifier. Kappa measure indicates the agreement between human coder and machine coder for coding the training set and rho measure indicates the generalizability of automatic coding. If the first round of training was not reliable therefore did not reach the intended thresholds for kappa and rho statistics (Kappa > 0.65 Rho < 0.05) then modifying the keywords and/or regular expressions is necessary to start a new round of revalidation. 4.1 Explicit Semantic Analysis Explicit Semantic Analysis (ESA) is a method to calculate semantic similarities between words and texts [22]. The basis of the calculations is a precalculated inverted index built from encyclopedic data. The index can be described as a matrix of terms and articles with the terms being reduced to their word stems. We can use the index to turn a term into a representative vector with tf-idf values for the corresponding articles. Using term vectors, we can calculate the cosine similarity. By adding the term vectors of all terms occurring in a text we can upscale this to compare texts with each other. In our implementation of this method, we opted to improve the performance by filtering the text’s terms first by using tf-idf and disregarding terms that score lower than 20% of the highest tf-idf score. For the association of projects with SDGs through ESA, we used the existing Wikipedia articles of all SDG as reference documents as a basis to calculate the corresponding text vectors. For any given project description, we can then calculate the corresponding text vector and the similarity with each SDG’s vector. The results are then ranked based on similarity. The similarity results are rarely ever close to 0 so that a cutoff for exclusion has to be determined and chosen. The underlying processing scheme is illustrated in Figure 2. Fig 2. Processing scheme for associating SDGs to CS projects through ESA 4.3 OSDG: Open-Source Approach to Classify Text Data As mentioned in the Introduction section, OSDG is an open-source initiative aimed to classify text data according to SDGs. In Pukelis et al. [23], the authors propose a way to integrate data from multiple sources into a single framework for SDG classification. OSDG combines the use of supervised machine learning models (e.g., it tooks the most significant features for each SDG), and from unsupervised machine learning looking at the most important words for each topic. More in concrete, the system builds an integrated ontology from the feature sets identified in previous research and then matches the ontology items to the Fields of Study from Microsoft Academic. In this study we use this tool as a way of comparison and validation, using the same dataset used with nCoder and ESA 4.4 BERT and machine learning models One of the most popular methods for text classification is the use of deep learning models. In this scenario, BERT (Bidirectional Encoder Representations from Transformers) has become one the most widely used tools in recent years to classify texts in different areas [24]. BERT is designed to pretrain deep bidirectional representations from unlabelled texts by jointly conditioning on both left and right context in all layers. One of the main issues when using deep learning models is the creation of a dataset that allows the training of that model so text can be classified into the corresponding categories accurately. However, nowadays we can find many sources with information that can be used to populate those training datasets. Using APIs or techniques such as Web Scraping, we can gather data that can be used to train deep learning models. Moreover, it would be easy to increase the number of texts of the dataset since information on the Internet is growing every day. There are prior studies in which the authors used Twitter to collect tweets regarding SDGs to train a deep learning model based on BERT [15]. In this study, 57,843 tweets were directly downloaded from the Twitter API by using English keywords, French keywords and Spanish keywords related to SDGs. The dataset was composed of unique tweets. After classifying 29,543 tweets, the results showed that around 25% of them were about SDG13, which talks about taking urgent action to combat climate change and its impact. 5 Results 5.1 Manual classification Figure 3 shows the results of the manual coding process (classification of selected 208 CS project descriptions into relevant SDGs) which we used as a baseline. Given the considerably large number of projects for manual coding across 17 SDGs it had not been possible to assign two people to do manual coding for every SDG, therefore, to report inter-rater reliability. However, the researchers and research assistants discussed regularly and specially when in doubt all discussed and re-coded until disagreements were solved. Fig 3. Manual coding results of CS project descriptions to indicate the presence of SDGs. 5.2 SDG Dependencies In the comparative analysis of SDG associations to the selected sample of projects, we have focused on SDGs as separate entities. However, SDGs and the ensuing classifications are partly overlapping. Based on our sample of classified projects, we can determine the overlap or similarity between SDGs. For the actual calculation, we use the Jaccard measure of similarity (see Figure 4). For two given SDGs X and Y, the Jaccard similarity would be calculated as the following proportion: The value of this measure would depend on the associations and co-occurrences determined by the different methods. For a baseline assessment of similarities, we rely on the results of the manual coding that we have also considered as a ground truth for comparing the other methods. Figure 5 shows the ensuing similarity matrix. Here, the very general and over-arching SDG #17 (Strengthen the means of implementation and revitalize the global partnership for sustainable development) has not been included since it was never assigned to one of the sample projects. For the given sample, the highest similarity (based on co-occurrences) is between SDGs #5 (Gender equality) and #8 (Decent work and economic growth). The similarity matrix can serve as a basis for clustering the SDGs by their overlaps. Figure 6 shows the cluster dendrogram that results from applying agglomerative hierarchical clustering, together with the ensuing silhouette values after a cut at height = 1.4. This cut gave the best overall silhouette value. Based on the silhouette values, the strongest clusters are 5-8-1 (Gender equality, Decent work and economic growth, No poverty) and 6-14-12 (Clean water and sanitation, Life below water, Responsible consumption and production) whereas SDGs 9 (Industry, innovation and infrastructure) and 16 (Peace, justice and strong institutions) are not strongly connected to other SDGs.It is important to consider that these are empirical findings in which the associations between SDGs depend on the actual overlaps in the given sample of 208 CS projects, with the basic assignments depending on human judgement. One might, e.g., have expected a stronger association between SDGs 2 (Zero hunger) and 3 (Good health and well-being), yet this is not backed by the orientation of the projects in our sample. The point here is to see the interdependencies of SDGs in practice as contrasted to “semantic expectations”. If we can further corroborate these dependencies, we can use them to improve our automatic detectors. E.g., if SDGs A and B are known to be closely related but our "detector" only finds A we may use this background knowledge to also infer B. 5.3 Automatic classification The same sample of project descriptions that were manually coded was also classified in relation to the 17 SDGs using nCoder. A detailed overview of the keywords used to train nCoder, Kappa and Rho statistics obtained using nCoder are presented in Appendix 1. In the following we present an overview of the results. In summary, we were able to obtain reliable classifications (reliable kappa and rho statistics) for SDG#2 (Zero hunger), SDG#3 (Good health and well-being) and SDG#6 (Clean water and sanitation). However, nCoder failed to classify project descriptions for the following SDGs: SDG#5 (Achieve gender equality and empower all women and girls), SDG#7 (Ensure access to affordable, reliable, sustainable and modern energy for all), SDG#9 (Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation), SDG#16 (Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels) and SDG#17 (Strengthen the means of implementation and revitalize the global partnership for sustainable development) due to lack of presence of the targeted concepts in the input data. For the other SDGs (e.g., SDG#1, SDG#4, SDG#8, SDG#10, SDG#11, SDG#12, SDG#13, SDG#14, SDG#15 - see details in Table 1) although we attempted multiple rounds of revalidation we were unable to improve kappa and rho statistics. Finally, we also calculated the F1-score based on successful classifications to report the performance of nCoder which reached an F1-score of 0.67 (with Precision = 0.58, Recall = 0.76). For ESA we used the same dataset of 208 projects considering the manually assigned SDGs we calculated the similarities between all projects and all SDGs. We then tried different thresholds (0.4, 0.3, 0.25, 0.2 and 0.15) and determined the counts of true positives, false positives, false negatives and true negatives for each. Based on this we calculated and compared Precision, Recall and F1 score for each threshold, which appeared to be optimized at 0.2 with an F1 score of 0.411 (with Precision=0.414 and Recall=0.408). Next, we compared the classification results obtained using different techniques. We also obtained SDGs classification of the 208 CS projects by manually checking each project description with OSDG as an external source for comparison. A comparison of the classification results obtained from the three different techniques (nCoder, ESA, OSDG) are presented in Figure 7 and Figure 8. As a baseline we used the results from manual coding (see Figure 3). OSDG classification achieved an F1-Score of 0.30 (with Precision=0.38 and Recall = 0.25) SDGS with more coincidence between methods are: SDG#3: Good health and well-being; SDG#4: Quality education (no coincidence with OSDG in this SDG); SDG#6: Clean water; SDG#12: Responsible consumption; SDG#13: Climate Action (again no coincidence with OSDG in this SDG); SDG#15: Life on Land. 6 Discussion The main aim of this research is to explore the advantages and limitations of text-classification techniques to understand their potential to classify data from CS projects with SDGs. Our main research question is : How can a data analytics approach based on web-based data mining and automatic classifiers contribute to the reporting of SDGs related to CS activities and projects? In this section, first we show how the application of automatic classifiers has allowed us to find interesting findings regarding the mapping of SDGs and CS. Second, we compare the techniques covered in this study by considering their advantages and limitations when applying each technique to classify CS project descriptions with SDGs. Despite that our dataset was randomly built we observe coincidences with previous results in the literature regarding which SDGs are more representative in CS. For instance, in Shulla’s et al. [25] it is stated that CS activities contributed mainly with SDG#4 (Quality Education), SDG#11 (Sustainable Cities and Communities), SDG#13 (Climate Action) and SDG#15 (Life on Land). Also, in the study done by Roldan-Álvarez et al. [15] SDG#13 was the one associated with 25% of the projects analyzed. These results match with the results obtained in our study, but there are interesting aspects that can be observed. In concrete SDG#4 and SDG#13 have very similar results when comparing manual coding, nCoder and ESA. It is interesting to observe the cases of SDG#10, SDG#11 and SDG#15 mainly we observed different results between methods. The case of SDG#10 (Reduced inequalities) is a curious case to be further investigated in the future. Similarly, to the case of SDG#4, SDG#10 seems to be a transversal SDG that can be associated with multiple disciplines. Around 42% of the projects in our dataset have been associated with this SDG with manual classification and nCoder, but the result is completely different with OSDG and ESA. As indicated before SDG#4 has been mentioned in previous literature, but we have not found evidence showing the connection between SDG#10 and CS in previous studies. In the case of SDG#11, a significant difference was observed between the results obtained with manual coding, nCoder and OSDG (similar) vs ESA (lower positive results). A similar case happens with SDG#15 but in this case the difference is less remarkable. The reasons behind this difference are related to the set of keywords or wikipedia articles used as a source of knowledge by each technique. In the case of SDG#11 and SDG#15 the results between nCoder and manual coding are similar indicating the keywords selected for training are adequate, in this case the problem would be associated with the content of the wikipedia article for these SDGs. A similar case is the one related to SDG#1 as observed in Figure 7, nCoder and ESA seem to incorrectly classify projects with SDG#1. In this case, the main reason behind this could be the fact that the SDG#1 itself covers a broad range of concepts, in comparison to the other SDGs which are more specific. Therefore, detecting all possible keywords associated with SDG#1 becomes a difficult task. It has particularly become problematic for nCoder classification as indicated in Figure 7 (deviations observed when compared to results obtained from nCoder vs. manual coding. The similarity between different SDGs (Figure 5) is also observed in our results. Most similar SDGS are: SDG#5 with SDG#8; SDG#6 and SDG#14; SDG#4 and SDG#10; SDG#3 and SDG#10. There is a lack of studies comparing similarity between different SDGs, for this reason we think this is an interesting line to be further explored as future work especially when we expand our analysis to the whole database of projects contained in the CS Track DB. The main aim of this study was to compare different automatic classifiers to map SDGs with CS, our experience with nCoder confirms the advantages identified by [21]. For large datasets manual coding for different codes can become difficult, as multiple inter-rater reliabilities need to be assessed to achieve acceptable reliability. nCoder automates the process of coding allowing researchers to discover concepts in data flexibly. In fact, in our study this process of having researcher/s iterating and adapting the set of keywords (Figure 1) made the process closer to the results obtained with manual coding. For this reason, we obtained better precision and recall results with nCoder when compared to the same results obtained with reference to with ESA technique. The other main advantage of nCoder is that explainability is quite high when compared to black box models (i.e., BERT). However, it is necessary to mention also the limitations experienced when applying nCoder in this study. One of the most relevant limitations is that overall, the process is time consuming. For instance, in this study nCoder didn't perform enough well for some specific SDGs (e.g., SDG#11 or SDG#15) classification. In these cases, the main limitations were due to the set of keywords used, or the limitations in the input dataset that did not consist of enough samples to carry out a proper training. Introducing additional keywords or enhancing the input dataset with more samples could have improved the results, however, the time constraints and lack of resources required to carry out re-validation eliminated us from conducting further experimentation. Moreover, at present it’s not possible to save a trained classifier that can be later re-used to classify new datasets. In the case of ESA one advantage is that previous manual coding is not necessary, it is fully based on automatic assignment/coding, therefore it requires less resources (i.e., time for manual coding) compared to nCoder. The same mechanism can be used for different classifications (e.g., research areas) after a corresponding comparison base is established. Another advantage is that thresholds for tf-idf and assignment can be modified and adapted to match the use case. This means that it can account for cases where one needs exactly or at least one assignment as well as for instances where the similarity needs to reach a certain threshold before assignment can be considered with little changes. However, optimization of thresholds takes time. A textual description is needed for each classifier, in this case Wikipedia articles of each individual SDG were used. But as observed in this study, the issue can always be with extensive or broad descriptions that sort of dilute the results. For instance, the projects ESA didn't catch might cover more than SDG#11 (i.e., other SDGs or research areas) which pulls down the textual similarity to the SDG#11 Wikipedia article. Finally with this technique the language used is also an issue, ESA is dependent on language of the base matrix (in this case english) so that would have to be set up for different languages if needed otherwise texts need to be translated. In this study we compared the results obtained from nCoder and ESA with an existing text-classification platform ‘OSDG’. When comparing the F1-Scores obtained with each technique, the ones from OSDG are lower than the ones obtained with nCoder or ESA. OSDG classification achieved an F1-Score of 0.30 (with Precision=0.38 and Recall = 0.25). ESA achieved a F1 score of 0.411 (with Precision=0.414 and Recall=0.408). And nCoder reached an F1-score of 0.67 (with Precision = 0.58, Recall = 0.76). However, it is necessary to indicate that in the case of nCoder it was not possible to consider the total of 17 SDGs to calculate the F1-score because SDG#5, SDG#7, SDG#9, SDG#16 and SDG#17. Despite OSDG achieving lower precision and recall results, one advantage of using this technique is the ease of use of its platform and results are generated instantly. Although BERT was not directly applied to the dataset used in this study, we would like to discuss the main advantages and limitations of this classifier based on the experience obtained in a previous study [15]. One of the main points of interest when using deep learning models, BERT in particular, is to create a balanced dataset in which each category is equally represented. In the case of the Twitter study regarding SDGs this was a limitation because the discussion around some of them is greater than in others then this produced an unbalanced dataset. For instance, according to our previous study, we can easily find tweets about SDG#13, but it is not that easy to gather tweets about SDG#9. When creating the training datasets this is an issue that needs to be tackled. Otherwise, predictions will not be accurate, and the number of wrong classifications will greatly increase. Therefore, the main limitation of BERT is the time required to prepare the initial sample for the training dataset, especially when there is not an original dataset containing positive and negative examples of CS projects classified for each SDG. 7 Future work and Conclusion This study contributes to the lack of research regarding the need of using data from Citizen Science projects to enhance the knowledge regarding the UN Sustainable Development Goals. The study shows how a process based on data analytics approach (combining web-based data mining and automatic classifiers) can contribute to the reporting of SDGs (RQ). First this study has shown that by using web-mining techniques a DB can be built as a central point of knowledge to avoid the problem of CS data dispersed on the web. Although this was not the main scope of this paper, our experience building the DB and the classification of the different CS project descriptions allows us to identify some future work. Now one major barrier is the poor quality of data collected. e.g., the different web page structures and different uses of metadata standards. In this line it is important to promote dialogue on data quality, data management including standards, metadata and interoperability are key actions. Therefore, in this context it is important to integrate initiatives such as the one developed by the Citizen Science Association Data & MetaData Working Group [18] by CS platform developers. But an alignment between the PPSR metadata standard with the SDG indicators is still work to be done. In this line, we suggest that it is key to define how to standardize the reporting of specific contributions/results from CS with the set of indicators proposed by the quantitative SDG framework. The main aim of this study was focused on understanding the advantages and limitations of text-classification techniques to enhance the understanding of the relationship between CS and SDGS. As described in the discussion section the three different techniques we used i.e., nCoder, ESA, OSDG, have several benefits and disadvantages. In summary, although the results obtained using nCoder are more aligned with the results of the manual classification the process can be overall time consuming and later using a trained classifier is not possible. In comparison the effort required from human coders is minimum for ESA, however the quality and the levels of details present in the reference article, i.e., Wikipedia, could impact the results. To this end, the paper contributes by sharing our experience in using different text classification techniques to classify CS project descriptions. On the other hand, deep learning models such as i.e., BERT is becoming the state-of-the-art model solution for multiple natural language processing tasks. Although obtaining satisfactory amounts of training data to train machine learning models is a challenge, specially in the case of SDGs where we need to classify data for 17 different categories, the use of advanced techniques such as BERT in future can provide more accurate results also considering multiple languages in the future. However, drawbacks inherited in black box models such as lack of explainability and interpretability should be taken into consideration. In addition to the advantages and limitations of the techniques explored in this paper, we have observed some interesting findings regarding the relationship between SDGs and CS. As indicated in the Introduction, several previous studies are focused on understanding the relationship of CS with one SDG. But in this study, we have observed that in most cases projects are associated with multiple SDGs. One possible explanation to be further explored is to analyze if the project has some SDGs as part of their own goals to be solved, but also other SDGs appear as part of the methodology or outcomes produced by the project. For instance, this would explain why SDG4 ‘Quality Education’ is one the most popular ones although most of the projects are not focused on scientific research of educational contexts. In this line, another future research line can be focused on identification of SDGs connected to other ones. Next steps in our research include to extend our dataset, and analyze the maximum number of projects included in our DB. This will allow us to identify which SDGs are the most representative ones in CS, relations between different SDGs, and how SDGs are related to different research areas associated with each project. Declarations Acknowledgment This work has been partially funded by the EU project ‘CS Track’ under the H2020 program (grant id: 87252) and the Ramón y Cajal programme of the Spanish Ministry of Science and Innovation (P. Santos). Author Contribution Patricia Santos has contributed to the leading of the manuscript, conceptual idea of the research line, to the analysis and writing of the paper.Ishari Amarashinghe has contributed to the conceptual idea of the research line, to the analysis and writing of the paper.Miriam Calvera, Cleo Schulten, Ulrich Hoope, David Roldán-Álvarez, Fernando Martínez-Martínez to the analysis of the datasets. References L. Fonseca, and F. Carvalho, “The reporting of SDGs by quality, environmental, and occupational health and safety-certified organizations” Sustainability , 11(20), 5797, 2019. J. Wu, S. Guo, H. Huang, W. Liu, and Y. Xiang, “Information and communications technologies for sustainable development goals: state-of-the-art, needs and perspectives. IEEE Communications Surveys & Tutorials , 20(3), 2389-2406, 2018. A. López-Vargas, M. Fuentes, and M. Vivar, “Challenges and opportunities of the internet of things for global development to achieve the United Nations sustainable development goals”. IEEE Access , 8, 37202-37213, 2020. S. Fritz, L. See, T. Carlson, M.M. Haklay, J.L. Oliver, D. Fraisl, D., R. Mondardini, M. Brocklehurst, L.A. Shanley, S. Schade, U. When, T. Abrate, J. Anstee, S. Arnorld, M. Billot, J. Campbell, J. Espey, M. Gold, G. Hager, S. He, L. Hepburn, A. Hsu, D. Long, J. Masó, I. McCallum, M. Muniafu, I. Moorthy, M. Obersteiner, A.J. Parker, M. Weisspflug and S. West “Citizen science and the United Nations sustainable development goals”. Nature Sustainability , 2(10), 922-930, 2019. D. Fraisl, J. Campbell, L. See, U. Wehn, J. Wardlaw, M. Gold, I. Moorthy, R. Arias, J. Piera, J.L. Oliver, J. Masó, M. Penker and S. Fritz, “Mapping citizen science contributions to the UN sustainable development goals”. Sustainability Science , 15(6), 1735-1751, 2020. M. V. Eitzel, J. L. Cappadonna, C. Santos-Lang, R.E. Duerr., A. Virapongse, S.E. West, C.C.M. Kyba, A. Bowser, C. B. Cooper, A. Sforzi, A. Nova-Metcalfe, E. S Harris, M. Thiel, M. Haklay, L. Ponciano, J. Roche, L. Ceccaroni, F. M. Shilling, D. Dörler, F. Heigl, T. Kiessling, B. Y. Davis and Q. Jiang, “Citizen science terminology matters: Exploring key terms”. Citizen science: Theory and practice , 2(1), 2017. F. Heigl, B. Kieslinger, K.T. Paul, J. Uhlik and D. Dörler, “Opinion: Toward an international definition of citizen science”. Proceedings of the National Academy of Sciences , 116(17), 8089-8092, 2019. S. Schade, M. Pelacho, T.C. van Noordwijk, K. Vohland, S. Hecker and M. Manzoni, “Citizen science and policy. In The science of citizen science” Springer, Cham, pp. 351-371, 2021. K. Vohland, C. Göbel, B. Balázs, E. Butkevičienė, M Daskolia, B. Duží, S. Hecker, M. Manzoni and S. Schade (2021). “Citizen Science in Europe”. The Science of Citizen Science, 35, 2021 H.Y. Liu, D. Dörler, F. Heigl, and S. Grossberndt, “Citizen Science Platforms”, The Science of Citizen Science, 439, 2021. L. Quinlivan, D.V. Chapmanand T. Sullivan, “Validating citizen science monitoring of ambient water quality for the United Nations sustainable development goals”. Science of the Total Environment , 699, 134255, 2021. S. Koffler, C. Barbiéri, N.P. Ghilardi-Lopes, J.N. Leocadio, B. Albertini, T.M. Francoy and A.M. Saraiva, “A buzz for sustainability and conservation: The growing potential of citizen science studies on bees”. Sustainability , 13(2), 959, 2021. European Commission, Joint Research Centre (JRC), “An inventory of citizen science activities for environmental policies”. European Commission, Joint Research Centre (JRC) [Dataset] 2018 PID: http://data.europa.eu/89h/jrc-citsci-10004 Bio Innovation Service. (2018). Citizen science for environmental policy: development of an EU-wide inventory and analysis of selected practices. Final report for the European Commission, DG Environment under the contract 070203/2017/768879/ETU/ENV. A. 3, in collaboration with Fundacion Ibercivis and The Natural History Museum. D. Roldán-Álvarez, F. Martínez-Martínez, E. Martín, and P.A. Haya, “Understanding Discussions of Citizen Science Around Sustainable Development Goals in Twitter,” IEEE Access , vol. 9, pp. 144106–144120, 2021, doi.org/10.1109/ACCESS.2021.3122086 M. Riegner, “Implementing the Data Revolution for the Post-2015 Sustainable Development Goals: Toward a Global Administrative Lawof Information,” World Bank Legal Rev. 7, 2016. U. Sturm, S. Schade, L. Ceccaroni, M. Gold, C. Kyba, B. Claramunt, M. Haklay, D. Kasperowski, A. Albert, J. Piera, J. Brier, C. Kullenberg, S. Luna, “Defining principles for mobile apps and platforms development in citizen science,” Research Ideas and Outcomes, vol . 3, 2017, doi.org/10.3897/rio.3.e21283 A. Bowser, “Standardizing Citizen Science?,” Proc. Biodiversity Information Science and Standards 1: e21123 , 2017, doi.org/10.3897/tdwgproceedings.1.21123 Citizen Science Association Data & Meta Data Working Group, “PPSR Core, A Data Standard for Public Participation in Scientific Research (Citizen Science),” https://core.citizenscience.org. 2021. Australia, S. D. S. N., “Getting started with the SDGs in Universities: A Guide for Universities. Higher Education Institutions, and the Academic Sector,” https://ap-unsdsn.org/wp-content/uploads/University-SDG-Guide_web.pdf. 2017. Z. Cai, A. Siebert-Evenstone, B. Eagan, D.W. Shaffer, X. Hu, and A.C Graesser, “nCoder+: a semantic tool for improving recall of nCoder coding,” Proc. Int. Conf. on Quantitative Ethnography, pp. 41–54, 2019. E. Gabrilovich and S. Markovitch, “Computing semantic relatedness using Wikipedia-based explicit semantic analysis,” Proc. Int. Joint Conf. on Artificial Intelligence, pp.1606–1611, 2007. L. Pukelis, N.B. Puig, M. Skrynik, and V. Stanciauskas, “OSDG--Open-Source Approach to Classify Text Data by UN Sustainable Development Goals (SDGs),” CoRR , vol. abs/2005.14569, arXiv preprint arXiv:2005.14569, 2020. J. Devlin, M. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” CoRR , vol. abs/1810.04805, arXiv preprint arXiv:1810.04805, 2018. K. Shulla, W.L. Filho, J.H. Sommer, A.L. Salvia, and C. Borgemeister, “Channels of collaboration for citizen science and the sustainable development goals,” Journal of Cleaner Production, vol. 264, 2020. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 15 Dec, 2024 Read the published version in International Journal of Data Science and Analytics → Version 1 posted Editorial decision: Revision requested 07 Oct, 2024 Reviews received at journal 25 Sep, 2024 Reviews received at journal 21 Aug, 2024 Reviewers agreed at journal 17 Aug, 2024 Reviewers agreed at journal 13 Aug, 2024 Reviewers agreed at journal 13 Aug, 2024 Reviewers agreed at journal 12 Aug, 2024 Reviewers invited by journal 12 Aug, 2024 Editor assigned by journal 06 Aug, 2024 Submission checks completed at journal 22 Jul, 2024 First submitted to journal 22 Jul, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4781489","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":341471094,"identity":"e0784ce8-4081-4242-910f-801af92557ce","order_by":0,"name":"Patricia Santos","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA/0lEQVRIiWNgGAWjYLCCBAYJOFuOgYHxAQMDGz71zEhaDjAwGANFDAhrgQOglsQGQlrM2c8f+/CgxkKegf/ws8cfKurSN9xuZmD4UHYYpxbLnmTmGQnHJAwbJNLMDQ6cOZy74c5hBsYZ53BrMTiQzAx0jARjgwSDmcTBtgO5G27kH2DmbcOj5fxjsBb7Bv7j34Ba6tINbiQzMP/Fp+UGxBagr3NAtjAngLUw4tFiOeOxMQPQL8ltEjllEmfOHDacCfTLwZ5z6Ti1mPMnPmb8UVNn289/fJtERUWdPN/tZsYHP8qscTsMxkBEhAQ4fnADA0whCUyhUTAKRsEoGNkAAFe6VQ6YvJNyAAAAAElFTkSuQmCC","orcid":"","institution":"Universitat Pompeu Fabra","correspondingAuthor":true,"prefix":"","firstName":"Patricia","middleName":"","lastName":"Santos","suffix":""},{"id":341471095,"identity":"aa7881a9-44e7-4a8b-9822-ceeefc6d9161","order_by":1,"name":"Ishari Amarashinghe","email":"","orcid":"","institution":"Universitat Pompeu Fabra","correspondingAuthor":false,"prefix":"","firstName":"Ishari","middleName":"","lastName":"Amarashinghe","suffix":""},{"id":341471096,"identity":"1779a6ab-2c48-4513-9e8d-10cc11a3dd24","order_by":2,"name":"Miriam Calvera-Isabal","email":"","orcid":"","institution":"Universitat Pompeu Fabra","correspondingAuthor":false,"prefix":"","firstName":"Miriam","middleName":"","lastName":"Calvera-Isabal","suffix":""},{"id":341471097,"identity":"42f36a83-801d-4afc-a6b5-91bbf9bb80b5","order_by":3,"name":"Cleo Shulten","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Cleo","middleName":"","lastName":"Shulten","suffix":""},{"id":341471098,"identity":"29f1589a-6877-4db1-bf38-c0bcf444751b","order_by":4,"name":"Ulrich Hoppe","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Ulrich","middleName":"","lastName":"Hoppe","suffix":""},{"id":341471099,"identity":"8e42cc3a-1ec8-4e83-94d8-f89dff0793bd","order_by":5,"name":"David Roldán-Álvarez","email":"","orcid":"","institution":"Universidad Rey Juan Carlos","correspondingAuthor":false,"prefix":"","firstName":"David","middleName":"","lastName":"Roldán-Álvarez","suffix":""},{"id":341471100,"identity":"689a8f3d-9e9c-42be-8c1f-ff25c0dc9a16","order_by":6,"name":"Fernando Martínez-Martínez","email":"","orcid":"","institution":"Universidad Rey Juan Carlos","correspondingAuthor":false,"prefix":"","firstName":"Fernando","middleName":"","lastName":"Martínez-Martínez","suffix":""}],"badges":[],"createdAt":"2024-07-22 11:29:43","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4781489/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4781489/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s41060-024-00695-7","type":"published","date":"2024-12-15T15:57:04+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":62779101,"identity":"76524feb-e2d0-43d9-b34b-16c2b717604a","added_by":"auto","created_at":"2024-08-19 11:18:04","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":80052,"visible":true,"origin":"","legend":"\u003cp\u003eDataset preparation and manual coding process\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-4781489/v1/838bfb3e78e69fc8ebfeaf2a.png"},{"id":62779512,"identity":"126565bf-0e4b-476c-b9f3-a5f4ca285691","added_by":"auto","created_at":"2024-08-19 11:26:04","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":73982,"visible":true,"origin":"","legend":"\u003cp\u003eProcessing scheme for associating SDGs to CS projects through ESA\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-4781489/v1/09098f395f0c47de73ec3dc3.png"},{"id":62779513,"identity":"a80f2e43-cfcc-4c86-bdfb-1d6890c7ded3","added_by":"auto","created_at":"2024-08-19 11:26:04","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":166915,"visible":true,"origin":"","legend":"\u003cp\u003eManual coding results of CS project descriptions to indicate the presence of SDGs.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-4781489/v1/d3b50fb2aaad7a164e3fba4f.png"},{"id":62778427,"identity":"1bd5986d-5f3d-4d9d-9760-f73ba406ec0b","added_by":"auto","created_at":"2024-08-19 11:10:04","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":37991,"visible":true,"origin":"","legend":"\u003cp\u003eJaccard measure of similarity (formula)\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-4781489/v1/a92f91177a7391ede8ce9d81.png"},{"id":62779104,"identity":"88640200-a96d-4774-8fd5-85703148d874","added_by":"auto","created_at":"2024-08-19 11:18:04","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":244528,"visible":true,"origin":"","legend":"\u003cp\u003eSDG similarity matrix based on associations through manual coding\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-4781489/v1/7e5bf7deed1d204f9d010491.png"},{"id":62778430,"identity":"5426bf39-864d-482f-90fc-59c907c1b2cd","added_by":"auto","created_at":"2024-08-19 11:10:04","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":38195,"visible":true,"origin":"","legend":"\u003cp\u003eCluster dendrogram and ensuing silhouette values (resulting from a cut at Height = 1.4)\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-4781489/v1/f3360cdff6c9cdd42b20b45e.png"},{"id":62778434,"identity":"6bffa598-31bc-44d8-957c-36757d141b98","added_by":"auto","created_at":"2024-08-19 11:10:05","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":140444,"visible":true,"origin":"","legend":"\u003cp\u003eSDG classification results - manually labeled positive samples vs. results from computational techniques\u003c/p\u003e","description":"","filename":"7.png","url":"https://assets-eu.researchsquare.com/files/rs-4781489/v1/bbc8d437eb7a4b0c71cc9a53.png"},{"id":62779514,"identity":"e8178793-5c43-4efb-a424-b7bec2fa8132","added_by":"auto","created_at":"2024-08-19 11:26:04","extension":"png","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":146747,"visible":true,"origin":"","legend":"\u003cp\u003eSDG classification results - manually labeled negative samples vs. results from computational techniques\u003c/p\u003e","description":"","filename":"8.png","url":"https://assets-eu.researchsquare.com/files/rs-4781489/v1/bd306f0d2260ba8082e74540.png"},{"id":71552348,"identity":"952223bb-6add-488b-b78f-45cf11baa66d","added_by":"auto","created_at":"2024-12-16 16:05:14","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1249245,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4781489/v1/6d8d3e7c-5651-40d7-bd85-9a399aefaba2.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Mapping Sustainable Development Goals to Citizen Science projects - a comparative evaluation of automatic classifiers","fulltext":[{"header":"1\tIntroduction ","content":"\u003cp\u003eThe seventeen Sustainable Development Goals (SDGs) formulated and set up by the United Nations General Assembly (UN-GA) in 2015 provide a framework of sustainability goals and targets that is universally accepted. The SDGs summarize priority action areas to help society achieve justice, prosperity, and environmental security (see: https://sdgs.un.org/goals). Improving the understanding of how the SDGs are implemented around the world is a priority task of the UN 2030 Agenda [1]. The potential of Information and Communication Technologies to achieve the SDGs have been explored by some authors [2],[3]. But one of the main problems is related to having access to data associated to SDGs. Data related to the SDGs is sourced primarily from global databases maintained by international organizations, national statistical offices, and other government agencies. This data is costly to obtain and still incomplete in detail and coverage. Fritz et al. [4] state that traditional data sources are not sufficient for measuring the achievement of SDGs so that it is necessary to identify new and additional information sources. In this line, recent studies show the value of using data from Citizen Science for improving the reporting of SDGs [5].\u003c/p\u003e\n\u003cp\u003eCitizen Science (CS) is a broad term that is interpreted differently depending on specific science cultures, research orientations, disciplines, or the types of citizen activities. As stated by Eitzel et al. [6] no single term is appropriate for all contexts and practices associated with the term CS. In this paper, we refer to the definition provided by the European Commission in the Science with and for Society (SwafS) work programme 2018-2020 as a baseline: \u0026ldquo;Citizen Science should be understood broadly, covering a range of different levels of participation, from raising public knowledge of science, encouraging citizens to participate in the scientific process by observing, gathering and processing data, right up to setting scientific agenda and co-designing and implementing science-related policies. It could also involve publication of results and teaching science\u0026rdquo;. According to Heigl et al [7], public participation in scientific projects is increasing around the world as part of projects and activities labeled as CS. Since 2010, CS has been explicitly placed in different European science policy frameworks aligned with the objectives of the Europe 2020 strategy and related to more specific areas such as the Digital Agenda, Science 2.0, Responsible Research and Innovation, Open Science, and the SDGs [8]. For this reason, Fritz et al in [4] pointed out the importance of considering the transformative role of CS as an engine for monitoring SDGs. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003eCS is a growing and flourishing practice of CS in Europe and across the world [9]. A strategic, scalable and pluralistic approach is needed to understand how CS is reported. Nowadays, multiple CS platforms, websites and social media accounts exist, and therefore the ensuing knowledge is dispersed on the Internet [10]. Therefore, although CS has the potential to support a better understanding of the status of SDGs, it is necessary to explore technical approaches to solve the problem of scattered data sources on the Internet and to support the automated analysis of large amounts of textual information. Currently, most of the studies or projects on inter-relating CS and the SDGs deal with limited samples of initiatives focused mainly on one SDG [11],[12],[13],[14]. Others are based on reporting how traditional (human-based) analytical methods can be applied to understand the connection between CS and SDGs [5]. So far, most of these studies are focused on topics related to environmental research/impact, but there is still a lack of knowledge of how CS contributions can be mapped to the total of the 17 SDGs.\u003c/p\u003e\n\u003cp\u003eIn this paper, we explore and exemplify opportunities for combining web-data mining techniques and automatic classifiers to enhance the understanding of the inter-relation between CS and the SDGs. This work is based on a cooperation and partnership in a SwafS-H2020 research project called: \u0026lsquo;CS Track: Expanding our knowledge on Citizen Science through analytics and analysis\u0026rsquo; (see: cstrack.eu). CS Track relies on a combination of web analytics techniques and classical social studies methods with the aim of broadening the knowledge about CS. The work done by Roldan-\u0026Aacute;lvarez et al [15] is a first example showing the benefits of applying social networks analysis (on Twitter) to connect data from CS and SDGs. Yet, as stated by these authors, Twitter blogosphere is a limited source of information related to CS activities, which are more often and, in more detail, represented in project websites and on specific CS online platforms. For this reason, one main goal of CS Track has been to build a central database aiming to compile a comprehensive collection of CS projects, essentially relying on available web data. Building such a DB is the first step to solve the problem of having the data from CS projects dispersed in multiple online sites. Once this is solved, the next step involves exploring how the textual data from CS projects reported online can be connected with the SDGs and therefore be analyzed. In this paper, we show the benefits of applying automatic classifiers to understand the connection between the data from CS projects with the SDGs. In particular, we compare the application of nCoder, ESA and BERT (more detail below).\u003c/p\u003e\n\u003cp\u003eThe goal of this article is to present opportunities, achievements, and future challenges in using computational analytics to better understand the connection between CS and the SDGs. The work in its status does not fully cover SDGs in CS, but it evaluates and shows the potential of the text-classification techniques for identifying SDGs in CS project descriptions and for assessing trends in connection of CS and SDGs based on available data. Accordingly, our main research question is: \u003cem\u003eHow can a data analytics approach based on web-based data mining and automatic classifiers contribute to the reporting of SDGs related to CS activities and projects?\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThe rest of this article is organized as follows: Section 2 reviews the state of the art. Section 3 analyzes emerging computational analytics techniques that can be applied for the purpose of this research. Section 4 presents preliminary results to show the potential of the techniques described in section 3. In section 5 the limitations and challenges faced during the study. The final section identifies future research lines to be further explored and formulates conclusions based on the work so far.\u003c/p\u003e"},{"header":"2\tBackground","content":"\u003ch1\u003e2.1 Traditional data sources to report SDGs\u003c/h1\u003e\n\u003cp\u003eDifferent reports from the United Nations, claim that a data revolution for sustainable development is needed. In general the data used to understand the SDGs comes from official statistics for governance, the central one is the UN Statistical Commission but also national and regional official institutions. The global indicator framework for Sustainable Development Goals was developed by the Inter-Agency and Expert Group on SDG Indicators (IAEG-SDGs) and agreed upon at the 48th session of the United Nations Statistical Commission held in March 2017 \u003csup\u003e\u003csup\u003e[1]\u003c/sup\u003e\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003eThis framework is refined annually and reviewed by the Statistical Commission. The global indicator framework is complemented by indicators at the regional and national levels, which will be developed by Member States. This includes a list of quantitative indicators aligned with the 17 SDGs.\u003c/p\u003e\n\u003cp\u003eAs stated by Riegner [16] one common problem in this process is that often it is difficult to have updated data, sometimes it is inaccurate, unavailable, or based on unreliable estimates. On the other hand, statistical capacities in many developing countries are insufficient. The author states that international standards need to be improved to be able to enhance these problems. In this line, \u0026ldquo;The data revolution report\u0026rdquo;\u003csup\u003e\u003csup\u003e[2]\u003c/sup\u003e\u003c/sup\u003e provides a reference point for efforts to implement the data revolution under the SDG framework. The report makes emphasis on the importance of agreeing on basic principles related to data revolution and adopting standards for openness and exchange of data and for the protection of human rights. Part of the recommendations to progress with the data revolution include the use of new data sources and big data analytics and the diffusion of information technologies.\u003c/p\u003e\n\u003cp\u003eIn the last lustrum, some authors have demonstrated the value of using data from CS for the SDGs [4],[5] among others already cited in the introduction. Fraisl et al [5] propose a methodology, to conduct a systematic review of the SDG indicators for CS, that has been applied as part of the manual coding process described in this paper. In this paper we propose an extension of this methodology by including the application of text-classification techniques. In this context there is no previous research proposing a solution in relation to the diffusion issue suggested by the data revolution report in the context of CS, and how the application of text-classification techniques have the potential to classify big amounts of CS data to SDGs.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.2 Citizen Science data\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe term Citizen Science was coined in the 1990s and has gained popularity since then [9]. One recommendation for future development of CS is to anchor the citizens in research and development, to understand better the positioning of CS with the global agenda of the SDGs. In the last decade, the number of citizen science programs, professional networks for CS stakeholders and online communities of citizen scientists have strongly increased, leading to the generation of several CS platforms. Some examples of CS Platforms are for instance: Zooniverse (see: https://www.zooniverse.org/), eu-citizen science (see: https://eu-citizen.science/) or Scistarter (see: https://scistarter.org/). According to Liu et al [10] CS platforms are: \u0026ldquo;web-based infrastructures with one single entrance point that contain one or several of the following functionalities: (1) present active citizen science projects and activities; (2) display citizen science data and information; (3) provide overall guidelines and tools that can be used to support citizen science projects and activities in general; (4) present good practice examples and lessons learned; and (5) offer relevant scientific outcomes for people who are involved or interested in citizen science\u0026rdquo;. At the same time these authors identify the following types of CS platforms: commercial; for specific projects; for specific scientific topics; national or continental platforms.\u003c/p\u003e\n\u003cp\u003eOther sources of information in CS are individual websites of projects, and (if any) associated social media accounts (e.g., Twitter, Facebook) In this paper, our attention is especially focused on CS platforms as the main source to find information summarizing the main goals of CS projects (i.e., project descriptions). We also discuss the potential of using data from Twitter to establish connections between CS and SDGs.\u003c/p\u003e\n\u003cp\u003eAccording to Sturm et al. [17], CS platforms can be developed as a technical framework designed to store data and information from multiple types of applications. In this context, the collaborative success for citizen science and public participation in scientific research (PPSR) led to the development of the PPSR_CORE metadata standard [18], [19]. The PPSR is proposed as the CS metadata standard with the main aim to provide a common consensus on how to structure the data from CS projects in technical platforms. Unfortunately, the PPSR_Core standard is not yet established as the common metadata standard in the CS community, therefore only few CS platforms now implement the standard at the level of the project descriptions, something that makes the automatic analysis of the content reported on existing platforms a challenge. In addition to this, the current version of the proposed metadata standard does not include any attribute related to the SDGs. As a consequence, there is no official standardized method at the moment to report data from CS contributions connected with the SDG quantitative framework.\u003c/p\u003e\n\u003cp\u003eOne of the most popular CS platforms, Scistarter, integrates a system called OSDG as a way to classify text data from projects\u0026rsquo; descriptions to SDGs. OSDG (see: https://osdg.ai/) is a citizen science initiative to (manually and automatically) classify textual data to SDGs. This initiative is not focused on classifying data from CS projects exclusively. In this paper, we have selected this system, among other text-classification techniques to understand the potential and limitations of these types of classifiers.\u003c/p\u003e\n\u003cp\u003eOur plan is to apply text-classification techniques to obtain first insights of the relationship between CS and SDGs. For this reason, before mapping specific quantitative contributions from CS with the SDGs indicators, we propose that it is necessary to first understand how existing CS projects are covering topics related to the SDGs. Opportunities to achieve this first step is what is covered in this paper. Once we are able to classify specific CS projects associated with specific SDGs, then the next step is to analyze the association of the derived contributions of the projects and the SDGs quantitative indicators.\u003c/p\u003e\n \u003cp\u003e[1] https://unstats.un.org/sdgs/indicators/indicators-list/\u003c/p\u003e\n \u003cp\u003e[2] https://www.undatarevolution.org/report/\u003c/p\u003e"},{"header":"3\tDataset Preparation","content":"\u003ch2\u003e3.1 Building a source of knowledge: the CS Track Database\u003c/h2\u003e\n\u003cp\u003eIn order to build our source of knowledge, with data from CS projects to be analyzed, a Database (DB) has been created by manually searching for existing CS platforms (e.g. https://eu-citizen.science/; https://scistarter.org/) containing CS projects (e.g. FLOODUP or COVID near you). First, we manually selected CS platforms from Europe and platforms containing projects conducted fully online. So far, a total of 56 platforms have been used to extract project descriptions, the list of CS platforms and projects is consistently updated for the duration of the project (2019-2022). As a first exercise to understand the potential of the techniques explored in this study, 16 CS platforms have been considered to extract and analyze CS project descriptions (more info about the dataset used below).\u003c/p\u003e\n\u003cp\u003eThe CS Track DB opens a new perspective on CS knowledge by observing and characterizing initiatives through a quantitative approach that relies on web-based data mining and network analytics. A crawler has been developed taking into account the different web structures of the selected CS platforms. The detailed process followed is out of the scope of this paper, but as a summary the main steps are as follows: (1) checking if a robots exclusion protocol allows data extraction (2) open URL of the websites of the platforms selected; (3) retrieve URL for specific CS projects listed on the selected platform, (4) open the URL (5) select, extract and classify the information from the website according to its HTML structure; (6) data cleaning and classification (following the Project Metadata Model (PMM) from PPRS_Core metadata standard) and (7) store the data into a single database organized by typology of data. To ensure GDPR compliance, a data anonymization process has also been applied.\u003c/p\u003e\n\u003cp\u003eSo far, the DB contains 4737 CS projects and the additional information that has been gathered in the DB is an information resource that can be analyzed on its own as well as serve as a useful starting point for other research, as the one proposed in this manuscript. The collection of CS projects in our DB reveals a complex picture (e.g., the unstructured reporting by CS platforms, the lack of information for some attribute, the projects that are too diverse) that challenges the analysis of specific topics of interest (i.e., SDGs covered by CS projects). In this research, the unit of analysis is the description of individual CS projects extracted from CS platforms. For this reason, we propose to explore the potential benefits (and limitations) of different techniques based on text-classification. In the following subsections we describe the process needed to apply the techniques selected, to proceed with this first exercise a subset of the total number of projects in the DB has been selected. \u003c/p\u003e\n\u003ch2\u003e3.2 Manual classification: hand-coding of the dataset\u003c/h2\u003e\n\u003cp\u003eWe decided to start with an initial subset of the data from our DB (see Figure 1). A total of 208 projects from 16 CS platforms were randomly selected with the following criteria: project descriptions should be in english; platforms should contain a list of projects situated in Europe or should be projects conducted online.\u003c/p\u003e\n\u003cp\u003eFollowing the initial steps defined by Fraisl et al. [5], we extracted and reviewed SDGs targets and indicators metadata from the official websites of the United Nations (i.e., https://unstats.un.org/sdgs/metadata/). This review was done by 3 researchers to identify a list of keywords to be applied for SDG classification purposes. For this study we used the list published by Monash University and Australia S.D.S.N. [20].\u003c/p\u003e\n\u003cp\u003eTwo authors of this paper and two research assistants (n=4) were involved in manual coding of the 208 project descriptions into relevant SDGs. In addition to the keywords selected in the previous step, new ones emerge from the manual coding process. The initial manual coding provided a ground truth against which the performance of the three methods (i.e nCoder, ESA and OSDF) could be evaluated. Results obtained via manual coding are presented in Figure 3.\u003c/p\u003e"},{"header":"4\tA comparison of different text classification techniques","content":"\u003cp\u003eIn this study three different text classification techniques are compared: First, nCoder has been selected because this technique is the closest one to the process followed for manual classification. nCoder automates the process of coding, allows researchers to discover concepts in data flexibly, so in this case researchers provided expert knowledge. Second, Explicit Semantic Analysis (ESA) because for this an external source of knowledge (in this case Wikipedia articles about each SDG) is used to to calculate semantic similarities between words and texts. Third, we compare our results with the existing tool \u0026lsquo;OSDG\u0026rsquo; also based on text-classification from multiple sources.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAdditionally, we discuss the results obtained with these three techniques with the results from a previous study [14] where BERT was used to classify CS data from Twitter to SDGs.\u003c/p\u003e\n\u003cp\u003eMore detail about the techniques compared and discussed in this manuscript continue as follows:\u003c/p\u003e\n\u003ch2\u003e4.1 nCoder\u003c/h2\u003e\n\u003cp\u003enCoder is a tool that enables to train classifiers that automatically label text according to some set of classification rules [21]. The tool is freely available online and an R package. The process of training text classifiers in nCoder consists of several steps. First, it is required to define the keywords and/or regular expressions to describe language patterns that we are looking for in the input datasets. nCoder uses a simple regular expression matching algorithm to automatically label the text data for a targeted concept. In this study, we used keywords that were derived from related sources (see section above) and also used a TF-IDF (term frequency-inverse document frequency) approach to understand relevant concepts. During the next step, nCoder prompts the human rater a randomly selected sample from the input dataset that consists of 80 samples (training set). The human rater is required to go through the 80 samples and to indicate the presence of the targeted concept, i.e., a particular SDG, by indicating yes/no. After manually coding 80 samples the tool provides statistics based on kappa and rho measures to indicate the validity of the automatic classifier. Kappa measure indicates the agreement between human coder and machine coder for coding the training set and rho measure indicates the generalizability of automatic coding. If the first round of training was not reliable therefore did not reach the intended thresholds for kappa and rho statistics (Kappa \u0026gt; 0.65 Rho \u0026lt; 0.05) then modifying the keywords and/or regular expressions is necessary to start a new round of revalidation.\u003c/p\u003e\n\u003ch2\u003e4.1 Explicit Semantic Analysis\u003c/h2\u003e\n\u003cp\u003eExplicit Semantic Analysis (ESA) is a method to calculate semantic similarities between words and texts [22]. The basis of the calculations is a precalculated inverted index built from encyclopedic data. The index can be described as a matrix of terms and articles with the terms being reduced to their word stems. We can use the index to turn a term into a representative vector with tf-idf values for the corresponding articles. Using term vectors, we can calculate the cosine similarity. By adding the term vectors of all terms occurring in a text we can upscale this to compare texts with each other. In our implementation of this method, we opted to improve the performance by filtering the text\u0026rsquo;s terms first by using tf-idf and disregarding terms that score lower than 20% of the highest tf-idf score.\u003c/p\u003e\n\u003cp\u003eFor the association of projects with SDGs through ESA, we used the existing Wikipedia articles of all SDG as reference documents as a basis to calculate the corresponding text vectors. For any given project description, we can then calculate the corresponding text vector and the similarity with each SDG\u0026rsquo;s vector. The results are then ranked based on similarity. The similarity results are rarely ever close to 0 so that a cutoff for exclusion has to be determined and chosen. The underlying processing scheme is illustrated in Figure 2.\u003c/p\u003e\n\u003cp\u003eFig 2. Processing scheme for associating SDGs to CS projects through ESA\u003c/p\u003e\n\u003ch2\u003e4.3 OSDG: Open-Source Approach to Classify Text Data\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eAs mentioned in the Introduction section, OSDG is an open-source initiative aimed to classify text data according to SDGs. In Pukelis et al. [23], the authors propose a way to integrate data from multiple sources into a single framework for SDG classification. OSDG combines the use of supervised machine learning models (e.g., it tooks the most significant features for each SDG), and from unsupervised machine learning looking at the most important words for each topic. More in concrete, the system builds an integrated ontology from the feature sets identified in previous research and then matches the ontology items to the Fields of Study from Microsoft Academic.\u003c/p\u003e\n\u003cp\u003eIn this study we use this tool as a way of comparison and validation, using the same dataset used with nCoder and ESA\u003c/p\u003e\n\u003ch2\u003e4.4 BERT and machine learning models\u003c/h2\u003e\n\u003cp\u003eOne of the most popular methods for text classification is the use of deep learning models. In this scenario, BERT (Bidirectional Encoder Representations from Transformers) has become one the most widely used tools in recent years to classify texts in different areas [24]. BERT is designed to pretrain deep bidirectional representations from unlabelled texts by jointly conditioning on both left and right context in all layers. One of the main issues when using deep learning models is the creation of a dataset that allows the training of that model so text can be classified into the corresponding categories accurately. However, nowadays we can find many sources with information that can be used to populate those training datasets. Using APIs or techniques such as Web Scraping, we can gather data that can be used to train deep learning models. Moreover, it would be easy to increase the number of texts of the dataset since information on the Internet is growing every day.\u003c/p\u003e\n\u003cp\u003eThere are prior studies in which the authors used Twitter to collect tweets regarding SDGs to train a deep learning model based on BERT [15]. In this study, 57,843 tweets were directly downloaded from the Twitter API by using English keywords, French keywords and Spanish keywords related to SDGs. The dataset was composed of unique tweets. After classifying 29,543 tweets, the results showed that around 25% of them were about SDG13, which talks about taking urgent action to combat climate change and its impact.\u003c/p\u003e"},{"header":"5\tResults","content":"\u003ch2\u003e5.1 Manual classification\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eFigure 3 shows the results of the manual coding process (classification of selected 208 CS project descriptions into relevant SDGs) which we used as a baseline. Given the considerably large number of projects for manual coding across 17 SDGs it had not been possible to assign two people to do manual coding for every SDG, therefore, to report inter-rater reliability. However, the researchers and research assistants discussed regularly and specially when in doubt all discussed and re-coded until disagreements were solved.\u003c/p\u003e\n\u003cp\u003eFig 3. Manual coding results of CS project descriptions to indicate the presence of SDGs.\u003c/p\u003e\n\u003ch2\u003e5.2 SDG Dependencies\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eIn the comparative analysis of SDG associations to the selected sample of projects, we have focused on SDGs as separate entities. However, SDGs and the ensuing classifications are partly overlapping. Based on our sample of classified projects, we can determine the overlap or similarity between SDGs. For the actual calculation, we use the Jaccard measure of similarity (see Figure 4). For two given SDGs X and Y, the Jaccard similarity would be calculated as the following proportion:\u003c/p\u003e\n\u003cp\u003eThe value of this measure would depend on the associations and co-occurrences determined by the different methods. For a baseline assessment of similarities, we rely on the results of the manual coding that we have also considered as a ground truth for comparing the other methods. Figure 5 shows the ensuing similarity matrix. Here, the very general and over-arching SDG #17 (Strengthen the means of implementation and revitalize the global partnership for sustainable development) has not been included since it was never assigned to one of the sample projects. For the given sample, the highest similarity (based on co-occurrences) is between SDGs #5 (Gender equality) and #8 (Decent work and economic growth).\u003c/p\u003e\n\u003cp\u003eThe similarity matrix can serve as a basis for clustering the SDGs by their overlaps. Figure 6 shows the cluster dendrogram that results from applying agglomerative hierarchical clustering, together with the ensuing silhouette values after a cut at height = 1.4. This cut gave the best overall silhouette value.\u003c/p\u003e\n\u003cp\u003eBased on the silhouette values, the strongest clusters are 5-8-1 (Gender equality, Decent work and economic growth, No poverty) and 6-14-12 (Clean water and sanitation, Life below water, Responsible consumption and production) whereas SDGs 9 (Industry, innovation and infrastructure) and 16 (Peace, justice and strong institutions) are not strongly connected to other SDGs.It is important to consider that these are empirical findings in which the associations between SDGs depend on the actual overlaps in the given sample of 208 CS projects, with the basic assignments depending on human judgement. One might, e.g., have expected a stronger association between SDGs 2 (Zero hunger) and 3 (Good health and well-being), yet this is not backed by the orientation of the projects in our sample. The point here is to see the interdependencies of SDGs in practice as contrasted to \u0026ldquo;semantic expectations\u0026rdquo;. If we can further corroborate these dependencies, we can use them to improve our automatic detectors. E.g., if SDGs A and B are known to be closely related but our \u0026quot;detector\u0026quot; only finds A we may use this background knowledge to also infer B.\u003c/p\u003e\n\u003ch2\u003e5.3 Automatic classification\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eThe same sample of project descriptions that were manually coded was also classified in relation to the 17 SDGs using nCoder. A detailed overview of the keywords used to train nCoder, Kappa and Rho statistics obtained using nCoder are presented in Appendix 1. In the following we present an overview of the results. In summary, we were able to obtain reliable classifications (reliable kappa and rho statistics) for SDG#2 (Zero hunger), SDG#3 (Good health and well-being) and SDG#6 (Clean water and sanitation). However, nCoder failed to classify project descriptions \u0026nbsp;for the following SDGs: SDG#5 (Achieve gender equality and empower all women and girls), SDG#7 (Ensure access to affordable, reliable, sustainable and modern energy for all), SDG#9 (Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation), SDG#16 (Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels) and SDG#17 (Strengthen the means of implementation and revitalize the global partnership for sustainable development) due to lack of presence of the targeted concepts in the input data. For the other SDGs (e.g., SDG#1, SDG#4, SDG#8, SDG#10, SDG#11, SDG#12, SDG#13, SDG#14, SDG#15 - see details in Table 1) although we attempted multiple rounds of revalidation we were unable to improve kappa and rho statistics. Finally, we also calculated the F1-score based on successful classifications to report the performance of nCoder \u0026nbsp;which reached an F1-score of 0.67 (with Precision = 0.58, Recall = 0.76).\u003c/p\u003e\n\u003cp\u003eFor ESA we used the same dataset of 208 projects considering the manually assigned SDGs we calculated the similarities between all projects and all SDGs. We then tried different thresholds (0.4, 0.3, 0.25, 0.2 and 0.15) and determined the counts of true positives, false positives, false negatives and true negatives for each. Based on this we calculated and compared Precision, Recall and F1 score for each threshold, which appeared to be optimized at 0.2 with an F1 score of 0.411 (with Precision=0.414 and Recall=0.408).\u003c/p\u003e\n\u003cp\u003eNext, we compared the classification results obtained using different techniques. We also obtained SDGs classification of the 208 CS projects by manually checking each project description with OSDG as an external source for comparison. A comparison of the classification results obtained from the three different techniques (nCoder, ESA, OSDG) are presented in Figure 7 and Figure 8. As a baseline we used the results from manual coding (see Figure 3). OSDG classification achieved an F1-Score of 0.30 (with Precision=0.38 and Recall = 0.25)\u003c/p\u003e\n\u003cp\u003eSDGS with more coincidence between methods are: SDG#3: Good health and well-being; SDG#4: Quality education (no coincidence with OSDG in this SDG); SDG#6: Clean water; SDG#12: Responsible consumption; SDG#13: Climate Action (again no coincidence with OSDG in this SDG); SDG#15: Life on Land.\u003c/p\u003e"},{"header":"6\tDiscussion","content":"\u003cp\u003eThe main aim of this research is to explore the advantages and limitations of text-classification techniques to understand their potential to classify data from CS projects with SDGs. Our main research question is\u003cem\u003e: How can a data analytics approach based on web-based data mining and automatic classifiers contribute to the reporting of SDGs related to CS activities and projects?\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eIn this section, first we show how the application of automatic classifiers has allowed us to find interesting findings regarding the mapping of SDGs and CS. Second, we compare the techniques covered in this study by considering their advantages and limitations when applying each technique to classify CS project descriptions with SDGs.\u003c/p\u003e\n\u003cp\u003eDespite that our dataset was randomly built we observe coincidences with previous results in the literature regarding which SDGs are more representative in CS. For instance, in Shulla’s et al. [25] it is stated that CS activities contributed mainly with SDG#4 (Quality Education), SDG#11 (Sustainable Cities and Communities), SDG#13 (Climate Action) and SDG#15 (Life on Land). Also, in the study done by Roldan-Álvarez et al. [15] SDG#13 was the one associated with 25% of the projects analyzed. These results match with the results obtained in our study, but there are interesting aspects that can be observed. In concrete SDG#4 and SDG#13 have very similar results when comparing manual coding, nCoder and ESA.\u003c/p\u003e\n\u003cp\u003eIt is interesting to observe the cases of SDG#10, SDG#11 and SDG#15 mainly we observed different results between methods. The case of SDG#10 (Reduced inequalities) is a curious case to be further investigated in the future. Similarly, to the case of SDG#4, SDG#10 seems to be a transversal SDG that can be associated with multiple disciplines. Around 42% of the projects in our dataset have been associated with this SDG with manual classification and nCoder, but the result is completely different with OSDG and ESA. As indicated before SDG#4 has been mentioned in previous literature, but we have not found evidence showing the connection between SDG#10 and CS in previous studies.\u003c/p\u003e\n\u003cp\u003eIn the case of SDG#11, a significant difference was observed between the results obtained with manual coding, nCoder and OSDG (similar) vs ESA (lower positive results). A similar case happens with SDG#15 but in this case the difference is less remarkable. The reasons behind this difference are related to the set of keywords or wikipedia articles used as a source of knowledge by each technique. In the case of SDG#11 and SDG#15 the results between nCoder and manual coding are similar indicating the keywords selected for training are adequate, in this case the problem would be associated with the content of the wikipedia article for these SDGs. A similar case is the one related to SDG#1 as observed in Figure 7, nCoder and ESA seem to incorrectly classify projects with SDG#1. In this case, the main reason behind this could be the fact that the SDG#1 itself covers a broad range of concepts, in comparison to the other SDGs which are more specific. Therefore, detecting all possible keywords associated with SDG#1 becomes a difficult task. It has particularly become problematic for nCoder classification as indicated in Figure 7 (deviations observed when compared to results obtained from nCoder vs. manual coding.\u003c/p\u003e\n\u003cp\u003eThe similarity between different SDGs (Figure 5) is also observed in our results. Most similar SDGS are: SDG#5 with SDG#8; SDG#6 and SDG#14; SDG#4 and SDG#10; SDG#3 and SDG#10. There is a lack of studies comparing similarity between different SDGs, for this reason we think this is an interesting line to be further explored as future work especially when we expand our analysis to the whole database of projects contained in the CS Track DB.\u003c/p\u003e\n\u003cp\u003eThe main aim of this study was to compare different automatic classifiers to map SDGs with CS, our experience with nCoder confirms the advantages identified by [21]. \u0026nbsp;For large datasets manual coding for different codes can become difficult, as multiple inter-rater reliabilities need to be assessed to achieve acceptable reliability. nCoder automates the process of coding allowing researchers to discover concepts in data flexibly. In fact, in our study this process of having researcher/s iterating and adapting the set of keywords (Figure 1) made the process closer to the results obtained with manual coding. For this reason, we obtained better precision and recall results with nCoder when compared to the same results obtained with reference to with ESA technique. The other main advantage of nCoder is that explainability is quite high when compared to black box models (i.e., BERT). However, it is necessary to mention also the limitations experienced when applying nCoder in this study. One of the most relevant limitations is that overall, the process is time consuming. \u0026nbsp;For instance, in this study nCoder didn't perform enough well for some specific SDGs (e.g., SDG#11 or SDG#15) classification. In these cases, the main limitations were due to the set of keywords used, or the limitations in the input dataset that did not consist of enough samples to carry out a proper training. \u0026nbsp;Introducing additional keywords or enhancing the input dataset with more samples could have improved the results, however, the time constraints and lack of resources required to carry out re-validation eliminated us from conducting further experimentation. Moreover, at present it’s not possible to save a trained classifier that can be later re-used to classify new datasets.\u003c/p\u003e\n\u003cp\u003eIn the case of ESA one advantage is that previous manual coding is not necessary, it is fully based on automatic assignment/coding, therefore it requires less resources (i.e., time for manual coding) compared to nCoder. The same mechanism can be used for different classifications (e.g., research areas) after a corresponding comparison base is established. Another advantage is that thresholds for tf-idf and assignment can be modified and adapted to match the use case. This means that it can account for cases where one needs exactly or at least one assignment as well as for instances where the similarity needs to reach a certain threshold before assignment can be considered with little changes. However, optimization of thresholds takes time. A textual description is needed for each classifier, in this case Wikipedia articles of each individual SDG were used. But as observed in this study, the issue can always be with extensive or broad descriptions that sort of dilute the results. For instance, the projects ESA didn't catch might cover more than SDG#11 (i.e., other SDGs or research areas) which pulls down the textual similarity to the SDG#11 Wikipedia article. Finally with this technique the language used is also an issue, ESA is dependent on language of the base matrix (in this case english) so that would have to be set up for different languages if needed otherwise texts need to be translated.\u003c/p\u003e\n\u003cp\u003eIn this study we compared the results obtained from nCoder and ESA with an existing text-classification platform ‘OSDG’. When comparing the F1-Scores obtained with each technique, the ones from OSDG are lower than the ones obtained with nCoder or ESA. OSDG classification achieved an F1-Score of 0.30 (with Precision=0.38 and Recall = 0.25). ESA achieved a F1 score of 0.411 (with Precision=0.414 and Recall=0.408). And nCoder reached an F1-score of 0.67 (with Precision = 0.58, Recall = 0.76). However, it is necessary to indicate that in the case of nCoder it was not possible to consider the total of 17 SDGs to calculate the F1-score because SDG#5, SDG#7, SDG#9, SDG#16 and SDG#17. Despite OSDG achieving lower precision and recall results, one advantage of using this technique is the ease of use of its platform and results are generated instantly.\u003c/p\u003e\n\u003cp\u003eAlthough BERT was not directly applied to the dataset used in this study, we would like to discuss the main advantages and limitations of this classifier based on the experience obtained in a previous study [15]. One of the main points of interest when using deep learning models, BERT in particular, is to create a balanced dataset in which each category is equally represented. In the case of the Twitter study regarding SDGs this was a limitation because the discussion around some of them is greater than in others then this produced an unbalanced dataset. For instance, according to our previous study, we can easily find tweets about SDG#13, but it is not that easy to gather tweets about SDG#9. When creating the training datasets this is an issue that needs to be tackled. Otherwise, predictions will not be accurate, and the number of wrong classifications will greatly increase. Therefore, the main limitation of BERT is the time required to prepare the initial sample for the training dataset, especially when there is not an original dataset containing positive and negative examples of CS projects classified for each SDG.\u003c/p\u003e"},{"header":"7\tFuture work and Conclusion","content":"\u003cp\u003eThis study contributes to the lack of research regarding the need of using data from Citizen Science projects to enhance the knowledge regarding the UN Sustainable Development Goals. The study shows how a process based on data analytics approach (combining web-based data mining and automatic classifiers) can contribute to the reporting of SDGs (RQ).\u003c/p\u003e\n\u003cp\u003eFirst this study has shown that by using web-mining techniques a DB can be built as a central point of knowledge to avoid the problem of CS data dispersed on the web. Although this was not the main scope of this paper, our experience building the DB and the classification of the different CS project descriptions allows us to identify some future work. Now one major barrier is the poor quality of data collected. e.g., the different web page structures and different uses of metadata standards. In this line it is important to promote dialogue on data quality, data management including standards, metadata and interoperability are key actions. Therefore, in this context it is important to integrate initiatives such as the one developed by the Citizen Science Association Data \u0026amp; MetaData Working Group [18] by CS platform developers. But an alignment between the PPSR metadata standard with the SDG indicators is still work to be done. In this line, we suggest that it is key to define how to standardize the reporting of specific contributions/results from CS with the set of indicators proposed by the quantitative SDG framework.\u003c/p\u003e\n\u003cp\u003eThe main aim of this study was focused on understanding the advantages and limitations of text-classification techniques to enhance the understanding of the relationship between CS and SDGS. \u0026nbsp;As described in the discussion section the three different techniques we used i.e., nCoder, ESA, OSDG, have several benefits and disadvantages. In summary, although the results obtained using nCoder are more aligned with the results of the manual classification the process can be overall time consuming and later using a trained classifier is not possible. In comparison the effort required from human coders is minimum for ESA, however the quality and the levels of details present in the reference article, i.e., Wikipedia, could impact the results. To this end, the paper contributes by sharing our experience in using different text classification techniques to classify CS project descriptions. On the other hand, deep learning models such as i.e., BERT is becoming the state-of-the-art model solution for multiple natural language processing tasks. Although obtaining satisfactory amounts of training data to train machine learning models is a challenge, specially in the case of SDGs where we need to classify data for 17 different categories, the use of advanced techniques such as BERT in future can provide more accurate results also considering multiple languages in the future. However, drawbacks inherited in black box models such as lack of explainability and interpretability should be taken into consideration.\u003c/p\u003e\n\u003cp\u003eIn addition to the advantages and limitations of the techniques explored in this paper, we have observed some interesting findings regarding the relationship between SDGs and CS. As indicated in the Introduction, several previous studies are focused on understanding the relationship of CS with one SDG. But in this study, we have observed that in most cases projects are associated with multiple SDGs. One possible explanation to be further explored is to analyze if the project has some SDGs as part of their own goals to be solved, but also other SDGs appear as part of the methodology or outcomes produced by the project. For instance, this would explain why SDG4 ‘Quality Education’ is one the most popular ones although most of the projects are not focused on scientific research of educational contexts. In this line, another future research line can be focused on identification of SDGs connected to other ones.\u003c/p\u003e\n\u003cp\u003eNext steps in our research include to extend our dataset, and analyze the maximum number of projects included in our DB. This will allow us to identify which SDGs are the most representative ones in CS, relations between different SDGs, and how SDGs are related to different research areas associated with each project.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eAcknowledgment\u003c/p\u003e\n\u003cp\u003eThis work has been partially funded by the EU project \u0026lsquo;CS Track\u0026rsquo; under the H2020 program (grant id: 87252) and the Ram\u0026oacute;n y Cajal programme of the Spanish Ministry of Science and Innovation (P. Santos).\u003c/p\u003e\n\u003cp\u003eAuthor Contribution\u003c/p\u003e\n\u003cp\u003ePatricia Santos has contributed to the leading of the manuscript, conceptual idea of the research line, to the analysis and writing of the paper.Ishari Amarashinghe has contributed to the conceptual idea of the research line, to the analysis and writing of the paper.Miriam Calvera, Cleo Schulten, Ulrich Hoope, David Rold\u0026aacute;n-\u0026Aacute;lvarez, Fernando Mart\u0026iacute;nez-Mart\u0026iacute;nez to the analysis of the datasets.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eL. Fonseca, and F. Carvalho, \u0026ldquo;The reporting of SDGs by quality, environmental, and occupational health and safety-certified organizations\u0026rdquo; \u003cem\u003eSustainability\u003c/em\u003e, 11(20), 5797, 2019.\u003c/li\u003e\n\u003cli\u003eJ. Wu, S. Guo, H. Huang, W. Liu, and Y. Xiang, \u0026ldquo;Information and communications technologies for sustainable development goals: state-of-the-art, needs and perspectives. \u003cem\u003eIEEE Communications Surveys \u0026amp; Tutorials\u003c/em\u003e, 20(3), 2389-2406, 2018.\u003c/li\u003e\n\u003cli\u003eA. L\u0026oacute;pez-Vargas, M. Fuentes, and M. Vivar, \u0026ldquo;Challenges and opportunities of the internet of things for global development to achieve the United Nations sustainable development goals\u0026rdquo;. \u003cem\u003eIEEE Access\u003c/em\u003e, 8, 37202-37213, 2020.\u003c/li\u003e\n\u003cli\u003eS. Fritz, L. See, T. Carlson, M.M. Haklay, J.L. Oliver, D. Fraisl, D., R. Mondardini, M. Brocklehurst, L.A. Shanley, S. Schade, U. When, T. Abrate, J. Anstee, S. Arnorld, M. Billot, J. Campbell, J. Espey, M. Gold, G. Hager, S. He, L. Hepburn, A. Hsu, D. Long, J. Mas\u0026oacute;, I. McCallum, M. Muniafu, I. Moorthy, M. Obersteiner, A.J. Parker, M. Weisspflug and S. West \u0026ldquo;Citizen science and the United Nations sustainable development goals\u0026rdquo;. \u003cem\u003eNature Sustainability\u003c/em\u003e, 2(10), 922-930, 2019.\u003c/li\u003e\n\u003cli\u003eD. Fraisl, J. Campbell, L. See, U. Wehn, J. Wardlaw, M. Gold, I. Moorthy, R. Arias, J. Piera, J.L. Oliver, J. Mas\u0026oacute;, M. Penker and S. Fritz, \u0026ldquo;Mapping citizen science contributions to the UN sustainable development goals\u0026rdquo;. \u003cem\u003eSustainability Science\u003c/em\u003e, 15(6), 1735-1751, 2020.\u003c/li\u003e\n\u003cli\u003eM. V. Eitzel, J. L. Cappadonna, C. Santos-Lang, R.E. Duerr., A. Virapongse, S.E. West, C.C.M. Kyba, A. Bowser, C. B. Cooper, A. Sforzi, A. Nova-Metcalfe, E. S Harris, M. Thiel, M. Haklay, L. Ponciano, J. Roche, L. Ceccaroni, F. M. Shilling, D. D\u0026ouml;rler, F. Heigl, T. Kiessling, B. Y. Davis and Q. Jiang, \u0026ldquo;Citizen science terminology matters: Exploring key terms\u0026rdquo;. \u003cem\u003eCitizen science: Theory and practice\u003c/em\u003e, 2(1), 2017.\u003c/li\u003e\n\u003cli\u003eF. Heigl, B. Kieslinger, K.T. Paul, J. Uhlik and D. D\u0026ouml;rler, \u0026ldquo;Opinion: Toward an international definition of citizen science\u0026rdquo;. \u003cem\u003eProceedings of the National Academy of Sciences\u003c/em\u003e, 116(17), 8089-8092, 2019.\u003c/li\u003e\n\u003cli\u003eS. Schade, M. Pelacho, T.C. van Noordwijk, K. Vohland, S. Hecker and M. Manzoni, \u0026ldquo;Citizen science and policy. In The science of citizen science\u0026rdquo; Springer, Cham, pp. 351-371, 2021.\u003c/li\u003e\n\u003cli\u003eK. Vohland, C. G\u0026ouml;bel, B. Bal\u0026aacute;zs, E. Butkevičienė, M Daskolia, B. Duž\u0026iacute;, S. Hecker, M. Manzoni and S. Schade (2021). \u0026ldquo;Citizen Science in Europe\u0026rdquo;. The Science of Citizen Science, 35, 2021\u003c/li\u003e\n\u003cli\u003eH.Y. Liu, D. D\u0026ouml;rler, F. Heigl, and S. Grossberndt, \u0026ldquo;Citizen Science Platforms\u0026rdquo;, The Science of Citizen Science, 439, 2021.\u003c/li\u003e\n\u003cli\u003eL. Quinlivan, D.V. Chapmanand T. Sullivan, \u0026ldquo;Validating citizen science monitoring of ambient water quality for the United Nations sustainable development goals\u0026rdquo;. \u003cem\u003eScience of the Total Environment\u003c/em\u003e, 699, 134255, 2021.\u003c/li\u003e\n\u003cli\u003eS. Koffler, C. Barbi\u0026eacute;ri, N.P. Ghilardi-Lopes, J.N. Leocadio, B. Albertini, T.M. Francoy and A.M. Saraiva, \u0026ldquo;A buzz for sustainability and conservation: The growing potential of citizen science studies on bees\u0026rdquo;. \u003cem\u003eSustainability\u003c/em\u003e, 13(2), 959, 2021.\u003c/li\u003e\n\u003cli\u003eEuropean Commission, Joint Research Centre (JRC), \u0026ldquo;An inventory of citizen science activities for environmental policies\u0026rdquo;. European Commission, Joint Research Centre (JRC) [Dataset] 2018 PID: http://data.europa.eu/89h/jrc-citsci-10004\u003c/li\u003e\n\u003cli\u003eBio Innovation Service. (2018). Citizen science for environmental policy: development of an EU-wide inventory and analysis of selected practices. Final report for the European Commission, DG Environment under the contract 070203/2017/768879/ETU/ENV. A. 3, in collaboration with Fundacion Ibercivis and The Natural History Museum.\u003c/li\u003e\n\u003cli\u003eD. Rold\u0026aacute;n-\u0026Aacute;lvarez, F. Mart\u0026iacute;nez-Mart\u0026iacute;nez, E. Mart\u0026iacute;n, and P.A. Haya, \u0026ldquo;Understanding Discussions of Citizen Science Around Sustainable Development Goals in Twitter,\u0026rdquo; \u003cem\u003eIEEE Access\u003c/em\u003e, vol. 9, pp. 144106\u0026ndash;144120, 2021, doi.org/10.1109/ACCESS.2021.3122086\u003c/li\u003e\n\u003cli\u003eM. Riegner, \u0026ldquo;Implementing the Data Revolution for the Post-2015 Sustainable Development Goals: Toward a Global Administrative Lawof Information,\u0026rdquo; World Bank Legal Rev. 7, 2016.\u003c/li\u003e\n\u003cli\u003eU. Sturm, S. Schade, L. Ceccaroni, M. Gold, C. Kyba, B. Claramunt, M. Haklay, D. Kasperowski, A. Albert, J. Piera, J. Brier, C. Kullenberg, S. Luna, \u0026ldquo;Defining principles for mobile apps and platforms development in citizen science,\u0026rdquo; \u003cem\u003eResearch Ideas and Outcomes, \u003c/em\u003evol\u003cem\u003e. \u003c/em\u003e3, 2017, doi.org/10.3897/rio.3.e21283\u003c/li\u003e\n\u003cli\u003eA. Bowser, \u0026ldquo;Standardizing Citizen Science?,\u0026rdquo; \u003cem\u003eProc. Biodiversity Information Science and Standards 1: e21123\u003c/em\u003e, 2017, doi.org/10.3897/tdwgproceedings.1.21123\u003c/li\u003e\n\u003cli\u003eCitizen Science Association Data \u0026amp; Meta Data Working Group, \u0026ldquo;PPSR Core, A Data Standard for Public Participation in Scientific Research (Citizen Science),\u0026rdquo; https://core.citizenscience.org. 2021.\u003c/li\u003e\n\u003cli\u003eAustralia, S. D. S. N., \u0026ldquo;Getting started with the SDGs in Universities: A Guide for Universities. Higher Education Institutions, and the Academic Sector,\u0026rdquo; https://ap-unsdsn.org/wp-content/uploads/University-SDG-Guide_web.pdf. 2017.\u003c/li\u003e\n\u003cli\u003eZ. Cai, A. Siebert-Evenstone, B. Eagan, D.W. Shaffer, X. Hu, and A.C Graesser, \u0026ldquo;nCoder+: a semantic tool for improving recall of nCoder coding,\u0026rdquo; \u003cem\u003eProc. Int. Conf. on Quantitative Ethnography, \u003c/em\u003epp. 41\u0026ndash;54, 2019.\u003c/li\u003e\n\u003cli\u003eE. Gabrilovich and S. Markovitch, \u0026ldquo;Computing semantic relatedness using Wikipedia-based explicit semantic analysis,\u0026rdquo; \u003cem\u003eProc. Int. Joint Conf. on Artificial Intelligence,\u003c/em\u003e pp.1606\u0026ndash;1611, 2007.\u003c/li\u003e\n\u003cli\u003eL. Pukelis, N.B. Puig, M. Skrynik, and V. Stanciauskas, \u0026ldquo;OSDG--Open-Source Approach to Classify Text Data by UN Sustainable Development Goals (SDGs),\u0026rdquo; \u003cem\u003eCoRR\u003c/em\u003e, vol. abs/2005.14569, arXiv preprint arXiv:2005.14569, 2020.\u003c/li\u003e\n\u003cli\u003eJ. Devlin, M. Chang, K. Lee, and K. Toutanova, \u0026ldquo;Bert: Pre-training of deep bidirectional transformers for language understanding,\u0026rdquo; \u003cem\u003eCoRR\u003c/em\u003e, vol. abs/1810.04805, arXiv preprint arXiv:1810.04805, 2018.\u003c/li\u003e\n\u003cli\u003eK. Shulla, W.L. Filho, J.H. Sommer, A.L. Salvia, and C. Borgemeister, \u0026ldquo;Channels of collaboration for citizen science and the sustainable development goals,\u0026rdquo; \u003cem\u003eJournal of Cleaner Production, \u003c/em\u003evol. 264, 2020.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-data-science-and-analytics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jdsa","sideBox":"Learn more about [International Journal of Data Science and Analytics](http://link.springer.com/journal/41060)","snPcode":"41060","submissionUrl":"https://submission.nature.com/new-submission/41060/3","title":"International Journal of Data Science and Analytics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Information Technology and Systems, Data mining, Web mining, Text analysis, Machine learning","lastPublishedDoi":"10.21203/rs.3.rs-4781489/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4781489/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eTraditional data sources provide insufficient knowledge for measuring the United Nations Sustainable Development Goals (SDGs). Data related to SDGs are sourced primarily from global databases maintained by international organizations, national statistical offices and other government agencies. Recent studies show the value of using data from Citizen Science (CS) for assessing the SDGs. There is an important online presence of CS programs, professional networks for CS and online communities of citizen scientists, leading to the generation of several CS platforms. In this context, the role of computational data science is key. This paper explores and exemplifies opportunities for combining web-data mining techniques and automatic classifiers to enhance the understanding of the inter-relation between CS and the SDGs. An analysis of different automatic classifiers is presented by comparing the results obtained from their application in a sample of 208 CS project descriptions. The results of this study indicate the benefits and limitations of these techniques (nCoder, ESA, OSDG and BERT), but also provides a discussion of the potential benefits of using data from CS projects to map the 17 SDGs.\u003c/p\u003e","manuscriptTitle":"Mapping Sustainable Development Goals to Citizen Science projects - a comparative evaluation of automatic classifiers","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-19 11:10:00","doi":"10.21203/rs.3.rs-4781489/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-10-07T08:26:37+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-09-25T11:54:10+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-08-21T15:07:28+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"332272856629711270991138714534134446459","date":"2024-08-17T09:51:20+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"178266247200872013846022831681650735254","date":"2024-08-13T11:03:17+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"278144576981238268177796495261760543315","date":"2024-08-13T04:52:34+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"231012595275044231874416003674461131780","date":"2024-08-12T13:05:36+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-08-12T12:54:00+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-08-06T20:21:11+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-07-23T00:51:16+00:00","index":"","fulltext":""},{"type":"submitted","content":"International Journal of Data Science and Analytics","date":"2024-07-22T11:28:24+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-data-science-and-analytics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jdsa","sideBox":"Learn more about [International Journal of Data Science and Analytics](http://link.springer.com/journal/41060)","snPcode":"41060","submissionUrl":"https://submission.nature.com/new-submission/41060/3","title":"International Journal of Data Science and Analytics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"cbb88ff8-d1d4-488e-84a8-f9df73f0c7aa","owner":[],"postedDate":"August 19th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2024-12-16T15:59:01+00:00","versionOfRecord":{"articleIdentity":"rs-4781489","link":"https://doi.org/10.1007/s41060-024-00695-7","journal":{"identity":"international-journal-of-data-science-and-analytics","isVorOnly":false,"title":"International Journal of Data Science and Analytics"},"publishedOn":"2024-12-15 15:57:04","publishedOnDateReadable":"December 15th, 2024"},"versionCreatedAt":"2024-08-19 11:10:00","video":"","vorDoi":"10.1007/s41060-024-00695-7","vorDoiUrl":"https://doi.org/10.1007/s41060-024-00695-7","workflowStages":[]},"version":"v1","identity":"rs-4781489","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4781489","identity":"rs-4781489","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}