A semi-automated approach to policy-relevant evidence synthesis: Combining natural language processing, causal mapping, and graph analytics for public policy

A semi-automated approach to policy-relevant evidence synthesis: Combining natural language processing, causal mapping, and graph analytics for public policy | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A semi-automated approach to policy-relevant evidence synthesis: Combining natural language processing, causal mapping, and graph analytics for public policy Rory Hooper, Nihit Goyal, Kornelis Blok, Lisa Scholten This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3285731/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Although causal evidence synthesis is critical for the policy sciences – whether it be analysis for policy or analysis of policy – its repeatable, systematic, and transparent execution remains challenging due to the growing volume, variety, and velocity of policy-relevant evidence generation as well as the complex web of relationships within which policies are usually situated. To address these shortcomings, we developed a novel, semi-automated approach to synthesizing causal evidence from policy-relevant documents. Specifically, we propose the use of natural language processing (NLP) for the extraction of causal evidence and subsequent homogenization or normalization of the varied text, causal mapping for the collation, visualization, and summarization of complex interdependencies within the policy system, and graph analytics for further investigation of the structure and dynamics of the causal map. We illustrate this approach by applying it to a collection of 28 articles on the emissions trading scheme (ETS), a policy instrument of increasing importance for climate change mitigation. In all, we find 300 variables and 284 cause-effect pairs in our input dataset (consisting of 4524 sentences), which are reduced to 70 unique variables and 119 cause-effect pairs after normalization. We create a causal map depicting these and analyze it subsequently to obtain systemic perspective as well as policy-relevant insight on the ETS that is broadly consistent with select manually conducted, previous meta-reviews of the policy instrument. We conclude that, despite its present limitations, this approach can help synthesize causal evidence for policy analysis, policymaking, and policy research. Evidence-informed policy Emissions trading schemes (ETS) Causal mapping Machine learning (ML) Natural language processing (NLP) Policy analysis Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 1 Introduction Transparent, timely, repeatable, and contextual synthesis of causal evidence is important for analyzing and informing every stage of the policy cycle. Policy research relies on effective summarization, synthesis, and mobilization of knowledge for or in the policy process. Studies that aim to facilitate evidence-based or evidence-informed policy-making, policy evaluation, and policy learning require at least some degree of evidence synthesis (Sanderson, 2002 ). Relatedly, studies conducting ex-ante policy assessment benefit from a careful aggregation of existing research to map a complex system (Freebairn et al., 2016 ). Even reviews of policy research – such as the Policy Studies Yearbook or Theories of the Policy Process – involve critical appraisal of evidence (Norman, 2023 ; Weible & Sabatier, 2018 ). Methods for evidence synthesis have, therefore, received much attention from policymakers, analysts, and researchers alike. However, several characteristics of the policy sciences make evidence synthesis challenging. First, the exponential growth in policy-relevant research increases resource and time requirements even as time-critical policy advice is more in demand (Bornmann & Mutz, 2015 ; Larsen & Von Ins, 2010 ; Nunez-Mir et al., 2016 ). Second, evidence is scattered across different policy areas, in academic and grey literature, wherein different terminologies are used to refer to similar phenomena (Goyal & Howlett, 2018 ; Saetren, 2005 ). Third, policies have (un)intended consequences on a variety of dimensions – such as programmatic, process, and political (Marsh & McConnell, 2010 ; Wakabayashi & Kimura, 2018 ) – all of which should be considered in evidence synthesis. Fourth, the process is complicated further due to different levels of policy analysis, spanning from cross-sectional, micro-level studies of individual policies to time-series, macro-level research on entire policy areas (Esty & Porter, 2005 ; Jiali, 1995 ; Thow et al., 2010 ; Warner & van Buuren, 2011 ). The field of evidence synthesis has witnessed rapid development and now consists of several methods that facilitate systematic source identification, repeatable evidence appraisal, and transparent reporting. On the one hand, methods such as thematic synthesis (Thomas & Harden, 2008 ), framework synthesis (Carroll et al., 2011 ), scoping reviews (Peters et al., 2020 ), meta-narrative reviews (Wong et al., 2013a ), and meta-ethnography (France et al., 2019 ) enable broad syntheses and critical appraisals of research, but do not seek to inform policy-making per se. On the other hand, methods such as systematic reviews (Pearson et al., 2015 ; Petticrew & Roberts, 2008 ), meta-analyses (Barza et al., 2009 ), and umbrella reviews (Fusar-Poli & Radua, 2018 ) help synthesize evidence on interventions in a critical, repeatable, and transparent manner. They can, however, be difficult to execute, limited in their ability to acknowledge diversity in evidence, simplistic in their treatment of complexity, and time-consuming (Haddaway et al., 2017 ). Although living systematic reviews (Millard et al., 2019 ) and rapid reviews permit execution in a time-sensitive situation (O’Leary et al., 2017 ), they do not address the other shortcomings. A more configurative approach is necessary to capture the underlying system complexity (Anderson et al., 2013 ). Realist syntheses, which focus on mechanisms of how interventions work (Macura et al., 2019 ; Wong et al., 2013b ), address complexity better (Greenhalgh et al., 2011 ). These can shed light on the underlying reasons behind success and facilitate theory-building or theory-testing. As they can also require significant resources, rapid realist reviews have been proposed (Saul et al., 2013 ). Meanwhile, meta-aggregation collates ‘lines of action’ in available evidence that can inform decision-making in an auditable, reliable, and transparent manner (Hannes & Lockwood, 2011 ). However, these methods, too, provide limited support in addressing diverse conceptual lenses of analyses in policy-relevant evidence. This study presents a semi-automated approach to extract, aggregate, map, and analyze causal evidence from policy-relevant literature. To address the challenges outlined above, we combine natural language processing (NLP), causal mapping, and graph analytics. NLP refers to computational techniques for analyzing and representing natural language text (Liddy, 2001 ), and is increasingly used in the policy sciences (El-Taliawi et al., 2021 ; Goyal & Howlett, 2019 , 2021 ). Meanwhile, causal mapping facilitates the organization and representation of causal evidence regarding interdependencies among parts of a system (Barbrook-Johnson & Penn, 2022 ), but has witnessed limited use in policy analysis. Finally, graph analytics – a field that is still underutilized for policy research – can help with systematic investigation of a causal map (Nguyen et al., 2013), a type of a graph. This article is structured as follows. In Section 2, we present an approach for semi-automated evidence synthesis. Subsequently, we illustrate the approach by synthesizing evidence on the emissions trading scheme (ETS) to create a causal map (Section 3). In Section 4, we describe and analyze the causal map to demonstrate its suitability for generating policy-relevant insight. Thereafter, we discuss the implications of this research (Section 5) and conclude the article (Section 6). 2 The potential of semi-automated causal evidence synthesis Causal evidence synthesis is a multi-step process. The first step is the collation of documents that contain evidence pertaining to a research area, topic, question, or hypothesis. Existing machine learning techniques such as classification and clustering can help in this step (van de Schoot et al., 2021 ). We, however, focus on the remainder of the process. Typically, the second step is the extraction and appraisal of evidence. While some protocols appraise causality based on research design – e.g., including only randomized control trials or meta-analyses of these (Burns et al., 2011 ) – such an approach excludes relevant evidence created through various qualitative, quantitative, and mixed-methods designs. An alternate approach is to extract causal claims from the relevant documents. Such claims might be present in the form of explicit causality (e.g. using causative adjectives, adverbs, or verbs), implicit causality (based on background knowledge and the line of reasoning), inter-sentence causality (cause and the effect are spread across multiple sentences), and embedded causality (a variable is included as a composite effect of one cause and the cause of another effect) (Yang et al., 2021 ). Due to the complexity of natural language and the growing volume of policy-relevant evidence, this is challenging even for a subject matter expert. While NLP is unlikely to substitute human intelligence, advances in the field can assist in reducing and redirecting human effort. The approaches to causality detection can be broadly classified into (top-down) co-occurrence-based methods and (bottom-up) causal relation extraction methods. Co-occurrence-based methods reduce a large volume of text into core concepts and then identify connections among these concepts (Han et al., 2019 ; Kim et al., 2016 ; Son et al., 2020 ). Causal relation extraction methods identify the connections within the text and then aggregate them (Asghar, 2016 ; Bach & Badaskar, 2007 ; Khoo & Na, 2006 ). The latter is generally more appropriate for evidence synthesis as its bottom-up nature covers even infrequently occurring variables and retains the stated relationships among variables (Barik et al., 2016 ). Causal relation extraction techniques can, in turn, be classified into knowledge-based, statistical machine learning, and deep learning techniques (Yang et al., 2021 ). Knowledge-based techniques rely on semantic and syntactic text characteristics, codified as patterns or rules, against which an algorithm can classify input text (e.g. a causative verb-noun pair) (Bui et al., 2010 ). These techniques perform well on simple text with explicit causality, but poorly on text with implicit causality (Beamer et al., 2008 ; Girju et al., 2009 ; Yang et al., 2021 ). Statistical machine learning identifies distinguishing characteristics based on labelled training data to classifying data, using techniques such as Bayesian inference or decision trees. Statistical machine learning can handle implicit causality well, but it suffers from low portability, i.e., poor performance when the test data is dissimilar from the training data (Asghar, 2016 ; Pakray & Gelbukh, 2014 ; Zhao et al., 2016 ). Finally, deep learning techniques utilize neural networks architectures for causal relationship extraction, which perform well even on complex text (Yang et al., 2021 ). In addition, they can leverage ‘transfer learning’ to enhance portability (Beltagy et al., 2019 ; Devlin et al., 2018 ; Kyriakakis et al., 2019 ). The next step is to make the evidence comparable through some form of homogenization or normalization. In case of a meta-analysis, a regression coefficient might be converted into a partial correlation coefficient to make the input comparable (Hansen et al., 2022 ). Whereas less relevant for qualitative evidence, a challenge for policy-relevant analysis is that the source material can span several (inter)disciplinary fields, with different vocabulary to refer to the same or similar phenomena (Goyal & Howlett, 2018 ; Saetren, 2005 ). For example, the use of ‘institution’ in governance studies might be synonymous with the use of ‘policy instrument’ in policy studies. While not a substitute for expert assessment, NLP techniques for measuring lexical or semantic similarity can be useful. Lexical similarity is measured using a dictionary-based approach (Cruanes et al., 2012 ), which are easier to implement, but do not consider the context in which a term is used. In contrast, semantic similarity is measured such that the distance between terms (or other units, such as sentences, paragraphs, documents) represents the likeness of their meaning (Liu et al., 2007 ). For complex text, semantic similarity, or a combination of lexical and semantic similarity, is likely to be a more robust approach (Inan, 2020 ). Thereafter, the evidence needs to be aggregated. For a quantitative evidence synthesis, a weighted mean effect size for the cause-effect relationship may be computed (Hansen et al., 2022 ). However, as policies often have consequences across many dimensions – program, process, and politics (McConnell, 2010 ) – that vary over time (Compton & ’t Hart, 2019 ; Goyal, 2021 ), it is essential to consider multiple effects for any cause. Further, relevant evidence might be available from diverse contexts varying in levels, geographies, scales, and time periods, making it important to highlight the relevant effects of contextual variables and internal mechanisms on the causal chain. In addition, the research designs of the sources could range from argumentative work to small-n case studies; medium or large-n observational, quasi-experimental, and experimental research; simulation modelling; and co-design or participatory knowledge. All of this lends itself to a more qualitative synthesis, for which in-depth information is usually summarized as tables and conceptual diagrams or synthesized in text. Collating and visually presenting evidence as in systems science can add value here. System mapping facilitates collation of existing evidence to answer a specific question, identify knowledge clusters, or articulate a knowledge gap (James et al., 2016 ). Similarly, systematic evidence mapping helps summarize, query, and visualize evidence, for example, using a database and/or a graphical representation (Peters et al., 2021 ; Wolffe et al., 2019 ). Causal mapping is likely to be especially useful for causal evidence synthesis. It combines causal chain analysis with the mapping of complex interrelationships, thereby facilitating collation of existing evidence or expert knowledge about (dynamic) causal interdependencies between parts of a system (Ackermann & Alexander, 2016 ). This allows to answer specific questions about effects of interest, to identify knowledge clusters, or articulate a knowledge gap (James et al., 2016 ). Causal mapping is well-suited for combining analysis of different scopes and allows for easy expansion when new information is introduced (Eden et al., 1992 ). As a causal map is a type of a directed knowledge graph, it can be investigated using graph analytics, broadly used here to refer to techniques in causal chain analysis, graph theory, and network analysis. Illustratively, topographic analysis creates insight based on the structural layout of the causal map. An assessment of the strength of the linkages, for example, indicates the number of studies providing evidence on specific cause-effect relationships (Montibeller & Belton, 2006 ). This can shed light not only on interventions that are supported by much evidence, but also on parts of the system that may warrant further investigation. Similarly, effect trees – a collection of downstream variables affected by a given factor, and cause trees – a collection of upstream variables that influence a factor – can help analyze the cause and consequences, respectively, of an intervention (Eden et al., 1992 ). Centrality analysis can provide insight into the likely importance of a variable – either perceived or real – based on whether, for example, it has several linkages to other variables (i.e., high degree centrality) or lies on the shortest path between several other variables (i.e., high betweenness centrality). Some of the other possibilities for investigating a causal map are causal inference analysis and causal loop analysis. Causal inference refers to the process of determining the causal impact of one factor on another (Axelrod, 1976 ), either the partial effect (the causal impact of one variable on another along a specific path), or the total effect (the overall impact of one variable on another taking all paths into consideration). Causal loop analysis involves the identification and description of feedback mechanisms within a causal map i.e. a positive or negative feedback within (a part of) the system, resulting in reinforcing or balancing behavior, respectively. Apart from aiding causal evidence synthesis, causal maps can also inform subsequent policy analysis, for example, in the form of Bayesian belief analysis or simulation modelling (Pullin et al., 2016 ). An overview of the semi-automated approach is shown in Fig. 1 . Beginning with a selection of relevant text, causal relations are extracted, similar factors are grouped, and an aggregate causal map is built. The map is then analyzed to derive policy-relevant insights. It is important to note that a variety of different algorithms can be used to achieve the desired outputs of each step. In this way algorithm selection can be made to best suit the given application, and the results of the method can keep pace with advancements in NLP. 3 Synthesizing policy-relevant evidence: An illustration using the Emissions Trading Scheme We illustrate the process of synthesizing policy-relevant evidence and creating a causal map through the semi-automated approach, using the case of the ETS. 3.1 Emissions trading schemes ETS, are policy instruments that aim to impose a cost on the emission of greenhouse gasses (GHGs). This policy involves setting a ‘cap’ on certain types of emissions, regulating bodies then divide the total allowable emission quantity into tradeable ‘allowances’ that are auctioned or allocated to the entities covered by the system. Polluters can buy and sell allowances but must surrender a quantity equivalent to their respective emissions at the end of pre-defined periods. Entities for whom abatement is relatively cheap have a financial incentive to reduce emissions, and entities for whom abatement is relatively expensive have the option to purchase allowances to satisfy their obligation. In this way, ETS provide a mechanism to reduce emissions, to specified levels, whilst encouraging the most cost-efficient abatement (ICAP, 2022b ). Evidence-synthesis from ETS policy analysis literature is encumbered by many of the limitations discussed earlier. First, there is an enormous literature base. Google Scholar returns over 23,100 results for a search of “ETS” and “policy analysis”. The terminology employed for discussing ETS performance varies across these studies. ETS policy studies often only examine a single performance dimension despite compelling analysis from programmatic, political and process perspectives. And finally, analysis is conducted at various levels of scope from case studies to reviews seeking to summarize the wholesale abatement impact of carbon pricing. As such, this field was selected to demonstrate the method. While several possibilities exist for compiling relevant source material, for this illustration we identified meta-reviews of ETS and selected constituent articles. This allowed us to compare the insights created by our approach against those in the meta-reviews. From the 14 meta-reviews on ETS identified in our search, we selected two: Green ( 2021 ), due to its explicit ex-post and quantitative direction encompassing studies from all major ETS jurisdictions, and Schmalensee & Stavins, ( 2017 ) due to its specific discussion on political considerations and system design factors, which reflect more process and political perspectives. Collating the papers within both reviews and removing those deemed irrelevant yielded a final total of 28 source papers, as shown in Table A1 ( Appendix ). 3.2 Extracting cause-effect relationships The first stage of our approach involves examining each sentence of the source material to determine whether it exhibits causality, and if so, the cause factor, effect factor, and the direction of the causal relation. Take, for example, the causal sentence “the higher emissions allowance price caused a decrease in coal power generation”. Here, ‘emissions allowance price’ is the causal factor, ‘coal power generation’ is the effect factor, the direction of the relationship is negative (i.e., an increase in the former results in a reduction in the latter). To semi-automate this process, we use a deep learning algorithm ‘Self-attentive BiLSTM-CRF wIth Transferred Embeddings’ or SCITE, Li et al. ( 2021 ). The algorithm was chosen given that it is open source, has an explicit focus on causal relations and performs highly on benchmark datasets. To derive causal relations using SCITE, the 28 documents were processed in three stages. First, we selected only the abstract, results, discussion, and conclusion sections of the articles. The literature review and methods were excluded because these often-returned irrelevant methodological links, and relations suggested in other literature instead of findings of that article. Next, the text was automatically segmented into sentences, cleaned (removing non-alphanumeric characters, separation of punctuation etc.) and further segmented into words. Subsequently, ‘layered embeddings’ – necessary for the algorithm to learn a representation of the input data (Levy & Goldberg, 2014 ) – were generated for each sentence (Akbik et al., 2018 ). The sentences and corresponding embeddings were processed through a pre-trained SCITE model resulting in a list of sentences deemed causal, and their suggested cause-effect relationships. An example causal relation returned by SCITE is shown in Fig. 2 . In all, our data consisted of 4542 sentences that were cleaned and provided as input to SCITE. Of these, 317 were deemed as causal by the algorithm. The raw output, however, did not indicate the direction of the cause (i.e., positive or negative) and included a high level of inaccuracy. This necessitated a manual review of each SCITE output to determine the true causal relations, causal pairs, and direction. This stage was also necessary to remove irrelevant causal sentences. Ultimately, 154 sentences were verified as causal and relevant. These sentences contained 284 causal pairs. Our validation of this step revealed that the pre-trained model of SCITE achieved 84% precision and ~ 38% recall for our data. Despite the current manual review stages required, this semi-automatic approach reduced the time taken to extract relations from a single paper from ~ 3 hours to ~ 20 minutes. 3.3 Clustering causal links Next, we concatenated the individual causal pairs, their contributing sentence, cause-effect factors, and relationship direction. To ensure backward traceability, we provided a unique cause-effect id (in the form [article number]-[sentence number][causal pair letter]) to each causal pair, which highlights the prevalence of similar causal pairs. For example, when talking about the use of coal power, one causal pair may include the factor ‘coal power generation’ whereas another ‘coal utilization’. Including both factors would quickly cloud the causal map with repetitive links. To address this issue, semantic clustering can help in grouping factors that have the same meaning. This would reduce the number of (duplicated) factors and increase the (legitimate) interconnections, resulting in a more cohesive causal map. While various NLP algorithms are applicable for this task, for this project we used the following process. First, ‘sentence embeddings’ were generated using SBERT (Reimers & Gurevych, 2019 ) to capture the semantic meaning of each factor phrase present in the causal pairs. These were then grouped using a density-based clustering algorithm, DBSCAN (Ester et al., 1996 ), with a cosine distance metric and a minimum cluster size of 2. This approach was chosen because it is well-suited to manage uneven cluster sizes and outliers, which are expected in the data. One overarching factor term is then chosen for each cluster and used to overwrite the factors present in that cluster where they occur in causal pairs (see Table 1 ). Table 1 An example of clustered causal pairs. Factors that have been clustered are shown in capital case. Id Raw cause effect pairs Clustered cause effect pairs 2-028A [european level commitments, +, 20-20-2010 targets] [European level commitments, +, 2020 EU CLIMATE ENERGY PACKAGE] 2-028B [20-20-2010 targets, +, renewable energy directive] [2020 EU CLIMATE ENERGY PACKAGE, +, RENEWABLE ENERGY DIRECTIVE] 2-028C [20-20-2010 targets, +, energy efficiency directive] [2020 EU CLIMATE ENERGY PACKAGE, +, energy efficiency directive] 2–029 [renewable energy directive, +, renewable energy utilisation] [RENEWABLE ENERGY DIRECTIVE, +, RENEWABLE ENERGY UTILISATION] 2-031A [carbon price, +, fuel-switching] [EMISSIONS ALLOWANCE PRICE, +, FUEL-SWITCHING] Table A1 The source articles on the ETS used for the synthesis. Contributing review paper Index Individual papers (Green, 2021 ) 1 (B. Anderson & Di Maria, 2011 ) 2 (Gloaguen & Alberola, 2013 ) 3 (Arimura & Abe, 2021 ) 4 (Bayer & Aklin, 2020 ) 5 (Bel & Joseph, 2015 ) 6 (Cullenward, 2014 ) 7 (Wagner et al., 2014 ) 8 (Jaraite-Kažukauske & Di Maria, 2016 ) 9 (Egenhofer et al., 2011 ) 10 (Ellerman et al., 2016 ) 11 (Kotnik et al., 2014 ) 12 (Fell & Maniloff, 2018 ) 13 (Dechezleprêtre et al., 2018 ) 14 (Ellerman & Buchner, 2008 ) 15 (Martin & Saikawa, 2017 ) 16 (Ellerman & McGuinness, 2008 ) 17 (Murray & Maniloff, 2015 ) 18 (Petrick & Wagner, 2014 ) 19 (Wakabayashi & Kimura, 2018 ) (Schmalensee & stavins, 2017 ) 20 (Sijm et al., 2011 ) 21 (Hibbard et al., 2015 ) 22 (Wing & Kolodziej, 2008 ) 23 (Ranson & Stavins, 2012 ) 24 (Ellerman & Buchner, 2007 ) 25 (Kruger et al., 2007 ) 26 (Convery & Redmond, 2007 ) 27 (Sartor et al., 2014 ) 28 (Löfgren et al., 2015 ) The initial collation of causal pairs resulted in the identification of 300 cause/effect factors in our dataset. Using semantic clustering, 230 of these cause/effect factors were grouped into 49 clusters. The remaining 70 factors were deemed unique. From the clustered factors, we could prune duplicate cause-effect pairs leading to 119 unique cause-effect pairs. 3.4 Building an aggregated causal map The next step is to generate the aggregate causal map. Using the clustered cause-effect pairs, individual factors can be added in the map as nodes and their causal links included as the edges between them. The causal map grows in complexity as more factors appear and connections between structures emerge. While based on the causal links identified in earlier stages, this map generation stage inevitably involves coder interpretation. Despite clustering factors, additional grouping may be warranted. Implicit structures are also likely to become more evident, the inclusion of which can allow for richer synthesis. For example, many relations discussed the impact of the free allocation of emission allowances and subsequent sales on the profits of power-generating firms. Initially, this behavior is captured by various links to and from ‘firm profits’. The same behavior may instead be captured by a stock-flow structure, whereby revenues are an inflow to profit and costs an outflow. This representation provides a clearer picture of the underlying dynamics. Figure 3 demonstrates how including the implicit stock-flow structure allows demarcation of factors influencing revenues and costs and their influence on profits. Another issue that can arise in this process is that of ‘intermediate variables’, whereby one causal chain depicts a connection from A à C and another from A à B à C. In such cases, it is often unclear whether the link A à C implies the existence of intermediate factor B, if it is unaware of factor B, it considers B irrelevant, or it suggests that A à B à C may only represent one of multiple paths to C, thereby partially contributing to the causal impact of A on C. Addressing this issue requires interpretation. In situations where an intermediate variable is suggested but not explicitly mentioned by a causal link, the analyst can refer to the contributing text segment to gain more context. In other cases, support cannot be found for an intermediate link, and an additional path is added to the causal map alongside the intermediate path, i.e., including A à B à C alongside A à C. What should be apparent is that model generation is an iterative process. As the map evolves, implicit structures emerge, and intermediate variables appear, frequent reorganization is warranted. There is no objectively complete causal map, rather it should be refined until all causal links have been incorporated, and the overall structure can be used to garner relevant insights. Connection to contributing text segments is also necessary to provide additional context when examining the causal map. By labelling the arrows and providing a reference table of supporting causal IDs, readers can examine each causal link in more detail, if desired. Our iterative process produced the final causal map shown in Fig. 5 . 4 The synthesized evidence: Probing the causal map on the emissions trading scheme In this section, we briefly describe the causal map and demonstrate how it can be further analyzed to create policy-relevant insight. Description of the aggregated ETS causal map Before describing the analysis of the causal map, it is worth describing some core structural elements of the graph. 4.1.1 Emissions allowance price inducing GHG emission reductions. This structure concerns the variables linking ‘emissions allowance price’ and ‘GHG emissions’. There is a clear influence of allowance price on emissions via fuel switching, elaborated by the coal-to-gas price ratio factor. This is consistent with the idea that allowance price will impact the cost of coal more than gas (as coal is more carbon-intensive), thereby inducing fuel switching. The link from allowance price to emissions via cost pass-through/production cost, energy sale price, and energy demand captures the idea that price increases are passed on to consumers who then reduce their energy demand as a result. Additionally, there is the path through emissions reduction investment, as higher allowance prices encourage polluters to enact measures to reduce their emissions. The map also conveys the expected impact that changes in any of these factors would have, through the sign of linkage. For example, a lower allowance price would: i) reduce the coal-to-gas price ratio and curtail fuel switching; ii) limit the energy sale price (due to lower production cost), thereby having lesser influence on energy demand reduction; and iii) lessen the emissions reduction investment. 4.1.2 Emissions leakage diminishing ETS efficacy. This structure concerns ‘emissions leakage’ and its connections, representing the factors contributing to leakage, and the different ways in which leakage impacts the system. One can observe the direct positive link from ‘ETS’ regulation to ‘emissions leakage’ to ‘GHG emissions’, reflecting the idea that ETS regulation incentivizes polluting activity to relocate outside of regulatory jurisdictions, so avoiding ETS-induced emissions reductions. Examining the structural location of emissions leakage also reveals its negative role in important causal chains. For example, consider again the path from allowance price through cost pass-through/production cost, energy demand to GHG emissions. This causal chain captures the idea that a higher allowance price reduces emissions. However, as both cost pass-through and energy production cost have a positive relationship with emissions leakage (which in turn has a negative relationship with GHG emissions), the effect of allowance price on emissions is mitigated, to some extent, by emissions leakage. 4.1.3 Free allocation of allowances leading to windfall profits. This concerns the ‘firm profit’ stock-flow structure and its connections with ‘free allocation of allowances'. This reflects behavior, particularly in the earlier phases of the EU ETS, whereby firms profited from the sale of freely distributed emissions allowances. The basic mechanism is clearly present with a path from free allocation to sale of allowances, revenues and, thereby, firm profit. Several factors elaborate the causes of free allocation, including relocation risks, competitiveness concerns and political demand for new entrant provisions. Additionally, the impacts of the firm ‘windfall profits’ can be seen, i.e., the reduction in competitiveness concerns, and an increasing regulatory threat concerning these profits. Analysis of the aggregated ETS causal map Analysis of the causal map can be conducted using topographic analysis, causal inference analysis, and causal loop analysis. While analysis has revealed interesting insights, these results are not exhaustive, but should typify potential insights. 4.2.1. Topographic analysis. The most supported connections on the map are not surprising. Links #34, #58 and #59 are each supported by nine articles and relate to obvious links between central factors in the system. This high degree of shared knowledge implies that these connections are relevant in explaining the ETS system. Conversely, #71 and #61 have only two articles supporting them. The apparent lack of knowledge regarding these relationships may suggest that further research is warranted. Consider the cause-and-effect trees of ETS system linkage. If an analyst were exploring the impact of linkage between ETS systems, the effect tree would highlight that linkage is expected to reduce compliance costs and lessen price volatility because of increased market thickness, however amongst the numerous other effects are also capital flows between systems and associated negative public perception. The analyst could use this knowledge to recommend measures to mitigate the negative downstream effects, sever the negative effect branches, or introduce reinforcing branches that promote the desired behavior. Figure 6 shows a network graph of the causal map, with nodes sized according to degree centrality and shaded according to betweenness. GHG emissions, for example, is positioned at the end of many causal paths contributing to its high degree. Allowance price, allowance demand, pass-through and firm profits have the highest betweenness. This reflects their position in the key causal paths within the system. Allowance demand has much higher betweenness than centrality indicating that it is important in determining system behavior, but that its influence is fairly narrow. This is consistent with the idea that allowance demand has a large impact on allowance price but does not directly influence other factors outside of this mechanism. 4.2.2. Causal inference analysis. Consider the path from ‘emissions allowance price’ to ‘coal-to-gas price ratio’, ‘fuel switching (coal to gas)’, and ‘GHG emissions’. The partial effect is negative, consistent with the idea that a greater allowance price will lead to a price disparity between coal and gas fuel sources which in turn leads to fuel switching from coal to gas, thereby reducing emissions. Article 2 supported the link between price ratio and fuel switching but did not include the connection between price-ratio and emissions allowance price. Article 16 did mention this relation but the link between price ration and fuel-switching was omitted. It is only by combining the causality of these two articles that the entire causal chain from allowance price to emissions becomes apparent. If a policymaker sought to encourage fuel-switching, an understanding of the aggregated path shows that increasing the allowance price would likely contribute to this end. Looking instead at the total effect, there is an obvious path from ‘cost pass-through’ to ‘GHG emissions’ via ‘energy sale price’, ‘energy demand’, and ‘energy generation emissions’ to ‘GHG emissions’, which has a negative partial effect indicating that greater cost pass-through would lead to emissions reductions. This is consistent with the idea that passing the cost on to consumers would reduce consumption and associated emissions. Such an effect is well-studied and understood, with aspects of this path covered by 15 articles. However, taking instead the path from ‘cost pass-through’ to ‘emissions leakage’ to ‘GHG emissions’ yields a positive partial effect, consistent with the idea that greater pass-through costs contribute to greater leakage and associated net emissions. The relationship between pass-through and leakage is suggested only in Article 27. In this case, the total effect between these two factors is thus undetermined. If targeting emissions reductions, a policymaker may examine the degree of cost pass-through. By only considering the energy sale price pathway, they may conclude that encouraging cost pass-through would be fruitful. Examining the total effect of cost pass-through on emissions would instead reveal that emissions leakage can mitigate the reduction effect somewhat. 4.2.3 Causal loop analysis. Consider the balancing loop between emissions allowance price and GHG emissions (Figure 6). Emissions reductions lessen allowance demand, there is a potential dampening effect on the allowance price if allowance supply cutbacks do not keep pace with these reductions, which in turn impacts emissions. Indeed a similar issue was experienced in the first phase of the EU ETS, an allowance oversupply contributed to an allowance price collapse (Schmalensee & Stavins, 2017). If future allowance supply caps are not set sufficiently low, then successful abatement efforts may reduce the allowance price thereby inducing a balancing effect on GHG emissions. This is undesirable when the goal is to maximize abatement. Consideration of this causal loop indicates that, alongside stringent allowance caps, a price collar could work to mitigate this issue. Comparisons with the meta-reviews It is useful to examine whether the causal map captures the insights of the meta-reviews from which the source material was selected. Our approach performs well in the identification of granular features contributing to high-level behavior. Green (2021), for example, notes the effect of the ETS on fuel switching as well as the (modest) contribution of fuel switching, energy efficiency, and reduced fuel consumption on GHG emissions, both of which are captured in the causal map and subsequent analysis. Further, Green discusses the mitigating role of consistently low carbon price on the impact of ETS on GHG emissions, along with the issues associated with offset credits. By tracing the effect of reducing allowance price, its negative impact on GHG emissions can clearly be seen. Looking at the ‘access to external offset credits’ factor, it can be seen that issues arise relating to market confidence and financial risks, which can negatively impact allowance price. However, our present approach does not reflect the counterfactual and quantitative inferences of this paper, nor how emission reductions vary by sector. Such insights are not visible in the current map because the factors are not sub-categorized according to their respective sectors. Schmalensee & Stavins (2017) do not present quantified insights, but rather discuss the performance of various ETS systems qualitatively. They explain the potentially large revenues that can be generated through allowance auctions, highlight the importance of free allowance allocations in gaining political support for ETS policy and note that these allocations are motivated by concerns of adverse competitiveness impacts. Within the causal map, all these structures are apparent. They further explain the importance of reducing price volatility to facilitate emission abatement, noting how financial conditions led to price instability. This, too, is interpretable from the causal map. First, the adverse impacts of the financial crisis can be seen through its reduction of allowance demand and the subsequent impact on allowance price. Through examination of the various causal paths from allowance price to GHG emissions, it can be interpreted that fluctuating price would in turn fluctuate emission abatement, although there is no explicit connection with the price volatility factor. The insights from this review that are not apparent in the causal map relate to the reflections on the differences in performance of different ETS implementations. For example, how carbon leakage is particularly concerning for subnational systems. Given that the causal map aggregates insights from across the various ETS systems, the differences in behavior across systems is not evident. 5 Discussion We have presented a novel, semi-automated, NLP approach to extract causal statements from policy analysis literature, aggregate them into a causal map of policy behavior, and derive policy-relevant insights. The results show that the method was able to address four shortcomings of state-of-the-art approaches for evidence synthesis for evidence-informed policy making as illustrated for ETS. Firstly, it provides a configurative approach to reviewing and synthesizing available documented causal evidence. This allows analysts to better capture complex mechanisms and interactions underlying observable policy effects and to understand its effectiveness within the policy system. Secondly, the semi-automated approach enables an analyst to do so with significantly less effort than traditional review and evidence syntheses methods, at high levels of accuracy. Using NLP is relatively easy and less time-consuming than manual text review, reducing the time to derive causal relations by a factor of 10, from ~ 3 hours to ~ 20 minutes per article. As a result, a greater number of sources can be considered with the same resources, increasing the evidence base and reducing potential biases arising from a more limited or narrow evidence base. The high precision achieved by the relation extraction algorithm also supports that a high-quality analysis can be obtained. Thirdly, the clustering algorithm helps with integrating policy evidence that is scattered across policy issues or areas. Lastly, the proposed approach can harmonize disconnected information from various levels of scope, performance perspectives, and taxonomies while also incorporating upstream and downstream effects into one comprehensive causal model. In combination, this equips analysts and policy makers with a more systemic understanding of a policy area beyond the direct cause-effect inferences that are typically obtained from traditional means of evidence synthesis. As demonstrated for ETS policy, the derived causal map captures most of the insights obtained from manual evidence syntheses as presented in the review articles by Schmalensee & Stavins ( 2017 ) and Green ( 2021 ). The comparison has highlighted some strengths and weaknesses of this causal map. The strengths include that the generated causal map can faithfully represent the granular features contributing to ETS system behavior and that most policy behavior as highlighted in the reviews is captured. The map performs well in distilling the factors and dynamics present in the disparate source material that are not easily obtained nor communicated otherwise. This can supplement traditional methods in presenting an explicit, parsimonious, aggregated systems perspective. The shortcomings of this representation relate to how well contextual information and counterfactuals are conveyed. Quantified insights and insights by sector were not represented explicitly in this causal map, however, they could be uncovered by referencing the contributing text segments. The map also overlooked some important counterfactual findings which proved to be important in one review paper. Despite these promising results, several limitations remain that should be addressed in future work to ensure high quality analysis and to realize the potential of this approach to evidence-informed policy. A major limitation concerns the low recall (~ 38%) achieved by the causal extractions algorithm we used. A consequence being that relevant causal links contributing to system behavior were probably missed. The low recall is likely explained by the data sources used (and the complexity of cause-effects therein, relative to the training data) rather than an inherent issue of the algorithm, which has been demonstrated to achieve recall rates of up to 86% on benchmark data sets (Li et al., 2021 ). Addressing this shortcoming, however, is complicated by the sparsity of training data (Asghar, 2016 ). Nevertheless, the capabilities of deep learning methods for causal relation extraction are advancing rapidly (Yang et al., 2021 ). Strategies to overcome existing limitations include utilizing increasingly more capable NLP-based causality extraction algorithms. Additionally, staged model building can help by starting out from a basic model that reflects widely accepted cause-effect relations, building the system map further by adding cause-effect relations from literature. Another limitation concerns the inability of the causal relation extraction algorithm to distinguish between hypothesized and empirically substantiated cause-effects, while also lacking consideration for counterfactuals. Recent developments using adversarial training for causality extraction models may be able to address this (Feder et al., 2021 ). A related challenge is the lack of a well-specified context within which the extracted cause-effect relations are deemed valid. It is possible to uncover additional contextual information by analyzing the underlying textual information present in the reference table, before creating the causal map. However, this is inconvenient especially when new information is collected iteratively, requiring manual updates of the map as regards link proximity and clustering, or other attributes to facilitate map interpretation. Again, developments in NLP may soon overcome these limitations. One promising area of development in this regard is word embedding techniques to facilitate contextual mapping of variables in a map (Pelevina et al., 2017 ; Selva Birunda & Kanniga Devi, 2021 ). Another area concerns recent advances in automating the workflow from causal relationship extraction via relations table building to creating a visual causal map (e.g. building onto Ancin-Murguzur & Hausner, 2021 ). This would reduce the need for iterative updating of the reference table and map, while facilitating interactive exploration by analysts and policy makers. Similarly, moving from semi-automation towards full automation would make it possible to analyze orders of magnitude more articles which would be necessary to achieve full coverage of a policy issue or area of interest with thousands of relevant publications as is the case for the ETS. Further to these important areas for improvement, future research may seek to use our approach to compare and contrast the causal maps that can be extracted from different sources and fields of evidence generation. This can help to expose conflicting information about causal paths from different sources of evidence. A directed network map representation allows to gauge contributions of various factors within and across causal chains, including potential for weighting of links or factors e.g., by frequency of mentioning, source type, soundness of the empirical basis, or authoritativeness of the source. The directed map representation furthermore provides additional analytic capacity. This includes topographic analyses using graph-theory based concepts to identify the most central factors as system levers or barriers to effective policy. Alternatively, causal reinforcing or balancing factors can be identified and explored, which is not commonly done in ETS policy analysis. In a broad field such as ETS policy, it is challenging to keep an overview of the many policies and their effects, both individually and in concert. Here, a causal map presentation allows to either zoom in to evaluate a specific policy, or to zoom out and identify feedback with other policies that may enhance or limit their effectiveness. It may also guide empirical validation of the key cause-effects identified. Beyond that, their expansion to system maps (i.e., that include possible policy actions, policy effects, and contextual variables that would influence the cause-effect pathway, see e.g., Enserink et al., 2022) would provide an ideal foundation for model-supported exploration of policy behavior and policy system change over time (e.g., using system dynamics or agent-based modelling). Altogether, these would result in a more comprehensive and complexity-proof evidence-base for policy evaluation and development. 6 Conclusion Evidence-based or evidence-informed policy relies on the ability to effectively summarize, synthesize, and mobilize knowledge for the policy process, yet characteristics of policy sciences as a field make evidence synthesis challenging in practice. The exponential growth in policy research significantly increases the resources necessary for evidence synthesis, evidence is often scattered across different policy areas and employing distinct terminology; policy impacts can be measured in a variety of dimensions, such as programmatic, process, and political, and at different levels from micro-level studies of individual policies to macro-level research on entire policy areas - all of which need to be taken into account when synthesizing evidence. To address these shortcomings, this article has introduced a novel analysis method that can semi-automatically derive and aggregate causal relations from policy analysis literature into a causal map of policy effects. Applying this method to a collection of 28 ETS literature sources produced a causal map consisting of 159 unique causal links. Evaluation of this result has demonstrated that the approach allows analysts to better capture complex mechanisms and interactions underlying observable policy effects with significantly less effort than traditional review and evidence synthesis methods. It also supports synthesis of policy evidence scattered across policy issues or areas and can harmonize disconnected information from various levels of scope, performance perspectives, and taxonomies. Comparison of insights obtained from the causal map against those from a manual review of the same source material has also demonstrated that most of the insights can be captured by this approach, whilst providing a more configurative perspective of the features contributing to policy behavior. Finally, in providing a causal map representation of a policy area, new tools for policy evaluation, such as topographic, causal inference, and causal loop analysis, become available to analysts. While promising in many respects, some notable limitations include the poor recall (~ 38%) achieved in this application of the method. This may contribute to structural gaps in the map and a lack of contextualization for cause-effect relations which can inhibit understanding of the nuance behind certain behavior. Regardless of algorithmic performance or the quality of insights obtained for ETS, the method and implementation presented in this study only represents a first step in combining NLP, causal mapping, and graph analytics for policy-relevant evidence synthesis. The proposed method, and future iterations, appears to contribute a promising new tool for policy analysts across domains, helping to provide a more comprehensive understanding of the factors and relations affecting policy and ultimately improving the evidence base on which to inform policy development. Declarations 7 Conflict of interest All authors declare that they have no conflicts of interest. References Ackermann, F., & Alexander, J. (2016). Researching complex projects: Using causal mapping to take a systems perspective. International Journal of Project Management , 34 (6), 891–901. https://doi.org/10.1016/j.ijproman.2016.04.001 Akbik, A., Blythe, D., & Vollgraf, R. (2018). Contextual string embeddings for sequence labeling. Proceedings of the 27th International Conference on Computational Linguistics , 1638–1649. Ancin-Murguzur, F. J., & Hausner, V. H. (2021). causalizeR: A text mining algorithm to identify causal relationships in scientific literature. PeerJ , 9 , e11850. https://doi.org/10.7717/peerj.11850 Anderson, B., & Di Maria, C. (2011). Abatement and Allocation in the Pilot Phase of the EU ETS. Environmental and Resource Economics , 48 (1), 83–103. https://doi.org/10.1007/s10640-010-9399-9 Anderson, L. M., Oliver, S. R., Michie, S., Rehfuess, E., Noyes, J., & Shemilt, I. (2013). Investigating complexity in systematic reviews of interventions by using a spectrum of methods. Journal of Clinical Epidemiology , 66 (11), 1223–1229. https://doi.org/10.1016/j.jclinepi.2013.06.014 Arimura, T. H., & Abe, T. (2021). The impact of the Tokyo emissions trading scheme on office buildings: What factor contributed to the emission reduction? Environmental Economics and Policy Studies , 23 (3), 517–533. https://doi.org/10.1007/s10018-020-00271-w Asghar, N. (2016). Automatic Extraction of Causal Relations from Natural Language Texts: A Comprehensive Survey . https://arxiv.org/abs/1605.07895v1 Axelrod, R. (1976). Structure of Decision: The Cognitive Maps of Political Elites . Princeton University Press. Bach, N., & Badaskar, S. (2007). A review of relation extraction. Literature Review for Language and Statistics II , 2 , 1–15. Barbrook-Johnson, P., & Penn, A. S. (2022). Systems Mapping: How to build and use causal models of systems . Springer Nature. Barik, B., Marsi, E., & Öztürk, P. (2016). Event Causality Extraction from Natural Science Literature. Research in Computing Science , 117 (1), 97–107. https://doi.org/10.13053/rcs-117-1-8 Barza, M., Trikalinos, T. A., & Lau, J. (2009). Statistical Considerations in Meta-analysis. Infectious Disease Clinics of North America , 23 (2), 195–210. https://doi.org/10.1016/j.idc.2009.01.003 Bayer, P., & Aklin, M. (2020). The European Union Emissions Trading System reduced CO2 emissions despite low prices. Proceedings of the National Academy of Sciences , 117 (16), 8804–8812. https://doi.org/10.1073/pnas.1918128117 Beamer, B., Rozovskaya, A., & Girju, R. (2008). Automatic Semantic Relation Extraction with Multiple Boundary Generation. AAAI , 824–829. Bel, G., & Joseph, S. (2015). Emission abatement: Untangling the impacts of the EU ETS and the economic crisis. Energy Economics , 49 , 531–539. https://doi.org/10.1016/j.eneco.2015.03.014 Beltagy, I., Lo, K., & Cohan, A. (2019). SciBERT: A Pretrained Language Model for Scientific Text. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , 3615–3620. https://doi.org/10.18653/v1/D19-1371 Bornmann, L., & Mutz, R. (2015). Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. Journal of the Association for Information Science and Technology , 66 (11), 2215–2222. Bui, Q.-C., Nualláin, B. Ó., Boucher, C. A., & Sloot, P. M. (2010). Extracting causal relations on HIV drug resistance from literature. BMC Bioinformatics , 11 (1), 101. https://doi.org/10.1186/1471-2105-11-101 Burns, P. B., Rohrich, R. J., & Chung, K. C. (2011). The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg , 128 (1), 305–310. https://doi.org/10.1097/PRS.0b013e318219c171 Carroll, C., Booth, A., & Cooper, K. (2011). A worked example of ‘best fit’ framework synthesis: A systematic review of views concerning the taking of some potential chemopreventive agents. BMC Medical Research Methodology , 11 . https://doi.org/10.1186/1471-2288-11-29 Compton, M., & ’t Hart, P. (2019). Great policy successes (First). Oxford University Press. Convery, F. J., & Redmond, L. (2007). Market and Price Developments in the European Union Emissions Trading Scheme. Review of Environmental Economics and Policy , 1 (1), 88–111. https://doi.org/10.1093/reep/rem010 Cruanes, J., Roma-Ferri, M. T., & Lloret, E. (2012). Measuring lexical similarity methods for textual mapping in nursing diagnoses in Spanish and SNOMED-CT . 180 , 255–259. https://doi.org/10.3233/978-1-61499-101-4-255 Cullenward, D. (2014). Leakage in California’s Carbon Market. The Electricity Journal , 27 (9), 36–48. https://doi.org/10.1016/j.tej.2014.09.014 Dechezleprêtre, A., Nachtigall, D., & Venmans, F. (2018). The joint impact of the European Union emissions trading system on carbon emissions and economic performance . OECD. https://doi.org/10.1787/4819b016-en Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv Preprint ArXiv:1810.04805 . Donald Nguyen, Andrew Lenharth, & Keshav Pingali. (2013). A lightweight infrastructure for graph analytics . Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, Farminton, Pennsylvania. Eden, C., Ackermann, F., & Cropper, S. (1992). The Analysis of Cause Maps. Journal of Management Studies , 29 (3), 309–324. https://doi.org/10.1111/j.1467-6486.1992.tb00667.x Egenhofer, C., Alessi, M., Georgiev, A., & Fujiwara, N. (2011). The EU Emissions Trading System and Climate Policy Towards 2050: Real Incentives to Reduce Emissions and Drive Innovation? (SSRN Scholarly Paper 1756736). Social Science Research Network. https://papers.ssrn.com/abstract=1756736 Ellerman, A. D., & Buchner, B. K. (2007). The European Union Emissions Trading Scheme: Origins, Allocation, and Early Results. Review of Environmental Economics and Policy , 1 (1), 66–87. https://doi.org/10.1093/reep/rem003 Ellerman, A. D., & Buchner, B. K. (2008). Over-Allocation or Abatement? A Preliminary Analysis of the EU ETS Based on the 2005–06 Emissions Data. Environmental and Resource Economics , 41 (2), 267–287. https://doi.org/10.1007/s10640-008-9191-2 Ellerman, A. D., Marcantonini, C., & Zaklan, A. (2016). The European Union Emissions Trading System: Ten Years and Counting. Review of Environmental Economics and Policy , 10 (1), 89–107. https://doi.org/10.1093/reep/rev014 Ellerman, A. D., & McGuinness, M. (2008). CO2 Abatement in the UK Power Sector: Evidence from the EU ETS Trial Period [Working Paper]. https://dspace.mit.edu/handle/1721.1/45654 El‐Taliawi, O. G., Goyal, N., & Howlett, M. (2021). Holding out the promise of Lasswell’s dream: Big data analytics in public policy research and teaching. Review of Policy Research , 38 (6), 640–660. Ester, M., Kriegel, H.-P., Sander, J., & Xu, X. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise. Kdd , 96 (34), 226–231. Esty, D. C., & Porter, M. E. (2005). National environmental performance: An empirical analysis of policy results and determinants. Environment and Development Economics , 10 (4), 391–434. Scopus. https://doi.org/10.1017/S1355770X05002275 European Commission. (2021). EU Emissions Trading System (EU ETS) . https://ec.europa.eu/clima/eu-action/eu-emissions-trading-system-eu-ets_en Feder, A., Oved, N., Shalit, U., & Reichart, R. (2021). CausaLM: Causal Model Explanation Through Counterfactual Language Models. Computational Linguistics , 47 (2), 333–386. https://doi.org/10.1162/coli_a_00404 Fell, H., & Maniloff, P. (2018). Leakage in regional environmental policy: The case of the regional greenhouse gas initiative. Journal of Environmental Economics and Management , 87 , 1–23. https://doi.org/10.1016/j.jeem.2017.10.007 France, E. F., Cunningham, M., Ring, N., Uny, I., Duncan, E. A. S., Jepson, R. G., Maxwell, M., Roberts, R. J., Turley, R. L., Booth, A., Britten, N., Flemming, K., Gallagher, I., Garside, R., Hannes, K., Lewin, S., Noblit, G. W., Pope, C., Thomas, J., … Noyes, J. (2019). Improving reporting of meta-ethnography: The eMERGe reporting guidance. Psycho-Oncology , 28 (3), 447–458. https://doi.org/10.1002/pon.4915 Freebairn, L., Atkinson, J., Kelly, P., McDonnell, G., & Rychetnik, L. (2016). Simulation modelling as a tool for knowledge mobilisation in health policy settings: A case study protocol. Health Research Policy and Systems , 14 (1). https://doi.org/10.1186/s12961-016-0143-y Fusar-Poli, P., & Radua, J. (2018). Ten simple rules for conducting umbrella reviews. Evidence-Based Mental Health , 21 (3), 95–100. https://doi.org/10.1136/ebmental-2018-300014 Girju, R., Nakov, P., Nastase, V., Szpakowicz, S., Turney, P., & Yuret, D. (2009). Classification of semantic relations between nominals. Language Resources and Evaluation , 43 (2), 105–121. https://doi.org/10.1007/s10579-009-9083-2 Gloaguen, O., & Alberola, E. (2013). Assessing the factors behind CO2 emissions changes over the phases 1 and 2 of the EU ETS: An econometric analysis. CDC Climat Research, Paris, France . Goyal, N. (2021). Explaining Policy Success Using the Multiple Streams Framework: Political Success Despite Programmatic Failure of the Solar Energy Policy in Gujarat, India. Politics & Policy , 49 (5), 1021–1060. https://doi.org/10.1111/polp.12426 Goyal, N., & Howlett, M. (2018). Lessons Learned and Not Learned: Bibliometric Analysis of Policy Learning. In C. A. Dunlop, C. M. Radaelli, & P. Trein (Eds.), Learning in Public Policy: Analysis, Modes and Outcomes (pp. 27–49). Springer International Publishing. Goyal, N., & Howlett, M. (2019). Combining internal and external evaluations within a multilevel evaluation framework: Computational text analysis of lessons from the Asian Development Bank. Evaluation , 25 (3), 366–380. https://doi.org/10.1177/1356389019827035 Goyal, N., & Howlett, M. (2021). “Measuring the Mix” of Policy Responses to COVID-19: Comparative Policy Analysis Using Topic Modelling. Journal of Comparative Policy Analysis: Research and Practice , 23 (2), 250–261. Green, J. F. (2021). Does carbon pricing reduce emissions? A review of ex-post analyses. Environmental Research Letters , 16 (4), 043004. https://doi.org/10.1088/1748-9326/abdae9 Greenhalgh, T., Wong, G., Westhorp, G., & Pawson, R. (2011). Protocol - realist and meta-narrative evidence synthesis: Evolving Standards (RAMESES). BMC Medical Research Methodology , 11 (1), 115. https://doi.org/10.1186/1471-2288-11-115 Haddaway, N. R., Land, M., & Macura, B. (2017). “A little learning is a dangerous thing”: A call for better understanding of the term ‘systematic review’. Environment International , 99 , 356–360. https://doi.org/10.1016/j.envint.2016.12.020 Han, H., Wang, Q., & Chen, C. (2019). Policy Text Analysis Based on Text Mining and Fuzzy Cognitive Map. 2019 15th International Conference on Computational Intelligence and Security (CIS) , 142–146. https://doi.org/10.1109/CIS.2019.00038 Hannes, K., & Lockwood, C. (2011). Pragmatism as the philosophical foundation for the Joanna Briggs meta-aggregative approach to qualitative evidence synthesis. Journal of Advanced Nursing , 67 (7), 1632–1642. https://doi.org/10.1111/j.1365-2648.2011.05636.x Hansen, C., Steinmetz, H., & Block, J. (2022). How to conduct a meta-analysis in eight steps: A practical guide. Management Review Quarterly , 72 (1), 1–19. https://doi.org/10.1007/s11301-021-00247-4 Hibbard, P. J., Okie, A. M., Tierney, S. F., & Darling, P. G. (2015). The economic impacts of the regional greenhouse gas initiative on nine northeast and mid-Atlantic states. Analysis Group, July . ICAP. (2022a). Emissions Trading Worldwide – International Carbon Action Partnership (ICAP) Status Report 2022 . 240. ICAP. (2022b). About Emissions Trading Systems . International Carbon Action Partnership. https://icapcarbonaction.com/en/about-emissions-trading-systems Inan, E. (2020). SimiT: A Text Similarity Method Using Lexicon and Dependency Representations. New Generation Computing , 38 (3), 509–530. https://doi.org/10.1007/s00354-020-00099-8 James, K. L., Randall, N. P., & Haddaway, N. R. (2016). A methodology for systematic mapping in environmental sciences. Environmental Evidence , 5 (1). https://doi.org/10.1186/s13750-016-0059-6 Jaraite-Kažukauske, J., & Di Maria, C. (2016). Did the EU ETS make a difference? An empirical assessment using Lithuanian firm-level data. The Energy Journal , 37 (1). Jiali, L. (1995). China’s one-child policy: How and how well has it worked? A case study of Hebei Province, 1979-88. Population & Development Review , 21 (3), 563–585. Scopus. Khoo, C. S., & Na, J.-C. (2006). Semantic relations in information science. Annual Review of Information Science and Technology , 40 (1), 157–228. Kim, J., Han, M., Lee, Y., & Park, Y. (2016). Futuristic data-driven scenario building: Incorporating text mining and fuzzy association rule mining into fuzzy cognitive map. Expert Systems with Applications , 57 , 311–323. Scopus. https://doi.org/10.1016/j.eswa.2016.03.043 Kotnik, Ž., Maja, K., & Škulj, D. (2014). The effect of taxation on greenhouse gas emissions. Transylvanian Review of Administrative Sciences , 10 (43), 168–185. Kruger, J., Oates, W. E., & Pizer, W. A. (2007). Decentralization in the EU Emissions Trading Scheme and Lessons for Global Policy. Review of Environmental Economics and Policy , 1 (1), 112–133. https://doi.org/10.1093/reep/rem009 Kyriakakis, M., Androutsopoulos, I., Saudabayev, A., & Ginés i Ametllé, J. (2019). Transfer Learning for Causal Sentence Detection. Proceedings of the 18th BioNLP Workshop and Shared Task , 292–297. https://doi.org/10.18653/v1/W19-5031 Larsen, P., & Von Ins, M. (2010). The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics , 84 (3), 575–603. Leining, C., Kerr, S., & Bruce-Brand, B. (2020). The New Zealand Emissions Trading Scheme: Critical review and future outlook for three design innovations. Climate Policy , 20 (2), 246–264. https://doi.org/10.1080/14693062.2019.1699773 Levy, O., & Goldberg, Y. (2014). Dependency-based word embeddings. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , 302–308. Li, Z., Li, Q., Zou, X., & Ren, J. (2021). Causality extraction based on self-attentive BiLSTM-CRF with transferred embeddings. Neurocomputing , 423 , 207–219. https://doi.org/10.1016/j.neucom.2020.08.078 Liddy, E. D. (2001). Natural language processing . Liu, X. Y., Zhou, Y. M., & Zheng, R. S. (2007). Measuring semantic similarity in WordNet . 3431-+. Löfgren, Å., Burtraw, D., Wråke, M., & Malinovskaya, A. (2015). Architecture of the EU emissions trading system in phase 3 and the distribution of allowance asset values . Macura, B., Suškevičs, M., Garside, R., Hannes, K., Rees, R., & Rodela, R. (2019). Systematic reviews of qualitative evidence for environmental policy and management: An overview of different methodological options. Environmental Evidence , 8 (1). https://doi.org/10.1186/s13750-019-0168-0 Marsh, D., & McConnell, A. (2010). Towards a Framework for Establishing Policy Success. Public Administration , 88 (2), 564–583. https://doi.org/10.1111/j.1467-9299.2009.01803.x Martin, G., & Saikawa, E. (2017). Effectiveness of state climate and energy policies in reducing power-sector CO2 emissions. Nature Climate Change , 7 (12), Article 12. https://doi.org/10.1038/s41558-017-0001-0 McConnell, A. (2010). Policy Success, Policy Failure and Grey Areas In-Between. Journal of Public Policy , 30 (3), 345–362. https://doi.org/10.1017/S0143814X10000152 Millard, T., Synnot, A., Elliott, J., Green, S., McDonald, S., & Turner, T. (2019). Feasibility and acceptability of living systematic reviews: Results from a mixed-methods evaluation. Systematic Reviews , 8 (1). https://doi.org/10.1186/s13643-019-1248-5 Montibeller, G., & Belton, V. (2006). Causal maps and the evaluation of decision options—A review. Journal of the Operational Research Society , 57 (7), 779–791. https://doi.org/10.1057/palgrave.jors.2602214 Murray, B. C., & Maniloff, P. T. (2015). Why have greenhouse emissions in RGGI states declined? An econometric attribution to economic, energy market, and policy factors. Energy Economics , 51 , 581–589. https://doi.org/10.1016/j.eneco.2015.07.013 Norman, E. R. (2023). Policy Studies Yearbook Annual Review 2022–23: Note from the editor and call for papers. Politics & Policy . Nunez-Mir, G. C., Iannone III, B. V., Pijanowski, B. C., Kong, N., & Fei, S. (2016). Automated content analysis: Addressing the big literature challenge in ecology and evolution. Methods in Ecology and Evolution , 7 (11), 1262–1272. https://doi.org/10.1111/2041-210X.12602 O’Leary, D. F., Casey, M., O’Connor, L., Stokes, D., Fealy, G. M., O’Brien, D., Smith, R., McNamara, M. S., & Egan, C. (2017). Using rapid reviews: An example from a study conducted to inform policy-making. Journal of Advanced Nursing , 73 (3), 742–752. https://doi.org/10.1111/jan.13231 Pakray, P., & Gelbukh, A. (2014). An Open-Domain Cause-Effect Relation Detection from Paired Nominals. In A. Gelbukh, F. C. Espinoza, & S. N. Galicia-Haro (Eds.), Nature-Inspired Computation and Machine Learning (pp. 263–271). Springer International Publishing. https://doi.org/10.1007/978-3-319-13650-9_24 Pearson, A., White, H., Bath-Hextall, F., Salmond, S., Apostolo, J., & Kirkpatrick, P. (2015). A mixed-methods approach to systematic reviews. International Journal of Evidence-Based Healthcare , 13 (3), 121–131. https://doi.org/10.1097/XEB.0000000000000052 Pelevina, M., Arefyev, N., Biemann, C., & Panchenko, A. (2017). Making Sense of Word Embeddings (arXiv:1708.03390). arXiv. https://doi.org/10.48550/arXiv.1708.03390 Peters, M. D. J., Marnie, C., Colquhoun, H., Garritty, C. M., Hempel, S., Horsley, T., Langlois, E. V., Lillie, E., O’Brien, K. K., Tunçalp, Ӧ., Wilson, M. G., Zarin, W., & Tricco, A. C. (2021). Scoping reviews: Reinforcing and advancing the methodology and application. Systematic Reviews , 10 (1). https://doi.org/10.1186/s13643-021-01821-3 Peters, M. D. J., Marnie, C., Tricco, A. C., Pollock, D., Munn, Z., Alexander, L., McInerney, P., Godfrey, C. M., & Khalil, H. (2020). Updated methodological guidance for the conduct of scoping reviews. JBI Evidence Synthesis , 18 (10), 2119–2126. https://doi.org/10.11124/JBIES-20-00167 Petrick, S., & Wagner, U. J. (2014). The Impact of Carbon Trading on Industry: Evidence from German Manufacturing Firms (SSRN Scholarly Paper 2389800). Social Science Research Network. https://doi.org/10.2139/ssrn.2389800 Petticrew, M., & Roberts, H. (2008). Systematic Reviews in the Social Sciences: A Practical Guide . Pullin, A., Frampton, G., Jongman, R., Kohl, C., Livoreil, B., Lux, A., Pataki, G., Petrokofsky, G., Podhora, A., Saarikoski, H., Santamaria, L., Schindler, S., Sousa-Pinto, I., Vandewalle, M., & Wittmer, H. (2016). Selecting appropriate methods of knowledge synthesis to inform biodiversity policy. Biodiversity and Conservation , 25 (7), 1285–1300. https://doi.org/10.1007/s10531-016-1131-9 Ranson, M., & Stavins, R. N. (2012). Post-Durban climate policy architecture based on linkage of cap-and-trade systems . National Bureau of Economic Research. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (arXiv:1908.10084). arXiv. http://arxiv.org/abs/1908.10084 Saetren, H. (2005). Facts and myths about research on public policy implementation: Out-of-fashion, allegedly dead, but still very much alive and relevant. Policy Studies Journal , 33 (4), 559–582. https://doi.org/10.1111/j.1541-0072.2005.00133.x Sanderson, I. (2002). Evaluation, policy learning and evidence-based policy making. Public Administration , 80 (1), 1–22. Sartor, O., Pallière, C., & Lecourt, S. (2014). Benchmark-based allocations in EU ETS Phase 3: An early assessment. Climate Policy , 14 (4), 507–524. https://doi.org/10.1080/14693062.2014.872888 Saul, J. E., Willis, C. D., Bitz, J., & Best, A. (2013). A time-responsive tool for informing policy making: Rapid realist review. Implementation Science , 8 (1). https://doi.org/10.1186/1748-5908-8-103 Schmalensee, R., & Stavins, R. N. (2017). Lessons Learned from Three Decades of Experience with Cap and Trade. Review of Environmental Economics and Policy , 11 (1), 59–79. https://doi.org/10.1093/reep/rew017 Selva Birunda, S., & Kanniga Devi, R. (2021). A Review on Word Embedding Techniques for Text Classification. In J. S. Raj, A. M. Iliyasu, R. Bestak, & Z. A. Baig (Eds.), Innovative Data Communication Technologies and Application (pp. 267–281). Springer. https://doi.org/10.1007/978-981-15-9651-3_23 Sijm, J., Neuhoff, K., & Chen, Y. (2011). CO2 cost pass-through and windfall profits in the power sector. Climate Policy , 6 (1), 49–72. https://doi.org/10.1080/14693062.2006.9685588 Son, C., Kim, J., & Kim, Y. (2020). Developing scenario-based technology roadmap in the big data era: An utilisation of fuzzy cognitive map and text mining techniques. Technology Analysis & Strategic Management , 32 (3), 272–291. https://doi.org/10.1080/09537325.2019.1654091 Thomas, J., & Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol , 8 , 45. https://doi.org/10.1186/1471-2288-8-45 Thow, A. M., Swinburn, B., Colagiuri, S., Diligolevu, M., Quested, C., Vivili, P., & Leeder, S. (2010). Trade and food policy: Case studies from three Pacific Island countries. Food Policy , 35 (6), 556–564. Scopus. https://doi.org/10.1016/j.foodpol.2010.06.005 van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., & Oberski, D. L. (2021). An open source machine learning framework for efficient and transparent systematic reviews. Nature Machine Intelligence , 3 (2), Article 2. https://doi.org/10.1038/s42256-020-00287-7 Wagner, U. J., Muûls, M., Martin, R., & Colmer, J. (2014). The causal effects of the European Union Emissions Trading Scheme: Evidence from French manufacturing plants. Fifth World Congress of Environmental and Resources Economists, Instanbul, Turkey . Wakabayashi, M., & Kimura, O. (2018). The impact of the Tokyo Metropolitan Emissions Trading Scheme on reducing greenhouse gas emissions: Findings from a facility-based study. Climate Policy , 18 (8), 1028–1043. https://doi.org/10.1080/14693062.2018.1437018 Warner, J., & van Buuren, A. (2011). Implementing room for the river: Narratives of success and failure in Kampen, the Netherlands. International Review of Administrative Sciences , 77 (4), 779–801. Scopus. https://doi.org/10.1177/0020852311419387 Weible, C. M., & Sabatier, P. A. (2018). Theories of the Policy Process . Taylor and Francis. Wing, I. S., & Kolodziej, M. (2008). The Regional Greenhouse Gas Initiative: Emission Leakage and the Effectiveness of Interstate Border Adjustments . Wolffe, T. A. M., Whaley, P., Halsall, C., Rooney, A. A., & Walker, V. R. (2019). Systematic evidence maps as a novel tool to support evidence-based decision-making in chemicals policy and risk management. Environ Int , 130 , 104871. https://doi.org/10.1016/j.envint.2019.05.065 Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013a). RAMESES publication standards: Meta-narrative reviews. BMC Med , 11 , 20. https://doi.org/10.1186/1741-7015-11-20 Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013b). RAMESES publication standards: Realist syntheses. BMC Medicine , 11 (1). https://doi.org/10.1186/1741-7015-11-21 World Bank. (2023). State and Trends of Carbon Pricing 2023 . https://doi.org/10.1596/39796 Yang, J., Han, S. C., & Poon, J. (2021). A Survey on Extraction of Causal Relations from Natural Language Text. ArXiv:2101.06426 [Cs] . http://arxiv.org/abs/2101.06426 Zhao, S., Liu, T., Zhao, S., Chen, Y., & Nie, J.-Y. (2016). Event causality extraction based on connectives analysis. Neurocomputing , 173 , 1943–1950. https://doi.org/10.1016/j.neucom.2015.09.066 Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3285731","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":228132108,"identity":"325b51ec-b824-4afc-bf66-d8fd19db5110","order_by":0,"name":"Rory Hooper","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6ElEQVRIiWNgGAWjYJACZgjF2GDwAUixsROvhbmhcAZICzPxWtgbPvMgcXEC8/beZ9IFFTZ5/NIHGzfb/Nomz8fMwPjhYw5uLTJnjptJzziTVizZl9hsnNt327CNmYFZcuY23FokJNLYpHnbDiduOMPYZpzbc5sRqIWNmZcYLfvPMLb/tuy5bU+8lg08jA3GDD9uJxLWwnOM2ZoH6BeJM4wNhr0Nt5PbmBmb8fuFvY3xNg8oxHrYHxj8+HPbdn5788EPH/FoAQIWCSCRAGYytoHJBrzqgYD5A1wLwx9CikfBKBgFo2AkAgCuX0qIIpROOAAAAABJRU5ErkJggg==","orcid":"","institution":"","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Rory","middleName":"","lastName":"Hooper","suffix":""},{"id":228132125,"identity":"67c4522e-4f79-454c-b6d4-5ec679bfabec","order_by":1,"name":"Nihit Goyal","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA50lEQVRIie2POwrCQBCGJyysTYytNolHWEkh4mVcLGxTScA3gTmDx8gRNgS0CaZNafAC8QZOfICNcUvB/Ypl/2U+/lkAg+EXYTao5213pgMcVufgi/JwrEgoUjifUhBNNTY8ayzsainDVjtJK1i54pjg4opjl/OOYkGDMoqcCS129EUmsUhw5nPOgO0bFJHagpSDjNVdSSV6EX1QS8lLDEjZYN2ioSxlXEgEUiZcS8mE8ntFGXWz02xALfVjg5JnfhWGa9fJp0kVzsdeh1vlxQ4/K68u6Ku3rD7MvbMGb6sxZjAYDP/JDQK5U24rEdOFAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-1025-7585","institution":"","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Nihit","middleName":"","lastName":"Goyal","suffix":""},{"id":228133354,"identity":"eb6e19d2-3101-4589-8344-01af2ea7f1ab","order_by":2,"name":"Kornelis Blok","email":"","orcid":"https://orcid.org/0000-0003-4869-6927","institution":"","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Kornelis","middleName":"","lastName":"Blok","suffix":""},{"id":228133355,"identity":"b4f2e02c-9d23-4630-adef-e21346f973f9","order_by":3,"name":"Lisa Scholten","email":"","orcid":"https://orcid.org/0000-0002-4487-758X","institution":"","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Lisa","middleName":"","lastName":"Scholten","suffix":""}],"badges":[],"createdAt":"2023-08-22 12:09:11","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":true,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false,"coiExplicitlySet":false},"doi":"10.21203/rs.3.rs-3285731/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3285731/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":42030103,"identity":"45619960-75b0-4d81-983b-821e287fd674","added_by":"auto","created_at":"2023-08-23 17:43:48","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":19989,"visible":true,"origin":"","legend":"\u003cp\u003eInput-output diagram of the four-step process\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-3285731/v1/bd4e5760bd82a84893ddcce7.png"},{"id":42030104,"identity":"4add8c4f-70b4-44c6-9e4f-6d7dd11507aa","added_by":"auto","created_at":"2023-08-23 17:43:49","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":22794,"visible":true,"origin":"","legend":"\u003cp\u003eExample SCITE output, article 4 sentence 49\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-3285731/v1/1021e59dff0b794ceb9e60a5.png"},{"id":42030107,"identity":"ee122d36-5214-4309-bc65-50c14121f4ef","added_by":"auto","created_at":"2023-08-23 17:43:49","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":39188,"visible":true,"origin":"","legend":"\u003cp\u003eExample of including implicit structures.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-3285731/v1/2df2c3d6c5accf38e31ddb1c.png"},{"id":42031464,"identity":"1ffa118c-ec1e-451d-9f87-79583312551a","added_by":"auto","created_at":"2023-08-23 17:51:49","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":595399,"visible":true,"origin":"","legend":"\u003cp\u003eFinal aggregated causal map\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-3285731/v1/387c340e0c4ca974393c4ec4.png"},{"id":42030108,"identity":"199294f1-1ee0-47ff-af45-b0812b38a7e7","added_by":"auto","created_at":"2023-08-23 17:43:49","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":358783,"visible":true,"origin":"","legend":"\u003cp\u003eNetwork graph representation of the causal map\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-3285731/v1/55a0f59dd59685983a7cfaa0.png"},{"id":42032304,"identity":"48168d63-cc7b-447a-b87b-acde5b7c601c","added_by":"auto","created_at":"2023-08-23 17:59:49","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":19816,"visible":true,"origin":"","legend":"\u003cp\u003eConsideration of allowance price collars, causal loop\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-3285731/v1/2b349807a3476d22c81af7b1.png"},{"id":42032307,"identity":"4f958f91-798f-42f5-ab5f-3e5901ddf23f","added_by":"auto","created_at":"2023-08-23 17:59:55","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1173083,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3285731/v1/07dc0e57-f4a6-4930-b443-00c68fb244d7.pdf"}],"financialInterests":"","formattedTitle":"\u003cp\u003eA semi-automated approach to policy-relevant evidence synthesis: Combining natural language processing, causal mapping, and graph analytics for public policy\u003c/p\u003e","fulltext":[{"header":"1 Introduction","content":"\u003cp\u003eTransparent, timely, repeatable, and contextual synthesis of causal evidence is important for analyzing and informing every stage of the policy cycle. Policy research relies on effective summarization, synthesis, and mobilization of knowledge \u003cem\u003efor\u003c/em\u003e or \u003cem\u003ein\u003c/em\u003e the policy process. Studies that aim to facilitate evidence-based or evidence-informed policy-making, policy evaluation, and policy learning require at least some degree of evidence synthesis (Sanderson, \u003cspan citationid=\"CR95\" class=\"CitationRef\"\u003e2002\u003c/span\u003e). Relatedly, studies conducting ex-ante policy assessment benefit from a careful aggregation of existing research to map a complex system (Freebairn et al., \u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e2016\u003c/span\u003e). Even reviews of policy research \u0026ndash; such as the Policy Studies Yearbook or Theories of the Policy Process \u0026ndash; involve critical appraisal of evidence (Norman, \u003cspan citationid=\"CR81\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Weible \u0026amp; Sabatier, \u003cspan citationid=\"CR108\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). Methods for evidence synthesis have, therefore, received much attention from policymakers, analysts, and researchers alike.\u003c/p\u003e \u003cp\u003eHowever, several characteristics of the policy sciences make evidence synthesis challenging. First, the exponential growth in policy-relevant research increases resource and time requirements even as time-critical policy advice is more in demand (Bornmann \u0026amp; Mutz, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Larsen \u0026amp; Von Ins, \u003cspan citationid=\"CR67\" class=\"CitationRef\"\u003e2010\u003c/span\u003e; Nunez-Mir et al., \u003cspan citationid=\"CR82\" class=\"CitationRef\"\u003e2016\u003c/span\u003e). Second, evidence is scattered across different policy areas, in academic and grey literature, wherein different terminologies are used to refer to similar phenomena (Goyal \u0026amp; Howlett, \u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e2018\u003c/span\u003e; Saetren, \u003cspan citationid=\"CR94\" class=\"CitationRef\"\u003e2005\u003c/span\u003e). Third, policies have (un)intended consequences on a variety of dimensions \u0026ndash; such as programmatic, process, and political (Marsh \u0026amp; McConnell, \u003cspan citationid=\"CR75\" class=\"CitationRef\"\u003e2010\u003c/span\u003e; Wakabayashi \u0026amp; Kimura, \u003cspan citationid=\"CR106\" class=\"CitationRef\"\u003e2018\u003c/span\u003e) \u0026ndash; all of which should be considered in evidence synthesis. Fourth, the process is complicated further due to different levels of policy analysis, spanning from cross-sectional, micro-level studies of individual policies to time-series, macro-level research on entire policy areas (Esty \u0026amp; Porter, \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e2005\u003c/span\u003e; Jiali, \u003cspan citationid=\"CR61\" class=\"CitationRef\"\u003e1995\u003c/span\u003e; Thow et al., \u003cspan citationid=\"CR103\" class=\"CitationRef\"\u003e2010\u003c/span\u003e; Warner \u0026amp; van Buuren, \u003cspan citationid=\"CR107\" class=\"CitationRef\"\u003e2011\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe field of evidence synthesis has witnessed rapid development and now consists of several methods that facilitate systematic source identification, repeatable evidence appraisal, and transparent reporting. On the one hand, methods such as thematic synthesis (Thomas \u0026amp; Harden, \u003cspan citationid=\"CR102\" class=\"CitationRef\"\u003e2008\u003c/span\u003e), framework synthesis (Carroll et al., \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2011\u003c/span\u003e), scoping reviews (Peters et al., \u003cspan citationid=\"CR88\" class=\"CitationRef\"\u003e2020\u003c/span\u003e), meta-narrative reviews (Wong et al., \u003cspan citationid=\"CR111\" class=\"CitationRef\"\u003e2013a\u003c/span\u003e), and meta-ethnography (France et al., \u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e2019\u003c/span\u003e) enable broad syntheses and critical appraisals of research, but do not seek to inform policy-making per se. On the other hand, methods such as systematic reviews (Pearson et al., \u003cspan citationid=\"CR85\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Petticrew \u0026amp; Roberts, \u003cspan citationid=\"CR90\" class=\"CitationRef\"\u003e2008\u003c/span\u003e), meta-analyses (Barza et al., \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2009\u003c/span\u003e), and umbrella reviews (Fusar-Poli \u0026amp; Radua, \u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e2018\u003c/span\u003e) help synthesize evidence on interventions in a critical, repeatable, and transparent manner. They can, however, be difficult to execute, limited in their ability to acknowledge diversity in evidence, simplistic in their treatment of complexity, and time-consuming (Haddaway et al., \u003cspan citationid=\"CR51\" class=\"CitationRef\"\u003e2017\u003c/span\u003e). Although living systematic reviews (Millard et al., \u003cspan citationid=\"CR78\" class=\"CitationRef\"\u003e2019\u003c/span\u003e) and rapid reviews permit execution in a time-sensitive situation (O\u0026rsquo;Leary et al., \u003cspan citationid=\"CR83\" class=\"CitationRef\"\u003e2017\u003c/span\u003e), they do not address the other shortcomings.\u003c/p\u003e \u003cp\u003eA more configurative approach is necessary to capture the underlying system complexity (Anderson et al., \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2013\u003c/span\u003e). Realist syntheses, which focus on mechanisms of how interventions work (Macura et al., \u003cspan citationid=\"CR74\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Wong et al., \u003cspan citationid=\"CR112\" class=\"CitationRef\"\u003e2013b\u003c/span\u003e), address complexity better (Greenhalgh et al., \u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e2011\u003c/span\u003e). These can shed light on the underlying reasons behind success and facilitate theory-building or theory-testing. As they can also require significant resources, rapid realist reviews have been proposed (Saul et al., \u003cspan citationid=\"CR97\" class=\"CitationRef\"\u003e2013\u003c/span\u003e). Meanwhile, meta-aggregation collates \u0026lsquo;lines of action\u0026rsquo; in available evidence that can inform decision-making in an auditable, reliable, and transparent manner (Hannes \u0026amp; Lockwood, \u003cspan citationid=\"CR53\" class=\"CitationRef\"\u003e2011\u003c/span\u003e). However, these methods, too, provide limited support in addressing diverse conceptual lenses of analyses in policy-relevant evidence.\u003c/p\u003e \u003cp\u003eThis study presents a semi-automated approach to extract, aggregate, map, and analyze causal evidence from policy-relevant literature. To address the challenges outlined above, we combine natural language processing (NLP), causal mapping, and graph analytics. NLP refers to computational techniques for analyzing and representing natural language text (Liddy, \u003cspan citationid=\"CR71\" class=\"CitationRef\"\u003e2001\u003c/span\u003e), and is increasingly used in the policy sciences (El-Taliawi et al., \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Goyal \u0026amp; Howlett, \u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e2019\u003c/span\u003e, \u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Meanwhile, causal mapping facilitates the organization and representation of causal evidence regarding interdependencies among parts of a system (Barbrook-Johnson \u0026amp; Penn, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2022\u003c/span\u003e), but has witnessed limited use in policy analysis. Finally, graph analytics \u0026ndash; a field that is still underutilized for policy research \u0026ndash; can help with systematic investigation of a causal map (Nguyen et al., 2013), a type of a graph.\u003c/p\u003e \u003cp\u003eThis article is structured as follows. In Section 2, we present an approach for semi-automated evidence synthesis. Subsequently, we illustrate the approach by synthesizing evidence on the emissions trading scheme (ETS) to create a causal map (Section 3). In Section 4, we describe and analyze the causal map to demonstrate its suitability for generating policy-relevant insight. Thereafter, we discuss the implications of this research (Section 5) and conclude the article (Section 6).\u003c/p\u003e"},{"header":"2 The potential of semi-automated causal evidence synthesis","content":"\u003cp\u003eCausal evidence synthesis is a multi-step process. The first step is the collation of documents that contain evidence pertaining to a research area, topic, question, or hypothesis. Existing machine learning techniques such as classification and clustering can help in this step (van de Schoot et al., \u003cspan citationid=\"CR104\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). We, however, focus on the remainder of the process.\u003c/p\u003e \u003cp\u003eTypically, the second step is the extraction and appraisal of evidence. While some protocols appraise causality based on research design \u0026ndash; e.g., including only randomized control trials or meta-analyses of these (Burns et al., \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2011\u003c/span\u003e) \u0026ndash; such an approach excludes relevant evidence created through various qualitative, quantitative, and mixed-methods designs. An alternate approach is to extract causal claims from the relevant documents. Such claims might be present in the form of explicit causality (e.g. using causative adjectives, adverbs, or verbs), implicit causality (based on background knowledge and the line of reasoning), inter-sentence causality (cause and the effect are spread across multiple sentences), and embedded causality (a variable is included as a composite effect of one cause and the cause of another effect) (Yang et al., \u003cspan citationid=\"CR114\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Due to the complexity of natural language and the growing volume of policy-relevant evidence, this is challenging even for a subject matter expert. While NLP is unlikely to substitute human intelligence, advances in the field can assist in reducing and redirecting human effort.\u003c/p\u003e \u003cp\u003eThe approaches to causality detection can be broadly classified into (top-down) co-occurrence-based methods and (bottom-up) causal relation extraction methods. Co-occurrence-based methods reduce a large volume of text into core concepts and then identify connections among these concepts (Han et al., \u003cspan citationid=\"CR52\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Kim et al., \u003cspan citationid=\"CR63\" class=\"CitationRef\"\u003e2016\u003c/span\u003e; Son et al., \u003cspan citationid=\"CR101\" class=\"CitationRef\"\u003e2020\u003c/span\u003e). Causal relation extraction methods identify the connections within the text and then aggregate them (Asghar, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2016\u003c/span\u003e; Bach \u0026amp; Badaskar, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2007\u003c/span\u003e; Khoo \u0026amp; Na, \u003cspan citationid=\"CR62\" class=\"CitationRef\"\u003e2006\u003c/span\u003e). The latter is generally more appropriate for evidence synthesis as its bottom-up nature covers even infrequently occurring variables and retains the stated relationships among variables (Barik et al., \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2016\u003c/span\u003e). Causal relation extraction techniques can, in turn, be classified into knowledge-based, statistical machine learning, and deep learning techniques (Yang et al., \u003cspan citationid=\"CR114\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Knowledge-based techniques rely on semantic and syntactic text characteristics, codified as patterns or rules, against which an algorithm can classify input text (e.g. a causative verb-noun pair) (Bui et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2010\u003c/span\u003e). These techniques perform well on simple text with explicit causality, but poorly on text with implicit causality (Beamer et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2008\u003c/span\u003e; Girju et al., \u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e2009\u003c/span\u003e; Yang et al., \u003cspan citationid=\"CR114\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Statistical machine learning identifies distinguishing characteristics based on labelled training data to classifying data, using techniques such as Bayesian inference or decision trees. Statistical machine learning can handle implicit causality well, but it suffers from low portability, i.e., poor performance when the test data is dissimilar from the training data (Asghar, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2016\u003c/span\u003e; Pakray \u0026amp; Gelbukh, \u003cspan citationid=\"CR84\" class=\"CitationRef\"\u003e2014\u003c/span\u003e; Zhao et al., \u003cspan citationid=\"CR115\" class=\"CitationRef\"\u003e2016\u003c/span\u003e). Finally, deep learning techniques utilize neural networks architectures for causal relationship extraction, which perform well even on complex text (Yang et al., \u003cspan citationid=\"CR114\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). In addition, they can leverage \u0026lsquo;transfer learning\u0026rsquo; to enhance portability (Beltagy et al., \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Devlin et al., \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2018\u003c/span\u003e; Kyriakakis et al., \u003cspan citationid=\"CR66\" class=\"CitationRef\"\u003e2019\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe next step is to make the evidence comparable through some form of homogenization or normalization. In case of a meta-analysis, a regression coefficient might be converted into a partial correlation coefficient to make the input comparable (Hansen et al., \u003cspan citationid=\"CR54\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). Whereas less relevant for qualitative evidence, a challenge for policy-relevant analysis is that the source material can span several (inter)disciplinary fields, with different vocabulary to refer to the same or similar phenomena (Goyal \u0026amp; Howlett, \u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e2018\u003c/span\u003e; Saetren, \u003cspan citationid=\"CR94\" class=\"CitationRef\"\u003e2005\u003c/span\u003e). For example, the use of \u0026lsquo;institution\u0026rsquo; in governance studies might be synonymous with the use of \u0026lsquo;policy instrument\u0026rsquo; in policy studies. While not a substitute for expert assessment, NLP techniques for measuring lexical or semantic similarity can be useful. Lexical similarity is measured using a dictionary-based approach (Cruanes et al., \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e2012\u003c/span\u003e), which are easier to implement, but do not consider the context in which a term is used. In contrast, semantic similarity is measured such that the distance between terms (or other units, such as sentences, paragraphs, documents) represents the likeness of their meaning (Liu et al., \u003cspan citationid=\"CR72\" class=\"CitationRef\"\u003e2007\u003c/span\u003e). For complex text, semantic similarity, or a combination of lexical and semantic similarity, is likely to be a more robust approach (Inan, \u003cspan citationid=\"CR58\" class=\"CitationRef\"\u003e2020\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThereafter, the evidence needs to be aggregated. For a quantitative evidence synthesis, a weighted mean effect size for the cause-effect relationship may be computed (Hansen et al., \u003cspan citationid=\"CR54\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). However, as policies often have consequences across many dimensions \u0026ndash; program, process, and politics (McConnell, \u003cspan citationid=\"CR77\" class=\"CitationRef\"\u003e2010\u003c/span\u003e) \u0026ndash; that vary over time (Compton \u0026amp; \u0026rsquo;t Hart, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Goyal, \u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e2021\u003c/span\u003e), it is essential to consider multiple effects for any cause. Further, relevant evidence might be available from diverse contexts varying in levels, geographies, scales, and time periods, making it important to highlight the relevant effects of contextual variables and internal mechanisms on the causal chain. In addition, the research designs of the sources could range from argumentative work to small-n case studies; medium or large-n observational, quasi-experimental, and experimental research; simulation modelling; and co-design or participatory knowledge. All of this lends itself to a more qualitative synthesis, for which in-depth information is usually summarized as tables and conceptual diagrams or synthesized in text. Collating and visually presenting evidence as in systems science can add value here.\u003c/p\u003e \u003cp\u003eSystem mapping facilitates collation of existing evidence to answer a specific question, identify knowledge clusters, or articulate a knowledge gap (James et al., \u003cspan citationid=\"CR59\" class=\"CitationRef\"\u003e2016\u003c/span\u003e). Similarly, systematic evidence mapping helps summarize, query, and visualize evidence, for example, using a database and/or a graphical representation (Peters et al., \u003cspan citationid=\"CR87\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Wolffe et al., \u003cspan citationid=\"CR110\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). Causal mapping is likely to be especially useful for causal evidence synthesis. It combines causal chain analysis with the mapping of complex interrelationships, thereby facilitating collation of existing evidence or expert knowledge about (dynamic) causal interdependencies between parts of a system (Ackermann \u0026amp; Alexander, \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2016\u003c/span\u003e). This allows to answer specific questions about effects of interest, to identify knowledge clusters, or articulate a knowledge gap (James et al., \u003cspan citationid=\"CR59\" class=\"CitationRef\"\u003e2016\u003c/span\u003e). Causal mapping is well-suited for combining analysis of different scopes and allows for easy expansion when new information is introduced (Eden et al., \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e1992\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eAs a causal map is a type of a directed knowledge graph, it can be investigated using graph analytics, broadly used here to refer to techniques in causal chain analysis, graph theory, and network analysis. Illustratively, topographic analysis creates insight based on the structural layout of the causal map. An assessment of the strength of the linkages, for example, indicates the number of studies providing evidence on specific cause-effect relationships (Montibeller \u0026amp; Belton, \u003cspan citationid=\"CR79\" class=\"CitationRef\"\u003e2006\u003c/span\u003e). This can shed light not only on interventions that are supported by much evidence, but also on parts of the system that may warrant further investigation. Similarly, effect trees \u0026ndash; a collection of downstream variables affected by a given factor, and cause trees \u0026ndash; a collection of upstream variables that influence a factor \u0026ndash; can help analyze the cause and consequences, respectively, of an intervention (Eden et al., \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e1992\u003c/span\u003e). Centrality analysis can provide insight into the likely importance of a variable \u0026ndash; either perceived or real \u0026ndash; based on whether, for example, it has several linkages to other variables (i.e., high degree centrality) or lies on the shortest path between several other variables (i.e., high betweenness centrality). Some of the other possibilities for investigating a causal map are causal inference analysis and causal loop analysis. Causal inference refers to the process of determining the causal impact of one factor on another (Axelrod, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e1976\u003c/span\u003e), either the partial effect (the causal impact of one variable on another along a specific path), or the total effect (the overall impact of one variable on another taking all paths into consideration). Causal loop analysis involves the identification and description of feedback mechanisms within a causal map i.e. a positive or negative feedback within (a part of) the system, resulting in reinforcing or balancing behavior, respectively. Apart from aiding causal evidence synthesis, causal maps can also inform subsequent policy analysis, for example, in the form of Bayesian belief analysis or simulation modelling (Pullin et al., \u003cspan citationid=\"CR91\" class=\"CitationRef\"\u003e2016\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eAn overview of the semi-automated approach is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. Beginning with a selection of relevant text, causal relations are extracted, similar factors are grouped, and an aggregate causal map is built. The map is then analyzed to derive policy-relevant insights. It is important to note that a variety of different algorithms can be used to achieve the desired outputs of each step. In this way algorithm selection can be made to best suit the given application, and the results of the method can keep pace with advancements in NLP.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e"},{"header":"3 Synthesizing policy-relevant evidence: An illustration using the Emissions Trading Scheme","content":"\u003cp\u003eWe illustrate the process of synthesizing policy-relevant evidence and creating a causal map through the semi-automated approach, using the case of the ETS.\u003c/p\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Emissions trading schemes\u003c/h2\u003e \u003cp\u003eETS, are policy instruments that aim to impose a cost on the emission of greenhouse gasses (GHGs). This policy involves setting a \u0026lsquo;cap\u0026rsquo; on certain types of emissions, regulating bodies then divide the total allowable emission quantity into tradeable \u0026lsquo;allowances\u0026rsquo; that are auctioned or allocated to the entities covered by the system. Polluters can buy and sell allowances but must surrender a quantity equivalent to their respective emissions at the end of pre-defined periods. Entities for whom abatement is relatively cheap have a financial incentive to reduce emissions, and entities for whom abatement is relatively expensive have the option to purchase allowances to satisfy their obligation. In this way, ETS provide a mechanism to reduce emissions, to specified levels, whilst encouraging the most cost-efficient abatement (ICAP, \u003cspan citationid=\"CR57\" class=\"CitationRef\"\u003e2022b\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eEvidence-synthesis from ETS policy analysis literature is encumbered by many of the limitations discussed earlier. First, there is an enormous literature base. Google Scholar returns over 23,100 results for a search of \u0026ldquo;ETS\u0026rdquo; and \u0026ldquo;policy analysis\u0026rdquo;. The terminology employed for discussing ETS performance varies across these studies. ETS policy studies often only examine a single performance dimension despite compelling analysis from programmatic, political and process perspectives. And finally, analysis is conducted at various levels of scope from case studies to reviews seeking to summarize the wholesale abatement impact of carbon pricing. As such, this field was selected to demonstrate the method.\u003c/p\u003e \u003cp\u003eWhile several possibilities exist for compiling relevant source material, for this illustration we identified meta-reviews of ETS and selected constituent articles. This allowed us to compare the insights created by our approach against those in the meta-reviews. From the 14 meta-reviews on ETS identified in our search, we selected two: Green (\u003cspan citationid=\"CR49\" class=\"CitationRef\"\u003e2021\u003c/span\u003e), due to its explicit ex-post and quantitative direction encompassing studies from all major ETS jurisdictions, and Schmalensee \u0026amp; Stavins, (\u003cspan citationid=\"CR98\" class=\"CitationRef\"\u003e2017\u003c/span\u003e) due to its specific discussion on political considerations and system design factors, which reflect more process and political perspectives. Collating the papers within both reviews and removing those deemed irrelevant yielded a final total of 28 source papers, as shown in Table \u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003eA1\u003c/span\u003e (\u003cspan refid=\"Sec14\" class=\"InternalRef\"\u003eAppendix\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Extracting cause-effect relationships\u003c/h2\u003e \u003cp\u003eThe first stage of our approach involves examining each sentence of the source material to determine whether it exhibits causality, and if so, the cause factor, effect factor, and the direction of the causal relation. Take, for example, the causal sentence \u0026ldquo;the higher emissions allowance price caused a decrease in coal power generation\u0026rdquo;. Here, \u0026lsquo;emissions allowance price\u0026rsquo; is the causal factor, \u0026lsquo;coal power generation\u0026rsquo; is the effect factor, the direction of the relationship is negative (i.e., an increase in the former results in a reduction in the latter). To semi-automate this process, we use a deep learning algorithm \u0026lsquo;Self-attentive BiLSTM-CRF wIth Transferred Embeddings\u0026rsquo; or SCITE, Li et al. (\u003cspan citationid=\"CR70\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). The algorithm was chosen given that it is open source, has an explicit focus on causal relations and performs highly on benchmark datasets.\u003c/p\u003e \u003cp\u003eTo derive causal relations using SCITE, the 28 documents were processed in three stages. First, we selected only the abstract, results, discussion, and \u003cspan refid=\"Sec13\" class=\"InternalRef\"\u003econclusion\u003c/span\u003e sections of the articles. The literature review and methods were excluded because these often-returned irrelevant methodological links, and relations suggested in other literature instead of findings of that article. Next, the text was automatically segmented into sentences, cleaned (removing non-alphanumeric characters, separation of punctuation etc.) and further segmented into words. Subsequently, \u0026lsquo;layered embeddings\u0026rsquo; \u0026ndash; necessary for the algorithm to learn a representation of the input data (Levy \u0026amp; Goldberg, \u003cspan citationid=\"CR69\" class=\"CitationRef\"\u003e2014\u003c/span\u003e) \u0026ndash; were generated for each sentence (Akbik et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). The sentences and corresponding embeddings were processed through a pre-trained SCITE model resulting in a list of sentences deemed causal, and their suggested cause-effect relationships. An example causal relation returned by SCITE is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eIn all, our data consisted of 4542 sentences that were cleaned and provided as input to SCITE. Of these, 317 were deemed as causal by the algorithm. The raw output, however, did not indicate the direction of the cause (i.e., positive or negative) and included a high level of inaccuracy. This necessitated a manual review of each SCITE output to determine the true causal relations, causal pairs, and direction. This stage was also necessary to remove irrelevant causal sentences. Ultimately, 154 sentences were verified as causal and relevant. These sentences contained 284 causal pairs. Our validation of this step revealed that the pre-trained model of SCITE achieved 84% precision and ~\u0026thinsp;38% recall for our data. Despite the current manual review stages required, this semi-automatic approach reduced the time taken to extract relations from a single paper from ~\u0026thinsp;3 hours to ~\u0026thinsp;20 minutes.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Clustering causal links\u003c/h2\u003e \u003cp\u003eNext, we concatenated the individual causal pairs, their contributing sentence, cause-effect factors, and relationship direction. To ensure backward traceability, we provided a unique cause-effect id (in the form [article number]-[sentence number][causal pair letter]) to each causal pair, which highlights the prevalence of similar causal pairs. For example, when talking about the use of coal power, one causal pair may include the factor \u0026lsquo;coal power generation\u0026rsquo; whereas another \u0026lsquo;coal utilization\u0026rsquo;. Including both factors would quickly cloud the causal map with repetitive links. To address this issue, semantic clustering can help in grouping factors that have the same meaning. This would reduce the number of (duplicated) factors and increase the (legitimate) interconnections, resulting in a more cohesive causal map.\u003c/p\u003e \u003cp\u003eWhile various NLP algorithms are applicable for this task, for this project we used the following process. First, \u0026lsquo;sentence embeddings\u0026rsquo; were generated using SBERT (Reimers \u0026amp; Gurevych, \u003cspan citationid=\"CR93\" class=\"CitationRef\"\u003e2019\u003c/span\u003e) to capture the semantic meaning of each factor phrase present in the causal pairs. These were then grouped using a density-based clustering algorithm, DBSCAN (Ester et al., \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e1996\u003c/span\u003e), with a cosine distance metric and a minimum cluster size of 2. This approach was chosen because it is well-suited to manage uneven cluster sizes and outliers, which are expected in the data. One overarching factor term is then chosen for each cluster and used to overwrite the factors present in that cluster where they occur in causal pairs (see Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eAn example of clustered causal pairs. Factors that have been clustered are shown in capital case.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eId\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRaw cause effect pairs\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eClustered cause effect pairs\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2-028A\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e[european level commitments, +, 20-20-2010 targets]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e[European level commitments, +, 2020 EU CLIMATE ENERGY PACKAGE]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2-028B\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e[20-20-2010 targets, +, renewable energy directive]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e[2020 EU CLIMATE ENERGY PACKAGE, +, RENEWABLE ENERGY DIRECTIVE]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2-028C\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e[20-20-2010 targets, +, energy efficiency directive]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e[2020 EU CLIMATE ENERGY PACKAGE, +, energy efficiency directive]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2\u0026ndash;029\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e[renewable energy directive, +, renewable energy utilisation]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e[RENEWABLE ENERGY DIRECTIVE, +, RENEWABLE ENERGY UTILISATION]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2-031A\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e[carbon price, +, fuel-switching]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e[EMISSIONS ALLOWANCE PRICE, +, FUEL-SWITCHING]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable A1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThe source articles on the ETS used for the synthesis.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eContributing review paper\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eIndex\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eIndividual papers\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"18\" rowspan=\"19\"\u003e \u003cp\u003e(Green, \u003cspan citationid=\"CR49\" class=\"CitationRef\"\u003e2021\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(B. Anderson \u0026amp; Di Maria, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2011\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Gloaguen \u0026amp; Alberola, \u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e2013\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Arimura \u0026amp; Abe, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2021\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Bayer \u0026amp; Aklin, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2020\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Bel \u0026amp; Joseph, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2015\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Cullenward, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2014\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Wagner et al., \u003cspan citationid=\"CR105\" class=\"CitationRef\"\u003e2014\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Jaraite-Kažukauske \u0026amp; Di Maria, \u003cspan citationid=\"CR60\" class=\"CitationRef\"\u003e2016\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Egenhofer et al., \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e2011\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Ellerman et al., \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e2016\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Kotnik et al., \u003cspan citationid=\"CR64\" class=\"CitationRef\"\u003e2014\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Fell \u0026amp; Maniloff, \u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e2018\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Dechezlepr\u0026ecirc;tre et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2018\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Ellerman \u0026amp; Buchner, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2008\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Martin \u0026amp; Saikawa, \u003cspan citationid=\"CR76\" class=\"CitationRef\"\u003e2017\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e16\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Ellerman \u0026amp; McGuinness, \u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e2008\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Murray \u0026amp; Maniloff, \u003cspan citationid=\"CR80\" class=\"CitationRef\"\u003e2015\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Petrick \u0026amp; Wagner, \u003cspan citationid=\"CR89\" class=\"CitationRef\"\u003e2014\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Wakabayashi \u0026amp; Kimura, \u003cspan citationid=\"CR106\" class=\"CitationRef\"\u003e2018\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"8\" rowspan=\"9\"\u003e \u003cp\u003e(Schmalensee \u0026amp; stavins, \u003cspan citationid=\"CR98\" class=\"CitationRef\"\u003e2017\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Sijm et al., \u003cspan citationid=\"CR100\" class=\"CitationRef\"\u003e2011\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Hibbard et al., \u003cspan citationid=\"CR55\" class=\"CitationRef\"\u003e2015\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Wing \u0026amp; Kolodziej, \u003cspan citationid=\"CR109\" class=\"CitationRef\"\u003e2008\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e23\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Ranson \u0026amp; Stavins, \u003cspan citationid=\"CR92\" class=\"CitationRef\"\u003e2012\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e24\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Ellerman \u0026amp; Buchner, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2007\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Kruger et al., \u003cspan citationid=\"CR65\" class=\"CitationRef\"\u003e2007\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Convery \u0026amp; Redmond, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2007\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e27\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(Sartor et al., \u003cspan citationid=\"CR96\" class=\"CitationRef\"\u003e2014\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e28\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e(L\u0026ouml;fgren et al., \u003cspan citationid=\"CR73\" class=\"CitationRef\"\u003e2015\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe initial collation of causal pairs resulted in the identification of 300 cause/effect factors in our dataset. Using semantic clustering, 230 of these cause/effect factors were grouped into 49 clusters. The remaining 70 factors were deemed unique. From the clustered factors, we could prune duplicate cause-effect pairs leading to 119 unique cause-effect pairs.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e3.4 Building an aggregated causal map\u003c/h2\u003e \u003cp\u003eThe next step is to generate the aggregate causal map. Using the clustered cause-effect pairs, individual factors can be added in the map as nodes and their causal links included as the edges between them. The causal map grows in complexity as more factors appear and connections between structures emerge.\u003c/p\u003e \u003cp\u003eWhile based on the causal links identified in earlier stages, this map generation stage inevitably involves coder interpretation. Despite clustering factors, additional grouping may be warranted. Implicit structures are also likely to become more evident, the inclusion of which can allow for richer synthesis. For example, many relations discussed the impact of the free allocation of emission allowances and subsequent sales on the profits of power-generating firms. Initially, this behavior is captured by various links to and from \u0026lsquo;firm profits\u0026rsquo;. The same behavior may instead be captured by a stock-flow structure, whereby revenues are an inflow to profit and costs an outflow. This representation provides a clearer picture of the underlying dynamics. Figure\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e demonstrates how including the implicit stock-flow structure allows demarcation of factors influencing revenues and costs and their influence on profits.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eAnother issue that can arise in this process is that of \u0026lsquo;intermediate variables\u0026rsquo;, whereby one causal chain depicts a connection from A \u0026agrave; C and another from A \u0026agrave; B \u0026agrave; C. In such cases, it is often unclear whether the link A \u0026agrave; C implies the existence of intermediate factor B, if it is unaware of factor B, it considers B irrelevant, or it suggests that A \u0026agrave; B \u0026agrave; C may only represent one of multiple paths to C, thereby partially contributing to the causal impact of A on C. Addressing this issue requires interpretation. In situations where an intermediate variable is suggested but not explicitly mentioned by a causal link, the analyst can refer to the contributing text segment to gain more context. In other cases, support cannot be found for an intermediate link, and an additional path is added to the causal map alongside the intermediate path, i.e., including A \u0026agrave; B \u0026agrave; C alongside A \u0026agrave; C.\u003c/p\u003e \u003cp\u003eWhat should be apparent is that model generation is an iterative process. As the map evolves, implicit structures emerge, and intermediate variables appear, frequent reorganization is warranted. There is no objectively complete causal map, rather it should be refined until all causal links have been incorporated, and the overall structure can be used to garner relevant insights. Connection to contributing text segments is also necessary to provide additional context when examining the causal map. By labelling the arrows and providing a reference table of supporting causal IDs, readers can examine each causal link in more detail, if desired.\u003c/p\u003e \u003cp\u003eOur iterative process produced the final causal map shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e.\u003c/p\u003e \u003c/div\u003e"},{"header":"4 The synthesized evidence: Probing the causal map on the emissions trading scheme","content":"\u003cp\u003eIn this section, we briefly describe the causal map and demonstrate how it can be further analyzed to create policy-relevant insight.\u003c/p\u003e\n\u003ch2\u003eDescription of the aggregated ETS causal map\u003c/h2\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eBefore describing the analysis of the causal map, it is worth describing some core structural elements of the graph.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e4.1.1 Emissions allowance price inducing GHG emission reductions.\u0026nbsp;This structure concerns the variables linking \u0026lsquo;emissions allowance price\u0026rsquo; and \u0026lsquo;GHG emissions\u0026rsquo;. There is a clear influence of allowance price on emissions via fuel switching, elaborated by the coal-to-gas price ratio factor. This is consistent with the idea that allowance price will impact the cost of coal more than gas (as coal is more carbon-intensive), thereby inducing fuel switching. The link from allowance price to emissions via cost pass-through/production cost, energy sale price, and energy demand captures the idea that price increases are passed on to consumers who then reduce their energy demand as a result. Additionally, there is the path through emissions reduction investment, as higher allowance prices encourage polluters to enact measures to reduce their emissions. The map also conveys the expected impact that changes in any of these factors would have, through the sign of linkage. For example, a lower allowance price would: i) reduce the coal-to-gas price ratio and curtail fuel switching; ii) limit the energy sale price (due to lower production cost), thereby having lesser influence on energy demand reduction; and iii) lessen the emissions reduction investment.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e4.1.2 Emissions leakage diminishing ETS efficacy.\u0026nbsp;This structure concerns \u0026lsquo;emissions leakage\u0026rsquo; and its connections, representing the factors contributing to leakage, and the different ways in which leakage impacts the system. One can observe the direct positive link from \u0026lsquo;ETS\u0026rsquo; regulation to \u0026lsquo;emissions leakage\u0026rsquo; to \u0026lsquo;GHG emissions\u0026rsquo;, reflecting the idea that ETS regulation incentivizes polluting activity to relocate outside of regulatory jurisdictions, so avoiding ETS-induced emissions reductions. Examining the structural location of emissions leakage also reveals its negative role in important causal chains. For example, consider again the path from allowance price through cost pass-through/production cost, energy demand to GHG emissions. This causal chain captures the idea that a higher allowance price reduces emissions. However, as both cost pass-through and energy production cost have a positive relationship with emissions leakage (which in turn has a negative relationship with GHG emissions), the effect of allowance price on emissions is mitigated, to some extent, by emissions leakage.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e4.1.3 Free allocation of allowances leading to windfall profits.\u0026nbsp;This concerns the \u0026lsquo;firm profit\u0026rsquo; stock-flow structure and its connections with \u0026lsquo;free allocation of allowances\u0026apos;. This reflects behavior, particularly in the earlier phases of the EU ETS, whereby firms profited from the sale of freely distributed emissions allowances. The basic mechanism is clearly present with a path from free allocation to sale of allowances, revenues and, thereby, firm profit. Several factors elaborate the causes of free allocation, including relocation risks, competitiveness concerns and political demand for new entrant provisions. Additionally, the impacts of the firm \u0026lsquo;windfall profits\u0026rsquo; can be seen, i.e., the reduction in competitiveness concerns, and an increasing regulatory threat concerning these profits.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eAnalysis of the aggregated ETS causal map\u003c/h2\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAnalysis of the causal map can be conducted using topographic analysis, causal inference analysis, and causal loop analysis. While analysis has revealed interesting insights, these results are not exhaustive, but should typify potential insights.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e4.2.1. Topographic analysis. The most supported connections on the map are not surprising. Links #34, #58 and #59 are each supported by nine articles and relate to obvious links between central factors in the system. This high degree of shared knowledge implies that these connections are relevant in explaining the ETS system. Conversely, #71 and #61 have only two articles supporting them. The apparent lack of knowledge regarding these relationships may suggest that further research is warranted.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eConsider the cause-and-effect trees of ETS system linkage. If an analyst were exploring the impact of linkage between ETS systems, the effect tree would highlight that linkage is expected to reduce compliance costs and lessen price volatility because of increased market thickness, however amongst the numerous other effects are also capital flows between systems and associated negative public perception. The analyst could use this knowledge to recommend measures to mitigate the negative downstream effects, sever the negative effect branches, or introduce reinforcing branches that promote the desired behavior.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFigure 6 shows a network graph of the causal map, with nodes sized according to degree centrality and shaded according to betweenness. GHG emissions, for example, is positioned at the end of many causal paths contributing to its high degree. Allowance price, allowance demand, pass-through and firm profits have the highest betweenness. This reflects their position in the key causal paths within the system. Allowance demand has much higher betweenness than centrality indicating that it is important in determining system behavior, but that its influence is fairly narrow. This is consistent with the idea that allowance demand has a large impact on allowance price but does not directly influence other factors outside of this mechanism.\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003cp\u003e4.2.2. Causal inference analysis. Consider the path from \u0026lsquo;emissions allowance price\u0026rsquo; to \u0026lsquo;coal-to-gas price ratio\u0026rsquo;, \u0026lsquo;fuel switching (coal to gas)\u0026rsquo;, and \u0026lsquo;GHG emissions\u0026rsquo;. The partial effect is negative, consistent with the idea that a greater allowance price will lead to a price disparity between coal and gas fuel sources which in turn leads to fuel switching from coal to gas, thereby reducing emissions. Article 2 supported the link between price ratio and fuel switching but did not include the connection between price-ratio and emissions allowance price. Article 16 did mention this relation but the link between price ration and fuel-switching was omitted. It is only by combining the causality of these two articles that the entire causal chain from allowance price to emissions becomes apparent. If a policymaker sought to encourage fuel-switching, an understanding of the aggregated path shows that increasing the allowance price would likely contribute to this end.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eLooking instead at the total effect, there is an obvious path from \u0026lsquo;cost pass-through\u0026rsquo; to \u0026lsquo;GHG emissions\u0026rsquo; via \u0026lsquo;energy sale price\u0026rsquo;, \u0026lsquo;energy demand\u0026rsquo;, and \u0026lsquo;energy generation emissions\u0026rsquo; to \u0026lsquo;GHG emissions\u0026rsquo;, which has a negative partial effect indicating that greater cost pass-through would lead to emissions reductions. This is consistent with the idea that passing the cost on to consumers would reduce consumption and associated emissions. Such an effect is well-studied and understood, with aspects of this path covered by 15 articles. However, taking instead the path from \u0026lsquo;cost pass-through\u0026rsquo; to \u0026lsquo;emissions leakage\u0026rsquo; to \u0026lsquo;GHG emissions\u0026rsquo; yields a positive partial effect, consistent with the idea that greater pass-through costs contribute to greater leakage and associated net emissions. The relationship between pass-through and leakage is suggested only in Article 27. In this case, the total effect between these two factors is thus undetermined. If targeting emissions reductions, a policymaker may examine the degree of cost pass-through. By only considering the energy sale price pathway, they may conclude that encouraging cost pass-through would be fruitful. Examining the total effect of cost pass-through on emissions would instead reveal that emissions leakage can mitigate the reduction effect somewhat.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e4.2.3 Causal loop analysis. Consider the balancing loop between emissions allowance price and GHG emissions (Figure 6). Emissions reductions lessen allowance demand, there is a potential dampening effect on the allowance price if allowance supply cutbacks do not keep pace with these reductions, which in turn impacts emissions. Indeed a similar issue was experienced in the first phase of the EU ETS, an allowance oversupply contributed to an allowance price collapse (Schmalensee \u0026amp; Stavins, 2017). If future allowance supply caps are not set sufficiently low, then successful abatement efforts may reduce the allowance price thereby inducing a balancing effect on GHG emissions. This is undesirable when the goal is to maximize abatement. Consideration of this causal loop indicates that, alongside stringent allowance caps, a price collar could work to mitigate this issue.\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003ch2\u003eComparisons with the meta-reviews\u003c/h2\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIt is useful to examine whether the causal map captures the insights of the meta-reviews from which the source material was selected. Our approach performs well in the identification of granular features contributing to high-level behavior.\u0026nbsp;Green (2021), for example, notes the effect of the ETS on fuel switching as well as the (modest) contribution of fuel switching, energy efficiency, and reduced fuel consumption on GHG emissions, both of which are captured in the causal map and subsequent analysis. Further,\u0026nbsp;Green\u0026nbsp;discusses the mitigating role of consistently low carbon price on the impact of ETS on GHG emissions, along with the issues associated with offset credits. By tracing the effect of reducing allowance price, its negative impact on GHG emissions can clearly be seen. Looking at the \u0026lsquo;access to external offset credits\u0026rsquo; factor, it can be seen that issues arise relating to market confidence and financial risks, which can negatively impact allowance price. However, our present approach does not reflect the counterfactual and quantitative inferences of this paper, nor how emission reductions vary by sector. Such insights are not visible in the current map because the factors are not sub-categorized according to their respective sectors.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eSchmalensee \u0026amp; Stavins (2017) do not present quantified insights, but rather discuss the performance of various ETS systems qualitatively. They explain the potentially large revenues that can be generated through allowance auctions, highlight the importance of free allowance allocations in gaining political support for ETS policy and note that these allocations are motivated by concerns of adverse competitiveness impacts. Within the causal map, all these structures are apparent. They further explain the importance of reducing price volatility to facilitate emission abatement, noting how financial conditions led to price instability. This, too, is interpretable from the causal map. First, the adverse impacts of the financial crisis can be seen through its reduction of allowance demand and the subsequent impact on allowance price. Through examination of the various causal paths from allowance price to GHG emissions, it can be interpreted that fluctuating price would in turn fluctuate emission abatement, although there is no explicit connection with the price volatility factor. The insights from this review that are not apparent in the causal map relate to the reflections on the differences in performance of different ETS implementations. For example, how carbon leakage is particularly concerning for subnational systems. Given that the causal map aggregates insights from across the various ETS systems, the differences in behavior across systems is not evident.\u003c/p\u003e"},{"header":"5 Discussion","content":"\u003cp\u003eWe have presented a novel, semi-automated, NLP approach to extract causal statements from policy analysis literature, aggregate them into a causal map of policy behavior, and derive policy-relevant insights. The results show that the method was able to address four shortcomings of state-of-the-art approaches for evidence synthesis for evidence-informed policy making as illustrated for ETS.\u003c/p\u003e \u003cp\u003eFirstly, it provides a configurative approach to reviewing and synthesizing available documented causal evidence. This allows analysts to better capture complex mechanisms and interactions underlying observable policy effects and to understand its effectiveness within the policy system. Secondly, the semi-automated approach enables an analyst to do so with significantly less effort than traditional review and evidence syntheses methods, at high levels of accuracy. Using NLP is relatively easy and less time-consuming than manual text review, reducing the time to derive causal relations by a factor of 10, from ~\u0026thinsp;3 hours to ~\u0026thinsp;20 minutes per article. As a result, a greater number of sources can be considered with the same resources, increasing the evidence base and reducing potential biases arising from a more limited or narrow evidence base. The high precision achieved by the relation extraction algorithm also supports that a high-quality analysis can be obtained. Thirdly, the clustering algorithm helps with integrating policy evidence that is scattered across policy issues or areas. Lastly, the proposed approach can harmonize disconnected information from various levels of scope, performance perspectives, and taxonomies while also incorporating upstream and downstream effects into one comprehensive causal model. In combination, this equips analysts and policy makers with a more systemic understanding of a policy area beyond the direct cause-effect inferences that are typically obtained from traditional means of evidence synthesis.\u003c/p\u003e \u003cp\u003eAs demonstrated for ETS policy, the derived causal map captures most of the insights obtained from manual evidence syntheses as presented in the review articles by Schmalensee \u0026amp; Stavins (\u003cspan citationid=\"CR98\" class=\"CitationRef\"\u003e2017\u003c/span\u003e) and Green (\u003cspan citationid=\"CR49\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). The comparison has highlighted some strengths and weaknesses of this causal map. The strengths include that the generated causal map can faithfully represent the granular features contributing to ETS system behavior and that most policy behavior as highlighted in the reviews is captured. The map performs well in distilling the factors and dynamics present in the disparate source material that are not easily obtained nor communicated otherwise. This can supplement traditional methods in presenting an explicit, parsimonious, aggregated systems perspective. The shortcomings of this representation relate to how well contextual information and counterfactuals are conveyed. Quantified insights and insights by sector were not represented explicitly in this causal map, however, they could be uncovered by referencing the contributing text segments. The map also overlooked some important counterfactual findings which proved to be important in one review paper.\u003c/p\u003e \u003cp\u003eDespite these promising results, several limitations remain that should be addressed in future work to ensure high quality analysis and to realize the potential of this approach to evidence-informed policy. A major limitation concerns the low recall (~\u0026thinsp;38%) achieved by the causal extractions algorithm we used. A consequence being that relevant causal links contributing to system behavior were probably missed. The low recall is likely explained by the data sources used (and the complexity of cause-effects therein, relative to the training data) rather than an inherent issue of the algorithm, which has been demonstrated to achieve recall rates of up to 86% on benchmark data sets (Li et al., \u003cspan citationid=\"CR70\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Addressing this shortcoming, however, is complicated by the sparsity of training data (Asghar, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2016\u003c/span\u003e). Nevertheless, the capabilities of deep learning methods for causal relation extraction are advancing rapidly (Yang et al., \u003cspan citationid=\"CR114\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Strategies to overcome existing limitations include utilizing increasingly more capable NLP-based causality extraction algorithms. Additionally, staged model building can help by starting out from a basic model that reflects widely accepted cause-effect relations, building the system map further by adding cause-effect relations from literature.\u003c/p\u003e \u003cp\u003eAnother limitation concerns the inability of the causal relation extraction algorithm to distinguish between hypothesized and empirically substantiated cause-effects, while also lacking consideration for counterfactuals. Recent developments using adversarial training for causality extraction models may be able to address this (Feder et al., \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). A related challenge is the lack of a well-specified context within which the extracted cause-effect relations are deemed valid. It is possible to uncover additional contextual information by analyzing the underlying textual information present in the reference table, before creating the causal map. However, this is inconvenient especially when new information is collected iteratively, requiring manual updates of the map as regards link proximity and clustering, or other attributes to facilitate map interpretation. Again, developments in NLP may soon overcome these limitations. One promising area of development in this regard is word embedding techniques to facilitate contextual mapping of variables in a map (Pelevina et al., \u003cspan citationid=\"CR86\" class=\"CitationRef\"\u003e2017\u003c/span\u003e; Selva Birunda \u0026amp; Kanniga Devi, \u003cspan citationid=\"CR99\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Another area concerns recent advances in automating the workflow from causal relationship extraction via relations table building to creating a visual causal map (e.g. building onto Ancin-Murguzur \u0026amp; Hausner, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). This would reduce the need for iterative updating of the reference table and map, while facilitating interactive exploration by analysts and policy makers. Similarly, moving from semi-automation towards full automation would make it possible to analyze orders of magnitude more articles which would be necessary to achieve full coverage of a policy issue or area of interest with thousands of relevant publications as is the case for the ETS.\u003c/p\u003e \u003cp\u003eFurther to these important areas for improvement, future research may seek to use our approach to compare and contrast the causal maps that can be extracted from different sources and fields of evidence generation. This can help to expose conflicting information about causal paths from different sources of evidence. A directed network map representation allows to gauge contributions of various factors within and across causal chains, including potential for weighting of links or factors e.g., by frequency of mentioning, source type, soundness of the empirical basis, or authoritativeness of the source. The directed map representation furthermore provides additional analytic capacity. This includes topographic analyses using graph-theory based concepts to identify the most central factors as system levers or barriers to effective policy. Alternatively, causal reinforcing or balancing factors can be identified and explored, which is not commonly done in ETS policy analysis. In a broad field such as ETS policy, it is challenging to keep an overview of the many policies and their effects, both individually and in concert. Here, a causal map presentation allows to either zoom in to evaluate a specific policy, or to zoom out and identify feedback with other policies that may enhance or limit their effectiveness. It may also guide empirical validation of the key cause-effects identified. Beyond that, their expansion to system maps (i.e., that include possible policy actions, policy effects, and contextual variables that would influence the cause-effect pathway, see e.g., Enserink et al., 2022) would provide an ideal foundation for model-supported exploration of policy behavior and policy system change over time (e.g., using system dynamics or agent-based modelling). Altogether, these would result in a more comprehensive and complexity-proof evidence-base for policy evaluation and development.\u003c/p\u003e"},{"header":"6 Conclusion","content":"\u003cp\u003eEvidence-based or evidence-informed policy relies on the ability to effectively summarize, synthesize, and mobilize knowledge for the policy process, yet characteristics of policy sciences as a field make evidence synthesis challenging in practice. The exponential growth in policy research significantly increases the resources necessary for evidence synthesis, evidence is often scattered across different policy areas and employing distinct terminology; policy impacts can be measured in a variety of dimensions, such as programmatic, process, and political, and at different levels from micro-level studies of individual policies to macro-level research on entire policy areas - all of which need to be taken into account when synthesizing evidence.\u003c/p\u003e \u003cp\u003eTo address these shortcomings, this article has introduced a novel analysis method that can semi-automatically derive and aggregate causal relations from policy analysis literature into a causal map of policy effects. Applying this method to a collection of 28 ETS literature sources produced a causal map consisting of 159 unique causal links. Evaluation of this result has demonstrated that the approach allows analysts to better capture complex mechanisms and interactions underlying observable policy effects with significantly less effort than traditional review and evidence synthesis methods. It also supports synthesis of policy evidence scattered across policy issues or areas and can harmonize disconnected information from various levels of scope, performance perspectives, and taxonomies. Comparison of insights obtained from the causal map against those from a manual review of the same source material has also demonstrated that most of the insights can be captured by this approach, whilst providing a more configurative perspective of the features contributing to policy behavior. Finally, in providing a causal map representation of a policy area, new tools for policy evaluation, such as topographic, causal inference, and causal loop analysis, become available to analysts. While promising in many respects, some notable limitations include the poor recall (~\u0026thinsp;38%) achieved in this application of the method. This may contribute to structural gaps in the map and a lack of contextualization for cause-effect relations which can inhibit understanding of the nuance behind certain behavior.\u003c/p\u003e \u003cp\u003eRegardless of algorithmic performance or the quality of insights obtained for ETS, the method and implementation presented in \u003cem\u003ethis\u003c/em\u003e study only represents a first step in combining NLP, causal mapping, and graph analytics for policy-relevant evidence synthesis. The proposed method, and future iterations, appears to contribute a promising new tool for policy analysts across domains, helping to provide a more comprehensive understanding of the factors and relations affecting policy and ultimately improving the evidence base on which to inform policy development.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e \u003ch2\u003e7 Conflict of interest\u003c/h2\u003e \u003cp\u003eAll authors declare that they have no conflicts of interest.\u003c/p\u003e \u003c/p\u003e"},{"header":"References","content":"\u003cp\u003eAckermann, F., \u0026amp; Alexander, J. (2016). Researching complex projects: Using causal mapping to take a systems perspective. \u003cem\u003eInternational Journal of Project Management\u003c/em\u003e, \u003cem\u003e34\u003c/em\u003e(6), 891\u0026ndash;901. https://doi.org/10.1016/j.ijproman.2016.04.001\u003c/p\u003e\n\u003cp\u003eAkbik, A., Blythe, D., \u0026amp; Vollgraf, R. (2018).\u0026nbsp;Contextual string embeddings for sequence labeling. \u003cem\u003eProceedings of the 27th International Conference on Computational Linguistics\u003c/em\u003e, 1638\u0026ndash;1649.\u003c/p\u003e\n\u003cp\u003eAncin-Murguzur, F. J., \u0026amp; Hausner, V. H. (2021). causalizeR: A text mining algorithm to identify causal relationships in scientific literature.\u0026nbsp;\u003cem\u003ePeerJ\u003c/em\u003e, \u003cem\u003e9\u003c/em\u003e, e11850. https://doi.org/10.7717/peerj.11850\u003c/p\u003e\n\u003cp\u003eAnderson, B., \u0026amp; Di Maria, C. (2011).\u0026nbsp;Abatement and Allocation in the Pilot Phase of the EU ETS. \u003cem\u003eEnvironmental and Resource Economics\u003c/em\u003e, \u003cem\u003e48\u003c/em\u003e(1), 83\u0026ndash;103. https://doi.org/10.1007/s10640-010-9399-9\u003c/p\u003e\n\u003cp\u003eAnderson, L. M., Oliver, S. R., Michie, S., Rehfuess, E., Noyes, J., \u0026amp; Shemilt, I. (2013). Investigating complexity in systematic reviews of interventions by using a spectrum of methods. \u003cem\u003eJournal of Clinical Epidemiology\u003c/em\u003e, \u003cem\u003e66\u003c/em\u003e(11), 1223\u0026ndash;1229. https://doi.org/10.1016/j.jclinepi.2013.06.014\u003c/p\u003e\n\u003cp\u003eArimura, T. H., \u0026amp; Abe, T. (2021). The impact of the Tokyo emissions trading scheme on office buildings: What factor contributed to the emission reduction? \u003cem\u003eEnvironmental Economics and Policy Studies\u003c/em\u003e, \u003cem\u003e23\u003c/em\u003e(3), 517\u0026ndash;533. https://doi.org/10.1007/s10018-020-00271-w\u003c/p\u003e\n\u003cp\u003eAsghar, N. (2016). \u003cem\u003eAutomatic Extraction of Causal Relations from Natural Language Texts: A Comprehensive Survey\u003c/em\u003e. https://arxiv.org/abs/1605.07895v1\u003c/p\u003e\n\u003cp\u003eAxelrod, R. (1976). \u003cem\u003eStructure of Decision: The Cognitive Maps of Political Elites\u003c/em\u003e. Princeton University Press.\u003c/p\u003e\n\u003cp\u003eBach, N., \u0026amp; Badaskar, S. (2007). A review of relation extraction. \u003cem\u003eLiterature Review for Language and Statistics II\u003c/em\u003e, \u003cem\u003e2\u003c/em\u003e, 1\u0026ndash;15.\u003c/p\u003e\n\u003cp\u003eBarbrook-Johnson, P., \u0026amp; Penn, A. S. (2022). \u003cem\u003eSystems Mapping: How to build and use causal models of systems\u003c/em\u003e. Springer Nature.\u003c/p\u003e\n\u003cp\u003eBarik, B., Marsi, E., \u0026amp; \u0026Ouml;zt\u0026uuml;rk, P. (2016). Event Causality Extraction from Natural Science Literature. \u003cem\u003eResearch in Computing Science\u003c/em\u003e, \u003cem\u003e117\u003c/em\u003e(1), 97\u0026ndash;107. https://doi.org/10.13053/rcs-117-1-8\u003c/p\u003e\n\u003cp\u003eBarza, M., Trikalinos, T. A., \u0026amp; Lau, J. (2009).\u0026nbsp;Statistical Considerations in Meta-analysis. \u003cem\u003eInfectious Disease Clinics of North America\u003c/em\u003e, \u003cem\u003e23\u003c/em\u003e(2), 195\u0026ndash;210. https://doi.org/10.1016/j.idc.2009.01.003\u003c/p\u003e\n\u003cp\u003eBayer, P., \u0026amp; Aklin, M. (2020). The European Union Emissions Trading System reduced CO2 emissions despite low prices. \u003cem\u003eProceedings of the National Academy of Sciences\u003c/em\u003e, \u003cem\u003e117\u003c/em\u003e(16), 8804\u0026ndash;8812. https://doi.org/10.1073/pnas.1918128117\u003c/p\u003e\n\u003cp\u003eBeamer, B., Rozovskaya, A., \u0026amp; Girju, R. (2008).\u0026nbsp;Automatic Semantic Relation Extraction with Multiple Boundary Generation. \u003cem\u003eAAAI\u003c/em\u003e, 824\u0026ndash;829.\u003c/p\u003e\n\u003cp\u003eBel, G., \u0026amp; Joseph, S. (2015). Emission abatement: Untangling the impacts of the EU ETS and the economic crisis.\u0026nbsp;\u003cem\u003eEnergy Economics\u003c/em\u003e, \u003cem\u003e49\u003c/em\u003e, 531\u0026ndash;539. https://doi.org/10.1016/j.eneco.2015.03.014\u003c/p\u003e\n\u003cp\u003eBeltagy, I., Lo, K., \u0026amp; Cohan, A. (2019).\u0026nbsp;SciBERT: A Pretrained Language Model for Scientific Text. \u003cem\u003eProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)\u003c/em\u003e, 3615\u0026ndash;3620. https://doi.org/10.18653/v1/D19-1371\u003c/p\u003e\n\u003cp\u003eBornmann, L., \u0026amp; Mutz, R. (2015). Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. \u003cem\u003eJournal of the Association for Information Science and Technology\u003c/em\u003e, \u003cem\u003e66\u003c/em\u003e(11), 2215\u0026ndash;2222.\u003c/p\u003e\n\u003cp\u003eBui, Q.-C., Nuall\u0026aacute;in, B. \u0026Oacute;., Boucher, C. A., \u0026amp; Sloot, P. M. (2010). Extracting causal relations on HIV drug resistance from literature. \u003cem\u003eBMC Bioinformatics\u003c/em\u003e, \u003cem\u003e11\u003c/em\u003e(1), 101. https://doi.org/10.1186/1471-2105-11-101\u003c/p\u003e\n\u003cp\u003eBurns, P. B., Rohrich, R. J., \u0026amp; Chung, K. C. (2011). The levels of evidence and their role in evidence-based medicine. \u003cem\u003ePlast Reconstr Surg\u003c/em\u003e, \u003cem\u003e128\u003c/em\u003e(1), 305\u0026ndash;310. https://doi.org/10.1097/PRS.0b013e318219c171\u003c/p\u003e\n\u003cp\u003eCarroll, C., Booth, A., \u0026amp; Cooper, K. (2011). A worked example of \u0026lsquo;best fit\u0026rsquo; framework synthesis: A systematic review of views concerning the taking of some potential chemopreventive agents. \u003cem\u003eBMC Medical Research Methodology\u003c/em\u003e, \u003cem\u003e11\u003c/em\u003e. https://doi.org/10.1186/1471-2288-11-29\u003c/p\u003e\n\u003cp\u003eCompton, M., \u0026amp; \u0026rsquo;t Hart, P. (2019). \u003cem\u003eGreat policy successes\u003c/em\u003e (First). Oxford University Press.\u003c/p\u003e\n\u003cp\u003eConvery, F. J., \u0026amp; Redmond, L. (2007). Market and Price Developments in the European Union Emissions Trading Scheme. \u003cem\u003eReview of Environmental Economics and Policy\u003c/em\u003e, \u003cem\u003e1\u003c/em\u003e(1), 88\u0026ndash;111. https://doi.org/10.1093/reep/rem010\u003c/p\u003e\n\u003cp\u003eCruanes, J., Roma-Ferri, M. T., \u0026amp; Lloret, E. (2012).\u0026nbsp;\u003cem\u003eMeasuring lexical similarity methods for textual mapping in nursing diagnoses in Spanish and SNOMED-CT\u003c/em\u003e. \u003cem\u003e180\u003c/em\u003e, 255\u0026ndash;259. https://doi.org/10.3233/978-1-61499-101-4-255\u003c/p\u003e\n\u003cp\u003eCullenward, D. (2014). Leakage in California\u0026rsquo;s Carbon Market. \u003cem\u003eThe Electricity Journal\u003c/em\u003e, \u003cem\u003e27\u003c/em\u003e(9), 36\u0026ndash;48. https://doi.org/10.1016/j.tej.2014.09.014\u003c/p\u003e\n\u003cp\u003eDechezlepr\u0026ecirc;tre, A., Nachtigall, D., \u0026amp; Venmans, F. (2018).\u0026nbsp;\u003cem\u003eThe joint impact of the European Union emissions trading system on carbon emissions and economic performance\u003c/em\u003e.\u0026nbsp;OECD. https://doi.org/10.1787/4819b016-en\u003c/p\u003e\n\u003cp\u003eDevlin, J., Chang, M.-W., Lee, K., \u0026amp; Toutanova, K. (2018).\u0026nbsp;Bert: Pre-training of deep bidirectional transformers for language understanding. \u003cem\u003eArXiv Preprint ArXiv:1810.04805\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eDonald Nguyen, Andrew Lenharth, \u0026amp; Keshav Pingali. (2013). \u003cem\u003eA lightweight infrastructure for graph analytics\u003c/em\u003e. Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, Farminton, Pennsylvania.\u003c/p\u003e\n\u003cp\u003eEden, C., Ackermann, F., \u0026amp; Cropper, S. (1992). The Analysis of Cause Maps. \u003cem\u003eJournal of Management Studies\u003c/em\u003e, \u003cem\u003e29\u003c/em\u003e(3), 309\u0026ndash;324. https://doi.org/10.1111/j.1467-6486.1992.tb00667.x\u003c/p\u003e\n\u003cp\u003eEgenhofer, C., Alessi, M., Georgiev, A., \u0026amp; Fujiwara, N. (2011). \u003cem\u003eThe EU Emissions Trading System and Climate Policy Towards 2050: Real Incentives to Reduce Emissions and Drive Innovation?\u003c/em\u003e (SSRN Scholarly Paper 1756736). Social Science Research Network. https://papers.ssrn.com/abstract=1756736\u003c/p\u003e\n\u003cp\u003eEllerman, A. D., \u0026amp; Buchner, B. K. (2007).\u0026nbsp;The European Union Emissions Trading Scheme: Origins, Allocation, and Early Results. \u003cem\u003eReview of Environmental Economics and Policy\u003c/em\u003e, \u003cem\u003e1\u003c/em\u003e(1), 66\u0026ndash;87. https://doi.org/10.1093/reep/rem003\u003c/p\u003e\n\u003cp\u003eEllerman, A. D., \u0026amp; Buchner, B. K. (2008).\u0026nbsp;Over-Allocation or Abatement? A Preliminary Analysis of the EU ETS Based on the 2005\u0026ndash;06 Emissions Data. \u003cem\u003eEnvironmental and Resource Economics\u003c/em\u003e, \u003cem\u003e41\u003c/em\u003e(2), 267\u0026ndash;287. https://doi.org/10.1007/s10640-008-9191-2\u003c/p\u003e\n\u003cp\u003eEllerman, A. D., Marcantonini, C., \u0026amp; Zaklan, A. (2016). The European Union Emissions Trading System: Ten Years and \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; Counting. \u003cem\u003eReview of Environmental Economics and Policy\u003c/em\u003e, \u003cem\u003e10\u003c/em\u003e(1), 89\u0026ndash;107. https://doi.org/10.1093/reep/rev014\u003c/p\u003e\n\u003cp\u003eEllerman, A. D., \u0026amp; McGuinness, M. (2008). \u003cem\u003eCO2 Abatement in the UK Power Sector: Evidence from the EU ETS Trial Period\u003c/em\u003e [Working Paper]. https://dspace.mit.edu/handle/1721.1/45654\u003c/p\u003e\n\u003cp\u003eEl‐Taliawi, O. G., Goyal, N., \u0026amp; Howlett, M. (2021). Holding out the promise of Lasswell\u0026rsquo;s dream: Big data analytics in public policy research and teaching. \u003cem\u003eReview of Policy Research\u003c/em\u003e, \u003cem\u003e38\u003c/em\u003e(6), 640\u0026ndash;660.\u003c/p\u003e\n\u003cp\u003eEster, M., Kriegel, H.-P., Sander, J., \u0026amp; Xu, X. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise. \u003cem\u003eKdd\u003c/em\u003e, \u003cem\u003e96\u003c/em\u003e(34), 226\u0026ndash;231.\u003c/p\u003e\n\u003cp\u003eEsty, D. C., \u0026amp; Porter, M. E. (2005). National environmental performance: An empirical analysis of policy results and determinants. \u003cem\u003eEnvironment and Development Economics\u003c/em\u003e, \u003cem\u003e10\u003c/em\u003e(4), 391\u0026ndash;434. Scopus. https://doi.org/10.1017/S1355770X05002275\u003c/p\u003e\n\u003cp\u003eEuropean Commission. (2021). \u003cem\u003eEU Emissions Trading System (EU ETS)\u003c/em\u003e. https://ec.europa.eu/clima/eu-action/eu-emissions-trading-system-eu-ets_en\u003c/p\u003e\n\u003cp\u003eFeder, A., Oved, N., Shalit, U., \u0026amp; Reichart, R. (2021). CausaLM: Causal Model Explanation Through Counterfactual Language Models. \u003cem\u003eComputational Linguistics\u003c/em\u003e, \u003cem\u003e47\u003c/em\u003e(2), 333\u0026ndash;386. https://doi.org/10.1162/coli_a_00404\u003c/p\u003e\n\u003cp\u003eFell, H., \u0026amp; Maniloff, P. (2018). Leakage in regional environmental policy: The case of the regional greenhouse gas initiative. \u003cem\u003eJournal of Environmental Economics and Management\u003c/em\u003e, \u003cem\u003e87\u003c/em\u003e, 1\u0026ndash;23. https://doi.org/10.1016/j.jeem.2017.10.007\u003c/p\u003e\n\u003cp\u003eFrance, E. F., Cunningham, M., Ring, N., Uny, I., Duncan, E. A. S., Jepson, R. G., Maxwell, M., Roberts, R. J., Turley, R. L., Booth, A., Britten, N., Flemming, K., Gallagher, I., Garside, R., Hannes, K., Lewin, S., Noblit, G. W., Pope, C., Thomas, J., \u0026hellip; Noyes, J. (2019). Improving reporting of meta-ethnography: The eMERGe reporting guidance. \u003cem\u003ePsycho-Oncology\u003c/em\u003e, \u003cem\u003e28\u003c/em\u003e(3), 447\u0026ndash;458. https://doi.org/10.1002/pon.4915\u003c/p\u003e\n\u003cp\u003eFreebairn, L., Atkinson, J., Kelly, P., McDonnell, G., \u0026amp; Rychetnik, L. (2016). Simulation modelling as a tool for knowledge mobilisation in health policy settings: A case study protocol. \u003cem\u003eHealth Research Policy and Systems\u003c/em\u003e, \u003cem\u003e14\u003c/em\u003e(1). https://doi.org/10.1186/s12961-016-0143-y\u003c/p\u003e\n\u003cp\u003eFusar-Poli, P., \u0026amp; Radua, J. (2018). Ten simple rules for conducting umbrella reviews. \u003cem\u003eEvidence-Based Mental Health\u003c/em\u003e, \u003cem\u003e21\u003c/em\u003e(3), 95\u0026ndash;100. https://doi.org/10.1136/ebmental-2018-300014\u003c/p\u003e\n\u003cp\u003eGirju, R., Nakov, P., Nastase, V., Szpakowicz, S., Turney, P., \u0026amp; Yuret, D. (2009). Classification of semantic relations between nominals. \u003cem\u003eLanguage Resources and Evaluation\u003c/em\u003e, \u003cem\u003e43\u003c/em\u003e(2), 105\u0026ndash;121. https://doi.org/10.1007/s10579-009-9083-2\u003c/p\u003e\n\u003cp\u003eGloaguen, O., \u0026amp; Alberola, E. (2013). Assessing the factors behind CO2 emissions changes over the phases 1 and 2 of the EU ETS: An econometric analysis. \u003cem\u003eCDC Climat Research, Paris, France\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eGoyal, N. (2021). Explaining Policy Success Using the Multiple Streams Framework: Political Success Despite Programmatic Failure of the Solar Energy Policy in Gujarat, India. \u003cem\u003ePolitics \u0026amp; Policy\u003c/em\u003e, \u003cem\u003e49\u003c/em\u003e(5), 1021\u0026ndash;1060. https://doi.org/10.1111/polp.12426\u003c/p\u003e\n\u003cp\u003eGoyal, N., \u0026amp; Howlett, M. (2018). Lessons Learned and Not Learned: Bibliometric Analysis of Policy Learning. In C. A. Dunlop, C. M. Radaelli, \u0026amp; P. Trein (Eds.), \u003cem\u003eLearning in Public Policy: Analysis, Modes and Outcomes\u003c/em\u003e (pp. 27\u0026ndash;49). Springer International Publishing.\u003c/p\u003e\n\u003cp\u003eGoyal, N., \u0026amp; Howlett, M. (2019). Combining internal and external evaluations within a multilevel evaluation framework: Computational text analysis of lessons from the Asian Development Bank. \u003cem\u003eEvaluation\u003c/em\u003e, \u003cem\u003e25\u003c/em\u003e(3), 366\u0026ndash;380. https://doi.org/10.1177/1356389019827035\u003c/p\u003e\n\u003cp\u003eGoyal, N., \u0026amp; Howlett, M. (2021). \u0026ldquo;Measuring the Mix\u0026rdquo; of Policy Responses to COVID-19: Comparative Policy Analysis Using Topic Modelling. \u003cem\u003eJournal of Comparative Policy Analysis: Research and Practice\u003c/em\u003e, \u003cem\u003e23\u003c/em\u003e(2), 250\u0026ndash;261.\u003c/p\u003e\n\u003cp\u003eGreen, J. F. (2021). Does carbon pricing reduce emissions? A review of ex-post analyses. \u003cem\u003eEnvironmental Research Letters\u003c/em\u003e, \u003cem\u003e16\u003c/em\u003e(4), 043004. https://doi.org/10.1088/1748-9326/abdae9\u003c/p\u003e\n\u003cp\u003eGreenhalgh, T., Wong, G., Westhorp, G., \u0026amp; Pawson, R. (2011). Protocol - realist and meta-narrative evidence synthesis: Evolving Standards (RAMESES). \u003cem\u003eBMC Medical Research Methodology\u003c/em\u003e, \u003cem\u003e11\u003c/em\u003e(1), 115. https://doi.org/10.1186/1471-2288-11-115\u003c/p\u003e\n\u003cp\u003eHaddaway, N. R., Land, M., \u0026amp; Macura, B. (2017). \u0026ldquo;A little learning is a dangerous thing\u0026rdquo;: A call for better understanding of the term \u0026lsquo;systematic review\u0026rsquo;. \u003cem\u003eEnvironment International\u003c/em\u003e, \u003cem\u003e99\u003c/em\u003e, 356\u0026ndash;360. https://doi.org/10.1016/j.envint.2016.12.020\u003c/p\u003e\n\u003cp\u003eHan, H., Wang, Q., \u0026amp; Chen, C. (2019). Policy Text Analysis Based on Text Mining and Fuzzy Cognitive Map. \u003cem\u003e2019 15th International Conference on Computational Intelligence and Security (CIS)\u003c/em\u003e, 142\u0026ndash;146. https://doi.org/10.1109/CIS.2019.00038\u003c/p\u003e\n\u003cp\u003eHannes, K., \u0026amp; Lockwood, C. (2011). Pragmatism as the philosophical foundation for the Joanna Briggs meta-aggregative approach to qualitative evidence synthesis. \u003cem\u003eJournal of Advanced Nursing\u003c/em\u003e, \u003cem\u003e67\u003c/em\u003e(7), 1632\u0026ndash;1642. https://doi.org/10.1111/j.1365-2648.2011.05636.x\u003c/p\u003e\n\u003cp\u003eHansen, C., Steinmetz, H., \u0026amp; Block, J. (2022). How to conduct a meta-analysis in eight steps: A practical guide. \u003cem\u003eManagement Review Quarterly\u003c/em\u003e, \u003cem\u003e72\u003c/em\u003e(1), 1\u0026ndash;19. https://doi.org/10.1007/s11301-021-00247-4\u003c/p\u003e\n\u003cp\u003eHibbard, P. J., Okie, A. M., Tierney, S. F., \u0026amp; Darling, P. G. (2015). The economic impacts of the regional greenhouse gas initiative on nine northeast and mid-Atlantic states. \u003cem\u003eAnalysis Group, July\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eICAP. (2022a). \u003cem\u003eEmissions Trading Worldwide \u0026ndash; International Carbon Action Partnership (ICAP) Status Report 2022\u003c/em\u003e. 240.\u003c/p\u003e\n\u003cp\u003eICAP. (2022b). \u003cem\u003eAbout Emissions Trading Systems\u003c/em\u003e. International Carbon Action Partnership. https://icapcarbonaction.com/en/about-emissions-trading-systems\u003c/p\u003e\n\u003cp\u003eInan, E. (2020). SimiT: A Text Similarity Method Using Lexicon and Dependency Representations. \u003cem\u003eNew Generation Computing\u003c/em\u003e, \u003cem\u003e38\u003c/em\u003e(3), 509\u0026ndash;530. https://doi.org/10.1007/s00354-020-00099-8\u003c/p\u003e\n\u003cp\u003eJames, K. L., Randall, N. P., \u0026amp; Haddaway, N. R. (2016). A methodology for systematic mapping in environmental sciences. \u003cem\u003eEnvironmental Evidence\u003c/em\u003e, \u003cem\u003e5\u003c/em\u003e(1). https://doi.org/10.1186/s13750-016-0059-6\u003c/p\u003e\n\u003cp\u003eJaraite-Kažukauske, J., \u0026amp; Di Maria, C. (2016). Did the EU ETS make a difference? An empirical assessment using Lithuanian firm-level data. \u003cem\u003eThe Energy Journal\u003c/em\u003e, \u003cem\u003e37\u003c/em\u003e(1).\u003c/p\u003e\n\u003cp\u003eJiali, L. (1995). China\u0026rsquo;s one-child policy: How and how well has it worked? A case study of Hebei Province, 1979-88. \u003cem\u003ePopulation \u0026amp; Development Review\u003c/em\u003e, \u003cem\u003e21\u003c/em\u003e(3), 563\u0026ndash;585. Scopus.\u003c/p\u003e\n\u003cp\u003eKhoo, C. S., \u0026amp; Na, J.-C. (2006).\u0026nbsp;Semantic relations in information science. \u003cem\u003eAnnual Review of Information Science and Technology\u003c/em\u003e, \u003cem\u003e40\u003c/em\u003e(1), 157\u0026ndash;228.\u003c/p\u003e\n\u003cp\u003eKim, J., Han, M., Lee, Y., \u0026amp; Park, Y. (2016). Futuristic data-driven scenario building: Incorporating text mining and fuzzy association rule mining into fuzzy cognitive map. \u003cem\u003eExpert Systems with Applications\u003c/em\u003e, \u003cem\u003e57\u003c/em\u003e, 311\u0026ndash;323. Scopus. https://doi.org/10.1016/j.eswa.2016.03.043\u003c/p\u003e\n\u003cp\u003eKotnik, Ž., Maja, K., \u0026amp; \u0026Scaron;kulj, D. (2014).\u0026nbsp;The effect of taxation on greenhouse gas emissions. \u003cem\u003eTransylvanian Review of Administrative Sciences\u003c/em\u003e, \u003cem\u003e10\u003c/em\u003e(43), 168\u0026ndash;185.\u003c/p\u003e\n\u003cp\u003eKruger, J., Oates, W. E., \u0026amp; Pizer, W. A. (2007). Decentralization in the EU Emissions Trading Scheme and Lessons for Global Policy. \u003cem\u003eReview of Environmental Economics and Policy\u003c/em\u003e, \u003cem\u003e1\u003c/em\u003e(1), 112\u0026ndash;133. https://doi.org/10.1093/reep/rem009\u003c/p\u003e\n\u003cp\u003eKyriakakis, M., Androutsopoulos, I., Saudabayev, A., \u0026amp; Gin\u0026eacute;s i Ametll\u0026eacute;, J. (2019). Transfer Learning for Causal Sentence Detection. \u003cem\u003eProceedings of the 18th BioNLP Workshop and Shared Task\u003c/em\u003e, 292\u0026ndash;297. https://doi.org/10.18653/v1/W19-5031\u003c/p\u003e\n\u003cp\u003eLarsen, P., \u0026amp; Von Ins, M. (2010).\u0026nbsp;The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. \u003cem\u003eScientometrics\u003c/em\u003e, \u003cem\u003e84\u003c/em\u003e(3), 575\u0026ndash;603.\u003c/p\u003e\n\u003cp\u003eLeining, C., Kerr, S., \u0026amp; Bruce-Brand, B. (2020). The New Zealand Emissions Trading Scheme: Critical review and future outlook for three design innovations. \u003cem\u003eClimate Policy\u003c/em\u003e, \u003cem\u003e20\u003c/em\u003e(2), 246\u0026ndash;264. https://doi.org/10.1080/14693062.2019.1699773\u003c/p\u003e\n\u003cp\u003eLevy, O., \u0026amp; Goldberg, Y. (2014). Dependency-based word embeddings. \u003cem\u003eProceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)\u003c/em\u003e, 302\u0026ndash;308.\u003c/p\u003e\n\u003cp\u003eLi, Z., Li, Q., Zou, X., \u0026amp; Ren, J. (2021). Causality extraction based on self-attentive BiLSTM-CRF with transferred embeddings. \u003cem\u003eNeurocomputing\u003c/em\u003e, \u003cem\u003e423\u003c/em\u003e, 207\u0026ndash;219. https://doi.org/10.1016/j.neucom.2020.08.078\u003c/p\u003e\n\u003cp\u003eLiddy, E. D. (2001). \u003cem\u003eNatural language processing\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eLiu, X. Y., Zhou, Y. M., \u0026amp; Zheng, R. S. (2007).\u0026nbsp;\u003cem\u003eMeasuring semantic similarity in WordNet\u003c/em\u003e. 3431-+.\u003c/p\u003e\n\u003cp\u003eL\u0026ouml;fgren, \u0026Aring;., Burtraw, D., Wr\u0026aring;ke, M., \u0026amp; Malinovskaya, A. (2015). \u003cem\u003eArchitecture of the EU emissions trading system in phase 3 and the distribution of allowance asset values\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eMacura, B., Su\u0026scaron;kevičs, M., Garside, R., Hannes, K., Rees, R., \u0026amp; Rodela, R. (2019).\u0026nbsp;Systematic reviews of qualitative evidence for environmental policy and management: An overview of different methodological options.\u0026nbsp;\u003cem\u003eEnvironmental Evidence\u003c/em\u003e, \u003cem\u003e8\u003c/em\u003e(1). https://doi.org/10.1186/s13750-019-0168-0\u003c/p\u003e\n\u003cp\u003eMarsh, D., \u0026amp; McConnell, A. (2010).\u0026nbsp;Towards a Framework for Establishing Policy Success. \u003cem\u003ePublic Administration\u003c/em\u003e, \u003cem\u003e88\u003c/em\u003e(2), 564\u0026ndash;583. https://doi.org/10.1111/j.1467-9299.2009.01803.x\u003c/p\u003e\n\u003cp\u003eMartin, G., \u0026amp; Saikawa, E. (2017). Effectiveness of state climate and energy policies in reducing power-sector CO2 emissions.\u0026nbsp;\u003cem\u003eNature Climate Change\u003c/em\u003e, \u003cem\u003e7\u003c/em\u003e(12), Article 12. https://doi.org/10.1038/s41558-017-0001-0\u003c/p\u003e\n\u003cp\u003eMcConnell, A. (2010).\u0026nbsp;Policy Success, Policy Failure and Grey Areas In-Between. \u003cem\u003eJournal of Public Policy\u003c/em\u003e, \u003cem\u003e30\u003c/em\u003e(3), 345\u0026ndash;362. https://doi.org/10.1017/S0143814X10000152\u003c/p\u003e\n\u003cp\u003eMillard, T., Synnot, A., Elliott, J., Green, S., McDonald, S., \u0026amp; Turner, T. (2019). Feasibility and acceptability of living systematic reviews: Results from a mixed-methods evaluation. \u003cem\u003eSystematic Reviews\u003c/em\u003e, \u003cem\u003e8\u003c/em\u003e(1). https://doi.org/10.1186/s13643-019-1248-5\u003c/p\u003e\n\u003cp\u003eMontibeller, G., \u0026amp; Belton, V. (2006). Causal maps and the evaluation of decision options\u0026mdash;A review. \u003cem\u003eJournal of the Operational Research Society\u003c/em\u003e, \u003cem\u003e57\u003c/em\u003e(7), 779\u0026ndash;791. https://doi.org/10.1057/palgrave.jors.2602214\u003c/p\u003e\n\u003cp\u003eMurray, B. C., \u0026amp; Maniloff, P. T. (2015). Why have greenhouse emissions in RGGI states declined? An econometric attribution to economic, energy market, and policy factors. \u003cem\u003eEnergy Economics\u003c/em\u003e, \u003cem\u003e51\u003c/em\u003e, 581\u0026ndash;589. https://doi.org/10.1016/j.eneco.2015.07.013\u003c/p\u003e\n\u003cp\u003eNorman, E. R. (2023). Policy Studies Yearbook Annual Review 2022\u0026ndash;23: Note from the editor and call for papers. \u003cem\u003ePolitics \u0026amp; Policy\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eNunez-Mir, G. C., Iannone III, B. V., Pijanowski, B. C., Kong, N., \u0026amp; Fei, S. (2016). Automated content analysis: Addressing the big literature challenge in ecology and evolution. \u003cem\u003eMethods in Ecology and Evolution\u003c/em\u003e, \u003cem\u003e7\u003c/em\u003e(11), 1262\u0026ndash;1272. https://doi.org/10.1111/2041-210X.12602\u003c/p\u003e\n\u003cp\u003eO\u0026rsquo;Leary, D. F., Casey, M., O\u0026rsquo;Connor, L., Stokes, D., Fealy, G. M., O\u0026rsquo;Brien, D., Smith, R., McNamara, M. S., \u0026amp; Egan, C. (2017). Using rapid reviews: An example from a study conducted to inform policy-making. \u003cem\u003eJournal of Advanced Nursing\u003c/em\u003e, \u003cem\u003e73\u003c/em\u003e(3), 742\u0026ndash;752. https://doi.org/10.1111/jan.13231\u003c/p\u003e\n\u003cp\u003ePakray, P., \u0026amp; Gelbukh, A. (2014). An Open-Domain Cause-Effect Relation Detection from Paired Nominals. In A. Gelbukh, F. C. Espinoza, \u0026amp; S. N. Galicia-Haro (Eds.), \u003cem\u003eNature-Inspired Computation and Machine Learning\u003c/em\u003e (pp. 263\u0026ndash;271). Springer International Publishing. https://doi.org/10.1007/978-3-319-13650-9_24\u003c/p\u003e\n\u003cp\u003ePearson, A., White, H., Bath-Hextall, F., Salmond, S., Apostolo, J., \u0026amp; Kirkpatrick, P. (2015). A mixed-methods approach to systematic reviews. \u003cem\u003eInternational Journal of Evidence-Based Healthcare\u003c/em\u003e, \u003cem\u003e13\u003c/em\u003e(3), 121\u0026ndash;131. https://doi.org/10.1097/XEB.0000000000000052\u003c/p\u003e\n\u003cp\u003ePelevina, M., Arefyev, N., Biemann, C., \u0026amp; Panchenko, A. (2017). \u003cem\u003eMaking Sense of Word Embeddings\u003c/em\u003e (arXiv:1708.03390). arXiv. https://doi.org/10.48550/arXiv.1708.03390\u003c/p\u003e\n\u003cp\u003ePeters, M. D. J., Marnie, C., Colquhoun, H., Garritty, C. M., Hempel, S., Horsley, T., Langlois, E. V., Lillie, E., O\u0026rsquo;Brien, K. K., Tun\u0026ccedil;alp, \u0026nbsp;Ӧ., Wilson, M. G., Zarin, W., \u0026amp; Tricco, A. C. (2021). Scoping reviews: Reinforcing and advancing the methodology and application. \u003cem\u003eSystematic Reviews\u003c/em\u003e, \u003cem\u003e10\u003c/em\u003e(1). https://doi.org/10.1186/s13643-021-01821-3\u003c/p\u003e\n\u003cp\u003ePeters, M. D. J., Marnie, C., Tricco, A. C., Pollock, D., Munn, Z., Alexander, L., McInerney, P., Godfrey, C. M., \u0026amp; Khalil, H. (2020). Updated methodological guidance for the conduct of scoping reviews. \u003cem\u003eJBI Evidence Synthesis\u003c/em\u003e, \u003cem\u003e18\u003c/em\u003e(10), 2119\u0026ndash;2126. https://doi.org/10.11124/JBIES-20-00167\u003c/p\u003e\n\u003cp\u003ePetrick, S., \u0026amp; Wagner, U. J. (2014). \u003cem\u003eThe Impact of Carbon Trading on Industry: Evidence from German Manufacturing Firms\u003c/em\u003e (SSRN Scholarly Paper 2389800). Social Science Research Network. https://doi.org/10.2139/ssrn.2389800\u003c/p\u003e\n\u003cp\u003ePetticrew, M., \u0026amp; Roberts, H. (2008). \u003cem\u003eSystematic Reviews in the Social Sciences: A Practical Guide\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003ePullin, A., Frampton, G., Jongman, R., Kohl, C., Livoreil, B., Lux, A., Pataki, G., Petrokofsky, G., Podhora, A., Saarikoski, H., Santamaria, L., Schindler, S., Sousa-Pinto, I., Vandewalle, M., \u0026amp; Wittmer, H. (2016). Selecting appropriate methods of knowledge synthesis to inform biodiversity policy. \u003cem\u003eBiodiversity and Conservation\u003c/em\u003e, \u003cem\u003e25\u003c/em\u003e(7), 1285\u0026ndash;1300. https://doi.org/10.1007/s10531-016-1131-9\u003c/p\u003e\n\u003cp\u003eRanson, M., \u0026amp; Stavins, R. N. (2012). \u003cem\u003ePost-Durban climate policy architecture based on linkage of cap-and-trade systems\u003c/em\u003e. National Bureau of Economic Research.\u003c/p\u003e\n\u003cp\u003eReimers, N., \u0026amp; Gurevych, I. (2019). \u003cem\u003eSentence-BERT: Sentence Embeddings using Siamese BERT-Networks\u003c/em\u003e (arXiv:1908.10084). arXiv. http://arxiv.org/abs/1908.10084\u003c/p\u003e\n\u003cp\u003eSaetren, H. (2005). Facts and myths about research on public policy implementation: Out-of-fashion, allegedly dead, but still very much alive and relevant. \u003cem\u003ePolicy Studies Journal\u003c/em\u003e, \u003cem\u003e33\u003c/em\u003e(4), 559\u0026ndash;582. https://doi.org/10.1111/j.1541-0072.2005.00133.x\u003c/p\u003e\n\u003cp\u003eSanderson, I. (2002). Evaluation, policy learning and evidence-based policy making. \u003cem\u003ePublic Administration\u003c/em\u003e, \u003cem\u003e80\u003c/em\u003e(1), 1\u0026ndash;22.\u003c/p\u003e\n\u003cp\u003eSartor, O., Palli\u0026egrave;re, C., \u0026amp; Lecourt, S. (2014). Benchmark-based allocations in EU ETS Phase 3: An early assessment. \u003cem\u003eClimate Policy\u003c/em\u003e, \u003cem\u003e14\u003c/em\u003e(4), 507\u0026ndash;524. https://doi.org/10.1080/14693062.2014.872888\u003c/p\u003e\n\u003cp\u003eSaul, J. E., Willis, C. D., Bitz, J., \u0026amp; Best, A. (2013). A time-responsive tool for informing policy making: Rapid realist review.\u0026nbsp;\u003cem\u003eImplementation Science\u003c/em\u003e, \u003cem\u003e8\u003c/em\u003e(1). https://doi.org/10.1186/1748-5908-8-103\u003c/p\u003e\n\u003cp\u003eSchmalensee, R., \u0026amp; Stavins, R. N. (2017).\u0026nbsp;Lessons Learned from Three Decades of Experience with Cap and \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; Trade. \u003cem\u003eReview of Environmental Economics and Policy\u003c/em\u003e, \u003cem\u003e11\u003c/em\u003e(1), 59\u0026ndash;79. https://doi.org/10.1093/reep/rew017\u003c/p\u003e\n\u003cp\u003eSelva Birunda, S., \u0026amp; Kanniga Devi, R. (2021).\u0026nbsp;A Review on Word Embedding Techniques for Text Classification. In J. S. Raj, A. M. Iliyasu, R. Bestak, \u0026amp; Z. A. Baig (Eds.), \u003cem\u003eInnovative Data Communication Technologies and Application\u003c/em\u003e (pp. 267\u0026ndash;281).\u0026nbsp;Springer. https://doi.org/10.1007/978-981-15-9651-3_23\u003c/p\u003e\n\u003cp\u003eSijm, J., Neuhoff, K., \u0026amp; Chen, Y. (2011).\u0026nbsp;CO2 cost pass-through and windfall profits in the power sector. \u003cem\u003eClimate Policy\u003c/em\u003e, \u003cem\u003e6\u003c/em\u003e(1), 49\u0026ndash;72. https://doi.org/10.1080/14693062.2006.9685588\u003c/p\u003e\n\u003cp\u003eSon, C., Kim, J., \u0026amp; Kim, Y. (2020). Developing scenario-based technology roadmap in the big data era: An utilisation of fuzzy cognitive map and text mining techniques. \u003cem\u003eTechnology Analysis \u0026amp; Strategic Management\u003c/em\u003e, \u003cem\u003e32\u003c/em\u003e(3), 272\u0026ndash;291. https://doi.org/10.1080/09537325.2019.1654091\u003c/p\u003e\n\u003cp\u003eThomas, J., \u0026amp; Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. \u003cem\u003eBMC Med Res Methodol\u003c/em\u003e, \u003cem\u003e8\u003c/em\u003e, 45. https://doi.org/10.1186/1471-2288-8-45\u003c/p\u003e\n\u003cp\u003eThow, A. M., Swinburn, B., Colagiuri, S., Diligolevu, M., Quested, C., Vivili, P., \u0026amp; Leeder, S. (2010). Trade and food policy: Case studies from three Pacific Island countries. \u003cem\u003eFood Policy\u003c/em\u003e, \u003cem\u003e35\u003c/em\u003e(6), 556\u0026ndash;564. Scopus. https://doi.org/10.1016/j.foodpol.2010.06.005\u003c/p\u003e\n\u003cp\u003evan de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., \u0026amp; Oberski, D. L. (2021). An open source machine learning framework for efficient and transparent systematic reviews.\u0026nbsp;\u003cem\u003eNature Machine Intelligence\u003c/em\u003e, \u003cem\u003e3\u003c/em\u003e(2), Article 2. https://doi.org/10.1038/s42256-020-00287-7\u003c/p\u003e\n\u003cp\u003eWagner, U. J., Mu\u0026ucirc;ls, M., Martin, R., \u0026amp; Colmer, J. (2014).\u0026nbsp;The causal effects of the European Union Emissions Trading Scheme: Evidence from French manufacturing plants. \u003cem\u003eFifth World Congress of Environmental and Resources Economists, Instanbul, Turkey\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eWakabayashi, M., \u0026amp; Kimura, O. (2018). The impact of the Tokyo Metropolitan Emissions Trading Scheme on reducing greenhouse gas emissions: Findings from a facility-based study. \u003cem\u003eClimate Policy\u003c/em\u003e, \u003cem\u003e18\u003c/em\u003e(8), 1028\u0026ndash;1043. https://doi.org/10.1080/14693062.2018.1437018\u003c/p\u003e\n\u003cp\u003eWarner, J., \u0026amp; van Buuren, A. (2011). Implementing room for the river: Narratives of success and failure in Kampen, the Netherlands. \u003cem\u003eInternational Review of Administrative Sciences\u003c/em\u003e, \u003cem\u003e77\u003c/em\u003e(4), 779\u0026ndash;801. Scopus. https://doi.org/10.1177/0020852311419387\u003c/p\u003e\n\u003cp\u003eWeible, C. M., \u0026amp; Sabatier, P. A. (2018). \u003cem\u003eTheories of the Policy Process\u003c/em\u003e. Taylor and Francis.\u003c/p\u003e\n\u003cp\u003eWing, I. S., \u0026amp; Kolodziej, M. (2008). \u003cem\u003eThe Regional Greenhouse Gas Initiative: Emission Leakage and the Effectiveness of Interstate Border Adjustments\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eWolffe, T. A. M., Whaley, P., Halsall, C., Rooney, A. A., \u0026amp; Walker, V. R. (2019). Systematic evidence maps as a novel tool to support evidence-based decision-making in chemicals policy and risk management. \u003cem\u003eEnviron Int\u003c/em\u003e, \u003cem\u003e130\u003c/em\u003e, 104871. https://doi.org/10.1016/j.envint.2019.05.065\u003c/p\u003e\n\u003cp\u003eWong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., \u0026amp; Pawson, R. (2013a). RAMESES publication standards: Meta-narrative reviews. \u003cem\u003eBMC Med\u003c/em\u003e, \u003cem\u003e11\u003c/em\u003e, 20. https://doi.org/10.1186/1741-7015-11-20\u003c/p\u003e\n\u003cp\u003eWong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., \u0026amp; Pawson, R. (2013b). RAMESES publication standards: Realist syntheses. \u003cem\u003eBMC Medicine\u003c/em\u003e, \u003cem\u003e11\u003c/em\u003e(1). https://doi.org/10.1186/1741-7015-11-21\u003c/p\u003e\n\u003cp\u003eWorld Bank. (2023). \u003cem\u003eState and Trends of Carbon Pricing 2023\u003c/em\u003e. https://doi.org/10.1596/39796\u003c/p\u003e\n\u003cp\u003eYang, J., Han, S. C., \u0026amp; Poon, J. (2021).\u0026nbsp;A Survey on Extraction of Causal Relations from Natural Language Text.\u0026nbsp;\u003cem\u003eArXiv:2101.06426 [Cs]\u003c/em\u003e. http://arxiv.org/abs/2101.06426\u003c/p\u003e\n\u003cp\u003eZhao, S., Liu, T., Zhao, S., Chen, Y., \u0026amp; Nie, J.-Y. (2016).\u0026nbsp;Event causality extraction based on connectives analysis. \u003cem\u003eNeurocomputing\u003c/em\u003e, \u003cem\u003e173\u003c/em\u003e, 1943\u0026ndash;1950. https://doi.org/10.1016/j.neucom.2015.09.066\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Evidence-informed policy, Emissions trading schemes (ETS), Causal mapping, Machine learning (ML), Natural language processing (NLP), Policy analysis","lastPublishedDoi":"10.21203/rs.3.rs-3285731/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3285731/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAlthough causal evidence synthesis is critical for the policy sciences \u0026ndash; whether it be analysis \u003cem\u003efor\u003c/em\u003e policy or analysis \u003cem\u003eof\u003c/em\u003e policy \u0026ndash; its repeatable, systematic, and transparent execution remains challenging due to the growing volume, variety, and velocity of policy-relevant evidence generation as well as the complex web of relationships within which policies are usually situated. To address these shortcomings, we developed a novel, semi-automated approach to synthesizing causal evidence from policy-relevant documents. Specifically, we propose the use of natural language processing (NLP) for the extraction of causal evidence and subsequent homogenization or normalization of the varied text, causal mapping for the collation, visualization, and summarization of complex interdependencies within the policy system, and graph analytics for further investigation of the structure and dynamics of the causal map. We illustrate this approach by applying it to a collection of 28 articles on the emissions trading scheme (ETS), a policy instrument of increasing importance for climate change mitigation. In all, we find 300 variables and 284 cause-effect pairs in our input dataset (consisting of 4524 sentences), which are reduced to 70 unique variables and 119 cause-effect pairs after normalization. We create a causal map depicting these and analyze it subsequently to obtain systemic perspective as well as policy-relevant insight on the ETS that is broadly consistent with select manually conducted, previous meta-reviews of the policy instrument. We conclude that, despite its present limitations, this approach can help synthesize causal evidence for policy analysis, policymaking, and policy research.\u003c/p\u003e","manuscriptTitle":"A semi-automated approach to policy-relevant evidence synthesis: Combining natural language processing, causal mapping, and graph analytics for public policy","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2023-08-23 17:43:44","doi":"10.21203/rs.3.rs-3285731/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"0d1f7483-3b79-4f80-946b-05ce781af55d","owner":[],"postedDate":"August 23rd, 2023","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2023-08-23T17:43:44+00:00","versionOfRecord":[],"versionCreatedAt":"2023-08-23 17:43:44","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3285731","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3285731","identity":"rs-3285731","version":["v1"]},"buildId":"FbvkV6FR0MCFSLy54lSbu","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}