Coincidence Analysis: A New Method for Causal Inference in Implementation Science

Coincidence Analysis: A New Method for Causal Inference in Implementation Science | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Help Center Sign In Submit a Preprint Cite Share Download PDF Methodology Coincidence Analysis: A New Method for Causal Inference in Implementation Science Rebecca Garr Whitaker, Nina Sperber, Michael Baumgartner, Alrik Thiem, and 5 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-58815/v2 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 11 Dec, 2020 Read the published version in Implementation Science → Version 2 posted 4 You are reading this latest preprint version Show more versions Abstract Background: Implementation of multifaceted interventions typically involves many diverse elements working together in interrelated ways, including intervention components, implementation strategies, and features of local context. Given this real-world complexity, implementation researchers may be interested in a new mathematical, cross-case method called Coincidence Analysis (CNA) that has been designed explicitly to support causal inference, answer research questions about combinations of conditions that are minimally necessary or sufficient for an outcome, and identify the possible presence of multiple causal paths to an outcome. CNA can be applied as a standalone method or in conjunction with other approaches, and can reveal new empirical findings related to implementation that might otherwise have gone undetected. Methods: We applied CNA to a publicly available dataset from Sweden with county-level data on human papillomavirus (HPV) vaccination campaigns and vaccination uptake in 2012 and 2014 and then compared CNA results to the published regression findings. Results: The original regression analysis found vaccination uptake was positively associated only with the availability of vaccines in schools. CNA produced different findings and uncovered an additional solution path: high vaccination rates were achieved by either (1) offering the vaccine in all schools or (2) a combination of offering the vaccine in some schools and media coverage. Conclusions. CNA offers a new comparative approach for researchers seeking to understand how implementation conditions work together and link to outcomes. Health Economics & Outcomes Research Coincidence Analysis Configurational Comparative Methods causal inference comparative analysis Figures Figure 1 Contributions To The Literature Coincidence Analysis (CNA) represents a new mathematical, cross-case method for researchers evaluating the implementation of complex interventions in dynamic settings. CNA can address multiple dimensions of real-world complexity, including conjunctivity (where several conditions must be jointly present to bring about an outcome) and equifinality (where different paths can lead to the same outcome). CNA can also detect causal chains, where conditions lead to an intermediary outcome, which then leads to the final outcome. Intentionally designed to investigate different hypotheses and uncover different properties of causal structures than more traditional approaches, CNA can identify implementation-related findings that might otherwise go undetected. Background One of the basic analytic challenges within implementation science is to study and understand implementation within real-world, dynamic settings. Implementation of multifaceted interventions typically involves many diverse elements working together in interrelated ways, including intervention components, implementation strategies, and features of local context. Furthermore, boundaries between an intervention, its implementation, and its contextual features can prove difficult to discern in practice. (1, 2) For researchers seeking to explain these complex relationships encountered in real-world settings, causal inference can play an important role. Since the mid-1980s, Configurational Comparative Methods (CCMs) have increasingly been recognized as effective methods for causal inference, especially in the social sciences. The cumulative number of CCM-related publications listed in the core collection of the Web of Science (3) has dramatically escalated in recent years, with more total publications appearing during the three-year period from 2017-2019 than in the entire preceding 22-year period between 1995-2016. CCMs have also started to make prominent appearances within the health services research and implementation science literatures. CCMs, for example, were used in a recent Cochrane Review to identify conditions directly linked with successful implementation of school-based interventions for asthma self-management (4); featured as an innovative member of the mixed-methods repertoire in a major methodological review in public health (5); highlighted as a central method in newly-published implementation science protocols(6, 7); and applied to determine different pathways for federally-qualified health centers to achieve patient-centered medical home status (8). CCMs were also featured in a dedicated chapter in the 2020 Handbook on Implementation Science (9). CCMs are designed to investigate different hypotheses and uncover different properties of causal structures than traditional regression analytical methods (RAMs).(10, 11) Qualitative Comparative Analysis (QCA) is one kind of CCM that, to date, has been most frequently applied in implementation science and health services research. The purpose of this article is to introduce a new CCM to the implementation research community: Coincidence Analysis. Coincidence Analysis (CNA) is a mathematical, cross-case approach that can be applied as a standalone method or in conjunction with other methods (including RAMs) to support causal inference, and is available via the R-package cna.(12-14) CNA offers a new cross-case method for implementation and health services researchers exploring causality when evaluating or implementing multifaceted interventions in complex contexts. Investigators applying CNA can conduct analyses across entire datasets to identify specific combinations of components and conditions that consistently lead to outcomes, and can be applied to large-n as well as small-n studies. Peer-reviewed, implementation-related work specifically involving CNA has started to emerge in implementation science, including podium presentations at major implementation conferences (15, 16); methods workshops dedicated specifically to CNA (17); published protocols (18); and full-length articles in established journals (19-23). CNA is a new comparative approach that can be used by the implementation research community to support causal inference, answer research questions about conditions that are minimally necessary or sufficient, and identify multiple causal paths to an outcome. We present this article in three parts. In part 1, we establish the theoretical foundation for CCMs, define CNA as a method within the CCM family and describe what CNA (and CCMs) uniquely offer. In part 2, we illustrate CNA by applying the method to a publicly available dataset that was originally analyzed using RAMs. In part 3, we offer guidance for reporting CNA design and results, and we discuss the limitations and challenges of CNA. In additional files accompanying this article we provide detailed descriptions of the steps and coding used to conduct the analysis [see Additional file 1] and the analytic dataset used [see Additional file 2] along with the R script [see Additional file 3] to allow for independent replication and validation of results. Part 1: Laying the Theoretical Foundation for CCMs Defining causal inference in CCMs. CNA is one method within a class of CCMs used to model complex patterns of conditions hypothesized to contribute to an outcome within a set of data. CCMs search for causal relations as defined by a regularity theory of causality, according to which a cause is a “difference-maker” of its effect within a fixed set of background conditions. More specifically, X is a cause of Y if there exists a fixed configuration of background factors Φsuch that, in Φ, a change in the value of X is systematically associated with a change in Y. If X does not make a difference to Y in any Φ, X is redundant to account for Y and, thus, not a cause of Y. The most influential theory defining causation along these lines is Mackie's INUS-theory, (24) with refinements by Graßhoff and May (25) and Baumgartner. (26) An INUS condition of an outcome Y is an Insufficient but Necessary part of a condition that is itself Unnecessary but Sufficient for Y. To use a common example for illustrating INUS conditions: not every fire is caused by a short circuit—fires can also be started by, for example, arson or lightning. However, a short circuit in combination with other conditions – e.g., presence of flammable material and absence of a suitably placed sprinkler – is sufficient for a fire. In this example, the short circuit is an INUS condition: it is a necessary, but itself insufficient, part of a sufficient, but itself unnecessary, condition for a fire. This particular causal path to a fire includes the combination of three specific conditions: presence of a short circuit, presence of flammable material, and absence of a sprinkler. All three of these conditions are difference-makers, for if one of them is missing, the fire does not occur along this causal path. Regularity theories account for the Boolean properties of causation, which encompass three dimensions of complexity. The first is conjunctivity: to bring about an outcome, several conditions must be jointly present. For example, in a study of high performance work practices and front line health care worker outcomes, Chuang and colleagues (27) found that no single high performance work practice was alone sufficient to produce the outcome of high job satisfaction. Instead, a configuration consisting of creative input, supervisor support, and team-based work practices together accounted for 65 percent of highly satisfied front-line health care workers. (27) Chuang and colleagues identified a second configuration that also led to high job satisfaction: supervisor support, incentive pay, team-based work and flexible work. (27) Both configurations resulted in high job satisfaction independently of each other. These configurations illustrate equifinality, a second dimension of complexity where different paths can lead to the same outcome. The third dimension of complexity is sequentiality: outcomes tend to produce further outcomes, propagating causal influence along causal chains. For instance, high job satisfaction of health care workers may, in turn, promote patient satisfaction. (28) Why use CCMs in implementation research? CCMs study different properties of causal structures than RAMs and thus are appropriate for exploring different types of hypotheses. RAMs examine statistical properties characterized by probabilistic or interventionist theories of causation. In the probabilistic framework, X is a cause of Y if, and only if the probability of Y given X is greater than the probability of Y alone and there does not exist a further factor, Z, that explains (i.e., neutralizes) the probabilistic dependence between X and Y. (29, 30) In the interventionist framework, X causes Y if, and only if there exists an intervention on X that changes the outcome Y while causes on other paths to Y are fixed. The interventionist theory of causation is counterfactual in that a case cannot simultaneously “receive” and “not receive” an intervention; instead, the intervention model maps possible values of Y onto possible values of X, focuses on how variables X and Y relate to one another, and generates average treatment effects over a population. (11, 31) Conversely, CCMs trace Boolean properties of causal structures as described by regularity theories of causation, according to which X is a cause of Y if, and only if X is an INUS condition of Y (see INUS definition above). (11, 24) CCMs study implication hypotheses that link specific values of variables as “X=χ i is (non-redundantly) sufficient/necessary for Y=γ i .” (11, 14). In this way, CCMs, including CNA, model the effect of conditions (e.g., high degree of X) on outcomes. This is a fundamentally different vantage point than the one adopted by RAMs which examine covariation hypotheses that link variables. Further, CCMs are case-oriented methods, in which observations consist of bounded, complex entities (e.g., organizations) that are considered as a whole (32) A case-based unit of analysis differs from the approach taken in RAMs, where cases are deconstructed into a series of variables, and estimates represent the net effect of a variable for the average case. As CNA and other CCMs employ a case-based approach and thus can be used to identify which interventions work in an array of contexts, they present opportunities for implementation and health services research questions in particular.. Different types of CCMs While CCMs have a common regularity theoretic foundation, different CCMs rely on different a priori conceptions of outcome and causal factors and build causal models in different ways. For example, Qualitative Comparative Analysis (QCA), in its standard implementation that uses the Quine-McCluskey (QMC) algorithm, (33, 34) requires identification of exactly one factor as outcome. It begins by identifying maximal sufficient and necessary conditions of the outcome, which are subsequently minimized using standard inference rules from Boolean algebra to arrive at a redundancy-free solution composed of INUS conditions of the outcome. (10) However, the QMC algorithm was not originally designed for causal inference. One consequence is that the non-observation of cases instantiating empirically possible configurations of the analyzed factors, also known as limited diversity, forces QMC to draw on counterfactual reasoning that goes beyond available data, sometimes requires assumptions contradicting the very causal structures under investigation(35), and regularly fails to completely eliminate redundancies in the presence of noise.(14) Moreover, QMC has built-in protocols for ambiguity reduction when multiple models fit the data equally well. Viable models are often eliminated to reduce ambiguity without justification, which is problematic for causal discovery.(36)[ Advantages of using CNA. Coincidence Analysis (CNA) is a new addition to the family of CCMs.(37, 38) It uses an algorithm specifically designed for causal inference, thus avoiding the problems mentioned above. In particular, it does not build causal models by means of a top-down approach that first searches for maximal sufficient and necessary conditions and then gradually minimizes them using the QMC algorithm. Rather, CNA employs a bottom-up approach that first tests single factor values for sufficiency and necessity, and then tests combinations of two, three, etc.(13, 14) All sufficient and necessary conditions revealed by this approach are automatically minimal and redundancy-free. Additionally, CNA is designed to treat any number of factors as endogenous and is therefore capable of analyzing causal chains, or common-cause structures.(39) For example, Baumgartner and Epple (2014) found in a Swiss policy analysis that certain population, economic or political characteristics in some areas led to a higher rate of prejudice and, in turn, discriminatory policy. (39) Analyzing causal chains may be advantageous if, for example, an intervention A occurs as a result of other factors but is not the ultimate outcome of interest. Identifying the full causal model, including which factors produce A on the path to the ultimate outcome of interest, is valuable when seeking to understand causal complexity. CNA is the only member of the CCM family that builds and evaluates models representing causal chains. Part 2: Demonstrating CNA using Publicly Available Data Data source In March 2016, Rehn and colleagues reported the impact of implementation strategies on human papillomavirus (HPV) "catch-up vaccination" uptake in Sweden among 5th and 6th grade girls.(40) The purpose of the original study was to estimate the impact of various information channels and delivery settings on county-level catch-up vaccine uptake to inform future vaccination campaigns in Sweden. The authors obtained county-level data on catch-up vaccinations and the eligible population from administrative data. They collected implementation strategies from county health care offices via an open-ended questionnaire emailed in 2012 asking respondents to list and describe “information channels” used to reach eligible girls and the settings in which they offered the vaccine. A subsequent phone interview was conducted in 2014 to update the lists. Rehn and colleagues used regression analysis to estimate county-level catch-up vaccine uptake as a function of information channels and delivery settings. The authors concluded that the availability of vaccines in schools explained differences in county-level vaccine uptake; no information channels were found to make a difference in uptake. Rehn and colleagues defined the outcome and predictor variables as follows: Outcome variable. County-level catch-up vaccine uptake was defined as the percent of eligible girls born between 1993-1998 who received at least one dose of vaccine by 2014. Predictor variables. Ten variables represented information channels and four variables represented the delivery settings where the vaccinations were available (some schools, all schools, primary health care centers, and other health care centers). All 14 factors were dichotomized with values of 1 (present) or 0 (absent). All county-level data on vaccine uptake, information channels and delivery settings used for the CNA illustration were reported in the article. Methods We re-analyzed the data using CNA. A step-by-step guide for conducting CNA, using this study as an illustration, is provided in a document accompanying this article [see Additional file 1] as well as the analytic dataset [see Additional file 2] and the R script [see Additional file 3] used in the analysis. Step 1: Define, calibrate and select the factors (i.e., outcomes and conditions) to create a data set. Vaccination rate represented the outcome of interest and ranged from 49% to 84% across the 21 counties. We selected 65% as the threshold defining “high” catch-up vaccination rates after conducting sensitivity analyses in which we varied the threshold for “high-uptake,” using two different existing break points in the data: the 65% cut-off generated the greatest diversity among cases for the conditions and the outcome and yielded a sufficiently high number of cases featuring the outcome. We coded the 21 counties into a new dichotomous outcome called HI_UPTAKE where 1 = "catch-up vaccination rate of 65% or higher" and 0 = "catch-up vaccination rate less than 65%." (See Additional file 1 for details on the rationale for each step in the analytic process). A secondary analysis identified conditions leading to the absence of the outcome (HI_UPTAKE = 0) because conditions that prevent the outcome may differ from those that contribute to the presence of the outcome. We prepared a dataset that included the uptake rates, delivery settings and implementation strategies as reported by Rehn and colleagues (see Additional file 2 to view the analytic dataset used in the Coincidence Analysis [see Additional file 2]). We transformed a number of factors from the original dataset for use with CNA because in the original dataset these factors had characteristics unsuitable for CCM processing. For instance, the original data set contained the factor “Primary health care centre” (PHC) that was constantly present in all 21 counties (cases). Constant factors like PHC can be automatically excluded as difference-makers. Another delivery setting, “Other health care center;” (HC) was eliminated given limited variation across cases. We combined “All schools” (“organized delivery of the vaccine in all schools in the county;” AS=1) and “Some schools” (“organized delivery of the vaccine in schools in some of the municipalities in the county…;” SS=1) into a new multi-value ordinal factor called "SCHOOLS," where SCHOOLS = 0 if AS = 0 and SS = 0; SCHOOLS = 1 if SS = 1 and AS = 0; and SCHOOLS = 2 if AS = 1. The resulting dataset included 12 potential explanatory factors. These factors could be combined into 6144 logically possible configurations, which could not be covered to an informative degree by the 21 cases included. Thus, the diversity index for the original data, i.e. the ratio of observed configurations to all possible configurations, was exceedingly small. The smaller the diversity index, the more challenging it is to draw informative configurational conclusions. To improve the diversity index, we included a subset of the 12 exogenous factors in our analysis. This is analogous to maximizing degrees of freedom in RAM. We selected schools (SCHOOLS) and four of the ten information channels to include: school-based information (SBI), media coverage (MC), social media (SM), and Cinema commercial/YouTube (CCY). Our rationale for choosing these four implementation strategies was that they were directly linked to school and digital media, two immersive domains that are dynamic and interactive where students and their parents commonly encounter new information, and thus likely to be effective channels for conveying information about why, where, how, and when to access vaccinations In our initial analysis plan, three of the seven cases (counties) exhibiting the outcome (high-uptake) instantiated exactly the same configuration of conditions, leaving only five observed configurations featuring HI_UPTAKE=1 out of a total of 48 logically possible configurations. As we could not justify removing one of the four selected information channels on a theoretical basis alone, but still wanted to decrease the number of overall factors in the analytic dataset (and thus increase the diversity index), we decided to assess if we could combine two of the information channels into a single “meta-factor” — a common approach in CCMs to reduce the number of conditions without eliminating either of the properties represented by these conditions from the analysis. There are six possible ways to pair four different information channels. Accordingly, we created six different datasets (i.e., analytic samples), each representing a different pairing of two channels, the two remaining channels and the outcome HI_UPTAKE. We coded the new meta-factor in each analytic sample with the value 1 if, and only if at least one of the two aggregated channels was present in a county (To view these six datasets, see Additional file 2.) Step 2. Perform CNA using the cna package in R.(13) Two parameters of fit – consistency and coverage – provide insight into the strength of the dependence between conditions and the outcome. Consistency, with a score ranging from 0 to 1, measures the degree to which the cases that instantiate a configuration or a whole model also instantiate the outcome. (10) Low consistency values indicate that the dependence between conditions or models and the outcome is far away from a strict Boolean (deterministic) dependence. Coverage scores range from 0 to 1 and represent the proportion of cases with the outcome that also instantiate a particular configuration or whole model. Coverage measures a given configuration’s or model's empirical importance based on the available data.(10) When applying CNA to the dataset, we set our minimum consistency and coverage scores to 1.0. (10)However, it is important to note that CNA, and CCMs generally, risk overfitting model solutions when searching for maximal consistency and coverage thresholds, which can lead to false positives.(41) . For this reason, consistency and coverage thresholds should only be set to 1 if researchers have strong reasons to assume that the data quality is very high (i.e. low levels of noise and measurement error); if that cannot be assumed, then researchers can apply a search strategy that systematically varies consistency and coverage thresholds to measure fit-robustness, or the degree to which a model solution agrees with other models identified at different consistency and coverage thresholds in the same dataset. (42) Results Step 3. Interpret results and refine model inputs if necessary. Like QCA, CNA encourages an iterative approach where researchers can run analyses, interpret results and redefine model inputs before finalizing a model set.(32) As such, we discuss our interpretation of the findings as our iterative analyses progress. Our analyses produced model ambiguity, meaning that the data were insufficient to determine exactly which causal structure was operative. Of the six analytic samples, three datasets yielded a total of five causal models for HI_UPTAKE=1, all of which featured maximal consistency and coverage scores (see the R replication script for a complete list of models). All models had the following identical terms as part of their solution, where "+" symbolizes the Boolean operator OR, "*" symbolizes AND, and "↔" expresses sufficiency and necessity: SCHOOLS=2 + SCHOOLS=1 * MC= 1 ↔ HI_UPTAKE=1 (1) The above expression (1) translates to: “counties had high catch-up vaccination rates if, and only if they offered vaccination in all schools OR offered vaccination in some (but not all) schools AND used a media coverage implementation strategy." All five causal models resulting from our analysis were supersets of (1), so we concluded that the factor values contained in (1) were causally relevant for high-uptake. However, (1) was not a complete model because it only achieved a consistency score of 0.875 (i.e., only 87.5% of counties with this configuration were high-uptake counties). Expression (1) covers a county, Skåne, that exhibited the configuration SCHOOLS=1 * MC=1 but was not high-uptake. The other counties with this configuration, SCHOOLS=1 * MC=1, were always associated with high-uptake of vaccines. Thus, other factors unique to Skåne must be missing from expression (1). While our data were insufficient to identify the factors that distinguished Skåne from other counties with the same configuration, the five complete models inferred by CNA provided five possible explanations. Interestingly, all five of these solution models combined SCHOOLS=1 * MC=1 with the absence of other information channels. Only when SCHOOLS=1 * MC=1 was accompanied by SM=0 or SBI=0 or CCY=0 was it associated with high uptake with 100% consistency, which could indicate these other information channels produced a “backfiring” effect. After further reviewing data for each county, we determined that the most plausible implementation strategy to backfire was cinema commercials and/or YouTube. Online media like YouTube can backfire because these open-source platforms contain unvalidated content that can appear automatically through newsfeeds or advertisements, potentially overriding legitimate health-related information. (43) Furthermore, we observed a negative relationship between the presence of CCY and the vaccination rate outcome in the underlying dataset. As Figure 1 shows, the presence of CCY was relatively well-represented in the overall dataset: 8 of the 21 cases had CCY=1. In 7 of these 8 cases, HI_UPTAKE=0. The lone exception was the county of Jonkoping, where the sufficient condition of SCHOOLS=2 was also present. For these reasons, we deemed the following complete model to be the most plausible: SCHOOLS=2 + SCHOOLS=1 * MC=1 * CCY=0 ↔ HI_UPTAKE=1 (2) Expression (2) translates to: “counties had high catch-up vaccination rates if, and only if they offered vaccination in all schools OR offered vaccination in some (but not all) schools AND used a media coverage implementation strategy but not cinema commercials/YouTube." Expression (2) had perfect consistency and coverage scores (1.0, respectively) and differentiated Skåne (a county without high uptake) from high uptake counties that also had vaccinations available at some schools and used media coverage. Figure 1 highlights the configurations instantiating this model in the data. Figure 1. Overall model identified by Coincidence Analysis with 100% consistency and 100% coverage. [insert Figure 1 here] Applying the same analytic approach to model the absence of the outcome yielded seven models for HI_UPTAKE=0 such that six of these seven models had a common core that corresponded exactly to the negation of the core of the positive models: SCHOOLS=0 + SCHOOLS=1 * MC=0 ↔ HI_UPTAKE=0 (3) Expression (3) translates to: “counties had no high catch-up vaccination rates if, and only if they did not offer vaccination in all schools OR offered vaccination in some (but not all) schools AND did not use a media coverage implementation strategy." Expression 3 exhibited 1.0 consistency and 0.93 coverage. Taken together, these results provide substantive evidence that media coverage is relevant for differentiating counties with and without high vaccination uptake, adding additional information to the regression results of the original study. Discussion The results of this CNA indicate that, under specific conditions, information channels made a difference for high vaccination uptake. This contrasts with the results of the regression analysis from the original study, which concluded that information channels made no difference in increasing vaccination uptake. Our results imply that the availability of vaccination in some schools is only sufficient for high vaccination rates if media coverage is employed and certain other communication channels are not used. In other words, the data contain enough evidence to infer that when vaccination is available at some but not all schools, availability must be complemented by media coverage to achieve high uptake. The data do not contain enough evidence, however, to have absolute certainty which communication channels should be avoided. Even so, cinema commercials and/or YouTube might be the most plausible information channel to avoid; YouTube in particular might backfire and reduce vaccination rates as a result of unsolicited content that undermines county-sanctioned media coverage on vaccines. Closely examining data from individual cases (counties) corroborate the theory of some conditions backfiring with respect to producing high vaccination uptake. Jonkoping provided vaccination in all schools, but only achieved vaccination uptake among 65% of eligible girls, as opposed to over 80% uptake achieved in the three other counties with vaccination availability at all schools. Notably, Jongkoping used two communication channels (CCY=1 and SM=1) that were absent in the other three counties (CCY=0 and SM=0). In sum, the CNA results indicate that whenever vaccination is available at only some schools, media coverage make a difference for high uptake. Furthermore, the common core of the resulting CNA models for HI_UPTAKE=0 indicate that a lack of media coverage when vaccinations are provided only in some schools make a difference for lower vaccine uptake. By systematically scrutinizing the configurations of implementation strategies associated with high-uptake and low-uptake counties, CNA extends the conclusions drawn from Rehn and colleagues’ regression model. The data on vaccination uptake in Sweden do not comprise multiple outcomes and, hence, CNA's capacity to uncover multi-outcome structures cannot be showcased with this example. Readers interested in multi-outcome discovery with CNA are, instead, referred to (39) and (44), the first of which finds a causal chain and the second a common-cause structure. Our approach to setting consistency and coverage thresholds in this study is consistent with the extant CNA literature. Of note, however, is that ensemble strategies have been newly proposed where consistency and coverage thresholds are systematically varied across a series of thresholds in order to measure “fit-robustness,” the degree to which a specific model agrees with other models identified at different consistency and coverage thresholds in the same dataset. (42) While ensemble approaches to setting consistency and coverage thresholds are still nascent, they appear highly promising as analytic strategies that can help mitigate the risk of overfitting models .(41) Part 3: Tips for Practitioners Recommendations for reporting analyses and results for CCMs Variation in reporting study design, analysis and results exists in previously published peer-reviewed CCM literature. To advance the methodological rigor with which CCMs are applied, we offer recommendations for describing design, analysis and results for CCMs. We also provide additional material in Additional files 1 and 2 accompanying this article to allow for independent replication of these analyses and findings [Additional file 1][Additional file 2]. We suggest that future studies and publications applying CCMs: 1) describe the rationale for using the CCM (e.g., CNA); 2) describe the rationale for selecting which factors (outcomes and conditions) to include in the analysis; 3) describe the process used to assign cases to factor values (45, 46) (e.g., high vaccination uptake); 4) specify the software (and version) used for analysis; 5) describe the iterative analyses used to refine factors (e.g., different approaches to calibration) and models (e.g., adding/dropping factors); 6) list the number of models generated in each iteration of the analysis and identify commonalities across models; 7) report consistency and coverage thresholds for final models along with ranges for models not part of the final model(s); and 8) describe the rationale for selecting the final model(s). In this article, we used conceptual knowledge of HPV vaccination uptake and the existence of variability across cases to select the factors to include in the initial analysis. Given the structure of the data, we relied on the original binary factor calibration for all but one factor (SCHOOLS), which was calibrated as a multi-value ordinal factor. Moreover, to improve the diversity index, we explored all mutually exclusive combinations of two factors representing information channels into a single factor. We analyzed six datasets using the cna package in R.(13) Five models reached perfect consistency and coverage and, thus, fit the data equally well. All models exhibited a common core, which we reported as determinate causal inference supported by our data. At the same time, we acknowledge that the common core did not amount to a complete causal model. Based on theoretical considerations, which we explicitly described, we then selected one of five viable completions as the most plausible overall model. We reported all relevant model fit scores for our preferred model and provide additional files with detailed descriptions of each step in the analytic process [see Additional file 1], the analytic dataset [see Additional file 2] and the R script [see Additional file 3] to allow for independent replication and verification of our results. Limitations of CNA CNA has several limitations of which implementation researchers should be aware. First, although CNA supports causal inference, there are limitations to the extent to which results may be generalized. Results can be confounded by unmeasured causes that are located on causal paths to outcomes that do not go through any measured factors. If the data cannot be assumed to be homogenous in confounder distributions—meaning that unmeasured factors do not affect all cases/configurations equally—generalization becomes problematic given the risk of over-interpreting the data or incorrectly inferring a causal relationship. As with other CCMs, familiarity with cases helps to evaluate generalizability—by, for example, justifying that cases included in the analysis are homogeneous with respect to potential confounders—and to interpret solutions generated by mathematical modeling. In the example dataset from Rehn and colleagues, it is possible a third factor may explain the negative relationship between cinema commercials and/or YouTube, for example. In this instance, researchers can further explore the relationship between these factors and HPV vaccination uptake through qualitative interviews or other methods to confirm findings. Second, CCMs rarely uncover the data-generating causal structures in full. Configurational data analyzed in observational studies tend to be fragmented (i.e., exhibit low diversity), so most logically possible combinations of conditions are not present in the observed cases. Under these circumstances, CCMs may reveal only portions of the underlying causal structures. Thus, the fact that some factor X is not contained in a CCM model must not be taken to mean that X is causally irrelevant. Unless there is reason to assume that the data are non-fragmented, the absence of X from a model can only mean that the data do not contain evidence for X's relevance (which is not the same as X's irrelevance). Conclusions CNA offers new insights of potential high interest to implementation researchers. We demonstrated the utility of CNA using data previously analyzed with RAMs. The authors of the original, RAM-based analysis indicated that offering vaccination in schools increased county-level vaccination uptake while no information channels significantly increased vaccination uptake.(40) By contrast, CNA results indicated that under specific conditions, information channels did make a difference for high vaccination uptake. Specifically, our results imply that higher vaccination rates are achieved by either (1) offering the vaccine in all schools or (2) offering the vaccine in some schools and using media coverage but not certain other communication channels. Compared to RAMs, CCMs have fundamentally different methodological goals and search for different properties of causal structures. CCMs and RAMs answer distinct types of questions. RAMs are useful for estimating the average influence of a specific variable on an outcome while holding other variables constant. CCMs are useful for identifying combinations of specific conditions that may be on the same or different causal paths (i.e., are minimally necessary or sufficient) to an outcome. Of the two CCMs, CNA was built expressly for causal inference and can be used to uncover causal chains underlying the data.(13, 14, 39) CNA has the potential to offer implementation researchers alternative and more nuanced knowledge about causal relationships when examining complex interventions in settings with interdependent or interrelated factors. In particular, CCMs are well-suited for implementation and health services research questions regarding the implementation of multifaceted interventions in complex, real-world settings, the dynamics of which can be influenced by many factors acting in combination. Declarations Ethics approval and consent to participate Not applicable Consent for publication Not applicable Availability of data and materials The full dataset used in this analysis along with detailed descriptions of each step of the analysis and the R script are all available for review and replication in additional files accompanying this article [see Additional file 1][see Additional file 2][Additional file 3]. The original dataset is publicly available within the following open-access article: Rehn M, Uhnoo I, Kühlmann-Berenzon S, Wallensten A, Sparén P, Netterlid E. Highest vaccine uptake after school-based delivery - a county-level evaluation of the implementation strategies for HPV catch-up vaccination in Sweden. PLoS One. 2016;11(3):e0149857. doi:10.1371/journal.pone.0149857. Competing interests The authors declare that they have no competing interests. Funding This secondary analysis was conducted by the investigators without additional funding. Authors' contributions RGW, SB, NS, LD and AS originated the idea for the project and convened the full team of co-authors. RGW found the published dataset that we used for secondary analysis. RGW, MB, and EJM took lead roles in conducting the Coincidence Analysis. MB wrote the R script provided in Additional file 3. RGW, SB, NS, LD, AS, MS, AT, DC and EJM participated in a series of discussions to review preliminary results from the Coincidence Analysis and provide direct feedback to inform ongoing analyses. RGW, SB, NS, LD, AS, MS, AT, DC and EJM participated in writing sections of the main manuscript plus the step-by-step guide provided in Additional file 1, as well as editing drafts. All authors read and approved the final manuscript. Acknowledgements The authors wish to thank the Regenstrief Institute for its pioneering role in recent years in hosting and organizing seminars that allow interested health researchers and implementation scientists to learn more about applying Configurational Comparative Methods (including Coincidence Analysis) in their work.Alrik Thiem gratefully acknowledges financial support from the Swiss National Science Foundation, grant number PP00P1_170442 . Sarah Birken’s effort was supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant KL2TR002490. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. Reference List Pfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A, Hofmann B, et al. Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework. Implement Sci. 2017;12. Council MR. A framework for development and evaluation of RCTs for complex interventions to improve health. 2000. database WoSCC. Philadelphia: Clarivate Analytics; 2020. Palinkas LA, Mendon SJ, Hamilton AB. Innovations in Mixed Methods Evaluations. Annu Rev Public Health. 2019;40:423-42. Harris K, Kneale D, Lasserson TJ, McDonald VM, Grigg J, Thomas J. School-based self-management interventions for asthma in children and adolescents: a mixed methods systematic review. Cochrane Database Syst Rev. 2019;1:CD011651. Rogal SS, Yakovchenko V, Morgan T, Bajaj JS, Gonzalez R, Park A, et al. Getting to implementation: a protocol for a Hybrid III stepped wedge cluster randomized evaluation of using data-driven implementation strategies to improve cirrhosis care for Veterans. Implement Sci. 2020;15(1):92. Bunger AC, Chuang E, Girth A, Lancaster KE, Gadel F, Himmeger M, et al. Establishing cross-systems collaborations for implementation: protocol for a longitudinal mixed methods study. Implement Sci. 2020;15(1):55. Mendel P, Chen EK, Green HD, Armstrong C, Timbie JW, Kress AM, et al. Pathways to Medical Home Recognition: A Qualitative Comparative Analysis of the PCMH Transformation Process. Health Serv Res. 2018;53(4):2523-46. Birken SA, Nilsen, P., Cragun, D. Configurational Comparative Methods. In: Sarah A. Birken PN, editor. Handbook on Implementation Science: Edward Elgar Publishing; 2020. p. 569. Thiem A. Conducting Configurational Comparative Research With Qualitative Comparative Analysis: A Hands-On Tutorial for Applied Evaluation Scholars and Practitioners. Am J Eval. 2017;38(3):420-33. Rohlfing I, Zuber CI. Check your truth conditions! Clarifying the relationship between theories of causation and social science methods for causal inference. Sociological Methods & Research. 2019:0049124119826156. Thiem A, Baumgartner M, Bol D. Still lost in translation! A correction of three misunderstandings between configurational comparativists and regressional analysts. Comparative Political Studies. 2016;49(6):742-74. Ambühl M, Baumgartner M, Epple R, Thiem A. cna: Causal Modeling with Coincidence Analysis. 2020. Baumgartner M, Ambühl M. Causal modeling with multi-value and fuzzy-set Coincidence Analysis. Political Science Research and Methods. 2020;8(3):526-42. Miech E, Rattray N, Damush T. Necessary but not sufficient: A multimethod study of the role of champions in heathcare-related implementation. Implement Sci. 2020;15. Cragun D, Rahm, AK. Coincidence Analysis: a methodology to identify contextual conditions influencing implementation across multiple settings. Implement Sci. 2019;14(Proceedings from the 11th Annual Conference on the Science of Dissemination and Implementation):27. Birken S, Damschroder, L., Miech, E., Cragun, D., editor Agenda for the 12th Annual Conference on the Science of Dissemination and Implementation. Dissemination and Implementation Conference; 2019; Arlington, VA. Rahm AK, Cragun D, Hunter JE, Epstein MM, Lowery J, Lu CY, et al. Implementing universal Lynch syndrome screening (IMPULSS): protocol for a multi-site study to identify strategies to implement, adapt, and sustain genomic medicine programs in different organizational contexts. Bmc Health Serv Res. 2018;18. Petrik AF, Green B, Schneider J, Miech EJ, Coury J, Retecki S, et al. Factors Influencing Implementation of a Colorectal Cancer Screening Improvement Program in Community Health Centers: an Applied Use of Configurational Comparative Methods. J Gen Intern Med. 2020;35(Suppl 2):815-22. Dy SM, Acton RM, Yuan CT, Hsu YJ, Lai AY, Marsteller J, et al. Association of Implementation and Social Network Factors With Patient Safety Culture in Medical Homes: A Coincidence Analysis. J Patient Saf. 2020. Hickman SE, Miech EJ, Stump TE, Fowler NR, Unroe KT. Identifying the Implementation Conditions Associated With Positive Outcomes in a Successful Nursing Facility Demonstration Project. Gerontologist. 2020. Yakovchenko V, Miech EJ, Chinman MJ, Chartier M, Gonzalez R, Kirchner JE, et al. Strategy Configurations Directly Linked to Higher Hepatitis C Virus Treatment Starts An Applied Use of Configurational Comparative Methods. Med Care. 2020;58(5):E31-E8. Moret W, Lorenzetti L. Realistic expectations: exploring the sustainability of graduation outcomes in a program for children affected by HIV in Kenya's Northern Arid Lands. Vulnerable Child You. 2020. Mackie JL. The Cement of the Universe: A Study of Causation. Oxford: Clarendon Press; 1974. Graßhoff G, & May, M. Causal regularities. Paderborn, Germany: Mentis-Verlag; 2001. 85-114 p. Baumgartner M. Regularity Theories Reassessed. Philosophia. 2008;36(3):327-54. Chuang E, Dill J, Morgan JC, Konrad TR. A Configurational Approach to the Relationship between High-Performance Work Practices and Frontline Health Care Worker Outcomes. Health Services Research. 2012;47(4):1460-81. Sikorska-Simmons E. Linking resident satisfaction to staff perceptions of the work environment in assisted living: A multilevel analysis. Gerontologist. 2006;46(5):590-8. Suppes P. A Probabilistic Theory of Causality: North-Holland Pub. Co.; 1970. 409-10 p. Baumgartner M. Causation. In: Dirk Berg-Schlosser BB, Leonardo Morlino, editor. The SAGE Handbook of Political Science. 01. London: SAGE Publications; 2020. p. 305-21. Woodward J. Making Things Happen: A Theory of Causal Explanation. New York, New York: Oxford University Press, Inc; 2003. Ragin CC. The Dialogue of Ideas and Evidence in Social Research. The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies. Okland, California: University of California Press; 1987. Quine WV. On Cores and Prime Implicants of Truth Functions. The American Mathematical Monthly. 1959;66(9):755-60. McCluskey EJ. Introduction tothe Theory of Switching Circuits: McGraw-Hill; 1965. 318 p. Baumgartner M. Parsimony and Causality. Qual Quant. 2015;49(2):839-56. Baumgartner M, Thiem A. Model Ambiguities in Configurational Comparative Research. Sociological Methods & Research. 2017;46(4):954-87. Baumgartner M. Uncovering deterministic causal structures: a Boolean approach. Synthese. 2009;170(1):71-96. Baumgartner M. Inferring Causal Complexity. Sociological Methods & Research. 2009;38(1):71-101. Baumgartner M, Epple R. A Coincidence Analysis of a Causal Chain The Swiss Minaret Vote. Sociological Methods & Research. 2014;43(2):280-312. Rehn M, Uhnoo I, Kuhlmann-Berenzon S, Wallensten A, Sparen P, Netterlid E. Highest Vaccine Uptake after School-Based Delivery - A County-Level Evaluation of the Implementation Strategies for HPV Catch-Up Vaccination in Sweden. PLoS One. 2016;11(3):e0149857. Arel-Bundock V. The Double Bind of Qualitative Comparative Analysis. Sociological Methods & Research. 2019:0049124119882460. Parkkinen V, Baumgartner M. Robustness and Model Selection in Configurational Causal Modeling. Sociological Methods & Research. 2020. Moorhead SA, Hazlett DE, Harrison L, Carroll JK, Irwin A, Hoving C. A new dimension of health care: systematic review of the uses, benefits, and limitations of social media for health communication. J Med Internet Res. 2013;15(4):e85. Haesebrouck T. Who follows whom? A coincidence analysis of military action, public opinion and threats. Journal of Peace Research. 2019;56(6):753-66. Thiem A. Membership function sensitivity of descriptive statistics in fuzzy-set relations. Int J Soc Res Method. 2014;17(6):625-42. Verkuilen J. Assigning membership in a fuzzy set analysis. Sociological Methods & Research. 2005;33(4):462-96. Supplementary Files CoincidenceAnalysisAdditionFile1wRefsAP11.17.docx Word .doc file. Includes detailed descriptions of each step performed in the Coincidence Analysis. 6CoincidenceAnalysisAdditionalfile2.docx Word .doc file. Provides the full analytic dataset used in this analysis to allow for independent replication and verification of results. 7CoincidenceAnalysisAdditionalfile3.docx Word .doc file. Provides the complete R script used in this analysis to support independent replication. Cite Share Download PDF Status: Published Journal Publication published 11 Dec, 2020 Read the published version in Implementation Science → Version 2 posted Editorial decision: Accept 29 Nov, 2020 Editor assigned by journal 19 Nov, 2020 Submission checks completed at journal 19 Nov, 2020 Editor invited by journal 19 Nov, 2020 You are reading this latest preprint version Show more versions Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About In Review Editorial Policies Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-58815","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Methodology","associatedPublications":[],"authors":[{"id":5130621,"identity":"b60d31e6-9e2d-4588-8f91-99ceea7d7655","order_by":0,"name":"Rebecca Garr Whitaker","email":"","orcid":"","institution":"Duke-Margolis Center for Health Policy","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Rebecca","middleName":"Garr","lastName":"Whitaker","suffix":""},{"id":5130622,"identity":"e6bbf925-08c5-49df-b314-fc6c8f94ae1f","order_by":1,"name":"Nina Sperber","email":"","orcid":"","institution":"Duke University School of Medicine","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Nina","middleName":"","lastName":"Sperber","suffix":""},{"id":5130623,"identity":"10c00d1a-3b88-4a44-9531-f0b559c48a3f","order_by":2,"name":"Michael Baumgartner","email":"","orcid":"","institution":"Universitetet i Bergen","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Michael","middleName":"","lastName":"Baumgartner","suffix":""},{"id":5130624,"identity":"4a2e9ca1-e63c-43c7-a256-968c61a73fce","order_by":3,"name":"Alrik Thiem","email":"","orcid":"","institution":"Universitat Luzern","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Alrik","middleName":"","lastName":"Thiem","suffix":""},{"id":5130625,"identity":"dd7dc687-dbb1-490d-b5a4-0fe94b16107c","order_by":4,"name":"Deborah Cragun","email":"","orcid":"","institution":"University of South Florida","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Deborah","middleName":"","lastName":"Cragun","suffix":""},{"id":5130626,"identity":"e72475de-6332-493c-abfe-96a96e075053","order_by":5,"name":"Laura Damschroder","email":"","orcid":"","institution":"VA Center for Clinical Management Research","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Laura","middleName":"","lastName":"Damschroder","suffix":""},{"id":5130627,"identity":"ddb4b8a7-2ebc-4b42-9b00-60fbaf32affa","order_by":6,"name":"Edward J. Miech","email":"","orcid":"","institution":"Regenstrief Institute Inc","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Edward","middleName":"J.","lastName":"Miech","suffix":""},{"id":5130628,"identity":"917f2437-8b1a-46fa-9eba-cca0e4ea8fd8","order_by":7,"name":"Alecia Slade","email":"","orcid":"","institution":"Avalere Health","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Alecia","middleName":"","lastName":"Slade","suffix":""},{"id":5130629,"identity":"e0a836eb-7feb-451d-a401-75ace3998ca2","order_by":8,"name":"Sarah A. Birken","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA9klEQVRIiWNgGAWjYDACCcYGxgYGBjkGBjYGZiBfhkECSPIQocUYpoWHh7AWBgaQlsQGorXIz25u+zij4k76huPHEh8XVNjx2Es3MD5424Zbi8Gdg80zN5x5lrvhTNph4xlnknl4ZA4wG87Fp0UisZnxYdvh3G0H0tukeduYgQ5LYAMy8DhsBkjLv8PpZueft//m/VcP0sL+G58WhhtALRsbDieY3Ug7xszbcBhsCzM+LSC/MM44dthw/41nydI8x47z8AANkZxzDo/DZrc/ZuypOSwv2Z9m+JmnplqOfUbywQ9vyvA4DAsAxdMoGAWjYBSMAooAAPH+U4tI7FngAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-0591-4800","institution":"University of North Carolina at Chapel Hill","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Sarah","middleName":"A.","lastName":"Birken","suffix":""}],"badges":[],"createdAt":"2020-08-13 11:22:16","currentVersionCode":2,"declarations":"","doi":"10.21203/rs.3.rs-58815/v2","doiUrl":"https://doi.org/10.21203/rs.3.rs-58815/v2","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s13012-020-01070-3","type":"published","date":"2020-12-11T15:01:54+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":3972834,"identity":"4da79436-ebc8-467a-ba9e-f37071ad104a","added_by":"auto","created_at":"2020-12-03 02:20:09","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":46074,"visible":true,"origin":"","legend":"Overall model identified by Coincidence Analysis with 100% consistency and 100% coverage.","description":"","filename":"1.PNG","url":"https://assets-eu.researchsquare.com/files/rs-58815/v2/b9eb71bc4f5de13b2e9ddd83.PNG"},{"id":13621663,"identity":"ee4818ff-6d17-4dbe-8b89-d4f331c9e839","added_by":"auto","created_at":"2021-09-17 07:11:34","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":274887,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-58815/v2/58fe3329-79f2-4ef2-8bdb-bdbbab8c9dbe.pdf"},{"id":3972833,"identity":"21435b49-75ec-4e3e-b208-7a1934ff01c6","added_by":"auto","created_at":"2020-12-03 02:20:09","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":38620,"visible":true,"origin":"","legend":"Word .doc file. Includes detailed descriptions of each step performed in the Coincidence Analysis.","description":"","filename":"CoincidenceAnalysisAdditionFile1wRefsAP11.17.docx","url":"https://assets-eu.researchsquare.com/files/rs-58815/v2/01e024a2e1fb6254ed289045.docx"},{"id":3972835,"identity":"c4984c97-d705-4ad7-8d86-eca3ae2d9f02","added_by":"auto","created_at":"2020-12-03 02:20:09","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":19641,"visible":true,"origin":"","legend":"Word .doc file. Provides the full analytic dataset used in this analysis to allow for independent replication and verification of results.","description":"","filename":"6CoincidenceAnalysisAdditionalfile2.docx","url":"https://assets-eu.researchsquare.com/files/rs-58815/v2/0a64f1645a5ea48890875d40.docx"},{"id":3972836,"identity":"3179e924-7e82-444e-9572-21103ce271be","added_by":"auto","created_at":"2020-12-03 02:20:09","extension":"docx","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":14701,"visible":true,"origin":"","legend":"Word .doc file. Provides the complete R script used in this analysis to support independent replication.","description":"","filename":"7CoincidenceAnalysisAdditionalfile3.docx","url":"https://assets-eu.researchsquare.com/files/rs-58815/v2/0e707bde9e58d92fe1735522.docx"}],"financialInterests":"","formattedTitle":"Coincidence Analysis: A New Method for Causal Inference in Implementation Science","fulltext":[{"header":"Contributions To The Literature","content":"\u003cul\u003e\n\u003cli\u003eCoincidence Analysis (CNA) represents a new mathematical, cross-case method for researchers evaluating the implementation of complex interventions in dynamic settings.\u003c/li\u003e\n\u003cli\u003eCNA can address multiple dimensions of real-world complexity, including conjunctivity (where several conditions must be jointly present to bring about an outcome) and equifinality (where different paths can lead to the same outcome). CNA can also detect causal chains, where conditions lead to an intermediary outcome, which then leads to the final outcome.\u003c/li\u003e\n\u003cli\u003eIntentionally designed to investigate different hypotheses and uncover different properties of causal structures than more traditional approaches, CNA can identify implementation-related findings that might otherwise go undetected.\u003c/li\u003e\n\u003c/ul\u003e"},{"header":"Background","content":"\u003cp\u003eOne of the basic analytic challenges within implementation science is to study and understand implementation within real-world, dynamic settings.\u0026nbsp; Implementation of multifaceted interventions typically involves many diverse elements working together in interrelated ways, including intervention components, implementation strategies, and features of local context.\u0026nbsp; Furthermore, boundaries between an intervention, its implementation, and its contextual features can prove difficult to discern in practice. (1, 2)\u003c/p\u003e\n\u003cp\u003eFor researchers seeking to explain these complex relationships encountered in real-world settings, causal inference can play an important role. Since the mid-1980s, Configurational Comparative Methods (CCMs) have increasingly been recognized as effective methods for causal inference, especially in the social sciences.\u0026nbsp; The cumulative number of CCM-related publications listed in the core collection of the Web of Science (3) has dramatically escalated in recent years, with more total publications appearing during the three-year period from 2017-2019 than in the entire preceding 22-year period between 1995-2016.\u003c/p\u003e\n\u003cp\u003eCCMs have also started to make prominent appearances within the health services research and implementation science literatures. CCMs, for example, were used in a recent Cochrane Review to identify conditions directly linked with successful implementation of school-based interventions for asthma self-management (4); featured as an innovative member of the mixed-methods repertoire in a major methodological review in public health (5); highlighted as a central method in newly-published implementation science protocols(6, 7);\u0026nbsp; and applied to determine different pathways for federally-qualified health centers to achieve patient-centered medical home status (8). CCMs were also featured in a dedicated chapter in the 2020 \u003cem\u003eHandbook on Implementation Science \u003c/em\u003e(9).\u003c/p\u003e\n\u003cp\u003eCCMs are designed to investigate different hypotheses and uncover different properties of causal structures than traditional regression analytical methods (RAMs).(10, 11) Qualitative Comparative Analysis (QCA) is one kind of CCM that, to date, has been most frequently applied in implementation science and health services research. The purpose of this article is to introduce a new CCM to the implementation research community: Coincidence Analysis.\u0026nbsp; Coincidence Analysis (CNA) is a mathematical, cross-case approach that can be applied as a standalone method or in conjunction with other methods (including RAMs) to support causal inference, and is available via the R-package cna.(12-14)\u003c/p\u003e\n\u003cp\u003eCNA offers a new cross-case method for implementation and health services researchers exploring causality when evaluating or implementing multifaceted interventions in complex contexts. Investigators applying CNA can conduct analyses across entire datasets to identify specific combinations of components and conditions that consistently lead to outcomes, and can be applied to large-n as well as small-n studies. Peer-reviewed, implementation-related work specifically involving CNA has started to emerge in implementation science, including podium presentations at major implementation conferences (15, 16); methods workshops dedicated specifically to CNA (17); published protocols (18); and full-length articles in established journals (19-23).\u003c/p\u003e\n\u003cp\u003eCNA is a new comparative approach that can be used by the implementation research community to support causal inference, answer research questions about conditions that are minimally necessary or sufficient, and identify multiple causal paths to an outcome. We present this article in three parts. In part 1, we establish the theoretical foundation for CCMs, define CNA as a method within the CCM family and describe what CNA (and CCMs) uniquely offer. In part 2, we illustrate CNA by applying the method to a publicly available dataset that was originally analyzed using RAMs. In part 3, we offer guidance for reporting CNA design and results, and we discuss the limitations and challenges of CNA. In additional files accompanying this article we provide detailed descriptions of the steps and coding used to conduct the analysis [see Additional file 1] and the analytic dataset used [see Additional file 2] along with the R script [see Additional file 3] to allow for independent replication and validation of results.\u003c/p\u003e\n\u003cp\u003ePart 1: Laying the Theoretical Foundation for CCMs\u003c/p\u003e\n\u003cp\u003eDefining causal inference in CCMs. CNA is one method within a class of CCMs used to model complex patterns of conditions hypothesized to contribute to an outcome within a set of data. CCMs search for causal relations as defined by a regularity theory of causality, according to which a cause is a \u0026ldquo;difference-maker\u0026rdquo; of its effect within a fixed set of background conditions. More specifically, X is a cause of Y if there exists a fixed configuration of background factors \u0026Phi;such that, in \u0026Phi;, a change in the value of X is systematically associated with a change in Y. If X does not make a difference to Y in any \u0026Phi;, X is redundant to account for Y and, thus, not a cause of Y. The most influential theory defining causation along these lines is Mackie's INUS-theory, (24) with refinements by Gra\u0026szlig;hoff and May (25) and Baumgartner. (26) An INUS condition of an outcome Y is an Insufficient but Necessary part of a condition that is itself Unnecessary but Sufficient for Y. To use a common example for illustrating INUS conditions: not every fire is caused by a short circuit\u0026mdash;fires can also be started by, for example, arson or lightning. However, a short circuit in combination with other conditions \u0026ndash; e.g., presence of flammable material and absence of a suitably placed sprinkler \u0026ndash; is sufficient for a fire. In this example, the short circuit is an INUS condition: it is a necessary, but itself insufficient, part of a sufficient, but itself unnecessary, condition for a fire. This particular causal path to a fire includes the combination of three specific conditions: presence of a short circuit, presence of flammable material, and absence of a sprinkler. All three of these conditions are difference-makers, for if one of them is missing, the fire does not occur along this causal path.\u003c/p\u003e\n\u003cp\u003eRegularity theories \u0026nbsp;account for the Boolean properties of causation, which encompass three dimensions of complexity. The first is conjunctivity: to bring about an outcome, several conditions must be jointly present. For example, in a study of high performance work practices and front line health care worker outcomes, Chuang and colleagues (27) found that no single high performance work practice was alone sufficient to produce the outcome of high job satisfaction. Instead, a configuration consisting of creative input, supervisor support, and team-based work practices together accounted for 65 percent of highly satisfied front-line health care workers. (27) Chuang and colleagues identified a second configuration that also led to high job satisfaction: supervisor support, incentive pay, team-based work and flexible work. (27) Both configurations resulted in high job satisfaction independently of each other. These configurations illustrate equifinality, a second dimension of complexity where different paths can lead to the same outcome. The third dimension of complexity is sequentiality: outcomes tend to produce further outcomes, propagating causal influence along causal chains. For instance, high job satisfaction of health care workers may, in turn, promote patient satisfaction. (28)\u003c/p\u003e\n\u003cp\u003eWhy use CCMs in implementation research? CCMs study different properties of causal structures than RAMs and thus are appropriate for exploring different types of hypotheses. RAMs examine statistical properties characterized by probabilistic or interventionist theories of causation.\u0026nbsp; In the probabilistic framework, X is a cause of Y if, and only if the probability of Y given X is greater than the probability of Y alone and there does not exist a further factor, Z, that explains (i.e., neutralizes) the probabilistic dependence between X and Y. (29, 30) \u0026nbsp;In the interventionist framework, X causes Y if, and only if there exists an intervention on X that changes the outcome Y while causes on other paths to Y are fixed. The interventionist theory of causation is counterfactual in that a case cannot simultaneously \u0026ldquo;receive\u0026rdquo; and \u0026ldquo;not receive\u0026rdquo; an intervention;\u0026nbsp; instead, the intervention model maps possible values of Y onto possible values of X, focuses on how variables X and Y relate to one another, and generates average treatment effects over a population. (11, 31)\u003c/p\u003e\n\u003cp\u003eConversely, CCMs trace Boolean properties of causal structures as described by regularity theories of causation, according to which X is a cause of Y if, and only if X is an INUS condition of Y (see INUS definition above). (11, 24)\u0026nbsp;\u0026nbsp; CCMs study implication hypotheses that link specific values of variables as \u0026ldquo;X=\u0026chi;\u003csub\u003ei\u003c/sub\u003e is (non-redundantly) sufficient/necessary for Y=\u0026gamma;\u003csub\u003ei\u003c/sub\u003e.\u0026rdquo; (11, 14). In this way, CCMs, including CNA, model the effect of conditions (e.g., high degree of X) on outcomes. This is a fundamentally different vantage point than the one adopted by RAMs which examine covariation hypotheses that link variables.\u0026nbsp; Further, CCMs are case-oriented methods, in which observations consist of bounded, complex entities (e.g., organizations) that are considered as a whole (32) A case-based unit of analysis differs from the approach taken in RAMs, where cases are deconstructed into a series of variables, and estimates represent the net effect of a variable for the average case. As CNA and other CCMs employ a case-based approach and thus can be used to identify which interventions work in an array of contexts, they present opportunities for implementation and health services research questions in particular.. \u003c/p\u003e\n\u003cp\u003eDifferent types of CCMs\u003c/p\u003e\n\u003cp\u003eWhile CCMs have a common regularity theoretic foundation, different CCMs rely on different a priori conceptions of outcome and causal factors and build causal models in different ways. For example, Qualitative Comparative Analysis (QCA), in its standard implementation that uses the Quine-McCluskey (QMC) algorithm, (33, 34) requires identification of exactly one factor as outcome. It begins by identifying maximal sufficient and necessary conditions of the outcome, which are subsequently minimized using standard inference rules from Boolean algebra to arrive at a redundancy-free solution composed of INUS conditions of the outcome. (10) However, the QMC algorithm was not originally designed for causal inference. One consequence is that the non-observation of cases instantiating empirically possible configurations of the analyzed factors, also known as limited diversity, forces QMC to draw on counterfactual reasoning that goes beyond available data, sometimes requires assumptions contradicting the very causal structures under investigation(35), and regularly fails to completely eliminate redundancies in the presence of noise.(14) Moreover, QMC has built-in protocols for ambiguity reduction when multiple models fit the data equally well. Viable models are often eliminated to reduce ambiguity without justification, which is problematic for causal discovery.(36)[\u003c/p\u003e\n\u003cp\u003eAdvantages of using CNA. Coincidence Analysis (CNA) is a new addition to the family of CCMs.(37, 38) It uses an algorithm specifically designed for causal inference, thus avoiding the problems mentioned above. In particular, it does not build causal models by means of a top-down approach that first searches for maximal sufficient and necessary conditions and then gradually minimizes them using the QMC algorithm. Rather, CNA employs a bottom-up approach that first tests single factor values for sufficiency and necessity, and then tests combinations of two, three, etc.(13, 14) All sufficient and necessary conditions revealed by this approach are automatically minimal and redundancy-free.\u003c/p\u003e\n\u003cp\u003eAdditionally, CNA is designed to treat any number of factors as endogenous and is therefore capable of analyzing causal chains, or common-cause structures.(39)\u0026nbsp; For example, Baumgartner and Epple (2014) found in a Swiss policy analysis that certain population, economic or political characteristics in some areas led to a higher rate of prejudice and, in turn, discriminatory policy. (39) Analyzing causal chains may be advantageous if, for example, an intervention A occurs as a result of other factors but is not the ultimate outcome of interest. Identifying the full causal model, including which factors produce A on the path to the ultimate outcome of interest, is valuable when seeking to understand causal complexity. CNA is the only member of the CCM family that builds and evaluates models representing causal chains.\u003c/p\u003e\n\u003cp\u003ePart 2: Demonstrating CNA using Publicly Available Data\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; \u003c/p\u003e\n\u003cp\u003eData source\u003c/p\u003e\n\u003cp\u003eIn March 2016, Rehn and colleagues reported the impact of implementation strategies on human papillomavirus (HPV) \"catch-up vaccination\" uptake in Sweden among 5th and 6th grade girls.(40) The purpose of the original study was to estimate the impact of various information channels and delivery settings on county-level catch-up vaccine uptake to inform future vaccination campaigns in Sweden.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe authors obtained county-level data on catch-up vaccinations and the eligible population from administrative data. They collected implementation strategies from county health care offices via an open-ended questionnaire emailed in 2012 asking respondents to list and describe \u0026ldquo;information channels\u0026rdquo; used to reach eligible girls and the settings in which they offered the vaccine. A subsequent phone interview was conducted in 2014 to update the lists.\u003c/p\u003e\n\u003cp\u003eRehn and colleagues used regression analysis to estimate county-level catch-up vaccine uptake as a function of information channels and delivery settings. The authors concluded that the availability of vaccines in schools explained differences in county-level vaccine uptake; no information channels were found to make a difference in uptake.\u003c/p\u003e\n\u003cp\u003eRehn and colleagues defined the outcome and predictor variables as follows:\u003c/p\u003e\n\u003cp\u003eOutcome variable. County-level catch-up vaccine uptake was defined as the percent of eligible girls born between 1993-1998 who received at least one dose of vaccine by 2014.\u003c/p\u003e\n\u003cp\u003ePredictor variables. Ten variables represented information channels and four variables represented the delivery settings where the vaccinations were available (some schools, all schools, primary health care centers, and other health care centers). All 14 factors were dichotomized with values of 1 (present) or 0 (absent).\u003c/p\u003e\n\u003cp\u003eAll county-level data on vaccine uptake, information channels and delivery settings used for the CNA illustration were reported in the article.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eWe re-analyzed the data using CNA.\u0026nbsp; A step-by-step guide for conducting CNA, using this study as an illustration, is provided in a document accompanying this article [see Additional file 1] as well as the analytic dataset [see Additional file 2] and the R script [see Additional file 3] used in the analysis.\u003c/p\u003e\n\u003cp\u003eStep 1: Define, calibrate and select the factors (i.e., outcomes and conditions) to create a data set. Vaccination rate represented the outcome of interest and ranged from 49% to 84% across the 21 counties. We selected 65% as the threshold defining \u0026ldquo;high\u0026rdquo; catch-up vaccination rates after conducting sensitivity analyses in which we varied the threshold for \u0026ldquo;high-uptake,\u0026rdquo; using two different existing break points in the data:\u0026nbsp; the 65% cut-off generated the greatest diversity among cases for the conditions and the outcome and yielded a sufficiently high number of cases featuring the outcome.\u0026nbsp; We coded the 21 counties into a new dichotomous outcome called HI_UPTAKE where 1 = \"catch-up vaccination rate of 65% or higher\" and 0 = \"catch-up vaccination rate less than 65%.\" (See Additional file 1 for details on the rationale for each step in the analytic process). A secondary analysis identified conditions leading to the absence of the outcome (HI_UPTAKE = 0) because conditions that prevent the outcome may differ from those that contribute to the presence of the outcome.\u003c/p\u003e\n\u003cp\u003eWe prepared a dataset that included the uptake rates, delivery settings and implementation strategies as reported by Rehn and colleagues (see Additional file 2 to view the analytic dataset used in the Coincidence Analysis [see Additional file 2]).\u0026nbsp; We transformed a number of factors from the original dataset for use with CNA because in the original dataset these factors had characteristics unsuitable for CCM processing. For instance, the original data set contained the factor \u0026ldquo;Primary health care centre\u0026rdquo; (PHC) that was constantly present in all 21 counties (cases). Constant factors like PHC can be automatically excluded as difference-makers. Another delivery setting, \u0026ldquo;Other health care center;\u0026rdquo; (HC) was eliminated given limited variation across cases. We combined \u0026ldquo;All schools\u0026rdquo; (\u0026ldquo;organized delivery of the vaccine in all schools in the county;\u0026rdquo; AS=1) and \u0026ldquo;Some schools\u0026rdquo; (\u0026ldquo;organized delivery of the vaccine in schools in some of the municipalities in the county\u0026hellip;;\u0026rdquo; SS=1) into a new multi-value ordinal factor called \"SCHOOLS,\" where SCHOOLS = 0 if AS = 0 and SS = 0; SCHOOLS = 1 if SS = 1 and AS = 0; and SCHOOLS = 2 if AS = 1.\u0026nbsp; The resulting dataset included 12 potential explanatory factors. These factors could be combined into 6144 logically possible configurations, which could not be covered to an informative degree by the 21 cases included. Thus, the diversity index for the original data, i.e. the ratio of observed configurations to all possible configurations, was exceedingly small. The smaller the diversity index, the more challenging it is to draw informative configurational conclusions. To improve the diversity index, we included a subset of the 12 exogenous factors in our analysis. This is analogous to maximizing degrees of freedom in RAM.\u003c/p\u003e\n\u003cp\u003eWe selected schools (SCHOOLS) and four of the ten information channels to include: school-based information (SBI), media coverage (MC), social media (SM), and Cinema commercial/YouTube (CCY). Our rationale for choosing these four implementation strategies was that they were directly linked to school and digital media, two immersive domains that are dynamic and interactive where students and their parents commonly encounter new information, \u0026nbsp;and thus likely to be effective channels \u0026nbsp;for conveying information \u0026nbsp;about why, where, how, and when to access vaccinations\u003c/p\u003e\n\u003cp\u003eIn our initial analysis plan, three of the seven cases (counties) exhibiting the outcome (high-uptake) instantiated exactly the same configuration of conditions, leaving only five observed configurations featuring HI_UPTAKE=1 out of a total of 48 logically possible configurations. As we could not justify removing one of the four selected information channels on a theoretical basis alone, but still wanted to decrease the number of overall factors in the analytic dataset (and thus increase the diversity index), we decided to assess if we could combine two of the information channels into a single \u0026nbsp;\u0026ldquo;meta-factor\u0026rdquo; \u0026mdash; a common approach in CCMs to reduce the number of conditions without eliminating either of the properties represented by these conditions from the analysis. \u0026nbsp;There are six possible ways to pair four different information channels.\u0026nbsp; Accordingly, we created six different datasets (i.e., analytic samples), each representing a different pairing of two channels, the two remaining channels and the outcome HI_UPTAKE. We coded the new meta-factor in each analytic sample with the value 1 if, and only if at least one of the two aggregated channels was present in a county (To view these six datasets, see Additional file 2.)\u0026nbsp; \u003c/p\u003e\n\u003cp\u003eStep 2. Perform CNA using the cna package in R.(13) Two parameters of fit \u0026ndash; consistency and coverage \u0026ndash; provide insight into the strength of the dependence between conditions and the outcome. Consistency, with a score ranging from 0 to 1, measures the degree to which the cases that instantiate a configuration or a whole model also instantiate the outcome. (10) Low consistency values indicate that the dependence between conditions or models and the outcome is far away from a strict Boolean (deterministic) dependence. Coverage scores range from 0 to 1 and represent the proportion of cases with the outcome that also instantiate a particular configuration or whole model. Coverage measures a given configuration\u0026rsquo;s or model's empirical importance based on the available data.(10) When applying CNA to the dataset, we set our minimum consistency and coverage scores to 1.0. (10)However, it is important to note that CNA, and CCMs generally, risk overfitting model solutions when searching for maximal consistency and coverage thresholds, which can lead to false positives.(41) . For this reason, consistency and coverage thresholds should only be set to 1 if researchers have strong reasons to assume that the data quality is very high (i.e. low levels of noise and measurement error); if that cannot be assumed, then researchers can apply a search strategy that systematically varies consistency and coverage thresholds to measure fit-robustness, or the degree to which a model solution agrees with other models identified at different consistency and coverage thresholds in the same dataset. (42)\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eStep 3. Interpret results and refine model inputs if necessary.\u003c/p\u003e\n\u003cp\u003eLike QCA, CNA encourages an iterative approach where researchers can run analyses, interpret results and redefine model inputs before finalizing a model set.(32) As such, we discuss our interpretation of the findings as our iterative analyses progress.\u003c/p\u003e\n\u003cp\u003eOur analyses produced model ambiguity, meaning that the data were insufficient to determine exactly which causal structure was operative. Of the six analytic samples, three datasets yielded a total of five causal models for HI_UPTAKE=1, all of which featured maximal consistency and coverage scores (see the R replication script for a complete list of models). All models had the following identical terms as part of their solution, where \"+\" symbolizes the Boolean operator OR, \"*\" symbolizes AND, and \"\u0026harr;\" expresses sufficiency and necessity:\u003c/p\u003e\n\u003cp\u003eSCHOOLS=2 \u0026nbsp;+ \u0026nbsp;SCHOOLS=1 * MC= 1\u0026nbsp;\u0026nbsp; \u0026harr;\u0026nbsp;\u0026nbsp;\u0026nbsp; HI_UPTAKE=1\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; (1)\u003c/p\u003e\n\u003cp\u003eThe above expression (1) translates to: \u0026ldquo;counties had high catch-up vaccination rates if, and only if they offered vaccination in all schools OR offered vaccination in some (but not all) schools AND used a media coverage implementation strategy.\" All five causal models resulting from our analysis were supersets of (1), so we concluded that the factor values contained in (1) were causally relevant for high-uptake. However, (1) was not a complete model because it only achieved a consistency score of 0.875 (i.e., only 87.5% of counties with this configuration were high-uptake counties).\u0026nbsp; Expression (1) covers a county, Sk\u0026aring;ne, that exhibited the configuration SCHOOLS=1 * MC=1 but was not high-uptake. The other counties with this configuration, SCHOOLS=1 * MC=1, were always associated with high-uptake of vaccines. Thus, other factors unique to Sk\u0026aring;ne must be missing from expression (1).\u003c/p\u003e\n\u003cp\u003eWhile our data were insufficient to identify the factors that distinguished Sk\u0026aring;ne from other counties with the same configuration, the five complete models inferred by CNA provided five possible explanations. Interestingly, all five of these solution models combined SCHOOLS=1 * MC=1 with the absence of other information channels. Only when SCHOOLS=1 * MC=1 was accompanied by SM=0 or SBI=0 or CCY=0 was it associated with high uptake with 100% consistency, which could indicate these other information channels produced a \u0026ldquo;backfiring\u0026rdquo; effect.\u003c/p\u003e\n\u003cp\u003eAfter further reviewing data for each county, we determined that the most plausible implementation strategy to backfire was cinema commercials and/or YouTube. Online media like YouTube can backfire because these open-source platforms contain unvalidated content that can appear automatically through newsfeeds or advertisements, potentially overriding legitimate health-related information. (43) Furthermore, we observed a negative relationship between the presence of CCY and the vaccination rate outcome in the underlying dataset.\u0026nbsp; As Figure 1 shows, the presence of CCY was relatively well-represented in the overall dataset: 8 of the 21 cases had CCY=1. In 7 of these 8 cases, HI_UPTAKE=0. The lone exception was the county of Jonkoping, where the sufficient condition of SCHOOLS=2 was also present. For these reasons, we deemed the following complete model to be the most plausible:\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eSCHOOLS=2 \u0026nbsp;+ \u0026nbsp;SCHOOLS=1 * MC=1 * CCY=0\u0026nbsp; \u0026harr; HI_UPTAKE=1 \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; (2)\u003c/p\u003e\n\u003cp\u003eExpression (2) translates to: \u0026ldquo;counties had high catch-up vaccination rates if, and only if they offered vaccination in all schools OR offered vaccination in some (but not all) schools AND used a media coverage implementation strategy but not cinema commercials/YouTube.\" Expression (2) had perfect consistency and coverage scores (1.0, respectively) and differentiated Sk\u0026aring;ne (a county without high uptake) from high uptake counties that also had vaccinations available at some schools and used media coverage. Figure 1 highlights the configurations instantiating this model in the data.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFigure 1. Overall model identified by Coincidence Analysis with 100% consistency and 100% coverage.\u003c/p\u003e\n\u003cp\u003e[insert Figure 1 here]\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eApplying the same analytic approach to model the absence of the outcome yielded seven models for HI_UPTAKE=0 such that six of these seven models had a common core that corresponded exactly to the negation of the core of the positive models:\u003c/p\u003e\n\u003cp\u003eSCHOOLS=0 + SCHOOLS=1 * MC=0\u0026nbsp; \u0026harr;\u0026nbsp; HI_UPTAKE=0\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; (3)\u003c/p\u003e\n\u003cp\u003eExpression (3) translates to: \u0026ldquo;counties had no high catch-up vaccination rates if, and only if they did not offer vaccination in all schools OR offered vaccination in some (but not all) schools AND did not use a media coverage implementation strategy.\" Expression 3 exhibited 1.0 consistency and 0.93 coverage. Taken together, these results provide substantive evidence that media coverage is relevant for differentiating counties with and without high vaccination uptake, adding additional information to the regression results of the original study.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThe results of this CNA indicate that, under specific conditions, information channels made a difference for high vaccination uptake. This contrasts with the results of the regression analysis from the original study, which concluded that information channels made no difference in increasing vaccination uptake. Our results imply that the availability of vaccination in some schools is only sufficient for high vaccination rates if media coverage is employed and certain other communication channels are not used. In other words, the data contain enough evidence to infer that when vaccination is available at some but not all schools, availability must be complemented by media coverage to achieve high uptake. The data do not contain enough evidence, however, to have absolute certainty which communication channels should be avoided. Even so, cinema commercials and/or YouTube might be the most plausible information channel to avoid; YouTube in particular might backfire and reduce vaccination rates as a result of unsolicited content that undermines county-sanctioned media coverage on vaccines.\u003c/p\u003e\n\u003cp\u003eClosely examining data from individual cases (counties) corroborate the theory of some conditions backfiring with respect to producing high vaccination uptake. Jonkoping provided vaccination in all schools, but only achieved vaccination uptake among 65% of eligible girls, as opposed to over 80% uptake achieved in the three other counties with vaccination availability at all schools.\u0026nbsp; Notably, Jongkoping used two communication channels (CCY=1 and SM=1) that were absent in the other three counties (CCY=0 and SM=0).\u003c/p\u003e\n\u003cp\u003eIn sum, the CNA results indicate that whenever vaccination is available at only some schools, media coverage make a difference for high uptake. Furthermore, the common core of the resulting CNA models for HI_UPTAKE=0 indicate that a lack of media coverage when vaccinations are provided only in some schools make a difference for lower vaccine uptake. By systematically scrutinizing the configurations of implementation strategies associated with high-uptake and low-uptake counties, CNA extends the conclusions drawn from Rehn and colleagues\u0026rsquo; regression model.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe data on vaccination uptake in Sweden do not comprise multiple outcomes and, hence, CNA's capacity to uncover multi-outcome structures cannot be showcased with this example. Readers interested in multi-outcome discovery with CNA are, instead, referred to (39) and (44), the first of which finds a causal chain and the second a common-cause structure.\u003c/p\u003e\n\u003cp\u003eOur approach to setting consistency and coverage thresholds in this study is consistent with the extant CNA literature.\u0026nbsp; Of note, however, is that ensemble strategies have been newly proposed where consistency and coverage thresholds are systematically varied across a series of thresholds in order to measure \u0026ldquo;fit-robustness,\u0026rdquo; the degree to which a specific model agrees with other models identified at different consistency and coverage thresholds in the same dataset. (42) While ensemble approaches to setting consistency and coverage thresholds are still nascent, they appear highly promising as analytic strategies that can help mitigate the risk of overfitting models .(41)\u003c/p\u003e\n\u003cp\u003ePart 3: Tips for Practitioners\u003c/p\u003e\n\u003cp\u003eRecommendations for reporting analyses and results for CCMs\u003c/p\u003e\n\u003cp\u003eVariation in reporting study design, analysis and results exists in previously published peer-reviewed CCM literature. To advance the methodological rigor with which CCMs are applied, we offer recommendations for describing design, analysis and results for CCMs. We also provide additional material in Additional files 1 and 2 accompanying this article to allow for independent replication of these analyses and findings [Additional file 1][Additional file 2].\u003c/p\u003e\n\u003cp\u003eWe suggest that future studies and publications applying CCMs: 1) describe the rationale for using the CCM (e.g., CNA); 2) describe the rationale for selecting which factors (outcomes and conditions) to include in the analysis; 3) describe the process used to assign cases to factor values (45, 46) (e.g., high vaccination uptake); 4) specify the software (and version) used for analysis; 5) describe the iterative analyses used to refine factors (e.g., different approaches to calibration) and models (e.g., adding/dropping factors); 6) list the number of models generated in each iteration of the analysis and identify commonalities across models; 7) report consistency and coverage thresholds for final models along with ranges for models not part of the final model(s); and 8) describe the rationale for selecting the final model(s).\u003c/p\u003e\n\u003cp\u003eIn this article, we used conceptual knowledge of HPV vaccination uptake and the existence of variability across cases to select the factors to include in the initial analysis. Given the structure of the data, we relied on the original binary factor calibration for all but one factor (SCHOOLS), which was calibrated as a multi-value ordinal factor. Moreover, to improve the diversity index, we explored all mutually exclusive combinations of two factors representing information channels into a single factor.\u0026nbsp; We analyzed six datasets using the cna package in R.(13) Five models reached perfect consistency and coverage and, thus, fit the data equally well. All models exhibited a common core, which we reported as \u0026nbsp;determinate causal inference supported by our data. At the same time, we acknowledge that the common core did not amount to a complete causal model. Based on theoretical considerations, which we explicitly described, we then selected one of five viable completions as the most plausible overall model. We reported all relevant model fit scores for our preferred model and provide additional files with detailed descriptions of each step in the analytic process [see Additional file 1], the analytic dataset [see Additional file 2] and the R script [see Additional file 3] to allow for independent replication and verification of our results.\u003c/p\u003e\n\u003cp\u003eLimitations of CNA\u003c/p\u003e\n\u003cp\u003eCNA has several limitations of which implementation researchers should be aware.\u0026nbsp; First, although CNA supports causal inference, there are limitations to the extent to which results may be generalized. Results can be confounded by unmeasured causes that are located on causal paths to outcomes that do not go through any measured factors. If the data cannot be assumed to be homogenous in confounder distributions\u0026mdash;meaning that unmeasured factors do not affect all cases/configurations equally\u0026mdash;generalization becomes problematic given the risk of over-interpreting the data or incorrectly inferring a causal relationship. As with other CCMs, familiarity with cases helps to evaluate generalizability\u0026mdash;by, for example, justifying that cases included in the analysis are homogeneous with respect to potential confounders\u0026mdash;and to interpret solutions generated by mathematical modeling.\u0026nbsp; In the example dataset from Rehn and colleagues, it is possible a third factor may explain the negative relationship between cinema commercials and/or YouTube, for example. In this instance, researchers can further explore the relationship between these factors and HPV vaccination uptake through qualitative interviews or other methods to confirm findings.\u003c/p\u003e\n\u003cp\u003eSecond, CCMs rarely uncover the data-generating causal structures in full. Configurational data analyzed in observational studies tend to be fragmented (i.e., exhibit low diversity), so most logically possible combinations of conditions are not present in the observed cases. Under these circumstances, CCMs may reveal only portions of the underlying causal structures. Thus, the fact that some factor X is not contained in a CCM model must not be taken to mean that X is causally irrelevant. Unless there is reason to assume that the data are non-fragmented, the absence of X from a model can only mean that the data do not contain evidence for X's relevance (which is not the same as X's irrelevance).\u003c/p\u003e"},{"header":"Conclusions","content":"\u003cp\u003eCNA offers new insights of potential high interest to implementation researchers.\u0026nbsp; We demonstrated the utility of CNA using data previously analyzed with RAMs. The authors of the original, RAM-based analysis indicated that offering vaccination in schools increased county-level vaccination uptake while no information channels significantly increased vaccination uptake.(40) By contrast, CNA results indicated that under specific conditions, information channels did make a difference for high vaccination uptake. Specifically, our results imply that higher vaccination rates are achieved by either (1) offering the vaccine in all schools or (2) offering the vaccine in some schools and using media coverage but not certain other communication channels.\u003c/p\u003e\n\u003cp\u003eCompared to RAMs, CCMs have fundamentally different methodological goals and search for different properties of causal structures. CCMs and RAMs answer distinct types of questions. RAMs are useful for estimating the average influence of a specific variable on an outcome while holding other variables constant. CCMs are useful for identifying combinations of specific conditions that may be on the same or different causal paths (i.e., are minimally necessary or sufficient) to an outcome. Of the two CCMs, CNA was built expressly for causal inference and can be used to uncover causal chains underlying the data.(13, 14, 39)\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eCNA has the potential to offer implementation researchers alternative and more nuanced knowledge about causal relationships when examining complex interventions in settings with interdependent or interrelated factors. In particular, CCMs are well-suited for implementation and health services research questions regarding the implementation of multifaceted interventions in complex, real-world settings, the dynamics of which can be influenced by many factors acting in combination.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eEthics approval and consent to participate\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eConsent for publication\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAvailability of data and materials\u003c/p\u003e\n\u003cp\u003eThe full dataset used in this analysis along with detailed descriptions of each step of the analysis and the R script are all available for review and replication in additional files accompanying this article [see Additional file 1][see Additional file 2][Additional file 3].\u0026nbsp; The original dataset is publicly available within the following open-access article: Rehn M, Uhnoo I, K\u0026uuml;hlmann-Berenzon S, Wallensten A, Spar\u0026eacute;n P, Netterlid E. Highest vaccine uptake after school-based delivery - a county-level evaluation of the implementation strategies for HPV catch-up vaccination in Sweden. PLoS One. 2016;11(3):e0149857. doi:10.1371/journal.pone.0149857.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eCompeting interests\u003c/p\u003e\n\u003cp\u003eThe authors declare that they have no competing interests.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFunding\u003c/p\u003e\n\u003cp\u003eThis secondary analysis was conducted by the investigators without additional funding.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAuthors' contributions\u003c/p\u003e\n\u003cp\u003eRGW, SB, NS, LD and AS originated the idea for the project and convened the full team of co-authors. RGW found the published dataset that we used for secondary analysis.\u0026nbsp; RGW, MB, and EJM took lead roles in conducting the Coincidence Analysis. MB wrote the R script provided in Additional file 3. RGW, SB, NS, LD, AS, MS, AT, DC and EJM participated in a series of discussions to review preliminary results from the Coincidence Analysis and provide direct feedback to inform ongoing analyses. RGW, SB, NS, LD, AS, MS, AT, DC and EJM participated in writing sections of the main manuscript plus the step-by-step guide provided in Additional file 1, as well as editing drafts.\u0026nbsp; All authors read and approved the final manuscript.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAcknowledgements\u003c/p\u003e\n\u003cp\u003eThe authors wish to thank the Regenstrief Institute for its pioneering role\u0026nbsp;in recent years\u0026nbsp;in hosting and organizing seminars that allow interested health researchers and implementation scientists to learn more about applying Configurational Comparative Methods (including Coincidence Analysis) in their work.Alrik Thiem gratefully acknowledges financial support from the Swiss National Science Foundation, grant number \u003ca href=\"https://www.mysnf.ch/grants/grant.aspx?id=0b25a7fb-40e9-415c-bfa7-b1ba38f1d47b\"\u003ePP00P1_170442\u003c/a\u003e. Sarah Birken\u0026rsquo;s effort was supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant KL2TR002490. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.\u003c/p\u003e"},{"header":"Reference List","content":"\u003col\u003e\n\u003cli\u003ePfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A, Hofmann B, et al. Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework. Implement Sci. 2017;12.\u003c/li\u003e\n\u003cli\u003eCouncil MR. A framework for development and evaluation of RCTs for complex interventions to improve health. 2000.\u003c/li\u003e\n\u003cli\u003edatabase WoSCC. Philadelphia: Clarivate Analytics; 2020.\u003c/li\u003e\n\u003cli\u003ePalinkas LA, Mendon SJ, Hamilton AB. Innovations in Mixed Methods Evaluations. Annu Rev Public Health. 2019;40:423-42.\u003c/li\u003e\n\u003cli\u003eHarris K, Kneale D, Lasserson TJ, McDonald VM, Grigg J, Thomas J. School-based self-management interventions for asthma in children and adolescents: a mixed methods systematic review. Cochrane Database Syst Rev. 2019;1:CD011651.\u003c/li\u003e\n\u003cli\u003eRogal SS, Yakovchenko V, Morgan T, Bajaj JS, Gonzalez R, Park A, et al. Getting to implementation: a protocol for a Hybrid III stepped wedge cluster randomized evaluation of using data-driven implementation strategies to improve cirrhosis care for Veterans. Implement Sci. 2020;15(1):92.\u003c/li\u003e\n\u003cli\u003eBunger AC, Chuang E, Girth A, Lancaster KE, Gadel F, Himmeger M, et al. Establishing cross-systems collaborations for implementation: protocol for a longitudinal mixed methods study. Implement Sci. 2020;15(1):55.\u003c/li\u003e\n\u003cli\u003eMendel P, Chen EK, Green HD, Armstrong C, Timbie JW, Kress AM, et al. Pathways to Medical Home Recognition: A Qualitative Comparative Analysis of the PCMH Transformation Process. Health Serv Res. 2018;53(4):2523-46.\u003c/li\u003e\n\u003cli\u003eBirken SA, Nilsen, P., Cragun, D. Configurational Comparative Methods. In: Sarah A. Birken PN, editor. Handbook on Implementation Science: Edward Elgar Publishing; 2020. p. 569.\u003c/li\u003e\n\u003cli\u003eThiem A. Conducting Configurational Comparative Research With Qualitative Comparative Analysis: A Hands-On Tutorial for Applied Evaluation Scholars and Practitioners. Am J Eval. 2017;38(3):420-33.\u003c/li\u003e\n\u003cli\u003eRohlfing I, Zuber CI. Check your truth conditions! Clarifying the relationship between theories of causation and social science methods for causal inference. Sociological Methods \u0026amp; Research. 2019:0049124119826156.\u003c/li\u003e\n\u003cli\u003eThiem A, Baumgartner M, Bol D. Still lost in translation! A correction of three misunderstandings between configurational comparativists and regressional analysts. Comparative Political Studies. 2016;49(6):742-74.\u003c/li\u003e\n\u003cli\u003eAmb\u0026uuml;hl M, Baumgartner M, Epple R, Thiem A. cna: Causal Modeling with Coincidence Analysis. 2020.\u003c/li\u003e\n\u003cli\u003eBaumgartner M, Amb\u0026uuml;hl M. Causal modeling with multi-value and fuzzy-set Coincidence Analysis. Political Science Research and Methods. 2020;8(3):526-42.\u003c/li\u003e\n\u003cli\u003eMiech E, Rattray N, Damush T. Necessary but not sufficient: A multimethod study of the role of champions in heathcare-related implementation. Implement Sci. 2020;15.\u003c/li\u003e\n\u003cli\u003eCragun D, Rahm, AK. Coincidence Analysis: a methodology to identify contextual conditions influencing implementation across multiple settings. Implement Sci. 2019;14(Proceedings from the 11th Annual Conference on the Science of Dissemination and Implementation):27.\u003c/li\u003e\n\u003cli\u003eBirken S, Damschroder, L., Miech, E., Cragun, D., editor Agenda for the 12th Annual Conference on the Science of Dissemination and Implementation. Dissemination and Implementation Conference; 2019; Arlington, VA.\u003c/li\u003e\n\u003cli\u003eRahm AK, Cragun D, Hunter JE, Epstein MM, Lowery J, Lu CY, et al. Implementing universal Lynch syndrome screening (IMPULSS): protocol for a multi-site study to identify strategies to implement, adapt, and sustain genomic medicine programs in different organizational contexts. Bmc Health Serv Res. 2018;18.\u003c/li\u003e\n\u003cli\u003ePetrik AF, Green B, Schneider J, Miech EJ, Coury J, Retecki S, et al. Factors Influencing Implementation of a Colorectal Cancer Screening Improvement Program in Community Health Centers: an Applied Use of Configurational Comparative Methods. J Gen Intern Med. 2020;35(Suppl 2):815-22.\u003c/li\u003e\n\u003cli\u003eDy SM, Acton RM, Yuan CT, Hsu YJ, Lai AY, Marsteller J, et al. Association of Implementation and Social Network Factors With Patient Safety Culture in Medical Homes: A Coincidence Analysis. J Patient Saf. 2020.\u003c/li\u003e\n\u003cli\u003eHickman SE, Miech EJ, Stump TE, Fowler NR, Unroe KT. Identifying the Implementation Conditions Associated With Positive Outcomes in a Successful Nursing Facility Demonstration Project. Gerontologist. 2020.\u003c/li\u003e\n\u003cli\u003eYakovchenko V, Miech EJ, Chinman MJ, Chartier M, Gonzalez R, Kirchner JE, et al. Strategy Configurations Directly Linked to Higher Hepatitis C Virus Treatment Starts An Applied Use of Configurational Comparative Methods. Med Care. 2020;58(5):E31-E8.\u003c/li\u003e\n\u003cli\u003eMoret W, Lorenzetti L. Realistic expectations: exploring the sustainability of graduation outcomes in a program for children affected by HIV in Kenya's Northern Arid Lands. Vulnerable Child You. 2020.\u003c/li\u003e\n\u003cli\u003eMackie JL. The Cement of the Universe: A Study of Causation. Oxford: Clarendon Press; 1974.\u003c/li\u003e\n\u003cli\u003eGra\u0026szlig;hoff G, \u0026amp; May, M. Causal regularities. Paderborn, Germany: Mentis-Verlag; 2001. 85-114 p.\u003c/li\u003e\n\u003cli\u003eBaumgartner M. Regularity Theories Reassessed. Philosophia. 2008;36(3):327-54.\u003c/li\u003e\n\u003cli\u003eChuang E, Dill J, Morgan JC, Konrad TR. A Configurational Approach to the Relationship between High-Performance Work Practices and Frontline Health Care Worker Outcomes. Health Services Research. 2012;47(4):1460-81.\u003c/li\u003e\n\u003cli\u003eSikorska-Simmons E. Linking resident satisfaction to staff perceptions of the work environment in assisted living: A multilevel analysis. Gerontologist. 2006;46(5):590-8.\u003c/li\u003e\n\u003cli\u003eSuppes P. A Probabilistic Theory of Causality: North-Holland Pub. Co.; 1970. 409-10 p.\u003c/li\u003e\n\u003cli\u003eBaumgartner M. Causation. In: Dirk Berg-Schlosser BB, Leonardo Morlino, editor. The SAGE Handbook of Political Science. 01. London: SAGE Publications; 2020. p. 305-21.\u003c/li\u003e\n\u003cli\u003eWoodward J. Making Things Happen: A Theory of Causal Explanation. New York, New York: Oxford University Press, Inc; 2003.\u003c/li\u003e\n\u003cli\u003eRagin CC. The Dialogue of Ideas and Evidence in Social Research. The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies. Okland, California: University of California Press; 1987.\u003c/li\u003e\n\u003cli\u003eQuine WV. On Cores and Prime Implicants of Truth Functions. The American Mathematical Monthly. 1959;66(9):755-60.\u003c/li\u003e\n\u003cli\u003eMcCluskey EJ. Introduction tothe Theory of Switching Circuits: McGraw-Hill; 1965. 318 p.\u003c/li\u003e\n\u003cli\u003eBaumgartner M. Parsimony and Causality. Qual Quant. 2015;49(2):839-56.\u003c/li\u003e\n\u003cli\u003eBaumgartner M, Thiem A. Model Ambiguities in Configurational Comparative Research. Sociological Methods \u0026amp; Research. 2017;46(4):954-87.\u003c/li\u003e\n\u003cli\u003eBaumgartner M. Uncovering deterministic causal structures: a Boolean approach. Synthese. 2009;170(1):71-96.\u003c/li\u003e\n\u003cli\u003eBaumgartner M. Inferring Causal Complexity. Sociological Methods \u0026amp; Research. 2009;38(1):71-101.\u003c/li\u003e\n\u003cli\u003eBaumgartner M, Epple R. A Coincidence Analysis of a Causal Chain The Swiss Minaret Vote. Sociological Methods \u0026amp; Research. 2014;43(2):280-312.\u003c/li\u003e\n\u003cli\u003eRehn M, Uhnoo I, Kuhlmann-Berenzon S, Wallensten A, Sparen P, Netterlid E. Highest Vaccine Uptake after School-Based Delivery - A County-Level Evaluation of the Implementation Strategies for HPV Catch-Up Vaccination in Sweden. PLoS One. 2016;11(3):e0149857.\u003c/li\u003e\n\u003cli\u003eArel-Bundock V. The Double Bind of Qualitative Comparative Analysis. Sociological Methods \u0026amp; Research. 2019:0049124119882460.\u003c/li\u003e\n\u003cli\u003eParkkinen V, Baumgartner M. Robustness and Model Selection in Configurational Causal Modeling. Sociological Methods \u0026amp; Research. 2020.\u003c/li\u003e\n\u003cli\u003eMoorhead SA, Hazlett DE, Harrison L, Carroll JK, Irwin A, Hoving C. A new dimension of health care: systematic review of the uses, benefits, and limitations of social media for health communication. J Med Internet Res. 2013;15(4):e85.\u003c/li\u003e\n\u003cli\u003eHaesebrouck T. Who follows whom? A coincidence analysis of military action, public opinion and threats. Journal of Peace Research. 2019;56(6):753-66.\u003c/li\u003e\n\u003cli\u003eThiem A. Membership function sensitivity of descriptive statistics in fuzzy-set relations. Int J Soc Res Method. 2014;17(6):625-42.\u003c/li\u003e\n\u003cli\u003eVerkuilen J. Assigning membership in a fuzzy set analysis. Sociological Methods \u0026amp; Research. 2005;33(4):462-96.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"implementation-science","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"imps","sideBox":"Learn more about [Implementation Science](http://implementationscience.biomedcentral.com)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/IMPS/default.aspx","title":"Implementation Science","twitterHandle":"@ImplementSci","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Coincidence Analysis, Configurational Comparative Methods, causal inference, comparative analysis","lastPublishedDoi":"10.21203/rs.3.rs-58815/v2","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-58815/v2","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBackground: Implementation of multifaceted interventions typically involves many diverse elements working together in interrelated ways, including intervention components, implementation strategies, and features of local context. Given this real-world complexity, implementation researchers may be interested in a new mathematical, cross-case method called Coincidence Analysis (CNA) that has been designed explicitly to support causal inference, answer research questions about combinations of conditions that are minimally necessary or sufficient for an outcome, and identify the possible presence of multiple causal paths to an outcome. CNA can be applied as a standalone method or in conjunction with other approaches, and can reveal new empirical findings related to implementation that might otherwise have gone undetected. \u003c/p\u003e\u003cp\u003eMethods: We applied CNA to a publicly available dataset from Sweden with county-level data on human papillomavirus (HPV) vaccination campaigns and vaccination uptake in 2012 and 2014 and then compared CNA results to the published regression findings. \u003c/p\u003e\u003cp\u003eResults: The original regression analysis found vaccination uptake was positively associated only with the availability of vaccines in schools. CNA produced different findings and uncovered an additional solution path: high vaccination rates were achieved by either (1) offering the vaccine in all schools or (2) a combination of offering the vaccine in some schools and media coverage. \u003c/p\u003e\u003cp\u003eConclusions. CNA offers a new comparative approach for researchers seeking to understand how implementation conditions work together and link to outcomes.\u003c/p\u003e","manuscriptTitle":"Coincidence Analysis: A New Method for Causal Inference in Implementation Science","msid":"","msnumber":"","nonDraftVersions":[{"code":2,"date":"2020-12-03 02:20:07","doi":"10.21203/rs.3.rs-58815/v2","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Accept","date":"2020-11-30T00:00:00+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2020-11-20T00:00:00+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2020-11-19T23:00:00+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2020-11-19T23:00:00+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"implementation-science","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"imps","sideBox":"Learn more about [Implementation Science](http://implementationscience.biomedcentral.com)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/IMPS/default.aspx","title":"Implementation Science","twitterHandle":"@ImplementSci","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}},{"code":1,"date":"2020-08-17 23:13:31","doi":"10.21203/rs.3.rs-58815/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Major revision","date":"2020-10-22T12:00:00+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2020-10-21T12:00:00+00:00","index":2,"fulltext":"Recommendation: Accept\nForm responses:\n---\n\nComments to Author:\n---\nThank you for the opportunity to review this interesting manuscript. It sheds light on a promising method for implementation researchers. The manuscript is well written; its structure supports the reader's understanding. It undoubtedly represents a significant contribution to the field!\n\nI have two very minor comments:\n\nManuscript page 7 : The fire examples illustrates perfectly the INUS condition concept. Great idea!\n\nManuscript page 12, lines 308-310: Why are there no references to support the statement?\n* Publons Reviewer Recognition. Springer Nature can send verification of this review directly to Publons (a subsidiary of Clarivate Analytics). If you would like to take advantage of this service, please click on the “Yes” option below. Your name, email address, title of the reviewed manuscript, name of the journal, and date of your review submission (the “Review Data”) will then be transmitted to Publons upon publication of the manuscript. If you have already registered at Publons, they will notify you of the receipt of this review and update your profile as per your settings and their policy. If you are not registered with Publons, you will receive an email from them asking you to register in order for them to be able to recognize your review on your new profile page. Publons may use the Review Data to generate derivative metadata for the benefit of Publons and you as a reviewer, carefully considering the sensitivity of such information. For example, Publons may verify your record as a reviewer by updating your profile published on its webservice if you have registered for such service or help editors to identify candidate reviewers. Please find the details of processing in Publons’ privacy policy https://publons.com/about/terms: **No**\n* Level of interest: **An exceptional article**\n* Quality of written English: **Acceptable**\n* Declaration of competing interests: **I declare that I have no competing interests**\n* I agree to the open peer review policy of the journal. I understand that my name will be included on my report to the authors and, if the manuscript is accepted for publication, my named report including any attachments I upload will be posted on the website along with the authors' responses. I agree for my report to be made available under an Open Access Creative Commons CC-BY license (http://creativecommons.org/licenses/by/4.0/). I understand that any comments which I do not wish to be included in my named report can be included as confidential comments to the editors, which will not be published.: **\nI agree to the open peer review policy of the journal**\n"},{"type":"editorInvitedReview","content":"","date":"2020-10-21T12:00:00+00:00","index":1,"fulltext":"Recommendation: Major Revision\nForm responses:\n---\n\nComments to Author:\n---\nOne of the most pressing challenges in implementation science relates to ways to evaluate implementation strategies, particularly multifaced strategies with components that may have interact with one another. This manuscript proposes the application of a new method, Coincidence Analysis (CNA), to solve this problem. They make use of a publicly available dataset including strategies to improve HPV vaccine uptake in which original regression-based analyses identified one effective strategy - offering vaccines in schools. This approach identified an additional pathway, the use of media coverage. The authors presented a compelling background with clear explanation and useful examples. They also present instructions and syntax for the method in a highly accessible analytic platform, which greatly enhances the likelihood of uptake among researchers and practitioners. Finally, they provide suggestions for reporting, which will promote standardized practice in the field. There are a few points of feedback that may provide additional clarity and enhance the value to Implementation Science readers.\nIn the methods section, it's not clear how the set of 4/10 information channels was selected relative to the other implementation strategies in the dataset, which someone must have thought would also have value in improving vaccine rates. Some additional clarity on the process by which disjunctive aggregation resulted in six datasets. Some examples and a conceptual figure of selection and disjunctive aggregation processes might help to clarify. I'm not sure that Figure 1 substantially contributes to the manuscript, and this may be a place for cutting if space is an issue.\nIn terms of causal inference, based on the example presented here, it seems like this method was oversold in its ability to support claims about causation, particularly with respect to third variables. For example, neighborhood income may drive the apparent association between availability in schools and vaccine uptake, with schools in more advantaged communities more able to offer vaccines, and greater primary care access for the families who live in those communities. Alternatively, there may be a third variable explaining the apparent negative effects of cinema or You Tube, for example, the public health practitioners who planned these implementation strategies may have felt that these communities had less access to traditional health communication channels and needed more innovative approaches. The results presented here do not seem strong enough to make the conclusion that You Tube should be avoided. Authors might do well to emphasize the need for replication or following up CNA with other types of designs (e.g., randomized trials, qualitative stakeholder interviews) to confirm findings.\nFinally, the ability to uncover causal chains was provided as a rationale for using CNA, and I was very interested to hear more about that, but details and examples were not provided.\nI thank you for the opportunity to learn more about this work, and am hopeful about the potential of this strategy to address what is certainly an urgent need in the field. I hope that authors find these comments helpful in revising their manuscript for publication.\n* Publons Reviewer Recognition. Springer Nature can send verification of this review directly to Publons (a subsidiary of Clarivate Analytics). If you would like to take advantage of this service, please click on the “Yes” option below. Your name, email address, title of the reviewed manuscript, name of the journal, and date of your review submission (the “Review Data”) will then be transmitted to Publons upon publication of the manuscript. If you have already registered at Publons, they will notify you of the receipt of this review and update your profile as per your settings and their policy. If you are not registered with Publons, you will receive an email from them asking you to register in order for them to be able to recognize your review on your new profile page. Publons may use the Review Data to generate derivative metadata for the benefit of Publons and you as a reviewer, carefully considering the sensitivity of such information. For example, Publons may verify your record as a reviewer by updating your profile published on its webservice if you have registered for such service or help editors to identify candidate reviewers. Please find the details of processing in Publons’ privacy policy https://publons.com/about/terms: **Yes**\n* Level of interest: **An article of importance in its field**\n* Quality of written English: **Acceptable**\n* Declaration of competing interests: **I declare that I have no competing interests**\n* I agree to the open peer review policy of the journal. I understand that my name will be included on my report to the authors and, if the manuscript is accepted for publication, my named report including any attachments I upload will be posted on the website along with the authors' responses. I agree for my report to be made available under an Open Access Creative Commons CC-BY license (http://creativecommons.org/licenses/by/4.0/). I understand that any comments which I do not wish to be included in my named report can be included as confidential comments to the editors, which will not be published.: **\nI agree to the open peer review policy of the journal**\n"},{"type":"editorInvitedReview","content":"","date":"2020-10-09T12:00:00+00:00","index":3,"fulltext":"Recommendation: Reviewer's comments unavailable pending editorial decision\n"},{"type":"reviewerAgreed","content":"","date":"2020-09-28T12:00:00+00:00","index":3,"fulltext":""},{"type":"reviewerAgreed","content":"","date":"2020-09-01T12:00:00+00:00","index":2,"fulltext":""},{"type":"reviewerAgreed","content":"","date":"2020-08-18T12:00:00+00:00","index":1,"fulltext":""},{"type":"reviewersInvited","content":"","date":"2020-08-17T12:00:00+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2020-08-13T12:00:00+00:00","index":"","fulltext":""},{"type":"submitted","content":"","date":"2020-08-12T12:00:00+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2020-08-12T12:00:00+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2020-08-12T12:00:00+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"implementation-science","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"imps","sideBox":"Learn more about [Implementation Science](http://implementationscience.biomedcentral.com)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/IMPS/default.aspx","title":"Implementation Science","twitterHandle":"@ImplementSci","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"124c958e-e6d1-4cc7-be6c-4e15b8799d2a","owner":[],"postedDate":"December 3rd, 2020","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":313223,"name":"Health Economics \u0026 Outcomes Research"}],"tags":[],"updatedAt":"2020-12-13T15:02:53+00:00","versionOfRecord":{"articleIdentity":"rs-58815","link":"https://doi.org/10.1186/s13012-020-01070-3","journal":{"identity":"implementation-science","isVorOnly":false,"title":"Implementation Science"},"publishedOn":"2020-12-11 15:01:54","publishedOnDateReadable":"December 11th, 2020"},"versionCreatedAt":"2020-12-03 02:20:07","video":"","vorDoi":"10.1186/s13012-020-01070-3","vorDoiUrl":"https://doi.org/10.1186/s13012-020-01070-3","workflowStages":[]},"version":"v2","identity":"rs-58815","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-58815","identity":"rs-58815","version":["v2"]},"buildId":"re_ckhLnmML6MCF96OHNJ","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}