A quantitative approach to evidence triangulation: development of a framework to address rigour and relevance

Triangulation is an approach to strengthening causal inference by integrating evidence from multiple sources. Most studies using triangulation have qualitatively examined whether different studies agree upon the presence of a causal effect, rather than estimated the effect by quantitatively integrating results. Here, we develop a framework for quantitative triangulation. We first address how to relate study specific research questions to an overall target causal question (relevance), and then assess the directions and magnitudes of bias in each study (rigour), before combining the

using meta-analysis adjusted for the biases. We illustrate our framework by triangulating evidence from randomized controlled trials (RCTs), Mendelian randomization (MR) and conventional multivariable regression (MVR) to estimate the effect of beta-carotene on coronary heart disease (CHD) and cardiovascular disease (CVD). Five RCTs and one MR study showed little evidence of a causal relationship between beta-carotene and CHD (relative risk (RR)=1.00 with 95% CI=0.98 to 1.01 and RR=1.02 with 95% CI=0.98 to 1.07, respectively). 13 MVR studies indicated that high intake of beta-carotene reduces CHD risk (RR=0.83 with 95% CI0.76 to 0.91). After applying our framework, the three study designs agreed that there is little evidence of an effect of beta-carotene intake on the risk of CHD (RR=1.01 with 95% CI=0.99 to 1.02). Findings were similar for CVD. Our framework shows how to address rigour and relevance quantitatively when triangulating evidence from different study designs. We highlight the importance of explicitly defining the target and study-specific research questions.

Aetiological epidemiology, triangulation, risk of bias, relevance, meta-analysis . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 3

Triangulation of epidemiological evidence has been defined as “The practice of strengthening causal inferences by integrating results from several approaches, where each approach has different (and assumed to be largely unrelated) key sources of potential bias”1. The motivation is that where different studies are subject to different biases plausibly acting in different directions, if the studies all demonstrate an effect, we have added confidence that the effect is real, even if the estimated magnitudes of effect vary across studies. Increasing interest in triangulation is accompanying the widening of available approaches for evaluating effects of exposures, such as genetics-based

(e.g., Mendelian randomization (MR))2 and target trial emulation analyses3. Applications of triangulation have so far taken a ‘qualitative’ approach, however, seeking primarily to establish the presence and direction of a causal relationship between an exposure and an outcome. Here, we develop a more quantitative approach, in which we seek to estimate the magnitude of the causal relationship. Quantifying relationships is important for several reasons, including the identification of safe levels of an exposure, the need to trade-off benefits and harms, and the development of interventions that might achieve worthwhile gains in health. To quantify a relationship, it is necessary to define the parameter(s) being estimated. This requires specification of the research question (the ‘target question’) in terms of the population, exposure and outcome of interest and the assumed nature of the relationship between the exposure and the outcome (e.g., linear or non-linear). Individual studies assembled for a triangulation exercise will rarely each provide an answer to the target question directly, for two broad reasons. First, the participants, exposures, and outcomes examined in the studies will typically differ from each other and from the target question. Second, the studies will suffer from different biases. These two realms have been described as ‘relevance’ and ‘rigour’, and have also been referred to as ‘external’ and ‘internal’ bias, respectively4. The aim of this paper is to develop and illustrate a framework for quantitatively triangulating evidence from studies taking different approaches to answer a specific target question. 5-7To deal with relevance (external bias), we transform or scale the results of the studies to match the target question, based on a combination of assumptions and external data. We deal with rigour (internal bias) by deriving adjustments for bias based on formal assessments of risk of bias using standard tools. These tools, widely used in systematic reviews, include the Risk of Bias 2 (RoB 2) tool for randomized controlled trials (RCTs)5 and the Risk Of Bias In Non-Randomized Studies tools for observational studies of intervention effects (ROBINS-I)6 and exposure effects (ROBINS-E)7. We undertake the analysis within a Bayesian framework, using informative prior distributions to achieve the bias adjustments4. We use the potential effect of beta-carotene on cardiovascular disease (CVD) risk as a case study, as previous research highlighted differences between RCT and observational estimates8. For our analyses, we seek recent published evidence from diverse study designs and approaches, using systematic reviews where available.

We develop our framework in terms of: defining the target causal question; assessing relevance of individual studies to this target question; assessing potential biases in the individual studies; and finally combining evidence from all studies whilst adjusting for identified bias and relevance (as shown in Figure 1). Defining the target causal question Defining the target causal question is the important first step of triangulation. We consider six key aspects in defining the target question: the population, exposure measure, exposure window, how . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 4 exposure is to be summarized over time (e.g., single exposure event, time invariant exposure, average exposure over 20 years in mid-life, or maximum dose of a treatment over a year), outcome measure, and the statistical parameter to be estimated9. Ideally, as with any study design, the causal question should be identified before identifying studies to be included. eTable 1 has a list of some different considerations for each of these aspects of the target causal question. Relevance: assessing and mapping results to a comparable metric The next step is to identify a pool of potentially related studies and assess their ‘relevance’, i.e., how directly each individual study maps onto the target causal question. First, we need to determine the research question addressed by each study. We assess this in terms of the same six aspects that define the target question. The population and outcome are generally derived easily from the eligibility criteria and outcome measurements used in the study. The exposure measure and duration in RCTs are straightforward to extract from the study description. Defining the exposure of interest for observational studies is more challenging because the assumed start time and exposure window are rarely articulated. It may be necessary to declare the exposure or time window to which it is assumed the observational study’s estimate relates without knowing what the original investigators intended. For example, a study relating body mass index (BMI) at baseline in UK Biobank to later risk of CVD10 may implicitly be considering average BMI during middle age as the exposure. Once the research question for each study has been identified, final decisions about inclusion of each study can be made – i.e., how closely does each study address the target causal question? Subjective decisions here include aspects such as whether to include studies using different exposures (e.g. percentage of fat mass instead of BMI) or in different populations. To be included, the effect estimate from a study must be mathematically relatable to the exposure effect of interest in the target question. This mathematical relationship could use information from a different source, providing the estimate is considered transportable to the study under consideration. For example, if the exposure of interest is BMI, then studies measuring percentage of fat mass could be included by using external information about the relationship between BMI and percentage of fat mass via their correlation or a mathematical formula for the conversion. Rigour (internal bias): risk-of-bias assessment Our framework uses rigorously developed, domain-based tools to assess risk of bias in each study contributing to the triangulation. Examples of such tools are RoB 25, ROBINS-I6 or ROBINS-E7 and the tool developed by Mamluk et al.11 to assess the risk of bias in RCTs, conventional observational studies of interventions and exposures, and MR studies, respectively. Different tools have different bias domains and levels of risk-of-bias judgements. Table 1 illustrates the bias domains assessed and risk-of-bias judgement options for three of these tools. Standard systematic review good practice should be followed, including having at least two independent assessors for risk of bias for each study. Combining relevant studies Our framework uses the open access triangulate R package12 to combine bias-corrected effect estimates for each study. triangulate12 provides a systematic way to integrate assessments from risk- of-bias tools and adjusts for bias and relevance concerns from multiple study designs using prior distributions, then pools the adjusted estimates using standard meta-analysis. . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 5 Prior distributions and bias adjustment The first step in implementing the bias correction is to postulate the type of bias in each domain, and whether the bias is likely to be additive or proportional. Additive bias is appropriate if it is plausible that the bias in the study has a subtractive or an additive effect on the true exposure effect. The additive bias either “Favours experimental” or “Favours comparator”, for subtractive and additive effects, respectively. Proportional bias acts multiplicatively on the estimated exposure effect so impacts only the magnitude and not the direction of any effect. It is either “Towards the null” or “Away from the null”. We allow bias to be “Unpredictable” if the direction of bias cannot be predicted. For example, consider bias adjustment in the effect of statin use on Alzheimer’s disease, if a study does not adjust for Apoε status as a confounder, it is predicted that this bias will shift the true effect towards favouring the comparator. If a study does not provide an analysis protocol, it might be suspected that the authors have cherry-picked their results and hence the bias adjustment would move the effect estimate proportionally towards the null (proportionally as it depends on the position of the effect estimate from the null). For more examples of defining type and direction of bias see Chapter 7 in McGuinness9. Algebraic details of the specification of additive and proportional biases are given in eAppendix 1b. As a final step, prior distributions need to be specified to quantify the likely extent of bias (based on judgements in Table 1), type of bias (additive or proportional) and direction of bias. We recommend including studies with “very high risk of bias”, as these study results can still be informative when they are contrasted with results of studies with different designs. To be consistent with the procedure proposed by the authors of triangulate9, we recommend specifying, for both additive and proportional bias, a prior distribution for “very high risk of bias” as double the value of mean and variance of the prior distribution for “high risk of bias”. For “unpredictable” bias, the prior distribution should likely have a mean of zero, but with a non-zero variance – thus “unpredictable” bias does not change the effect estimate but increases its uncertainty. Application to case-study Defining the target causal question To illustrate our framework, we aimed to investigate the effect of dietary beta-carotene intake on CVD risk. We defined our target population as middle-aged adults (above age 40), since the risk of CVD increases exponentially after age 4013. We additionally limited the population to those initially free from known (diagnosed) CVD, as occurrence of a CVD event may change patient's diets. We chose dietary beta-carotene as the exposure (rather than circulating beta-carotene) because it is directly modifiable. We considered an average increase of 5,000 µg in dietary beta-carotene per day. We assumed that a short-term intervention of this magnitude would not be effective, so we decided to consider an exposure that was sustained over time. For this illustrative example we have used a 20 year period in mid-life14. Our main outcome of interest is any CVD, including both fatal and non- fatal events. A secondary outcome is coronary heart disease (CHD), the most commonly occurring sub-type of CVD, including fatal and non-fatal events15. Our causal question is therefore: in a general population of adults (above age 40) initially free of known CVD, what is the effect of an average increase of 5,000 µg per day of dietary beta-carotene on the incidence of a CVD [or CHD] event expressed as a risk ratio (RR) assumed constant over a 20-year period? . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 6 Identification of relevant studies We searched the Scopus database in February 2023 to identify systematic reviews of the effect of beta-carotene on CVD or CHD using the following search terms within title, abstract or

("vitamin A" or "retinol" or "Provitamin A" or carotene or carotenes or carotenoid or "β- carotene" or "beta-carotene" or "lycopene") and (“coronary heart disease” or “heart disease” or CHD or “cardiovascular disease” or CVD) and ("meta-analysis" or "systematic review"). We selected the most recent systematic review covering each study design (RCTs or observational studies). We did not exclude studies of circulating beta-carotene, as we assume that the effect of dietary beta- carotene will act via its impact on circulating beta-carotene. We retrieved 19 studies for the risk of CVD and CHD with beta-carotene (dietary and circulating) from the systematic review of conventional observational studies16. From a systematic review of RCTs17, we extracted eight studies for the effect of supplemental beta-carotene on risk of CVD and CHD. We did not identify any systematic reviews of MR studies. Therefore, we searched for individual MR studies of the potential effects of dietary intake of beta-carotene on CVD/CHD using Scopus, as previously and a similar search but the term “Mendelian randomization” replacing the systematic review terms (search run February 2023). We identified one MR study18 from 2021 for the effect of circulating beta-carotene on risk of CHD, which included three estimates. eAppendix 2a gives further details about study exclusion criteria and data transformation in the systematic reviews, as well as details of instrument selection, samples for instrument-exposure and instrument-outcome association in the MR study. Relevance: assessing and converting results to a comparable metric In this section we assess ‘relevance’ in terms of the target population, exposure measure, exposure window, definition of any summary measures of exposure over time, outcome measure and the statistical parameter to be estimated. eTables 2 and 3 gives the country, population, age range, follow-up time, exposure, and outcome measure of each study. We first sought to specify the research question for each study. The exposure measure is given in each observational study (dietary or circulating beta-carotene). However, the assumed exposure window was not articulated in any study. Therefore, we decided that the exposure window for each observational study was from the time of recruitment to the study, and for RCTs and MR study from time of randomization. This choice also matched the analyses performed in the studies most closely. eAppendix 2b provides further discussion of potential bias on the choice of exposure window in conventional observational studies. The summary of exposure over time was also not specified in any study, therefore we decided that, for observational studies, the summary exposure is the daily average beta-carotene dietary intake over time. For RCTs, the summary exposure would be the average additional beta- carotene dietary intake during the trial period. For MR study in our example, we considered the summary exposure to be lifetime cumulative beta-carotene dietary intake, as it is assumed that the genetic instrument acts on the exposure from conception/foetal development onwards19, as the study only had beta-carotene measured at baseline and not measured at different time points. In each study, the outcome was whether an individual was diagnosed with CVD and CHD defined by International Classification of Diseases either version 9 or 10. . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 7 Our parameter of interest is the RR for the outcome, given an increase in dietary beta- carotene. As CVD and CHD are rare events, we have assumed that odds ratios, hazard ratios, incidence rate ratios and RRs would be approximately equal. To map results to a comparable metric, we transformed dosage of exposure level of each study to our pre-specified exposure, “5,000 µg per day of dietary beta-carotene”. We converted circulating beta-carotene to dietary beta-carotene for the observational and MR studies that used circulating beta-carotene as their exposure, using results of a meta-analysis of the correlation between dietary and plasma beta-carotene20. We assumed that dietary beta-carotene intake would only affect outcomes via its effect on circulating beta-carotene. For each RCT, we converted risk ratio for beta-carotene supplements compared to controls to per 5,000 µg/day. See more details in eAppendix 2c. Rigour (internal bias): risk-of-bias assessment We used the tools RoB 25, ROBINS-E7 and developed by Mamluk et al.11 to assess the risk of bias in RCTs, conventional observational studies and MR studies, respectively. Two assessors independently assessed risk of bias for each study. After assessing all studies, assessors reached a consensus judgement for each study. Any discrepancies were discussed with a third assessor. For the assessment of observational studies using ROBINS-E, after discussions with content experts we agreed on a list of important confounders to consider in every study (eTable 5). We agreed that all studies would be judged to be at “Very high risk of bias” for Domain 1 (Risk of bias due to confounding) unless the study had adjusted for a minimal set of confounders. The minimal set of confounders was age, sex, socioeconomic background, ethnicity, BMI and dietary features (including vitamins via food and supplements or dietary intake measured as energy intake). Combining relevant studies For meta-analysis without a bias correction, we estimated a combined RR across the studies using random effects meta-analysis of logarithmic (to base e) RRs, with a restricted maximum likelihood estimator to estimate between-studies heterogeneity. We used the robviz R package for meta- analysis and risk-of-bias visualisation 21. We used the RR for all incident (i.e., both fatal and non-fatal) CVD (or CHD) if a study provided both mortality and all incidences. Prior distribution and bias adjustment Since our purpose here is illustrative, we used the default priors (Table 2) in triangulate12 to adjust for the extent of bias (derived from 4): the prior distributions corresponds to the different levels of bias judgements. Note that for additive biases, the sign of the distribution is defined by the absolute direction of bias, with “Favours comparator” having a positive sign and “Favours experimental” having a negative sign (e.g., N(-0.09, 0.05)). Furthermore, in this example “Favours comparator” is favouring lower exposure and “Favours experimental” is favouring higher exposure. We obtained bias-corrected effect estimates for each study and performed a random effect meta-analysis with bias-corrected effect estimates via triangulate12. All analyses were implemented in R version 4.2.2.

We present the results for CHD in this section. The results for CVD are in eAppendix 2e, as the findings for CVD and CHD are similar. . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 8 Description of studies For the analysis of dietary beta-carotene on risk of CHD specifically, there were three RCTs22-24, seven conventional observational studies25-29 and one MR study18 (within provided three effect estimates from separate instrument-outcome data), shown in Table 3. Five studies defined their endpoint as patients experiencing any CHD events, one study with CHD mortality, and four studies with all MI events; three studies have separate numbers for all CHD and all MI events. Risk-of-bias

of risk-of-bias assessments for studies reporting CHD as an outcome are presented alongside the study results in Figure 2. We judged two out of the three RCTs to have some concerns around bias, because of insufficient detail about randomization methods in bias domain 1 (D1). One RCT was judged to have a high risk of bias due to missing data (D3), and another had some concerns over the possibility of selection of the reported result (D5). The conventional observational studies were mainly judged to have high risk of bias overall. All the studies had adjusted for the minimal set of essential confounders (eTable 5), hence had high risk of bias due to confounding (D1) rather than very high risk of bias (as discussed above). We judged that not adjusting for confounding would result in additive bias, i.e., could shift the estimate in both direction and magnitude, and most likely the bias is in favour of showing a greater benefit of beta- carotene intake. Risk of bias from measurement of the exposure (D2) was considered high or very high for three of the studies. Although a food frequency questionnaire (FFQ) had been used to assess the exposure, we considered it unlikely that this would provide an accurate measure of an individual’s beta-carotene intake due to high day-to-day variability and poor recall30. We judged D2 to have proportional bias, as classical random measurement error is more likely to pull the effect towards the null. Pandey et al.27 was the only observational study to have a very high risk of bias overall, arising from measurement of exposure (D2), 27 since they did not implement the commonly used FFQ and beta-carotene was estimated from intake of other vitamins. Some studies were judged to have moderate risk of bias in selection of the reported result (D7) because of the lack of a pre- determined analysis plan, and therefore possible that bias was introduced by changing some of the analyses after seeing the results. Each domain does not necessarily have the same type of bias for all studies. For example, most studies were assigned “unpredictable” for proportional bias in D7, as we did not know the investigator’s prior beliefs. However, for Klipstein-Grobusch et al.28, we assigned “favours experimental” for additive bias in D7 as they focused on only one ‘statistically significant’ result. Note that because Klipstein-Grobusch et al.28 is assigned as “unpredictable” with “favours experimental”, this will not adjust the effect estimate but increases its uncertainty instead. All the MR studies have similar risk-of-bias judgements for each domain, apart from CARDIoGRAMplusC4D which had high risk of bias in D5. This is due to inclusion of non-European ancestry for the instrument-outcome association when the instrument-exposure associations are of only European descents. Meta-analyses without bias correction Meta-analyses of results unadjusted for bias from RCTs and MR studies provided no evidence of an effect of beta-carotene on CHD and did not show evidence of heterogeneity between studies (left panel of Figure 2). Conventional observational studies gave a pooled RR of 0.83 with a 95% CI from 0.76 to 0.91, without evidence of heterogeneity. . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 9 Meta-analyses with bias-corrected estimates Meta-analysing the bias-adjusted effect estimates gave an estimated null effect of beta-carotene on CHD from RCT, conventional observational, and MR studies (right panel of Figure 2). The pooled RRs were 1.00 (95% CI=0.99 to 1.02), 0.97 (95% CI=0.75 to 1.24) and 1.02 (95% CI=0.98 to 1.06) from RCTs, MVR and MR analyses, respectively (Figure 2). There was one RCT and one MR analysis with high risk of bias, but we judged the direction of bias to be unpredictable in both cases, so that the adjusted effect estimate was unchanged by inclusion of these studies (but with a wider CI).

Most of the published work on triangulation in epidemiology has focused on the qualitative comparison of study estimates from different study designs1,31. By contrast, the focus of our paper is on a quantitative approach to triangulation with different study designs. Our focus on bias correction and quantifying effects is an important extension of the approach initially described by Lawlor et al.1. We have developed and illustrated a framework to identify the target question and the relevance of each study to this and used formal risk-of-bias tools to identify different sources of internal bias. We have also shown how to address the potential biases using the R package triangulate12 by stating the expected magnitude and direction of bias for each study through informative prior distributions for internal bias. Our illustrative example demonstrates how risk of bias correction quantification can be applied in triangulation and our previously developed open access R package makes this efficient. In this example, even with uncertainty around the generalisability and simplicity of the default priors, our results provide strong evidence that dietary intake of beta-carotene does not affect risk of any CVD or of CHD. It is common practice in meta-analysis to restrict the analysis to studies considered to be most relevant and at lowest risk of bias, for example by excluding studies that do not meet strict eligibility criteria or are assessed to be at high risk of bias (although sensitivity analyses might be conducted to evaluate the influence of high risk of bias results). In a triangulation framework, it is generally preferable to take a broader perspective, including all studies in the analysis and systematically addressing their limitations in the synthesis. Quantitative bias analysis is one approach to doing this32-34, for example by adjusting study results according to the direction and magnitude of bias, possibly using elicited expert opinions about the biases4,35,36. Expert opinion4,37,38 can come in different forms: different choices of bias terms via sensitivity analysis35, distributions of bias terms4 or modelling “ignorance” about the bias term39. The triangulate R package12 is based on the method proposed by Turner et al.4, in which elicited distribution of bias/relevance terms based on expert opinion are incorporated into the analysis. We make the important assumption, for the sake of illustrating the methods, that the default priors from the R package triangulate12, which are derived from a study examining the effectiveness of routine anti-D prophylaxis in Rhesus negative women, 4 are applicable to the effect of beta-carotene on cardiovascular health. Relatedly, we assume exchangeability for the bias distributions, so that each study design has the same bias distributions, which may not be the case (the mean and uncertainty around bias may differ between RCTs, conventional observational studies, and MR studies). Furthermore, the bias distributions are assumed to be the same for each domain and independent between domains. Previous meta-epidemiologic studies of RCTs showed evidence that the effect of bias may be different for different domains40, and the domains are unlikely to be independent in practice4. For example, the bias arising from measurement of exposure could be positively correlated with bias due to missing data, because dietary beta-carotene is . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 10 measured from FFQ, which is time-consuming if it has many food items, and the participants are less likely to answer all the questions thoroughly. Potential future work would be to formulate relevant prior distributions for bias using combinations of meta-epidemiologic studies and expert opinion38,41. These approaches depend on available data and much of the methodological research has been done on clinical trials. Even though imposing prior distributions on bias is challenging, failure to acknowledge bias and assuming all effect estimates are unbiased is even more problematic and may lead to false conclusions, as seems to have been the case in some systematic reviews of RCTs42. Our experience suggests that careful a priori definition of the research question (e.g., the causal effect of interest) is vital when embarking on a triangulation exercise. During the implementation of ROBINS-E, we noticed that most of the primary studies did not clearly define their research question and rarely covered the six criteria; the population, exposure measure, exposure window, outcome, definition of any summary measures of exposure over time, and the statistical parameter to be estimated. No cohort study discussed the exposure window or summary measure of exposure over time that they were examining. Consequently, we had to theorise the research questions as we perceived them from study data rather than from authors’ stated intentions. In future, we advocate specification of clearly defined causal questions in epidemiologic studies of the effects of exposures. Similar arguments for a clearly defined causal question were also discussed by Munafò and Davey Smith43, who also argued that triangulation can enable more informative prospective studies. Judging the relevance of different studies to the target question is potentially more subjective than judging risk of bias. For some aspects, quantitative approaches can be used – for example, we included studies that measured circulatory beta-carotene and used correlation estimated from a large meta-analysis of dietary and plasma beta-carotene20 to convert the RR for circulatory to dietary beta-carotene. Similarly, in our case study, some studies only included male participants while our research question applied to both sexes. We judged that sex was not a moderator of our target effect, which meant we included these studies with no further quantitative adjustment. The triangulate R package12 has assessment of relevance in aetiological studies that covers the domains of population (or participants), exposure, and outcome (or endpoint)9. For each study, as with bias assessment, the assessor could judge level and direction of relevance in each of these domains. Further work is needed to establish guidance for assessing relevance. We note that in our case study, we considered measurement error of exposure to be classical random measurement error, and this would cause bias towards the null. However, this assumes no measurement error in all confounders44. For example, in a sample of health-conscious individuals could potentially over-report their frequency of physical activity and dietary beta- carotene intake, and therefore this measurement error is biasing the effect estimate away from the null. Our case study highlights the need for a more thorough risk-of-bias tool for MR. The risk-of- bias tool for MR by Mamluk et al. 11 was designed for their own systematic review, unlike RoB 2 and ROBINS-E which are products of a team of multiple experts and many years of piloting. The Mamluk et al. tool also does not include domains for measurement error, sensitivity analysis, or missing data 45. All the studies in our illustrative example aligned in their conclusions after bias adjustment. In this case, all the evidence we considered suggests that there is little evidence of an effect of beta- carotene intake on risk of CVD or CHD. In many cases, however, there may remain heterogeneity in the bias-adjusted estimates. As with any meta-analysis, the reasons for this should be examined, . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 11 considering whether the heterogeneity is the result of study-level variables. For example, if sex moderates the beta-carotene-CVD relationship, this could cause heterogeneity between studies. An alternative approach that overcomes this problem would be to use different statistical approaches within the same study1 such as within sibship analyses, target trial emulation and negative controls. However, this depends on having access to the individual-level data for each study, and the structure of the study supporting alternative approaches. In general, heterogeneity between effect estimates would give rise to further investigation, as it clearly indicates that the effect of interest cannot be reliably estimated from the evidence available. The applied example was aimed to illustrate our framework; therefore, we assumed that the two systematic reviews have selected all the relevant papers and followed best practices46. We also acknowledge that these reviews are published in 2018 and therefore will not include new evidence. We would recommend the triangulation efforts should ideally build on top of existing evidence synthesis best practices, e.g. comprehensive systematic/umbrella reviews to identify all studies. In conclusion, triangulation is a challenging and still-developing area of quantitative bias analysis in epidemiology. With an ever-growing number of studies, triangulation will become more difficult but also more important, and thus robust methods are crucial. We have developed a framework for quantitative triangulation, including guidance for the process of identifying and addressing biases and differences in relevance, and illustrated how prior distributions can empirically lay out researchers’ assumptions about these factors. We have also demonstrated the future of triangulation is dependent on better and more consistent reporting. . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 12 Table 1: Bias domains and risk-of-bias judgements for each study design Study Design Randomized controlled trials Observational studies of exposure effects Mendelian randomization Tools RoB 2 ROBINS-E Mamluk et al.11 Domains D1 Randomization process Confounding Weak instrument D2 Deviations from intended interventions Measurement of the exposure Genetic confounding D3 Missing outcome data Selection of participants into the study (or into the analysis) Other confounding D4 Measurement of the outcome Post-exposure interventions Additional direct effects between instrumental variable and outcome (exclusion restriction assumption) D5 Selection of the reported

Missing data Selection of participants D6 - Measurement of outcomes - D7 - Selection of the reported

- Judgements Low / Some concerns / High Low / Some concerns / High / Very high Low / Moderate / High RoB 2: Risk of Bias 2, ROBINS-E: Risk Of Bias In Non-Randomized Studies tools for observational studies of exposure effects. Table 2: Prior distributions mapped to different extents of bias - Prior values are defined as log-normal distributions, N(µ, σ2) with mean (µ) and variance (σ2). Bias Level Additive bias (“Favours experimental”/ “Favours comparator”) Proportional bias (“Towards null”/ “Away from null”) “Unpredictable” for additive bias “Unpredictable” for proportional bias Low - - - - Moderate N(0.09, 0.05) N(0.03, 0.016) N(0, 0.05) N(0, 0.016) High N(0.18, 0.1) N(0.06, 0.032) N(0, 0.1) N(0, 0.032) Very high N(0.36, 0.2) N(0.12, 0.064) N(0, 0.2) N(0, 0.064) Table 3: Characteristics of included studies where endpoint is coronary heart disease (CHD) First author Study design Outcome* Exposure summary Median age (years)^ Follow-up (years) Relative risk* (95% CI) Randomized controlled trials Hennekens 22 RCT (Physicians’ Health Study) All MI events 50 mg/2d 40-84 12.9 (Dur. 12) 0.96 (0.85, 1.08) Tornwall 23 RCT (The Alpha- Tocopherol, Beta- Carotene Cancer Prevention Study) All CHD events 20 mg/d 57 12.1 (Dur. 6.1) 1.03 (0.92, 1.16) All MI events 1.07 (0.91, 1.24) . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 13 Cook 24 RCT (Women’s Antioxidant Cardiovascular Study) All CHD events 50 mg/2d 60.6 9.4 (Dur. 9.4) 1.00 (0.89, 1.13) All MI events 0.97 (0.77, 1.23) Observational studies Osganian 25 Cohort study (Nurses’ Health Study) All CHD events Dietary 50 12 0.83 (0.72, 0.96) Todd 26 Cohort study (Scottish Heart Health Study) All CHD events Dietary 50 7.5 0.87 (0.68, 1.12) Pandey 27 Cohort study (Western Electric Company Study) CHD mortality Dietary 40-55 21 0.84 (0.66, 1.09) Klipstein- Grobusch 28 Cohort study (The Rotterdam Study) All MI events Dietary 67.45 4 0.22 (0.07, 0.68) Koh 29 Nested case-control study (Singapore Chinese Health Study) All CHD events Plasma 69.2 (7.5) 6.5 0.95 (0.66, 1.37) Karppi 47 Cohort study (Kuopio Ischaemic Heart Disease Risk Factor Study) All MI events Plasma 57.1 11.5 0.64 (0.45, 0.93) Hak 48 Nested case-control study (Physicians’ Health Study) All MI events Plasma 58 6.3 0.82 (0.63, 1.07) Mendelian randomization studies Luo 18 Mendelian randomization study All CHD events Plasma Multiple GWAS~ Multiple GWAS~ 1.05 (0.95, 1.17)a 1.02 (0.93, 1.11)b 1.03 (0.84, 1.26)c CHD, coronary heart disease; MI, myocardial infarction; CI, confidence interval; Dur., intervention duration. *Relative risk is extracted from systematic reviews, not individual studies. ^this varies between studies, some provides median, mean or range. ~Multiple GWAS: summary statistic for instrument-exposure is from the Nurses’ Health Study, and instrument-outcome are from (a) Coronary Artery Disease Genome-Wide Replication and Meta-analysis plus the Coronary Artery Disease Genetics, (b) UK Biobank and (c) FinnGen study. . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 14 Figure 1: Flow chart demonstrating our proposed triangulation framework. . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 15 Figure 2: Unadjusted (left panel) and bias-corrected (right panel) random effects meta-analysis for CHD events as endpoint with risk-of-bias assessment (middle panel). See Table 1 for description of D1-D7 of each bias assessment tool. O is the overall judgment of the risk of bias. RCT, randomized controlled trial; Obs, conventional observational study; MR, Mendelian randomization; CI, confidence intervals; RE, random effects; p, p-value of the pooled effect estimate; I2, heterogeneity variance / total variance; τ 2, estimated amount of heterogeneity. The colours and symbols are shown in the legend on the plot.

1 Lawlor, D. A., Tilling, K. & Davey Smith, G. Triangulation in aetiological epidemiology. International Journal of Epidemiology 45, 1866-1886 (2016). 2 Lawlor, D. A., Harbord, R. M., Sterne, J. A., Timpson, N. & Davey Smith, G. Mendelian randomization: using genes as instruments for making causal inferences in epidemiology. Statistics in medicine 27, 1133-1163 (2008). 3 Hernán, M. A., Wang, W. & Leaf, D. E. Target Trial Emulation: A Framework for Causal Inference From Observational Data. JAMA 328, 2446-2447 (2022). https://doi.org/10.1001/jama.2022.21383 4 Turner, R. M., Spiegelhalter, D. J., Smith, G. C. & Thompson, S. G. Bias modelling in evidence synthesis. Journal of the Royal Statistical Society: Series A (Statistics in Society) 172, 21-47 (2009). 5 Sterne, J. A. et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 366 (2019). 6 Sterne, J. A. et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355, i4919 (2016). https://doi.org/10.1136/bmj.i4919 7 ROBINS-E Development Group (Higgins J, M. R., Rooney A, Taylor K, Thayer K, Silva R, Lemeris C, Akl A, Arroyave W, Bateson T, Berkman N, Demers P, Forastiere F, Glenn B, Hróbjartsson A, Kirrane E, LaKind J, Luben T, Lunn R, McAleenan A, McGuinness L, Meerpohl . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 16 J, Mehta S, Nachman R, Obbagy J, O'Connor A, Radke E, Savović J, Schubauer-Berigan M, Schwingl P, Schunemann H, Shea B, Steenland K, Stewart T, Straif K, Tilling K, Verbeek V, Vermeulen R, Viswanathan M, Zahm S, Sterne J). Risk Of Bias In Non-randomized Studies - of Exposure (ROBINS-E). (2022). 8 Egger, M., Schneider, M. & Davey Smith, G. Meta-analysis Spurious precision? Meta-analysis of observational studies. BMJ 316, 140-144 (1998). 9 McGuinness, L. A. Chapter 7: Aetiological triangulation across new and existing evidence sources: a qualitative and quantitative approach, (2021). 10 Sudlow, C. et al. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS medicine 12, e1001779 (2015). 11 Mamluk, L. et al. Evidence of detrimental effects of prenatal alcohol exposure on offspring birthweight and neurodevelopment from a systematic review of quasi-experimental studies. International Journal of Epidemiology 49, 1972-1995 (2020). 12 McGuinness, L. A. & Shapland, C. Y. mcguinlu/triangulate: Initial release (v0.1.0), (2024). 13 Wallisch, C. et al. Re-estimation improved the performance of two Framingham cardiovascular risk equations and the pooled cohort equations: a nationwide registry analysis. Scientific reports 10, 8140 (2020). 14 Hernán, M. A. & Robins, J. M. (CRC Boca Raton, FL, 2010). 15 Mendis, S., Puska, P., Norrving, B. & Organization, W. H. Global atlas on cardiovascular disease prevention and control. (World Health Organization, 2011). 16 Aune, D. et al. Dietary intake and blood concentrations of antioxidants and the risk of cardiovascular disease, total cancer, and all-cause mortality: a systematic review and dose- response meta-analysis of prospective studies. The American journal of clinical nutrition 108, 1069-1091 (2018). 17 Jenkins, D. J. et al. Supplemental vitamins and minerals for CVD prevention and treatment. Journal of the American College of Cardiology 71, 2570-2584 (2018). 18 Luo, J., le Cessie, S., van Heemst, D. & Noordam, R. Diet-derived circulating antioxidants and risk of coronary heart disease: a Mendelian randomization study. Journal of the American College of Cardiology 77, 45-54 (2021). 19 Morris, T. T. et al. Interpretation of Mendelian randomization using a single measure of an exposure that varies over time. International Journal of Epidemiology 51, 1899-1909 (2022). https://doi.org/10.1093/ije/dyac136 20 Burrows, T. L. et al. Plasma carotenoid levels as biomarkers of dietary carotenoid consumption: A systematic review of the validation studies. Journal of Nutrition & Intermediary Metabolism 2, 15-64 (2015). 21 McGuinness, L. A. & Higgins, J. P. Risk‐of‐bias VISualization (robvis): an R package and Shiny web app for visualizing risk‐of‐bias assessments. Research synthesis methods 12, 55-61 (2021). 22 Hennekens, C. H. et al. Lack of effect of long-term supplementation with beta carotene on the incidence of malignant neoplasms and cardiovascular disease. New England Journal of Medicine 334, 1145-1149 (1996). 23 Törnwall, M. E. et al. Effect of α-tocopherol and β-carotene supplementation on coronary heart disease during the 6-year post-trial follow-up in the ATBC study. European heart journal 25, 1171-1178 (2004). 24 Cook, N. R. et al. A randomized factorial trial of vitamins C and E and beta carotene in the secondary prevention of cardiovascular events in women: results from the Women's Antioxidant Cardiovascular Study. Archives of internal medicine 167, 1610-1618 (2007). 25 Osganian, S. K. et al. Dietary carotenoids and risk of coronary artery disease in women. The American journal of clinical nutrition 77, 1390-1399 (2003). . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 17 26 Todd, S., Woodward, M., Tunstall-Pedoe, H. & Bolton-Smith, C. Dietary antioxidant vitamins and fiber in the etiology of cardiovascular disease and all-causes mortality: results from the Scottish Heart Health Study. American Journal of Epidemiology 150, 1073-1080 (1999). 27 Pandey, D. K., Shekelle, R., Selwyn, B. J., Tangney, C. & Stamler, J. Dietary vitamin C and β- carotene and risk of death in middle-aged men: the Western Electric study. American Journal of Epidemiology 142, 1269-1278 (1995). 28 Klipstein-Grobusch, K. et al. Dietary antioxidants and risk of myocardial infarction in the elderly: the Rotterdam Study. The American journal of clinical nutrition 69, 261-266 (1999). 29 Koh, W.-P. et al. Plasma carotenoids and risk of acute myocardial infarction in the Singapore Chinese Health Study. Nutrition, Metabolism and Cardiovascular Diseases 21, 685-690 (2011). 30 Natarajan, L. et al. Validity and Systematic Error in Measuring Carotenoid Consumption with Dietary Self-report Instruments. American Journal of Epidemiology 163, 770-778 (2006). https://doi.org/10.1093/aje/kwj082 31 Matthay, E. C. et al. Alternative causal inference methods in population health research: evaluating tradeoffs and triangulating evidence. SSM-Population Health 10, 100526 (2020). 32 Dias, S., Sutton, A. J., Welton, N. J. & Ades, A. Evidence synthesis for decision making 3: heterogeneity—subgroups, meta-regression, bias, and bias-adjustment. Medical Decision Making 33, 618-640 (2013). 33 Lash, T. L. et al. Good practices for quantitative bias analysis. International Journal of Epidemiology 43, 1969-1985 (2014). https://doi.org/10.1093/ije/dyu149 34 Stone, J. C., Glass, K., Munn, Z., Tugwell, P. & Doi, S. A. R. Comparison of bias adjustment

in meta-analysis suggests that quality effects modeling may have less limitations than other approaches. Journal of Clinical Epidemiology 117, 36-45 (2020). https://doi.org/10.1016/j.jclinepi.2019.09.010 35 Spiegelhalter, D. J. & Best, N. G. Bayesian approaches to multiple sources of evidence and uncertainty in complex cost‐effectiveness modelling. Statistics in medicine 22, 3687-3709 (2003). 36 Verde, P. E. A bias‐corrected meta‐analysis model for combining, studies of different types and quality. Biometrical Journal 63, 406-422 (2021). 37 Van der Bles, A. M. et al. Communicating uncertainty about facts, numbers and science. Royal Society open science 6, 181870 (2019). 38 Rhodes, K. M. et al. Adjusting trial results for biases in meta‐analysis: combining data‐based evidence on bias with detailed trial assessment. Journal of the Royal Statistical Society: Series A (Statistics in Society) 183, 193-209 (2020). 39 Raices Cruz, I., Troffaes, M. C., Lindström, J. & Sahlin, U. A robust Bayesian bias‐adjusted random effects model for consideration of uncertainty about bias terms in evidence synthesis. Statistics in Medicine (2022). 40 Savović, J. et al. Association between risk-of-bias assessments and results of randomized trials in Cochrane reviews: the ROBES meta-epidemiologic study. American Journal of Epidemiology 187, 1113-1122 (2018). 41 Welton, N. J., Ades, A. E., Carlin, J., Altman, D. & Sterne, J. Models for potentially biased evidence in meta‐analysis using empirically based priors. Journal of the Royal Statistical Society: Series A (Statistics in Society) 172, 119-136 (2009). 42 Katikireddi, S. V., Egan, M. & Petticrew, M. How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? A methodological study. J Epidemiol Community Health 69, 189-195 (2015). 43 Munafò, M. R. & Davey Smith, G. Repeating experiments is not enough. Nature 553, 399-401 (2018). 44 Rutter, C. E., Millard, L. A. C., Borges, M. C. & Lawlor, D. A. Exploring regression dilution bias using repeat measurements of 2858 variables in ≤49 000 UK Biobank participants. . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint 18 International Journal of Epidemiology 52, 1545-1556 (2023). https://doi.org/10.1093/ije/dyad082 45 Spiga, F. et al. Tools for assessing quality and risk of bias in Mendelian randomization studies: a systematic review. International Journal of Epidemiology (2022). https://doi.org/10.1093/ije/dyac149 46 Higgins, J. P. T. et al. Cochrane book series (Wiley-Blackwell, Hoboken, NJ., 2019). 47 Karppi, J., Laukkanen, J. A., Mäkikallio, T. H. & Kurl, S. Low serum lycopene and β-carotene increase risk of acute myocardial infarction in men. The European Journal of Public Health 22, 835-840 (2012). 48 Hak, A. E. et al. Plasma carotenoids and tocopherols and risk of myocardial infarction in a low-risk population of US male physicians. Circulation 108, 802-807 (2003). . CC-BY-NC-ND 4.0 International licenseIt is made available under a perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in(which was not certified by peer review)preprint The copyright holder for thisthis version posted September 23, 2024. ; https://doi.org/10.1101/2024.09.20.24314046doi: medRxiv preprint