Bias Estimation In Study Design: A Meta-Epidemiological Analysis of Transcatheter Versus Surgical Aortic Valve Replacement

Bias Estimation In Study Design: A Meta-Epidemiological Analysis of Transcatheter Versus Surgical Aortic Valve Replacement | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research article Bias Estimation In Study Design: A Meta-Epidemiological Analysis of Transcatheter Versus Surgical Aortic Valve Replacement Saerom Youn, Shannon Avery Wong, Caitlin Chrystoja, George Tomlinson, and 7 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-71534/v1 This work is licensed under a CC BY 4.0 License Status: Under Revision Version 1 posted 8 You are reading this latest preprint version Abstract Objective : To estimate the bias associated with specific nonrandomized study attributes among studies comparing transcatheter aortic valve implantation with surgical aortic valve replacement for the treatment of severe aortic stenosis. Data sources and study selection : We searched 7 databases from inception to June 2017: Medline, Medline In-Process/ePubs, Embase, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, Scopus, and Web of Science. We included all RCTs and nonrandomized studies that reported outcomes of interest. Data extraction and synthesis : We categorized studies according to study design, and evaluated 41 nonrandomized study attributes as potential sources of bias. We calculated odds ratios (OR) and other effect measures with 95% confidence intervals (CI) using random effects models. Main outcomes : One month postoperative mortality, and length of stay. Bias was defined as the difference in estimates of treatment effects between nonrandomized studies and high quality (low risk of bias) RCTs, which were considered to provide “gold standard” estimates. Results : We included 6 RCTs and 87 nonrandomized studies. Surgical risk scores were similar for comparison groups in RCTs, but were higher for patients having transcatheter aortic valve implantation in nonrandomized studies. Nonrandomized studies underestimated the benefit of transcatheter aortic valve implantation compared with RCTs. For example, nonrandomized studies without adjustment estimated a higher risk of postoperative mortality for transcatheter aortic valve implantation compared with surgical aortic valve replacement (OR 1.43 [95% CI, 1.26 to 1.62]) than high quality RCTs (OR 0.78 [95% CI, 0.54 to 1.11). Nonrandomized studies using propensity score matching (OR 1.13 [95% CI, 0.85 to 1.52]) and regression modelling (OR 0.68 [95% CI, 0.57 to 0.81]) to adjust results estimated treatment effects closer to high quality RCTs. Nonrandomized studies describing losses to follow-up estimated treatment effects that were significantly closer to high quality RCT than nonrandomized studies that did not. Conclusion : Studies with different attributes produce different estimates of treatment effects. Study design attributes related to the completeness of follow-up may explain biased treatment estimates in nonrandomized studies, as in the case of aortic valve replacement where high-risk patients were preferentially selected for the newer (transcatheter) procedure. Surgery General Surgery Medline Medline In-Process/ePubs postoperative mortality Bias Figures Figure 1 Figure 2 Figure 3 Introduction Frameworks of study designs often specify hierarchies based on the likelihood of estimating biased treatment effects, with well-designed randomized controlled trials (RCT) and their meta-analyses considered to provide the least biased estimates. 1 – 3 However, there are limited RCTs of non-drug technologies such as medical devices and surgical techniques, 45 leading to widespread dependence on non-randomized studies for the evaluation of non-drug health technologies. Not surprisingly, there is variation in the treatment effects estimated by different study designs, with non-randomized studies frequently reporting larger benefits for the experimental treatment than RCTs. 6 – 12 Differences in the conclusions of non-randomized studies and RCTs vary according to the clinical context. 13 – 16 Among RCTs, study quality is associated with estimates of treatment effects; lower quality RCTs often overestimate the benefit of an experimental procedure as compared to high quality RCTs. 17 – 21 The relationship between study attributes and biased treatment effects is less clear for nonrandomized studies—a better understanding of this relationship would help inform readers, providers, patients, and policy makers, especially when data from high-quality RCTs are not available . There are many nonrandomized studies and RCTs comparing transcatheter and surgical aortic valve replacement for the treatment of aortic stenosis, providing an ideal opportunity to study the influence of study designs and characteristics on estimates of treatment effectiveness. We sought to empirically explore the direction and magnitude of bias associated with different study attributes using a meta-epidemiological analysis of published studies. Methods The study was performed in accordance with the PRISMA guidelines for meta-epidemiological studies. 22 Clinical context We studied transcatheter and surgical aortic valve replacement for aortic stenosis because there were both high quality RCTs and a large number of non-randomized studies. Transcatheter aortic valve implantation is a relatively new technique, and its safety and efficacy is of current clinical interest. Study selection We included all RCTs that randomly assigned patients to transcatheter or surgical aortic valve replacement and followed patients over time. We also included all comparative cohort studies that reported primary data on outcomes of interest after transcatheter or surgical aortic valve replacement. We excluded non-randomized studies that were not comparative cohort studies, defined the population by excluding the outcome of interest, combined patients from RCTs and non-randomized studies, conference abstracts, poster presentations, non-peer reviewed publications, unpublished literature, systematic reviews that lacked primary data, and studies that used other surgical aortic valve replacement methods (e.g., minimally invasive, sutureless). For multiple publications using the identical cohort we included the publication with the most representative sample, determined by sample size or duration of follow up. Data sources We searched Medline, Medline In-Process/ePubs, Embase, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, Scopus, and Web of Science from inception to June 2017 (eTable 1). We used DistillerSR (Evidence Partners, Ottawa, Canada) to check for duplicate citations, and to screen titles, abstracts, and full text. Data extraction A single reviewer collected study characteristics, patient characteristics, and outcomes of interest; questions were resolved by consensus among the study team. Agreement of re-abstracted outcomes for a sample of 15 nonrandomized studies (17%) by a second reviewer demonstrated excellent inter-rater reliability (ICC 0.99 [95% CI, 0.98 to 0.99]). 23 Study characteristics We collected study sample size, publication year and country, surgical approach, and the study time period. We collected surgical risk scores (e.g., EuroSCORE II) as a measure of potential selection bias among comparison groups. Outcomes We defined postoperative mortality as death due to any cause within 1-month or in hospital after the procedure regardless of location. We defined length of stay as the number of days the patient stayed in the hospital after the procedure. We extracted the necessary components of each outcome to calculate the pooled estimates of treatment effects. We calculated missing data points using given information where possible. Explanatory variables: Study designs We categorized studies into 8 groups according to study design: (1) All (all RCT and nonrandomized studies), (2) All RCT, (3) High quality RCT, (4) Low quality RCT, (5) All non-randomized studies, (6) Nonrandomized studies without adjustment, (7) Nonrandomized studies adjusted using propensity score matching (PSM), and (8) Nonrandomized studies adjusted using regression. RCTs were divided into high or low quality RCTs based on the Cochrane Risk Of Bias (ROB) tool 24 based on the content of the published articles; authors were not contacted for additional information (eTable 3). No RCT blinded study participants; hence RCTs that satisfied all other criteria were categorized as high quality. Non-randomized studies reported unadjusted estimates, adjusted estimates, or both. Non-randomized studies estimates were pooled into 3 groups: without adjustment, adjusted using PSM, and adjusted using regression. Finally, we previously developed a set of 41 non-randomized studies attributes that could bias studies (Appendix B). These attributes were based on existing frameworks of bias and quality assessment tools for nonrandomized studies, and were extensively pilot tested and iteratively developed for clarity and reliability. Data synthesis Study characteristics We compared overall study characteristics between RCTs and non-randomized studies using descriptive statistics. To combine continuous variables across studies, the weighted mean of estimates was calculated, and the pooled standard deviation (SD) was either calculated directly (where reported) or imputed from the pooled variance of included studies in the relevant group if missing 25 . Pooled estimates of treatment effects The effect of treatment on postoperative mortality was estimated using odds ratio (OR). OR < 1 indicated lower risk of death for transcatheter aortic valve implantation. For Bayesian RCTs, we assumed the median estimate represented the percentage with events. 26,27 The treatment effect on length of stay was estimated using mean difference (MD, with values < 0 indicating shorter length of stay for transcatheter aortic valve implantation). All effect sizes were pooled using a random effects model to account for potential between-study heterogeneity. For postoperative mortality, we used the DerSimonian-Laird method, 28 with the exception of estimates that incorporated adjusted ORs from nonrandomized studies adjusted using regression, which were calculated using the generic inverse variance method 25 . For length of stay, we used the inverse variance method. 25 All pooled estimates were presented visually using forest plots with point estimates and 95% CI. Estimates from high-quality RCTs were considered to represent the “gold standard” treatment effects. We evaluated the impact of the 41 nonrandomized study attributes on estimates of treatment effect by calculating the ratio of odds ratios (ROR) for postoperative mortality and difference of mean differences (DMD) for length of stay with 95% CI using random effects meta regression. The ROR is the ratio of the OR in one group of studies and the OR in another group of studies 18 ; the DMD is the difference between MD reported in one group of studies and the MD in another group of studies. 29 We compared the pooled estimates between study categories, and also between nonrandomized studies with attributes hypothesized to be associated with bias. ROR < 1 and DMD < 0 indicated that studies with ‘better’ study characteristics favored transcatheter aortic valve implantation. All statistical analyses were conducted using R studio version 1.0.136 (2016). 30 The analysis of whether the attributes of nonrandomized studies were associated with statistical differences in pooled effect sizes was an exploratory analysis; a less restrictive 2-sided P value of 0.10 was used to determine potentially important attributes. In all other analyses a P value of 0.05 or less was considered statistically significant. P values for comparisons of estimates between types of study were those of the ROR or DMD for the comparison. Results Study selection Of 2,061 RCTs identified in our search, six (described in 15 publications) met the inclusion criteria, and of 10,409 nonrandomized studies, 87 (described in 88 publications) met the inclusion criteria (Fig. 1 and eTable 2). We included four additional nonrandomized studies from publications that were not identified in the initial search. Study Characteristics The six RCTs included 5,352 patients, and the 87 non-randomized studies included 239,433 patients (Table 1 ). RCTs and nonrandomized studies had similar years of publication, were conducted mostly in Europe and North America, and often used multiple surgical approaches. Table 1 Descriptive characteristics of included studies RCTs NRSs Number of studies 6 87 Total number of patients 5352 239433 Year published* 2014 (2012, 2016) 2014 (2012, 2016) Region Europe 2 (33.3%) 47 (54.0%) North America 2 (33.3%) 16 (18.4%) Asia 0 7 (8.0%) Other 0 3 (3.5%) Multiple 2 (33.3%) 2 (2.3%) Unclear 0 12 (13.8%) TAVI Approach Any 5 (83.3%) 57 (65.5%) Transfemoral 0 10 (11.5%) Transapical 1 (16.7%) 13 (14.9%) Other 0 7 (8.0%) TAVI SAVR TAVI SAVR Number of patients 2771 2581 78254 161179 Year enrolment began 2010 (2008, 2011) (n = 6) 2010 (2008, 2011) (n = 6) 2009 (2006–2011) (n = 79) 2007 (2005–2009) (n = 76) Year enrolment ended 2012 (2011, 2013) (n = 6) 2012 (2011, 2013) (n = 6) 2012 (2010–2013) (n = 75) 2012 (2010, 2013) (n = 72) Baseline surgical risk† STS 6.13 ± 2.25 (n = 5) 6.20 ± 2.32 (n = 5) 9.83 ± 5.03 (n = 34) 6.76 ± 3.68 (n = 33) EuroSCORE I NA NA 18.25 ± 8.61 (n = 8) 11.16 ± 5.26 (n = 8) LogEuroSCORE 16.21 ± 8.77 (n = 5) 16.30 ± 8.64 (n = 5) 22.32 ± 11.29 (n = 44) 14.19 ± 8.73 (n = 44) EuroSCORE II NA NA 8.52 ± 6.58 (n = 5) 8.09 ± 5.74 (n = 5) NYHA 2.75 (n = 3) 2.74 (n = 3) 3.40 (n = 12) 2.62 (n = 12) Abbreviations: RCT, Randomized Controlled Trial; NRS, Nonrandomized Study; TAVI, Transcatheter Aortic Valve Implantation; SAVR, Surgical Aortic Valve Replacement; STS, Society of Thoracic Surgeons; NYHA, New York Heart Association; NA, Not Applicable. All continuous variables are reported as either median (25th, 75th percentile) or mean ± SD. All discrete variables are reported as n (%). Values describing the characteristics of patients in each arm of the studies are followed by the number of studies each category that reported the value (n). * For studies with multiple publications, the year of the first publication was used. † STS, EuroSCORE I, LogEuroSCORE and EuroSCORE II are measures of predicted operative mortality. NYHA classifies the extent of heart failure into 4 classes I to IV, with I being least severe and IV being most severe. The numbers indicate the weighted average NYHA class of each cohort. ‘Other’ TAVI approaches included non-iliofemoral, transthoracic, or transvascular approaches. a) Postoperative mortality The proportion of studies including patients of all surgical risk categories was higher in non-randomized studies than RCTs (67.8% vs 33.3%,). In general, transcatheter aortic valve implantation subjects in nonrandomized studies had higher surgical risk compared to transcatheter aortic valve implantation subjects in RCTs or surgical aortic valve replacement subjects in nonrandomized studies and RCTs. Three RCTs satisfied modified ROB assessment criteria for “high quality” and three were “low quality” (Appendix C). Comparison Of Treatment Effects Between Rcts And Non-randomized Studies For postoperative mortality, nonrandomized studies adjusted using regression significantly favored transcatheter aortic valve implantation (OR 0.68 [95% CI, 0.57 to 0.81], P for comparison with high quality RCT 0.61). High quality RCTs (OR 0.78 [95% CI, 0.54 to 1.11]), low quality RCTs (OR, 0.8 [95% CI, 0.58 to 1.65], P for comparison with high quality RCT 0.48) and nonrandomized studies adjusted using PSM (OR, 1.13 [95% CI, 0.85 to 1.52], P for comparison with high quality RCT 0.18) found no statistical difference, while nonrandomized studies without adjustment significantly favored surgical aortic valve replacement (OR, 1.43 [95% CI, 1.26 to 1.62], P for comparison with high quality RCT 0.01). For length of stay, all categories of study design except for PSM-adjusted nonrandomized studies significantly favored transcatheter aortic valve implantation. However, there were differences in the magnitudes of the pooled point estimates. High quality RCTs reported a point estimate for the length of stay in the transcatheter group (MD -4.50 [95% CI, -5.05 to -3.96]) that was about 1.5 days shorter than low quality RCTs (MD -2.87 [95% CI, -5.13 to -0.61], P for comparison 0.26), nonrandomized studies adjusted using PSM (MD -3.01 [95% CI, -6.01 to 0], P for comparison 0.62), and nonrandomized studies without adjustment (MD -3.06 [95% CI, -3.89 to -2.24], P for comparison 0.33). No nonrandomized studies adjusted length of stay using regression. Influence of non-randomized study characteristics on estimates of treatment effect For each outcome, some attributes of nonrandomized studies were significantly (P < 0.10) associated with pooled estimates of treatment effect closer to those from high quality RCTs (Fig. 3 ). For postoperative mortality, these attributes were: losses to follow up described (P = 0.05), follow up equal in duration (P = 0.10), and conflict of interest disclosure for non-first/last authors (P = 0.10). For length of stay, these attributes were: losses to follow up described (P = 0.08), missing data addressed (P = 0.09), and outcome measured from interviews (P = 0.06). Discussion When comparing estimates of treatment effects in randomized and nonrandomized studies comparing transcatheter aortic valve implantation with surgical aortic valve replacement, we found that point estimates of the effect sizes of study designs with lower risk of bias tended to show larger benefit for transcatheter aortic valve implantation than study designs with higher risk of bias. Statistical adjustment using regression, but not propensity score matching, brought estimated effect sizes closer to high quality RCTs for postoperative mortality. Among nonrandomized studies, accounting for loss to follow up was associated with estimates of treatment effect closer to those from RCTs. Our findings are consistent with meta-analyses that found RCTs favored transcatheter aortic valve implantation more than nonrandomized studies with respect to postoperative mortality. 31 Interestingly, while meta-epidemiological studies of other clinical topics found that lower quality studies tend to overestimate the benefit of newer treatments, 19–21,32−36 higher risk of bias studies of transcatheter aortic valve implantation underestimated treatment benefit. There are several possible reasons for the discrepancy. Our analysis included recent studies, which are generally of higher quality and follow better reporting guidelines than older studies. 37 The difference may also be specific to the clinical context we studied. Allocation of patients to treatment groups is highly influenced by differences in case-mix. 38 The surgical risk of postoperative mortality was higher in patients who had transcatheter aortic valve implantation in nonrandomized studies, presumably because the transcatheter procedure was largely restricted to patients who were too high risk for surgical aortic valve replacement in the early years of its clinical use. This situation is different than other clinical situations, where newer or innovative procedures are preferentially used in lower-risk patients. 39 In our study, propensity score matching did not consistently shift estimates from nonrandomized studies closer to RCT estimates. Others found that propensity score-matched effect sizes from nonrandomized studies were closer to RCTs than regression modeling 38 . Of the various design attributes of nonrandomized studies we analyzed, studies that described loss to follow up yielded estimates of treatment effects that were closer to high quality RCTs. Study attributes related to baseline characteristics did not substantially affect effect estimates. Loss to follow up is a major source of selection bias in cohort studies; it is associated with socioeconomic status, 40 – 47 substance abuse, 41 smoking, 45,48−51 alcohol abuse, 43 , 52 physical inactivity, 49 , 52 , 53 and poor diet. 52 Quantifying the extent of bias due to loss to follow up may be helpful in understanding biased estimation of treatment effects in nonrandomized studies. Our study had important strengths. We focused on a single clinical question, allowing us to focus on the influence of study characteristics on estimated treatment effects without introducing other sources of variation from studying a heterogeneous group of interventions. We stratified RCTs by risk of bias, instead of pooling all RCTs together. Studies comparing transcatheter and surgical aortic valve replacement included several high quality RCTs, and many recent large and well-reported nonrandomized studies, enabling us to disentangle the influence of study quality and study characteristics on estimated treatment effects. Thirteen nonrandomized studies reported both adjusted and unadjusted treatment effects. Surgical risk scores allowed us to examine confounding by indication. Our study has limitations. Our literature review may not have included every potentially eligible study. However, this would not affect the internal consistency and generalizability of our findings, which focused on differential estimates between RCTs and nonrandomized studies, rather than estimating the independent treatment effect of aortic valve replacement techniques. Although we limited our analysis to a single clinical question, there is nevertheless substantial heterogeneity among the articles we analyzed that must be taken into account. Although we categorized studies by design, different studies included subjects from very different clinical populations (e.g., a high-risk population is completely different from an intermediate risk population). However, this situation is typical of the medical literature, and pooled measures of effect are commonly reported in meta-analyses even when clinical heterogeneity exists among included studies. Although some of the high-quality RCTs were designed as non-inferiority studies, they would still be expected to provide unbiased estimates of the relative effectiveness of transcatheter aortic valve implantation with respect to the outcomes we evaluated. We specified a priori a liberal P value threshold of 0.10 and performed multiple univariate comparisons to identify nonrandomized study attributes potentially associated with biased effect estimates. The intent of these exploratory analyses was to generate hypotheses about these study attributes for future analyses rather than test specific hypotheses. Many of these attributes are correlated, and further research could test specific hypothesis regarding the effect of a limited number of pre-specified attributes on bias. Further studies on the reliability of measured attributes of non-randomized studies and how they influence effect estimates compared with RCTs will help improve the interpretation of the results of nonrandomized studies. Finally, while a single reviewer collected the data for this study, analyses of inter-rater reliability demonstrated excellent correlation among a sample of key variables that were re-abstracted by a second reviewer. We found that higher quality studies reported a larger benefit than lower quality studies for transcatheter aortic valve replacement compared with surgical valve replacement, although differences were not statistically significant. While adjusted estimates of treatment effects in nonrandomized studies were generally closer to high quality RCT estimates, propensity score snatching and regression modelling varied in the extent to which they were able to adjust effect estimates closer to RCT estimates. Risk adjustment methods may not reliably account for biases in nonrandomized studies. Consideration of loss to follow up appears to be an important attribute of nonrandomized studies with respect to estimating treatment effects that are closer to RCT estimates. Declarations Ethics approval and consent to participate : The study was exempted from ethics review by the Research Ethics Board at the University of Toronto. Consent for publication : Not applicable. Funding : The study was funded by the Canadian Institutes of Health Research (CIHR) operating grant MOP-136787. Competing interests : The authors declare that they have no competing interests. Availability of data and materials : The datasets during and/or analysed during the current study available from the corresponding author on reasonable request. Author’s contributions : SY conducted data screening, collection, analysis, and interpretation. SY drafted and revised the manuscript. SAW collected data. CC assisted with data collection and analysis. GT assisted with data analysis and interpretation. HCW, CMB, ARG, NNB and LS assisted with data interpretation. JT provided administrative assistance. DRU is the corresponding author. All authors approved of the submitted version. Acknowledgement : We thank Eugene Kim, BEE JD (University of Ottawa) for help with coding and editing; Dr. Douglas Lee, MD PhD (Institute of Health Policy Management and Evaluation, University of Toronto) and Dr. Asim Cheema, MD PhD (Institute of Medical Sciences, University of Toronto) for helpful comments; the Human Brain Project (HBP) for supporting the presentation of project at the 4 th HBP School – Future computing: Brain Science and Artificial Intelligence. None received compensation for their roles in the study. References Sackett DL. Clinical Epidemiology: A Basic Science for Clinical Medicine . Little, Brown; 1991. Brighton B, Bhandari M, Tornetta P, Felson DT. Hierarchy of evidence: from case reports to randomized controlled trials. Clin Orthop Relat Res. 2003;Aug(413):19–24. doi: 10.1097/01.blo.0000079323.41006.12 . Burns P, Rohrich R, Chong K. The Levels of Evidence and their role in Evidence-Based Medicine. Plast Reconstr Surg. 2011;128(1):305–10. doi: 10.1097/PRS.0b013e318219c171.The . Wente MN, Seiler CM, Uhl W, Büchler MW. Perspectives of evidence-based surgery. Dig Surg. 2003;20(4):263–9. doi: 10.1159/000071183 . Califf RM, Zarin DA, Kramer JM, Sherman RE, Aberle LH, Tasneem A. Characteristics of clinical trials registered in ClinicalTrials.gov, 2007–2010. Jama. 2012;307(17):1838–47. doi: 10.1001/jama.2012.3424 . Huynh T, Perron S, O’Loughlin J, et al. Comparison of primary percutaneous coronary intervention and fibrinolytic therapy in ST-segment-elevation myocardial infarction: bayesian hierarchical meta-analyses of randomized controlled trials and observational studies. Circulation. 2009;119(24):3101–9. doi: 10.1161/CIRCULATIONAHA.108.793745 . Ioannidis JP, Haidich AB, Pappa M, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA. 2001;286(7):821–30. doi: 10.1001/jama.286.7.821 . Nicolaides K, Brizot MDL, Patel F, Snijders R. Comparison of chorionic villus sampling and amniocentesis for fetal karyotyping at 10–13 weeks’ gestation. Obstet Gynecol Surv. 1995;50(2):96–7. doi: 10.1097/00006254-199502000-00008 . Jha P, Flather M, Lonn E, Farkouh M, Yusuf S. The antioxidant vitamins and cardiovascular disease. A critical review of epidemiologic and clinical trial data. Ann Intern Med. 1995;123(11):860–72. http://www.ncbi.nlm.nih.gov/pubmed/7486470 . Pyorala S, Huttunen NP, Uhari M. A review and meta-analysis of hormonal treatment of cryptorchidism. J Clin Endocrinol Metab. 1995;80(9):2795–9. doi: 10.1210/jcem.80.9.7673426 . Kunz R, Vist GE, Oxman AD. Randomisation to protect against selection bias in healthcare trials. Cochrane Database Syst Rev. 2007;(2). doi: 10.1002/14651858.MR000012.pub2 . Kirtane AJ, Gupta A, Iyengar S, et al. Safety and efficacy of drug-eluting and bare metal stents: Comprehensive meta-analysis of randomized trials and observational studies. Circulation. 2009;119(25):3198–206. doi: 10.1161/CIRCULATIONAHA.108.826479 . Odgaard-Jensen J, Vist G, Timmer A, et al. Randomisation to protect against selection bias in healthcare trials (Review). Cochrane database Syst Rev. 2011;4:MR000012. doi: 10.1002/14651858.MR000012.pub3.www.cochranelibrary.com . Shikata S, Nakayama T, Noguchi Y, Taji Y, Yamagishi H. Comparison of effects in randomized controlled trials with observational studies in digestive surgery. Ann Surg. 2006;244(5):668–76. doi: 10.1097/01.sla.0000225356.04304.bc . Antman K, Amato D, Wood W, et al. Selection bias in clinical trials. J Clin Oncol. 1985;3(8):1142–7. . Britton A, McKee M, Black N, McPherson K, Sanderson C, Bain C. Choosing between randomised and non-randomised studies:a systematic review. Heal Technol Assess. 1998;2(1998):13. doi: 10.1136/bmj.317.7167.1258a . 1–136. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273(5):408–12. doi: 10.1001/jama.273.5.408 . Wood L, Egger M, Gluud LL, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: Meta-epidemiological study. BMJ. 2008;336(7644):601–5. doi: 10.1136/bmj.39465.451748.AD . Moher D, Pham B, Jones A, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet. 1998;352(9128):609–13. doi: 10.1016/S0140-6736(98)01085-X . Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med. 2001;135(11):982–9. doi: 10.7326/0003-4819-149-3-200808050-00023 . Pildal J, Hróbjartsson A, Jörgensen KJ, Hilden J, Altman DG, Gøtzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007;36(4):847–57. doi: 10.1093/ije/dym087 . Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22(4):139–42. doi: 10.1136/ebmed-2017-110713 . Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016. doi: 10.1016/j.jcm.2016.02.012 . Higgins JPT, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. Br Med J. 2011;343:889–93. doi: 10.1136/bmj.d5928 . Higgins J, S G, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [Updated March 2011]. The Cochrane Collaboration. . Published 2011. Wan X, Wang W, Liu J, Tong T. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol. 2014. doi: 10.1186/1471-2288-14-135 . Hozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol. 2005. doi: 10.1186/1471-2288-5-13 . DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986. doi: 10.1016/0197-2456(86)90046-2 . Page MJ, Higgins JPT, Clayton G, Sterne JAC, Hróbjartsson A, Savović J. Empirical evidence of study design biases in randomized trials: Systematic review of meta-epidemiological studies. PLoS One. 2016. doi: 10.1371/journal.pone.0159267 . RStudio Team. RStudio: Integrated Development for R. 2016. http://www.rstudio.com/ . Gargiulo G, Sannino A, Capodanno D, et al. Transcatheter aortic valve implantation versus surgical aortic valve replacement: A Systematic review and meta-analysis. Ann Intern Med. 2016;165(5):334–44. doi: 10.7326/M16-0060 . Hrobjartsson A, Thomsen A, Emanuelsson F, et al. Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. BMJ. 2012;344:e1119. doi: 10.1136/bmj.e1119 . Hróbjartsson A, Thomsen ASS, Emanuelsson F, et al. Observer bias in randomized clinical trials with time-to-event outcomes: Systematic review of trials with both blinded and non-blinded outcome assessors. Int J Epidemiol. 2014;43(3):937–48. doi: 10.1093/ije/dyt270 . Hartling L, Ospina M, Liang Y, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339(oct19 1):b4012–2. doi: 10.1136/bmj.b4012 . Tierney JF, Stewart LA. Investigating patient exclusion bias in meta-analysis. Int J Epidemiol. 2005;34(1):79–87. doi: 10.1093/ije/dyh300 . Nüesch E, Trelle S, Reichenbach S, et al. The effects of excluding patients from the analysis in randomised controlled trials: Meta-epidemiological study. BMJ. 2009;339(7722):679–83. doi: 10.1136/bmj.b3244 . Mcculloch P, Feinberg J, Philippou Y, et al. Progress in clinical research in surgery and IDEAL. Lancet. 2018;0(0). doi: 10.1016/S0140-6736(18)30102-8 . Deeks JJ, Dinnes J, D’Amico R, et al. Evaluating non-randomised intervention studies. Health Technol Assess (Rockv). 2003;7(27). doi: 10.3310/hta7270 . Amer MA, Herbison GP, Smith MD, Grainger SH, Khoo CH, McCall J. Bias in surgical randomised trials: a meta-epidemiological study using laparoscopic versus open surgery as an example. In: Abstracts of the 25th Cochrane Colloquium, Edinburgh, UK. Cochrane Database of Systematic Reviews; 2018:9 Suppl 1. Tin ST, Woodward A, Ameratunga S. Estimating bias from loss to follow-up in a prospective cohort study of bicycle crash injuries. Inj Prev. 2014;20(5):322–9. doi: 10.1136/injuryprev-2013-040997 . Corrigan JD, Harrison-Felix C, Bogner J, Dijkers M, Terrill MS, Whiteneck G. Systematic bias in traumatic brain injury outcome studies because of loss to follow-up. Arch Phys Med Rehabil. 2003;84(2):153–60. doi: 10.1053/apmr.2003.50093 . Osler M, Kriegbaum M, Christensen U, Lund R, Nybo Andersen AM. Loss to follow up did not bias associations between early life factors and adult depression. J Clin Epidemiol. 2008;61(9):958–63. doi: 10.1016/j.jclinepi.2007.11.005 . Osler M, Kriegbaum M, Christensen U, Holstein B, Nybo Andersen AM. Rapid Report on Methodology: Does Loss to Follow-up in a Cohort Study Bias Associations Between Early Life Factors and Lifestyle-Related Health Outcomes? Ann Epidemiol. 2008;18(5):422–4. doi: 10.1016/j.annepidem.2007.12.008 . Wolke D, Waylen A, Samara M, et al. Selective drop-out in longitudinal studies and non-biased prediction of behaviour disorders. Br J Psychiatry. 2009;195(3):249–56. doi: 10.1192/bjp.bp.108.053751 . Greene N, Greenland S, Olsen J, Nohr EA. Estimating bias from loss to follow-up in the danish national birth cohort. Epidemiology. 2011;22(6):815–22. doi: 10.1097/EDE.0b013e31822939fd . Carter KN, Imlach-Gunasekara F, McKenzie SK, Blakely T. Differential loss of participants does not necessarily cause selection bias. Aust N Z J Public Health. 2012;36(3):218–22. doi: 10.1111/j.1753-6405.2012.00867.x . Howe LD, Tilling K, Galobardes B, Lawlor DA. Loss to follow-up in cohort studies: Bias in estimates of socioeconomic inequalities. Epidemiology. 2013;24(1):1–9. doi: 10.1097/EDE.0b013e31827623b1 . Nilsen RM, Vollset SE, Gjessing HK, et al. Self-selection and bias in a large prospective pregnancy cohort in Norway. Paediatr Perinat Epidemiol. 2009;23(6):597–608. doi: 10.1111/j.1365-3016.2009.01062.x . Bjertness E, Sagatun Å, Green K, Lien L, Søgaard AJ, Selmer R. Response rates and selection problems, with emphasis on mental health variables and DNA sampling, in large population-based, cross-sectional and longitudinal studies of adolescents in Norway. BMC Public Health. 2010;10. doi: 10.1186/1471-2458-10-602 . Barchielli A, Balzi D. Nine-year follow-up of a survey on smoking habits in Florence (Italy): higher mortality among non-responders. Int J Epidemiol. 2002;31(5):1038–42. doi: 10.1093/ije/31.5.1038 . Verlato G, Melotti R, Olivieri M, et al. Asthmatics and ex-smokers respond early, heavy smokers respond late to mailed surveys in Italy. Respir Med. 2010;104(2):172–9. doi: 10.1016/j.rmed.2009.09.022 . Kypri K, Samaranayaka A, Connor J, Langley JD, Maclennan B. Non-response bias in a web-based health behaviour survey of New Zealand tertiary students. Prev Med (Baltim). 2011;53(4–5):274–7. doi: 10.1016/j.ypmed.2011.07.017 . Van Loon AJM, Tijhuis M, Picavet HSJ, Surtees PG, Ormel J. Survey non-response in the Netherlands: Effects on prevalence estimates and associations. Ann Epidemiol. 2003;13(2):105–10. doi: 10.1016/S1047-2797(02)00257-0 . Supplementary Files 200902PRISMA.doc 200902OnlineSupplement.docx Cite Share Download PDF Status: Under Revision Version 1 posted Editorial decision: Minor revision 23 Dec, 2020 Review # 1 received at journal 17 Oct, 2020 Reviewer # 1 agreed at journal 09 Oct, 2020 Reviewers invited by journal 08 Oct, 2020 Editor assigned by journal 02 Oct, 2020 Submission checks completed at journal 22 Sep, 2020 Editor invited by journal 18 Sep, 2020 First submitted to journal 02 Sep, 2020 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-71534","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research article","associatedPublications":[],"authors":[{"id":2640327,"identity":"0911d5aa-abe9-4207-a5f1-10916dbbcca2","order_by":0,"name":"Saerom Youn","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA9ElEQVRIiWNgGAWjYNCCCgk5fvYGBgYe4rWcsTGW7DlAihbGtrTEDTcSiNQiPyP5mHQB22HGhpuvEx+8+cOQuOF4A+OHH3i0GNxIS5OewXOYmXF27mbDuW1ALWcOMEv24NMikWMmzSNxmI1ZOnebNG8DQ+K2GwlseF0oPwOkxeAwD5vk2e2/ef5AtDD+weeZGyAtCWkSPBK825h52CBamPHZYnDmWbI1zwEbAwme3M2Sc9skjPefOdgsLYPPYe3JB2/z/pOo33/87MYPb/7YyM5sbz748Q0+hwkkoHAlgJixAZ8GBgb+A/jlR8EoGAWjYBQwAADLq02Ak8CX7QAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0001-5745-954X","institution":"Schulich School of Medicine and Dentistry, University of Western Ontario","correspondingAuthor":true,"prefix":"","firstName":"Saerom","middleName":"","lastName":"Youn","suffix":""},{"id":2640328,"identity":"f4c8d836-ee01-47ad-a266-30cb3874d2a5","order_by":1,"name":"Shannon Avery Wong","email":"","orcid":"","institution":"James Cook University","correspondingAuthor":false,"prefix":"","firstName":"Shannon","middleName":"Avery","lastName":"Wong","suffix":""},{"id":2640329,"identity":"4751dadb-cd74-4b83-9eb7-3c899c9a445b","order_by":2,"name":"Caitlin Chrystoja","email":"","orcid":"","institution":"Institute of Health Policy Management and Evaluation, University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Caitlin","middleName":"","lastName":"Chrystoja","suffix":""},{"id":2640330,"identity":"2bbb5714-9af0-4e34-a14c-702b2a633600","order_by":3,"name":"George Tomlinson","email":"","orcid":"","institution":"Institute of Health Policy Management, and Evaluation, University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"George","middleName":"","lastName":"Tomlinson","suffix":""},{"id":2640331,"identity":"b15c1829-f2b6-413c-8ebe-bd773112c231","order_by":4,"name":"Harindra C Wijeysundera","email":"","orcid":"","institution":"Institute of Health Policy Management and Evaluation, University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Harindra","middleName":"C","lastName":"Wijeysundera","suffix":""},{"id":2640332,"identity":"527fc877-0453-453c-a4b6-19aa82b48e51","order_by":5,"name":"Chaim M Bell","email":"","orcid":"","institution":"Institute of Health Policy Management and Evaluation, University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Chaim","middleName":"M","lastName":"Bell","suffix":""},{"id":2640333,"identity":"0f1649c9-6475-4c8f-9355-a2399354ac44","order_by":6,"name":"Anna R Gagliardi","email":"","orcid":"","institution":"Institute of Health Policy Management and Evaluation, University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Anna","middleName":"R","lastName":"Gagliardi","suffix":""},{"id":2640334,"identity":"814c5608-150d-4dc4-a671-1350b4b99ee3","order_by":7,"name":"Nancy N Baxter","email":"","orcid":"","institution":"Institute of Health Policy Management and Evaluation, University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Nancy","middleName":"N","lastName":"Baxter","suffix":""},{"id":2640335,"identity":"2d0f9095-0bb2-47a9-b01b-6eaa0351e7f3","order_by":8,"name":"Lakhir Sandhu","email":"","orcid":"","institution":"Department of Surgery, University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Lakhir","middleName":"","lastName":"Sandhu","suffix":""},{"id":2640336,"identity":"7bc0e1e6-46d4-415d-b997-3465e079f665","order_by":9,"name":"Julie Takata","email":"","orcid":"","institution":"Women's College Hospital Research Institute","correspondingAuthor":false,"prefix":"","firstName":"Julie","middleName":"","lastName":"Takata","suffix":""},{"id":2640337,"identity":"221bc641-3713-4431-b3d5-121849a06693","order_by":10,"name":"David Robert Urbach","email":"","orcid":"","institution":"Institute of Health Policy Management and Evaluation","correspondingAuthor":false,"prefix":"","firstName":"David","middleName":"Robert","lastName":"Urbach","suffix":""}],"badges":[],"createdAt":"2020-09-03 11:39:16","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-71534/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-71534/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":2578426,"identity":"847c150b-c36f-436d-8f1f-9a3fd45287de","added_by":"auto","created_at":"2020-09-24 14:53:41","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":61610,"visible":true,"origin":"","legend":"Flow diagram of literature search and screening to identify eligible studies\nRCT, randomized controlled trials; NRS, nonrandomized studies \n","description":"","filename":"1.JPG","url":"https://assets-eu.researchsquare.com/files/rs-71534/v1/1.JPG"},{"id":2578427,"identity":"7a40c76c-32a1-4223-8975-969c04e6da72","added_by":"auto","created_at":"2020-09-24 14:53:41","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":64528,"visible":true,"origin":"","legend":"Pooled estimates of treatment effects in different study designs\nAbbreviations: RCT, Randomized Controlled Trial; NRS, Non-Randomized Studies; PSM, Propensity Score Matching; OR, Odds Ratio; CI, Confidence Interval; MD, Mean Difference.\nEarly means ≤30 days post-op. Diamond is the point estimate of treatment effect. Horizontal lines are 95% CI. Odds ratios were plotted in log scale. Treatment effects were plotted to exact values, but were reported rounded to 2 decimal places.\n","description":"","filename":"2.JPG","url":"https://assets-eu.researchsquare.com/files/rs-71534/v1/2.JPG"},{"id":2578428,"identity":"cd246e90-65d3-464a-8456-ff49f2b07970","added_by":"auto","created_at":"2020-09-24 14:53:41","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":367869,"visible":true,"origin":"","legend":"Comparison of pooled estimates of treatment effect in NRSs stratified by specific NRS attributes\nAbbreviations: NRS, Non-Randomized Studies; OR, Odds Ratio; MD, Mean Difference; CI, Confidence Interval; COI, Conflict of Interest; IRB, Institutional Review Board; AS, aortic stenosis.\nAttributes were ordered by increasing Ratio of Odds Ratios (RORs) and Difference in Mean Differences (DMD) between the pooled estimates in each stratification. Diamond is the point estimate. Horizontal lines are 95% CI. Odds ratios were plotted in log scale. Treatment effects are plotted to exact values, but are reported rounded to 2 decimal places. Studies column shows how many studies out of total NRS were pooled to produce each estimate. Grey and white bars group rows that belong to each attribute for easier visibility. \n","description":"","filename":"3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-71534/v1/3.jpg"},{"id":13597929,"identity":"252cbb0f-da32-4047-a45d-6e527a531c55","added_by":"auto","created_at":"2021-09-17 05:34:03","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":595746,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-71534/v1/4e6d6e7d-1182-4382-bc9c-573beda45ff8.pdf"},{"id":2578430,"identity":"390de4c9-a7fb-46b9-aace-c96199c41be9","added_by":"auto","created_at":"2020-09-24 14:53:41","extension":"doc","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":65536,"visible":true,"origin":"","legend":"","description":"","filename":"200902PRISMA.doc","url":"https://assets-eu.researchsquare.com/files/rs-71534/v1/200902PRISMA.doc"},{"id":2578431,"identity":"1faafbb9-c592-4351-a68e-c5fcde19aeac","added_by":"auto","created_at":"2020-09-24 14:53:42","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":67515,"visible":true,"origin":"","legend":"","description":"","filename":"200902OnlineSupplement.docx","url":"https://assets-eu.researchsquare.com/files/rs-71534/v1/200902OnlineSupplement.docx"}],"financialInterests":"","formattedTitle":"\u003cp\u003eBias Estimation In Study Design: A Meta-Epidemiological Analysis of Transcatheter Versus Surgical Aortic Valve Replacement\u003c/p\u003e","fulltext":[{"header":"Introduction","content":" \u003cp\u003eFrameworks of study designs often specify hierarchies based on the likelihood of estimating biased treatment effects, with well-designed randomized controlled trials (RCT) and their meta-analyses considered to provide the least biased estimates.\u003csup\u003e\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e However, there are limited RCTs of non-drug technologies such as medical devices and surgical techniques,\u003csup\u003e\u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e45\u003c/span\u003e\u003c/sup\u003e leading to widespread dependence on non-randomized studies for the evaluation of non-drug health technologies.\u003c/p\u003e \u003cp\u003eNot surprisingly, there is variation in the treatment effects estimated by different study designs, with non-randomized studies frequently reporting larger benefits for the experimental treatment than RCTs.\u003csup\u003e\u003cspan additionalcitationids=\"CR7 CR8 CR9 CR10 CR11\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e Differences in the conclusions of non-randomized studies and RCTs vary according to the clinical context.\u003csup\u003e\u003cspan additionalcitationids=\"CR14 CR15\" citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e Among RCTs, study quality is associated with estimates of treatment effects; lower quality RCTs often overestimate the benefit of an experimental procedure as compared to high quality RCTs.\u003csup\u003e\u003cspan additionalcitationids=\"CR18 CR19 CR20\" citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e The relationship between study attributes and biased treatment effects is less clear for nonrandomized studies\u0026mdash;a better understanding of this relationship would help inform readers, providers, patients, and policy makers, especially when data from high-quality RCTs are not available .\u003c/p\u003e \u003cp\u003eThere are many nonrandomized studies and RCTs comparing transcatheter and surgical aortic valve replacement for the treatment of aortic stenosis, providing an ideal opportunity to study the influence of study designs and characteristics on estimates of treatment effectiveness. We sought to empirically explore the direction and magnitude of bias associated with different study attributes using a meta-epidemiological analysis of published studies.\u003c/p\u003e "},{"header":"Methods","content":"\u003cp\u003eThe study was performed in accordance with the PRISMA guidelines for meta-epidemiological studies.\u003csup\u003e22\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eClinical context\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe studied transcatheter and surgical aortic valve replacement for aortic stenosis because there were both high quality RCTs and a large number of non-randomized studies. Transcatheter aortic valve implantation is a relatively new technique, and its safety and efficacy is of current clinical interest.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStudy selection\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe included all RCTs that randomly assigned patients to transcatheter or surgical aortic valve replacement and followed patients over time. We also included all comparative cohort studies that reported primary data on outcomes of interest after transcatheter or surgical aortic valve replacement.\u003c/p\u003e\n\u003cp\u003eWe excluded non-randomized studies that were not comparative cohort studies, defined the population by excluding the outcome of interest, combined patients from RCTs and non-randomized studies, conference abstracts, poster presentations, non-peer reviewed publications, unpublished literature, systematic reviews that lacked primary data, and studies that used other surgical aortic valve replacement methods (e.g., minimally invasive, sutureless).\u003c/p\u003e\n\u003cp\u003eFor multiple publications using the identical cohort we included the publication with the most representative sample, determined by sample size or duration of follow up.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData sources\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe searched Medline, Medline In-Process/ePubs, Embase, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, Scopus, and Web of Science from inception to June 2017 (eTable 1). We used DistillerSR (Evidence Partners, Ottawa, Canada) to check for duplicate citations, and to screen titles, abstracts, and full text.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData extraction\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eA single reviewer collected study characteristics, patient characteristics, and outcomes of interest; questions were resolved by consensus among the study team. Agreement of re-abstracted outcomes for a sample of 15 nonrandomized studies (17%) by a second reviewer demonstrated excellent inter-rater reliability (ICC 0.99 [95% CI, 0.98 to 0.99]).\u003csup\u003e23\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eStudy characteristics\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eWe collected study sample size, publication year and country, surgical approach, and the study time period. We collected surgical risk scores (e.g., EuroSCORE II) as a measure of potential selection bias among comparison groups.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eOutcomes\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eWe defined postoperative mortality as death due to any cause within 1-month or in hospital after the procedure regardless of location. We defined length of stay as the number of days the patient stayed in the hospital after the procedure. We extracted the necessary components of each outcome to calculate the pooled estimates of treatment effects. We calculated missing data points using given information where possible.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eExplanatory variables: Study designs\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eWe categorized studies into 8 groups according to study design: (1) All (all RCT and nonrandomized studies), (2) All RCT, (3) High quality RCT, (4) Low quality RCT, (5) All non-randomized studies, (6) Nonrandomized studies without adjustment, (7) Nonrandomized studies adjusted using propensity score matching (PSM), and (8) Nonrandomized studies adjusted using regression.\u003c/p\u003e\n\u003cp\u003eRCTs were divided into high or low quality RCTs based on the Cochrane Risk Of Bias (ROB) tool\u003csup\u003e24\u003c/sup\u003e based on the content of the published articles; authors were not contacted for additional information (eTable 3). No RCT blinded study participants; hence RCTs that satisfied all other criteria were categorized as high quality. Non-randomized studies reported unadjusted estimates, adjusted estimates, or both. Non-randomized studies estimates were pooled into 3 groups: without adjustment, adjusted using PSM, and adjusted using regression.\u003c/p\u003e\n\u003cp\u003eFinally, we previously developed a set of 41 non-randomized studies attributes that could bias studies (Appendix B). These attributes were based on existing frameworks of bias and quality assessment tools for nonrandomized studies, and were extensively pilot tested and iteratively developed for clarity and reliability.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData synthesis\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eStudy characteristics\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eWe compared overall study characteristics between RCTs and non-randomized studies using descriptive statistics. To combine continuous variables across studies, the weighted mean of estimates was calculated, and the pooled standard deviation (SD) was either calculated directly (where reported) or imputed from the pooled variance of included studies in the relevant group if missing \u003csup\u003e25\u003c/sup\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003ePooled estimates of treatment effects\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThe effect of treatment on postoperative mortality was estimated using odds ratio (OR). OR \u0026lt; 1 indicated lower risk of death for transcatheter aortic valve implantation. For Bayesian RCTs, we assumed the median estimate represented the percentage with events.\u003csup\u003e26,27\u003c/sup\u003e The treatment effect on length of stay was estimated using mean difference (MD, with values \u0026lt; 0 indicating shorter length of stay for transcatheter aortic valve implantation).\u003c/p\u003e\n\u003cp\u003eAll effect sizes were pooled using a random effects model to account for potential between-study heterogeneity. For postoperative mortality, we used the DerSimonian-Laird method,\u003csup\u003e28\u003c/sup\u003e with the exception of estimates that incorporated adjusted ORs from nonrandomized studies adjusted using regression, which were calculated using the generic inverse variance method \u003csup\u003e25\u003c/sup\u003e. For length of stay, we used the inverse variance method.\u003csup\u003e25\u003c/sup\u003e All pooled estimates were presented visually using forest plots with point estimates and 95% CI. Estimates from high-quality RCTs were considered to represent the \u0026ldquo;gold standard\u0026rdquo; treatment effects.\u003c/p\u003e\n\u003cp\u003eWe evaluated the impact of the 41 nonrandomized study attributes on estimates of treatment effect by calculating the ratio of odds ratios (ROR) for postoperative mortality and difference of mean differences (DMD) for length of stay with 95% CI using random effects meta regression. The ROR is the ratio of the OR in one group of studies and the OR in another group of studies\u003csup\u003e18\u003c/sup\u003e; the DMD is the difference between MD reported in one group of studies and the MD in another group of studies.\u003csup\u003e29\u003c/sup\u003e We compared the pooled estimates between study categories, and also between nonrandomized studies with attributes hypothesized to be associated with bias. ROR \u0026lt; 1 and DMD \u0026lt; 0 indicated that studies with \u0026lsquo;better\u0026rsquo; study characteristics favored transcatheter aortic valve implantation.\u003c/p\u003e\n\u003cp\u003eAll statistical analyses were conducted using R studio version 1.0.136 (2016).\u003csup\u003e30\u003c/sup\u003e The analysis of whether the attributes of nonrandomized studies were associated with statistical differences in pooled effect sizes was an exploratory analysis; a less restrictive 2-sided P value of 0.10 was used to determine potentially important attributes. In all other analyses a P value of 0.05 or less was considered statistically significant. P values for comparisons of estimates between types of study were those of the ROR or DMD for the comparison.\u003c/p\u003e"},{"header":"Results","content":" \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eStudy selection\u003c/h2\u003e \u003cp\u003eOf 2,061 RCTs identified in our search, six (described in 15 publications) met the inclusion criteria, and of 10,409 nonrandomized studies, 87 (described in 88 publications) met the inclusion criteria (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e and eTable 2). We included four additional nonrandomized studies from publications that were not identified in the initial search.\u003c/p\u003e \u003c/div\u003e \n\u003ch2\u003eStudy Characteristics\u003c/h2\u003e\n \u003cp\u003eThe six RCTs included 5,352 patients, and the 87 non-randomized studies included 239,433 patients (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). RCTs and nonrandomized studies had similar years of publication, were conducted mostly in Europe and North America, and often used multiple surgical approaches.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eDescriptive characteristics of included studies\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRCTs\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eNRSs\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNumber of studies\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTotal number of patients\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e5352\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e239433\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eYear published*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2014 (2012, 2016)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2014 (2012, 2016)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRegion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEurope\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2 (33.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e47 (54.0%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNorth America\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2 (33.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e16 (18.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAsia\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e7 (8.0%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOther\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3 (3.5%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMultiple\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2 (33.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2 (2.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eUnclear\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e12 (13.8%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTAVI Approach\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAny\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e5 (83.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e57 (65.5%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTransfemoral\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e10 (11.5%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTransapical\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1 (16.7%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e13 (14.9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOther\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e7 (8.0%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTAVI\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSAVR\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTAVI\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eSAVR\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNumber of patients\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2771\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2581\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e78254\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e161179\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eYear enrolment began\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2010 (2008, 2011) (n\u0026thinsp;=\u0026thinsp;6)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2010 (2008, 2011) (n\u0026thinsp;=\u0026thinsp;6)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2009 (2006\u0026ndash;2011) (n\u0026thinsp;=\u0026thinsp;79)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e2007 (2005\u0026ndash;2009) (n\u0026thinsp;=\u0026thinsp;76)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eYear enrolment ended\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2012 (2011, 2013) (n\u0026thinsp;=\u0026thinsp;6)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2012 (2011, 2013) (n\u0026thinsp;=\u0026thinsp;6)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2012 (2010\u0026ndash;2013) (n\u0026thinsp;=\u0026thinsp;75)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e2012 (2010, 2013) (n\u0026thinsp;=\u0026thinsp;72)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBaseline surgical risk\u0026dagger;\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSTS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e6.13\u0026thinsp;\u0026plusmn;\u0026thinsp;2.25 (n\u0026thinsp;=\u0026thinsp;5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6.20\u0026thinsp;\u0026plusmn;\u0026thinsp;2.32 (n\u0026thinsp;=\u0026thinsp;5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e9.83\u0026thinsp;\u0026plusmn;\u0026thinsp;5.03 (n\u0026thinsp;=\u0026thinsp;34)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e6.76\u0026thinsp;\u0026plusmn;\u0026thinsp;3.68 (n\u0026thinsp;=\u0026thinsp;33)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEuroSCORE I\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e18.25\u0026thinsp;\u0026plusmn;\u0026thinsp;8.61 (n\u0026thinsp;=\u0026thinsp;8)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e11.16\u0026thinsp;\u0026plusmn;\u0026thinsp;5.26 (n\u0026thinsp;=\u0026thinsp;8)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLogEuroSCORE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e16.21\u0026thinsp;\u0026plusmn;\u0026thinsp;8.77 (n\u0026thinsp;=\u0026thinsp;5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e16.30\u0026thinsp;\u0026plusmn;\u0026thinsp;8.64 (n\u0026thinsp;=\u0026thinsp;5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e22.32\u0026thinsp;\u0026plusmn;\u0026thinsp;11.29 (n\u0026thinsp;=\u0026thinsp;44)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e14.19\u0026thinsp;\u0026plusmn;\u0026thinsp;8.73 (n\u0026thinsp;=\u0026thinsp;44)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEuroSCORE II\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eNA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e8.52\u0026thinsp;\u0026plusmn;\u0026thinsp;6.58 (n\u0026thinsp;=\u0026thinsp;5)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e8.09\u0026thinsp;\u0026plusmn;\u0026thinsp;5.74 (n\u0026thinsp;=\u0026thinsp;5)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNYHA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2.75 (n\u0026thinsp;=\u0026thinsp;3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2.74 (n\u0026thinsp;=\u0026thinsp;3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3.40 (n\u0026thinsp;=\u0026thinsp;12)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e2.62 (n\u0026thinsp;=\u0026thinsp;12)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003eAbbreviations: RCT, Randomized Controlled Trial; NRS, Nonrandomized Study; TAVI, Transcatheter Aortic Valve Implantation; SAVR, Surgical Aortic Valve Replacement; STS, Society of Thoracic Surgeons; NYHA, New York Heart Association; NA, Not Applicable.\u003c/td\u003e\u003c/tr\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003eAll continuous variables are reported as either median (25th, 75th percentile) or mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD. All discrete variables are reported as n (%).\u003c/td\u003e\u003c/tr\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003eValues describing the characteristics of patients in each arm of the studies are followed by the number of studies each category that reported the value (n).\u003c/td\u003e\u003c/tr\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003e* For studies with multiple publications, the year of the first publication was used.\u003c/td\u003e\u003c/tr\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003e\u0026dagger; STS, EuroSCORE I, LogEuroSCORE and EuroSCORE II are measures of predicted operative mortality. NYHA classifies the extent of heart failure into 4 classes I to IV, with I being least severe and IV being most severe. The numbers indicate the weighted average NYHA class of each cohort. \u0026lsquo;Other\u0026rsquo; TAVI approaches included non-iliofemoral, transthoracic, or transvascular approaches.\u003c/td\u003e\u003c/tr\u003e \u003ctr\u003e\u003ctd colspan=\"5\"\u003ea) Postoperative mortality\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe proportion of studies including patients of all surgical risk categories was higher in non-randomized studies than RCTs (67.8% vs 33.3%,). In general, transcatheter aortic valve implantation subjects in nonrandomized studies had higher surgical risk compared to transcatheter aortic valve implantation subjects in RCTs or surgical aortic valve replacement subjects in nonrandomized studies and RCTs.\u003c/p\u003e \u003cp\u003eThree RCTs satisfied modified ROB assessment criteria for \u0026ldquo;high quality\u0026rdquo; and three were \u0026ldquo;low quality\u0026rdquo; (Appendix C).\u003c/p\u003e \n\u003ch2\u003eComparison Of Treatment Effects Between Rcts And Non-randomized Studies\u003c/h2\u003e\n \u003cp\u003eFor postoperative mortality, nonrandomized studies adjusted using regression significantly favored transcatheter aortic valve implantation (OR 0.68 [95% CI, 0.57 to 0.81], P for comparison with high quality RCT 0.61). High quality RCTs (OR 0.78 [95% CI, 0.54 to 1.11]), low quality RCTs (OR, 0.8 [95% CI, 0.58 to 1.65], P for comparison with high quality RCT 0.48) and nonrandomized studies adjusted using PSM (OR, 1.13 [95% CI, 0.85 to 1.52], P for comparison with high quality RCT 0.18) found no statistical difference, while nonrandomized studies without adjustment significantly favored surgical aortic valve replacement (OR, 1.43 [95% CI, 1.26 to 1.62], P for comparison with high quality RCT 0.01).\u003c/p\u003e \u003cp\u003eFor length of stay, all categories of study design except for PSM-adjusted nonrandomized studies significantly favored transcatheter aortic valve implantation. However, there were differences in the magnitudes of the pooled point estimates. High quality RCTs reported a point estimate for the length of stay in the transcatheter group (MD -4.50 [95% CI, -5.05 to -3.96]) that was about 1.5 days shorter than low quality RCTs (MD -2.87 [95% CI, -5.13 to -0.61], P for comparison 0.26), nonrandomized studies adjusted using PSM (MD -3.01 [95% CI, -6.01 to 0], P for comparison 0.62), and nonrandomized studies without adjustment (MD -3.06 [95% CI, -3.89 to -2.24], P for comparison 0.33). No nonrandomized studies adjusted length of stay using regression.\u003c/p\u003e \u003cp\u003e \u003cb\u003eInfluence of non-randomized study characteristics on estimates of treatment effect\u003c/b\u003e \u003c/p\u003e \u003cp\u003eFor each outcome, some attributes of nonrandomized studies were significantly (P\u0026thinsp;\u0026lt;\u0026thinsp;0.10) associated with pooled estimates of treatment effect closer to those from high quality RCTs (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). For postoperative mortality, these attributes were: losses to follow up described (P\u0026thinsp;=\u0026thinsp;0.05), follow up equal in duration (P\u0026thinsp;=\u0026thinsp;0.10), and conflict of interest disclosure for non-first/last authors (P\u0026thinsp;=\u0026thinsp;0.10). For length of stay, these attributes were: losses to follow up described (P\u0026thinsp;=\u0026thinsp;0.08), missing data addressed (P\u0026thinsp;=\u0026thinsp;0.09), and outcome measured from interviews (P\u0026thinsp;=\u0026thinsp;0.06).\u003c/p\u003e "},{"header":"Discussion","content":" \u003cp\u003eWhen comparing estimates of treatment effects in randomized and nonrandomized studies comparing transcatheter aortic valve implantation with surgical aortic valve replacement, we found that point estimates of the effect sizes of study designs with lower risk of bias tended to show larger benefit for transcatheter aortic valve implantation than study designs with higher risk of bias. Statistical adjustment using regression, but not propensity score matching, brought estimated effect sizes closer to high quality RCTs for postoperative mortality. Among nonrandomized studies, accounting for loss to follow up was associated with estimates of treatment effect closer to those from RCTs.\u003c/p\u003e \u003cp\u003eOur findings are consistent with meta-analyses that found RCTs favored transcatheter aortic valve implantation more than nonrandomized studies with respect to postoperative mortality.\u003csup\u003e\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u003c/sup\u003e Interestingly, while meta-epidemiological studies of other clinical topics found that lower quality studies tend to \u003cem\u003eoverestimate\u003c/em\u003e the benefit of newer treatments,\u003csup\u003e19\u0026ndash;21,32\u0026minus;36\u003c/sup\u003e higher risk of bias studies of transcatheter aortic valve implantation \u003cem\u003eunderestimated\u003c/em\u003e treatment benefit. There are several possible reasons for the discrepancy. Our analysis included recent studies, which are generally of higher quality and follow better reporting guidelines than older studies.\u003csup\u003e\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e\u003c/sup\u003e The difference may also be specific to the clinical context we studied. Allocation of patients to treatment groups is highly influenced by differences in case-mix.\u003csup\u003e\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e\u003c/sup\u003e The surgical risk of postoperative mortality was higher in patients who had transcatheter aortic valve implantation in nonrandomized studies, presumably because the transcatheter procedure was largely restricted to patients who were too high risk for surgical aortic valve replacement in the early years of its clinical use. This situation is different than other clinical situations, where newer or innovative procedures are preferentially used in lower-risk patients.\u003csup\u003e\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eIn our study, propensity score matching did not consistently shift estimates from nonrandomized studies closer to RCT estimates. Others found that propensity score-matched effect sizes from nonrandomized studies were closer to RCTs than regression modeling \u003csup\u003e\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eOf the various design attributes of nonrandomized studies we analyzed, studies that described loss to follow up yielded estimates of treatment effects that were closer to high quality RCTs. Study attributes related to baseline characteristics did not substantially affect effect estimates. Loss to follow up is a major source of selection bias in cohort studies; it is associated with socioeconomic status,\u003csup\u003e\u003cspan additionalcitationids=\"CR41 CR42 CR43 CR44 CR45 CR46\" citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e47\u003c/span\u003e\u003c/sup\u003e substance abuse,\u003csup\u003e\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e\u003c/sup\u003e smoking,\u003csup\u003e45,48\u0026minus;51\u003c/sup\u003e alcohol abuse,\u003csup\u003e\u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e,\u003cspan citationid=\"CR52\" class=\"CitationRef\"\u003e52\u003c/span\u003e\u003c/sup\u003e physical inactivity,\u003csup\u003e\u003cspan citationid=\"CR49\" class=\"CitationRef\"\u003e49\u003c/span\u003e,\u003cspan citationid=\"CR52\" class=\"CitationRef\"\u003e52\u003c/span\u003e,\u003cspan citationid=\"CR53\" class=\"CitationRef\"\u003e53\u003c/span\u003e\u003c/sup\u003e and poor diet.\u003csup\u003e\u003cspan citationid=\"CR52\" class=\"CitationRef\"\u003e52\u003c/span\u003e\u003c/sup\u003e Quantifying the extent of bias due to loss to follow up may be helpful in understanding biased estimation of treatment effects in nonrandomized studies. Our study had important strengths. We focused on a single clinical question, allowing us to focus on the influence of study characteristics on estimated treatment effects without introducing other sources of variation from studying a heterogeneous group of interventions. We stratified RCTs by risk of bias, instead of pooling all RCTs together. Studies comparing transcatheter and surgical aortic valve replacement included several high quality RCTs, and many recent large and well-reported nonrandomized studies, enabling us to disentangle the influence of study quality and study characteristics on estimated treatment effects. Thirteen nonrandomized studies reported both adjusted and unadjusted treatment effects. Surgical risk scores allowed us to examine confounding by indication.\u003c/p\u003e \u003cp\u003eOur study has limitations. Our literature review may not have included every potentially eligible study. However, this would not affect the internal consistency and generalizability of our findings, which focused on differential estimates between RCTs and nonrandomized studies, rather than estimating the independent treatment effect of aortic valve replacement techniques. Although we limited our analysis to a single clinical question, there is nevertheless substantial heterogeneity among the articles we analyzed that must be taken into account. Although we categorized studies by design, different studies included subjects from very different clinical populations (e.g., a high-risk population is completely different from an intermediate risk population). However, this situation is typical of the medical literature, and pooled measures of effect are commonly reported in meta-analyses even when clinical heterogeneity exists among included studies. Although some of the high-quality RCTs were designed as non-inferiority studies, they would still be expected to provide unbiased estimates of the relative effectiveness of transcatheter aortic valve implantation with respect to the outcomes we evaluated.\u003c/p\u003e \u003cp\u003eWe specified a priori a liberal P value threshold of 0.10 and performed multiple univariate comparisons to identify nonrandomized study attributes potentially associated with biased effect estimates. The intent of these exploratory analyses was to generate hypotheses about these study attributes for future analyses rather than test specific hypotheses. Many of these attributes are correlated, and further research could test specific hypothesis regarding the effect of a limited number of pre-specified attributes on bias. Further studies on the reliability of measured attributes of non-randomized studies and how they influence effect estimates compared with RCTs will help improve the interpretation of the results of nonrandomized studies. Finally, while a single reviewer collected the data for this study, analyses of inter-rater reliability demonstrated excellent correlation among a sample of key variables that were re-abstracted by a second reviewer.\u003c/p\u003e \u003cp\u003eWe found that higher quality studies reported a larger benefit than lower quality studies for transcatheter aortic valve replacement compared with surgical valve replacement, although differences were not statistically significant. While adjusted estimates of treatment effects in nonrandomized studies were generally closer to high quality RCT estimates, propensity score snatching and regression modelling varied in the extent to which they were able to adjust effect estimates closer to RCT estimates. Risk adjustment methods may not reliably account for biases in nonrandomized studies. Consideration of loss to follow up appears to be an important attribute of nonrandomized studies with respect to estimating treatment effects that are closer to RCT estimates.\u003c/p\u003e "},{"header":"Declarations","content":"\u003cp\u003e\u003cem\u003eEthics approval and consent to participate\u003c/em\u003e: The study was exempted from ethics review by the Research Ethics Board at the University of Toronto.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eConsent for publication\u003c/em\u003e: Not applicable.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eFunding\u003c/em\u003e: The study was funded by the Canadian Institutes of Health Research (CIHR) operating grant MOP-136787.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eCompeting interests\u003c/em\u003e: The authors declare that they have no competing interests.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eAvailability of data and materials\u003c/em\u003e: The datasets during and/or analysed during the current study available from the corresponding author on reasonable request.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eAuthor\u0026rsquo;s contributions\u003c/em\u003e: SY conducted data screening, collection, analysis, and interpretation. SY drafted and revised the manuscript. SAW collected data. CC assisted with data collection and analysis. GT assisted with data analysis and interpretation. HCW, CMB, ARG, NNB and LS assisted with data interpretation. JT provided administrative assistance. DRU is the corresponding author. All authors approved of the submitted version.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eAcknowledgement\u003c/em\u003e: We thank Eugene Kim, BEE JD (University of Ottawa) for help with coding and editing; Dr. Douglas Lee, MD PhD (Institute of Health Policy Management and Evaluation, University of Toronto) and Dr. Asim Cheema, MD PhD (Institute of Medical Sciences, University of Toronto) for helpful comments; the Human Brain Project (HBP) for supporting the presentation of project at the 4\u003csup\u003eth\u003c/sup\u003e HBP School \u0026ndash; Future computing: Brain Science and Artificial Intelligence. None received compensation for their roles in the study.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e \u003cspan\u003eSackett DL. \u003cem\u003eClinical Epidemiology: A Basic Science for Clinical Medicine\u003c/em\u003e. Little, Brown; 1991.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eBrighton B, Bhandari M, Tornetta P, Felson DT. Hierarchy of evidence: from case reports to randomized controlled trials. Clin Orthop Relat Res. 2003;Aug(413):19\u0026ndash;24. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1097/01.blo.0000079323.41006.12\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eBurns P, Rohrich R, Chong K. The Levels of Evidence and their role in Evidence-Based Medicine. Plast Reconstr Surg. 2011;128(1):305\u0026ndash;10. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1097/PRS.0b013e318219c171.The\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eWente MN, Seiler CM, Uhl W, B\u0026uuml;chler MW. Perspectives of evidence-based surgery. Dig Surg. 2003;20(4):263\u0026ndash;9. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1159/000071183\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eCaliff RM, Zarin DA, Kramer JM, Sherman RE, Aberle LH, Tasneem A. Characteristics of clinical trials registered in ClinicalTrials.gov, 2007\u0026ndash;2010. Jama. 2012;307(17):1838\u0026ndash;47. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1001/jama.2012.3424\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHuynh T, Perron S, O\u0026rsquo;Loughlin J, et al. Comparison of primary percutaneous coronary intervention and fibrinolytic therapy in ST-segment-elevation myocardial infarction: bayesian hierarchical meta-analyses of randomized controlled trials and observational studies. Circulation. 2009;119(24):3101\u0026ndash;9. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1161/CIRCULATIONAHA.108.793745\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eIoannidis JP, Haidich AB, Pappa M, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA. 2001;286(7):821\u0026ndash;30. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1001/jama.286.7.821\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eNicolaides K, Brizot MDL, Patel F, Snijders R. Comparison of chorionic villus sampling and amniocentesis for fetal karyotyping at 10\u0026ndash;13 weeks\u0026rsquo; gestation. Obstet Gynecol Surv. 1995;50(2):96\u0026ndash;7. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1097/00006254-199502000-00008\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eJha P, Flather M, Lonn E, Farkouh M, Yusuf S. The antioxidant vitamins and cardiovascular disease. A critical review of epidemiologic and clinical trial data. Ann Intern Med. 1995;123(11):860\u0026ndash;72. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.ncbi.nlm.nih.gov/pubmed/7486470\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003ePyorala S, Huttunen NP, Uhari M. A review and meta-analysis of hormonal treatment of cryptorchidism. J Clin Endocrinol Metab. 1995;80(9):2795\u0026ndash;9. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1210/jcem.80.9.7673426\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eKunz R, Vist GE, Oxman AD. Randomisation to protect against selection bias in healthcare trials. Cochrane Database Syst Rev. 2007;(2). doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1002/14651858.MR000012.pub2\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eKirtane AJ, Gupta A, Iyengar S, et al. Safety and efficacy of drug-eluting and bare metal stents: Comprehensive meta-analysis of randomized trials and observational studies. Circulation. 2009;119(25):3198\u0026ndash;206. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1161/CIRCULATIONAHA.108.826479\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eOdgaard-Jensen J, Vist G, Timmer A, et al. Randomisation to protect against selection bias in healthcare trials (Review). Cochrane database Syst Rev. 2011;4:MR000012. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1002/14651858.MR000012.pub3.www.cochranelibrary.com\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eShikata S, Nakayama T, Noguchi Y, Taji Y, Yamagishi H. Comparison of effects in randomized controlled trials with observational studies in digestive surgery. Ann Surg. 2006;244(5):668\u0026ndash;76. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1097/01.sla.0000225356.04304.bc\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAntman K, Amato D, Wood W, et al. Selection bias in clinical trials. J Clin Oncol. 1985;3(8):1142\u0026ndash;7. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eBritton A, McKee M, Black N, McPherson K, Sanderson C, Bain C. Choosing between randomised and non-randomised studies:a systematic review. Heal Technol Assess. 1998;2(1998):13. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1136/bmj.317.7167.1258a\u003c/span\u003e\u003c/span\u003e. 1\u0026ndash;136.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eSchulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273(5):408\u0026ndash;12. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1001/jama.273.5.408\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eWood L, Egger M, Gluud LL, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: Meta-epidemiological study. BMJ. 2008;336(7644):601\u0026ndash;5. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1136/bmj.39465.451748.AD\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eMoher D, Pham B, Jones A, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet. 1998;352(9128):609\u0026ndash;13. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/S0140-6736(98)01085-X\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eKjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med. 2001;135(11):982\u0026ndash;9. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.7326/0003-4819-149-3-200808050-00023\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003ePildal J, Hr\u0026oacute;bjartsson A, J\u0026ouml;rgensen KJ, Hilden J, Altman DG, G\u0026oslash;tzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007;36(4):847\u0026ndash;57. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/ije/dym087\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eMurad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22(4):139\u0026ndash;42. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1136/ebmed-2017-110713\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eKoo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jcm.2016.02.012\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHiggins JPT, Altman DG, G\u0026oslash;tzsche PC, et al. The Cochrane Collaboration\u0026rsquo;s tool for assessing risk of bias in randomised trials. Br Med J. 2011;343:889\u0026ndash;93. doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1136/bmj.d5928\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHiggins J, S G, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [Updated March 2011]. The Cochrane Collaboration. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e\u003c/span\u003e\u003c/span\u003e. Published 2011.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eWan X, Wang W, Liu J, Tong T. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol. 2014. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1186/1471-2288-14-135\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol. 2005. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1186/1471-2288-5-13\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eDerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/0197-2456(86)90046-2\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003ePage MJ, Higgins JPT, Clayton G, Sterne JAC, Hr\u0026oacute;bjartsson A, Savović J. Empirical evidence of study design biases in randomized trials: Systematic review of meta-epidemiological studies. PLoS One. 2016. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1371/journal.pone.0159267\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eRStudio Team. RStudio: Integrated Development for R. 2016. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.rstudio.com/\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eGargiulo G, Sannino A, Capodanno D, et al. Transcatheter aortic valve implantation versus surgical aortic valve replacement: A Systematic review and meta-analysis. Ann Intern Med. 2016;165(5):334\u0026ndash;44. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.7326/M16-0060\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHrobjartsson A, Thomsen A, Emanuelsson F, et al. Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. BMJ. 2012;344:e1119. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1136/bmj.e1119\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHr\u0026oacute;bjartsson A, Thomsen ASS, Emanuelsson F, et al. Observer bias in randomized clinical trials with time-to-event outcomes: Systematic review of trials with both blinded and non-blinded outcome assessors. Int J Epidemiol. 2014;43(3):937\u0026ndash;48. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/ije/dyt270\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHartling L, Ospina M, Liang Y, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339(oct19 1):b4012\u0026ndash;2. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1136/bmj.b4012\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eTierney JF, Stewart LA. Investigating patient exclusion bias in meta-analysis. Int J Epidemiol. 2005;34(1):79\u0026ndash;87. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/ije/dyh300\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eN\u0026uuml;esch E, Trelle S, Reichenbach S, et al. The effects of excluding patients from the analysis in randomised controlled trials: Meta-epidemiological study. BMJ. 2009;339(7722):679\u0026ndash;83. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1136/bmj.b3244\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eMcculloch P, Feinberg J, Philippou Y, et al. Progress in clinical research in surgery and IDEAL. Lancet. 2018;0(0). doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/S0140-6736(18)30102-8\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eDeeks JJ, Dinnes J, D\u0026rsquo;Amico R, et al. Evaluating non-randomised intervention studies. Health Technol Assess (Rockv). 2003;7(27). doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3310/hta7270\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAmer MA, Herbison GP, Smith MD, Grainger SH, Khoo CH, McCall J. Bias in surgical randomised trials: a meta-epidemiological study using laparoscopic versus open surgery as an example. In: \u003cem\u003eAbstracts of the 25th Cochrane Colloquium, Edinburgh, UK.\u003c/em\u003e Cochrane Database of Systematic Reviews; 2018:9 Suppl 1.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eTin ST, Woodward A, Ameratunga S. Estimating bias from loss to follow-up in a prospective cohort study of bicycle crash injuries. Inj Prev. 2014;20(5):322\u0026ndash;9. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1136/injuryprev-2013-040997\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eCorrigan JD, Harrison-Felix C, Bogner J, Dijkers M, Terrill MS, Whiteneck G. Systematic bias in traumatic brain injury outcome studies because of loss to follow-up. Arch Phys Med Rehabil. 2003;84(2):153\u0026ndash;60. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1053/apmr.2003.50093\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eOsler M, Kriegbaum M, Christensen U, Lund R, Nybo Andersen AM. Loss to follow up did not bias associations between early life factors and adult depression. J Clin Epidemiol. 2008;61(9):958\u0026ndash;63. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jclinepi.2007.11.005\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eOsler M, Kriegbaum M, Christensen U, Holstein B, Nybo Andersen AM. Rapid Report on Methodology: Does Loss to Follow-up in a Cohort Study Bias Associations Between Early Life Factors and Lifestyle-Related Health Outcomes? Ann Epidemiol. 2008;18(5):422\u0026ndash;4. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.annepidem.2007.12.008\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eWolke D, Waylen A, Samara M, et al. Selective drop-out in longitudinal studies and non-biased prediction of behaviour disorders. Br J Psychiatry. 2009;195(3):249\u0026ndash;56. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1192/bjp.bp.108.053751\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eGreene N, Greenland S, Olsen J, Nohr EA. Estimating bias from loss to follow-up in the danish national birth cohort. Epidemiology. 2011;22(6):815\u0026ndash;22. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1097/EDE.0b013e31822939fd\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eCarter KN, Imlach-Gunasekara F, McKenzie SK, Blakely T. Differential loss of participants does not necessarily cause selection bias. Aust N Z J Public Health. 2012;36(3):218\u0026ndash;22. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1111/j.1753-6405.2012.00867.x\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHowe LD, Tilling K, Galobardes B, Lawlor DA. Loss to follow-up in cohort studies: Bias in estimates of socioeconomic inequalities. Epidemiology. 2013;24(1):1\u0026ndash;9. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1097/EDE.0b013e31827623b1\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eNilsen RM, Vollset SE, Gjessing HK, et al. Self-selection and bias in a large prospective pregnancy cohort in Norway. Paediatr Perinat Epidemiol. 2009;23(6):597\u0026ndash;608. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1111/j.1365-3016.2009.01062.x\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eBjertness E, Sagatun \u0026Aring;, Green K, Lien L, S\u0026oslash;gaard AJ, Selmer R. Response rates and selection problems, with emphasis on mental health variables and DNA sampling, in large population-based, cross-sectional and longitudinal studies of adolescents in Norway. BMC Public Health. 2010;10. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1186/1471-2458-10-602\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eBarchielli A, Balzi D. Nine-year follow-up of a survey on smoking habits in Florence (Italy): higher mortality among non-responders. Int J Epidemiol. 2002;31(5):1038\u0026ndash;42. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/ije/31.5.1038\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eVerlato G, Melotti R, Olivieri M, et al. Asthmatics and ex-smokers respond early, heavy smokers respond late to mailed surveys in Italy. Respir Med. 2010;104(2):172\u0026ndash;9. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.rmed.2009.09.022\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eKypri K, Samaranayaka A, Connor J, Langley JD, Maclennan B. Non-response bias in a web-based health behaviour survey of New Zealand tertiary students. Prev Med (Baltim). 2011;53(4\u0026ndash;5):274\u0026ndash;7. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.ypmed.2011.07.017\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eVan Loon AJM, Tijhuis M, Picavet HSJ, Surtees PG, Ormel J. Survey non-response in the Netherlands: Effects on prevalence estimates and associations. Ann Epidemiol. 2003;13(2):105\u0026ndash;10. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/S1047-2797(02)00257-0\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e \u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-surgery","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"bsur","sideBox":"Learn more about [BMC Surgery](http://bmcsurg.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/bsur/default.aspx","title":"BMC Surgery","twitterHandle":"@BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Medline, Medline In-Process/ePubs, postoperative mortality, Bias ","lastPublishedDoi":"10.21203/rs.3.rs-71534/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-71534/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eObjective\u003c/strong\u003e: To estimate the bias associated with specific nonrandomized study attributes among studies comparing transcatheter aortic valve implantation with surgical aortic valve replacement for the treatment of severe aortic stenosis.\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eData sources and study selection\u003c/strong\u003e: We searched 7 databases from inception to June 2017: Medline, Medline In-Process/ePubs, Embase, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, Scopus, and Web of Science. We included all RCTs and nonrandomized studies that reported outcomes of interest.\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eData extraction and synthesis\u003c/strong\u003e: We categorized studies according to study design, and evaluated 41 nonrandomized study attributes as potential sources of bias. We calculated odds ratios (OR) and other effect measures with 95% confidence intervals (CI) using random effects models.\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eMain outcomes\u003c/strong\u003e: One month postoperative mortality, and length of stay. Bias was defined as the difference in estimates of treatment effects between nonrandomized studies and high quality (low risk of bias) RCTs, which were considered to provide “gold standard” estimates.\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e: We included 6 RCTs and 87 nonrandomized studies. Surgical risk scores were similar for comparison groups in RCTs, but were higher for patients having transcatheter aortic valve implantation in nonrandomized studies. Nonrandomized studies underestimated the benefit of transcatheter aortic valve implantation compared with RCTs. For example, nonrandomized studies without adjustment estimated a higher risk of postoperative mortality for transcatheter aortic valve implantation compared with surgical aortic valve replacement (OR 1.43 [95% CI, 1.26 to 1.62]) than high quality RCTs (OR 0.78 [95% CI, 0.54 to 1.11). Nonrandomized studies using propensity score matching (OR 1.13 [95% CI, 0.85 to 1.52]) and regression modelling (OR 0.68 [95% CI, 0.57 to 0.81]) to adjust results estimated treatment effects closer to high quality RCTs. Nonrandomized studies describing losses to follow-up estimated treatment effects that were significantly closer to high quality RCT than nonrandomized studies that did not.\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eConclusion\u003c/strong\u003e: Studies with different attributes produce different estimates of treatment effects. Study design attributes related to the completeness of follow-up may explain biased treatment estimates in nonrandomized studies, as in the case of aortic valve replacement where high-risk patients were preferentially selected for the newer (transcatheter) procedure.\u003c/p\u003e","manuscriptTitle":"Bias Estimation In Study Design: A Meta-Epidemiological Analysis of Transcatheter Versus Surgical Aortic Valve Replacement","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2020-09-24 14:53:39","doi":"10.21203/rs.3.rs-71534/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Minor revision","date":"2020-12-24T00:00:00+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2020-10-17T12:00:00+00:00","index":1,"fulltext":"Recommendation: Accept after minor essential revisions\nForm responses:\n---\n\nComments to Author:\n---\nThanks for inviting me to review this paper. I read the manuscript by Saerom Youn\net al. with much interest. The Authors showed a meta-epidemiological analysis of transcatheter\nversus surgical aortic valve replacement with an analysis of the bias between RCTs and\nnonrandomized study. this paper is very interesting and the statistical analysis is well done.\nI would recommend adding a summary flow chart of the research methodology you performed.\nAdd to the discussion section, references to other studies with similar statistics and objectives in the literature.\n* Publons Reviewer Recognition. Springer Nature can send verification of this review directly to Publons (a subsidiary of Clarivate Analytics). If you would like to take advantage of this service, please click on the “Yes” option below. Your name, email address, title of the reviewed manuscript, name of the journal, and date of your review submission (the “Review Data”) will then be transmitted to Publons upon publication of the manuscript. If you have already registered at Publons, they will notify you of the receipt of this review and update your profile as per your settings and their policy. If you are not registered with Publons, you will receive an email from them asking you to register in order for them to be able to recognize your review on your new profile page. Publons may use the Review Data to generate derivative metadata for the benefit of Publons and you as a reviewer, carefully considering the sensitivity of such information. For example, Publons may verify your record as a reviewer by updating your profile published on its webservice if you have registered for such service or help editors to identify candidate reviewers. Please find the details of processing in Publons’ privacy policy https://publons.com/about/terms: **No**\n* Declaration of competing interests: **I declare that I have no competing interests**\n* Reviewer Publication Consent. I agree for my report to be made available under an Open Access Creative Commons CC-BY License (http://creativecommons.org/licenses/by/4.0) if this manuscript is accepted for publication. Any comments that I do not wish to be included in the published report have been included as confidential comments to the editor, which will not be published.: **I agree to the terms of the CC-BY 4.0 license; please do not publish my name with my report. (default)**\n* Is the study design appropriate to answer the research question (including the use of appropriate controls), and are the conclusions supported by the evidence presented?: **Yes**\n* Are the methods sufficiently described to allow the study to be repeated?: **Yes**\n* Is the use of statistics and treatment of uncertainties appropriate?: **Yes**\n* Is the presentation of the work clear?: **Yes**\n* Are the images in this manuscript (including electrophoretic gels and blots) free from apparent manipulation?: **Yes**\n"},{"type":"reviewerAgreed","content":"","date":"2020-10-09T12:00:00+00:00","index":1,"fulltext":""},{"type":"reviewersInvited","content":"","date":"2020-10-08T12:00:00+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2020-10-02T12:00:00+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2020-09-22T12:00:00+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2020-09-18T12:00:00+00:00","index":"","fulltext":""},{"type":"submitted","content":"","date":"2020-09-02T12:00:00+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-surgery","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"bsur","sideBox":"Learn more about [BMC Surgery](http://bmcsurg.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/bsur/default.aspx","title":"BMC Surgery","twitterHandle":"@BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"b2f43310-1a6c-47b1-a715-6ee0211118f6","owner":[],"postedDate":"September 24th, 2020","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"in-revision","subjectAreas":[{"id":600795,"name":"Surgery"},{"id":600796,"name":"General Surgery"}],"tags":[],"updatedAt":"2020-12-25T16:12:34+00:00","versionOfRecord":[],"versionCreatedAt":"2020-09-24 14:53:39","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-71534","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-71534","identity":"rs-71534","version":["v1"]},"buildId":"_2-kVJe1T_tPrBINL-cwx","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}