Effects of allocation concealment and blinding in trials addressing treatments for COVID-19: A methods study

Objective Assess the impact of allocation concealment and blinding on the results of trials addressing COVID-19 therapeutics. Data sources World Health Organization (WHO) COVID-19 database and the Living Overview of the Evidence (L-OVE) COVID-19 platform by the Epistemonikos Foundation (up to February 4 th 2022) Methods We included trials that compared drug treatments, antiviral antibodies and cellular therapies with placebo or standard care. For the five most commonly reported outcomes, if sufficient data were available, we performed random-effects meta-regression comparing the results of trials with and without allocation concealment and trials in which both healthcare providers and patients were blinded with trials in which healthcare providers and/or patients were aware of the intervention. A ratio of odds ratios (ROR) > 1 or a difference in mean difference (DMD) > 0 indicates that trials without allocation concealment or open-label trials produced larger effects than trials with allocation concealment or blinded trials. Results As of February 4 th 2022, we have identified 488 trials addressing COVID-19 drug treatments and antiviral antibodies and cellular therapies. Of these, 436 trials reported on one or more of our outcomes of interest and were included in our analyses. We found that trials without allocation concealment probably overestimate mortality (ROR 1.14 [95% CI 0.92 to 1.41]), need for mechanical ventilation (ROR 1.26 [95% CI 0.97 to 1.64]), admission to hospital (ROR 1.93 [95% CI 0.83 to 4.48]), duration of hospitalization (DMD 1.94 [95% CI 0.86 to 3.02]), and duration of mechanical ventilation (DMD 2.64 [95% CI −0.90 to 6.18]), but results were imprecise. We did not find compelling evidence that double-blind and open-label trials produce consistently different results for mortality (ROR 1.00 [95% CI 0.87 to 1.15]), need for mechanical ventilation (ROR 1.03 [95% CI 0.84 to 1.26]), and duration of hospitalization (DMD 0.47 days [95% CI −0.38 to 1.32]). We found that open-label trials may overestimate the beneficial effects of interventions for hospitalizations (ROR 1.87 [95% CI 0.95 to 3.67] and duration of mechanical ventilation (DMD 1.02 days [95% CI −1.30 to 3.35]), but results were imprecise. Conclusion We found compelling evidence that, compared to trials with allocation concealment, trials without allocation concealment may overestimate the beneficial effects of treatments. We did not find evidence that trials without blinding addressing COVID-19 interventions produce consistently different results from trials with blinding. Our results suggest that consideration of blinding status may not be sufficient to judge risk of bias due to imbalances in co-interventions. Evidence users may consider evidence of differences in co-interventions between trial arms when judging the trustworthiness of open-label trials. We suggest, however, evidence users to remain skeptical of trials without allocation concealment. What’s new? key findings Trials without blinding did not produce consistently different results from trials with blinding. Additional information Previous studies have had conflicting results with regards to the effects of blinding on trial results. Our study supports the assertion that results from blinded trials may not differ significantly from unblinded ones. Implications Our study suggest that risk of bias assessment of blinding needs to be more nuanced and that lack of blinding may not be a definite indication of risk of bias.