Retrospective analysis of Covid-19 hospitalization modelling scenarios which guided policy response in France

A bstract During the COVID-19 pandemic, epidemiological modelling has played a key role in public debate and policy making for anticipating the epidemic trajectory, as well as proposing and evaluating non-pharmaceuticals interventions. Despite its importance, evaluations of models’ ability to accurately represent the evolution of the disease remain scarce. Robust and systematic evaluation is needed to assess models. We investigate the following research question : were the COVID-19 scenarios proposed by modellers during the pandemic to policy-makers relevant for decision making ? To answer this, we conduct a retrospective assessment of modelling reports which guided policy response in France in 2020-2022. After systematically verifying the scenarios hypotheses (e.g., exclusion of no-lockdown scenarios when a lockdown was effectively in place), we find that out of 10 reports, reality was below the best-case scenario in 6 reports; within the best-case / worst case scenarios range in 3 reports; above the worst-case scenario in 1 report. Best-case scenarios were the closest to reality, but often came from report with a large span between best-case and worst-case scenarios beyond 2 weeks, precluding certainty about future outcomes at the time of publishing. Our results hint a systematic overestimation bias for these particular models used to anticipate epidemic evolution, which can be of importance if such models are used to contractually estimate the effectiveness of non pharmaceutical interventions. To our knowledge, this is the only national systematic retrospective assessment of COVID-19 pandemic scenarios assessing hospital burden; such an approach should be reproduced in other countries whenever possible. Graphical Abstract Reality (black line) compared to prospective scenarios (colored lines) which informed policy during the COVID-19 pandemic in France for Intensive Care Units (top) and New Hospital Admissions (bottom). Colors indicate the error between reality and scenarios, expressed as a percentage of the 1st wave peak (horizontal dashed line).