From metric to action: An evaluation framework to translate infectious disease forecasts into policy decisions

Abstract Decisions for infectious disease outbreaks are difficult and consequential, required to be made in the face of considerable time and societal pressure and great uncertainty. Public health decisions can be supported by probabilistic forecasts – predictions of the future value of an epidemiological quantity, together with uncertainty. Forecasting is challenged by noisy, incomplete, and delayed data alongside non-linear and changing dynamics. Evaluation metrics for infectious disease forecasts often focus on the forecaster’s perspective; improving calibration and sharpness of forecasts. Currently, there are no systematic evaluation protocols to explicitly measure a forecast’s “value” – its ability to provide actionable insights for decision-makers. We here develop a systematic forecast evaluation framework for informing epidemic decision-making, focusing on three aspects: i) translating forecasts and popular evaluation metrics into public-health-relevant quantities; ii) defining evaluation metrics for decision-makers; iii) linking predictability of an epidemic to the value of forecasts for decision-making. By melding concepts from weather forecasting, information theory, and decision theory, our framework bridges conceptual gaps between forecasters and decision-makers. We illustrate the framework with an application to forecasts of weekly incident COVID-19 cases and find that ensemble models often provided the most value for decision-makers with varying levels of risk appetite. Focusing forecast evaluations on the decision-maker provides a new perspective for infectious disease modellers with the hope for improved public health decision-making in future outbreaks and epidemics. Significance statement Infectious disease forecasts are often used to support public health decisions, yet forecasts are typically evaluated using statistical measures that do not necessarily reflect their practical value; to inform decisions. We develop a new framework that connects forecast evaluation to decision-making by incorporating risk preferences and policy-relevant outcomes in evaluations. Drawing on concepts from decision theory, information theory, and weather forecasting, we introduce metrics that prioritise the decision-maker in evaluating forecasting models. Applying our framework to COVID-19 forecasts, we demonstrate that models performing best on average may not provide the greatest value for specific decisions or risk preferences. This work provides a foundation for aligning infectious disease forecasting with real-world decision-making and improving the use of forecasting in public health. Competing Interest Statement The authors have declared no competing interest. Funding Statement C.M. is supported by a studentship from the UK's Engineering and Physical Sciences Research Council. M.U.G.K. and C.A.D. acknowledge funding from the Oxford Martin School Programme in Digital Pandemic Preparedness. Grants from the National Secretary of Science FID-2024-146 and FID-2024-147. M.U.G.K. acknowledges funding from The Rockefeller Foundation (PC-2022-POP-005), Google.org, the Oxford Martin School Programme in Pandemic Genomics, European Union's Horizon Europe programme projects MOOD (No. 874850) and E4Warning (No. 101086640), Wellcome Trust grants 303666/Z/23/Z, 226052/Z/22/Z & 228186/Z/23/Z, the United Kingdom Research and Innovation (No. APP8583), the Medical Research Foundation (MRF-RG-ICCH-2022-100069), UK International Development (301542-403), the Bill \& Melinda Gates Foundation (INV-063472) and Novo Nordisk Foundation (NNF24OC0094346). C.A.D. is supported by the UK National Institute for Health Research Health Protection Research Unit (NIHR HPRU) in Emerging and Zoonotic Infections in partnership with Public Health England (PHE), Grant Number: HPRU200907. The funders had no role in the study design, analysis, or interpretation of results. Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Footnotes We have updated the manuscript in a condensed form for publication. Data Availability All code and data used in our analysis are available at https://github.com/cathalmills/forecast_evaluation.