Challenges in estimating the impact of vaccination with sparse data

ABSTRACT Background The synthetic control (SC) model is a powerful tool to quantify the population-level impact of vaccines, because it can adjust for trends unrelated to vaccination using a composite of control diseases. Because vaccine impact studies are often conducted using smaller subnational datasets, we evaluated the performance of SC models with sparse time series data. To obtain more robust estimates of vaccine effects from noisy time series, we proposed a possible alternative approach, “STL+PCA” method (seasonal-trend decomposition plus principal component analysis), which first extracts smoothed trends from the control time series and uses them to adjust the outcome. Methods Using both the SC and STL+PCA models, we estimated the impact of 10-valent pneumococcal conjugate vaccine (PCV10) on pneumonia hospitalizations among cases <12 months and 80+ years of age during 2004-2014 at the subnational level in Brazil. The performance of these models was also compared using simulation analyses. Results The SC model was able to adjust for trends unrelated to PCV10 in larger states but not in smaller states. The simulation analysis confirmed that the SC model failed to select an appropriate set of control diseases when the time series were sparse and noisy, thereby generating biased estimates of the impact of vaccination when secular trends were present. The STL+PCA approach decreased bias in the estimates for smaller populations. Conclusions Estimates from the SC model might be biased when data are sparse. The STL+PCA model provides more accurate evaluations of vaccine impact in smaller populations.