The fractions skill score for ensemble forecast verification

The Fractions Skill Score (FSS) is a neighbourhood verification method originally designed to verify deterministic forecasts of binary events. Previous studies employed different approaches for computing an ensemble-based FSS for probabilistic forecast verification. We show that the formulation of an ensemble-based FSS substantially affects verification results. Comparing four possible approaches, we determine how different ensemble-based FSS variants depend on ensemble size, neighbourhood size, and forecast event frequency of occurrence. We demonstrate that only one ensemble-based FSS, which we call the probabilistic FSS (pFSS), is well-behaved and reasonably dependent on ensemble size. Furthermore, we derive a relationship that allows one to predict how the pFSS behaves with ensemble size. The proposed relationship is very similar to a known result for the Brier Skill Score. Our study uses high-resolution 1000-member ensemble precipitation forecasts from a high-impact weather period. The large ensemble enables us to study the influence of ensemble and neighbourhood size on forecast skill by deriving probabilistic skilful spatial scales.