Confidence in Comparing Two Models with F1 Measure Based on Block-regularized m x 2 Cross Validation

Abstract In the task of comparing two models, evaluating the confidence of a model outperforming another one in terms of F1 measure is a challenging problem from a probability perspective. Existing works used the distributions of the averaged cross-validated estimators of F1 measure over confusion matrices to induce different estimators of the confidence. However, such estimators of the confidence frequently possess inherent biases because of the deviation of the distributions of the averaged cross-validated estimators. To overcome the weakness, in this study, a voted confusion matrix is obtained through aggregating m models in a block-regularized m x 2 cross validation with a majority-vote technique. Over the voted confusion matrix, an accurate distribution of the estimator of F1 measure and a novel estimator of the confidence are developed. Several theoretical results are obtained. (a) The confidence is controlled by the signal-to-noise ratio (SNR) of the estimator of F1 measure; (b) a higher value of the SNR probably indicates a better confidence; and (c) the SNR of the voted estimator increases with an increasing m and reaches an upper bound when m tends to infinity. Finally, extensive simulated and real-world experiments are conducted to verify the validity of the proposed method.