AUCReshaping: Improved sensitivity at high-specificity

Abstract The performance of deep-learning (DL) systems are generally measured using the Area under the Receiver-Operating-Curve (AU-ROC); however, the holistic nature of this metric fails to account for the performance, at a specific range(s) of sensitivity and specificity, at which system is intended to operate. Therefore, it is seen that two systems with the same AU-ROC perform very differently in the field. The issue is especially exaggerated for anomaly detection tasks: a very popular application of DL systems in various areas of research spanning medical imaging, industrial automation, manufacturing, cyber security, fraud detection, drug research, etc. This is primarily due to the fact that the datasets used to train these systems are heavily imbalanced, and the abnormality class usually has a highly skewed misclassification cost compared to the normal class. Traditionally, DL systems account for this by weighting the cost function or designing for different operating points on the ROC curve. This approach achieves reasonable results in most instances but, the system does not actively attempt to maximize performance for the chosen operating point. In this paper, we propose a novel function called AUCReshaping which reshapes the ROC curve only in the desired region by optimizing the sensitivity at a desired specificity. The reshaping is implemented through an adaptive and iterative boosting manner that allows the network to focus on relevant samples during learning. We mainly studied the impact of AUCReshaping on an abnormality detection Chest X-Ray (CXR) task, followed by a breast mammogram and credit card fraud detection task. The results demonstrate an increase of 2-40% in sensitivity at high-specificity, for a binary classification task.