Image embeddings extracted from CNNs outperform other transfer learning approaches in chest radiographs’ classification

Abstract Purpose To identify the best transfer learning approach in the identification of the most frequent abnormalities on chest radiograph (CXR) using embeddings extracted from pre-trained convolutional neural networks (CNNs). Explainable AI (XAI) model was applied to interpret black-box models’ predictions and assess its performance. Methods and Materials: Seven CNNs were trained on CheXpert. Three different transfer learning approaches were thereafter applied to a local dataset. Ensembling of the classification results was performed using simple and entropy-weighted averaging. Model performance was assessed by using the area under the curve (AUC). We applied Grad-CAM (an XAI model) to produce a saliency map, highlighting areas of the image relevant to the classification of the output class, both on single images and on a 200 CXRs sample. Grad-CAM maps were compared to regions of interest extracted manually. The training time was recorded. Results The best transfer learning model was the one that used image embeddings and Random Forest with simple averaging, whit an average AUC of 0.856. Grad-CAM maps showed that the models were focusing on specific features of each CXR. The training time was less than five minutes. Conclusions CNNs pre-trained on a large public dataset of medical images can be exploited as feature extractors for a task of interest. The extracted image embeddings contain relevant information to train an additional classifier with good performance on an independent dataset demonstrating to be the optimal transfer learning strategy, overcoming the need for large private datasets, high computational resources, and long training times.