Brain Tumor Classification from MRI using Vision Transformers Ensembling

Abstract Automated classification of brain tumors plays an important role in supporting radiologists in decision making. Recently, Vision Transformer (ViT) based deep neural network architectures have gained attention in the computer vision research domain owing to the tremendous success of transformer models in natural language processing. However, studies involving vision transformers for various tasks in the medical imaging domain, including in the field of neuroimaging, are still growing. Many methods have been developed for the classification of brain tumors using traditional machine learning and deep learning methods. In particular, there are several convolutional neural network based transfer learning approaches for achieving good tumor classification accuracy. In this study, pretrained and finetuned ViT models on the ImageNet were adopted for the classification task. A brain tumor dataset from figshare consisting of 3064 T1-weighted contrast-enhanced (CE) magnetic resonance imaging (MRI) slices with meningioma, glioma, and pituitary tumor was used for cross-validation and testing of ensembled ViT models ability for 3-class classification task. The ensemble of all four ViT models B/16, B/32, L/16, and L/32, has demonstrated an overall testing accuracy of 98.7% at 384 × 384 resolution. Therefore, an ensemble of ViT models could be deployed for the computer-aided diagnosis of brain tumors based on T1w CE MRI leading to radiologist relief.