Independent assessment of a deep learning system for lymph node metastasis detection on the Augmented Reality Microscope

Several machine learning algorithms have demonstrated high predictive capability in the identification of cancer within digitized pathology slides. The Augmented Reality Microscope (ARM) has allowed these algorithms to be seamlessly integrated within the current pathology workflow by overlaying their inferences onto its microscopic field of view in real time. In this paper, we present an independent assessment of the LYmph Node Assistant (LYNA) models, state-of-the-art algorithms for the identification of breast cancer metastases in lymph node biopsies which have been optimized for usage at different ARM magnifications. We assessed the models on a set of 40 whole slide images at the commonly used objective magnifications of 10x, 20x, and 40x. We analyzed the performance of the models across clinically relevant subclasses of tissue, including metastatic breast cancer, lymphocytes, histiocytes, veins, and fat. We also analyzed the models’ performance on potential types of contaminant tissue such as endometrial carcinoma and papillary thyroid cancer. Each model obtained overall AUC values of approximately 0.98, accuracy values of approximately 0.94, and sensitivity values above 0.88 at classifying small regions of a field of view as benign or cancerous. Across tissue subclasses, the models performed most accurately on fat and blood, and least accurately on histiocytes, germinal centers, and sinus. The models also struggled with the identification of isolated tumor cells, especially at lower magnifications. After testing, we manually reviewed the discrepancies between model predictions and ground truth in order to understand the causes of error. We introduce a distinction between proper and improper ground truth to allow for analysis in cases of uncertain annotations or on tasks with low inter-rater reliability. Taken together, these methods comprise a novel approach for exploratory model analysis over complex anatomic pathology data in which precise ground truth is difficult to establish.