Deep learning can predict prognosis and endocrine therapy response in breast cancer patients from H&E staining based on estrogen receptor signaling activity

Estrogen receptor (ER) expression status has long been a main factor for endocrine therapy. Deep learning methods can predict ER expression status by hematoxylin-and-eosin (H&E) staining. Since ER signaling activity has been found to be prognostic and is related to endocrine therapy responsiveness, we determined whether deep learning methods and whole-slide H&E-stained images could be used to predict ER signaling activity to determine prognosis in patients with breast cancer. ER signaling activity was determined using the Hallmark Estrogen Response Early gene set from the Molecular Signature Database (MSigDB). The data were fed into ResNet50 with three additional fully connected layers to predict the ER signaling activity of the samples, with ER signaling activity higher than the quantile 0.5. The trained model predicted that ER+/HER2- breast cancer patients with higher ER signaling activity had longer disease-free survival ( p  = 0.00415) and disease-specific survival durations ( p  = 0.00887). In conclusion, a convolutional deep neural network can predict prognosis and endocrine therapy response in breast cancer patients based on ER signaling activity using whole-slide H&E-stained images of tumors.