SHIFT: speedy histological-to-immunofluorescent translation of whole slide images enabled by deep learning

ABSTRACT Spatially-resolved molecular profiling by immunostaining tissue sections is a key feature in cancer diagnosis, subtyping, and treatment, where it complements routine histopathological evaluation by clarifying tumor phenotypes. In this work, we present a deep learning-based method called speedy histological-to-immunofluorescent translation (SHIFT) which takes histologic images of hematoxylin and eosin-stained tissue as input, then in near-real time returns inferred virtual immunofluorescence (IF) images that accurately depict the underlying distribution of phenotypes without requiring immunostaining of the tissue being tested. We show that deep learning-extracted feature representations of histological images can guide representative sample selection, which improves SHIFT generalizability. SHIFT could serve as an efficient preliminary, auxiliary, or substitute for IF by delivering multiplexed virtual IF images for a fraction of the cost and in a fraction of the time required by nascent multiplexed imaging technologies. KEY POINTS Spatially-resolved molecular profiling is an essential complement to histopathological evaluation of cancer tissues. Information obtained by immunofluorescence imaging is encoded by features in histological images. SHIFT leverages previously unappreciated features in histological images to facilitate virtual immunofluorescence staining. Feature representations of images guide sample selection, improving model generalizability.