Weak supervision of H&E slides reveals systems-level biology and functional states that govern therapeutic resistance

ABSTRACT Precision oncology lacks scalable tools to assess, at the patient level, systems-level tumor microenvironment (TME) programs driving therapeutic resistance. To address this gap, we trained a weakly-supervised deep learning model, using routine H&E slides as input, to derive quantitative activity for therapeutically-relevant TME phenotypes, spanning immune, metabolic, and tumor cell-intrinsic programs. Using 3111 breast cancer H&E WSIs with matched bulk transcriptomics, our model accurately infers these biological states (AUROC>0.80; PCC>0.64). Validation spanned three levels: (i) tissue-matched multiplexed immunofluorescence, showing concordance between inferred functional states and immune cell fractions (p=0.006-0.106), (ii) blinded reader assessments, confirming localization of phenotype-specific morphology (p<3×10 -5 ), and (iii) multi-institutional patient cohorts, where model-derived phenotypes stratified for clinical response (p<0.045). Despite relying on slide-level labels for training, our model’s attention mechanism identifies focal regions of tumor tissue that drive the overarching clinical phenotype or treatment response. By extracting spatially resolved TME biology from routine histology, this strategy can be applied to massive legacy biobanks to enable discovery of new morpho-molecular mediators of resistance across real-world patient populations. STATEMENT OF SIGNIFICANCE Multi-Omics is too resource-intensive for everyday clinical use. Using slide-level labels, weakly supervised deep learning infers quantitative and spatially resolved TME phenotypes directly from H&E slides. By highlighting high-attention regions of tumor tissue that drive therapeutic efficacy, this strategy can serve as a discovery engine to identify morpho-molecular mediators of resistance.