Cross-modal Graph Contrastive Learning with Cellular Images

Abstract Constructing discriminative representations of molecules lies at the core of a number of domains such as drug discovery, chemistry and medicine. State-of-the-art methods employ graph neural networks (GNNs) and self-supervised learning (SSL) to learn the structural representations from unlabeled data, which can then be fine-tuned for downstream tasks. Albeit powerful, these methods that are pre-trained solely on molecular structures often struggle with the tasks involved in intricate biological processes. To cope with this challenge, we propose using high-content cell microscopy images to assist in learning molecular representation. The fundamental rationale of our method is to leverage the correspondence between molecular topological structures and the caused perturbations at the phenotypic level. By incorporating cross-modal pre-training with multiple types of contrastive loss functions within a unified framework, our model can efficiently learn generic and informative representations from cellular images, which are complementary to molecular structures. Extensive experiments demonstrated that the proposed model transfers non-trivially to a range of downstream tasks and is often competitive with the existing SSL baselines, e.g., a 15.4\% absolute Hit@10 gains in graph-image retrieval task and a 4.0% absolute AUC improvement in clinical outcome predictions. Further zero-shot experiments with cDNA-induced images suggest that our approach has significant potential in bridging small molecule therapeutic regimens with other therapeutic modalities through cellular embeddings