Achieving high-resolution single-cell segmentation in convoluted cancer spheroids via Bayesian optimization and deep-learning

ABSTRACT Understanding cellular function within 3D multicellular spheroids is essential for advancing cancer research, particularly in studying cell-stromal interactions as potential targets for novel drug therapies. However, accurate single-cell segmentation in 3D cultures is challenging due to dense cell clustering and the impracticality of manual annotations. We present a high-throughput (HT) 3D single-cell analysis pipeline that combines optimized wet-lab conditions, deep learning (DL)-based segmentation models, and Bayesian optimization to address these challenges. By using live-cell nuclear and cytoplasmic dyes, we achieved clear and uniform staining of cell populations in renal cancer and immune T-cell monocultures and cocultures, improving single-cell detection in spheroids. Our pipeline integrates image preprocessing and DL models based on 3DUnet architecture, enabling robust segmentation of densely packed 3D structures. Bayesian optimization, guided by a custom objective function, was employed to refine segmentation parameters and improve quality based on biologically relevant criteria. The pipeline successfully segments cells under various drug treatments, revealing drug-induced changes not detectable by bulk conventional assays. This approach has potential for application to more complex biological samples, including, organoid co-cultures, diverse drug treatments, and integration with additional immunostaining assays, paving the way for detailed HT analyses of single-cell responses.