Nuclei-Net: A multi-stage fusion model for nuclei segmentation in microscopy images

Abstract In this study, we proposed Nuclei-Net: A multi-stage fusion model for segmenting nuclei in microscopy images. Our proposed model works in two stages. In the first stage, we incorporated a deep learning (DL) based model namely Mask Region-Based-CNN (Mask-RCNN) for the coarse segmentation of the nuclei. However, it can be observed that the predicted mask of Mask-RCNN is unable to segment some complex overlapping nuclei boundaries. So, in the second stage of Nuclei-Net, we detect the complex overlapping nuclei and pass only those overlapping nuclei for fine segmentation. We proposed a marker generation method for each nucleus in a clump. These markers serve as seed points for the marker-controlled watershed algorithm. This algorithm fine segments those overlapping nuclei which were failed by Mask R-CNN. Finally, we fuse the results from the two stages to develop the final segmentation map. Our proposed Nuclei-Net is trained and tested on Kaggle 2018 Data Science Bowl dataset and compared with some standard DL models. Our proposed model achieved a DSC of 83.3%, AJI of 70.03%, Pre of 89.80%, and Rec of 87.20% respectively