“E Pluribus Unum”: Prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology (C3RO) Crowdsourced Initiative for Multi-Observer Segmentation

OBJECTIVE Contouring Collaborative for Consensus in Radiation Oncology (C3RO) is a crowdsourced challenge engaging radiation oncologists across various expertise levels in segmentation. A challenge in artificial intelligence (AI) development is the paucity of multi-expert datasets; consequently, we sought to characterize whether aggregate segmentations generated from multiple non-experts could meet or exceed recognized expert agreement. MATERIALS AND METHODS Participants who contoured ≥1 region of interest (ROI) for the breast, sarcoma, head and neck (H&N), gynecologic (GYN), or gastrointestinal (GI) challenge were identified as a non-expert or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. STAPLE non-expert ROIs were evaluated against STAPLE expert contours using Dice Similarity Coefficient (DSC). The expert interobserver DSC (IODSC expert ) was calculated as an acceptability threshold between STAPLE non-expert and STAPLE expert . To determine the number of non-experts required to match the IODSC expert for each ROI, a single consensus contour was generated using variable numbers of non-experts and then compared to the IODSC expert . RESULTS For all cases, the DSC for STAPLE non-expert versus STAPLE expert were higher than comparator expert IODSC expert for most ROIs. The minimum number of non-expert segmentations needed for a consensus ROI to achieve IODSC expert acceptability criteria ranged between 2-4 for breast, 3-5 for sarcoma, 3-5 for H&N, 3-5 for GYN ROIs, and 3 for GI ROIs. DISCUSSION AND CONCLUSION Multiple non-expert-generated consensus ROIs met or exceeded expert-derived acceptability thresholds. 5 non-experts could potentially generate consensus segmentations for most ROIs with performance approximating experts, suggesting non-expert segmentations as feasible cost-effective AI inputs.