Human-AI Collaborative Multi-modal Multi-rater Learning for Endometriosis Diagnosis

[47,48]. 2.3. Multi-rater Learning Multi-rater learning is a technique designed to train a classifier using noisy labels gathered from multiple annotators. The challenge lies in how to derive a “clean” label from these imperfect labels. Traditional approaches often rely on majority voting [49] and the expectation-maximization (EM) algorithm [50, 51]. Rodrigues et al. [52] introduce an end-to-end deep neural network (DNN) that incorporates a crowd layer to model the annotator-specific transition matrix, enabling the direct training of a DNN with crowdsourced labels. Alternatively, Chen et al. [53] suggest a probabilistic model that learns an interpretable transition matrix unique to each annotator. Meanwhile, Guan et al. [54] employ multiple output layers in the classifier and learn combination weights to aggregate the results. More recently, CROWDLAB [55] has set the state of the art in multi-rater learning by using multiple noisy-label samples and predictions with a model trained via label noise learning. Despite the promise of multi-rater learning in leveraging multiple noisy labels per training sample, it falls short by overlooking the Human-AI Collaborative Multi-modal Multi-rater Learning 5 concept of human-AI collaboration and multi-modal learning. 2.4. Imaging-based Endometriosis Detection One crucial indicator to detect endometriosis is the obliteration of the Pouch of Douglas (POD) [7,8]. However, the development of an AI model that can classify such indicator hinges on the availability of precise POD obliteration annotations from T1/T2 MRIs, a task that is challenging because even experienced clinicians often face uncertainty in identifying this sign. Despite these challenges, there have been some efforts to train multi-modal MRI AI models for POD obliteration classification. For example, Zhang et al. [11] proposed a method to transfer knowledge from ultrasound to MRI for classifying POD obliteration, and Butler et al. [8] explored self-supervised pre-training for multi- modal POD obliteration classification. However, these methods have not been validated against ground-truth labels obtained from surgical reports, making it difficult to assess their clinical validity. Nevertheless, for all of the aforementioned related work, none of the methods deal with human-AI collaboration, multi-modal classification, and multi-rater learning simultaneously, particularly for classifying endometriosis. In this paper, we propose the HAICOMM model to address this research gap. 3. Methodology The training of our HAICOMM methodology is depicted in Fig. 2. The first stage consists of pre-training a multi-modal encoder using a large unlabelled T1/T2 MRI dataset, with a self-supervised learning mechanism [56] (see frame (a) in Fig. 2). Subsequently, for training the proposed human-AI classifier HAICOMM, we first need to estimate the pseudo ground truth label from the the multiple “noisy” labels available for each pair of T1/T2 MRI training images. We rely on CrowdLab [12] to produce such pseudo ground truth labels (see frame (b) in Fig. 2). Next, the T1/T2 MRI images with multi-rater (manual) labels are fed into their multi-modal encoders. The embeddings from the multi-modal and label encoders are combined to produce the final prediction that is trained to match the pseudo ground truth label (see frame (c) in Fig. 2). We provide details about each of these training stages below. 3.1. Multi-modal Encoder Pre-training The MRI encoder of the HAICOMM model is pre-trained with the Masked Autoencoder (MAE) self-supervised learning method [56]. For this pre-training, we use a dataset denoted as DP = {x′(i) t1 }Mt1 i=1 S{x′(i) t2 }Mt2 i=1 , with x′(i) t1 , x′(i) t2 ∈ X ⊂ RH×W ×D denoting the T1 and T2 MRI volumes of size H × W × D. It is worth noting that the number of unlabeled images, M = Mt1 + Mt2, far exceeds the number of labeled images, denoted as N (i.e., M >> N ), of the datasets that will be defined in Sections 3.2 and 3.3. Human-AI Collaborative Multi-modal Multi-rater Learning 6 P(i) from Eq.6 Pseudo Label MAE Encoder FT1 from Eq.5 MAE DecoderOriginal Image Masked Image Reconstruction (a) HAICOMM Pretraining from Eq.1 ... Multi-rater Labels (c) Multi-modal Human-AI Collaborative Classification Concat Fts Fr from Eq.5 FC layers Loss Minimization from Eq.7 ... (b) Multi-rater Learning Majority Voting Majority Vote Classifier CrowdLab Pseudo Labels T1 T2 Voted labels Multi-rater Labels ⊕ ... with dataset DP DMV from Eq.2 D’T from Eq.3 FT2 from Eq.5 Figure 2: The framework of HAICOMM. The MRI encoders of HAICOMM are: (a) firstly pre-trained with a Masked Autoencoder (MAE) model; then (b) the pseudo clean labels are estimated from the multi-rater learning process; next, (c) the T1 and T2 data, along with the human-produced multi-rater labels are entered into respective feature extraction encoders – the features from three sources are fused for the final prediction. In the figure, “FC” means fully-connected, “Fts” represents features, “Concat” denotes concatenation, and “ ⊕” is the concatenation operation. Following the 3D Vision Transformer [57], the architecture of 3D-MAE follows an asymmetric encoder-decoder setup. The encoder, parameterized by ϕ, is represented by gϕ : X → F , which receives visible patches along with positional embeddings that are processed through a 3D Vision Transformer to produce features in the space F. The resulting features are subsequently directed to the decoder, parameterized by ψ and denoted by fψ : F → X , which reconstructs the original volume with the masked volume tokens. In the MRI pre-training, our objective is to minimize the mean squared error (MSE) of the reconstruction of the original masked patches. Formally, we have: ϕ∗, ψ∗ = arg minϕ,ψ 1 M PMt1 i=1 fψ  gϕ  x′(i) t1  −x′(i) t1 2 2 +PMt2 i=1 fψ  gϕ  x′(i) t2  −x′(i) t2 2 2  , (1) where ∥ · ∥ 2 denotes the L2-norm. The optimization in (1) jointly learns the encoder parameters ϕ and the decoder parameters ψ to minimize the reconstruction error of the masked patches across all training samples. For the subsequent training and evaluation of the human-AI collaborative classifier, we adopt the pre-trained feature extractor gϕ∗(.), as explained below in Sec. 3.3. Human-AI Collaborative Multi-modal Multi-rater Learning 7 3.2. Multi-rater Learning The training of our human-AI collaborative classifier requires each pair of T1/T2 MRI training images to have a single pseudo clean label estimated from the multiple “noisy” training labels. The multi-modal multi-rater dataset is denoted by DT = {(x(i) t1 , x(i) t2 , y(i))}N i=1 with N samples, where the multi-rater label has K binary annotations denoted as y(i) ∈ Y ⊂ { 0, 1}K, provided by the K clinicians who annotated the training images in DT . With the multi-rater labels, we first perform majority vote to fetch the most frequently appearing labels per training sample. Let us present the majority vote operation as h : Y → { 0, 1}. Then, we have the mapping from multi-rater labels to the majority label for each multi-modal sample, forming the following majority voting dataset: DM V = {(x(i) t1 , x(i) t2 , ˆy(i))}N i=1, where ˆy(i) = h(y(i)), (2) where ˆy(i) ∈ {0, 1}. With such generated consensus labels from majority vote, we train a classifier fθ : X × X → [0, 1] that takes the T1 and T2 MRI volumes to optimize a standard binary cross-entropy objective function. This trained classifier, together with DT will then be used for the filtering process of multi-rater cleaning technique CrowdLab [12], denoted by z : [0, 1] × Y → { 0, 1}, which generates a pseudo clean label for each sample. Formally, CrowdLab’s pseudo labeling process is defined by: y(i) = z(fθ(x(i) t1 , x(i) t2 ), y(i)), (3) where y(i) ∈ {0, 1} denotes CrowdLab’s estimate for the “clean” label of the i-th sample, which is referred to as the pseudo ground-truth label. 3.3. Multi-modal Human-AI Collaborative Classification Given the pseudo clean labels y(i) from (3), the dataset for the final model training is defined as: D′T = {(x(i) t1 , x(i) t2 , y(i), y(i)}N i=1. (4) The pre-trained encoder from (1), parameterized by ϕ∗, is utilized to initialize the T1 and T2 MRI feature extractors, respectively defined byg ˆϕ : X → F and gϕ′ : X → F , for the classification task. We also need to define a new feature extractor for processing the manual labels with a learnable module defined as gγ : Y → F . Such manual labels are used in the human-AI collaborative module. Hence, these extractors produce the T1, T2 and rater features as: F(i) t1 = g ˆϕ(x(i) t1 ), F(i) t2 = gϕ′(x(i) t2 ), F(i) r = gγ(y(i)). (5) These three feature maps are then concatenated and fed into a learnable linear projection πη : F × F × F → F , parameterized by η to predict: p(i) = σ  πη(F(i) t1 ⊕ F(i) t2 ⊕ F(i) r )  , (6) Human-AI Collaborative Multi-modal Multi-rater Learning 8 where p(i) ∈ [0, 1]2 denotes the probabilistic prediction, ⊕ represents the concatenation operator, and σ is the softmax function. Finally, the training of the whole model is performed by minimizing the binary cross-entropy loss, as follows: ˆϕ∗, ϕ′∗, γ∗, η∗ = arg min ˆϕ,ϕ′,γ,η − 1 N NX i=1 y(i) log(p(i)) + (1 − y(i)) log(1 − p(i)) ! , (7) where the p(i) is the predicted probability of the positive class of i-th sample (i.e., the 2nd dimension p(i) in (6)), and y(i) is the pseudo clean label in D′ T . The goal is to learn the optimal parameters ˆϕ∗, ϕ′∗, γ∗, and η∗ that minimize this loss across all N samples. The testing process consists of taking the input T1/T2 MRI images and labels y from clinicians to output p from (6). 4. Experimental Settings 4.1. Endometriosis Dataset We first introduce our multi-modal multi-rater dataset annotated with imaging and surgery-based POD obliteration labels for the diagnosis of endometriosis. For the pre- training stage, we collected 5,867 unlabeled T1 MRI volumes and 8,984 unlabeled T2 MRI volumes from patients aged between 18 and 45 years old, where the volumes show female pelvis scans obtained from various MRI machines with varying resolutions. The pre-training pelvic MRI scans were obtained from the South Australian Medical Imaging (SAMI) service. For this paper, we included only pelvic MRI scans of female patients acquired using either T1- or T2-weighted imaging sequences. These scans were collected from multiple centers using various equipments, including the 1.5T Siemens Aera, 3T Siemens TrioTim, 1.5T Philips Achieva, 1.5T Philips Ingenia, and 1.5T Philips Intera. The T1 and T2 datasets include a wide variety of sequences, where each scan is acquired using a combination of the following parameters: spin echo sequences (e.g., Turbo Spin Echo, TSE) or gradient echo sequences (e.g., Volumetric Interpolated Breath- hold Examination, VIBE); orientation in either 2D planes (axial, sagittal, coronal) or 3D sequences (e.g., SPC); with or without fat saturation/suppression techniques (e.g., DIXON, SPIR, SPAIR); and with or without gadolinium contrast enhancement. We show the statistics of the size distribution information of the pre-training dataset in Table 1. As described in Section 3.1, this dataset is used as containing 14,851 independent scans for the Masked Autoencoder (MAE) self-supervised learning. To standardize the data, the volumes were resampled to 1mm×1mm×3mm voxels. Also, we apply 3D contrast-limited adaptive histogram equalization (CLAHE) to enhance image local contrast and refine edge definitions. We choose CLAHE, because it produced the best classification results among the different data pre-processing methods tested on an internal validation process (e.g., voxel value truncate.). For the training of the human-AI collaborative POD obliteration classifier, we collected 82 pairs of T1/T2 MRI volumes with patients aged 18 to 45 years old. These Human-AI Collaborative Multi-modal Multi-rater Learning 9 Table 1: Statistics of the volume size distribution of the pre-training dataset. (a) T1 pre-training dataset Dim Mean Std Max Min 0 326.67 93.73 550 66 1 271.97 88.78 530 60 2 79.64 24.5 430 50 (b) T2 pre-training dataset Dim Mean Std Max Min 0 245.78 113.33 530 60 1 208.47 93.87 504 50 2 77.01 23.99 430 50 scans were obtained across multiple clinical sites, with each case annotated by three experienced clinicians who work in clinics specialized in the imaging-based diagnosis of endometriosis. The scans were acquired using a standardized protocol for endometriosis examination and they show a specific region surrounding the uterus, which is the area where the signs of POD obliteration are more visible. MRIs from patients with a history of hysterectomy or those with large pelvic lesions that limited adequate assessment of the POD were excluded. The scanners are of the following models: SIEMENS Aera, SIEMENS Espree, SIEMENS MAGNETOM Sola, and SIEMENS MAGNETOM Vida. The volume types are: T1-weighted MRI images, obtained using a Volume Interpolated Breath-hold Examination technique with Dixon fat-water separation, acquired in the transverse plane; and T2-weighted MRI images, acquired using the SPACE sequence, characterized by non-saturation techniques, taken in the coronal plane, and with isotropic voxel dimensions. All 3D volumes were reoriented to a Right-Anterior-Superior (RAS) coordinate system to standardize anatomical alignment, then resampled with an output spacing. Similar to the pre-training, to standardize the data, the volumes were resampled to 1mm × 1mm × 3mm voxels. We show the statistics of the size distribution information of the POD obliteration dataset in Table 2 Table 2: Statistics of the volume size distribution of the POD obliteration dataset. (a) T1 modality in the dataset Dim Mean Std Max Min Depth 383.91 91.88 576 260 Height 312.10 72.61 464 250 Width 72.16 8.46 80 51 (b) T2 modality in the dataset Dim Mean Std Max Min Depth 389.80 18.40 448 384 Height 192.30 26.22 336 160 Width 289.71 19.37 354 192 This training set with 82 volumes is subdivided into 62 volumes for training and 20 for validation that are used for cross validation for hyper-parameter selection. We further collect 30 cases that contain ground truth annotation of POD obliteration from surgical reports. These cases serve as gold standards for testing. We show in Table 6 the distribution of normal vs abnormal (i.e., POD obliteration) images in the training and testing sets. Note that while in the testing set, we have ground truth from surgery, in the training set, we only have the labelers’ annotation. We also use CLAHE pre-processing in this dataset. Similar to the pre-training, to standardize the data, the volumes were Human-AI Collaborative Multi-modal Multi-rater Learning 10 resampled to 1mm × 1mm × 3mm voxels. Table 3: Number of normal and abnormal (i.e., with POD obliteration) images in the training and testing sets. In the training set, ‘A1’ to ‘A3’ are the three annotators, ‘MV’ represents the annotations from majority voting, and ‘CL’ denotes the annotations from CrowdLab [55]. In the testing set, we have the grount truth (GT) from surgery. POD Training Set Testing Set A1 A2 A3 MV CL GT Normal 62 48 36 51 47 15 Abnormal 20 34 46 31 34 15 4.2. Implementation Details For model pre-training, the input volumes are cropped and possibly zero-padded to achieve the dimension of 64 × 128 × 128 voxels. To maintain consistency with the pre- training dataset, the endometriosis training and testing samples are manually centered at the uterine region which is the most important region for the detection of POD obliteration. Later we center crop each volume to the same dimensions as pre-training data, i.e. 64 × 128 × 128 voxels, for training and testing. In both pre-training and the training of the human-AI collaborative POD obliteration classifier, the multi-modal encoder for each modality is a transformer with 12 blocks. The majority vote classifier has a 3D-ResNet50 as its backbone network [58]. For the human-AI collaborative POD obliteration classifier training, we use 5 epochs for model optimization warming up. AdamW optimizer and base learning rate of 1e-3 with cosine annealing [59] learning rate tuning strategy are adopted. Three multi-rater labels from three different annotators are incorporated into the training process. In the testing phase, the scans are also cropped in the uterine regions and the clinical surgical results serve as the ground truth for accurate evaluation. Using a cross-validation procedure, with a training set containing 62 volumes and a validation set comprising 20 volumes, we observe signs of overfitting after 60 epochs. To prevent this, we implemented early stopping and halted the training at the 60th epoch. Note that the majority voting is only produced for the consensus pseudo label required for the training of the model fθ(.) to be used by CrowdLab, as explained in Eq. 3. Once the pseudo clean labels are generated by CrowdLab, the majority voting will no longer be needed. All experiments in the paper are run on an NVIDIA GeForce RTX 3090. The main Python libraries used in our implementation are: torch 1.7.1+cu110, torchvision 0.8.2+cu110, scikit-image and scipy. Human-AI Collaborative Multi-modal Multi-rater Learning 11 Figure 3: The ROC Curve of HAICOMM models and its counterparts. 4.3. Quantitative Evaluation Settings We compare the performance of our proposed HAICOMM with respect to the following models: 1) purely manual annotation from the three expert clinicians via majority voting; 2) models trained with noisy-label learning techniques (SSR [60] and ProMix [61]) using the noisy labels from one of the annotators (GT1, GT2, GT3); 3) models trained from labels produced by the multi-rater learning CrowdLab [12] (in the table denoted as models w/ CL); and 4) human-AI classifiers using the three annotators (models w/ HAIC). In terms of evaluation metrics, we adopt Accuracy and Area Under the ROC Curve (AUROC). We also show an ablation study that compares the performance of each annotator without using the system, with the performance of different combinations of annotators to be used in the human-AI collaboration, and the performance of the proposed HAICOMM. 5. Results and Discussion 5.1. Model Performance The performance results in Table 4 show that the proposed HAICOMM outperforms other competing models by a large margin across the accuracy and AUC measures. Relative improvements vary from 9.10% to 54.83% on accuracy and 19.37% and 64.75% on AUROC. The standard deviation is calculated by inference time bootstrapping. Also, we provide the the ROC Curve of HAICOMM models and its counterparts SSR and ProMix with CrowdLab labels in Fig. 3. There are interesting points to observe in the results from Table 4. First, the multi-rater learning tends to be more accurate than noisy label learning. The manual Human-AI Collaborative Multi-modal Multi-rater Learning 12 Table 4: Performance comparison of HAICOMM and other models on the test set in terms of accuracy and AUROC (and respective improvements of HAICOMM) with the testing ground-truth labels from surgical reports. The standard deviation is calculated by inference time bootstrapping. “Majority Vote” denotes a purely manual classification using the majority vote of the three annotators. SSR and ProMix w/ “GT1”, “GT2” and “GT3” mean noisy-label learning models trained with labels from annotators #1, #2 and #3, respectively. “CL” denotes noisy-label learning models trained with labels y(i) from Eq. (3) produced by the multi-rater learning method CrowdLab. “HAIC” represents a model that collaborates with the annotators and that is trained with labels from CrowdLab. The best results for each column are in bold.

Models Accuracy Improvement AUROC Improvement Human Majority Vote 0.70±0.00 14.29% - - SSR w/ GT1 0.58±0.02 37.93% 0.58±0.01 53.83% Noisy SSR w/ GT2 0.60±0.04 33.33% 0.58±0.01 51.67% Label SSR w/ GT3 0.57±0.02 41.17% 0.56±0.06 57.07% Learning ProMix w/ GT1 0.53±0.02 50.01% 0.54±0.04 64.75% ProMix w/ GT2 0.62±0.03 29.72% 0.58±0.02 52.63% ProMix w/ GT3 0.52±0.05 54.83% 0.57±0.02 56.54% Multi- SSR w/ CL 0.62±0.02 29.72% 0.59±0.01 50.82% rater ProMix w/ CL 0.65±0.04 23.08% 0.54±0.03 63.32% HAIC SSR w/ HAIC 0.68±0.08 17.08% 0.74±0.01 19.37% ProMix w/ HAIC0.67±0.06 20.00% 0.74±0.04 19.98% Ours HAICOMM 0.80±0.04 - 0.89±0.06 - annotation without any assistance from the model fθ(.) in Eq. 3, shows a relatively low accuracy of 0.7, motivating the importance of the proposed human-AI collaboration. Also, when noisy-label learning models are designed to collaborate with humans, we can see large performance improvements, such as shown by “SSR w/ HAIC” and “ProMix w/ HAIC”. However, the proposed HAICOMM still obtains much higher accuracy and AUROC. Additionally, the proposed HAICOMM shows a much simpler training algorithm than “SSR w/ HAIC” and “ProMix w/ HAIC”. To summarize, the proposed model outperforms not only the ensemble of experts (Majority Vote), but also the top-performing multi-rater learning model (SSR w/CL and ProMix w/ CL), as well as the best noisy-label learning methods (SSR and ProMix), even after adding human AI collaboration (SSR w/ HAIC and ProMix w/ HAIC) by a large margin. 5.2. Human-AI Collaborative Multi-modal Multi-rater Ablation Study The first three rows of Table 5 present the accuracy of each of the three annotators. The next rows show HAICOMM without relying on any human collaboration (w/o HAIC), then the next 6 rows show different combinations of annotators for the human- AI collaboration process. This is followed by two rows showing HAICOMM with single Human-AI Collaborative Multi-modal Multi-rater Learning 13 Table 5: Accuracy and AUROC performance analyses of HAICOMM and its variants on the test set. “R1”, “R2” and “R3” denote models trained with input annotations from annotators #1, #2 and #3, respectively. “HAIC” represents model trained with multi- rater labels inputs for human-AI collaborations. T1 and T2 Only w/ HAIC represent single-modality HAICOMM approaches. The best results for each column are in bold. Models AccuracyAUROC Labels from Rater #1 (R1)0.67 - Labels from Rater #2 (R2)0.73 - Labels from Rater #3 (R3)0.70 - HAICOMM w/o HAIC 0.63 0.59 HAICOMM w/ R1 0.77 0.70 HAICOMM w/ R2 0.80 0.84 HAICOMM w/ R3 0.57 0.60 HAICOMM w/ R1,2 0.77 0.87 HAICOMM w/ R2,3 0.73 0.87 HAICOMM w/ R1,3 0.63 0.67 T1 Only w/ HAIC 0.67 0.81 T2 Only w/ HAIC 0.77 0.88 HAICOMM 0.80 0.89 modality data (either T1 or T2) in the input. Last row shows the HAICOMM results. Note that the collaboration with annotators almost always improve over the result of HAICOMM w/o HAIC, and it also improves the accuracy for most of the annotators (particularly R1 and R2). Interestingly, we found that the model with R2 inputs performs the best among with single rater labels. The model with combination inputs of R1 and R3 performs the worst. This may suggest that R2 provides relatively more accurate labels compared to R1 and R3. This phenomenon resonates with the fact that R2 provides the most accurate labels among three raters (as shown in the first three rows). This table also shows that both single modality results with HAIC (with T2 being much better than T1) have worse results than the multi-modal HAICOMM, which provides evidence of the need for multi-modal analysis in the classification of POD obliteration. 5.3. Analyses We conduct a qualitative analysis of HAICOMM. In Figure 4, (a) and (b) are the input T1 and T2 MRIs, respectively. The table below shows the predictions by the three raters (Rater #1,#2,#3), then the predictions by SSR and ProMix trained with Rater #1’s labels and CROWDLAB’s labels (SSR w GT1, ProMix w GT1, SSR w CL GT, ProMix w CL GT). Next, we show SSR and ProMix trained with CROWDLAB’s labels and relying on human-AI collaborative classification (SSR w HAIC, ProMix w HAIC), followed by the result from our HAICOMM, and the ground truth label from Human-AI Collaborative Multi-modal Multi-rater Learning 14 surgical data. This figure shows that the proposed HAICOMM model can generate correct labels, with other approaches showing incorrect predictions. For (c) and (d), we show that the proposed HAICOMM model produced a correct label while most of other

failed (only ProMix w GT1 and HAICOMM predict the surgical ground truth label correctly). (a) Modality T1 (b) Modality T2 Models Rater #1 Rater #2 Rater #3 SSR w GT1 ProMix w GT1 SSR w CL GT ProMix w CL GT SSR w HAIC ProMix w HAIC HAICO MM Surgic al Preds 0 1 0 0 0 0 0 0 0 1 1 (c) Modality T1 (d) Modality T2 Models Rater #1 Rater #2 Rater #3 SSR w GT1 ProMix w GT1 SSR w CL GT ProMix w CL GT SSR w HAIC ProMix w HAIC HAICO MM Surgic al Preds 1 1 0 0 1 0 0 0 0 1 1 Figure 4: Qualitative analysis of HAICOMM. Each row (a),(b) and (c),(d) presents the input T1 and T2 MRI images, with corresponding tables below showing predictions from three human raters (Rater #1, #2, and #3). The tables also display predictions from SSR and ProMix models trained using labels from Rater #1 (SSR w/ GT1, ProMix w/ GT1) and CROWDLAB (SSR w/ CL GT, ProMix w/ CL GT). Following these, we provide the predictions from SSR and ProMix models trained on CROWDLAB labels and utilizing human-AI collaborative classification (SSR w/ HAIC, ProMix w/ HAIC). Finally, we present results from our HAICOMM model, along with the ground truth label based on surgical data. In terms of the normal and abnormal volumes, they are sampled from distributions with equivalent condition / equipment / resolution, so they have similar properties. We calculate the statistics of the volume size distribution of the normal / abnormal (for the training data, we use majority vote results as the label) ones in Table 7 and Table 8. Moreover, we plot the distribution of manufacturer model of scanners for normal and abnormal in Figure 5 and the distribution of magnetic field strength for normal and Human-AI Collaborative Multi-modal Multi-rater Learning 15 abnormal in Figure 6. Table 6: Number of normal and abnormal (i.e., with POD obliteration) volumes in the training and testing sets. In the training set, ‘A1’ to ‘A3’ are the three annotators, ‘MV’ represents the annotations from majority voting, and ‘CL’ denotes the annotations from CrowdLab [55]. In the testing set, we have the grount truth (GT) from surgery. POD Training Set Testing Set A1 A2 A3 MV CL GT Normal 62 48 36 51 47 15 Abnormal20 34 46 31 34 15 Table 7: Statistics of the volume size distribution of the T1 MRIs of POD obliteration dataset. (a) Normal Statistics Dim Mean Std Max Min Depth 382.73 91.05 576 260 Height 313.67 74.57 464 260 Width 72.44 7.81 80 52 (b) Abnormal Statistics Dim Mean Std Max Min Depth 381.71 90.31 576 320 Height 309.35 71.95 464 250 Width 72.07 8.67 80 51 Table 8: Statistics of the volume size distribution of the T2 MRIs of POD obliteration dataset. (a) Normal Statistics Dim Mean Std Max Min Depth 389.82 18.54 448 384 Height 190.06 17.64 256 176 Width 292.36 13.90 336 288 (b) Abnormal Statistics Dim Mean Std Max Min Depth 389.79 18.40 448 384 Height 193.05 28.50 336 160 Width 288.82 20.83 354 192 6. Conclusion and Future Work In this paper, we proposed the Human-AI Collaborative Multi-modal Multi-rater Learning (HAICOMM) methodology for an imaging-based endometriosis classification. It integrates the capabilities of machine learning models and multiple human labels to enhance the classification accuracy of POD obliteration from T1/T2 MRIs. Evaluation on our endometriosis dataset demonstrates the efficacy of the HAICOMM model, surpassing ensemble clinician predictions, noisy-label learning approaches, and a multi- rater learning method. This underscores the potential of collaborative efforts between AI and human clinicians in diagnosing and managing endometriosis and other complex Human-AI Collaborative Multi-modal Multi-rater Learning 16 (a) (b) Figure 5: Distribution of manufacturer model of scanners for the normal (a) and abnormal (b) datasets. (a) (b) Figure 6: Distribution of magnetic field strength of scanners for the normal (a) and abnormal (b) datasets. medical conditions. To the best of knowledge, we are the first to propose the multi- modal multi-rater classification task. Furthermore, our endometriosis dataset is the first in the field to enable the development of multi-modal multi-rater classifiers. One potential limitation of our method is the dataset size. Currently, we are dedicated to collect more data from different clinical sources to expand the dataset. The use of such multiple clinical sources will require the exploration of domain adaptation techniques to enable a better flexibility of the method to work in multiple domains. Beyond this issue, the need for specific labellers for training and testing is another potential limitation. We plan to address this issue with the development of techniques that work with a variable set of labellers during training and testing. Another interesting direction is the collection of new datasets for other multi-modal multi-rater clinical problems to enable the evaluation of our HAICOMM in a different task. One more potential future direction is the development of a detector of region of interests, which will require dense (i.e., pixel-wise) annotations of the training and testing sets and the collection of significantly larger training sets. Human-AI Collaborative Multi-modal Multi-rater Learning 17

[1] A. S. Lagana, F. M. Salmeri, H. Ban Frangeˇ z, F. Ghezzi, E. Vrtaˇ cnik-Bokal, R. Granese, Evaluation of m1 and m2 macrophages in ovarian endometriomas from women affected by endometriosis at different stages of the disease, Gynecological Endocrinology 36 (5) (2020) 441–444. [2] K. Moss, J. Doust, H. Homer, I. Rowlands, R. Hockey, G. Mishra, Delayed diagnosis of endometriosis disadvantages women in art: a retrospective population linked data study, Human Reproduction 36 (12) (2021) 3074–3082. [3] A. I. of Health, Welfare, Endometriosis in Australia: Prevalence and Hospitalisations: In Focus, Australian Institute of Health and Welfare, 2019. [4] C. M. Becker, et al., Eshre guideline: endometriosis, Human reproduction open 2022 (2) (2022) hoac009. [5] A. W. Horne, S. A. Missmer, Pathophysiology, diagnosis, and management of endometriosis, bmj 379 (2022). [6] A. M. Soliman, H. Yang, E. X. Du, C. Kelley, C. Winkel, The direct and indirect costs associated with endometriosis: a systematic literature review, Human reproduction 31 (4) (2016) 712–722. [7] K. Kinkel, K. A. Frei, C. Balleyguier, C. Chapron, Diagnosis of endometriosis with imaging: a review, European radiology 16 (2006) 285–298. [8] D. Butler, H. Wang, Y. Zhang, M.-S. To, G. Condous, M. Leonardi, S. Knox, J. Avery, L. Hull, G. Carneiro, The effectiveness of self-supervised pre-training for multi-modal endometriosis classification*†, 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) (2023) 1–5. URL https://api.semanticscholar.org/CorpusID:266196804 [9] M. L. Kataoka, et al., Posterior cul-de-sac obliteration associated with endometriosis: Mr imaging evaluation, Radiology 234 (3) (2005) 815–823. [10] T. Indrielle-Kelly, et al., Diagnostic accuracy of ultrasound and mri in the mapping of deep pelvic endometriosis using the international deep endometriosis analysis (idea) consensus, BioMed research international 2020 (2020). [11] Y. Zhang, H. Wang, D. Butler, M.-S. To, J. Avery, M. L. Hull, G. Carneiro, Distilling missing modality knowledge from ultrasound for endometriosis diagnosis with magnetic resonance images, in: 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), IEEE, 2023. doi:10.1109/isbi53787.2023.10230667. URL http://dx.doi.org/10.1109/ISBI53787.2023.10230667 [12] H. W. Goh, U. Tkachenko, J. Mueller, C. C. Cleanlab, Crowdlab: Supervised learning to infer consensus labels and quality scores for data with multiple annotators, arXiv preprint arXiv:2210.06812 (2022). [13] B. Wilder, E. Horvitz, E. Kamar, Learning to complement humans, in: Proceedings of the Twenty- Ninth International Joint Conference on Artificial Intelligence, IJCAI’20, 2021. [14] Z. Lu, M. Yin, Human reliance on machine learning models when performance feedback is limited: Heuristics and risks, in: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, pp. 1–16. [15] K. Weitz, D. Schiller, R. Schlagowski, T. Huber, E. Andr´ e, ” do you trust me?” increasing user- trust by integrating virtual agents in explainable ai interaction design, in: Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, 2019, pp. 7–9. [16] A. Rosenfeld, M. D. Solbach, J. K. Tsotsos, Totally looks like-how humans compare, compared to machines, 2018, pp. 1961–1964. [17] T. Serre, Deep learning: the good, the bad, and the ugly, Annual review of vision science 5 (2019) 399–426. [18] E. Kamar, S. Hacker, E. Horvitz, Combining human and machine intelligence in large-scale crowdsourcing., in: AAMAS, Vol. 12, 2012, pp. 467–474. [19] G. Bansal, B. Nushi, E. Kamar, E. Horvitz, D. S. Weld, Is the most accurate ai the best teammate? optimizing ai for teamwork, in: AAAI, Vol. 35, 2021, pp. 11405–11414. Human-AI Collaborative Multi-modal Multi-rater Learning 18 [20] K. Vodrahalli, R. Daneshjou, T. Gerstenberg, J. Zou, Do humans trust advice more if it comes from ai? an analysis of human-ai interactions, in: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, 2022, pp. 763–777. [21] X. Wu, L. Xiao, Y. Sun, J. Zhang, T. Ma, L. He, A survey of human-in-the-loop for machine learning, Future Gener. Comput. Syst. 135 (C) (2022) 364–381. doi:10.1016/j.future.2022. 05.014. URL https://doi.org/10.1016/j.future.2022.05.014 [22] M. F. Pradier, J. Zazo, S. Parbhoo, R. H. Perlis, M. Zazzi, F. Doshi-Velez, Preferential mixture-of- experts: Interpretable models that rely on human expertise as much as possible, AMIA Summits on Translational Science Proceedings 2021 (2021) 525. [23] C. Cortes, G. DeSalvo, M. Mohri, Learning with rejection, in: Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, Springer, 2016, pp. 67–82. [24] D. Madras, T. Pitassi, R. Zemel, Predict responsibly: improving fairness and accuracy by learning to defer, NeurIPS 31 (2018). [25] H. Narasimhan, W. Jitkrittum, A. K. Menon, A. Rawat, S. Kumar, Post-hoc estimators for learning to defer to an expert, NeurIPS 35 (2022) 29292–29304. [26] M. Raghu, K. Blumer, G. Corrado, J. Kleinberg, Z. Obermeyer, S. Mullainathan, The algorithmic automation problem: Prediction, triage, and human effort, arXiv preprint arXiv:1903.12220 (2019). [27] N. Okati, A. De, M. Rodriguez, Differentiable learning under triage, NeurIPS 34 (2021) 9140–9151. [28] R. Verma, D. Barrej´ on, E. Nalisnick, On the calibration of learning to defer to multiple experts, in: Workshop on Human-Machine Collaboration and Teaming in International Confere of Machine Learning, 2022. [29] A. Mao, C. Mohri, M. Mohri, Y. Zhong, Two-stage learning to defer with multiple experts, in: NeurIPS, 2023. [30] P. Hemmer, S. Schellhammer, M. V¨ ossing, J. Jakubik, G. Satzger, Forming effective human-ai teams: Building machine learning models that complement the capabilities of multiple experts, International Joint Conferences on Artificial Intelligence Organization, 2022, pp. 2478–2484, main Track. doi:10.24963/ijcai.2022/344. URL https://doi.org/10.24963/ijcai.2022/344 [31] M. Steyvers, H. Tejeda, G. Kerrigan, P. Smyth, Bayesian modeling of human–ai complementarity, Proceedings of the National Academy of Sciences 119 (11) (2022) e2111547119. [32] G. Kerrigan, P. Smyth, M. Steyvers, Combining human predictions with model probabilities via confusion matrices and calibration, NeurIPS 34 (2021) 4421–4434. [33] Z. Zhang, K. Wells, G. Carneiro, Learning to complement with multiple humans (lecomh): Integrating multi-rater and noisy-label learning into human-ai collaboration, arXiv preprint arXiv:2311.13172 (2023). [34] M. Liu, J. Wei, Y. Liu, J. Davis, Do humans and machines have the same eyes? human-machine perceptual differences on image classification, arXiv preprint arXiv:2304.08733 (2023). [35] Q. Dou, Q. Liu, P. A. Heng, B. Glocker, Unpaired multi-modal segmentation via knowledge distillation, in: IEEE Transactions on Medical Imaging, 2020. [36] M. Monteiro, L. Le Folgoc, D. Coelho de Castro, N. Pawlowski, B. Marques, K. Kamnitsas, M. van der Wilk, B. Glocker, Stochastic segmentation networks: Modelling spatially correlated aleatoric uncertainty, Advances in Neural Information Processing Systems 33 (2020) 12756– 12767. [37] Z. Han, C. Zhang, H. Fu, J. T. Zhou, Trusted multi-view classification, arXiv preprint arXiv:2102.02051 (2021). [38] H. Wang, J. Zhang, Y. Chen, C. Ma, J. Avery, L. Hull, G. Carneiro, Uncertainty-aware multi-modal learning via cross-modal random network prediction, in: European Conference on Computer Vision, Springer, 2022, pp. 200–217. Human-AI Collaborative Multi-modal Multi-rater Learning 19 [39] H. Wang, C. Ma, Y. Liu, Y. Chen, Y. Tian, J. Avery, L. Hull, G. Carneiro, Enhancing multi-modal learning: Meta-learned cross-modal knowledge distillation for handling missing modalities, arXiv preprint arXiv:2405.07155 (2024). [40] H. Wang, C. Ma, J. Zhang, Y. Zhang, J. Avery, L. Hull, G. Carneiro, Learnable cross- modal knowledge distillation for multi-modal learning with missing modality, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2023, pp. 216–226. [41] H. Wang, Y. Chen, C. Ma, J. Avery, L. Hull, G. Carneiro, Multi-modal learning with missing modality via shared-specific feature modelling, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15878–15887. [42] Y. Wang, W. Huang, F. Sun, T. Xu, Y. Rong, J. Huang, Deep multimodal fusion by channel exchanging, Advances in Neural Information Processing Systems 33 (2020) 4835–4845. [43] M. Patrick, Y. M. Asano, P. Kuznetsova, R. Fong, J. F. Henriques, G. Zweig, A. Vedaldi, Multi- modal self-supervision from generalized data transformations, arXiv preprint arXiv:2003.04298 (2020). [44] M. Patrick, P.-Y. Huang, I. Misra, F. Metze, A. Vedaldi, Y. M. Asano, J. F. Henriques, Space- time crop & attend: Improving cross-modal video representation learning, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10560–10572. [45] H. Chen, W. Xie, T. Afouras, A. Nagrani, A. Vedaldi, A. Zisserman, Localizing visual sounds the hard way, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 16867–16876. [46] X. Jia, X.-Y. Jing, X. Zhu, S. Chen, B. Du, Z. Cai, Z. He, D. Yue, Semi-supervised multi-view deep discriminant representation learning, IEEE transactions on pattern analysis and machine intelligence 43 (7) (2020) 2496–2509. [47] H.-Y. Lee, H.-Y. Tseng, J.-B. Huang, M. Singh, M.-H. Yang, Diverse image-to-image translation via disentangled representations, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 35–51. [48] X. Liu, P. Sanchez, S. Thermos, A. Q. O’Neil, S. A. Tsaftaris, Learning disentangled representations in the imaging domain, Medical Image Analysis (2022) 102516. [49] Z.-H. Zhou, Ensemble methods: foundations and algorithms, CRC press, 2012. [50] F. Rodrigues, F. Pereira, B. Ribeiro, Gaussian process classification and active learning with multiple annotators, in: ICML, PMLR, 2014, pp. 433–441. [51] V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, L. Moy, Learning from crowds., Journal of machine learning research 11 (4) (2010). [52] F. Rodrigues, F. Pereira, Deep learning from crowds, Vol. 32, 2018. [53] Z. Chen, H. Wang, H. Sun, P. Chen, T. Han, X. Liu, J. Yang, Structured probabilistic end-to-end learning from crowds, 2021, pp. 1512–1518. [54] M. Guan, V. Gulshan, A. Dai, G. Hinton, Who said what: Modeling individual labelers improves classification, in: AAAI, Vol. 32, 2018. [55] H. W. Goh, U. Tkachenko, J. Mueller, Crowdlab: Supervised learning to infer consensus labels and quality scores for data with multiple annotators (2023). arXiv:2210.06812. [56] K. He, X. Chen, S. Xie, Y. Li, P. Doll´ ar, R. Girshick, Masked autoencoders are scalable vision learners, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16000–16009. [57] A. Dosovitskiy, et al., An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.11929 (2020). [58] K. Hara, H. Kataoka, Y. Satoh, Learning spatio-temporal features with 3d residual networks for action recognition, in: Proceedings of the IEEE international conference on computer vision workshops, 2017, pp. 3154–3160. [59] I. Loshchilov, F. Hutter, Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint arXiv:1608.03983 (2016). Human-AI Collaborative Multi-modal Multi-rater Learning 20 [60] C. Feng, G. Tzimiropoulos, I. Patras, SSR: an efficient and robust framework for learning with unknown label noise, in: 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022, BMVA Press, 2022, p. 372. URL https://bmvc2022.mpi-inf.mpg.de/372/ [61] R. Xiao, Y. Dong, H. Wang, L. Feng, R. Wu, G. Chen, J. Zhao, Promix: combating label noise via maximizing clean sample utility, in: Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, E. Elkind, Ed. International Joint Conferences on Artificial Intelligence Organization, Vol. 8, 2023, pp. 4442–4450.