{"paper_id":"d2fa526d-50bb-4c3f-99df-5dd24842caeb","body_text":"Human-AI Collaborative Multi-modal Multi-rater\nLearning for Endometriosis Diagnosis\nHu Wang, David Butler, Yuan Zhang, Jodie Avery, Steven\nKnox, Congbo Ma, Louise Hull, Gustavo Carneiro\nThe University of Adelaide, Adelaide, Australia\nBenson Radiology, Adelaide, Australia\nMacquarie University, Sydney, Australia\nUniversity of Surrey, Guildford, United Kingdom\nAbstract. Endometriosis, affecting about 10% of individuals assigned female at\nbirth, is challenging to diagnose and manage. Diagnosis typically involves the\nidentification of various signs of the disease using either laparoscopic surgery or the\nanalysis of T1/T2 MRI images, with the latter being quicker and cheaper but less\naccurate. A key diagnostic sign of endometriosis is the obliteration of the Pouch\nof Douglas (POD). However, even experienced clinicians struggle with accurately\nclassifying POD obliteration from MRI images, which complicates the training of\nreliable AI models. In this paper, we introduce the H uman-AI Collaborative Multi-\nmodal Multi-rater Learning (HAICOMM) methodology to address the challenge above.\nHAICOMM is the first method that explores three important aspects of this problem:\n1) multi-rater learning to extract a cleaner label from the multiple “noisy” labels\navailable per training sample; 2) multi-modal learning to leverage the presence of\nT1/T2 MRI images for training and testing; and 3) human-AI collaboration to build a\nsystem that leverages the predictions from clinicians and the AI model to provide more\naccurate classification than standalone clinicians and AI models. Presenting results on\nthe multi-rater T1/T2 MRI endometriosis dataset that we collected to validate our\nmethodology, the proposed HAICOMM model outperforms an ensemble of clinicians,\nnoisy-label learning models, and multi-rater learning methods.\n1. Introduction\nEndometriosis is characterized by the abnormal growth of endometrial-like tissue outside\nthe uterus, often leading to distressing symptoms such as chronic pain, prolonged\nmenstrual bleeding, and infertility [1, 2]. Despite its prevalence in around 10% of\nindividuals assigned female at birth [3], diagnosing endometriosis has been a hard\ncondition to diagnose. Conventional diagnostic methods primarily rely on invasive\nlaparoscopy, a surgical procedure that involves the insertion of a slender camera through\na small incision in the abdomen to visually inspect the pelvic region [4]. This diagnostic\nmethod, while effective, presents substantial drawbacks. Chief among them is the\nsignificant delay (averaging 6.4 years [3]) that patients endure before receiving a formal\narXiv:2409.02046v3  [cs.CV]  25 Oct 2024\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 2\ndiagnosis. This long waiting period lowers the quality of life for those afflicted by the\ncondition [5]. Furthermore, the extensive reliance on laparoscopy escalates healthcare\ncosts, imposing a considerable burden on both healthcare systems and patients [6].\nThese challenges underscore the pressing need for innovative imaging-based diagnostic\nsolutions that can mitigate these issues while enhancing patient care.\nThe T1 and T2 modalities of Magnetic Resonance Imaging (MRI) are among the\nmost recommended medical imaging methods for diagnosing endometriosis given their\neffectiveness to visualize many signs of the condition. One of the most important\nsigns associated with the condition is the Pouch of Douglas (POD) Obliteration [7, 8].\nDeveloping an AI model capable of classifying POD obliteration has the potential to\nfacilitate the widespread adoption of imaging-based diagnosis and enhance diagnosis\naccuracy and consistency. However, training such a model relies on acquiring precise\nPOD Obliteration annotations from T1/T2 MRIs, which is a challenging task because\neven experienced clinicians may lack certainty regarding the presence of the sign. In\nfact, the uncertainty in the manual POD obliteration classification from T1/T2 MRI is\nremarkably low, with only 61.4% to 71.9% accuracy [9,10]. To demonstrate the difficulty\nin the detection of POD obliteration, we present Figure 1, showing T1-weighted (b) and\n(d) paired with T2-weighted (a) and (c) MR images in the sagittal plane. The (a) and (b)\npair shows a normal POD case, while the (c) and (d) pair shows POD obliteration, with\nthe red arrow pointing to significant adhesion and distortion, indicating the loss of the\nsoft tissue plane separating the uterine fundus from the bowel in the POD. Nevertheless,\nthere have been some attempts to train machine learning models for POD obliteration\ndetection [8,11]. Despite their reported accuracy, these methods have not been validated\nagainst ground truth labels from surgical reports, making it challenging to assess their\nclinical value in the detection of POD obliteration. Therefore, a major research question\nin this problem is if it is possible to design innovative training and testing methodologies\nthat can lead to highly accurate and clinically useful POD obliteration classification\nresults.\nThere are many important aspects of this problem that can be leveraged in\norder to formulate an innovative solution to produce an accurate POD obliteration\nclassifier. First, the uncertain manual classification by clinicians can lead to training\nsets that contain multiple “noisy” labels per training sample (with each label being\nproduced by a different clinician), which can be explored by multi-rater learning\nmechanisms [12]. Second, given that clinicians and AI models may not be highly\naccurate, the combination of their predictions may lead to more accurate predictions\n– such idea is studied by human-AI collaborative classification [13]. Third, similarly\nto previous approaches [8, 11], it is important to explore the complementarity of the\nmultiple MRI modalities.\nIn this paper, we explore the three points listed above to propose the innovative\nHuman-AI Collaborative Multi-modal Multi-rater Learning (HAICOMM) methodology.\nHAICOMM is the first method in the field that simultaneously explores multi-rater\nlearning to provide a clean training label from the multiple “noisy” labels produced\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 3\n(a) (b)\n(c) (d)\nFigure 1: Paired sagittal MR images, with T1-weighted (b, d) and T2-weighted (a,\nc) scans. The (a, b) pair shows a normal POD, while the (c, d) pair reveals POD\nobliteration, highlighted by a red arrow indicating significant adhesion and tissue\ndistortion, demonstrating the loss of the soft tissue plane separating the uterine fundus\nfrom the bowel.\nby clinicians, multi-modal learning to leverage the presence of T1/T2 MRI images,\nand human-AI collaboration to build a system that synergises predictions from both\nclinicians and the AI model. The contributions of this paper are:\n• The first human-AI collaborative multi-modal multi-rater learning methodology\nthat produces a highly accurate POD obliteration classifier from T1/T2 MRIs;\n• The first multi-modal multi-rater dataset annotated with imaging and surgery-\nbased POD obliteration labels for the diagnosis of endometriosis.\nExperiments on our proposed endometriosis dataset shows that our HAICOMM model\npresents more accurate POD classification than predictions produced by an ensemble of\nclinicians, by noisy-label learning methods, and by multi-rater learning methods.\n2. Literature Review\n2.1. Human-AI Collaboration\nHuman-AI Collaboration (HAIC) integrates the unique strengths of human experts and\nAI systems, resulting in improved model capabilities and performance when compared to\nstandalone AI systems [14,15]. The motivation behind HAIC arises from research [16–18]\nthat highlights the limitations of traditional isolated AI methods, overlooking the\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 4\npotential of human-AI collaboration. To overcome these limitations, researchers have\nproposed various strategies to enhance human-AI collaboration [19–22]. Two key\nstrategies within HAIC have emerged: learning to defer and learning to complement.\nLearning to defer (l2D), which evolved from the concept of learning to reject [23, 24],\nfocuses on optimizing the decision of whether to defer prediction to either the expert or\nthe AI system. Researchers have investigated several L2D approaches [25–27], initially\nin single-expert scenarios but later extending to multi-user collaborations [28–30]. On\nthe other hand, learning to complement [13] focuses on maximizing the expected utility\nof combined human-AI decisions, and various frameworks have been proposed to model\nhuman-AI complementarity [19,31–34].\n2.2. Multi-modal Learning\nMulti-modal learning has become increasingly crucial in various fields, including medical\nimage analysis and computer vision. It combines data from different sources to provide\na more comprehensive understanding of tasks. In medical image analysis, several\ninnovative methods have been developed. These include a chilopod-shaped architecture\nusing modality-dependent feature normalization and knowledge distillation [35], a pixel-\nwise coherence approach modeling aleatoric uncertainty [36], a trusted multi-view\nclassifier using the Dirichlet distribution [37], and an uncertainty-aware model based on\ncross-modal random network prediction [38]. Wang et al. [39–41] also tried to approach\nthe missing modality issues in the multi-modal learning scenario. Computer vision\nhas seen advancements in multi-modal learning as well. Researchers have combined\nchannel exchanging with multi-modal learning [42], applied self-supervised learning\nto improve performance [43, 44], enhanced video-and-sound source localization [45],\nintroduced a model for multi-view learning [46], and explored feature disentanglement\nmethods [47,48].\n2.3. Multi-rater Learning\nMulti-rater learning is a technique designed to train a classifier using noisy labels\ngathered from multiple annotators. The challenge lies in how to derive a “clean” label\nfrom these imperfect labels. Traditional approaches often rely on majority voting [49]\nand the expectation-maximization (EM) algorithm [50, 51]. Rodrigues et al. [52]\nintroduce an end-to-end deep neural network (DNN) that incorporates a crowd layer to\nmodel the annotator-specific transition matrix, enabling the direct training of a DNN\nwith crowdsourced labels. Alternatively, Chen et al. [53] suggest a probabilistic model\nthat learns an interpretable transition matrix unique to each annotator. Meanwhile,\nGuan et al. [54] employ multiple output layers in the classifier and learn combination\nweights to aggregate the results. More recently, CROWDLAB [55] has set the state\nof the art in multi-rater learning by using multiple noisy-label samples and predictions\nwith a model trained via label noise learning. Despite the promise of multi-rater learning\nin leveraging multiple noisy labels per training sample, it falls short by overlooking the\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 5\nconcept of human-AI collaboration and multi-modal learning.\n2.4. Imaging-based Endometriosis Detection\nOne crucial indicator to detect endometriosis is the obliteration of the Pouch of Douglas\n(POD) [7,8]. However, the development of an AI model that can classify such indicator\nhinges on the availability of precise POD obliteration annotations from T1/T2 MRIs,\na task that is challenging because even experienced clinicians often face uncertainty in\nidentifying this sign. Despite these challenges, there have been some efforts to train\nmulti-modal MRI AI models for POD obliteration classification. For example, Zhang et\nal. [11] proposed a method to transfer knowledge from ultrasound to MRI for classifying\nPOD obliteration, and Butler et al. [8] explored self-supervised pre-training for multi-\nmodal POD obliteration classification. However, these methods have not been validated\nagainst ground-truth labels obtained from surgical reports, making it difficult to assess\ntheir clinical validity.\nNevertheless, for all of the aforementioned related work, none of the methods\ndeal with human-AI collaboration, multi-modal classification, and multi-rater learning\nsimultaneously, particularly for classifying endometriosis. In this paper, we propose the\nHAICOMM model to address this research gap.\n3. Methodology\nThe training of our HAICOMM methodology is depicted in Fig. 2. The first stage\nconsists of pre-training a multi-modal encoder using a large unlabelled T1/T2 MRI\ndataset, with a self-supervised learning mechanism [56] (see frame (a) in Fig. 2).\nSubsequently, for training the proposed human-AI classifier HAICOMM, we first need\nto estimate the pseudo ground truth label from the the multiple “noisy” labels available\nfor each pair of T1/T2 MRI training images. We rely on CrowdLab [12] to produce such\npseudo ground truth labels (see frame (b) in Fig. 2). Next, the T1/T2 MRI images with\nmulti-rater (manual) labels are fed into their multi-modal encoders. The embeddings\nfrom the multi-modal and label encoders are combined to produce the final prediction\nthat is trained to match the pseudo ground truth label (see frame (c) in Fig. 2). We\nprovide details about each of these training stages below.\n3.1. Multi-modal Encoder Pre-training\nThe MRI encoder of the HAICOMM model is pre-trained with the Masked Autoencoder\n(MAE) self-supervised learning method [56]. For this pre-training, we use a dataset\ndenoted as DP = {x′(i)\nt1 }Mt1\ni=1\nS{x′(i)\nt2 }Mt2\ni=1 , with x′(i)\nt1 , x′(i)\nt2 ∈ X ⊂ RH×W ×D denoting the\nT1 and T2 MRI volumes of size H × W × D. It is worth noting that the number of\nunlabeled images, M = Mt1 + Mt2, far exceeds the number of labeled images, denoted\nas N (i.e., M >> N ), of the datasets that will be defined in Sections 3.2 and 3.3.\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 6\nP(i) from Eq.6\nPseudo Label\nMAE Encoder\nFT1 from Eq.5\nMAE DecoderOriginal Image Masked Image Reconstruction\n(a) HAICOMM Pretraining from Eq.1\n...\nMulti-rater Labels\n(c) Multi-modal Human-AI \nCollaborative Classification\nConcat Fts\nFr from Eq.5 \nFC layers\nLoss \nMinimization \nfrom Eq.7\n...\n(b) Multi-rater Learning\nMajority Voting\nMajority Vote \nClassifier\nCrowdLab\nPseudo Labels\nT1\nT2\nVoted labels\nMulti-rater Labels\n⊕\n...\nwith dataset DP\nDMV     from Eq.2\nD’T\nfrom Eq.3\nFT2 from Eq.5\nFigure 2: The framework of HAICOMM. The MRI encoders of HAICOMM are: (a)\nfirstly pre-trained with a Masked Autoencoder (MAE) model; then (b) the pseudo clean\nlabels are estimated from the multi-rater learning process; next, (c) the T1 and T2 data,\nalong with the human-produced multi-rater labels are entered into respective feature\nextraction encoders – the features from three sources are fused for the final prediction.\nIn the figure, “FC” means fully-connected, “Fts” represents features, “Concat” denotes\nconcatenation, and “ ⊕” is the concatenation operation.\nFollowing the 3D Vision Transformer [57], the architecture of 3D-MAE follows an\nasymmetric encoder-decoder setup. The encoder, parameterized by ϕ, is represented\nby gϕ : X → F , which receives visible patches along with positional embeddings that\nare processed through a 3D Vision Transformer to produce features in the space F.\nThe resulting features are subsequently directed to the decoder, parameterized by ψ\nand denoted by fψ : F → X , which reconstructs the original volume with the masked\nvolume tokens. In the MRI pre-training, our objective is to minimize the mean squared\nerror (MSE) of the reconstruction of the original masked patches. Formally, we have:\nϕ∗, ψ∗ = arg minϕ,ψ 1\nM\n\u0012PMt1\ni=1\n\r\r\rfψ\n\u0010\ngϕ\n\u0010\nx′(i)\nt1\n\u0011\u0011\n−x′(i)\nt1\n\r\r\r\n2\n2\n+PMt2\ni=1\n\r\r\rfψ\n\u0010\ngϕ\n\u0010\nx′(i)\nt2\n\u0011\u0011\n−x′(i)\nt2\n\r\r\r\n2\n2\n\u0013\n, (1)\nwhere ∥ · ∥ 2 denotes the L2-norm. The optimization in (1) jointly learns the encoder\nparameters ϕ and the decoder parameters ψ to minimize the reconstruction error of the\nmasked patches across all training samples. For the subsequent training and evaluation\nof the human-AI collaborative classifier, we adopt the pre-trained feature extractor\ngϕ∗(.), as explained below in Sec. 3.3.\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 7\n3.2. Multi-rater Learning\nThe training of our human-AI collaborative classifier requires each pair of T1/T2\nMRI training images to have a single pseudo clean label estimated from the multiple\n“noisy” training labels. The multi-modal multi-rater dataset is denoted by DT =\n{(x(i)\nt1 , x(i)\nt2 , y(i))}N\ni=1 with N samples, where the multi-rater label has K binary\nannotations denoted as y(i) ∈ Y ⊂ { 0, 1}K, provided by the K clinicians who annotated\nthe training images in DT .\nWith the multi-rater labels, we first perform majority vote to fetch the most\nfrequently appearing labels per training sample. Let us present the majority vote\noperation as h : Y → { 0, 1}. Then, we have the mapping from multi-rater labels to\nthe majority label for each multi-modal sample, forming the following majority voting\ndataset:\nDM V = {(x(i)\nt1 , x(i)\nt2 , ˆy(i))}N\ni=1, where ˆy(i) = h(y(i)), (2)\nwhere ˆy(i) ∈ {0, 1}. With such generated consensus labels from majority vote, we train\na classifier fθ : X × X → [0, 1] that takes the T1 and T2 MRI volumes to optimize\na standard binary cross-entropy objective function. This trained classifier, together\nwith DT will then be used for the filtering process of multi-rater cleaning technique\nCrowdLab [12], denoted by z : [0, 1] × Y → { 0, 1}, which generates a pseudo clean label\nfor each sample. Formally, CrowdLab’s pseudo labeling process is defined by:\ny(i) = z(fθ(x(i)\nt1 , x(i)\nt2 ), y(i)), (3)\nwhere y(i) ∈ {0, 1} denotes CrowdLab’s estimate for the “clean” label of the i-th sample,\nwhich is referred to as the pseudo ground-truth label.\n3.3. Multi-modal Human-AI Collaborative Classification\nGiven the pseudo clean labels y(i) from (3), the dataset for the final model training is\ndefined as:\nD′T = {(x(i)\nt1 , x(i)\nt2 , y(i), y(i)}N\ni=1. (4)\nThe pre-trained encoder from (1), parameterized by ϕ∗, is utilized to initialize the\nT1 and T2 MRI feature extractors, respectively defined byg ˆϕ : X → F and gϕ′ : X → F ,\nfor the classification task. We also need to define a new feature extractor for processing\nthe manual labels with a learnable module defined as gγ : Y → F . Such manual labels\nare used in the human-AI collaborative module. Hence, these extractors produce the\nT1, T2 and rater features as:\nF(i)\nt1 = g ˆϕ(x(i)\nt1 ),\nF(i)\nt2 = gϕ′(x(i)\nt2 ),\nF(i)\nr = gγ(y(i)).\n(5)\nThese three feature maps are then concatenated and fed into a learnable linear projection\nπη : F × F × F → F , parameterized by η to predict:\np(i) = σ\n\u0010\nπη(F(i)\nt1 ⊕ F(i)\nt2 ⊕ F(i)\nr )\n\u0011\n, (6)\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 8\nwhere p(i) ∈ [0, 1]2 denotes the probabilistic prediction, ⊕ represents the concatenation\noperator, and σ is the softmax function.\nFinally, the training of the whole model is performed by minimizing the binary\ncross-entropy loss, as follows:\nˆϕ∗, ϕ′∗, γ∗, η∗ = arg min\nˆϕ,ϕ′,γ,η\n− 1\nN\n NX\ni=1\ny(i) log(p(i)) + (1 − y(i)) log(1 − p(i))\n!\n, (7)\nwhere the p(i) is the predicted probability of the positive class of i-th sample (i.e., the\n2nd dimension p(i) in (6)), and y(i) is the pseudo clean label in D′\nT . The goal is to learn\nthe optimal parameters ˆϕ∗, ϕ′∗, γ∗, and η∗ that minimize this loss across all N samples.\nThe testing process consists of taking the input T1/T2 MRI images and labels y\nfrom clinicians to output p from (6).\n4. Experimental Settings\n4.1. Endometriosis Dataset\nWe first introduce our multi-modal multi-rater dataset annotated with imaging and\nsurgery-based POD obliteration labels for the diagnosis of endometriosis. For the pre-\ntraining stage, we collected 5,867 unlabeled T1 MRI volumes and 8,984 unlabeled T2\nMRI volumes from patients aged between 18 and 45 years old, where the volumes show\nfemale pelvis scans obtained from various MRI machines with varying resolutions. The\npre-training pelvic MRI scans were obtained from the South Australian Medical Imaging\n(SAMI) service. For this paper, we included only pelvic MRI scans of female patients\nacquired using either T1- or T2-weighted imaging sequences. These scans were collected\nfrom multiple centers using various equipments, including the 1.5T Siemens Aera, 3T\nSiemens TrioTim, 1.5T Philips Achieva, 1.5T Philips Ingenia, and 1.5T Philips Intera.\nThe T1 and T2 datasets include a wide variety of sequences, where each scan is acquired\nusing a combination of the following parameters: spin echo sequences (e.g., Turbo\nSpin Echo, TSE) or gradient echo sequences (e.g., Volumetric Interpolated Breath-\nhold Examination, VIBE); orientation in either 2D planes (axial, sagittal, coronal)\nor 3D sequences (e.g., SPC); with or without fat saturation/suppression techniques\n(e.g., DIXON, SPIR, SPAIR); and with or without gadolinium contrast enhancement.\nWe show the statistics of the size distribution information of the pre-training dataset\nin Table 1. As described in Section 3.1, this dataset is used as containing 14,851\nindependent scans for the Masked Autoencoder (MAE) self-supervised learning. To\nstandardize the data, the volumes were resampled to 1mm×1mm×3mm voxels. Also, we\napply 3D contrast-limited adaptive histogram equalization (CLAHE) to enhance image\nlocal contrast and refine edge definitions. We choose CLAHE, because it produced the\nbest classification results among the different data pre-processing methods tested on an\ninternal validation process (e.g., voxel value truncate.).\nFor the training of the human-AI collaborative POD obliteration classifier, we\ncollected 82 pairs of T1/T2 MRI volumes with patients aged 18 to 45 years old. These\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 9\nTable 1: Statistics of the volume size distribution of the pre-training dataset.\n(a) T1 pre-training dataset\nDim Mean Std Max Min\n0 326.67 93.73 550 66\n1 271.97 88.78 530 60\n2 79.64 24.5 430 50\n(b) T2 pre-training dataset\nDim Mean Std Max Min\n0 245.78 113.33 530 60\n1 208.47 93.87 504 50\n2 77.01 23.99 430 50\nscans were obtained across multiple clinical sites, with each case annotated by three\nexperienced clinicians who work in clinics specialized in the imaging-based diagnosis of\nendometriosis. The scans were acquired using a standardized protocol for endometriosis\nexamination and they show a specific region surrounding the uterus, which is the area\nwhere the signs of POD obliteration are more visible. MRIs from patients with a history\nof hysterectomy or those with large pelvic lesions that limited adequate assessment of\nthe POD were excluded. The scanners are of the following models: SIEMENS Aera,\nSIEMENS Espree, SIEMENS MAGNETOM Sola, and SIEMENS MAGNETOM Vida.\nThe volume types are: T1-weighted MRI images, obtained using a Volume Interpolated\nBreath-hold Examination technique with Dixon fat-water separation, acquired in the\ntransverse plane; and T2-weighted MRI images, acquired using the SPACE sequence,\ncharacterized by non-saturation techniques, taken in the coronal plane, and with\nisotropic voxel dimensions. All 3D volumes were reoriented to a Right-Anterior-Superior\n(RAS) coordinate system to standardize anatomical alignment, then resampled with an\noutput spacing. Similar to the pre-training, to standardize the data, the volumes were\nresampled to 1mm × 1mm × 3mm voxels. We show the statistics of the size distribution\ninformation of the POD obliteration dataset in Table 2\nTable 2: Statistics of the volume size distribution of the POD obliteration dataset.\n(a) T1 modality in the dataset\nDim Mean Std Max Min\nDepth 383.91 91.88 576 260\nHeight 312.10 72.61 464 250\nWidth 72.16 8.46 80 51\n(b) T2 modality in the dataset\nDim Mean Std Max Min\nDepth 389.80 18.40 448 384\nHeight 192.30 26.22 336 160\nWidth 289.71 19.37 354 192\nThis training set with 82 volumes is subdivided into 62 volumes for training and\n20 for validation that are used for cross validation for hyper-parameter selection. We\nfurther collect 30 cases that contain ground truth annotation of POD obliteration from\nsurgical reports. These cases serve as gold standards for testing. We show in Table 6 the\ndistribution of normal vs abnormal (i.e., POD obliteration) images in the training and\ntesting sets. Note that while in the testing set, we have ground truth from surgery, in the\ntraining set, we only have the labelers’ annotation. We also use CLAHE pre-processing\nin this dataset. Similar to the pre-training, to standardize the data, the volumes were\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 10\nresampled to 1mm × 1mm × 3mm voxels.\nTable 3: Number of normal and abnormal (i.e., with POD obliteration) images in the\ntraining and testing sets. In the training set, ‘A1’ to ‘A3’ are the three annotators, ‘MV’\nrepresents the annotations from majority voting, and ‘CL’ denotes the annotations from\nCrowdLab [55]. In the testing set, we have the grount truth (GT) from surgery.\nPOD Training Set Testing Set\nA1 A2 A3 MV CL GT\nNormal 62 48 36 51 47 15\nAbnormal 20 34 46 31 34 15\n4.2. Implementation Details\nFor model pre-training, the input volumes are cropped and possibly zero-padded to\nachieve the dimension of 64 × 128 × 128 voxels. To maintain consistency with the pre-\ntraining dataset, the endometriosis training and testing samples are manually centered\nat the uterine region which is the most important region for the detection of POD\nobliteration. Later we center crop each volume to the same dimensions as pre-training\ndata, i.e. 64 × 128 × 128 voxels, for training and testing. In both pre-training and\nthe training of the human-AI collaborative POD obliteration classifier, the multi-modal\nencoder for each modality is a transformer with 12 blocks. The majority vote classifier\nhas a 3D-ResNet50 as its backbone network [58]. For the human-AI collaborative POD\nobliteration classifier training, we use 5 epochs for model optimization warming up.\nAdamW optimizer and base learning rate of 1e-3 with cosine annealing [59] learning rate\ntuning strategy are adopted. Three multi-rater labels from three different annotators are\nincorporated into the training process. In the testing phase, the scans are also cropped\nin the uterine regions and the clinical surgical results serve as the ground truth for\naccurate evaluation. Using a cross-validation procedure, with a training set containing\n62 volumes and a validation set comprising 20 volumes, we observe signs of overfitting\nafter 60 epochs. To prevent this, we implemented early stopping and halted the training\nat the 60th epoch. Note that the majority voting is only produced for the consensus\npseudo label required for the training of the model fθ(.) to be used by CrowdLab, as\nexplained in Eq. 3. Once the pseudo clean labels are generated by CrowdLab, the\nmajority voting will no longer be needed. All experiments in the paper are run on an\nNVIDIA GeForce RTX 3090. The main Python libraries used in our implementation\nare: torch 1.7.1+cu110, torchvision 0.8.2+cu110, scikit-image and scipy.\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 11\nFigure 3: The ROC Curve of HAICOMM models and its counterparts.\n4.3. Quantitative Evaluation Settings\nWe compare the performance of our proposed HAICOMM with respect to the\nfollowing models: 1) purely manual annotation from the three expert clinicians via\nmajority voting; 2) models trained with noisy-label learning techniques (SSR [60] and\nProMix [61]) using the noisy labels from one of the annotators (GT1, GT2, GT3); 3)\nmodels trained from labels produced by the multi-rater learning CrowdLab [12] (in the\ntable denoted as models w/ CL); and 4) human-AI classifiers using the three annotators\n(models w/ HAIC). In terms of evaluation metrics, we adopt Accuracy and Area Under\nthe ROC Curve (AUROC).\nWe also show an ablation study that compares the performance of each annotator\nwithout using the system, with the performance of different combinations of annotators\nto be used in the human-AI collaboration, and the performance of the proposed\nHAICOMM.\n5. Results and Discussion\n5.1. Model Performance\nThe performance results in Table 4 show that the proposed HAICOMM outperforms\nother competing models by a large margin across the accuracy and AUC measures.\nRelative improvements vary from 9.10% to 54.83% on accuracy and 19.37% and 64.75%\non AUROC. The standard deviation is calculated by inference time bootstrapping. Also,\nwe provide the the ROC Curve of HAICOMM models and its counterparts SSR and\nProMix with CrowdLab labels in Fig. 3.\nThere are interesting points to observe in the results from Table 4. First, the\nmulti-rater learning tends to be more accurate than noisy label learning. The manual\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 12\nTable 4: Performance comparison of HAICOMM and other models on the test set in\nterms of accuracy and AUROC (and respective improvements of HAICOMM) with the\ntesting ground-truth labels from surgical reports. The standard deviation is calculated\nby inference time bootstrapping. “Majority Vote” denotes a purely manual classification\nusing the majority vote of the three annotators. SSR and ProMix w/ “GT1”, “GT2”\nand “GT3” mean noisy-label learning models trained with labels from annotators #1,\n#2 and #3, respectively. “CL” denotes noisy-label learning models trained with labels\ny(i) from Eq. (3) produced by the multi-rater learning method CrowdLab. “HAIC”\nrepresents a model that collaborates with the annotators and that is trained with labels\nfrom CrowdLab. The best results for each column are in bold.\nMethods Models Accuracy Improvement AUROC Improvement\nHuman Majority Vote 0.70±0.00 14.29% - -\nSSR w/ GT1 0.58±0.02 37.93% 0.58±0.01 53.83%\nNoisy SSR w/ GT2 0.60±0.04 33.33% 0.58±0.01 51.67%\nLabel SSR w/ GT3 0.57±0.02 41.17% 0.56±0.06 57.07%\nLearning ProMix w/ GT1 0.53±0.02 50.01% 0.54±0.04 64.75%\nProMix w/ GT2 0.62±0.03 29.72% 0.58±0.02 52.63%\nProMix w/ GT3 0.52±0.05 54.83% 0.57±0.02 56.54%\nMulti- SSR w/ CL 0.62±0.02 29.72% 0.59±0.01 50.82%\nrater ProMix w/ CL 0.65±0.04 23.08% 0.54±0.03 63.32%\nHAIC SSR w/ HAIC 0.68±0.08 17.08% 0.74±0.01 19.37%\nProMix w/ HAIC0.67±0.06 20.00% 0.74±0.04 19.98%\nOurs HAICOMM 0.80±0.04 - 0.89±0.06 -\nannotation without any assistance from the model fθ(.) in Eq. 3, shows a relatively low\naccuracy of 0.7, motivating the importance of the proposed human-AI collaboration.\nAlso, when noisy-label learning models are designed to collaborate with humans, we can\nsee large performance improvements, such as shown by “SSR w/ HAIC” and “ProMix\nw/ HAIC”. However, the proposed HAICOMM still obtains much higher accuracy\nand AUROC. Additionally, the proposed HAICOMM shows a much simpler training\nalgorithm than “SSR w/ HAIC” and “ProMix w/ HAIC”. To summarize, the proposed\nmodel outperforms not only the ensemble of experts (Majority Vote), but also the\ntop-performing multi-rater learning model (SSR w/CL and ProMix w/ CL), as well as\nthe best noisy-label learning methods (SSR and ProMix), even after adding human AI\ncollaboration (SSR w/ HAIC and ProMix w/ HAIC) by a large margin.\n5.2. Human-AI Collaborative Multi-modal Multi-rater Ablation Study\nThe first three rows of Table 5 present the accuracy of each of the three annotators.\nThe next rows show HAICOMM without relying on any human collaboration (w/o\nHAIC), then the next 6 rows show different combinations of annotators for the human-\nAI collaboration process. This is followed by two rows showing HAICOMM with single\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 13\nTable 5: Accuracy and AUROC performance analyses of HAICOMM and its variants on\nthe test set. “R1”, “R2” and “R3” denote models trained with input annotations from\nannotators #1, #2 and #3, respectively. “HAIC” represents model trained with multi-\nrater labels inputs for human-AI collaborations. T1 and T2 Only w/ HAIC represent\nsingle-modality HAICOMM approaches. The best results for each column are in bold.\nModels AccuracyAUROC\nLabels from Rater #1 (R1)0.67 -\nLabels from Rater #2 (R2)0.73 -\nLabels from Rater #3 (R3)0.70 -\nHAICOMM w/o HAIC 0.63 0.59\nHAICOMM w/ R1 0.77 0.70\nHAICOMM w/ R2 0.80 0.84\nHAICOMM w/ R3 0.57 0.60\nHAICOMM w/ R1,2 0.77 0.87\nHAICOMM w/ R2,3 0.73 0.87\nHAICOMM w/ R1,3 0.63 0.67\nT1 Only w/ HAIC 0.67 0.81\nT2 Only w/ HAIC 0.77 0.88\nHAICOMM 0.80 0.89\nmodality data (either T1 or T2) in the input. Last row shows the HAICOMM results.\nNote that the collaboration with annotators almost always improve over the result of\nHAICOMM w/o HAIC, and it also improves the accuracy for most of the annotators\n(particularly R1 and R2). Interestingly, we found that the model with R2 inputs\nperforms the best among with single rater labels. The model with combination inputs\nof R1 and R3 performs the worst. This may suggest that R2 provides relatively more\naccurate labels compared to R1 and R3. This phenomenon resonates with the fact\nthat R2 provides the most accurate labels among three raters (as shown in the first\nthree rows). This table also shows that both single modality results with HAIC (with\nT2 being much better than T1) have worse results than the multi-modal HAICOMM,\nwhich provides evidence of the need for multi-modal analysis in the classification of\nPOD obliteration.\n5.3. Analyses\nWe conduct a qualitative analysis of HAICOMM. In Figure 4, (a) and (b) are the\ninput T1 and T2 MRIs, respectively. The table below shows the predictions by the\nthree raters (Rater #1,#2,#3), then the predictions by SSR and ProMix trained with\nRater #1’s labels and CROWDLAB’s labels (SSR w GT1, ProMix w GT1, SSR w CL\nGT, ProMix w CL GT). Next, we show SSR and ProMix trained with CROWDLAB’s\nlabels and relying on human-AI collaborative classification (SSR w HAIC, ProMix w\nHAIC), followed by the result from our HAICOMM, and the ground truth label from\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 14\nsurgical data. This figure shows that the proposed HAICOMM model can generate\ncorrect labels, with other approaches showing incorrect predictions. For (c) and (d), we\nshow that the proposed HAICOMM model produced a correct label while most of other\nmethods failed (only ProMix w GT1 and HAICOMM predict the surgical ground truth\nlabel correctly).\n(a) Modality T1 (b) Modality T2\nModels Rater \n#1\nRater \n#2\nRater \n#3\nSSR w \nGT1\nProMix w \nGT1\nSSR w \nCL GT\nProMix w \nCL GT\nSSR w \nHAIC\nProMix w \nHAIC\nHAICO\nMM\nSurgic\nal\nPreds 0 1 0 0 0 0 0 0 0 1 1\n(c) Modality T1 (d) Modality T2\nModels Rater \n#1\nRater \n#2\nRater \n#3\nSSR w \nGT1\nProMix w \nGT1\nSSR w \nCL GT\nProMix w \nCL GT\nSSR w \nHAIC\nProMix w \nHAIC\nHAICO\nMM\nSurgic\nal\nPreds 1 1 0 0 1 0 0 0 0 1 1\nFigure 4: Qualitative analysis of HAICOMM. Each row (a),(b) and (c),(d) presents the\ninput T1 and T2 MRI images, with corresponding tables below showing predictions from\nthree human raters (Rater #1, #2, and #3). The tables also display predictions from\nSSR and ProMix models trained using labels from Rater #1 (SSR w/ GT1, ProMix\nw/ GT1) and CROWDLAB (SSR w/ CL GT, ProMix w/ CL GT). Following these, we\nprovide the predictions from SSR and ProMix models trained on CROWDLAB labels\nand utilizing human-AI collaborative classification (SSR w/ HAIC, ProMix w/ HAIC).\nFinally, we present results from our HAICOMM model, along with the ground truth\nlabel based on surgical data.\nIn terms of the normal and abnormal volumes, they are sampled from distributions\nwith equivalent condition / equipment / resolution, so they have similar properties. We\ncalculate the statistics of the volume size distribution of the normal / abnormal (for the\ntraining data, we use majority vote results as the label) ones in Table 7 and Table 8.\nMoreover, we plot the distribution of manufacturer model of scanners for normal and\nabnormal in Figure 5 and the distribution of magnetic field strength for normal and\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 15\nabnormal in Figure 6.\nTable 6: Number of normal and abnormal (i.e., with POD obliteration) volumes in the\ntraining and testing sets. In the training set, ‘A1’ to ‘A3’ are the three annotators, ‘MV’\nrepresents the annotations from majority voting, and ‘CL’ denotes the annotations from\nCrowdLab [55]. In the testing set, we have the grount truth (GT) from surgery.\nPOD Training Set Testing Set\nA1 A2 A3 MV CL GT\nNormal 62 48 36 51 47 15\nAbnormal20 34 46 31 34 15\nTable 7: Statistics of the volume size distribution of the T1 MRIs of POD obliteration\ndataset.\n(a) Normal Statistics\nDim Mean Std Max Min\nDepth 382.73 91.05 576 260\nHeight 313.67 74.57 464 260\nWidth 72.44 7.81 80 52\n(b) Abnormal Statistics\nDim Mean Std Max Min\nDepth 381.71 90.31 576 320\nHeight 309.35 71.95 464 250\nWidth 72.07 8.67 80 51\nTable 8: Statistics of the volume size distribution of the T2 MRIs of POD obliteration\ndataset.\n(a) Normal Statistics\nDim Mean Std Max Min\nDepth 389.82 18.54 448 384\nHeight 190.06 17.64 256 176\nWidth 292.36 13.90 336 288\n(b) Abnormal Statistics\nDim Mean Std Max Min\nDepth 389.79 18.40 448 384\nHeight 193.05 28.50 336 160\nWidth 288.82 20.83 354 192\n6. Conclusion and Future Work\nIn this paper, we proposed the Human-AI Collaborative Multi-modal Multi-rater\nLearning (HAICOMM) methodology for an imaging-based endometriosis classification.\nIt integrates the capabilities of machine learning models and multiple human labels to\nenhance the classification accuracy of POD obliteration from T1/T2 MRIs. Evaluation\non our endometriosis dataset demonstrates the efficacy of the HAICOMM model,\nsurpassing ensemble clinician predictions, noisy-label learning approaches, and a multi-\nrater learning method. This underscores the potential of collaborative efforts between\nAI and human clinicians in diagnosing and managing endometriosis and other complex\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 16\n(a) (b)\nFigure 5: Distribution of manufacturer model of scanners for the normal (a) and\nabnormal (b) datasets.\n(a) (b)\nFigure 6: Distribution of magnetic field strength of scanners for the normal (a) and\nabnormal (b) datasets.\nmedical conditions. To the best of knowledge, we are the first to propose the multi-\nmodal multi-rater classification task. Furthermore, our endometriosis dataset is the first\nin the field to enable the development of multi-modal multi-rater classifiers.\nOne potential limitation of our method is the dataset size. Currently, we are\ndedicated to collect more data from different clinical sources to expand the dataset. The\nuse of such multiple clinical sources will require the exploration of domain adaptation\ntechniques to enable a better flexibility of the method to work in multiple domains.\nBeyond this issue, the need for specific labellers for training and testing is another\npotential limitation. We plan to address this issue with the development of techniques\nthat work with a variable set of labellers during training and testing. Another interesting\ndirection is the collection of new datasets for other multi-modal multi-rater clinical\nproblems to enable the evaluation of our HAICOMM in a different task. One more\npotential future direction is the development of a detector of region of interests, which\nwill require dense (i.e., pixel-wise) annotations of the training and testing sets and the\ncollection of significantly larger training sets.\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 17\nReferences\n[1] A. S. Lagana, F. M. Salmeri, H. Ban Frangeˇ z, F. Ghezzi, E. Vrtaˇ cnik-Bokal, R. Granese, Evaluation\nof m1 and m2 macrophages in ovarian endometriomas from women affected by endometriosis at\ndifferent stages of the disease, Gynecological Endocrinology 36 (5) (2020) 441–444.\n[2] K. Moss, J. Doust, H. Homer, I. Rowlands, R. Hockey, G. Mishra, Delayed diagnosis of\nendometriosis disadvantages women in art: a retrospective population linked data study, Human\nReproduction 36 (12) (2021) 3074–3082.\n[3] A. I. of Health, Welfare, Endometriosis in Australia: Prevalence and Hospitalisations: In Focus,\nAustralian Institute of Health and Welfare, 2019.\n[4] C. M. Becker, et al., Eshre guideline: endometriosis, Human reproduction open 2022 (2) (2022)\nhoac009.\n[5] A. W. Horne, S. A. Missmer, Pathophysiology, diagnosis, and management of endometriosis, bmj\n379 (2022).\n[6] A. M. Soliman, H. Yang, E. X. Du, C. Kelley, C. Winkel, The direct and indirect costs associated\nwith endometriosis: a systematic literature review, Human reproduction 31 (4) (2016) 712–722.\n[7] K. Kinkel, K. A. Frei, C. Balleyguier, C. Chapron, Diagnosis of endometriosis with imaging: a\nreview, European radiology 16 (2006) 285–298.\n[8] D. Butler, H. Wang, Y. Zhang, M.-S. To, G. Condous, M. Leonardi, S. Knox, J. Avery, L. Hull,\nG. Carneiro, The effectiveness of self-supervised pre-training for multi-modal endometriosis\nclassification*†, 2023 45th Annual International Conference of the IEEE Engineering in Medicine\n& Biology Society (EMBC) (2023) 1–5.\nURL https://api.semanticscholar.org/CorpusID:266196804\n[9] M. L. Kataoka, et al., Posterior cul-de-sac obliteration associated with endometriosis: Mr imaging\nevaluation, Radiology 234 (3) (2005) 815–823.\n[10] T. Indrielle-Kelly, et al., Diagnostic accuracy of ultrasound and mri in the mapping of deep pelvic\nendometriosis using the international deep endometriosis analysis (idea) consensus, BioMed\nresearch international 2020 (2020).\n[11] Y. Zhang, H. Wang, D. Butler, M.-S. To, J. Avery, M. L. Hull, G. Carneiro, Distilling missing\nmodality knowledge from ultrasound for endometriosis diagnosis with magnetic resonance\nimages, in: 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), IEEE,\n2023. doi:10.1109/isbi53787.2023.10230667.\nURL http://dx.doi.org/10.1109/ISBI53787.2023.10230667\n[12] H. W. Goh, U. Tkachenko, J. Mueller, C. C. Cleanlab, Crowdlab: Supervised learning to\ninfer consensus labels and quality scores for data with multiple annotators, arXiv preprint\narXiv:2210.06812 (2022).\n[13] B. Wilder, E. Horvitz, E. Kamar, Learning to complement humans, in: Proceedings of the Twenty-\nNinth International Joint Conference on Artificial Intelligence, IJCAI’20, 2021.\n[14] Z. Lu, M. Yin, Human reliance on machine learning models when performance feedback is\nlimited: Heuristics and risks, in: Proceedings of the 2021 CHI Conference on Human Factors in\nComputing Systems, 2021, pp. 1–16.\n[15] K. Weitz, D. Schiller, R. Schlagowski, T. Huber, E. Andr´ e, ” do you trust me?” increasing user-\ntrust by integrating virtual agents in explainable ai interaction design, in: Proceedings of the\n19th ACM International Conference on Intelligent Virtual Agents, 2019, pp. 7–9.\n[16] A. Rosenfeld, M. D. Solbach, J. K. Tsotsos, Totally looks like-how humans compare, compared to\nmachines, 2018, pp. 1961–1964.\n[17] T. Serre, Deep learning: the good, the bad, and the ugly, Annual review of vision science 5 (2019)\n399–426.\n[18] E. Kamar, S. Hacker, E. Horvitz, Combining human and machine intelligence in large-scale\ncrowdsourcing., in: AAMAS, Vol. 12, 2012, pp. 467–474.\n[19] G. Bansal, B. Nushi, E. Kamar, E. Horvitz, D. S. Weld, Is the most accurate ai the best teammate?\noptimizing ai for teamwork, in: AAAI, Vol. 35, 2021, pp. 11405–11414.\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 18\n[20] K. Vodrahalli, R. Daneshjou, T. Gerstenberg, J. Zou, Do humans trust advice more if it comes from\nai? an analysis of human-ai interactions, in: Proceedings of the 2022 AAAI/ACM Conference\non AI, Ethics, and Society, 2022, pp. 763–777.\n[21] X. Wu, L. Xiao, Y. Sun, J. Zhang, T. Ma, L. He, A survey of human-in-the-loop for machine\nlearning, Future Gener. Comput. Syst. 135 (C) (2022) 364–381. doi:10.1016/j.future.2022.\n05.014.\nURL https://doi.org/10.1016/j.future.2022.05.014\n[22] M. F. Pradier, J. Zazo, S. Parbhoo, R. H. Perlis, M. Zazzi, F. Doshi-Velez, Preferential mixture-of-\nexperts: Interpretable models that rely on human expertise as much as possible, AMIA Summits\non Translational Science Proceedings 2021 (2021) 525.\n[23] C. Cortes, G. DeSalvo, M. Mohri, Learning with rejection, in: Algorithmic Learning Theory: 27th\nInternational Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, Springer,\n2016, pp. 67–82.\n[24] D. Madras, T. Pitassi, R. Zemel, Predict responsibly: improving fairness and accuracy by learning\nto defer, NeurIPS 31 (2018).\n[25] H. Narasimhan, W. Jitkrittum, A. K. Menon, A. Rawat, S. Kumar, Post-hoc estimators for learning\nto defer to an expert, NeurIPS 35 (2022) 29292–29304.\n[26] M. Raghu, K. Blumer, G. Corrado, J. Kleinberg, Z. Obermeyer, S. Mullainathan, The algorithmic\nautomation problem: Prediction, triage, and human effort, arXiv preprint arXiv:1903.12220\n(2019).\n[27] N. Okati, A. De, M. Rodriguez, Differentiable learning under triage, NeurIPS 34 (2021) 9140–9151.\n[28] R. Verma, D. Barrej´ on, E. Nalisnick, On the calibration of learning to defer to multiple experts, in:\nWorkshop on Human-Machine Collaboration and Teaming in International Confere of Machine\nLearning, 2022.\n[29] A. Mao, C. Mohri, M. Mohri, Y. Zhong, Two-stage learning to defer with multiple experts, in:\nNeurIPS, 2023.\n[30] P. Hemmer, S. Schellhammer, M. V¨ ossing, J. Jakubik, G. Satzger, Forming effective human-ai\nteams: Building machine learning models that complement the capabilities of multiple experts,\nInternational Joint Conferences on Artificial Intelligence Organization, 2022, pp. 2478–2484,\nmain Track. doi:10.24963/ijcai.2022/344.\nURL https://doi.org/10.24963/ijcai.2022/344\n[31] M. Steyvers, H. Tejeda, G. Kerrigan, P. Smyth, Bayesian modeling of human–ai complementarity,\nProceedings of the National Academy of Sciences 119 (11) (2022) e2111547119.\n[32] G. Kerrigan, P. Smyth, M. Steyvers, Combining human predictions with model probabilities via\nconfusion matrices and calibration, NeurIPS 34 (2021) 4421–4434.\n[33] Z. Zhang, K. Wells, G. Carneiro, Learning to complement with multiple humans (lecomh):\nIntegrating multi-rater and noisy-label learning into human-ai collaboration, arXiv preprint\narXiv:2311.13172 (2023).\n[34] M. Liu, J. Wei, Y. Liu, J. Davis, Do humans and machines have the same eyes? human-machine\nperceptual differences on image classification, arXiv preprint arXiv:2304.08733 (2023).\n[35] Q. Dou, Q. Liu, P. A. Heng, B. Glocker, Unpaired multi-modal segmentation via knowledge\ndistillation, in: IEEE Transactions on Medical Imaging, 2020.\n[36] M. Monteiro, L. Le Folgoc, D. Coelho de Castro, N. Pawlowski, B. Marques, K. Kamnitsas,\nM. van der Wilk, B. Glocker, Stochastic segmentation networks: Modelling spatially correlated\naleatoric uncertainty, Advances in Neural Information Processing Systems 33 (2020) 12756–\n12767.\n[37] Z. Han, C. Zhang, H. Fu, J. T. Zhou, Trusted multi-view classification, arXiv preprint\narXiv:2102.02051 (2021).\n[38] H. Wang, J. Zhang, Y. Chen, C. Ma, J. Avery, L. Hull, G. Carneiro, Uncertainty-aware multi-modal\nlearning via cross-modal random network prediction, in: European Conference on Computer\nVision, Springer, 2022, pp. 200–217.\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 19\n[39] H. Wang, C. Ma, Y. Liu, Y. Chen, Y. Tian, J. Avery, L. Hull, G. Carneiro, Enhancing multi-modal\nlearning: Meta-learned cross-modal knowledge distillation for handling missing modalities, arXiv\npreprint arXiv:2405.07155 (2024).\n[40] H. Wang, C. Ma, J. Zhang, Y. Zhang, J. Avery, L. Hull, G. Carneiro, Learnable cross-\nmodal knowledge distillation for multi-modal learning with missing modality, in: International\nConference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2023,\npp. 216–226.\n[41] H. Wang, Y. Chen, C. Ma, J. Avery, L. Hull, G. Carneiro, Multi-modal learning with missing\nmodality via shared-specific feature modelling, in: Proceedings of the IEEE/CVF Conference\non Computer Vision and Pattern Recognition, 2023, pp. 15878–15887.\n[42] Y. Wang, W. Huang, F. Sun, T. Xu, Y. Rong, J. Huang, Deep multimodal fusion by channel\nexchanging, Advances in Neural Information Processing Systems 33 (2020) 4835–4845.\n[43] M. Patrick, Y. M. Asano, P. Kuznetsova, R. Fong, J. F. Henriques, G. Zweig, A. Vedaldi, Multi-\nmodal self-supervision from generalized data transformations, arXiv preprint arXiv:2003.04298\n(2020).\n[44] M. Patrick, P.-Y. Huang, I. Misra, F. Metze, A. Vedaldi, Y. M. Asano, J. F. Henriques, Space-\ntime crop & attend: Improving cross-modal video representation learning, in: Proceedings of\nthe IEEE/CVF International Conference on Computer Vision, 2021, pp. 10560–10572.\n[45] H. Chen, W. Xie, T. Afouras, A. Nagrani, A. Vedaldi, A. Zisserman, Localizing visual sounds\nthe hard way, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern\nRecognition, 2021, pp. 16867–16876.\n[46] X. Jia, X.-Y. Jing, X. Zhu, S. Chen, B. Du, Z. Cai, Z. He, D. Yue, Semi-supervised multi-view\ndeep discriminant representation learning, IEEE transactions on pattern analysis and machine\nintelligence 43 (7) (2020) 2496–2509.\n[47] H.-Y. Lee, H.-Y. Tseng, J.-B. Huang, M. Singh, M.-H. Yang, Diverse image-to-image translation\nvia disentangled representations, in: Proceedings of the European conference on computer vision\n(ECCV), 2018, pp. 35–51.\n[48] X. Liu, P. Sanchez, S. Thermos, A. Q. O’Neil, S. A. Tsaftaris, Learning disentangled\nrepresentations in the imaging domain, Medical Image Analysis (2022) 102516.\n[49] Z.-H. Zhou, Ensemble methods: foundations and algorithms, CRC press, 2012.\n[50] F. Rodrigues, F. Pereira, B. Ribeiro, Gaussian process classification and active learning with\nmultiple annotators, in: ICML, PMLR, 2014, pp. 433–441.\n[51] V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, L. Moy, Learning from\ncrowds., Journal of machine learning research 11 (4) (2010).\n[52] F. Rodrigues, F. Pereira, Deep learning from crowds, Vol. 32, 2018.\n[53] Z. Chen, H. Wang, H. Sun, P. Chen, T. Han, X. Liu, J. Yang, Structured probabilistic end-to-end\nlearning from crowds, 2021, pp. 1512–1518.\n[54] M. Guan, V. Gulshan, A. Dai, G. Hinton, Who said what: Modeling individual labelers improves\nclassification, in: AAAI, Vol. 32, 2018.\n[55] H. W. Goh, U. Tkachenko, J. Mueller, Crowdlab: Supervised learning to infer consensus labels\nand quality scores for data with multiple annotators (2023). arXiv:2210.06812.\n[56] K. He, X. Chen, S. Xie, Y. Li, P. Doll´ ar, R. Girshick, Masked autoencoders are scalable\nvision learners, in: Proceedings of the IEEE/CVF conference on computer vision and pattern\nrecognition, 2022, pp. 16000–16009.\n[57] A. Dosovitskiy, et al., An image is worth 16x16 words: Transformers for image recognition at scale,\narXiv preprint arXiv:2010.11929 (2020).\n[58] K. Hara, H. Kataoka, Y. Satoh, Learning spatio-temporal features with 3d residual networks for\naction recognition, in: Proceedings of the IEEE international conference on computer vision\nworkshops, 2017, pp. 3154–3160.\n[59] I. Loshchilov, F. Hutter, Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint\narXiv:1608.03983 (2016).\n\nHuman-AI Collaborative Multi-modal Multi-rater Learning 20\n[60] C. Feng, G. Tzimiropoulos, I. Patras, SSR: an efficient and robust framework for learning with\nunknown label noise, in: 33rd British Machine Vision Conference 2022, BMVC 2022, London,\nUK, November 21-24, 2022, BMVA Press, 2022, p. 372.\nURL https://bmvc2022.mpi-inf.mpg.de/372/\n[61] R. Xiao, Y. Dong, H. Wang, L. Feng, R. Wu, G. Chen, J. Zhao, Promix: combating label noise\nvia maximizing clean sample utility, in: Proceedings of the Thirty-Second International Joint\nConference on Artificial Intelligence, IJCAI-23, E. Elkind, Ed. International Joint Conferences\non Artificial Intelligence Organization, Vol. 8, 2023, pp. 4442–4450.","source_license":"CC0","license_restricted":false}