RDCorpus: labeled medical records for the timely detection of rare diseases

RDCorpus: labeled medical records for the timely detection of rare diseases | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article RDCorpus: labeled medical records for the timely detection of rare diseases Matias Rolando, Victor Raggio, Hugo Naya, Lucia Spangenberg, Leticia Cagnina This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4795232/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 26 Feb, 2025 Read the published version in Scientific Reports → Version 1 posted 11 You are reading this latest preprint version Abstract Rare diseases (RD) are a group of pathologies that individually affect less than 1 in 2000 people but collectively impact around 7% of the world's population. Most of them affect children, are chronic and progressive, and have no specific treatment. RD patients face diagnostic challenges, with an average diagnosis time of 5 years, multiple specialist visits, and invasive procedures. This ‘diagnostic odyssey’ can be detrimental to their health. Machine learning (ML) has the potential to improve healthcare by providing more personalized and accurate patient management, diagnoses, and in some cases, treatments. Leveraging the MIMIC-III database and additional medical notes from different sources such as in-house data, PubMed and chatGPT, we propose a labeled dataset for early RD detection in hospital settings. Applying various supervised ML methods, including logistic regression, decision trees, support vector machine (SVM), deep learning methods (LSTM and CNN), and Transformers (BERT), we validated the use of the proposed resource, achieving 92.7% F-measure and a 96% AUC using SVM. These findings highlight the potential of ML in redirecting RD patients towards more accurate diagnostic pathways and presents a corpus that can be used for future development and refinements. Biological sciences/Genetics/Genome Biological sciences/Genetics Health sciences/Health care Health sciences/Medical research/Genetics research Physical sciences/Engineering/Biomedical engineering Figures Figure 1 Figure 2 Figure 3 1 Introduction Rare diseases (RD) are pathologies that affect less than 1 in 2000 people 1 . Although separately each disease is very rare, taken together, there are around 7000 different disorders, they involve a large number of patients (7% of the world population) 2 . Most of them affect children, have a high impact on quality of life and life expectancy, are generally chronic and progressive, and most of them have no specific treatment. Moreover, by their infrequency nature, they are a diagnostic challenge. On average, it takes five years from the onset of symptoms to diagnosis, and it takes a mean of about seven visits to different specialists and dozens of studies, some of them invasive or requiring general anesthesia 3 ; this is generally referred to as the “diagnostic odyssey”. According to a recent report by Globalgenes ( https://globalgenes.org/rare-facts ) 40% of general practitioners and 24% of specialists state that they do not have the time to work on these diagnoses, which contributes to the difficulty of a time-efficient diagnostic process. It is important to obtain a time-efficient diagnosis to avoid a detrimental disease progression. RD patients are extremely vulnerable and neglected by the healthcare system since there is usually no global, nationwide strategy to tackle and fund this problem efficiently. Genomic approaches such as whole exome or genome sequencing have improved the diagnosis rate in RD patients 4,5 . Depending on the existing pathology and method used, the diagnosis rate can vary from 30 to 50% 6,7,8 . It is of imperative importance to include these types of tests at the right moment in the diagnostic algorithm. Machine learning (ML) is a powerful tool in several fields of healthcare, allowing for the development of more personalized and accurate patient management, diagnoses and treatments 9 , 10,11 . The application of ML methods has the potential to improve healthcare and create more efficient, reliable, and cost-effective treatments 12 , 13,14 . The advent of large scale language models Transformer-based such as BERT 15 , GPT-3 15,16 or T5 17 trained on a massive amount of text data has shown to outperform a wide range of natural language processing tasks including text summarization, sentiment analysis, question answering and language translation. The efforts of ML researchers have shifted towards the fine-tuning of freely available versions of these models with relatively small specialized datasets (few-shot learning). Hence, the importance of generating such smaller corpora related to particular fields of interest and making them available to the community. In this study we aim to propose a resource to contribute to the research in the detection of RD patients in an early stage in their passage through a hospital. For this, we have made use of the large, freely-available, MIMIC-III database 18 of de-identified health-related data of over forty thousand patients who stayed in critical care units at the Beth Israel Deaconess Medical Center. MIMIC-III includes information of patients such as test results, medications, mortality and discharge summaries. Besides, we have collected medical notes from other sources such as case reports from Pubmed, our own records and diagnostics automatically generated by chatGPT. We explored a plethora of methods applied to the proposed corpus, including classical supervised machine learning (SVM), deep learning (LSTM and CNN), and Transformers (BERT). Using SVM, we identified RD patients with 92.7% of F-measure and an AUC of 96%. This information could be used to redirect them to a more accurate diagnostic algorithm. 2 Data and Methods 2.1 Proposed corpus With the aim to get diversity, we used discharge summaries from MIMIC-III, some medical education notes from the School of Medicine in Uruguay, a few diagnosis reports from the URUGENOMES project (urugenomes.org), some clinical records scrapped from PubMed and a few clinical records “created” by chatGPT. MIMIC-III database has been used in previous studies related to the classification of diseases 18 and constitutes a valuable resource for research. This large and freely-available corpus has more than 2000K de-identified data related to 46K patients of critical care units of the Beth Israel Deaconess Medical Center (Boston, Massachusetts) between the years 2001 and 2012. The information contained is about demographics, vital sign measurements made at the bedside, test results of laboratories, procedures performed, medications, caregiver notes, imaging reports, data of mortality and discharge summaries. For our study, we used the discharge summaries of patients because these have interesting information about the diagnosis that doctors arrived at during their stay at the hospital. As we need to label some clinical notes as RD for consideration in our proposed dataset, we considered a published article 19 in which the authors found RD in a subset of discharge summaries taken from MIMIC-III. They used a two-steps approach. First, tokens appearing in the text are linked to medical concepts of the Unified Medical Language System (UMLS) with the SemEHR tool. The results were refined using particular rules for removing abbreviations and text-UMLS pairs with low mention frequencies in the clinical notes. Contextual representations of the pairs were obtained with BlueBERT and used to train a Logistic Regressor for a binary classification to confirm clinic mentions. Secondly, the UMLS concepts were linked to Orphanet Rare Disease Ontology (ORDO) and thus, the authors created a gold standard dataset with 1073 mentions: 146 of rare and 927 of common diseases. We searched for the discharge summary notes corresponding to these mentions and thus, obtained 65 notes containing clues of RD and the remaining 247, common diseases. We also randomly selected additional 100 clinical notes of MIMIC-III corresponding to the admission stage of patients. This is to introduce variability to the medical texts besides to augment the number of common diseases. We checked that those diagnoses were not labeled as rare (according to 19 ). Finally, our proposed dataset has 412 records containing medical notes from MIMIC-III. We also used medical education summaries collected from Oficina del Libro of the Medicine School at Universidad de la República (Uruguay) 20 which contain diagnostics related to cardiology, hematology, neurology and internal medicine. After a rigorous review performed by an expert (medical geneticist), we labeled the 98 notes in common (64) and rare (34) diseases. Those clinical texts were translated from spanish to english. In order to balance both classes of diagnosis, we included 32 RDs obtained from clinical records of a previous project, URUGENOMES (urugenomes.org) 4,21 and 277 from PubMed. For the last, we scrapped the PubMed platform ( https://pubmed.ncbi.nlm.nih.gov/ ) searching for case reports of RD in free articles. As obtaining medical notes on rare diseases is challenging, we generated some ones using chatGPT ( https://chat.openai.com ). Thus, we included 10 RD clinical records and 13 common diseases generated by the tool and these were manually curated by a clinical geneticist, adding 23 diagnostics to the corpus. Our complete corpus finally has 842 clinical records, 418 rare and 424 common. It is available under: https://sites.google.com/view/leticia-cagnina/research and can be free-used for research issues performing the corresponding citation. No preprocessing is done to preserve the data’s genuine characteristics. The records collected constitute a balanced corpus which is more reliable to work with. Table 1 shows the main characteristics of the dataset. The vocabulary is large, with more than half of the words unique. This could be expected since the texts are related to specific medical issues (names of medications, lab tests, symptoms, diseases, abbreviations used by doctors, etc.). The diagnoses are written in 4 sentences on average although the variability of lengths is high: most clinical notes have only one sentence (possibly the diagnosis is written like a paragraph), just one with 340 (short sentences) and the rest with values oscillating the 2 and 70 sentences. Table 1 Dataset statistics. diagnosis vocabulary words Word average sentences Sentence average Sentence min Sentence max corpus 842 41765 787277 935 3576 4 1 340 common 424 29671 474648 1119 2753 6 1 340 rare 418 21686 312629 747 823 1 1 39 Table 1 (two bottom rows) shows the characteristics of the corpus separated by classes. As mentioned this corpus is balanced so it has a similar amount of records of RD and non-RD (418 vs 424). Although the size of vocabulary used is not very different, it seems that clinical notes of common diseases are much longer than ones for RD (474K vs 312k). The same occurs with the number of sentences: RD are written using one third of those included in common diagnostics. In fact, most of the notes of RD use two sentences while those of common are slightly more extensive (6 on average). Only one record has a maximum number of sentences of 340 which corresponds to a common disease but the rest of this class are around 1 and 70. Moreover, the RD diagnosis has between 1 and 39 sentences as maximum. We also analyzed the contents of the clinical notes by using word clouds, which show the frequency of the words in each class. Most used words in common diseases notes are regular medical terms such as blood, pain and patient (Fig. 1 A). Also some words related to non specific treatment: capsule, daily and hours. Interestingly the word cloud corresponding to RD notes (Fig. 1 B) shows advanced studies and serious issues found in words such as tomography, examination, tumor and mass. Top 50 more representative words in the whole proposed dataset are in Supplementary Figure S1 . Figure 1 C shows the proportion of medical notes collected from the different sources. Note that no requirement was stated in the time to collect the record and thus, the diversity of the distribution of words (log-scaled) is high. The distribution corresponding to RD is approximately normal which is suitable for classification models (Fig. 1 D). However, some outliers above 3500 words can be observed for this class. The kernel density estimation is shown with a narrow kurtosis (leptokurtic) and central tendency around 500. The distribution for the class of common diseases seems to be bimodal although the central tendency is around 100 without major outliers (Fig. 1 D). 2.2 Machine Learning models We considered TF-IDF (Term Frequency Inverse Document Frequency) weighting and boolean schemes. We applied either L2 or L1 normalization, we limited the number of features considered (5000 or 10000) and considered either words or trigrams as features. After vectorization, several classic classification models were tested: Support Vector Machines (SVM), Logistic Regression (LR) and Decision Trees (DT). For each baseline model we performed 30 independent runs with different dataset partitions considering 80% for training and 20% to test. The best results were obtained with the following hyperparameters: LR used Solver Newton-CG, TF-IDF norm L2 representation and max features 10000; SVM included a sigmoid kernel, TFIDF norm L1 representation; maximum depth of DT was 23 and considered TFIDF norm L2 representation. We also include models based on deep neural networks such as the Long-Short Term Memory (LSTM) 22 and Convolutional Neural Network (CNN) 23 , model architectures are shown in Fig. 2 . The input of these models are static dense representations of words, that is, embeddings. After several experiments, we decided to use a combination of our own pre-trained vectors (obtained from signs and symptoms of rare diseases) and others specific for the task to solve in our study (obtained from https://github.com/yao8839836/obesity ). We used two embedding layers with the pre-trained word vectors which are concatenated to obtain a 400 dimensional higher-level representation of each input text. After processing the input with the specific architecture of each model, the output is obtained as the result of a fully connected softmax layer to perform the classification using the probability distribution over the output labels (RD vs. common). The LSTM includes a layer with 64 units (Fig. 2 A) while the CNN only 4 1D-convolutional layers for extracting 70 filters with different sizes of kernels (varying between 1 and 4) (Fig. 2 B). After applying max pooling operation to each feature map, the outputs are concatenated, flatted and passed to a dense 64-unit layer. Previous to the output layer, a dropout operation is performed to reduce overfitting (rate 0.2). The only change introduced to the previous architecture to construct the CNN + LSTM model, is the inclusion of a LSTM layer between the concatenation of the 1D-convolutional and the dense layer (Fig. 2 C). The number of units in the dense layer and filters is lower (here 50). Finally, the architecture of the LSTM + CNN is obtained by embedding the CNN between the LSTM and the output layer (Fig. 2 D). We removed the 64-unit dense layer and the dropout operation to simplify the ended model. We also reduced the size of the filters and the units in the LSTM (10 and 16 respectively). For each deep neural network architecture the input layer is the text to classify. All architectures are shown in Fig. 2 A-D).These architectures were obtained as the best after testing several models with different configurations of hyperparameters. Large language models are pre-trained using large amounts of data (in an unsupervised way) and usually fine-tuned (in a supervised way) with specific data depending on the task to solve. An example of such models is the Bidirectional Encoder Representations from Transformers 15 (BERT). Unlike the models we proposed before, BERT uses contextualized word representation of the input which feeds several stackedTransformer encoders. We select BERT for our experiments because, beyond its good performance in various NLP tasks, there are several models pre-trained with biomedical texts. We compare a base version of BERT with Bio_ClinicalBERT. The latter is a fine-tuned (with medical conditions data) version of the first domain-specific BERT based model pre-trained on large scale biomedical corpora, named BioBert 24 . The model is available in sid321axn/Bio_ClinicalBERT-finetuned-medicalcondition · Hugging Face . BERT was pre-trained on English Wikipedia and General BooksCorpus while Bio_ClinicalBERT, besides the same as BERT, was pre-trained with PubMed Abstracts and PMC Full-text articles (that is, biomedical domain-specific texts) and fine-tuned with clinical conditions of diseases. 3 Results We analyze the results of the different models implemented and show the one that obtained the best performance in detecting RD from medical notes. Then, we perform an error analysis on the systematically misclassified records to highlight some characteristics about the proposed corpus. 3.1 The SVM better predicts RD patients on discharge summary texts Table 2 summarizes the results of applying all models and considering the metrics Accuracy (useful when the problem has balanced classes), the standard F-measure and Area Under Curve Receiver Operator Characteristic (AUC) for complementing the evaluation of the performance. First rows in Table 2 shows for each one of the baseline models the results obtained with the best configuration. The results suggest that SVM outperforms the models obtained with the baseline classifiers but also the advanced ones (LSTM, CNN and BERT) when F-measure is considered (0.927). Regarding the results of deep models (Table 2 , rows 4–7), CNN and LSTM perform similarly with a small difference between metrics: CNN is better than LSTM in F-measure (0.906 and 0.898, respectively) but LSTM is better than CNN in accuracy and AUC (0.899 and 0.893 in accuracy, respectively). It is interesting to note that LSTM outperforms the combination of LSTM and CNN models which would indicate that LSTM can learn long-term dependencies from sequences of higher-level representations without help of convolutional operations. Figure 3 A shows the confusion matrix of the best model (SVM). The class 0 corresponds to common disease and class 1 to RD. Only one common disease was misclassified and 11 RD were wrongly classified as common. Figure 3 B the ROC curve obtained from the data. The last 2 rows of Table 2 show the results obtained with the Transformer-based models. BioBert obtained better performance than BERT considering F-measure (0.873 vs 0.852) and the same behavior with the other metrics. The reason is probably because BioBERT was trained with biomedical data. BioBert performs quite similarly to all models, demonstrating that for this particular task, the complexity of transformed-based models is not synonymous with better results. We chose the AUC metric in model analysis as it provides valuable insights into the model's ability to distinguish between classes and its impact on specific errors like false positives and false negatives, as demonstrated in previous studies 25 . In our case, higher AUC means how better the model is at distinguishing between patients with a RD and not. LSTM, CNN and SVM classifiers obtained the highest AUC (0.95, 0.94 and 0.96, respectively) indicating their ability to correctly classify diagnosis with RD with relatively small models (in comparison to the transformer-based ones). Table 2 Best results obtained with each method for the proposed corpus. Accuracy F-measure AUC LR 0.923 0.922 0.925 DT 0.888 0.893 0.887 SVM 0.929 0.927 0.960 CNN 0.893 0.906 0.944 LSTM 0.899 0.898 0.952 CNN + LSTM 0.893 0.877 0.922 LSTM + CNN 0.888 0.887 0.920 BioBERT 0.882 0.873 0.881 BERT 0.858 0.852 0.858 3.2. Systematically misclassified clinical records The most frequent error of our model is that RDs are misclassified as common diseases. In almost all models (baseline and not) a median of ~ 87.25% of all misclassified records correspond to RD wrongly classified as common. Most frequently misclassified clinical records are in Table S1 (several tabs corresponding to each model). A median of 80.95% of misclassified RD corresponds to MIMIC-III clinical records that were previously classified by another study 19 . The rest (mostly) correspond to common diseases that were translated from Spanish. Focusing on the SVM classification, all misclassified records ( 11 ) were RD (RD classified as common). After careful consideration of an expert we found that most of them were actually common diseases, but mislabeled as RD in the corpus since the clinical records describe a large amount of complications of common diseases probably in elderly patients. Hence, the clinical text becomes very long, complicated with several interactions with procedures, medical specialties staff, drugs, interventions, and so on (Table S1 , tab SVM with expert comments on each clinical text), which might be an explanation of the misclassification. Even though the corpus might have some noise regarding the labels (which is a realistic scenario in the context of several applications) the classifiers are able to perform fairly well in practice. 4 Discussion RDs are difficult to detect, to diagnose and to treat. Patients with RD have to navigate the healthcare system patiently, inefficiently and with economical (and psychological) costs. Timely diagnosis, hence early strategies to assess the disease, might be of great importance to control the impact on the patient and the family. The diagnosis pipeline of RD is different from other diseases and very frequently includes consultation with geneticists and molecular studies for proper diagnosis. An early detection of the presence of a RD might be a substantial improvement in many cases. Our method aims to timely detect RD patients from medical records (discharge summaries) obtained from many sources, specially from an emergency unit. When the discharge summary of a patient classifies as a potential RD a flag could be raised in the hospital system and measurements could be set in place, such as consultation with geneticist and other specialists, molecular analysis, improving the time until diagnosis. An accuracy around 90% implies that in 90% of the cases the classification is correct, and the RD flag should be raised. The remaining 10% of cases correspond to individuals that have a common disease but they were classified as RD. The impact of such an error would be mostly an economical loss for a public hospital, since additional unnecessary consultations and/or laboratory examination might be done, however, the savings generated by the remaining 90% most likely outweigh the cost of these additional consultations. Besides, upon manual reexamination most of these false calls would be easily detected. On the other hand, an error misclassifying RD as common would have a higher impact on patients well-being (patients would go through the standard algorithmic path, hence probably a diagnostic odyssey) and costs would be even bigger. Future work relies on the fine-tuning of models that are already close to the clinical aspects of the problem, such as Bio_ClinicalBERT (freely available), using our corpus. This might improve our results by better understanding the technical words and their contexts. Also, the tested models are not strong in the explicativeness. In some models, we are not able to understand why a specific clinical record or discharge summary is classified as RD. Understanding the results of the classification process would improve our knowledge on RD in general, and also, how to write discharge summaries so that models would work properly. Additionally, we intend to expand the corpus. The inclusion of more reliable RD clinical records and manual curation of those already included, are going to improve downstream results. Finally, we believe that this valuable corpus is in line with the trend of few-shot learning for classification above all in the biomedical domain and we would try other transformer-based methods for few-shot identification of rare diseases. 5 Conclusions We presented a corpus for the classification of rare diseases from clinical notes. We showed a detailed exploratory analysis of the data collected and concluded with a balanced dataset with a similar number of notes labeled as containing RD or not. To test the proposed resource, we performed a comparative study of different models for the classification of rare diseases, the classical SVM, the artificial neural networks LSTM and CNN and, the recent transformer-based BERT. SVM performs the best with a F-measure of 0.927. Thus, we conclude that the SVM-based model is able to accurately predict rare diseases based on the clinical record of the patients, hence enabling the possibility to be included as a warning and a lead to a more accurate diagnostic path. By making the corpus available we encourage future applications to be developed and refined. In addition to helping mitigate the lack of annotated data for the identification of RD, this resource can be safely used for few-shot machine learning algorithms in classification as well as other tasks. Declarations Data availability statement The corpus is available under https://sites.google.com/view/leticia-cagnina/research Acknowledgements This study was partially funded by BID (Banco Iberomericano de desarrollo) in the context of the URUGENOMES Project (Proyecto ATN/KK-L4584-“Fortalecimiento de las capacidades técnicas y humanas para las exportaciones de servicios genómicos”). Additionally, support was obtained from the CONICET, Short Research Stages program given to Leticia Cagnina. PEDECIBA under Grant Number: Alicuotas INNOVA II under Grant Number: DCI‑ALA /2011/23‑502. Author contributions MR: Data curation, formal analysis, Roles/Writing - original draft VR: Data curation, Writing - review & editing HN: Validation, Methodology, Writing - review & editing LC: Supervision, Funding acquisition, Roles/Writing - original draft, Methodology LS: Supervision, Roles/Writing - original draft, Project administration, Data curation References The Voice of 12,000 Patients. Experiences and Expectations of Rare Disease Patients on Diagnosis and Care in Europe. (EURORDIS - Rare Diseases Eu, 2009). Sireau, N. Rare Diseases: Challenges and Opportunities for Social Entrepreneurs. (Routledge, 2017). Yan, X., He, S. & Dong, D. Determining How Far an Adult Rare Disease Patient Needs to Travel for a Definitive Diagnosis: A Cross-Sectional Examination of the 2018 National Rare Disease Survey in China. Int. J. Environ. Res. Public Health 17, (2020). Raggio, V. et al. Whole genome sequencing reveals a frameshift mutation and a large deletion in YY1AP1 in a girl with a panvascular artery disease. Hum. Genomics 15, 28 (2021). Meyer, E. J. et al. CBG Montevideo: A Clinically Novel Mutation Leading to Haploinsufficiency of Corticosteroid-binding Globulin. J Endocr Soc 5, bvab115 (2021). Della Mina, E. et al. Improving molecular diagnosis in epilepsy by a dedicated high-throughput sequencing platform. Eur. J. Hum. Genet. 23, 354–362 (2015). Liu, H.-Y. et al. Diagnostic and clinical utility of whole genome sequencing in a cohort of undiagnosed Chinese families with rare diseases. Sci. Rep. 9, 19365 (2019). Clark, M. M. et al. Meta-analysis of the diagnostic and clinical utility of genome and exome sequencing and chromosomal microarray in children with suspected genetic diseases. NPJ Genom Med 3, 16 (2018). Esteva, A. et al. A guide to deep learning in healthcare. Nat. Med. 25, 24–29 (2019). Hwang, S. & Lee, B. Machine learning-based prediction of critical illness in children visiting the emergency department. PLoS One 17, e0264184 (2022). Hatachi, T. et al. Machine Learning-Based Prediction of Hospital Admission Among Children in an Emergency Care Center. Pediatr. Emerg. Care 39, 80–86 (2023). Gulshan, V. et al. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA 316, 2402–2410 (2016). Golden, J. A. Deep Learning Algorithms for Detection of Lymph Node Metastases From Breast Cancer: Helping Artificial Intelligence Be Seen. JAMA: the journal of the American Medical Association vol. 318 2184–2186 (2017). Doshi-Velez, F., Ge, Y. & Kohane, I. Comorbidity clusters in autism spectrum disorders: an electronic health record time-series analysis. Pediatrics 133, e54–63 (2014). Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4171–4186. om B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen,Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. (2020). Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 140, (2020). Johnson, A. E. W. et al. MIMIC-III, a freely accessible critical care database. Sci Data 3, 160035 (2016). Dong, H. et al. Rare Disease Identification from Clinical Notes with Ontologies and Weak Supervision. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2021, 2294–2298 (2021). Fernando López Bello, Hugo Naya, Víctor Raggio, Aiala Rosá. From medical records to research papers: A literature analysis pipeline for supporting medical genomic diagnosis processes. Informatics in Medicine Unlocked 15, 100181 (2019). Spangenberg, L. et al. Novel frameshift mutation in PURA gene causes severe encephalopathy of unclear cause. Mol Genet Genomic Med 9, e1622 (2021). Sepp Hochreiter, J. S. Long short-term memory. Neural Comput. 9, 1735–1780. LeCun et al, Y. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Comput. 1, 541–551. Lee, J. et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 1234–1240 (2020). Fawcett, P. F. Robust Classification for Imprecise Environments. 42, 203–231. Additional Declarations No competing interests reported. Supplementary Files FigureS1corpuswc.png TableS1wrongpredictionsTREEonly11647VR.xlsx Cite Share Download PDF Status: Published Journal Publication published 26 Feb, 2025 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 16 Jan, 2025 Reviews received at journal 12 Jan, 2025 Reviewers agreed at journal 06 Jan, 2025 Reviews received at journal 30 Aug, 2024 Reviewers agreed at journal 31 Jul, 2024 Reviewers agreed at journal 26 Jul, 2024 Reviewers invited by journal 26 Jul, 2024 Editor assigned by journal 26 Jul, 2024 Editor invited by journal 26 Jul, 2024 Submission checks completed at journal 26 Jul, 2024 First submitted to journal 24 Jul, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4795232","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":342762364,"identity":"96eb1fee-c12e-4748-bac3-4a23470ef040","order_by":0,"name":"Matias Rolando","email":"","orcid":"","institution":"Institut Pasteur de Montevideo","correspondingAuthor":false,"prefix":"","firstName":"Matias","middleName":"","lastName":"Rolando","suffix":""},{"id":342762365,"identity":"bb635bb1-ba24-4ce1-a46f-1b005e411770","order_by":1,"name":"Victor Raggio","email":"","orcid":"","institution":"Universidad de la República","correspondingAuthor":false,"prefix":"","firstName":"Victor","middleName":"","lastName":"Raggio","suffix":""},{"id":342762366,"identity":"51cd903a-0d4d-4a02-879f-17937f674da1","order_by":2,"name":"Hugo Naya","email":"","orcid":"","institution":"Institut Pasteur de Montevideo","correspondingAuthor":false,"prefix":"","firstName":"Hugo","middleName":"","lastName":"Naya","suffix":""},{"id":342762367,"identity":"036189ad-1ea9-4540-a622-0d633771624e","order_by":3,"name":"Lucia Spangenberg","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA10lEQVRIiWNgGAWjYBACPiA+wFCBEGBsIKSFDazlDJRFtBYGxjaStEgkPzzwc94defn5zcekKxhsZDcc4DGTwK8lzeBg77ZnhhuOsaVJnmFIMyZCSw7DAd5thxk3sPGYSTYwHE4kSsvBv3MO289v4/8G1PKfOC2HeRsOJzYc42EDajlAhBaeZwaHZY49S95wLM3YssEg2XjmYbZiC3xa+NmTH398U3PHdn7z4Yc3GyrsZPuON2+8gU8LFByA0gZAzMzAgtdhaFoggPkDEVpGwSgYBaNg5AAA9whJ8NOmpJoAAAAASUVORK5CYII=","orcid":"","institution":"Institut Pasteur de Montevideo","correspondingAuthor":true,"prefix":"","firstName":"Lucia","middleName":"","lastName":"Spangenberg","suffix":""},{"id":342762368,"identity":"cfdcfba6-7bf0-4053-ab37-dc754fc4e8c5","order_by":4,"name":"Leticia Cagnina","email":"","orcid":"","institution":"Universidad Nacional de San Luis","correspondingAuthor":false,"prefix":"","firstName":"Leticia","middleName":"","lastName":"Cagnina","suffix":""}],"badges":[],"createdAt":"2024-07-24 12:06:32","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4795232/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4795232/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-025-90450-0","type":"published","date":"2025-02-26T15:57:32+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":63419959,"identity":"c884dd51-f188-4137-a259-0e94b8d0c8db","added_by":"auto","created_at":"2024-08-28 02:34:05","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":361889,"visible":true,"origin":"","legend":"\u003cp\u003eCorpus description. \u003cstrong\u003eA.\u003c/strong\u003e Word cloud for the Common class. \u003cstrong\u003eB.\u003c/strong\u003e Word cloud for the Rare class. \u003cstrong\u003eC.\u003c/strong\u003e Number of records collected from different sources for the proposed dataset. \u003cstrong\u003eD.\u003c/strong\u003e Distribution of text length (in words) for each class.\u003c/p\u003e","description":"","filename":"image1.png","url":"https://assets-eu.researchsquare.com/files/rs-4795232/v1/70ee1fddfc83197e79b637c3.png"},{"id":63419963,"identity":"22a5c2a6-670d-4e8a-9bf9-5cd87f6395a3","added_by":"auto","created_at":"2024-08-28 02:34:05","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":319025,"visible":true,"origin":"","legend":"\u003cp\u003eModel architecture description. \u003cstrong\u003eA.\u003c/strong\u003e LSTM model. \u003cstrong\u003eB.\u003c/strong\u003e CNN model. \u003cstrong\u003eC.\u003c/strong\u003e CNN+LSTM model. \u003cstrong\u003eD\u003c/strong\u003e. LSTM+CNN model.\u003c/p\u003e","description":"","filename":"image2.png","url":"https://assets-eu.researchsquare.com/files/rs-4795232/v1/99c8167151e616064cd95f6f.png"},{"id":63419960,"identity":"ab945199-482a-4a72-857b-9594b4b9cecb","added_by":"auto","created_at":"2024-08-28 02:34:05","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":35899,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eSVM model metrics. A \u003c/strong\u003econfusion matrix. \u003cstrong\u003eB.\u003c/strong\u003e ROC curve.\u003c/p\u003e","description":"","filename":"image3.png","url":"https://assets-eu.researchsquare.com/files/rs-4795232/v1/69c97af0ac0f9882b602a2cb.png"},{"id":77622449,"identity":"838e4e4e-124d-42ed-be55-8c2b68df2aef","added_by":"auto","created_at":"2025-03-03 16:07:00","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1235723,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4795232/v1/c7d3c525-f5f0-4c67-88f4-35ce240973ba.pdf"},{"id":63419961,"identity":"0d9e6634-6a79-49c7-9ad9-90b889479dda","added_by":"auto","created_at":"2024-08-28 02:34:05","extension":"png","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":157176,"visible":true,"origin":"","legend":"","description":"","filename":"FigureS1corpuswc.png","url":"https://assets-eu.researchsquare.com/files/rs-4795232/v1/62080031cc14a9a4f9391273.png"},{"id":63420897,"identity":"1c666e78-8c4a-4e9b-9f51-6ef6980edc63","added_by":"auto","created_at":"2024-08-28 02:42:05","extension":"xlsx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":222866,"visible":true,"origin":"","legend":"","description":"","filename":"TableS1wrongpredictionsTREEonly11647VR.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-4795232/v1/2fcaa908e0234e516b9e5a37.xlsx"}],"financialInterests":"No competing interests reported.","formattedTitle":"RDCorpus: labeled medical records for the timely detection of rare diseases","fulltext":[{"header":"1 Introduction","content":"\u003cp\u003eRare diseases (RD) are pathologies that affect less than 1 in 2000 people\u003csup\u003e1\u003c/sup\u003e. Although separately each disease is very rare, taken together, there are around 7000 different disorders, they involve a large number of patients (7% of the world population)\u003csup\u003e2\u003c/sup\u003e. Most of them affect children, have a high impact on quality of life and life expectancy, are generally chronic and progressive, and most of them have no specific treatment. Moreover, by their infrequency nature, they are a diagnostic challenge. On average, it takes five years from the onset of symptoms to diagnosis, and it takes a mean of about seven visits to different specialists and dozens of studies, some of them invasive or requiring general anesthesia \u003csup\u003e3\u003c/sup\u003e; this is generally referred to as the \u0026ldquo;diagnostic odyssey\u0026rdquo;. According to a recent report by Globalgenes (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://globalgenes.org/rare-facts\u003c/span\u003e\u003cspan address=\"https://globalgenes.org/rare-facts\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e) 40% of general practitioners and 24% of specialists state that they do not have the time to work on these diagnoses, which contributes to the difficulty of a time-efficient diagnostic process. It is important to obtain a time-efficient diagnosis to avoid a detrimental disease progression.\u003c/p\u003e \u003cp\u003eRD patients are extremely vulnerable and neglected by the healthcare system since there is usually no global, nationwide strategy to tackle and fund this problem efficiently. Genomic approaches such as whole exome or genome sequencing have improved the diagnosis rate in RD patients\u003csup\u003e4,5\u003c/sup\u003e. Depending on the existing pathology and method used, the diagnosis rate can vary from 30 to 50%\u003csup\u003e6,7,8\u003c/sup\u003e. It is of imperative importance to include these types of tests at the right moment in the diagnostic algorithm.\u003c/p\u003e \u003cp\u003eMachine learning (ML) is a powerful tool in several fields of healthcare, allowing for the development of more personalized and accurate patient management, diagnoses and treatments\u003csup\u003e9\u003c/sup\u003e,\u003csup\u003e10,11\u003c/sup\u003e. The application of ML methods has the potential to improve healthcare and create more efficient, reliable, and cost-effective treatments\u003csup\u003e12\u003c/sup\u003e,\u003csup\u003e13,14\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eThe advent of large scale language models Transformer-based such as BERT\u003csup\u003e15\u003c/sup\u003e, GPT-3\u003csup\u003e15,16\u003c/sup\u003e or T5\u003csup\u003e17\u003c/sup\u003e trained on a massive amount of text data has shown to outperform a wide range of natural language processing tasks including text summarization, sentiment analysis, question answering and language translation. The efforts of ML researchers have shifted towards the fine-tuning of freely available versions of these models with relatively small specialized datasets (few-shot learning). Hence, the importance of generating such smaller corpora related to particular fields of interest and making them available to the community.\u003c/p\u003e \u003cp\u003eIn this study we aim to propose a resource to contribute to the research in the detection of RD patients in an early stage in their passage through a hospital. For this, we have made use of the large, freely-available, MIMIC-III database\u003csup\u003e18\u003c/sup\u003e of de-identified health-related data of over forty thousand patients who stayed in critical care units at the Beth Israel Deaconess Medical Center. MIMIC-III includes information of patients such as test results, medications, mortality and discharge summaries. Besides, we have collected medical notes from other sources such as case reports from Pubmed, our own records and diagnostics automatically generated by chatGPT.\u003c/p\u003e \u003cp\u003eWe explored a plethora of methods applied to the proposed corpus, including classical supervised machine learning (SVM), deep learning (LSTM and CNN), and Transformers (BERT). Using SVM, we identified RD patients with 92.7% of F-measure and an AUC of 96%. This information could be used to redirect them to a more accurate diagnostic algorithm.\u003c/p\u003e"},{"header":"2 Data and Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Proposed corpus\u003c/h2\u003e \u003cp\u003eWith the aim to get diversity, we used discharge summaries from MIMIC-III, some medical education notes from the School of Medicine in Uruguay, a few diagnosis reports from the URUGENOMES project (urugenomes.org), some clinical records scrapped from PubMed and a few clinical records \u0026ldquo;created\u0026rdquo; by chatGPT.\u003c/p\u003e \u003cp\u003eMIMIC-III database has been used in previous studies related to the classification of diseases\u003csup\u003e18\u003c/sup\u003e and constitutes a valuable resource for research. This large and freely-available corpus has more than 2000K de-identified data related to 46K patients of critical care units of the Beth Israel Deaconess Medical Center (Boston, Massachusetts) between the years 2001 and 2012. The information contained is about demographics, vital sign measurements made at the bedside, test results of laboratories, procedures performed, medications, caregiver notes, imaging reports, data of mortality and discharge summaries. For our study, we used the discharge summaries of patients because these have interesting information about the diagnosis that doctors arrived at during their stay at the hospital. As we need to label some clinical notes as RD for consideration in our proposed dataset, we considered a published article\u003csup\u003e19\u003c/sup\u003e in which the authors found RD in a subset of discharge summaries taken from MIMIC-III. They used a two-steps approach. First, tokens appearing in the text are linked to medical concepts of the Unified Medical Language System (UMLS) with the SemEHR tool. The results were refined using particular rules for removing abbreviations and text-UMLS pairs with low mention frequencies in the clinical notes. Contextual representations of the pairs were obtained with BlueBERT and used to train a Logistic Regressor for a binary classification to confirm clinic mentions. Secondly, the UMLS concepts were linked to Orphanet Rare Disease Ontology (ORDO) and thus, the authors created a gold standard dataset with 1073 mentions: 146 of rare and 927 of common diseases. We searched for the discharge summary notes corresponding to these mentions and thus, obtained 65 notes containing clues of RD and the remaining 247, common diseases. We also randomly selected additional 100 clinical notes of MIMIC-III corresponding to the admission stage of patients. This is to introduce variability to the medical texts besides to augment the number of common diseases. We checked that those diagnoses were not labeled as rare (according to\u003csup\u003e19\u003c/sup\u003e). Finally, our proposed dataset has 412 records containing medical notes from MIMIC-III.\u003c/p\u003e \u003cp\u003eWe also used medical education summaries collected from \u003cem\u003eOficina del Libro\u003c/em\u003e of the Medicine School at Universidad de la Rep\u0026uacute;blica (Uruguay)\u003csup\u003e20\u003c/sup\u003e which contain diagnostics related to cardiology, hematology, neurology and internal medicine. After a rigorous review performed by an expert (medical geneticist), we labeled the 98 notes in common (64) and rare (34) diseases. Those clinical texts were translated from spanish to english.\u003c/p\u003e \u003cp\u003eIn order to balance both classes of diagnosis, we included 32 RDs obtained from clinical records of a previous project, URUGENOMES (urugenomes.org)\u003csup\u003e4,21\u003c/sup\u003e and 277 from PubMed. For the last, we scrapped the PubMed platform (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://pubmed.ncbi.nlm.nih.gov/\u003c/span\u003e\u003cspan address=\"https://pubmed.ncbi.nlm.nih.gov/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e)\u003c/span\u003e searching for case reports of RD in free articles.\u003c/p\u003e \u003cp\u003eAs obtaining medical notes on rare diseases is challenging, we generated some ones using chatGPT (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://chat.openai.com\u003c/span\u003e\u003cspan address=\"https://chat.openai.com\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e). Thus, we included 10 RD clinical records and 13 common diseases generated by the tool and these were manually curated by a clinical geneticist, adding 23 diagnostics to the corpus. Our complete corpus finally has 842 clinical records, 418 rare and 424 common. It is available under: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://sites.google.com/view/leticia-cagnina/research\u003c/span\u003e\u003cspan address=\"https://sites.google.com/view/leticia-cagnina/research\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e and can be free-used for research issues performing the corresponding citation.\u003c/p\u003e \u003cp\u003eNo preprocessing is done to preserve the data\u0026rsquo;s genuine characteristics. The records collected constitute a balanced corpus which is more reliable to work with. Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e shows the main characteristics of the dataset. The vocabulary is large, with more than half of the words unique. This could be expected since the texts are related to specific medical issues (names of medications, lab tests, symptoms, diseases, abbreviations used by doctors, etc.). The diagnoses are written in 4 sentences on average although the variability of lengths is high: most clinical notes have only one sentence (possibly the diagnosis is written like a paragraph), just one with 340 (short sentences) and the rest with values oscillating the 2 and 70 sentences.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eDataset statistics.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"9\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003ediagnosis\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003evocabulary\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003ewords\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eWord average\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003esentences\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eSentence average\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eSentence min\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c9\"\u003e \u003cp\u003eSentence max\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ecorpus\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e842\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e41765\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e787277\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e935\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e3576\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e340\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003ecommon\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e424\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e29671\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e474648\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e1119\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e2753\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e340\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003erare\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e418\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e21686\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e312629\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e747\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e823\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e39\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e (two bottom rows) shows the characteristics of the corpus separated by classes. As mentioned this corpus is balanced so it has a similar amount of records of RD and non-RD (418 vs 424). Although the size of vocabulary used is not very different, it seems that clinical notes of common diseases are much longer than ones for RD (474K vs 312k). The same occurs with the number of sentences: RD are written using one third of those included in common diagnostics. In fact, most of the notes of RD use two sentences while those of common are slightly more extensive (6 on average). Only one record has a maximum number of sentences of 340 which corresponds to a common disease but the rest of this class are around 1 and 70. Moreover, the RD diagnosis has between 1 and 39 sentences as maximum.\u003c/p\u003e \u003cp\u003eWe also analyzed the contents of the clinical notes by using word clouds, which show the frequency of the words in each class. Most used words in common diseases notes are regular medical terms such as blood, pain and patient (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eA). Also some words related to non specific treatment: capsule, daily and hours. Interestingly the word cloud corresponding to RD notes (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eB) shows advanced studies and serious issues found in words such as tomography, examination, tumor and mass. Top 50 more representative words in the whole proposed dataset are in Supplementary Figure \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eC shows the proportion of medical notes collected from the different sources. Note that no requirement was stated in the time to collect the record and thus, the diversity of the distribution of words (log-scaled) is high. The distribution corresponding to RD is approximately normal which is suitable for classification models (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eD). However, some outliers above 3500 words can be observed for this class. The kernel density estimation is shown with a narrow kurtosis (leptokurtic) and central tendency around 500. The distribution for the class of common diseases seems to be bimodal although the central tendency is around 100 without major outliers (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eD).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Machine Learning models\u003c/h2\u003e \u003cp\u003eWe considered TF-IDF (Term Frequency Inverse Document Frequency) weighting and boolean schemes. We applied either L2 or L1 normalization, we limited the number of features considered (5000 or 10000) and considered either words or trigrams as features. After vectorization, several classic classification models were tested: Support Vector Machines (SVM), Logistic Regression (LR) and Decision Trees (DT). For each baseline model we performed 30 independent runs with different dataset partitions considering 80% for training and 20% to test. The best results were obtained with the following hyperparameters: LR used Solver Newton-CG, TF-IDF norm L2 representation and max features 10000; SVM included a sigmoid kernel, TFIDF norm L1 representation; maximum depth of DT was 23 and considered TFIDF norm L2 representation.\u003c/p\u003e \u003cp\u003eWe also include models based on deep neural networks such as the Long-Short Term Memory (LSTM)\u003csup\u003e22\u003c/sup\u003e and Convolutional Neural Network (CNN)\u003csup\u003e23\u003c/sup\u003e, model architectures are shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. The input of these models are static dense representations of words, that is, embeddings. After several experiments, we decided to use a combination of our own pre-trained vectors (obtained from signs and symptoms of rare diseases) and others specific for the task to solve in our study (obtained from \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/yao8839836/obesity\u003c/span\u003e\u003cspan address=\"https://github.com/yao8839836/obesity\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003e).\u003c/span\u003e We used two embedding layers with the pre-trained word vectors which are concatenated to obtain a 400 dimensional higher-level representation of each input text. After processing the input with the specific architecture of each model, the output is obtained as the result of a fully connected softmax layer to perform the classification using the probability distribution over the output labels (RD vs. common). The LSTM includes a layer with 64 units (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eA) while the CNN only 4 1D-convolutional layers for extracting 70 filters with different sizes of kernels (varying between 1 and 4) (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eB). After applying max pooling operation to each feature map, the outputs are concatenated, flatted and passed to a dense 64-unit layer. Previous to the output layer, a dropout operation is performed to reduce overfitting (rate 0.2). The only change introduced to the previous architecture to construct the CNN\u0026thinsp;+\u0026thinsp;LSTM model, is the inclusion of a LSTM layer between the concatenation of the 1D-convolutional and the dense layer (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eC). The number of units in the dense layer and filters is lower (here 50). Finally, the architecture of the LSTM\u0026thinsp;+\u0026thinsp;CNN is obtained by embedding the CNN between the LSTM and the output layer (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eD). We removed the 64-unit dense layer and the dropout operation to simplify the ended model. We also reduced the size of the filters and the units in the LSTM (10 and 16 respectively).\u003c/p\u003e \u003cp\u003eFor each deep neural network architecture the input layer is the text to classify. All architectures are shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eA-D).These architectures were obtained as the best after testing several models with different configurations of hyperparameters.\u003c/p\u003e \u003cp\u003eLarge language models are pre-trained using large amounts of data (in an unsupervised way) and usually fine-tuned (in a supervised way) with specific data depending on the task to solve. An example of such models is the Bidirectional Encoder Representations from Transformers\u003csup\u003e15\u003c/sup\u003e (BERT). Unlike the models we proposed before, BERT uses contextualized word representation of the input which feeds several stackedTransformer encoders. We select BERT for our experiments because, beyond its good performance in various NLP tasks, there are several models pre-trained with biomedical texts. We compare a base version of BERT with Bio_ClinicalBERT. The latter is a fine-tuned (with medical conditions data) version of the first domain-specific BERT based model pre-trained on large scale biomedical corpora, named BioBert\u003csup\u003e24\u003c/sup\u003e. The model is available in \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003esid321axn/Bio_ClinicalBERT-finetuned-medicalcondition \u0026middot; Hugging Face\u003c/span\u003e. BERT was pre-trained on English Wikipedia and General BooksCorpus while Bio_ClinicalBERT, besides the same as BERT, was pre-trained with PubMed Abstracts and PMC Full-text articles (that is, biomedical domain-specific texts) and fine-tuned with clinical conditions of diseases.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"3 Results","content":"\u003cp\u003eWe analyze the results of the different models implemented and show the one that obtained the best performance in detecting RD from medical notes. Then, we perform an error analysis on the systematically misclassified records to highlight some characteristics about the proposed corpus.\u003c/p\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e3.1 The SVM better predicts RD patients on discharge summary texts\u003c/h2\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e summarizes the results of applying all models and considering the metrics Accuracy (useful when the problem has balanced classes), the standard F-measure and Area Under Curve Receiver Operator Characteristic (AUC) for complementing the evaluation of the performance.\u003c/p\u003e \u003cp\u003eFirst rows in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e shows for each one of the baseline models the results obtained with the best configuration. The results suggest that SVM outperforms the models obtained with the baseline classifiers but also the advanced ones (LSTM, CNN and BERT) when F-measure is considered (0.927).\u003c/p\u003e \u003cp\u003eRegarding the results of deep models (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, rows 4\u0026ndash;7), CNN and LSTM perform similarly with a small difference between metrics: CNN is better than LSTM in F-measure (0.906 and 0.898, respectively) but LSTM is better than CNN in accuracy and AUC (0.899 and 0.893 in accuracy, respectively). It is interesting to note that LSTM outperforms the combination of LSTM and CNN models which would indicate that LSTM can learn long-term dependencies from sequences of higher-level representations without help of convolutional operations. Figure\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eA shows the confusion matrix of the best model (SVM). The class 0 corresponds to common disease and class 1 to RD. Only one common disease was misclassified and 11 RD were wrongly classified as common. Figure\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eB the ROC curve obtained from the data.\u003c/p\u003e \u003cp\u003eThe last 2 rows of Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e show the results obtained with the Transformer-based models. BioBert obtained better performance than BERT considering F-measure (0.873 vs 0.852) and the same behavior with the other metrics. The reason is probably because BioBERT was trained with biomedical data. BioBert performs quite similarly to all models, demonstrating that for this particular task, the complexity of transformed-based models is not synonymous with better results.\u003c/p\u003e \u003cp\u003eWe chose the AUC metric in model analysis as it provides valuable insights into the model's ability to distinguish between classes and its impact on specific errors like false positives and false negatives, as demonstrated in previous studies\u003csup\u003e25\u003c/sup\u003e. In our case, higher AUC means how better the model is at distinguishing between patients with a RD and not. LSTM, CNN and SVM classifiers obtained the highest AUC (0.95, 0.94 and 0.96, respectively) indicating their ability to correctly classify diagnosis with RD with relatively small models (in comparison to the transformer-based ones).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eBest results obtained with each method for the proposed corpus.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eF-measure\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAUC\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLR\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.923\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.922\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.925\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.888\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.893\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.887\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSVM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.929\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.927\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.960\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.893\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.906\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.944\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLSTM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.899\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.898\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.952\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNN\u0026thinsp;+\u0026thinsp;LSTM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.893\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.877\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.922\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLSTM\u0026thinsp;+\u0026thinsp;CNN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.888\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.887\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.920\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBioBERT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.882\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.873\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.881\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBERT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.858\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.852\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.858\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e3.2. Systematically misclassified clinical records\u003c/h2\u003e \u003cp\u003eThe most frequent error of our model is that RDs are misclassified as common diseases. In almost all models (baseline and not) a median of ~\u0026thinsp;87.25% of all misclassified records correspond to RD wrongly classified as common. Most frequently misclassified clinical records are in Table \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e (several tabs corresponding to each model). A median of 80.95% of misclassified RD corresponds to MIMIC-III clinical records that were previously classified by another study\u003csup\u003e19\u003c/sup\u003e. The rest (mostly) correspond to common diseases that were translated from Spanish. Focusing on the SVM classification, all misclassified records (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e) were RD (RD classified as common). After careful consideration of an expert we found that most of them were actually common diseases, but mislabeled as RD in the corpus since the clinical records describe a large amount of complications of common diseases probably in elderly patients. Hence, the clinical text becomes very long, complicated with several interactions with procedures, medical specialties staff, drugs, interventions, and so on (Table \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e, tab SVM with expert comments on each clinical text), which might be an explanation of the misclassification. Even though the corpus might have some noise regarding the labels (which is a realistic scenario in the context of several applications) the classifiers are able to perform fairly well in practice.\u003c/p\u003e \u003c/div\u003e"},{"header":"4 Discussion","content":"\u003cp\u003eRDs are difficult to detect, to diagnose and to treat. Patients with RD have to navigate the healthcare system patiently, inefficiently and with economical (and psychological) costs. Timely diagnosis, hence early strategies to assess the disease, might be of great importance to control the impact on the patient and the family.\u003c/p\u003e \u003cp\u003eThe diagnosis pipeline of RD is different from other diseases and very frequently includes consultation with geneticists and molecular studies for proper diagnosis. An early detection of the presence of a RD might be a substantial improvement in many cases.\u003c/p\u003e \u003cp\u003eOur method aims to timely detect RD patients from medical records (discharge summaries) obtained from many sources, specially from an emergency unit. When the discharge summary of a patient classifies as a potential RD a flag could be raised in the hospital system and measurements could be set in place, such as consultation with geneticist and other specialists, molecular analysis, improving the time until diagnosis.\u003c/p\u003e \u003cp\u003eAn accuracy around 90% implies that in 90% of the cases the classification is correct, and the RD flag should be raised. The remaining 10% of cases correspond to individuals that have a common disease but they were classified as RD. The impact of such an error would be mostly an economical loss for a public hospital, since additional unnecessary consultations and/or laboratory examination might be done, however, the savings generated by the remaining 90% most likely outweigh the cost of these additional consultations. Besides, upon manual reexamination most of these false calls would be easily detected. On the other hand, an error misclassifying RD as common would have a higher impact on patients well-being (patients would go through the standard algorithmic path, hence probably a diagnostic odyssey) and costs would be even bigger.\u003c/p\u003e \u003cp\u003eFuture work relies on the fine-tuning of models that are already close to the clinical aspects of the problem, such as Bio_ClinicalBERT (freely available), using our corpus. This might improve our results by better understanding the technical words and their contexts.\u003c/p\u003e \u003cp\u003eAlso, the tested models are not strong in the explicativeness. In some models, we are not able to understand why a specific clinical record or discharge summary is classified as RD. Understanding the results of the classification process would improve our knowledge on RD in general, and also, how to write discharge summaries so that models would work properly.\u003c/p\u003e \u003cp\u003eAdditionally, we intend to expand the corpus. The inclusion of more reliable RD clinical records and manual curation of those already included, are going to improve downstream results.\u003c/p\u003e \u003cp\u003eFinally, we believe that this valuable corpus is in line with the trend of few-shot learning for classification above all in the biomedical domain and we would try other transformer-based methods for few-shot identification of rare diseases.\u003c/p\u003e"},{"header":"5 Conclusions","content":"\u003cp\u003eWe presented a corpus for the classification of rare diseases from clinical notes. We showed a detailed exploratory analysis of the data collected and concluded with a balanced dataset with a similar number of notes labeled as containing RD or not.\u003c/p\u003e \u003cp\u003eTo test the proposed resource, we performed a comparative study of different models for the classification of rare diseases, the classical SVM, the artificial neural networks LSTM and CNN and, the recent transformer-based BERT. SVM performs the best with a F-measure of 0.927.\u003c/p\u003e \u003cp\u003eThus, we conclude that the SVM-based model is able to accurately predict rare diseases based on the clinical record of the patients, hence enabling the possibility to be included as a warning and a lead to a more accurate diagnostic path.\u003c/p\u003e \u003cp\u003eBy making the corpus available we encourage future applications to be developed and refined. In addition to helping mitigate the lack of annotated data for the identification of RD, this resource can be safely used for few-shot machine learning algorithms in classification as well as other tasks.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eData availability statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe corpus is available under https://sites.google.com/view/leticia-cagnina/research\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study was partially funded by BID (Banco Iberomericano de desarrollo) in the context of the URUGENOMES Project (Proyecto ATN/KK-L4584-\u0026ldquo;Fortalecimiento de las capacidades técnicas y humanas para las exportaciones de servicios genómicos\u0026rdquo;). Additionally, support was obtained from the CONICET, Short Research Stages program given to Leticia Cagnina.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003ePEDECIBA under Grant Number: Alicuotas\u003c/p\u003e\n\u003cp\u003eINNOVA II under Grant Number: DCI‑ALA /2011/23‑502.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eMR: Data curation, formal analysis, Roles/Writing - original draft\u003c/p\u003e\n\u003cp\u003eVR: Data curation, Writing - review \u0026amp; editing\u003c/p\u003e\n\u003cp\u003eHN: Validation, Methodology, Writing - review \u0026amp; editing\u003c/p\u003e\n\u003cp\u003eLC: Supervision, Funding acquisition, Roles/Writing - original draft, Methodology\u003c/p\u003e\n\u003cp\u003eLS: Supervision, Roles/Writing - original draft, Project administration, Data curation\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eThe Voice of 12,000 Patients. Experiences and Expectations of Rare Disease Patients on Diagnosis and Care in Europe. (EURORDIS - Rare Diseases Eu, 2009).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSireau, N. Rare Diseases: Challenges and Opportunities for Social Entrepreneurs. (Routledge, 2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYan, X., He, S. \u0026amp; Dong, D. Determining How Far an Adult Rare Disease Patient Needs to Travel for a Definitive Diagnosis: A Cross-Sectional Examination of the 2018 National Rare Disease Survey in China. Int. J. Environ. Res. Public Health 17, (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRaggio, V. et al. Whole genome sequencing reveals a frameshift mutation and a large deletion in YY1AP1 in a girl with a panvascular artery disease. Hum. Genomics 15, 28 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMeyer, E. J. et al. CBG Montevideo: A Clinically Novel Mutation Leading to Haploinsufficiency of Corticosteroid-binding Globulin. J Endocr Soc 5, bvab115 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDella Mina, E. et al. Improving molecular diagnosis in epilepsy by a dedicated high-throughput sequencing platform. Eur. J. Hum. Genet. 23, 354\u0026ndash;362 (2015).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu, H.-Y. et al. Diagnostic and clinical utility of whole genome sequencing in a cohort of undiagnosed Chinese families with rare diseases. Sci. Rep. 9, 19365 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eClark, M. M. et al. Meta-analysis of the diagnostic and clinical utility of genome and exome sequencing and chromosomal microarray in children with suspected genetic diseases. NPJ Genom Med 3, 16 (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEsteva, A. et al. A guide to deep learning in healthcare. Nat. Med. 25, 24\u0026ndash;29 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHwang, S. \u0026amp; Lee, B. Machine learning-based prediction of critical illness in children visiting the emergency department. PLoS One 17, e0264184 (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHatachi, T. et al. Machine Learning-Based Prediction of Hospital Admission Among Children in an Emergency Care Center. Pediatr. Emerg. Care 39, 80\u0026ndash;86 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGulshan, V. et al. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA 316, 2402\u0026ndash;2410 (2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGolden, J. A. Deep Learning Algorithms for Detection of Lymph Node Metastases From Breast Cancer: Helping Artificial Intelligence Be Seen. JAMA: the journal of the American Medical Association vol. 318 2184\u0026ndash;2186 (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDoshi-Velez, F., Ge, Y. \u0026amp; Kohane, I. Comorbidity clusters in autism spectrum disorders: an electronic health record time-series analysis. Pediatrics 133, e54\u0026ndash;63 (2014).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4171\u0026ndash;4186.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen,Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 140, (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJohnson, A. E. W. et al. MIMIC-III, a freely accessible critical care database. Sci Data 3, 160035 (2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDong, H. et al. Rare Disease Identification from Clinical Notes with Ontologies and Weak Supervision. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2021, 2294\u0026ndash;2298 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFernando L\u0026oacute;pez Bello, Hugo Naya, V\u0026iacute;ctor Raggio, Aiala Ros\u0026aacute;. From medical records to research papers: A literature analysis pipeline for supporting medical genomic diagnosis processes. Informatics in Medicine Unlocked 15, 100181 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSpangenberg, L. et al. Novel frameshift mutation in PURA gene causes severe encephalopathy of unclear cause. Mol Genet Genomic Med 9, e1622 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSepp Hochreiter, J. S. Long short-term memory. Neural Comput. 9, 1735\u0026ndash;1780.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLeCun et al, Y. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Comput. 1, 541\u0026ndash;551.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLee, J. et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 1234\u0026ndash;1240 (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFawcett, P. F. Robust Classification for Imprecise Environments. 42, 203\u0026ndash;231.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-4795232/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4795232/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eRare diseases (RD) are a group of pathologies that individually affect less than 1 in 2000 people but collectively impact around 7% of the world's population. Most of them affect children, are chronic and progressive, and have no specific treatment. RD patients face diagnostic challenges, with an average diagnosis time of 5 years, multiple specialist visits, and invasive procedures. This \u0026lsquo;diagnostic odyssey\u0026rsquo; can be detrimental to their health. Machine learning (ML) has the potential to improve healthcare by providing more personalized and accurate patient management, diagnoses, and in some cases, treatments.\u003c/p\u003e \u003cp\u003eLeveraging the MIMIC-III database and additional medical notes from different sources such as in-house data, PubMed and chatGPT, we propose a labeled dataset for early RD detection in hospital settings.\u003c/p\u003e \u003cp\u003eApplying various supervised ML methods, including logistic regression, decision trees, support vector machine (SVM), deep learning methods (LSTM and CNN), and Transformers (BERT), we validated the use of the proposed resource, achieving 92.7% F-measure and a 96% AUC using SVM.\u003c/p\u003e \u003cp\u003eThese findings highlight the potential of ML in redirecting RD patients towards more accurate diagnostic pathways and presents a corpus that can be used for future development and refinements.\u003c/p\u003e","manuscriptTitle":"RDCorpus: labeled medical records for the timely detection of rare diseases","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-28 02:34:00","doi":"10.21203/rs.3.rs-4795232/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-01-16T06:28:07+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-01-13T04:06:46+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"279517447325876974177233968170857317547","date":"2025-01-06T23:29:42+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-08-30T15:08:24+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"74027946884658189070630771559643330462","date":"2024-07-31T15:20:13+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"294721917230863943968064692872051366885","date":"2024-07-26T16:34:15+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-07-26T15:15:50+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-07-26T15:08:32+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2024-07-26T04:58:33+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-07-26T04:35:32+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2024-07-24T12:03:01+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"b1a57940-80e9-4973-be3d-f0110868d4ce","owner":[],"postedDate":"August 28th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":36314764,"name":"Biological sciences/Genetics/Genome"},{"id":36314765,"name":"Biological sciences/Genetics"},{"id":36314766,"name":"Health sciences/Health care"},{"id":36314767,"name":"Health sciences/Medical research/Genetics research"},{"id":36314768,"name":"Physical sciences/Engineering/Biomedical engineering"}],"tags":[],"updatedAt":"2025-03-03T16:01:14+00:00","versionOfRecord":{"articleIdentity":"rs-4795232","link":"https://doi.org/10.1038/s41598-025-90450-0","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2025-02-26 15:57:32","publishedOnDateReadable":"February 26th, 2025"},"versionCreatedAt":"2024-08-28 02:34:00","video":"","vorDoi":"10.1038/s41598-025-90450-0","vorDoiUrl":"https://doi.org/10.1038/s41598-025-90450-0","workflowStages":[]},"version":"v1","identity":"rs-4795232","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4795232","identity":"rs-4795232","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}