Application of Machine Learning to Distinguish IBS from Crohn’s Disease in Underdeveloped Regions

Application of Machine Learning to Distinguish IBS from Crohn’s Disease in Underdeveloped Regions | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Case Report Application of Machine Learning to Distinguish IBS from Crohn’s Disease in Underdeveloped Regions Varun Bhargav Kurra, Jennifer Otiono This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9658515/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background and Aims: Crohn’s disease, a chronic inflammatory bowel disease (IBD), and Irritable Bowel Syndrome (IBS) are gastrointestinal conditions that share overlapping symptoms. Due to their similarities, ~10-30% of individuals diagnosed with Crohn’s disease are initially misdiagnosed with Irritable Bowel Syndrome (IBS), with the crisis being especially severe in underdeveloped countries. Artificial intelligence (AI) and its subset, Machine Learning (ML), are becoming essential technologies in medical specialties. This quantitative research paper examines the conditions of Crohn’s disease and Irritable Bowel Syndrome (IBS) misdiagnosis in underdeveloped countries, and develops a Machine Learning (ML) model that improves diagnostic accuracy and availability against diagnostic conditions in underdeveloped regions. Methods: Data from 383,220 unique patients were extracted from the Dataset MIMIC-IV(v3.1). Pandas and Numpy were used to load, filter, and organize the data. ClinicalBERT was used for the vectorization of the clinical notes. L2 regularization, dropout, Leaky ReLU, and early-stopping were utilized to minimize overfitting. ML models were trained, tuned, evaluated, and Bayesian optimization was implemented on the finalized model. Results: The finalized model received an accuracy of 74.41%, an F1-score of 73.40%, and an AUC of 79.56%. After Bayesian Optimization, the model’s accuracy reached 78%. Conclusions: Underdeveloped countries lack affordable medical testing, as 55% of Crohn’s disease patients in underdeveloped countries are unable to afford diagnostic testing. LLMICs also face a shortage of specialists as they report an average gastroenterologist density of 0.005-0.02 per 100,000 inhabitants. Therefore, the possibility and necessity of implementing a low-cost predictive ML application are evident. Artificial Intelligence and Machine Learning Gastroenterology & Hepatology Health Economics & Outcomes Research Epidemiology Crohn’s Disease Irritable Bowel Syndrome (IBS) Artificial Intelligence (AI) Machine Learning (ML) Machine-Learning-Based Disease Diagnosis (MLBDD) Clinical Notes Laboratory Data Figures Figure 1 Figure 2 I. INTRODUCTION Crohn’s disease, one of the two IBD diseases, and Irritable Bowel Syndrome share common symptoms, and both affect one’s digestive tract, making maintaining diagnostic accuracy challenging. 1 To assist the common misdiagnosis, the average diagnostic delays (correct diagnosis after misdiagnosis) can range from 2 months to 8 years in IBD, with Crohn’s disease averaging longer delays than Ulcerative Colitis, the 2nd IBD disease. 2 The diagnostic delay of Crohn’s disease leads to the necessity of stronger medication, hospital admissions, and expensive surgeries, imposing heavy cost burdens on those who can’t afford the cost. 3 This research investigated whether AI models leveraging clinical notes and laboratory data can accurately distinguish between IBS and Crohn’s disease to improve diagnostic accuracy and reduce misdiagnosis. The hypothesis is that integrating clinical notes and laboratory data will improve diagnostic performance compared to traditional methods of diagnostic analysis. Through this research, we seek to demonstrate that AI can enhance early detection of gastrointestinal disorders and provide underdeveloped countries that lack medical diagnostic tools with a high-impact and low-cost solution for improving the misdiagnosis crisis between IBS and Crohn’s disease to prevent elongated diagnostic delays. The findings of this case study could inform healthcare systems, alter clinical guidelines, and serve as a foundation for the future application of AI-driven predictive diagnostic tools in the field of gastroenterology. II. LITERARY REVIEW Crohn’s Disease and IBS Introduction Crohn’s disease is a chronic autoimmune condition that inflames and irritates one’s digestive tract. Crohn’s disease and Ulcerative Colitis are the 2 primary forms of inflammatory bowel disease (IBD). Symptoms of Crohn’s disease may develop gradually or suddenly, and may be mild or severe. A “flare” is a term used in Crohn’s disease to describe when symptoms occur. The etiology of Crohn’s disease is unknown, but it’s widely believed that genetic predisposition and immune system irregularities contribute to the development of the disease. Crohn’s disease affects more than 750,000 people in the United States and approximately 6 to 8 million people globally. 4 Irritable bowel syndrome (IBS) is a common condition that affects the gastrointestinal tract, the stomach, and the intestines. Symptoms of IBS tend to be present for an extended period, making IBS an ongoing condition that needs long-term care. Similar to Crohn’s disease, the causes of IBS are unknown, but factors that reportedly contribute to the syndrome include muscle contractions in the intestine, issues with the nervous system, severe infections, early-life stress, and changes in gut microbes. The “flares” of IBS may be triggered by food and anxiety, although these triggers don’t initially cause the symptoms to occur, but rather enhance their effect. 5 A 2015 clinical review conducted by Chey et al. 6 concluded that approximately 12% (7% to 21%) of people globally, with roughly 24-45 million people suffering from IBS. Although IBS affects 10%-15% of adults in the United States, only 5%-7% of adults are initially diagnosed with the syndrome. IBS is also known to affect more women, with twice as many women obtaining the disease as men annually. 7 IBS and Crohn’s disease are both known for their “flare-remission” cycle, with “flare-ups” signifying the period of increased symptom severity. “Flare-ups” are when symptoms of the disease become more noticeable after a period of remission, with its causation relating to a dysfunctional immune response where one’s immune system attempts to “attack” germs that enter the body, but uncontrollably harms bodily tissues instead. Inflammation happens when an autoimmune response occurs, when the immune system “attacks” one’s body cells inappropriately while attempting to remove germs from the body. Inflammation is a central signal that the “attack” is ongoing and the “Flare-up” is active. Once the bodily threat is gone, the immune system flows into a state of repair, and inflammation may begin to lessen, but in certain cases, inflammation doesn’t completely heal. The “flare-ups” in both conditions are typically short-term but vary in duration, lasting a few hours to weeks in IBS and from a few days to months in Crohn’s disease. 8,9 During periods of remission, symptoms are reduced or absent, but the underlying disease is still present. Although both IBS and Crohn’s disease follow a “flare-remission” pattern, they vary significantly in their nature and underlying fundamentals. Crohn’s disease is an organic IBD characterized by visible inflammation, ulceration, and structural tissue damage commonly found in the ileocolon. Unlike Crohn’s disease, IBS is a functional disorder, classified as a “syndrome”, meaning that patients experience real symptoms but no observable damage occurs. Patients with IBS don’t experience inflammation or structural bowel damage, which patients with Crohn’s disease do. Overlapping Symptoms and Delays IBS and IBD, including Crohn’s disease and Ulcerative colitis, have an overlap of apparent symptoms, making patients susceptible to frequent misdiagnosis. The overlapping symptoms occur as both conditions affect the gastrointestinal tract, which has a narrow repertoire of symptoms and a limited range of ways it can manifest bacteria from diseases. As a result, multiple gastrointestinal conditions, such as IBS and Crohn’s disease, may appear with the same clinical features. Patients with either disorder may experience changes in bowel habits, typically involving diarrhea along with pain, bloating, and rectal mucus discharge. A systematic review by Halpin and Ford (2012) analyzed studies of patients with IBD who had answered symptom questionnaires for IBS. Their findings revealed that approximately 35% to 40% of patients with IBD reported IBS-related symptoms, and patients with IBD were ~5 times more likely to have symptoms compatible with IBS than healthy controls. 10 Their studies also concluded that the prevalence of IBS-type symptoms in IBD was slightly higher in Crohn’s disease than in Ulcerative colitis, with the odds ratio being 1.62, meaning that patients with Crohn’s disease had a 62% higher chance of reporting IBS-compatible symptoms. One might assume that the underlying symptoms persist during remission, allowing for an accurate and clear diagnosis. However, this isn’t the case, as Dr. Keeley M Fairbrass of the Leeds Gastroenterology Institute explains, “IBS-type symptoms co-exist in IBD patients even during periods of deep remission.” 11 A 2020 meta-analysis conducted by Ford 11 of 27 studies (15 Crohn’s Disease) published between January 1, 2012, and May 11, 2020, which included 3169 patients with IBD in remission, concluded that IBS-type symptoms among patients with IBD who were in remission were 32.5%, ranging from 11.2% to 63.6%. The analysis also found that among the 15 studies of patients with Crohn’s disease in remission, the prevalence of IBS-type symptoms was 36.6%, averaging 4.1% higher than in IBD patients as a whole. In conclusion, the prevalence of IBS-type symptoms occurs in ~35% to 40% of patients with IBD, and 32.5% of patients with IBD in remission, with Crohn’s disease patients averaging a 63.6% higher likelihood of obtaining IBS-type symptoms than Ulcerative colitis, providing a solid basis for frequent misdiagnosis. With IBS and Crohn’s disease patients obtaining overlapping symptoms, diagnostic delays prevailed. A 2023 meta-analysis conducted by Jayasoorya et al. 12 of 101 EMBASE [Excerpta Medica dataBASE] and Medline studies representing over 112,194 patients with IBD (CD = 59,359; UC = 52,835) concluded that the median time to Crohn’s disease diagnosis was 8 months (Interquartile Range [IQR]: 5 months to 15.2 months). In high-income countries, this was 6.2 months (IQR: 5 months to 12.3 months) compared to 11.7 months (IQR: 8.3 months to 18 months) in low-income countries. Another study consisting of 46 provider surveys and 15 low-income countries reported that most patients had symptoms of Crohn’s disease 6-24 months before diagnosis, and 26% to 50% of patients lived in rural areas, exacerbating the lack of medical resources and efficiencies in rural, low-income provinces. In conclusion, patients with Crohn’s disease in low-income countries experience an average diagnostic delay roughly 2x longer than in high-income settings. Misdiagnosis: Crises between Crohn’s Disease and IBS The most significant cause of diagnostic delays is widespread misdiagnosis. A 2014 case-controlled study published in the United European Gastroenterology (UEG) journal, consisting of over 20,193 IBD cases, revealed that IBD patients were more than 3 times as likely as controls to have a prior record of IBS. 15% of IBD cases compared to 5% of controls had IBS encoded in their medical records before diagnosis, with 11% of IBD cases (cf. 5% of controls) having the code of IBS for over a year before IBD, and over 6% of cases (cf. 3% of controls) maintaining the disease for over 5 years. The study concluded that globally, approximately 10% of IBD patients are initially misdiagnosed with IBS, and in the case of 3% of patients, the misdiagnosis persisted for 5 or more years. 13,14 Along with the 10% risk of initial misdiagnosis, a large longitudinal study analyzing 9,341 incident cases found that individuals with IBS had an 8.6-fold increased risk of receiving an IBD diagnosis compared to non-IBS controls (238.1 vs. 27.8 cases per 100,000 person-years, p < 0.0001). In a subset of well-defined IBS cases, the risk increased to 15 times that of the non-IBS cases, highlighting the potential of initial misdiagnosis and early undetected IBD among IBS patients. Overall, the global rate of Crohn’s disease to IBS misdiagnosis can be stated as 10%, with 3% of total patients being misdiagnosed for 5 years or more. Additionally, IBS patients have an 8.6-15 times higher risk than non-IBS patients of obtaining IBD, highlighting a significant and ongoing misdiagnosis crisis. Misdiagnosis: Crises in Underdeveloped Countries The misdiagnosis crisis is a notable burden in high-income countries. Still, the situation turns to catastrophe in low-income nations due to unaffordable costs and scarcity of diagnostic tools and medical professionals. Although the crisis is exacerbated in low-income countries by a scarcity of equipment/professionals and unaffordable costs, concrete statistical data on Crohn’s disease misdiagnosis as IBS and diagnostic delay remain unavailable. A scoping review of 4486 publications conducted by Rajbhandari et al. 15 from 79 LLMICs (Low and Lower-Middle-Income-Countries) revealed that only 21 LLMICs have publications on Crohn’s disease, indicating major under-diagnosis or under-reporting and outlining the scarcity of statistical data in underdeveloped regions regarding Crohn’s disease. On top of the statistically high Crohn’s disease to IBS misdiagnosis rate of 33.5%, estimated through a statistical analysis of Crohn’s disease to IBS ratios, factors such as diagnostic affordability and resource availability also make an impact on underdeveloped countries. Firstly, regarding diagnostic cost, a cross-sectional survey of gastroenterology providers from 15 underdeveloped countries conducted by Ng et al. 16 reported that 52.4% of Crohn’s disease patients in Asia and 60.9% of patients in Africa could not afford diagnostic testing to distinguish between Crohn’s disease and other infectious diseases. Secondly, addressing the scarcity of diagnostic resource availability, analyzing Africa’s metrics reveals a shortfall in equipment and specialists. It should be noted that the gold-standard method of Crohn’s disease diagnosis is endoscopy, including colonoscopy and ileoscopy. A 2022 report on the progress of expanding gastrointestinal endoscopy in LLMIC African nations, conducted by Rajhbandari et al. 17 revealed that Eastern Africa’s endoscopy capacity consisted of approximately 0.12 endoscopists, 0.12 gastroscopes, and 0.09 colonoscopes available per 100,000 inhabitants, showcasing a stark deficit of diagnostic equipment. Addressing the aspect of the availability of medical specialties, the survey also disclosed that in several underdeveloped African countries, such as Malawi and Rwanda, there was less than 1 medically trained gastroenterologist in each country, serving more than 10 million inhabitants, and fewer than 11 endoscopy centers, showing the extreme specialist shortage in LLMICs, especially sub-Saharan Africa. 16 Similarly, Ruma Rajbhandari, a Harvard professor and lead Crohn’s disease researcher, stated that “researchers found ‘very few gastroenterologists – less than 20’ for Ethiopia, which has a population of more than 115 million.” 18 Overall, through a statistical analysis of the quantity of gastroenterologists operating in Africa, a generalizable average gastroenterologist density of 0.005 to 0.02 per 100,000 inhabitants, or 1 specialist per 5-20 million inhabitants, can be established. In conclusion, the scarcity of resources and specialists is widely apparent in African LLMIC nations, with 0.12 endoscopists, 0.12 gastroscopes, and 0.09 colonscopes available per 100,000 inhabitants, and an average gastroenterologist density of 0.005 to 0.02 per 100,000 inhabitants being generalizable metrics between underdeveloped nations. The following scarcity of resources and specialists highlights the leading cause of an inflated misdiagnosis rate of 33.5% between Crohn’s disease and IBS in underdeveloped countries. Predictive AI Introduction With nations desperately seeking alternative, available, and low-cost diagnostic solutions 19,20 , Artificial Intelligence’s (AI’s) reputable background in the field of predictive healthcare stands as the most comprehensive and promising approach to revolutionize early detection in underdeveloped regions. AI has experienced an unexpected surge in healthcare over the past 2 decades due to the development of Machine Learning (ML) and Deep Learning (DL), a subset of ML, the digitization of healthcare records, and a quantifiable rise in medical data. The global AI in the healthcare market, once valued at $1.1 billion in 2016, grew to $22.4 billion in 2023, representing a rise of approximately 1,779%. Between the years of 2022 and 2023 alone, the market grew by over 45%, from $15.4 billion to $22.4 billion. 21 Currently, the global AI in healthcare market was valued at roughly $26.57 billion in 2024 and is projected to reach $187.69 billion by 2030, growing at a compound annual growth rate (CAGR) of over 38.62% from 2025 to 2030, outlining AI’s rapid integration into the healthcare sector. 22 A 2020 longitudinal bibliometric analysis of healthcare-related publications conducted by Guo et al. 23 revealed that the growth of research papers on healthcare-related AI significantly increased to 45.15% from 2014 to 2019, compared to the average of 17.02% per year since 1995. Fueling the rise of AI in modern healthcare, predictive AI, utilizing ML algorithms to analyze medical data, has risen in popularity and is currently being applied in early disease detection, patient outcome prediction, and resource optimization. The utilization of AI in healthcare allows for increased diagnostic accuracy, cost savings, and accessibility. It’s estimated that the wider adoption of AI in healthcare could lead to savings of 5% to 10% in US healthcare spending, around $200 to $360 billion annually. 24 Along with increased cost savings, it’s estimated that AI applications reduce more than 86% of errors made by healthcare workers, saving more than 250,000 lives per year. 25 AI can be utilized in regions where medical professionals are scarcely available, especially when it comes to diagnostic specialties. In the cases of many underdeveloped regions, AI-based predictive ML models have been extensively tested and evaluated in the cases of Tuberculosis (TB) and Breast Cancer screenings. A cost-effectiveness analysis conducted by Nsengiyumva et al. 26 of TB screening from chest radiograph (x-ray) interpretation using AI in Pakistan and Bangladesh observed the triage for TB in underdeveloped, high-burden programs using 5 AI models on 23,954 patients in Dhaka, Bangladesh. The usage of the AI model resulted in the following metrics: AUC up to 90.8% (90% sensitivity), qXR specificity of 74.3%, and CAD4TB of 72.9%, meeting WHO triage targets. Triage strategies with AI-based CXR analysis were projected to lower costs by 19% to 37% and average 3% to 4% Disability-Adjusted Life Years (DALYs), signifying major economic impact and enhanced diagnostic rates upon usage of the predictive AI model. Similarly, an observational study conducted by Kakileti et al. 27 on the clinical efficacy of Thermalytix on detecting Breast Cancer in 470 (a)symptomatic women revealed that the predictive-AI tool resulted in an overall AUC of 0.90, a sensitivity of 91.02%, and a specificity of 82.39%, outlining its strong performance. Correspondingly, a real-world evaluation conducted by Adapa et al. 28 of Thermalytix for population-level Breast Cancer screening in 183 locations in Punjab revealed that screening by Thermalytix delivered an age-adjusted incidence (AAI) of 120.94/100,000 (Crude rate: 0.18%), a threefold improvement compared to the NPCDCS recommended CBE-based screening, which resulted in a nationwide (Punjab) AAI of 31.2/100,000 (Crude rate: 0.03%) and between 38-48/100,000 (Crude rate: 0.04–0.05%) in 2022. Overall, the Thermalytix AI-based Breast Cancer screening tool’s metrics distinguish the fact that predictive AI models can provide low-cost and highly effective solutions to medical diagnostic issues.In underdeveloped regions where financial burdens, a lack of equipment, and medical inaccuracies hinder inhabitants from receiving proper medical care, the implementation of modern AI technologies stands as an effective solution. In summary, in underdeveloped regions, commonly LLMICs, the accurate diagnosis and distinction between Crohn’s disease and IBS is hampered due to high costs, unavailability of diagnostic tools, and a lack of medical professionals, heavily lowering diagnostic accuracy and increasing chances of misdiagnosis. These implications make it clear that an alternative diagnostic method for differentiating and diagnosing Cohn’s disease and IBS is needed in underdeveloped regions, ultimately reducing financial burdens, removing the heavy need for medical professionals, and saving lives. AI’s application to the Crohn’s disease and IBS misdiagnosis crisis in underdeveloped regions is an essential step in rectifying existing inadequate diagnostic methods. This research focuses on training, tuning, and evaluating an ML model to differentiate between Crohn’s disease and IBS using clinical notes and laboratory data to be applicable in underdeveloped regions. III. DATA & METHODOLOGY To build the IBS and Crohn’s Disease detection ML model, assessments of medical-grade datasets were conducted to align with rigorous benchmarks. Criteria for the database were established as the industry standards for modern-day datasets. The criteria implemented in the dataset assessments are data (a) quality, (b) structure, (c) usability, and (d) patient privacy. After the criteria were determined, the web was searched for datasets that not only aligned with the criteria but also had noticeable recognition in the inflammatory bowel disease research sector. The database of MIMIC looked to be the most notable, specifically the dataset MIMIC-IV (v3.1) and its subset, MIMIC-IV-Note (De-identified free-text clinical notes). MIMIC-IV (v3.1) is a large, open-sourced medical database developed by the MIT Laboratory for Computational Physiology (MLCP) in alliance with Beth Israel Deaconess Medical Center (BIDMC). The database contains de-identified data from both intensive care unit (ICU) and medical-surgeon care (Hospital) patients admitted between 2008 and 2019. The dataset has over 383,220 unique patients, 94,458 ICU stays, and 546,028 hospital admissions. The MIMIC-IV (v3.1) database aligns with the assigned data criteria in all 4 sections. In the data quality sector, the database had internationally encoded procedures and diagnoses following ICD (International Classification of Diseases) and CPT (Common Procedure Terminology) regulations, along with having consistent data validation and longitudinal records of patients. In the structure sector, the data was organized into multiple tables and matrices with labeled metadata, along with the use of standard terminologies such as ICD, LOINC, and CPT codes. In the usability sector, the database was publicly available, had heavy documentation and support from PhysioNet, and was compatible with modern-day software (Ex, Pandas & Tensorflow). Finally, in the patient privacy sector, it followed the regulated de-identification process by removing 18 identifiers mandated by HIPAA, applied data shifting (same ranges but different time frames), and required the completion of human research training and the signing of a data usage agreement. Overall, this database excelled at all measures of the criteria established for this research and was chosen as the database that would be utilized for further continuation. After the process of assessing databases was complete, MIMIC-IV (v3.1) was secured as the database in use for the entirety of this research, and the data-application process began. The following steps were taken to gain access and authorization for this database: (a) create a PhysioNet account & agree to the terms of use, (b) fill in the application form to be a credentialed user on the platform, (c) complete the required CITI program training (“Data or Specimens only Research”) and upload the completion report (PDF) into PhysioNet’s submission form, (d) wait for approval from PhysioNet’s team to confirm your credentials and submissions through verification software, and (e) sign the data usage agreement form (security & confidentiality agreement). Once access to the MIMIC-IV (v3.1) database was granted, all the metadata and files will be available for download in an organized fashion at the bottom of the dataset's page. The ZIP file for the dataset was 9.8 GB with a 9.9 GB total uncompressed size. One limitation of this research was the amount of storage space (RAM & Memory) available, making downloading a 9.9 GB database as a whole unattainable. By utilizing MIMIC-IV (v3.1)’s option to download individual files from the folders listed in MIMIC-IV’s database, downloading individual files needed in the creation of the ML model instead of the entire dataset would be an effective way to bypass research limitations. The following files were downloaded from the dataset MIMIC-IV (v3.1): (a) admissions.csv.gz, (b) d_labitems.csv.gz, (c) diagnoses_icd.csv.gz, (d) labevents.csv.gz, and (e) patients.csv.gz. The following files were downloaded from the dataset MIMIC-IV-Note (Deidentified free-text clinical notes): (a) discharge.csv.gz and (b) discharge_detail.csv.gz. The process of downloading each file individually rather than the entire ZIP file saved more than 61% of the storage space, reduced RAM (Random Access Memory) usage, and saved processing time. After the needed data files from the MIMIC-IV database were properly downloaded (~3.5 Hours) and stored in an organized manner in a secure folder named “IBS_Crohns_Research”, the process of ETL (Extract, Transform, and Load) was followed by utilizing software through Python. Firstly, Pandas (data manipulation) and NumPy (array operations) software were installed in the terminal and imported through the code. Pandas’ read feature was used to load the CSV files of the following: Diagnoses, for patient ICD codes, Patients, for demographics, Admissions, to connect patient IDs to hospital visits, and D_labitems, to link lab IIDs to test names. Pandas was then used to simultaneously load and merge the following files: Discharge (text) and Discharge_Detail (metadata). The files were merged into a single dataframe and a Subject ID so the medical notes could be properly linked to each patient. Then, all the patients were filtered through the inclusion/exclusion criteria of having a minimum value of 90% of data complete and not containing outlying, skewed data values. The patients were then separated based on the ICD-10 diagnosis codes to obtain only patients with IBD (K58) and Crohn’s Disease (K50). From there, binary labels for disease classification were assigned through Pandas by assigning the integer 0 for IBS and 1 for Crohn’s Disease. All the patient’s labels were then combined to initiate a ‘labels’ DataFrame to be used as a future target variable. Then, Pandas was used to load and filter the laboratory events with a chunk-based format due to the large file size (2.59 GB). The chunk size optimized for the data file was 10^6 (1,000,000 lines) of data at a time. The loading loop consisted of each chunk being filtered to only include IBS and Crohn’s Disease patients based on ICD codes (K58 & K50), concatenation of the filtered results occurred into the ‘labs’ DataFrame, and the next chunk was loaded into the program. From there, each lab was merged with its matching textual names based on its SubjectIDs from the d_labitems file. Then, Pandas and NumPy were used to clean ‘valuenum’ (numerical result of lab-test or measurement) and drop NaNs (Not a Numbers). The ‘valuenum’ measure was generated by taking the average of all laboratory tests conducted. From there, the data table was pivoted from a lab-based table (raw data table) to a patient-based table by grouping all labs by patient, with each row containing a unique patient through merging common patient IDs, and each column containing a separate lab test. Lastly, Pandas was used to filter the discharge notes for IBS and Crohn’s Disease patients, group notes into a patient-based table, and concatenate notes into a single text string per patient for the vectorization process. After the data extraction, filtration, and organization were complete, the optimization of data preprocessing began. Firstly, the vectorization of the clinical notes was initiated through the ClinicalBERT tokenizer and model, which generated mean-pooled embeddings and merged the vectorized clinical notes and lab features with labels. Vectorization was essential in the development of the ML model, as raw natural language processing (NLP) models operate through numerical representations called embedding rather than string-form text. ClinicalBERT, a BERT model fine-tuned on clinical text data and medical notes, was chosen over the built-in TF-IDF model as it had named entity recognition (NER) and medical natural language inference (MedNLI), giving it the ability to classify medical entities (diseases, medications, and lab values) and determine the relationship between 2 clinical statements. In contrast, TF-IDF lacks functionality as it merely tracks the frequency of the words instead of assessing the text through a deeper analysis. After vectorizing the text data with ClinicalBERT, the numerical embedding was combined with structured features from each patient’s laboratory results, which were compressed using principal component analysis (PCA) to reduce dimensionality by only including the most important dimensions. Once the data preprocessing ended, the data was split for ML model training. The golden split ratio of data in medical applications was applied, partitioning 70% for initial training, 15% for model tuning, and 15% for final evaluation. The ML model was then defined through empirical testing as a sequential model, where training data is processed in a specific order, consisting of 4 layers with 1, 64, 128, and 256 corresponding neurons on each layer. After initial testing to analyze data quality and quantity, it was found that there were 2,126 Crohn’s disease patients included in the dataset, but only 1,367 IBS patients, highlighting an imbalance of data, potentially causing biased model predictions with poor performance on the minority side. To reduce these potential roadblocks, class weights were added to handle imbalance through Sklearn’s weight module, which automatically calculates optimized weights through the “compute_class_weight” built-in function. After class weights were appended in the model training, more testing discovered moderately high levels of overfitting in the model’s training, when the model learns the training data too well (including its noise) and fails to generalize to new & unseen data. The overfitting was likely caused by the complex similarities between Crohn’s disease and IBS, insufficient training data to cover all aspects of diagnosis, and overtraining. In all, four advanced techniques, including L2 regularization (ridge regression), dropout, Leaky ReLU, and early stopping, were utilized to minimize overfitting, ultimately increasing the accuracy and boosting performance metrics in the model. L2 regularization was applied with the following cost equation: Loss + λ * (1/2) * Σ(w_i)^2 The “loss” represents the original loss function of the model (mean squared error and cross-entropy loss), the “λ” (lambda) is a hyperparameter that controls the strength of the regularization, and “Σ(w_i)^2” is the sum of the squares of all the model's weights (coefficients). L2 regularization, dropout, and Leaky ReLU were applied on layers 2-4 with a penalty term of 0.001 and a dropout rate of 0.5 on each layer. Early stopping was also used in the model training with a patience value of 10, implying that if the validation loss hadn’t increased in the previous 10 epochs, then the training would automatically cease. Once the initial ML model training occurred, the model was tuned on the validation set and evaluated on the test set. After the ML model was developed, Bayesian optimization was implemented through hyperparameter tuning, efficiently finding optimal hyperparameters for ML models and constructing a probabilistic model. Bayesian optimization ran a total of 50 epochs with a variety of hyperparameters being evaluated on each set, allowing for enhanced model performance. After the optimization is complete, all 50 tuned models are filtered through a certain performance metric, area under the curve (AUC), and the model with the finest performance on that metric will be saved as the final model. Through all sections of the research, from the model training, tuning, evaluating, and optimizing, adequate graphs were collected through the Matplotlib package, a comprehensive library for creating static and interactive visualizations in Python. The final model was saved in a “.keras” file, and all post-processed data, analytical results, and ML models were stored in their dedicated research repositories/folders. IV. RESULTS A total of 3,493 patient records (2,126 Crohn's, 1,367 IBS) were included in the development of the ML model. The model was trained using 70% of patient data, with 15% splittage for both tuning and final evaluation. The model was optimized to reduce diagnostic bias and overfitting and was fine-tuned with 53 epochs of Bayesian Optimization with Hyperparameter Search. TABLE 1 Model Metrics (Accuracy, F1-score, and AUC) Fine-tuned ML Model Metrics: Accuracy, F1-score, and AUC, signifying high plausibility for Predictive AI to enter the Medical Field of Gastroenterology The final ClinicalBERT + PCA + L2 regularization + Bayesian optimization + deep neural network (DNN) model achieved an overall accuracy of 74.41%, F1-score of 73.40%, and an AUC of 79.56%. The model was filtered for AUC during Bayesian optimization, but was previously set to accuracy. The highest accuracy rating the model received on the test set was 78%. FIGURE 1 Bayesian Optimization: Model Performance Comparison Bayesian Optimization Individual Epoch Metric Comparison: The bar-graph shows the trials in order of computer operations, with the initial model being 1st and the model with the highest AUC being last. FIGURE 2 Bayesian Optimization: Model ROC AUC over Trials Bayesian Optimization ROC AUC over Trials: lineplot on the metrics of Bayesian Optimization’s corresponding effects on the AUC of the model over time as trials continue, with the orange dotted line keeping track of the model with the Highest ROC AUC. VI. DISCUSSION The ML model, which differentiates Crohn’s disease and IBS, had an accuracy of 74.41%, an F1-score of 73.40%, and an AUC of 79.56% (Table 1). Comparing the model’s AUC, the metric used to distinguish between 2 classes (Crohn’s & IBS), to the global average diagnostic accuracy rate (90%), we can see that the model generated through this case-study doesn’t empirically align with the global average, with roughly a 10% diagnostic gap between the 2 diseases. Although the diagnostic gap is apparent, we must acknowledge its primary usage in underdeveloped countries and how it serves as a low-cost, highly available, and accessible alternative to traditional diagnosis. Roughly 55% of Crohn’s disease patients in underdeveloped countries are unable to afford diagnostic testing (52.4% in Asia, 60.9% in Africa), 0.12 endoscopists, 0.12 gastroscopes, and 0.09 colonscopes available per 100,000 inhabitants, and an average gastroenterologist density of 0.005 to 0.02 per 100,000 inhabitants (commonly 1 per country in low-income regions). It’s reasonable to infer that the impact that the ML model would make on the inhabitants of underdeveloped countries, such as giving ~55% of patients the ability to afford low-cost diagnosis, and allowing hundreds of thousands of patients to have access to diagnostic methods. One potential issue one may notice is that the model’s diagnostic accuracy is lower than the global average. Still, when focusing on underdeveloped regions and their lack of data on the misdiagnosis rate between Crohn’s disease and IBS, the misdiagnosis rate is certainly illuminated in underdeveloped regions compared to the global average. A majority of patients may not be able to receive a diagnostic analysis due to cost or availability-related burdens, which the implementation of the ML model would solve. In the end, it’s safe to imply that the application of the AI model in underdeveloped regions where patients actively face unaffordable costs and the scarcity of diagnostic equipment/gastroenterologists heavily outweighs the diagnostic accuracy difference between the model and the global average. Although the model may be limited by the quality of the dataset in use and the quantity of the data, as a large amount of data is needed to train an ML model on a complex medical intersection such as Crohn’s disease and IBS, the finished AI model is still exceptionally applicable in real-world applications in underdeveloped regions. Future work may be made to gather a wider range of data with finer quality and quantity to reduce overfitting, and re-train the model using DNNs to boost the ROC AUC of the model or assess the current model’s implications when applied to real-world scenarios. Overall, these results highlight the potential application of ML-powered AI models as an alternative diagnostic method to traditional Crohn’s disease and IBS diagnosis. VII. CONCLUSION This case study developed and analyzed an ML model that combined ClinicalBERT embeddings of clinical notes with PCA-reduced lab data to classify the distinction between Crohn’s disease and IBS using L2 regularization, dropout, Leaky ReLU, early-stopping, and Bayesian optimization, achieving 79.56% (~80%) AUC and an accuracy of 74.41%. The model solves critical issues powering the misdiagnosis crisis between Crohn’s disease and IBS in underdeveloped countries by providing a low-cost and highly available solution to traditional diagnostic methods, which require the use of expensive laboratory testing, scarcely available equipment, and specialized medical professionals (gastroenterologists). This research presents a case study that demonstrates the potential of utilizing ML models in the gastrointestinal intersection between Crohn’s disease and IBS to reduce misdiagnosis rates and increase diagnostic availability for more affected patients. Such models form the basis of modernized clinical decision support tools, which significantly improve patient outcomes. Declarations Ethical Considerations This study utilized data from the Medical Information Mart for Intensive Care IV (MIMIC-IV) database. The MIMIC-IV database contains fully de-identified patient data and was approved by the Institutional Review Boards of the Massachusetts Institute of Technology (MIT) and Beth Israel Deaconess Medical Center (BIDMC). As all data were de-identified and publicly available to credentialed users under a data use agreement, this study did not involve human subjects research as defined by U.S. federal regulations, and additional institutional review board (IRB) approval was not required. Conflicts of Interest Varun Kurra declares no conflicts of interest. Author Contributions Varun Kurra gathered and extracted the data, developed Predictive ML models, conducted the case study, interpreted the results, and wrote the manuscript. Funding This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Data Availability The data used in this study were obtained from the dataset MIMIC-IV (v3.1), available publicly through PhysioNet. Credentialled registration and CITI training are pre-requisites to obtaining access to the dataset. No new data were generated through this case-study. References Warner D (2025) IBS vs. Crohn’s disease: What to know. Medicalnewstoday.com. Published March 19, 2024. Accessed August 4. https://www.medicalnewstoday.com/articles/ibs-vs-crohns Cross E, Saunders B, Farmer AD, Prior JA (2023) Diagnostic delay in adult inflammatory bowel disease: A systematic review. Indian J Gastroenterology: Official J Indian Soc Gastroenterol 42(1):40–52. https://doi.org/10.1007/s12664-022-01303-x New study finds delayed diagnosis of inflammatory bowel disease is linked to worse clinical outcomes (2023) crohnsandcolitis org uk Published January 13, https://crohnsandcolitis.org.uk/news-stories/news-items/new-study-shows-delayed-diagnosis-of-inflammatory-bowel-disease-is-linked-to-worse-clinical-outcomes Cleveland Clinic (2023) Crohn’s disease. Clevel Clin Published April 12, https://my.clevelandclinic.org/health/diseases/9357-crohns-disease Clinic Staff M (2024) Irritable bowel syndrome - Symptoms and causes. Mayo Clinic. Published October 11. https://www.mayoclinic.org/diseases-conditions/irritable-bowel-syndrome/symptoms-causes/syc-20360016 Chey WD, Kurlander J, Eswaran S (2015) Irritable bowel syndrome: A clinical review. JAMA 313(9):949. https://doi.org/10.1001/jama.2015.0954 Berg S (2023) What doctors wish patients knew about irritable bowel syndrome. American Medical Association. Published June 23. https://www.ama-assn.org/delivering-care/prevention-wellness/what-doctors-wish-patients-knew-about-irritable-bowel-syndrome Clinic ACE, Clinic ACE (2025) Recognizing and Dealing with a Crohn’s Flare-Up | Chandler, Arizona. Arizona Colorectal Experts. Published March 29, 2024. Accessed August 5. https://aceclinic.org/recognizing-and-dealing-with-a-crohns-flare-up/ Admin (2021) Colorectal Clinic of Tampa Bay. Colorectal Clinic of Tampa Bay. Published January 22, Accessed August 5, 2025. https://www.tampacolorectal.com/blog/how-to-calm-an-ibs-flare-up Ford AC (2020) Overlap Between Irritable Bowel Syndrome and Inflammatory Bowel Disease. Gastroenterol Hepatol 16(4):211. https://pmc.ncbi.nlm.nih.gov/articles/PMC8132685/ Burns E (2021) Doubling Up? IBS Symptoms in Patients With IBD in Remission - Ulcerative Colitis Peer to Peer. Medpagetoday.com. Published August 12, Accessed August 5, 2025. https://www.medpagetoday.com/resource-centers/ulcerative-colitis-crohns-disease/doubling-up-ibs-symptoms-patients-ibd-remission/3418 Jayasooriya N, Baillie S, Blackwell J et al (2023) Systematic review with meta-analysis: Time to diagnosis and the impact of delayed diagnosis on clinical outcomes in inflammatory bowel disease. Aliment Pharmacol Ther 57(6). https://doi.org/10.1111/apt.17370 New study reveals 1 (2014) in 10 Inflammatory Bowel Disease patients are misdiagnosed with Irritable Bowel Syndrome. ueg eu Published Dec 15, https://ueg.eu/a/201 Card TR, Siffledeen J, Fleming KM (2014) Are IBD patients more likely to have a prior diagnosis of irritable bowel syndrome? Report of a case-control study in the General Practice Research Database. United Eur Gastroenterol J 2(6):505–512. https://doi.org/10.1177/2050640614554217 Rajbhandari R, Blakemore S, Gupta N et al (2020) Crohn’s disease in low and lower-middle income countries: A scoping review. World J Gastroenterol 26(43):6891–6908. https://doi.org/10.3748/wjg.v26.i43.6891 Ruma Rajbhandari, Blakemore S, Gupta N et al (2022) Crohn’s Disease Among the Poorest Billion: Burden of Crohn’s Disease in Low- and Lower-Middle-Income Countries. Dig Dis Sci 68(4):1226–1236. https://doi.org/10.1007/s10620-022-07675-6 Gaur A, Bharadwaj HR, Dalal P, Ahmed M (2024) Recent progress and future directions for expanding gastrointestinal endoscopy in low- and middle‐income African nations. JGH Open 8(9). https://doi.org/10.1002/jgh3.70026 Sharing my (2022) story is a superpower: the doctor living without a cure in Ethiopia. Guardian Published Oct 27, https://www.theguardian.com/global-development/2022/oct/27/doctor-fasika-teferra-superpower-hope-crohns-ethiopia-acc Schönborn C, Levy M, Jaeger MD et al (2025) Unmet health-related needs in patients with Crohn’s disease in Belgium: a mixed-methods study. Archives Public Health 83(1). https://doi.org/10.1186/s13690-025-01632-1 Current Research Initiatives Crohn’s & Colitis Foundation. https://www.crohnscolitisfoundation.org/research/current-research-initiatives AIPRM. 50 + AI in Healthcare Statistics (2024) Aiprm.com. Published July 8, 2024. https://www.aiprm.com/ai-in-healthcare-statistics/ Grand View Research (2023) https://www.grandviewresearch.com/industry-analysis/artificial-intelligence-ai-healthcare-market Guo Y, Hao Z, Zhao S, Gong J, Yang F (2020) Artificial Intelligence in Health Care: Bibliometric Analysis. J Med Internet Res 22(7):e18228. https://doi.org/10.2196/18228 Sahni N, Stein G, Zemmel R, Cutler DM (2023) The Potential Impact of Artificial Intelligence on Healthcare Spending. Natl Bureau Economic Res Published January 1, https://www.nber.org/papers/w30857 Ghezzi M, How AI in Healthcare could Save over 250,000 Lives Each Year and become a $ 188 Billion Market by 2030. The Journal of mHealth. Published November 6, 2023. https://thejournalofmhealth.com/how-ai-in-healthcare-could-save-over-250000-lives-each-year-and-become-a-188-billion-market-by-2030/ Nsengiyumva NP, Hussain H, Oxlade O et al (2021) Triage of Persons With Tuberculosis Symptoms Using Artificial Intelligence–Based Chest Radiograph Interpretation: A Cost-Effectiveness Analysis. Open Forum Infect Dis 8(12). https://doi.org/10.1093/ofid/ofab567 Kakileti ST, Madhu HJ, Krishnan L, Manjunath G, Sampangi S, Ramprakash HV (2020) Observational Study to Evaluate the Clinical Efficacy of Thermalytix for Detecting Breast Cancer in Symptomatic and Asymptomatic Women. JCO Global Oncol 61472–1480. https://doi.org/10.1200/go.20.00168 Karthik Adapa, Gupta A, Singh S et al (2025) A real-world evaluation of an innovative artificial intelligence tool for population-level breast cancer screening. npj Digit Med 8(1). https://doi.org/10.1038/s41746-024-01368-2 Additional Declarations The authors declare no competing interests. Supplementary Files FinalAccuracyPlot.png Final Predictive Model Accuracy Plot FinalLossPlot.png Final Predictive Model Loss Plot L2LossPlot.png L2 Regularization Loss Plot Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9658515","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Case Report","associatedPublications":[],"authors":[{"id":637146415,"identity":"fd7579f8-e5eb-4133-ae44-7e6cb6f0813e","order_by":0,"name":"Varun Bhargav Kurra","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABOUlEQVRIie2RMUvDQBSAIwdxebWOF1KSv3DlIB3M5h9JKCRLIoJLNgNCXCKu18m/UBddHE4Cdol7QAeLkClDQBALHTxTh0pSdRS8D473HtzHvfdOUSSSPwheBdtQzYyLxGlKJM7gB8Wjfaw6XxRoMr5RyVyNAfmdorGwrBdJ5pCH9PVpceO558yfvxxGNphx7/a5bis69unkLPEPyOP99TAtA5cVAdVZ7gHhO2PS8YqBPaRAvndEeHiFgUdhXAQq6iWZ6BMsvEHZWubInfKg1JZCuSj8slHMGEZvHYouFATRvjthgaoDD8Jp4ViNonCwOsdPS4QGkVgy3I3ogHvHl3lFdfiYJQOK87aCZ6Kxioiv3D4p5xUfU2MmNgaRbZin6bCO2so66irsOp81+v76mtLv6F8ikUj+Ne/OxmmTS6N1bwAAAABJRU5ErkJggg==","orcid":"","institution":"Parkway West Highschool","correspondingAuthor":true,"prefix":"","firstName":"Varun","middleName":"Bhargav","lastName":"Kurra","suffix":""},{"id":637149501,"identity":"c92e374a-345f-4236-a032-224c2236c64f","order_by":1,"name":"Jennifer Otiono","email":"","orcid":"https://orcid.org/0000-0002-3837-3297","institution":"Cornell University","correspondingAuthor":false,"prefix":"","firstName":"Jennifer","middleName":"","lastName":"Otiono","suffix":""}],"badges":[],"createdAt":"2026-05-09 00:46:46","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-9658515/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9658515/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":109121060,"identity":"aed7b7a5-b10a-4ead-89d4-a578d008440f","added_by":"auto","created_at":"2026-05-12 17:19:24","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":53317,"visible":true,"origin":"","legend":"\u003cp\u003eBayesian Optimization: Model Performance Comparison\u003c/p\u003e\n\u003cp\u003eBayesian Optimization Individual Epoch Metric Comparison: The bar-graph shows the trials in order of computer operations, with the initial model being 1st and the model with the highest AUC being last.\u003c/p\u003e","description":"","filename":"Picture1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9658515/v1/4b9ec207a3bfe28341996e37.jpg"},{"id":109205000,"identity":"f0955f38-96fb-4b0a-bda8-05b15df0c42b","added_by":"auto","created_at":"2026-05-13 15:03:10","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":36841,"visible":true,"origin":"","legend":"\u003cp\u003eBayesian Optimization: Model ROC AUC over Trials\u003c/p\u003e\n\u003cp\u003eBayesian Optimization ROC AUC over Trials: lineplot on the metrics of Bayesian Optimization’s corresponding effects on the AUC of the model over time as trials continue, with the orange dotted line keeping track of the model with the Highest ROC AUC.\u003c/p\u003e","description":"","filename":"Picture2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9658515/v1/baa99379fccc31a655162289.jpg"},{"id":109207950,"identity":"a75325c3-5e86-4f66-88aa-5b3d15eea3a8","added_by":"auto","created_at":"2026-05-13 15:22:37","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":307006,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9658515/v1/369f9229-0b7f-4d76-8dac-12f7a0a1b63a.pdf"},{"id":109204855,"identity":"12b59cb5-b7f6-4aa7-88fa-8fb19f7b46e8","added_by":"auto","created_at":"2026-05-13 15:02:36","extension":"png","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":30966,"visible":true,"origin":"","legend":"\u003cp\u003eFinal Predictive Model Accuracy Plot\u003c/p\u003e","description":"","filename":"FinalAccuracyPlot.png","url":"https://assets-eu.researchsquare.com/files/rs-9658515/v1/1a202ea7cb14588788e02d5e.png"},{"id":109204990,"identity":"c5ce0613-e6c8-42d0-84af-0409af314c9d","added_by":"auto","created_at":"2026-05-13 15:03:07","extension":"png","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":26365,"visible":true,"origin":"","legend":"\u003cp\u003eFinal Predictive Model Loss Plot\u003c/p\u003e","description":"","filename":"FinalLossPlot.png","url":"https://assets-eu.researchsquare.com/files/rs-9658515/v1/15a364bb65b4b31a923331fc.png"},{"id":109121064,"identity":"5ba62571-d926-4caf-8899-490823e5cc24","added_by":"auto","created_at":"2026-05-12 17:19:24","extension":"png","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":155211,"visible":true,"origin":"","legend":"\u003cp\u003eL2 Regularization Loss Plot\u003c/p\u003e","description":"","filename":"L2LossPlot.png","url":"https://assets-eu.researchsquare.com/files/rs-9658515/v1/ab5686008822dea4a9c8f288.png"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eApplication of Machine Learning to Distinguish IBS from Crohn’s Disease in Underdeveloped Regions\u003c/p\u003e","fulltext":[{"header":"I. INTRODUCTION","content":"\u003cp\u003eCrohn’s disease, one of the two IBD diseases, and Irritable Bowel Syndrome share common symptoms, and both affect one’s digestive tract, making maintaining diagnostic accuracy challenging.\u003csup\u003e1\u003c/sup\u003e To assist the common misdiagnosis, the average diagnostic delays (correct diagnosis after misdiagnosis) can range from 2 months to 8 years in IBD, with Crohn’s disease averaging longer delays than Ulcerative Colitis, the 2nd IBD disease.\u003csup\u003e2\u003c/sup\u003e The diagnostic delay of Crohn’s disease leads to the necessity of stronger medication, hospital admissions, and expensive surgeries, imposing heavy cost burdens on those who can’t afford the cost.\u003csup\u003e3\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eThis research investigated whether AI models leveraging clinical notes and laboratory data can accurately distinguish between IBS and Crohn’s disease to improve diagnostic accuracy and reduce misdiagnosis. The hypothesis is that integrating clinical notes and laboratory data will improve diagnostic performance compared to traditional methods of diagnostic analysis. Through this research, we seek to demonstrate that AI can enhance early detection of gastrointestinal disorders and provide underdeveloped countries that lack medical diagnostic tools with a high-impact and low-cost solution for improving the misdiagnosis crisis between IBS and Crohn’s disease to prevent elongated diagnostic delays. The findings of this case study could inform healthcare systems, alter clinical guidelines, and serve as a foundation for the future application of AI-driven predictive diagnostic tools in the field of gastroenterology.\u003c/p\u003e"},{"header":"II. LITERARY REVIEW","content":"\u003cp\u003e\u003cstrong\u003eCrohn’s Disease and IBS Introduction\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eCrohn’s disease is a chronic autoimmune condition that inflames and irritates one’s digestive tract. Crohn’s disease and Ulcerative Colitis are the 2 primary forms of inflammatory bowel disease (IBD). Symptoms of Crohn’s disease may develop gradually or suddenly, and may be mild or severe. A “flare” is a term used in Crohn’s disease to describe when symptoms occur. The etiology of Crohn’s disease is unknown, but it’s widely believed that genetic predisposition and immune system irregularities contribute to the development of the disease. Crohn’s disease affects more than 750,000 people in the United States and approximately 6 to 8 million people globally.\u003csup\u003e4\u003c/sup\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIrritable bowel syndrome (IBS) is a common condition that affects the gastrointestinal tract, the stomach, and the intestines. Symptoms of IBS tend to be present for an extended period, making IBS an ongoing condition that needs long-term care. Similar to Crohn’s disease, the causes of IBS are unknown, but factors that reportedly contribute to the syndrome include muscle contractions in the intestine, issues with the nervous system, severe infections, early-life stress, and changes in gut microbes. The “flares” of IBS may be triggered by food and anxiety, although these triggers don’t initially cause the symptoms to occur, but rather enhance their effect.\u003csup\u003e5\u003c/sup\u003e A 2015 clinical review conducted by Chey et al.\u003csup\u003e6\u003c/sup\u003e concluded that approximately 12% (7% to 21%) of people globally, with roughly 24-45 million people suffering from IBS. Although IBS affects 10%-15% of adults in the United States, only 5%-7% of adults are initially diagnosed with the syndrome. IBS is also known to affect more women, with twice as many women obtaining the disease as men annually.\u003csup\u003e7\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eIBS and Crohn’s disease are both known for their “flare-remission” cycle, with “flare-ups” signifying the period of increased symptom severity. “Flare-ups” are when symptoms of the disease become more noticeable after a period of remission, with its causation relating to a dysfunctional immune response where one’s immune system attempts to “attack” germs that enter the body, but uncontrollably harms bodily tissues instead. Inflammation happens when an autoimmune response occurs, when the immune system “attacks” one’s body cells inappropriately while attempting to remove germs from the body. Inflammation is a central signal that the “attack” is ongoing and the “Flare-up” is active. Once the bodily threat is gone, the immune system flows into a state of repair, and inflammation may begin to lessen, but in certain cases, inflammation doesn’t completely heal. The “flare-ups” in both conditions are typically short-term but vary in duration, lasting a few hours to weeks in IBS and from a few days to months in Crohn’s disease.\u003csup\u003e8,9\u003c/sup\u003e During periods of remission, symptoms are reduced or absent, but the underlying disease is still present.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAlthough both IBS and Crohn’s disease follow a “flare-remission” pattern, they vary significantly in their nature and underlying fundamentals. Crohn’s disease is an organic IBD characterized by visible inflammation, ulceration, and structural tissue damage commonly found in the ileocolon. Unlike Crohn’s disease, IBS is a functional disorder, classified as a “syndrome”, meaning that patients experience real symptoms but no observable damage occurs. Patients with IBS don’t experience inflammation or structural bowel damage, which patients with Crohn’s disease do.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eOverlapping Symptoms and Delays\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIBS and IBD, including Crohn’s disease and Ulcerative colitis, have an overlap of apparent symptoms, making patients susceptible to frequent misdiagnosis. The overlapping symptoms occur as both conditions affect the gastrointestinal tract, which has a narrow repertoire of symptoms and a limited range of ways it can manifest bacteria from diseases. As a result, multiple gastrointestinal conditions, such as IBS and Crohn’s disease, may appear with the same clinical features. Patients with either disorder may experience changes in bowel habits, typically involving diarrhea along with pain, bloating, and rectal mucus discharge. A systematic review by Halpin and Ford (2012) analyzed studies of patients with IBD who had answered symptom questionnaires for IBS. Their findings revealed that approximately 35% to 40% of patients with IBD reported IBS-related symptoms, and patients with IBD were ~5 times more likely to have symptoms compatible with IBS than healthy controls.\u003csup\u003e10\u003c/sup\u003e Their studies also concluded that the prevalence of IBS-type symptoms in IBD was slightly higher in Crohn’s disease than in Ulcerative colitis, with the odds ratio being 1.62, meaning that patients with Crohn’s disease had a 62% higher chance of reporting IBS-compatible symptoms. One might assume that the underlying symptoms persist during remission, allowing for an accurate and clear diagnosis. However, this isn’t the case, as Dr. Keeley M Fairbrass of the Leeds Gastroenterology Institute explains, “IBS-type symptoms co-exist in IBD patients even during periods of deep remission.”\u003csup\u003e11\u003c/sup\u003e A 2020 meta-analysis conducted by Ford\u003csup\u003e11\u003c/sup\u003e of 27 studies (15 Crohn’s Disease) published between January 1, 2012, and May 11, 2020, which included 3169 patients with IBD in remission, concluded that IBS-type symptoms among patients with IBD who were in remission were 32.5%, ranging from 11.2% to 63.6%. The analysis also found that among the 15 studies of patients with Crohn’s disease in remission, the prevalence of IBS-type symptoms was 36.6%, averaging 4.1% higher than in IBD patients as a whole. In conclusion, the prevalence of IBS-type symptoms occurs in ~35% to 40% of patients with IBD, and 32.5% of patients with IBD in remission, with Crohn’s disease patients averaging a 63.6% higher likelihood of obtaining IBS-type symptoms than Ulcerative colitis, providing a solid basis for frequent misdiagnosis.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWith IBS and Crohn’s disease patients obtaining overlapping symptoms, diagnostic delays prevailed. A 2023 meta-analysis conducted by Jayasoorya et al.\u003csup\u003e12\u003c/sup\u003e of 101 EMBASE [Excerpta Medica dataBASE] and Medline studies representing over 112,194 patients with IBD (CD = 59,359; UC = 52,835) concluded that the median time to Crohn’s disease diagnosis was 8 months (Interquartile Range [IQR]: 5 months to 15.2 months). In high-income countries, this was 6.2 months (IQR: 5 months to 12.3 months) compared to 11.7 months (IQR: 8.3 months to 18 months) in low-income countries. Another study consisting of 46 provider surveys and 15 low-income countries reported that most patients had symptoms of Crohn’s disease 6-24 months before diagnosis, and 26% to 50% of patients lived in rural areas, exacerbating the lack of medical resources and efficiencies in rural, low-income provinces. In conclusion, patients with Crohn’s disease in low-income countries experience an average diagnostic delay roughly 2x longer than in high-income settings.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMisdiagnosis: Crises between Crohn’s Disease and IBS\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe most significant cause of diagnostic delays is widespread misdiagnosis. A 2014 case-controlled study published in the United European Gastroenterology (UEG) journal, consisting of over 20,193 IBD cases, revealed that IBD patients were more than 3 times as likely as controls to have a prior record of IBS. 15% of IBD cases compared to 5% of controls had IBS encoded in their medical records before diagnosis, with 11% of IBD cases (cf. 5% of controls) having the code of IBS for over a year before IBD, and over 6% of cases (cf. 3% of controls) maintaining the disease for over 5 years. The study concluded that globally, approximately 10% of IBD patients are initially misdiagnosed with IBS, and in the case of 3% of patients, the misdiagnosis persisted for 5 or more years.\u003csup\u003e13,14\u003c/sup\u003e Along with the 10% risk of initial misdiagnosis, a large longitudinal study analyzing 9,341 incident cases found that individuals with IBS had an 8.6-fold increased risk of receiving an IBD diagnosis compared to non-IBS controls (238.1 vs. 27.8 cases per 100,000 person-years, p \u0026lt; 0.0001). In a subset of well-defined IBS cases, the risk increased to 15 times that of the non-IBS cases, highlighting the potential of initial misdiagnosis and early undetected IBD among IBS patients. Overall, the global rate of Crohn’s disease to IBS misdiagnosis can be stated as 10%, with 3% of total patients being misdiagnosed for 5 years or more. Additionally, IBS patients have an 8.6-15 times higher risk than non-IBS patients of obtaining IBD, highlighting a significant and ongoing misdiagnosis crisis.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMisdiagnosis: Crises in Underdeveloped Countries\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe misdiagnosis crisis is a notable burden in high-income countries. Still, the situation turns to catastrophe in low-income nations due to unaffordable costs and scarcity of diagnostic tools and medical professionals. Although the crisis is exacerbated in low-income countries by a scarcity of equipment/professionals and unaffordable costs, concrete statistical data on Crohn’s disease misdiagnosis as IBS and diagnostic delay remain unavailable. A scoping review of 4486 publications conducted by Rajbhandari et al.\u003csup\u003e15\u003c/sup\u003e from 79 LLMICs (Low and Lower-Middle-Income-Countries) revealed that only 21 LLMICs have publications on Crohn’s disease, indicating major under-diagnosis or under-reporting and outlining the scarcity of statistical data in underdeveloped regions regarding Crohn’s disease.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;On top of the statistically high Crohn’s disease to IBS misdiagnosis rate of 33.5%, estimated through a statistical analysis of Crohn’s disease to IBS ratios, factors such as diagnostic affordability and resource availability also make an impact on underdeveloped countries. Firstly, regarding diagnostic cost, a cross-sectional survey of gastroenterology providers from 15 underdeveloped countries conducted by Ng et al.\u003csup\u003e16\u003c/sup\u003e reported that 52.4% of Crohn’s disease patients in Asia and 60.9% of patients in Africa could not afford diagnostic testing to distinguish between Crohn’s disease and other infectious diseases. Secondly, addressing the scarcity of diagnostic resource availability, analyzing Africa’s metrics reveals a shortfall in equipment and specialists. It should be noted that the gold-standard method of Crohn’s disease diagnosis is endoscopy, including colonoscopy and ileoscopy. A 2022 report on the progress of expanding gastrointestinal endoscopy in LLMIC African nations, conducted by Rajhbandari et al.\u003csup\u003e17\u003c/sup\u003e revealed that Eastern Africa’s endoscopy capacity consisted of approximately 0.12 endoscopists, 0.12 gastroscopes, and 0.09 colonoscopes available per 100,000 inhabitants, showcasing a stark deficit of diagnostic equipment. Addressing the aspect of the availability of medical specialties, the survey also disclosed that in several underdeveloped African countries, such as Malawi and Rwanda, there was less than 1 medically trained gastroenterologist in each country, serving more than 10 million inhabitants, and fewer than 11 endoscopy centers, showing the extreme specialist shortage in LLMICs, especially sub-Saharan Africa.\u003csup\u003e16\u003c/sup\u003e Similarly, Ruma Rajbhandari, a Harvard professor and lead Crohn’s disease researcher, stated that “researchers found ‘very few gastroenterologists – less than 20’ for Ethiopia, which has a population of more than 115 million.”\u003csup\u003e18\u003c/sup\u003e Overall, through a statistical analysis of the quantity of gastroenterologists operating in Africa, a generalizable average gastroenterologist density of 0.005 to 0.02 per 100,000 inhabitants, or 1 specialist per 5-20 million inhabitants, can be established. In conclusion, the scarcity of resources and specialists is widely apparent in African LLMIC nations, with 0.12 endoscopists, 0.12 gastroscopes, and 0.09 colonscopes available per 100,000 inhabitants, and an average gastroenterologist density of 0.005 to 0.02 per 100,000 inhabitants being generalizable metrics between underdeveloped nations. The following scarcity of resources and specialists highlights the leading cause of an inflated misdiagnosis rate of 33.5% between Crohn’s disease and IBS in underdeveloped countries.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePredictive AI Introduction\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;With nations desperately seeking alternative, available, and low-cost diagnostic solutions\u003csup\u003e19,20\u003c/sup\u003e, Artificial Intelligence’s (AI’s) reputable background in the field of predictive healthcare stands as the most comprehensive and promising approach to revolutionize early detection in underdeveloped regions. AI has experienced an unexpected surge in healthcare over the past 2 decades due to the development of Machine Learning (ML) and Deep Learning (DL), a subset of ML, the digitization of healthcare records, and a quantifiable rise in medical data. The global AI in the healthcare market, once valued at $1.1 billion in 2016, grew to $22.4 billion in 2023, representing a rise of approximately 1,779%. Between the years of 2022 and 2023 alone, the market grew by over 45%, from $15.4 billion to $22.4 billion.\u003csup\u003e21\u003c/sup\u003e Currently, the global AI in healthcare market was valued at roughly $26.57 billion in 2024 and is projected to reach $187.69 billion by 2030, growing at a compound annual growth rate (CAGR) of over 38.62% from 2025 to 2030, outlining AI’s rapid integration into the healthcare sector.\u003csup\u003e22\u003c/sup\u003e A 2020 longitudinal bibliometric analysis of healthcare-related publications conducted by Guo et al.\u003csup\u003e23\u003c/sup\u003e revealed that the growth of research papers on healthcare-related AI significantly increased to 45.15% from 2014 to 2019, compared to the average of 17.02% per year since 1995. Fueling the rise of AI in modern healthcare, predictive AI, utilizing ML algorithms to analyze medical data, has risen in popularity and is currently being applied in early disease detection, patient outcome prediction, and resource optimization. The utilization of AI in healthcare allows for increased diagnostic accuracy, cost savings, and accessibility. It’s estimated that the wider adoption of AI in healthcare could lead to savings of 5% to 10% in US healthcare spending, around $200 to $360 billion annually.\u003csup\u003e24\u003c/sup\u003e Along with increased cost savings, it’s estimated that AI applications reduce more than 86% of errors made by healthcare workers, saving more than 250,000 lives per year.\u003csup\u003e25\u003c/sup\u003e AI can be utilized in regions where medical professionals are scarcely available, especially when it comes to diagnostic specialties. In the cases of many underdeveloped regions, AI-based predictive ML models have been extensively tested and evaluated in the cases of Tuberculosis (TB) and Breast Cancer screenings. A cost-effectiveness analysis conducted by Nsengiyumva et al.\u003csup\u003e26\u003c/sup\u003e of TB screening from chest radiograph (x-ray) interpretation using AI in Pakistan and Bangladesh observed the triage for TB in underdeveloped, high-burden programs using 5 AI models on 23,954 patients in Dhaka, Bangladesh. The usage of the AI model resulted in the following metrics: AUC up to 90.8% (90% sensitivity), qXR specificity of 74.3%, and CAD4TB of 72.9%, meeting WHO triage targets. Triage strategies with AI-based CXR analysis were projected to lower costs by 19% to 37% and average 3% to 4% Disability-Adjusted Life Years (DALYs), signifying major economic impact and enhanced diagnostic rates upon usage of the predictive AI model. Similarly, an observational study conducted by Kakileti et al.\u003csup\u003e27\u0026nbsp;\u003c/sup\u003eon the clinical efficacy of Thermalytix on detecting Breast Cancer in 470 (a)symptomatic women revealed that the predictive-AI tool resulted in an overall AUC of 0.90, a sensitivity of 91.02%, and a specificity of 82.39%, outlining its strong performance. Correspondingly, a real-world evaluation conducted by Adapa et al.\u003csup\u003e28\u0026nbsp;\u003c/sup\u003eof Thermalytix for population-level Breast Cancer screening in 183 locations in Punjab revealed that screening by Thermalytix delivered an age-adjusted incidence (AAI) of 120.94/100,000 (Crude rate: 0.18%), a threefold improvement compared to the NPCDCS recommended CBE-based screening, which resulted in a nationwide (Punjab) AAI of 31.2/100,000 (Crude rate: 0.03%) and between 38-48/100,000 (Crude rate: 0.04–0.05%) in 2022. Overall, the Thermalytix AI-based Breast Cancer screening tool’s metrics distinguish the fact that predictive AI models can provide low-cost and highly effective solutions to medical diagnostic issues.In underdeveloped regions where financial burdens, a lack of equipment, and medical inaccuracies hinder inhabitants from receiving proper medical care, the implementation of modern AI technologies stands as an effective solution.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; In summary, in underdeveloped regions, commonly LLMICs, the accurate diagnosis and distinction between Crohn’s disease and IBS is hampered due to high costs, unavailability of diagnostic tools, and a lack of medical professionals, heavily lowering diagnostic accuracy and increasing chances of misdiagnosis. These implications make it clear that an alternative diagnostic method for differentiating and diagnosing Cohn’s disease and IBS is needed in underdeveloped regions, ultimately reducing financial burdens, removing the heavy need for medical professionals, and saving lives. AI’s application to the Crohn’s disease and IBS misdiagnosis crisis in underdeveloped regions is an essential step in rectifying existing inadequate diagnostic methods. This research focuses on training, tuning, and evaluating an ML model to differentiate between Crohn’s disease and IBS using clinical notes and laboratory data to be applicable in underdeveloped regions.\u0026nbsp;\u003c/p\u003e"},{"header":"III. DATA \u0026 METHODOLOGY","content":"\u003cp\u003eTo build the IBS and Crohn\u0026rsquo;s Disease detection ML model, assessments of medical-grade datasets were conducted to align with rigorous benchmarks. Criteria for the database were established as the industry standards for modern-day datasets. The criteria implemented in the dataset assessments are data (a) quality, (b) structure, (c) usability, and (d) patient privacy.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAfter the criteria were determined, the web was searched for datasets that not only aligned with the criteria but also had noticeable recognition in the inflammatory bowel disease research sector. The database of MIMIC looked to be the most notable, specifically the dataset MIMIC-IV (v3.1) and its subset, MIMIC-IV-Note (De-identified free-text clinical notes). MIMIC-IV (v3.1) is a large, open-sourced medical database developed by the MIT Laboratory for Computational Physiology (MLCP) in alliance with Beth Israel Deaconess Medical Center (BIDMC). The database contains de-identified data from both intensive care unit (ICU) and medical-surgeon care (Hospital) patients admitted between 2008 and 2019. The dataset has over 383,220 unique patients, 94,458 ICU stays, and 546,028 hospital admissions. The MIMIC-IV (v3.1) database aligns with the assigned data criteria in all 4 sections. In the data quality sector, the database had internationally encoded procedures and diagnoses following ICD (International Classification of Diseases) and CPT (Common Procedure Terminology) regulations, along with having consistent data validation and longitudinal records of patients. In the structure sector, the data was organized into multiple tables and matrices with labeled metadata, along with the use of standard terminologies such as ICD, LOINC, and CPT codes. In the usability sector, the database was publicly available, had heavy documentation and support from PhysioNet, and was compatible with modern-day software (Ex, Pandas \u0026amp; Tensorflow). Finally, in the patient privacy sector, it followed the regulated de-identification process by removing 18 identifiers mandated by HIPAA, applied data shifting (same ranges but different time frames), and required the completion of human research training and the signing of a data usage agreement. Overall, this database excelled at all measures of the criteria established for this research and was chosen as the database that would be utilized for further continuation.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAfter the process of assessing databases was complete, MIMIC-IV (v3.1) was secured as the database in use for the entirety of this research, and the data-application process began. The following steps were taken to gain access and authorization for this database: (a) create a PhysioNet account \u0026amp; agree to the terms of use, (b) fill in the application form to be a credentialed user on the platform, (c) complete the required CITI program training (\u0026ldquo;Data or Specimens only Research\u0026rdquo;) and upload the completion report (PDF) into PhysioNet\u0026rsquo;s submission form, (d) wait for approval from PhysioNet\u0026rsquo;s team to confirm your credentials and submissions through verification software, and (e) sign the data usage agreement form (security \u0026amp; confidentiality agreement).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eOnce access to the MIMIC-IV (v3.1) database was granted, all the metadata and files will be available for download in an organized fashion at the bottom of the dataset\u0026apos;s page. The ZIP file for the dataset was 9.8 GB with a 9.9 GB total uncompressed size. One limitation of this research was the amount of storage space (RAM \u0026amp; Memory) available, making downloading a 9.9 GB database as a whole unattainable. By utilizing MIMIC-IV (v3.1)\u0026rsquo;s option to download individual files from the folders listed in MIMIC-IV\u0026rsquo;s database, downloading individual files needed in the creation of the ML model instead of the entire dataset would be an effective way to bypass research limitations. The following files were downloaded from the dataset MIMIC-IV (v3.1): (a) admissions.csv.gz, (b) d_labitems.csv.gz, (c) diagnoses_icd.csv.gz, (d) labevents.csv.gz, and (e) patients.csv.gz. The following files were downloaded from the dataset MIMIC-IV-Note (Deidentified free-text clinical notes): (a) discharge.csv.gz and (b) discharge_detail.csv.gz. The process of downloading each file individually rather than the entire ZIP file saved more than 61% of the storage space, reduced RAM (Random Access Memory) usage, and saved processing time.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAfter the needed data files from the MIMIC-IV database were properly downloaded (~3.5 Hours) and stored in an organized manner in a secure folder named \u0026ldquo;IBS_Crohns_Research\u0026rdquo;, the process of ETL (Extract, Transform, and Load) was followed by utilizing software through Python. Firstly, Pandas (data manipulation) and NumPy (array operations) software were installed in the terminal and imported through the code. Pandas\u0026rsquo; read feature was used to load the CSV files of the following: Diagnoses, for patient ICD codes, Patients, for demographics, Admissions, to connect patient IDs to hospital visits, and D_labitems, to link lab IIDs to test names. Pandas was then used to simultaneously load and merge the following files: Discharge (text) and Discharge_Detail (metadata). The files were merged into a single dataframe and a Subject ID so the medical notes could be properly linked to each patient. Then, all the patients were filtered through the inclusion/exclusion criteria of having a minimum value of 90% of data complete and not containing outlying, skewed data values. \u0026nbsp;The patients were then separated based on the ICD-10 diagnosis codes to obtain only patients with IBD (K58) and Crohn\u0026rsquo;s Disease (K50). From there, binary labels for disease classification were assigned through Pandas by assigning the integer 0 for IBS and 1 for Crohn\u0026rsquo;s Disease. All the patient\u0026rsquo;s labels were then combined to initiate a \u0026lsquo;labels\u0026rsquo; DataFrame to be used as a future target variable. Then, Pandas was used to load and filter the laboratory events with a chunk-based format due to the large file size (2.59 GB). The chunk size optimized for the data file was 10^6 (1,000,000 lines) of data at a time. The loading loop consisted of each chunk being filtered to only include IBS and Crohn\u0026rsquo;s Disease patients based on ICD codes (K58 \u0026amp; K50), concatenation of the filtered results occurred into the \u0026lsquo;labs\u0026rsquo; DataFrame, and the next chunk was loaded into the program. From there, each lab was merged with its matching textual names based on its SubjectIDs from the d_labitems file. Then, Pandas and NumPy were used to clean \u0026lsquo;valuenum\u0026rsquo; (numerical result of lab-test or measurement) and drop NaNs (Not a Numbers). The \u0026lsquo;valuenum\u0026rsquo; measure was generated by taking the average of all laboratory tests conducted. From there, the data table was pivoted from a lab-based table (raw data table) to a patient-based table by grouping all labs by patient, with each row containing a unique patient through merging common patient IDs, and each column containing a separate lab test. Lastly, Pandas was used to filter the discharge notes for IBS and Crohn\u0026rsquo;s Disease patients, group notes into a patient-based table, and concatenate notes into a single text string per patient for the vectorization process.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAfter the data extraction, filtration, and organization were complete, the optimization of data preprocessing began. Firstly, the vectorization of the clinical notes was initiated through the ClinicalBERT tokenizer and model, which generated mean-pooled embeddings and merged the vectorized clinical notes and lab features with labels. Vectorization was essential in the development of the ML model, as raw natural language processing (NLP) models operate through numerical representations called embedding rather than string-form text. ClinicalBERT, a BERT model fine-tuned on clinical text data and medical notes, was chosen over the built-in TF-IDF model as it had named entity recognition (NER) and medical natural language inference (MedNLI), giving it the ability to classify medical entities (diseases, medications, and lab values) and determine the relationship between 2 clinical statements. In contrast, TF-IDF lacks functionality as it merely tracks the frequency of the words instead of assessing the text through a deeper analysis. After vectorizing the text data with ClinicalBERT, the numerical embedding was combined with structured features from each patient\u0026rsquo;s laboratory results, which were compressed using principal component analysis (PCA) to reduce dimensionality by only including the most important dimensions.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eOnce the data preprocessing ended, the data was split for ML model training. The golden split ratio of data in medical applications was applied, partitioning 70% for initial training, 15% for model tuning, and 15% for final evaluation. The ML model was then defined through empirical testing as a sequential model, where training data is processed in a specific order, consisting of 4 layers with 1, 64, 128, and 256 corresponding neurons on each layer. After initial testing to analyze data quality and quantity, it was found that there were 2,126 Crohn\u0026rsquo;s disease patients included in the dataset, but only 1,367 IBS patients, highlighting an imbalance of data, potentially causing biased model predictions with poor performance on the minority side. To reduce these potential roadblocks, class weights were added to handle imbalance through Sklearn\u0026rsquo;s weight module, which automatically calculates optimized weights through the \u0026ldquo;compute_class_weight\u0026rdquo; built-in function. After class weights were appended in the model training, more testing discovered moderately high levels of overfitting in the model\u0026rsquo;s training, when the model learns the training data too well (including its noise) and fails to generalize to new \u0026amp; unseen data. The overfitting was likely caused by the complex similarities between Crohn\u0026rsquo;s disease and IBS, insufficient training data to cover all aspects of diagnosis, and overtraining. In all, four advanced techniques, including L2 regularization (ridge regression), dropout, Leaky ReLU, and early stopping, were utilized to minimize overfitting, ultimately increasing the accuracy and boosting performance metrics in the model. L2 regularization was applied with the following cost equation:\u003c/p\u003e\n\u003cp\u003eLoss + \u0026lambda; * (1/2) * \u0026Sigma;(w_i)^2\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;The \u0026ldquo;loss\u0026rdquo; represents the original loss function of the model (mean squared error and cross-entropy loss), the \u0026ldquo;\u0026lambda;\u0026rdquo; (lambda) is a hyperparameter that controls the strength of the regularization, and \u0026ldquo;\u0026Sigma;(w_i)^2\u0026rdquo; is the sum of the squares of all the model\u0026apos;s weights (coefficients). L2 regularization, dropout, and Leaky ReLU were applied on layers 2-4 with a penalty term of 0.001 and a dropout rate of 0.5 on each layer. Early stopping was also used in the model training with a patience value of 10, implying that if the validation loss hadn\u0026rsquo;t increased in the previous 10 epochs, then the training would automatically cease. Once the initial ML model training occurred, the model was tuned on the validation set and evaluated on the test set.\u003c/p\u003e\n\u003cp\u003eAfter the ML model was developed, Bayesian optimization was implemented through hyperparameter tuning, efficiently finding optimal hyperparameters for ML models and constructing a probabilistic model. Bayesian optimization ran a total of 50 epochs with a variety of hyperparameters being evaluated on each set, allowing for enhanced model performance. After the optimization is complete, all 50 tuned models are filtered through a certain performance metric, area under the curve (AUC), and the model with the finest performance on that metric will be saved as the final model.\u003c/p\u003e\n\u003cp\u003eThrough all sections of the research, from the model training, tuning, evaluating, and optimizing, adequate graphs were collected through the Matplotlib package, a comprehensive library for creating static and interactive visualizations in Python. The final model was saved in a \u0026ldquo;.keras\u0026rdquo; file, and all post-processed data, analytical results, and ML models were stored in their dedicated research repositories/folders.\u003c/p\u003e"},{"header":"IV. RESULTS","content":"\u003cp\u003eA total of 3,493 patient records (2,126 Crohn\u0026apos;s, 1,367 IBS) were included in the development of the ML model. The model was trained using 70% of patient data, with 15% splittage for both tuning and final evaluation. The model was optimized to reduce diagnostic bias and overfitting and was fine-tuned with 53 epochs of Bayesian Optimization with Hyperparameter Search.\u003c/p\u003e\n\u003cp\u003eTABLE 1\u003c/p\u003e\n\u003cp\u003eModel Metrics (Accuracy, F1-score, and AUC)\u003c/p\u003e\n\u003cp\u003e\u003cimg src=\"https://myfiles.space/user_files/69519_bce2c0439cd956a6/69519_custom_files/img1778600111.png\" style=\"width: 421px;\"\u003e\u003c/p\u003e\n\u003cp\u003eFine-tuned ML Model Metrics: Accuracy, F1-score, and AUC, signifying high plausibility for Predictive AI to enter the Medical Field of Gastroenterology\u003c/p\u003e\n\u003cp\u003eThe final ClinicalBERT + PCA + L2 regularization + Bayesian optimization + deep neural network (DNN) model achieved an overall accuracy of 74.41%, F1-score of 73.40%, and an AUC of 79.56%. The model was filtered for AUC during Bayesian optimization, but was previously set to accuracy. The highest accuracy rating the model received on the test set was 78%.\u003c/p\u003e\n\u003cp\u003eFIGURE 1\u003c/p\u003e\n\u003cp\u003eBayesian Optimization: Model Performance Comparison\u003c/p\u003e\n\u003cp\u003eBayesian Optimization Individual Epoch Metric Comparison: The bar-graph shows the trials in order of computer operations, with the initial model being 1st and the model with the highest AUC being last.\u003c/p\u003e\n\u003cp\u003eFIGURE 2\u003c/p\u003e\n\u003cp\u003eBayesian Optimization: Model ROC AUC over Trials\u003c/p\u003e\n\u003cp\u003eBayesian Optimization ROC AUC over Trials: lineplot on the metrics of Bayesian Optimization\u0026rsquo;s corresponding effects on the AUC of the model over time as trials continue, with the orange dotted line keeping track of the model with the Highest ROC AUC.\u003c/p\u003e"},{"header":"VI. DISCUSSION","content":"\u003cp\u003eThe ML model, which differentiates Crohn’s disease and IBS, had an accuracy of 74.41%, an F1-score of 73.40%, and an AUC of 79.56% (Table 1). Comparing the model’s AUC, the metric used to distinguish between 2 classes (Crohn’s \u0026amp; IBS), to the global average diagnostic accuracy rate (90%), we can see that the model generated through this case-study doesn’t empirically align with the global average, with roughly a 10% diagnostic gap between the 2 diseases. Although the diagnostic gap is apparent, we must acknowledge its primary usage in underdeveloped countries and how it serves as a low-cost, highly available, and accessible alternative to traditional diagnosis. Roughly 55% of Crohn’s disease patients in underdeveloped countries are unable to afford diagnostic testing (52.4% in Asia, 60.9% in Africa), 0.12 endoscopists, 0.12 gastroscopes, and 0.09 colonscopes available per 100,000 inhabitants, and an average gastroenterologist density of 0.005 to 0.02 per 100,000 inhabitants (commonly 1 per country in low-income regions). It’s reasonable to infer that the impact that the ML model would make on the inhabitants of underdeveloped countries, such as giving ~55% of patients the ability to afford low-cost diagnosis, and allowing hundreds of thousands of patients to have access to diagnostic methods. One potential issue one may notice is that the model’s diagnostic accuracy is lower than the global average. Still, when focusing on underdeveloped regions and their lack of data on the misdiagnosis rate between Crohn’s disease and IBS, the misdiagnosis rate is certainly illuminated in underdeveloped regions compared to the global average. A majority of patients may not be able to receive a diagnostic analysis due to cost or availability-related burdens, which the implementation of the ML model would solve. In the end, it’s safe to imply that the application of the AI model in underdeveloped regions where patients actively face unaffordable costs and the scarcity of diagnostic equipment/gastroenterologists heavily outweighs the diagnostic accuracy difference between the model and the global average. Although the model may be limited by the quality of the dataset in use and the quantity of the data, as a large amount of data is needed to train an ML model on a complex medical intersection such as Crohn’s disease and IBS, the finished AI model is still exceptionally applicable in real-world applications in underdeveloped regions. Future work may be made to gather a wider range of data with finer quality and quantity to reduce overfitting, and re-train the model using DNNs to boost the ROC AUC of the model or assess the current model’s implications when applied to real-world scenarios. Overall, these results highlight the potential application of ML-powered AI models as an alternative diagnostic method to traditional Crohn’s disease and IBS diagnosis.\u003c/p\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e"},{"header":"VII. CONCLUSION","content":"\u003cp\u003eThis case study developed and analyzed an ML model that combined ClinicalBERT embeddings of clinical notes with PCA-reduced lab data to classify the distinction between Crohn\u0026rsquo;s disease and IBS using L2 regularization, dropout, Leaky ReLU, early-stopping, and Bayesian optimization, achieving 79.56% (~80%) AUC and an accuracy of 74.41%. The model solves critical issues powering the misdiagnosis crisis between Crohn\u0026rsquo;s disease and IBS in underdeveloped countries by providing a low-cost and highly available solution to traditional diagnostic methods, which require the use of expensive laboratory testing, scarcely available equipment, and specialized medical professionals (gastroenterologists). This research presents a case study that demonstrates the potential of utilizing ML models in the gastrointestinal intersection between Crohn\u0026rsquo;s disease and IBS to reduce misdiagnosis rates and increase diagnostic availability for more affected patients. Such models form the basis of modernized clinical decision support tools, which significantly improve patient outcomes.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthical Considerations\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study utilized data from the Medical Information Mart for Intensive Care IV (MIMIC-IV) database. The MIMIC-IV database contains fully de-identified patient data and was approved by the Institutional Review Boards of the Massachusetts Institute of Technology (MIT) and Beth Israel Deaconess Medical Center (BIDMC). As all data were de-identified and publicly available to credentialed users under a data use agreement, this study did not involve human subjects research as defined by U.S. federal regulations, and additional institutional review board (IRB) approval was not required.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConflicts of Interest\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eVarun Kurra declares no conflicts of interest.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor Contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eVarun Kurra gathered and extracted the data, developed Predictive ML models, conducted the case study, interpreted the results, and wrote the manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData Availability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe data used in this study were obtained from the dataset MIMIC-IV (v3.1), available publicly through PhysioNet. Credentialled registration and CITI training are pre-requisites to obtaining access to the dataset. No new data were generated through this case-study.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eWarner D (2025) IBS vs. Crohn\u0026rsquo;s disease: What to know. Medicalnewstoday.com. Published March 19, 2024. Accessed August 4. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.medicalnewstoday.com/articles/ibs-vs-crohns\u003c/span\u003e\u003cspan address=\"https://www.medicalnewstoday.com/articles/ibs-vs-crohns\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCross E, Saunders B, Farmer AD, Prior JA (2023) Diagnostic delay in adult inflammatory bowel disease: A systematic review. Indian J Gastroenterology: Official J Indian Soc Gastroenterol 42(1):40\u0026ndash;52. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s12664-022-01303-x\u003c/span\u003e\u003cspan address=\"10.1007/s12664-022-01303-x\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNew study finds delayed diagnosis of inflammatory bowel disease is linked to worse clinical outcomes (2023) crohnsandcolitis org uk Published January 13, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://crohnsandcolitis.org.uk/news-stories/news-items/new-study-shows-delayed-diagnosis-of-inflammatory-bowel-disease-is-linked-to-worse-clinical-outcomes\u003c/span\u003e\u003cspan address=\"https://crohnsandcolitis.org.uk/news-stories/news-items/new-study-shows-delayed-diagnosis-of-inflammatory-bowel-disease-is-linked-to-worse-clinical-outcomes\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCleveland Clinic (2023) Crohn\u0026rsquo;s disease. Clevel Clin Published April 12, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://my.clevelandclinic.org/health/diseases/9357-crohns-disease\u003c/span\u003e\u003cspan address=\"https://my.clevelandclinic.org/health/diseases/9357-crohns-disease\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eClinic Staff M (2024) Irritable bowel syndrome - Symptoms and causes. Mayo Clinic. Published October 11. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.mayoclinic.org/diseases-conditions/irritable-bowel-syndrome/symptoms-causes/syc-20360016\u003c/span\u003e\u003cspan address=\"https://www.mayoclinic.org/diseases-conditions/irritable-bowel-syndrome/symptoms-causes/syc-20360016\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChey WD, Kurlander J, Eswaran S (2015) Irritable bowel syndrome: A clinical review. JAMA 313(9):949. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1001/jama.2015.0954\u003c/span\u003e\u003cspan address=\"10.1001/jama.2015.0954\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBerg S (2023) What doctors wish patients knew about irritable bowel syndrome. American Medical Association. Published June 23. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.ama-assn.org/delivering-care/prevention-wellness/what-doctors-wish-patients-knew-about-irritable-bowel-syndrome\u003c/span\u003e\u003cspan address=\"https://www.ama-assn.org/delivering-care/prevention-wellness/what-doctors-wish-patients-knew-about-irritable-bowel-syndrome\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eClinic ACE, Clinic ACE (2025) Recognizing and Dealing with a Crohn\u0026rsquo;s Flare-Up | Chandler, Arizona. Arizona Colorectal Experts. Published March 29, 2024. Accessed August 5. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://aceclinic.org/recognizing-and-dealing-with-a-crohns-flare-up/\u003c/span\u003e\u003cspan address=\"https://aceclinic.org/recognizing-and-dealing-with-a-crohns-flare-up/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAdmin (2021) Colorectal Clinic of Tampa Bay. Colorectal Clinic of Tampa Bay. Published January 22, Accessed August 5, 2025. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.tampacolorectal.com/blog/how-to-calm-an-ibs-flare-up\u003c/span\u003e\u003cspan address=\"https://www.tampacolorectal.com/blog/how-to-calm-an-ibs-flare-up\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFord AC (2020) Overlap Between Irritable Bowel Syndrome and Inflammatory Bowel Disease. Gastroenterol Hepatol 16(4):211. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://pmc.ncbi.nlm.nih.gov/articles/PMC8132685/\u003c/span\u003e\u003cspan address=\"https://pmc.ncbi.nlm.nih.gov/articles/PMC8132685/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBurns E (2021) Doubling Up? IBS Symptoms in Patients With IBD in Remission - Ulcerative Colitis Peer to Peer. Medpagetoday.com. Published August 12, Accessed August 5, 2025. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.medpagetoday.com/resource-centers/ulcerative-colitis-crohns-disease/doubling-up-ibs-symptoms-patients-ibd-remission/3418\u003c/span\u003e\u003cspan address=\"https://www.medpagetoday.com/resource-centers/ulcerative-colitis-crohns-disease/doubling-up-ibs-symptoms-patients-ibd-remission/3418\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJayasooriya N, Baillie S, Blackwell J et al (2023) Systematic review with meta-analysis: Time to diagnosis and the impact of delayed diagnosis on clinical outcomes in inflammatory bowel disease. Aliment Pharmacol Ther 57(6). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/apt.17370\u003c/span\u003e\u003cspan address=\"10.1111/apt.17370\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNew study reveals 1 (2014) in 10 Inflammatory Bowel Disease patients are misdiagnosed with Irritable Bowel Syndrome. ueg eu Published Dec 15, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ueg.eu/a/201\u003c/span\u003e\u003cspan address=\"https://ueg.eu/a/201\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCard TR, Siffledeen J, Fleming KM (2014) Are IBD patients more likely to have a prior diagnosis of irritable bowel syndrome? Report of a case-control study in the General Practice Research Database. United Eur Gastroenterol J 2(6):505\u0026ndash;512. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1177/2050640614554217\u003c/span\u003e\u003cspan address=\"10.1177/2050640614554217\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRajbhandari R, Blakemore S, Gupta N et al (2020) Crohn\u0026rsquo;s disease in low and lower-middle income countries: A scoping review. World J Gastroenterol 26(43):6891\u0026ndash;6908. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3748/wjg.v26.i43.6891\u003c/span\u003e\u003cspan address=\"10.3748/wjg.v26.i43.6891\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRuma Rajbhandari, Blakemore S, Gupta N et al (2022) Crohn\u0026rsquo;s Disease Among the Poorest Billion: Burden of Crohn\u0026rsquo;s Disease in Low- and Lower-Middle-Income Countries. Dig Dis Sci 68(4):1226\u0026ndash;1236. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10620-022-07675-6\u003c/span\u003e\u003cspan address=\"10.1007/s10620-022-07675-6\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGaur A, Bharadwaj HR, Dalal P, Ahmed M (2024) Recent progress and future directions for expanding gastrointestinal endoscopy in low- and middle‐income African nations. JGH Open 8(9). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/jgh3.70026\u003c/span\u003e\u003cspan address=\"10.1002/jgh3.70026\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSharing my (2022) story is a superpower: the doctor living without a cure in Ethiopia. Guardian Published Oct 27, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.theguardian.com/global-development/2022/oct/27/doctor-fasika-teferra-superpower-hope-crohns-ethiopia-acc\u003c/span\u003e\u003cspan address=\"https://www.theguardian.com/global-development/2022/oct/27/doctor-fasika-teferra-superpower-hope-crohns-ethiopia-acc\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSch\u0026ouml;nborn C, Levy M, Jaeger MD et al (2025) Unmet health-related needs in patients with Crohn\u0026rsquo;s disease in Belgium: a mixed-methods study. Archives Public Health 83(1). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s13690-025-01632-1\u003c/span\u003e\u003cspan address=\"10.1186/s13690-025-01632-1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCurrent Research Initiatives Crohn\u0026rsquo;s \u0026amp; Colitis Foundation. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.crohnscolitisfoundation.org/research/current-research-initiatives\u003c/span\u003e\u003cspan address=\"https://www.crohnscolitisfoundation.org/research/current-research-initiatives\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAIPRM. 50\u0026thinsp;+\u0026thinsp;AI in Healthcare Statistics (2024) Aiprm.com. Published July 8, 2024. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.aiprm.com/ai-in-healthcare-statistics/\u003c/span\u003e\u003cspan address=\"https://www.aiprm.com/ai-in-healthcare-statistics/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGrand View Research (2023) \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.grandviewresearch.com/industry-analysis/artificial-intelligence-ai-healthcare-market\u003c/span\u003e\u003cspan address=\"https://www.grandviewresearch.com/industry-analysis/artificial-intelligence-ai-healthcare-market\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGuo Y, Hao Z, Zhao S, Gong J, Yang F (2020) Artificial Intelligence in Health Care: Bibliometric Analysis. J Med Internet Res 22(7):e18228. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2196/18228\u003c/span\u003e\u003cspan address=\"10.2196/18228\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSahni N, Stein G, Zemmel R, Cutler DM (2023) The Potential Impact of Artificial Intelligence on Healthcare Spending. Natl Bureau Economic Res Published January 1, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.nber.org/papers/w30857\u003c/span\u003e\u003cspan address=\"https://www.nber.org/papers/w30857\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGhezzi M, How AI in Healthcare could Save over 250,000 Lives Each Year and become a \u003cspan\u003e$\u003c/span\u003e188 Billion Market by 2030. The Journal of mHealth. Published November 6, 2023. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://thejournalofmhealth.com/how-ai-in-healthcare-could-save-over-250000-lives-each-year-and-become-a-188-billion-market-by-2030/\u003c/span\u003e\u003cspan address=\"https://thejournalofmhealth.com/how-ai-in-healthcare-could-save-over-250000-lives-each-year-and-become-a-188-billion-market-by-2030/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNsengiyumva NP, Hussain H, Oxlade O et al (2021) Triage of Persons With Tuberculosis Symptoms Using Artificial Intelligence\u0026ndash;Based Chest Radiograph Interpretation: A Cost-Effectiveness Analysis. Open Forum Infect Dis 8(12). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/ofid/ofab567\u003c/span\u003e\u003cspan address=\"10.1093/ofid/ofab567\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKakileti ST, Madhu HJ, Krishnan L, Manjunath G, Sampangi S, Ramprakash HV (2020) Observational Study to Evaluate the Clinical Efficacy of Thermalytix for Detecting Breast Cancer in Symptomatic and Asymptomatic Women. JCO Global Oncol 61472\u0026ndash;1480. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1200/go.20.00168\u003c/span\u003e\u003cspan address=\"10.1200/go.20.00168\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKarthik Adapa, Gupta A, Singh S et al (2025) A real-world evaluation of an innovative artificial intelligence tool for population-level breast cancer screening. npj Digit Med 8(1). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41746-024-01368-2\u003c/span\u003e\u003cspan address=\"10.1038/s41746-024-01368-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Lumiere Scholars","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Crohn’s Disease, Irritable Bowel Syndrome (IBS), Artificial Intelligence (AI), Machine Learning (ML), Machine-Learning-Based Disease Diagnosis (MLBDD), Clinical Notes, Laboratory Data ","lastPublishedDoi":"10.21203/rs.3.rs-9658515/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9658515/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cem\u003e\u003cstrong\u003eBackground and Aims:\u003c/strong\u003e\u003c/em\u003e\u003cstrong\u003e \u003c/strong\u003eCrohn’s disease, a chronic inflammatory bowel disease (IBD), and Irritable Bowel Syndrome (IBS) are gastrointestinal conditions that share overlapping symptoms. Due to their similarities, ~10-30% of individuals diagnosed with Crohn’s disease are initially misdiagnosed with Irritable Bowel Syndrome (IBS), with the crisis being especially severe in underdeveloped countries. Artificial intelligence (AI) and its subset, Machine Learning (ML), are becoming essential technologies in medical specialties. This quantitative research paper examines the conditions of Crohn’s disease and Irritable Bowel Syndrome (IBS) misdiagnosis in underdeveloped countries, and develops a Machine Learning (ML) model that improves diagnostic accuracy and availability against diagnostic conditions in underdeveloped regions.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e\u003cstrong\u003eMethods:\u003c/strong\u003e\u003c/em\u003e Data from 383,220 unique patients were extracted from the Dataset MIMIC-IV(v3.1). Pandas and Numpy were used to load, filter, and organize the data. ClinicalBERT was used for the vectorization of the clinical notes. L2 regularization, dropout, Leaky ReLU, and early-stopping were utilized to minimize overfitting. ML models were trained, tuned, evaluated, and Bayesian optimization was implemented on the finalized model.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e\u003cstrong\u003eResults:\u003c/strong\u003e\u003c/em\u003e The finalized model received an accuracy of 74.41%, an F1-score of 73.40%, and an AUC of 79.56%. After Bayesian Optimization, the model’s accuracy reached 78%.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e\u003cstrong\u003eConclusions:\u003c/strong\u003e\u003c/em\u003e Underdeveloped countries lack affordable medical testing, as 55% of Crohn’s disease patients in underdeveloped countries are unable to afford diagnostic testing. LLMICs also face a shortage of specialists as they report an average gastroenterologist density of 0.005-0.02 per 100,000 inhabitants. Therefore, the possibility and necessity of implementing a low-cost predictive ML application are evident.\u003c/p\u003e","manuscriptTitle":"Application of Machine Learning to Distinguish IBS from Crohn’s Disease in Underdeveloped Regions","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-05-12 17:19:20","doi":"10.21203/rs.3.rs-9658515/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"477626ce-b631-4fda-bfab-06eb3ad362f7","owner":[],"postedDate":"May 12th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":67821032,"name":"Artificial Intelligence and Machine Learning"},{"id":67821033,"name":"Gastroenterology \u0026 Hepatology"},{"id":67821034,"name":"Health Economics \u0026 Outcomes Research"},{"id":67821035,"name":"Epidemiology"}],"tags":[],"updatedAt":"2026-05-12T17:19:20+00:00","versionOfRecord":[],"versionCreatedAt":"2026-05-12 17:19:20","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9658515","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9658515","identity":"rs-9658515","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}