Identification of primary sclerosing cholangitis: ICD-10 code validation and comparison with a large language model approach

Identification of primary sclerosing cholangitis: ICD-10 code validation and comparison with a large language model approach | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Identification of primary sclerosing cholangitis: ICD-10 code validation and comparison with a large language model approach Melinda Wang, Mai Dao, Molly Delk, Cynthia Fenton, Michelle Y. Li, and 4 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9152968/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract Background Retrospective studies investigating primary sclerosing cholangitis (PSC) have been limited by the absence of a PSC-specific diagnostic code. In 2018, a new PSC-specific ICD-10 code was introduced. Aims We aimed to validate the new ICD-10 code and compare it to other methods of identifying patients with PSC. Methods All gastroenterology/hepatology and primary clinic notes and discharge summaries were extracted from UCSF Epic Clarity database and potential PSC patients were identified using natural language processing (NLP). PSC diagnosis was determined by physician adjudication through chart review. LASSO regression was used to develop and internally validate a PSC prediction model. Separately, we tested large language model’s (LLM) ability to distinguish PSC from non-PSC patients. Results Among 867 patients identified using NLP, 226 (26%) patients were adjudicated to have a true PSC diagnosis. The LASSO model selected ICD-10 code, alkaline phosphatase > 120 IU/L, ursodiol use, inflammatory bowel disease, and history of cholangitis. ICD-10 code alone had a c-statistic of 0.87, sensitivity 87.6%, and PPV 68.8%. The LASSO model had a c-statistic of 0.92, sensitivity 87.4%, and PPV 70.7%. LLM had a c-statistic 0.77, sensitivity 91.7%, and PPV 51.0%. Conclusions The PSC-specific ICD-10 code had excellent discriminatory capacity for identifying patients with PSC. While an optimized PSC prediction algorithm had slightly improved test characteristics, ICD-10 code alone was sufficient in identifying patients with PSC, supporting the use of the ICD-10 code in future database studies of PSC. In contrast, LLM had inferior discrimination compared to either ICD-10 code or the prediction model. primary sclerosing cholangitis algorithm ICD database artificial intelligence Figures Figure 1 Introduction Primary sclerosing cholangitis (PSC) is a rare, progressive cholestatic liver disease that causes inflammation and fibrosis of the biliary tree, often leading to cirrhosis and need for liver transplantation. 1 – 8 Patients with PSC can develop life-threatening complications such as bacterial cholangitis and malignancies, including cholangiocarcinoma and other gastrointestinal-related cancers. 1 – 3 PSC is associated with high healthcare utilization and often poor health-related quality of life. 4 – 8 Unfortunately, there are currently no effective therapies that alter the natural history of PSC. 1 – 3 Large databases can be helpful to study rare diseases such as PSC (which has a prevalence of 0-31.7 per 100,000 persons) 9 , however, prior diagnostic codes for PSC have not been specific and encompassed other biliary diseases like acute cholangitis. 10 Thus, prior PSC-related studies have depended on single-center experiences with cases often adjudicated by manual chart review. 10 – 13 Recognizing a need for a PSC-specific code, an ICD-10 code for PSC was introduced in October 2018 (ICD-10 K83.01). Newer administrative database studies using the new PSC-specific ICD-10 code demonstrate similar demographic characteristics to prior epidemiological studies. 14 , 15 The new PSC-specific ICD-10 code, however, has not yet been validated. 8 With recent technological advancements, artificial intelligence systems such as large language models (LLMs) are also reshaping the landscape of clinical research. By leveraging deep learning techniques, LLMs can extract and synthesize disease-related information from unstructured electronic health record text, substantially enhancing the analytical power and efficacy of clinical research. Recent studies have demonstrated that LLMs can outperform ICD-10 codes in identifying disease from unstructured text. 16 , 17 Beyond simple disease labeling, LLMs offer the potential to capture nuanced clinical features and generate probabilistic differential diagnoses directly from narrative notes. This capacity positions LLMs as a powerful complementary approach to structured coding systems in disease identification and prediction. Given these rapid advances, we sought to validate the new PSC-specific ICD-10 code, to develop a prediction model for PSC including the ICD-10 code, and to compare the discrimination of the ICD-10 code alone to the prediction model using clinical data, and to processing by a large language model (LLM). Methods Gastroenterology, hepatology and primary care clinic notes and discharge summaries between 10/1/2018 (when the new PSC ICD-10 code was introduced) and 6/1/2023 for patients ≥ 18 years were extracted as unstructured data from the UCSF Clarity Clinical Data Warehouse database. All patients with the terms “PSC” or “primary sclerosing cholangitis” within the text were identified using natural language processing (NLP) with regularized expressions. Notes with “PSC” embedded within text, preceded by “does not have,” “without,” “no,” “not,” “denies,” or “rule out,” and followed by “also increases risk” were excluded. All patients with the new PSC ICD-10 code (K83.01) were also identified and included in the data extraction. Clinical data including maximum laboratory values (alkaline phosphatase, IgG4, AMA, ANA, IgG, SMA, SLA), magnetic resonance cholangiopancreatography (MRCP) imaging reports, endoscopic retrograde cholangiopancreatography (ERCP) procedure reports, medications (e.g., ursodiol), additional diagnoses by ICD-10 code (inflammatory bowel disease (IBD), autoimmune hepatitis, primary biliary cholangitis, acute cholangitis, cirrhosis, colorectal cancer, cholangiocarcinoma) and pathology reports from liver biopsies were extracted for all patients from the Clarity database (Supplemental Table 1). Three physicians (MW, MD, MD), who were blinded to the presence/absence of the PSC ICD-10 code, reviewed the extracted data to adjudicate the presence or absence of PSC for each patient. A true PSC diagnosis was defined as an elevated alkaline phosphatase (> 120 IU/mL) and either radiographic (MRCP), endoscopic (ERCP), and/or biopsy evidence of PSC. For patients who did not meet criteria for PSC as defined above but had documentation from a gastroenterologist or hepatologist confirming a PSC diagnosis (e.g., MRCP or liver biopsy performed at an outside institution and therefore not able to be extracted automatically from the Clarity database) (n = 36), an attending hepatologist (ML) provided final adjudication of presence/absence of PSC diagnosis. For the prediction model, the dataset was split randomly into a training set (70%) and a test set (30%). We constructed a LASSO regression model using the following covariates: ICD-10 code, elevated alkaline phosphatase, ursodiol treatment, and the presence of inflammatory bowel disease, primary biliary cholangitis, autoimmune hepatitis, and/or bacterial cholangitis based on compatible ICD-10 codes. We utilized 10-fold cross-validation to determine the optimal lambda regularization parameter (using the 1-standard error rule). 18 The resulting model was then internally validated in the independent test set. To enable calculation of sensitivity and positive predictive value (PPV), the optimal cutoff for the LASSO regression model was calculated by maximizing Youden’s index (the j-statistic). We also tested the ability of a large language model (LLM) to detect and confirm the presence of PSC in clinical notes. For this exercise, we used protected health information (PHI)-compliant versions of the Microsoft Azure OpenAI Generative Pre-Trained Transformer-4 Omni (GPT-4o) available at our institution. These models are deployed within UCSF’s secure “Versa” platform, which has been previously described and used to analyze clinical documentation in hepatology. 16 GPT-4o was prompted in a zero-shot fashion (e.g., no example instructions or answers were given to the LLM). We iteratively refined the prompt until the LLM’s accuracy was greater than 90% in a random 5% sample of the total number of notes identified by NLP. The final prompt used is provided in the supplemental materials (Supplementary Text 1). Finally, the c-statistic, sensitivity, and PPV were calculated for the ICD code alone, the LASSO regression model, and the LLM. Results In total, 867 patients were identified through NLP to have (1) an inpatient or outpatient note with the term “PSC” or “primary sclerosing cholangitis”, and/or (2) the PSC ICD-10 code within the study period. Of these patients, 288 (33%) had the PSC ICD-10 code. After physician review of laboratory, radiographic, and biopsy data, 226 (26%) patients were adjudicated to have true PSC. Table 1 displays clinical characteristics comparing the PSC (n = 226) and non-PSC (n = 579) groups in our study. Compared to non-PSC patients, PSC patients were younger (44 vs 52 years, p < 0.01) and had greater prevalence of PSC-related conditions including cirrhosis (61% vs 44%, p < 0.01), inflammatory bowel disease (76% vs 44%, p < 0.01), acute cholangitis (97% vs 42%, p < 0.01), and cholangiocarcinoma (11% vs 4%, p < 0.01). PSC patients also had significantly higher alkaline phosphatase levels (550 vs 273, p < 0.01) and were significantly more likely to have a history of ursodiol treatment (75% vs 41%, p < 0.01). The ICD-10 code alone had a sensitivity of 87.6% (198/226; 95%CI 82.6–91.6) and a positive predictive value of 68.8% (198/288; 95%CI 63.1–74.1). The c-statistic for the ICD-10 code as a solitary predictor of PSC was 0.87 (95%CI 0.84–0.89). LASSO regression eliminated PBC and AIH from the final model (i.e., the coefficients for PBC diagnosis and AIH diagnosis were reduced to zero). Based on the LASSO regression coefficients, the probability of having a true PSC diagnosis was defined as follows: The c-statistic in the test set was 0.92 (95%CI 0.90–0.95). On internal validation, there was minimal model optimism (c-statistic in the test set 0.91 (95%CI 0.87–0.95)). The optimal cutpoint identified by maximizing Youden’s index was 0.50 (i.e., the model estimating a 50% probability of a true PSC diagnosis). Using this dichotomized cutoff, the LASSO regression model had a sensitivity of 87.4% (195/223; 95%CI 82.4–91.5%) and positive predictive value of 70.7% (195/276; 95%CI 64.9–76.0). Figure 1 shows ROC curves for ICD-10 only and PSC prediction model. LLM alone had a sensitivity of 91.7% (187/204; 95%CI 87.0-95.1) and a positive predictive value of 51.0% (187/367; 95%CI 45.7%-56.2). C-statistic for LLM alone was 0.77 (95%CI 0.75–0.81). Table 2 shows test characteristics for the ICD-10 code alone, the LASSO regression model, and the LLM. Table 2 Test Performance of ICD only, LASSO regression model, and large language model for identifying PSC C-statistic ICD-code only LASSO Regression Model Large Language Model 0.87 (0.84–0.89) 0.92 (0.90–0.95) 0.77 (0.75–0.81) Sensitivity 87.6 (82.6–91.6) 87.4 (82.4–91.5) 91.7 (87.0-95.1) Specificity 86.0 (83.0-88.6) 86.4 (83.4–89.1) 63.8 (59.4–68.0) Positive Predictive Value 68.8 (63.1–74.1) 70.7 (64.9–76.0) 51.0 (45.7–56.2) Discussion Prior identification of patients with PSC in administrative datasets required the use of multiple ICD-9 codes, including a cholangitis-specific ICD code and an IBD-related ICD code. 10 , 19 Despite the use of multiple diagnostic codes to create an optimal algorithm, this method only had a sensitivity of 56% and positive predictive value of 59%. 10 The ICD-9 code for cholangitis alone had a sensitivity of 83.7% and PPV of 7.2%. 10 These prior data suggested that with the ICD codes available, it was not possible to accurately identify patients with PSC. With the development of a new ICD-10 code specific to PSC, we observed markedly improved sensitivity and positive predictive value using the ICD-10 code alone as a predictor of PSC. We investigated whether we could optimize an algorithm for predicting PSC by including additional variables that we considered a priori to be associated with a true PSC diagnosis (e.g., elevated alkaline phosphatase, ursodiol treatment, concurrent inflammatory bowel disease, etc.). Though we developed and internally validated a prediction model with excellent discrimination, the model’s performance was not meaningfully improved over the ICD-code alone (c-statistics of 0.91 vs 0.87, sensitivities of 87.9% vs 87.6%, and PPVs of 70.3% vs 68.8%, respectively. Based on our data, it seems clear that the ICD code alone is sufficient for use in database-related research. Notably, the LLM yielded substantially lower sensitivity and discrimination. Compared to ICD-10 codes that are more inclusive for billing purposes, our prompting of the LLM required explicit documentation of PSC to generate a positive prediction. This likely explains GPT-4o’s conservative classification compared to ICD-10 code alone. In addition, in our implementation, the LLM was limited to evaluating unstructured text in clinic notes but did not have access to other structured (i.e. laboratory tests, diagnostic codes) and unstructured (i.e. MRCP reports, ERCP reports, pathology reports) that adjudicators in this study used to determine the presence/absence of PSC. Providing these data to LLM may improve LLM performance but given that ICD-10 code alone demonstrated strong discrimination and is easier to extract from the electronic health record, the added complexity of feeding multiple structured and unstructured data elements did not appear justified. The PSC prediction model included additional variables including alkaline phosphatase, ursodiol use, and associated diagnoses including IBD and acute cholangitis. The prevalence of IBD in the PSC patients included in this study was 76%, in line with the reported prevalence of 70–80% in the general PSC population. 20 In addition, acute bacterial cholangitis is a classic complication among patients with PSC. 21 Ursodiol was also incorporated in the optimal PSC prediction model – this is because while there are no current therapies to treat PSC, many patients with PSC are prescribed ursodiol. 3 Studies demonstrate that while ursodiol may improve liver tests, ursodiol does not improve survival and may lead to increased rates of adverse events. 22 With new potential therapies for PSC on the horizon, 23 rates of ursodiol use among patients with PSC may decrease, making this variable less effective in a PSC prediction model. Finally, autoimmune hepatitis and PBC were eliminated by LASSO regression. The prevalence of autoimmune hepatitis overlap in our PSC cohort (15%) is similar to what has been reported in the general PSC population (7–14%) though notably the “prevalence” of PBC overlap (58%) far exceeds case reports in the literature. 24 As these diagnoses were made based on ICD-10 codes, we strongly suspect that most PSC patients in our cohort experienced miscoding for one or more billable events (e.g., the PBC diagnosis code was inputted instead of the PSC code) leading to the very high proportion of PSC patients also carrying a PBC diagnosis code. Furthermore, our data showing expected prevalence of IBD and autoimmune hepatitis in our PSC cohort supports the validity of our approach to validate true PSC diagnoses. PSC has been recognized as a spectrum of disease ranging from small duct PSC to classic, large duct PSC. While patients with large duct PSC have classic diagnostic features to assist with diagnosis, small duct PSC, a rarer form of PSC with better prognosis, can often be challenging to diagnosis. 25 The definition of PSC in this population included classic diagnostic features of large duct PSC and also hepatologist/gastroenterologist statement of PSC even without known diagnostic features. This definition attempts to capture patients with PSC across the spectrum of disease. However, given diversity of PSC presentations, future studies may benefit from evaluating the validity of the ICD-10 code within a larger and more diverse administrative dataset. This study has several limitations. First, data were limited to a single center experience which does not capture different coding practices across systems. Second, variables used within the PSC optimization model were limited by variables available in the electronic health record. Finally, the study cohort included patients with positive ICD-10 code for PSC and patients identified in gastroenterology/hepatology and primary care notes or discharge summaries to have “PSC” or similar iterations in the notes by national language processing. Thus, negative predictive value could not be calculated within this study population and positive predictive value must be interpreted with caution as the population is enriched with patients likely to have PSC which would artificially inflate the positive predictive value. In conclusion, we validated the new PSC-specific ICD-10 code and demonstrated use of either the ICD-10 code alone or an optimized PSC prediction model including additional clinical variables are effective in identifying patients with PSC in administrative databases. Further, an LLM approach solely using unstructured data from notes was inferior. Future studies can safely utilize the ICD-10 code for investigating PSC. Abbreviations PSC; primary sclerosing cholangitis NLP; national language processing PPV; positive predictive value AMA; anti-mitochondrial antibody ANA; anti-nuclear antibody SMA; smooth muscle antibody SLA; soluble liver antigen antibody MRCP; magnetic resonance cholangiopancreatography ERCP; endoscopic retrograde cholangiopancreatography IBD; inflammatory bowel disease PHI; patient health information Declarations Funding: The authors of this study were supported by a PSC Partners Young Investigator Award (Li), K23DK139455 (National Institute of Diabetes and Digestive and Kidney Diseases, Ge), P30DK026743 (UCSF Liver Center Grant, Ge and Lai), T32DK060414 (National Institute of Diabetes and Digestive and Kidney Diseases , Wang) and R01DK133527 (National Institute of Diabetes and Digestive and Kidney Diseases, Lai), K24AG080021 (National Institute on Aging, Lai). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or any other funding agencies. The funding agencies played no role in the analysis of the data or the preparation of this manuscript. Competing Interests: The authors of this manuscript have the following potential conflicts of interest to disclose as described by: Dr. Jin Ge previously received research support (to institution) from Merck and Co. He previously served on an advisory board for Gilead Sciences and previously consulted for Madrigal Pharmaceuticals and Astellas Pharmaceuticals/Iota Biosciences. Dr. Jennifer C. Lai receives research support (to institution) from Lipocene and Gore Medical; serves on an advisory board for Boehringer Ingelheim, Cosmo; and consults for Novo Nordisk. Dr. Michael Li receives research support (to institution) from FUJIFILM Wako and previously consulted for Ono Pharmaceuticals. Author Contribution Statement: MW: study design, data analysis, data interpretation, manuscript writing, funding MD: data analysis, data interpretation, manuscript editing MD: data analysis, data interpretation, manuscript editing CF: data acquisition, data interpretation, manuscript editing MYL: data analysis, data interpretation, manuscript editing JRB: data interpretation, manuscript editing JG: study design, data interpretation, manuscript editing, funding JCL: data interpretation, manuscript editing, funding ML: study design, data analysis, data interpretation, manuscript editing, funding References Boonstra K, Beuers U, Ponsioen CY. Epidemiology of primary sclerosing cholangitis and primary biliary cirrhosis: a systematic review. Journal of hepatology. 2012;56(5):1181–8. Dyson JK, Beuers U, Jones DE, Lohse AW, Hudson M. Primary sclerosing cholangitis. The Lancet. 2018;391(10139):2547–59. Chapman R, Fevery J, Kalloo A, Nagorney DM, Boberg KM, Shneider B, et al. Diagnosis and management of primary sclerosing cholangitis. Hepatology. 2010;51(2):660–78. Haapamäki J, Tenca A, Sintonen H, Barner-Rasmussen N, Färkkilä MA. Health‐related quality of life among patients with primary sclerosing cholangitis. Liver international. 2015;35(9):2194–201. Mol B, van Munster K, Bogaards J, Weersma R, Inderson A, de Groof J, et al. Health-related quality of life in patients with primary sclerosing cholangitis: a longitudinal population-based cohort study. Journal of Hepatology. 2022;77:S338-S9. Trivedi PJ, Bowlus CL, Yimam KK, Razavi H, Estes C. Epidemiology, natural history, and outcomes of primary sclerosing cholangitis: a systematic review of population-based studies. Clinical Gastroenterology and Hepatology. 2022;20(8):1687–700. e4. Toy E, Balasubramanian S, Selmi C, Li C-S, Bowlus CL. The prevalence, incidence and natural history of primary sclerosing cholangitis in an ethnically diverse population. BMC gastroenterology. 2011;11:1–7. Wang M, Auerbach A, Oreper SM, Hohmann SF, Lai JC, Rubin JB. Leveraging a new ICD-10 diagnosis code to characterize hospitalized patients with primary sclerosing cholangitis. Clinical Gastroenterology and Hepatology. 2022. Mehta TI, Weissman S, Fung BM, Sotiriadis J, Lindor KD, Tabibian JH. Global incidence, prevalence and features of primary sclerosing cholangitis: a systematic review and meta-analysis. Liver international. 2021;41(10):2418–26. Molodecky NA, Myers RP, Barkema HW, Quan H, Kaplan GG. Validity of administrative data for the diagnosis of primary sclerosing cholangitis: a population-based study. Liver International. 2011;31(5):712–20. Ponsioen CY, Assis DN, Boberg KM, Bowlus CL, Deneau M, Thorburn D, et al. Defining primary sclerosing cholangitis: results from an international primary sclerosing cholangitis study group consensus process. Gastroenterology. 2021;161(6):1764–75. e5. Bowlus CL, Li CS, Karlsen TH, Lie BA, Selmi C. Primary sclerosing cholangitis in genetically diverse populations listed for liver transplantation: unique clinical and human leukocyte antigen associations. Liver Transplantation. 2010;16(11):1324–30. Are VS, Vilar-Gomez E, Gromski MA, Akisik F, Lammert C, Ghabril M, et al. Racial differences in primary sclerosing cholangitis mortality is associated with community socioeconomic status. Liver International. 2021;41(11):2703–11. Wang M, Auerbach A, Oreper SM, Hohmann SF, Lai JC, Rubin JB. Leveraging a new ICD-10 diagnosis code to characterize hospitalized patients with primary sclerosing cholangitis. Clinical Gastroenterology and Hepatology. 2023;21(10):2692–4. Wang M, Harris A, McCulloch CE, Wadhwani SI, Lai JC, Rubin JB. Racial differences in primary sclerosing cholangitis: A retrospective cohort study leveraging a new ICD-10 code. Annals of Hepatology. 2025;30(1):101901. Far AT, Bastani A, Lee A, Gologorskaya O, Huang CY, Pletcher MJ, et al. Evaluating the positive predictive value of code-based identification of cirrhosis and its complications utilizing GPT-4. Hepatology. 2025;81(6):1753–63. McMurry AJ, Phelan D, Dixon BE, Geva A, Gottlieb D, Jones JR, et al. Large Language Model Symptom Identification From Clinical Text: Multicenter Study. J Med Internet Res. 2025;27:e72984. Friedman JH, Hastie T, Tibshirani R. Regularization paths for generalized linear models via coordinate descent. Journal of statistical software. 2010;33:1–22. Nguyen A, Torabi Sagvand B, Alizadeh M, Nguyen C, Scott W, von Rosenvinge EC. Primary sclerosing cholangitis and pancreatic cancer: a retrospective cohort study of United States veterans. Frontiers in Gastroenterology. 2023;1:1076788. Rossi RE, Conte D, Massironi S. Primary sclerosing cholangitis associated with inflammatory bowel disease: an update. European journal of gastroenterology & hepatology. 2016;28(2):123–31. Loftus E, Harewood G, Loftus C, Tremaine W, Harmsen W, Zinsmeister A, et al. PSC-IBD: a unique form of inflammatory bowel disease associated with primary sclerosing cholangitis. Gut. 2005;54(1):91–6. Lindor KD, Kowdley KV, Luketic VA, Harrison ME, McCashland T, Befeler AS, et al. High-dose ursodeoxycholic acid for the treatment of primary sclerosing cholangitis. Hepatology. 2009;50(3):808–14. Abbas N, Quraishi MN, Trivedi P. Emerging drugs for the treatment of primary sclerosing cholangitis. Current opinion in pharmacology. 2022;62:23–35. Boberg KM, Chapman RW, Hirschfield GM, Lohse AW, Manns MP, Schrumpf E. Overlap syndromes: The International Autoimmune Hepatitis Group (IAIHG) position statement on a controversial issue. Journal of Hepatology. 2011;54(2):374–85. Deliwala S, Sundus S, Haykal T, Elbedawi MM, Bachuwa G. Small duct primary sclerosing cholangitis: an underdiagnosed cause of chronic liver disease and cirrhosis. Cureus. 2020;12(3). Table 1 Table 1 is available in the Supplementary Files section. Additional Declarations Competing interest reported. The authors of this manuscript have the following potential conflicts of interest to disclose as described by: • Dr. Jin Ge previously received research support (to institution) from Merck and Co. He previously served on an advisory board for Gilead Sciences and previously consulted for Madrigal Pharmaceuticals and Astellas Pharmaceuticals/Iota Biosciences. • Dr. Jennifer C. Lai receives research support (to institution) from Lipocene and Gore Medical; serves on an advisory board for Boehringer Ingelheim, Cosmo; and consults for Novo Nordisk. • Dr. Michael Li receives research support (to institution) from FUJIFILM Wako and previously consulted for Ono Pharmaceuticals. Supplementary Files PSCValidationSupplemental.docx Table1.docx Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 29 Apr, 2026 Reviews received at journal 20 Apr, 2026 Reviews received at journal 18 Apr, 2026 Reviewers agreed at journal 31 Mar, 2026 Reviewers agreed at journal 30 Mar, 2026 Reviewers invited by journal 30 Mar, 2026 Editor assigned by journal 18 Mar, 2026 Submission checks completed at journal 18 Mar, 2026 First submitted to journal 17 Mar, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9152968","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":615211817,"identity":"5de95fd2-71cf-4f4d-8cb4-459f31ef9f71","order_by":0,"name":"Melinda Wang","email":"","orcid":"","institution":"University of California-San Francisco","correspondingAuthor":false,"prefix":"","firstName":"Melinda","middleName":"","lastName":"Wang","suffix":""},{"id":615211818,"identity":"98c6b9e2-b432-4b3b-aaf2-3263869c81ae","order_by":1,"name":"Mai Dao","email":"","orcid":"","institution":"University of California-San Francisco","correspondingAuthor":false,"prefix":"","firstName":"Mai","middleName":"","lastName":"Dao","suffix":""},{"id":615211819,"identity":"9e3c592c-fc42-4e15-a54e-21496752ed1f","order_by":2,"name":"Molly Delk","email":"","orcid":"","institution":"Tulane University","correspondingAuthor":false,"prefix":"","firstName":"Molly","middleName":"","lastName":"Delk","suffix":""},{"id":615211820,"identity":"2fd1c5c1-4a1f-4dd4-9168-8d26d1db98d7","order_by":3,"name":"Cynthia Fenton","email":"","orcid":"","institution":"University of California-San Francisco","correspondingAuthor":false,"prefix":"","firstName":"Cynthia","middleName":"","lastName":"Fenton","suffix":""},{"id":615211821,"identity":"354230cb-0d08-470f-bb98-7b2b57fab652","order_by":4,"name":"Michelle Y. Li","email":"","orcid":"","institution":"University of California-San Francisco","correspondingAuthor":false,"prefix":"","firstName":"Michelle","middleName":"Y.","lastName":"Li","suffix":""},{"id":615211822,"identity":"52d6e72d-84be-4ebf-b4ca-cb57896fbc40","order_by":5,"name":"Jessica B. Rubin","email":"","orcid":"","institution":"University of California-San Francisco","correspondingAuthor":false,"prefix":"","firstName":"Jessica","middleName":"B.","lastName":"Rubin","suffix":""},{"id":615211823,"identity":"2c056e8c-4034-4dd1-8bb5-4a74ad302760","order_by":6,"name":"Jin Ge","email":"","orcid":"","institution":"University of California-San Francisco","correspondingAuthor":false,"prefix":"","firstName":"Jin","middleName":"","lastName":"Ge","suffix":""},{"id":615211824,"identity":"badcb346-4554-4d3a-be04-a0fef54beb32","order_by":7,"name":"Jennifer C. Lai","email":"","orcid":"","institution":"University of California-San Francisco","correspondingAuthor":false,"prefix":"","firstName":"Jennifer","middleName":"C.","lastName":"Lai","suffix":""},{"id":615211825,"identity":"6319a825-500c-497f-b492-b7dcbc70dfb8","order_by":8,"name":"Michael Li","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA1ElEQVRIiWNgGAWjYBACg8MMbEDKgoGPmfkAkCEHxAn4tVhCtEgwsDGzgZQaE9ZifwCmhYHHgDgtZseZnz34UQHUws7zTeLHHwMGfvYcA/xaDrOZG/acATmMd5tkb5sBg2TPG0JaeNgkeNsgWqQZG/4wGNwgYIsBUIvk338gLTzPpBmADrMnRos0bwNYC5s0A5sBg4EEQS1sZtIyxyR4gIFsbAn0C4/EmWcF+LWcP/xM8k2NjRw//+GHN4AhJsffnrwBrxYY4MFgjIJRMApGwSigAAAAlQ83Il5qwZgAAAAASUVORK5CYII=","orcid":"","institution":"University of California-San Francisco","correspondingAuthor":true,"prefix":"","firstName":"Michael","middleName":"","lastName":"Li","suffix":""}],"badges":[],"createdAt":"2026-03-17 23:53:41","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9152968/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9152968/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":105908933,"identity":"04e8633a-36e8-4732-8b08-f2ec3b4b89fc","added_by":"auto","created_at":"2026-04-01 10:40:25","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":25257,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eROC curves for PSC prediction model vs ICD-10 code alone\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"PSCValidationFigure.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9152968/v1/9ce032eed65365121c12a5b7.jpg"},{"id":107479843,"identity":"66e1372c-099c-4525-8358-85f25ea2723d","added_by":"auto","created_at":"2026-04-22 01:55:03","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":273135,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9152968/v1/883fcca7-c414-4e33-a434-f00714905111.pdf"},{"id":105908732,"identity":"a26ff036-d428-4b7d-b874-93a2b6735990","added_by":"auto","created_at":"2026-04-01 10:39:35","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":17650,"visible":true,"origin":"","legend":"","description":"","filename":"PSCValidationSupplemental.docx","url":"https://assets-eu.researchsquare.com/files/rs-9152968/v1/f1ad90de5b1c5d31e4961b9b.docx"},{"id":105908901,"identity":"8bb09626-7429-4a0f-a0c0-f436a05f9c8a","added_by":"auto","created_at":"2026-04-01 10:39:56","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":16490,"visible":true,"origin":"","legend":"","description":"","filename":"Table1.docx","url":"https://assets-eu.researchsquare.com/files/rs-9152968/v1/ce15b7b13c003ff80779fb98.docx"}],"financialInterests":"Competing interest reported. The authors of this manuscript have the following potential conflicts of interest to disclose as described by:\n•\tDr. Jin Ge previously received research support (to institution) from Merck and Co. He previously served on an advisory board for Gilead Sciences and previously consulted for Madrigal Pharmaceuticals and Astellas Pharmaceuticals/Iota Biosciences.\n•\tDr. Jennifer C. Lai receives research support (to institution) from Lipocene and Gore Medical; serves on an advisory board for Boehringer Ingelheim, Cosmo; and consults for Novo Nordisk.\n•\tDr. Michael Li receives research support (to institution) from FUJIFILM Wako and previously consulted for Ono Pharmaceuticals.","formattedTitle":"Identification of primary sclerosing cholangitis: ICD-10 code validation and comparison with a large language model approach","fulltext":[{"header":"Introduction","content":"\u003cp\u003ePrimary sclerosing cholangitis (PSC) is a rare, progressive cholestatic liver disease that causes inflammation and fibrosis of the biliary tree, often leading to cirrhosis and need for liver transplantation.\u003csup\u003e\u003cspan additionalcitationids=\"CR2 CR3 CR4 CR5 CR6 CR7\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e Patients with PSC can develop life-threatening complications such as bacterial cholangitis and malignancies, including cholangiocarcinoma and other gastrointestinal-related cancers.\u003csup\u003e\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e PSC is associated with high healthcare utilization and often poor health-related quality of life.\u003csup\u003e\u003cspan additionalcitationids=\"CR5 CR6 CR7\" citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e Unfortunately, there are currently no effective therapies that alter the natural history of PSC.\u003csup\u003e\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e Large databases can be helpful to study rare diseases such as PSC (which has a prevalence of 0-31.7 per 100,000 persons)\u003csup\u003e\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e, however, prior diagnostic codes for PSC have not been specific and encompassed other biliary diseases like acute cholangitis.\u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u003c/sup\u003e Thus, prior PSC-related studies have depended on single-center experiences with cases often adjudicated by manual chart review.\u003csup\u003e\u003cspan additionalcitationids=\"CR11 CR12\" citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eRecognizing a need for a PSC-specific code, an ICD-10 code for PSC was introduced in October 2018 (ICD-10 K83.01). Newer administrative database studies using the new PSC-specific ICD-10 code demonstrate similar demographic characteristics to prior epidemiological studies.\u003csup\u003e\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e The new PSC-specific ICD-10 code, however, has not yet been validated.\u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e With recent technological advancements, artificial intelligence systems such as large language models (LLMs) are also reshaping the landscape of clinical research. By leveraging deep learning techniques, LLMs can extract and synthesize disease-related information from unstructured electronic health record text, substantially enhancing the analytical power and efficacy of clinical research. Recent studies have demonstrated that LLMs can outperform ICD-10 codes in identifying disease from unstructured text.\u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e Beyond simple disease labeling, LLMs offer the potential to capture nuanced clinical features and generate probabilistic differential diagnoses directly from narrative notes. This capacity positions LLMs as a powerful complementary approach to structured coding systems in disease identification and prediction. Given these rapid advances, we sought to validate the new PSC-specific ICD-10 code, to develop a prediction model for PSC including the ICD-10 code, and to compare the discrimination of the ICD-10 code alone to the prediction model using clinical data, and to processing by a large language model (LLM).\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eGastroenterology, hepatology and primary care clinic notes and discharge summaries between 10/1/2018 (when the new PSC ICD-10 code was introduced) and 6/1/2023 for patients\u0026thinsp;\u0026ge;\u0026thinsp;18 years were extracted as unstructured data from the UCSF Clarity Clinical Data Warehouse database. All patients with the terms \u0026ldquo;PSC\u0026rdquo; or \u0026ldquo;primary sclerosing cholangitis\u0026rdquo; within the text were identified using natural language processing (NLP) with regularized expressions. Notes with \u0026ldquo;PSC\u0026rdquo; embedded within text, preceded by \u0026ldquo;does not have,\u0026rdquo; \u0026ldquo;without,\u0026rdquo; \u0026ldquo;no,\u0026rdquo; \u0026ldquo;not,\u0026rdquo; \u0026ldquo;denies,\u0026rdquo; or \u0026ldquo;rule out,\u0026rdquo; and followed by \u0026ldquo;also increases risk\u0026rdquo; were excluded. All patients with the new PSC ICD-10 code (K83.01) were also identified and included in the data extraction. Clinical data including maximum laboratory values (alkaline phosphatase, IgG4, AMA, ANA, IgG, SMA, SLA), magnetic resonance cholangiopancreatography (MRCP) imaging reports, endoscopic retrograde cholangiopancreatography (ERCP) procedure reports, medications (e.g., ursodiol), additional diagnoses by ICD-10 code (inflammatory bowel disease (IBD), autoimmune hepatitis, primary biliary cholangitis, acute cholangitis, cirrhosis, colorectal cancer, cholangiocarcinoma) and pathology reports from liver biopsies were extracted for all patients from the Clarity database (Supplemental Table\u0026nbsp;1). Three physicians (MW, MD, MD), who were blinded to the presence/absence of the PSC ICD-10 code, reviewed the extracted data to adjudicate the presence or absence of PSC for each patient. A true PSC diagnosis was defined as an elevated alkaline phosphatase (\u0026gt;\u0026thinsp;120 IU/mL) and either radiographic (MRCP), endoscopic (ERCP), and/or biopsy evidence of PSC. For patients who did not meet criteria for PSC as defined above but had documentation from a gastroenterologist or hepatologist confirming a PSC diagnosis (e.g., MRCP or liver biopsy performed at an outside institution and therefore not able to be extracted automatically from the Clarity database) (n\u0026thinsp;=\u0026thinsp;36), an attending hepatologist (ML) provided final adjudication of presence/absence of PSC diagnosis.\u003c/p\u003e \u003cp\u003eFor the prediction model, the dataset was split randomly into a training set (70%) and a test set (30%). We constructed a LASSO regression model using the following covariates: ICD-10 code, elevated alkaline phosphatase, ursodiol treatment, and the presence of inflammatory bowel disease, primary biliary cholangitis, autoimmune hepatitis, and/or bacterial cholangitis based on compatible ICD-10 codes. We utilized 10-fold cross-validation to determine the optimal lambda regularization parameter (using the 1-standard error rule).\u003csup\u003e\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e The resulting model was then internally validated in the independent test set. To enable calculation of sensitivity and positive predictive value (PPV), the optimal cutoff for the LASSO regression model was calculated by maximizing Youden\u0026rsquo;s index (the j-statistic).\u003c/p\u003e \u003cp\u003eWe also tested the ability of a large language model (LLM) to detect and confirm the presence of PSC in clinical notes. For this exercise, we used protected health information (PHI)-compliant versions of the Microsoft Azure OpenAI Generative Pre-Trained Transformer-4 Omni (GPT-4o) available at our institution. These models are deployed within UCSF\u0026rsquo;s secure \u0026ldquo;Versa\u0026rdquo; platform, which has been previously described and used to analyze clinical documentation in hepatology.\u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e GPT-4o was prompted in a zero-shot fashion (e.g., no example instructions or answers were given to the LLM). We iteratively refined the prompt until the LLM\u0026rsquo;s accuracy was greater than 90% in a random 5% sample of the total number of notes identified by NLP. The final prompt used is provided in the supplemental materials (Supplementary Text 1). Finally, the c-statistic, sensitivity, and PPV were calculated for the ICD code alone, the LASSO regression model, and the LLM.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eIn total, 867 patients were identified through NLP to have (1) an inpatient or outpatient note with the term \u0026ldquo;PSC\u0026rdquo; or \u0026ldquo;primary sclerosing cholangitis\u0026rdquo;, and/or (2) the PSC ICD-10 code within the study period. Of these patients, 288 (33%) had the PSC ICD-10 code. After physician review of laboratory, radiographic, and biopsy data, 226 (26%) patients were adjudicated to have true PSC. Table 1 displays clinical characteristics comparing the PSC (n\u0026thinsp;=\u0026thinsp;226) and non-PSC (n\u0026thinsp;=\u0026thinsp;579) groups in our study. Compared to non-PSC patients, PSC patients were younger (44 vs 52 years, p\u0026thinsp;\u0026lt;\u0026thinsp;0.01) and had greater prevalence of PSC-related conditions including cirrhosis (61% vs 44%, p\u0026thinsp;\u0026lt;\u0026thinsp;0.01), inflammatory bowel disease (76% vs 44%, p\u0026thinsp;\u0026lt;\u0026thinsp;0.01), acute cholangitis (97% vs 42%, p\u0026thinsp;\u0026lt;\u0026thinsp;0.01), and cholangiocarcinoma (11% vs 4%, p\u0026thinsp;\u0026lt;\u0026thinsp;0.01). PSC patients also had significantly higher alkaline phosphatase levels (550 vs 273, p\u0026thinsp;\u0026lt;\u0026thinsp;0.01) and were significantly more likely to have a history of ursodiol treatment (75% vs 41%, p\u0026thinsp;\u0026lt;\u0026thinsp;0.01).\u003c/p\u003e\n\u003cdiv\u003e\n \u003cdiv align=\"left\" colname=\"c4\" colnum=\"4\"\u003eThe ICD-10 code alone had a sensitivity of 87.6% (198/226; 95%CI 82.6\u0026ndash;91.6) and a positive predictive value of 68.8% (198/288; 95%CI 63.1\u0026ndash;74.1). The c-statistic for the ICD-10 code as a solitary predictor of PSC was 0.87 (95%CI 0.84\u0026ndash;0.89).\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003eLASSO regression eliminated PBC and AIH from the final model (i.e., the coefficients for PBC diagnosis and AIH diagnosis were reduced to zero). Based on the LASSO regression coefficients, the probability of having a true PSC diagnosis was defined as follows:\u003c/p\u003e\n\u003cdiv id=\"Equa\"\u003e\n \u003cdiv format=\"TEX\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e\u003cimg src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAApcAAABSCAYAAAAB1v+/AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAFiUAABYlAUlSJPAAACqOSURBVHhe7Z3/bxtF/v+f/vxO0nX4iUPVKev7oYJTULtpEYRKQaLroyd0J+DWXBFColKxr0K6bwk4VCdEabqG9qQTJE5FperEYffeIBA6m8SVUqm2ojRxKlvXih8ua6GC+Gk3IfAHzOcHPKPd2V1/SZzQ0NdDWiU7Mzs7X14z89rXfHGEMcZAEARBEARBED3g/8kOBEEQBEEQBLFZSLkkCIIgCIIgegYplwRBEARBEETPIOWSIAiCIAiC6Bl3tHKZyWQQiUQQjUZlr66IRCKIRCKYmZmRvUKpVCriuUajIXvfERSLRcRiMUQiEQwPD8vegs3knyAIgiAIYjNsWrnkCov7SiQScBxHDrppxsbGkE6nceTIEdmrK2zbBgA8/PDDspegWCwiEomI9I+MjMA0TaiqisHBwdBwPxaVSgVHjx7Fxx9/jHK5jIGBATmIoJP894qJiYmWiu5PFffHCL+i0SgmJiY84RzHQSqVQjQaFeHi8bhPniqVChKJRMu4CIIgCOJOZNPKZaFQAABYlgXGGMrlMkqlEk6dOiUH3RIrKysYHR2Vnbviiy++gKIoGBoakr0EN2/ehKIoPiVN0zTPfVC4fD6PeDzuCbfdnD59GslkEkNDQxgZGcHs7KwcRPDFF18AQMv894qVlRWoqio7bxuxWAyVSkV23nFGRkaQTqehqioYY2CMYXJyEmfOnEE+nxfhjh07huXlZVSrVTDGUKvVsLS05JOnxx57DPv374dt27BtG6qqYn19XYQhCIIgiDuVTSuXN2/e9Fj1RkZGcPDgwZ5PIS8tLeGBBx6Qnbvi1q1bOHjwoOzsYWxsDGtrax63+fl5n2Irh3McB+fOncOBAwc84baTer2Oubk5HD58WPYKZGFhAbquy87bwuzsrEeZ2k7y+Twsy8K+fftkrx+FlZUVj5X9xIkTAIDbt28Lt7m5Ofz5z38W7WZoaMgjT41GA8899xxM08TY2BgGBgYwMDCAZ599dkc+DgiCIAhiy7BNous6S6fT4t62baYoinDL5XJMURSmaRpjjLF0Os0URWG6rotnarUaMwyDAWAAmGEYzLZtjz8AZts203VdhHHjjkNRFJbNZj3vYIwxwzCYaZoiTYqisHK5LPz5+wuFguc5AKxWq4WGK5fLwo1f/N3pdJoB8JSRaZo+N5lCocA0TRP5kcPyONyXO41BtMs/66Ac5fzUajWmqipTFIVZluUpC16HQTIAgKmqyizLEnFz3HnXNI2ZpunLP4fLg/sql8u+dOVyOaaqKkNT1Hn5JZNJT3wIqH+5LmR/GTkOXiZuN1VVmaZpgflnjLFkMslUVZWdCYIgCGLXsCnl0rZtz6BpWRYzDEMM6JZlsWQyKQbXdDrNarWaR1mwLIspisKSySSzbZvZts1UVfUoE9lsVigZtm2L+LhiFBRHkELG3Xh6dV33KE48Xrdiy93cBIWzLIsB8CgLhUKB5XI5oci4cadfplAoMAAsm80y5lKuZaWGK22dwsuIx6uqKjNNU/i3K8dCocBqtRrTdV0oZTysO++maQpFUpYB0zRZuVwWz8hlIOed38t5d5NMJn1KIv9AQVOBzOVyrFwuM03TWC6XE/XSrv5N02Sqqop0Gobh+2hxw9PL4+Dv5OXB4covAJbL5Tx+PN2y/BIEQRDEbmJTyiUfSN2XYRg+C1oul2MIsawZhuEbeGWlj4dxD/puxSRowJctTFxBcw/ksnKRy+V8aZHDsJBw3DIWBFc83cqwHKebIMUYTcXMTTKZbBmPG55/rrSxgDLqpBwZY0LR59hNizInmUz60s8VYXcZQFLGWfN97riDFD4ZVVV9ChpzvcOdZzeyUupWipnr+VqtxmzbZoVCgSmKEvguDrfK8sutrMvYti3Cu+PkeZYVb4IgCILYTWxKuUyn06EKlZtkMumbxuYEDf6yRU1WcPjgy5VVeXDmipR7QOfWTze6rnvSFaQU6boeqNTJ4QzD8FnP3CjNKWbWfF5Wqjhy3liAcsqRy6kV2WzWY+UMKqNOypGHc6clnU577lVV9SmksiIcpIzzvLvLJqje3PA0BpUnV2jDkOVK0zRPvXKLM1cSDcPw5UtG07SWchCE/DEVZOkmCIIgiN3Gpjb0XL58uaPjgUqlkm9DDJobUgB4NupUKhVYliU2ntTrdayvr+PQoUMizIcffghd1zE0NCR2CN9///3Cf3JyErque3beXr161ZPWRqOBubk5nDx5UriVSiU8+uij4h7NjRePPPKIxy0oXKlUarmx5siRI7h9+zbq9Tp+/vOfe441cnPr1i1A2tH90UcfQVEUjIyMCLd6vQ7LsnxpC0PO/+LioqeMOi1HXmf33Xcf0NxM09/fL9LWaDRgWZanvtAsn9/+9rfi/tq1azAMwxNmYWEBAETZOI6D1157raWMLS4u+o6J4ly7di30WVmuisUiqtWqp17n5+eh6zoYY1hbW0M+n8eTTz7pisVLo9FAtVptKQdByGnv7+/33BMEQRDEbqRr5ZIrgT//+c9lLw9c2Qg6W3FoaAiKogilolKp4KmnnhJH66CpPADA119/DQCYmZnB9PQ0TNMEXErOrVu34DgOJiYmhLJRr9fFgeGlUgmWZQHNNBmGAcMwfEpRX18fUqkUHMcRihQA4RYUDgDW19fR19eHRqMReBzR/v37USqVMDk5ibGxMdlbwMuJK3szMzMYHx/H1NSUJxwvF7fC2QpZwf/kk09w4MABUUadluP3338PNBWgTCaDa9euefKztLQk/k+lUkCIDHClNJ/Pi7zu3bsXaCp+jUYD58+fRzQaxS9/+UvxnMzGxgYURQGaZeU+IF7Osxt+LBOaZf3VV18BAPr6+jAxMQHHcXDgwAGsrq6KOm531NRHH30ESAq6zMTEhOecymKxiHw+j1deeUW4PfHEE0DzxwM4+XwesVhM3BMEQRDEHY9symwHny6Up0hl2k1N8ulRNHcPy9O8fDMGD6Prum/tJl+3xv34WlA+xVmr1cRUNp/ilN9jNzcSuadKO3Vjrl3LQVPCrDnlqyiKL+1BpJs76tHcLR0UX9D6yDD41LH73bws3NPA7cqRuabow9LF86lpmnhfkAzwHelBazfR3ITTasqbw9dB8mf4FL57vWQQlmWJeuRp0DSNqa7NO+70IGQ9MYdP6fMrDL6UhIcLK0e3zKMpV/JSDIIgCIK4k4mwHxRGYptIpVI4fPgwEomE7LUpotEoJicnxRmKu51MJoO9e/d6yieVSmF5eRnLy8uesARBEARB3Pl0PS1OdAb/mb9eKpZ86pdPn/4UGB8fx8bGhrjnyx/eeOMNTziCIAiCIHYHpFxuA8PDwzh06BB+/etf90yxjMfjePvtt/HBBx/4NoLsZnK5HN5++23xG9rvv/8+CoVCyw00BEEQBEHcudC0OEEQBEEQBNEzyHJJEARBEARB9AxSLgmCIAiCIIieQcolQRAEQRAE0TNIuSQIgiAIgiB6BimXBEEQBEEQRM8g5ZIgCIIgCILoGaRcEgRBEARBED2DlEuCIAiCIAiiZ5BySRAEQRAEQfQMUi4JgiAIgiCInkHKJUEQBEEQBNEzSLkkCIIgCIIgegYplwRB7HoqlQoSiQTq9brsRdzhZDIZpFIpNBoN2WvHIPkhiNZ0205JuSR+dIaHhzEzMwM0O/lIJCKuSqUiB79raDQaSKVSyOfzsteOkc/n7/g6yOfz+OMf/4j33nsPQ0NDsjexw7jbcCcD0djYGH7/+9/jyJEjP4pydzfJTyaTQSQSQSwW87lFo1FP2G6Jx+OIRCJIpVKy1x0Bl0k+1vwYuNvGjyHrW6HrdsoIYhsoFApM13XZ2UetVmMAmG3bjDHGyuUyA8DK5bIclLFmeE3TRPhOMU2TmaYpO9+x1Go1pqoqq9VqslfH2LbNTNNkqqoyAExRFJZOp+VgAl728tWqLhRFCfXvBF3Xt/R8uVxmiqL45KFcLjPDMEQedF0PLcuwfANguVxODs5qtRpLJpNMURSWzWZl701jmqZ4ryyrcl0CYJqmefIkp11RFJZMJn1lsxPwtHZDuVxmqqp2ld5yudxRPxNGmPywZh4URQks61a0axeb7cN6ha7rLJlMetzS6TQzDMPjthmC2oymaS37nZ3Ctm0GoON63C4KhQJTFEV23jV02k7Jckn0FMdxkEgkcPToUdkrkJmZGSSTSQwMDMhePjKZDB566CFUq9WOwu9WHMfB6OgoLl26tCVLyrFjx/Dll1/i+vXrYIzh1VdfxZkzZ9paQhljnmtkZEQOAsdxcPz4cayvr8teO8rp06cxOTnpkYdKpYKnnnoKJ0+eBGMMtVoNS0tLGB8f9zzrJplMevJsmiZUVUUikfCEy+fzGB0dxeHDh1GtVnHixAmP/1YYGxuDpmnQdR1jY2PC3XEcxONx/N///R8uXboExhhyuRyq1SruueceEa5QKAAALMsCYwwffPABlpeXcejQITiOI8LtFEeOHJGdWjIyMgJFUXDx4kXZa9sIkh80+5qzZ8/i6tWrsG0bqqpidHS0bTm2axd3Sh8m9ysrKyvYv3+/x22z7Nu3z3NfrVaxd+9ej9uPwRdffAEE5H2nuXnzJg4ePCg7d00qlUImk5Gdt52O26msbRLEVjAMg9VqNWaaZluLgm3bvi98bkWSv/rT6TQrFArCv1t2k+XSNM2eWBFkeNmFfbl3U7bJZJJls9nAuuqGrVguLctiAJhlWbKXj3Q63XHeGGOBVslcLscURQktvzC6sVIB8L2XW0nlOGTrR5C1kNepbE3abgzD8OWjE7LZrC8PrdiK5bKV/Kiq6rG28RmWduXYql1stg8rFAo9tfwF9QFB6e2WctMKfKfSyZi0E+i6vuWxiFvHC4WC7LUjdNJOyXJ5FyOvbwy7ullzl8/nO/4yvHLlCqLRaKBlTOatt97Ck08+KTvvGI7jIJVKIRqNIhKJIB6Py0F6xoULF/Cb3/xGdt40juOgWCzi3XffRS6X67h+wsjn81hbW+up1W4zLC0tAQAGBwdlLx/r6+vQNE12DoRbdp955hmPeyqVwquvvtpx+cXjcTQaDWGhardhhLezJ554Qrg1Gg1MT09jamrKZ+laW1vz3N+4ccNnLeRt6/bt2x53mUqlguHhYdHm3evm6vU6EomE8EskEj4LXiaTQSwWE89Wq1U8/PDDnjByPEFWlwceeACWZfni3w7C5KfRaMCyLDz66KPCjdd5q3Js1y667cNmZmYwMzODvr4+9Pf3o1gstl3P6K6HeDyOiYkJFItF4V+pVKAoikeGuUxyiyNfg+l+l9z/TUxMIB6Pe+ppYWHBI388nuHhYeE2MTEhnucW+bD+tFgsCpmMRqOefLiR40ylUoFx3rhxA48//jjy+Tyi0Sii0ahvbJNlVJb1er3uaSexWAyVSgX1eh2xWAzRaBSNRgPFYlHcy+leWlrCI488ItIdi8V88t6qHrn1e319HUePHvW1JXe5xWIxzMzMYGJiQvg7zdlFnodoNOpZhyrXtVyO6LCdknJ5FzMyMuKbAg26OlH+NsO5c+dw+vRp2fmOw3Ec/OIXv4CiKPjf//6HWq2Gubk5X8fUC/jAdv/998tem6JSqeDee+/F0aNH8e2337aM95577oGiKJ4OXVYAGo0Gzp07h/fee8/j/mNw+/Zt6LouO/uo1+uYnp7G+++/L3sF8vrrr+PVV1/1KHPFYlFMdbrLp9XmgKmpKVy8eBGrq6uIx+N44YUXWiqmn3/+OVRV9Sg7fOrJrXCGUSqVcPjwYdm5LcViEY899hiOHz8OviRAURSgWd+jo6OIRqOwbRu2baNareL8+fPi+YmJCZw9e1ZM2SuKgrW1NU9eK5UKRkdHMTo6CsYYstlsy2UKfApzOwmTn2+++QYA0NfXJ3thfn5edgK2qV2cOHEC/f39ePfdd3H27Fn85z//wZtvvikHE8j18Pjjj+PMmTM4dOiQCLOwsOCbkl1cXISmaULeb9y4AdM0MTU1JcLE43EsLy+jWq2CMYaVlRU4juNpI/Pz856p9bGxMei6LhTOfD6PX/7ylzBNEysrKzh16hSmpqZQKBQwNzcnnkNTgXrllVfw97//HYwxHDlyBP/4xz88YRAS51/+8pfAPrpUKuHLL7/ExsYG1tbWEI1GsbCwIPw7kfXx8XEMDw+DMQbbtgEA9913HyYnJ3H9+nWsr6/j4sWL+Oqrr7C6uupLd71ex/r6Oj7//HP86le/gmVZsCzLI+/t6nFsbAy5XA6qqooxmi+jKRaLOHr0qGjLly5dwssvv+z5ULp48SIsy4Jt26Js+/v7gU2MdS3bqWzKJIhe0G4Kgpv15am+sGlxTqdTSrquMwRs0HBfrdLnJplMimlqvpmj1ZSA/J6gK2xapFX+tpIny7JYMplkaFG2Mnx6zz0VKG9s4PGZptlRvHJ6g66wspFpJ2PMtfSi3XQmJ5fLMbg2mHHM5mabbDYr/Hj5yFOMnFqtxnRdF1Os7cpH0zTfRgtd19vmkbmmbeUpXi5PrabPVFUNLXPDMJimaR43d5r4e915C1rWoaqqmCYvl8uBeeV+cnxueD20uzohTH7C0gDAly9ON+2Cx98JpmkyTdPEVKosl5ygeshms4F1J9e1YRgsmUwKeZXTm81mfX21YRi+qXr5/axZ77Ls8f6Tx5fL5Tz9KV+uUKvVmG3bYgNMqzacTqd9y0/c6eHl416qIU8rt5N1HkbXdV87Y64NQ+50yjIm9xk8XTzdndZjMpkMbD+KonjqV46fuaa05bpiXYx1XIaD4uB0JuHETxIuIO2uVgIUhtyoZMIaRzuh5f7dYm5hzaW7LPjOx7BOfqtsNn+dwDu/bspB0zTR2fCOMewKq7NWBA1mncIH3jBs2+46flVVA+WSKzUyrcqTD0K6rjPbtsUAHgSvG3kgbhW/m7A1UMnmes0wuLwFDZas+X557aRbGTUD1nkahuFJs9zP6Lrui5PTrv3LlLew5jJMfviA3GlddNsuOm3j2WyWZbNZVnYpqUGyyVrUQycKoNLcEY9m/yajB+wulxWzoDxxJVHuK+WPGXks4G0NzRMPDMPw1YWMpmkepY7XIW9vXEGW/WVlVJZLOa22bTOjeQqFLAuFQiGwDtwfJPK9rDh2Wo9BynZQW5bj5/AyNgzDVwZuWQgb6zppp+0lnCA2gdlCueQDadBA205ogzqxTjA3qVy2S0+v4e8LatC9IKhTbIWqqp7OUGarZdOt8ueGWxmDsG2baZrmibtVPlhI58zh75L9EDAgycgWlSDCLKayJSIMo2l9csPjbJU+PsgEEWRF4WXE265s2Ql6ptU7ZLqV//IWlMtW8qNIx3ZxRamdksORy8BNt31YJxt6OqmHoPe6w7WScbcMcmXaTVB/n8vlfIqN2yrJka2bcl7awccTd7pli1uQUtdpeQWNU0EfIOl02tMGeV7dSqAibRSUP8TkvAeli7vJ9SS3M9u2fcqxG+7PZYvnN0xu3fCwrdoprbkktoWNjQ2srq7KzgCAjz76CJqmtVx/FsZ3330HNNfH7AR8kfutW7eAHTjY/L777gParWXpEHmhN/+fb1ThC+75epp4PI5MJiMWac/MzMCyLJw8eVLE4YaH+/rrr2WvHYGvHQuShT/84Q/485//LNYLO46DUqkk/IMWqp8+fRqGYfg2eKC55lFRFLzzzjsi35lMBoqitF0Puby87NuMI3Pp0iUA8IVLJBK4cOGCyGOj0UAikfCt9SyVSvj1r38twmQyGTz33HNIp9OhG0wA4MEHHwRcmzqKxaJY/D80NARFUcS6NH7EUzKZFG13cHAQq6urcBwHlUoFi4uLQFOO+QYL+R2VSkVsdpJZWFiAqqq+ctgOWslPMpnE9PQ06vU6HMfBO++8A03TxIYcue24adcuuu3DnnzySbz11luys4dO6oH3YXzDBprrLdFcf8/l+MqVK8jn86KPU1UVN27cAJp9wsbGBlRVheM4Qla4f71eF/3MtWvXMDAw4HFbWlrybCiq1+uwLAt9fX0inQcOHBB5QXNdpdxW3Vy5cgUA8P333wPNNE5PT4s2hWb7GB0dFfeffPIJDhw4gHq9jpmZmY5kPRaLCRl2HwHGuXz5sthk12g0YBgGdF0Xx5nx9ZbujW6XL1/Ggw8+KH6sopN65Pns7+9HpVIRdcmPe6rX62g0Gjh//jxisRj27t0r4k+lUp6+g6+tRpdjXUftVNY2CWIr8LUvaJrWVVX1fYWqquoz6XPCvp5yuRzTXWsOFUVheouDsWU2a7lkriNoeH42G0+ntCqfbigUCmIKB81pDne58i9d7mY2pwnd4VtZalTX4eyd1oPMViyXLGQNWaFQEHmQLw6k9alBFgKZWnNNGo/LaB67tVV4PQDwWR/d03D8kuWeH7PELz6V2CovbvjzSsCh67nmejiEyH6tedi/28KqaRpTpR8AcKexlVxpmuZ7RyvKW7BcshD5Yc1yTzbXKPO6dluK5LbjJqxdbLUPa0Un9VBuHhWkudaHyhY9Xk9uSynv/7iF0bIskXYuK9ya6Z5mDXJLutb0MZf1THFNs9vN5Szusm9VRjxOXu5yP8fbdpA8yvlsJety3bn9uZXS3ZbkKWV5ap4160hxTXF3Uo+WZYl0uuXHXW68HafTaU/8/J7nI6i9c/+gMuB00k5JuSR2FN7BhZnTyyHK5VYxt6Bc7jRmwBTTT5WtKpft5InYPfA1a93U5VaVS5Kf3Q9Xen9MgpYA/FTptJ3StDixo3z44YdIJBKtzenbwNjYmOdXT+5kXnrpJSwtLQVOuf3UmJ2d3dJRVyMjI5iamvKduUfsLur1Ol555RV8/PHHXfUNIyMjmJ2dlZ07huRnd9NoHt0WdGzUTsKXAPzU6aadknJJ7BiO42B6errlGjDih3V3V69exYsvvijW+BDhJBIJ/P3vf8cf/vAHKq9dCD8svFQqbWod9lYh+dm98IPwP/zwQ9lrRymVSlhdXf1JGwS6bacR9sP6I4K4I6hUKnjsscfEfblc3pJlazfTaDTwzjvv4PDhw77fuCYIgiCIOxVSLgmCIAiCIIieQdPiBEEQBEEQRM8g5ZIgCIIgCILoGaRcEgRBEARBED2DlEuCIAiCIAiiZ5BySRAEQRAEQfQMUi4JgiAIgiCInkHKJUEQBEEQBNEzSLkkCIIgCIIgegYplwRBEARBEETPIOWSIAiCIAiC6BmkXBIEQRAEQRA9g5RLYluoVCpIJBKo1+uyF3EHkslkkEql0Gg0ZK8dg2QmHKof4k7gTpBDYpfACKLH5HI5pmkas21b9iK2QLlcZgAYAGZZluzdFbquMwAsmUwKt3K5zFRVZbVazRN2J/gpywwva8MwZK9Q7pb6cefvx8Y0TQaAKYoie3VML9vonUqQHFqWxXRdZ7lczhOWuWSZX8TdAdU04WGrjb9cLjNFUXyDUK1WE52MoigsnU57/GX4YMY7pGQy6YlT7rD4paqqJ56fGqZp+vJYKBQYAF+ZtwOAbzDgA0c3cZmmyUzTlJ07JkxmbNtmyWRS1K1hGL4wbizLYslkkimKImRBzh+nUCgwXddl556gaZpPvjVNY9ls1uPWjl7Vj67rrFwuy84d06v6CWqzch2Uy2VmGIbw1zRtR5XpdDrd1UdAEEFtdKfppk+wbZsBYIVCQfYKJUgOuTyEfTBw5Z24O6BpcaKnnD59GpOTkxgYGBBujuNgdHQUg4ODYIzhs88+w5kzZ5DJZDzPcjKZDM6dO4fLly+DMYZcLofp6WlcvHjRE86yLDQ/kGDbNhRFwenTpz1hfoocOXLEc3/z5k0oiuIp807Zt2+f535kZASKovjKejsJkhkAOHbsGJaXl2HbNizLQrVaxbFjxzxh3KiqiqGhIaytrcG2bWiahueee84zhec4DhKJBI4ePep5tpdUq1Xs3bvX5/bwww973Drh4MGDnvvdXD8AYJqmaLOMMczOzgq/SqWCp556CidPngRjDJZlYX19HU8//bQnju1kZWUF+/fvl527RtM02WlHCeoT8vk84vG4JxwAfPHFFwCAn/3sZ7JXKEFyODAwgKmpKTQaDczMzHjCE3chsrZJ3N1sRSQsy2IImA7K5XK+r2jDMLr6utc0zWflcJPNZjuazrJtm2maJjvvGgzD6NoCFgS3RgWRzWa7qputWC7DZIa7u60p2Ww2MGwYpmn68mgYBqvVasw0zZbyJJNOp7uy7Lip1Wpdt6tareZLO6fb+tmK5bKX9aPretdyout6R5bEXk3ZA9h0WXF0Xe9JG+0lvN+TLepbIUwOC4VCoKWbLJd3F2S5JHrG0tISAGBwcNDjfu3aNWia5vmK3r9/PyzL8oRrxfr6Og4cOCA7C95++228+uqrsrOgXq8jlUoBzS/ser0e+BWPZtjh4WFEo1HP5oVMJoPh4WGgaQGLx+OIRCIoFovimUgkgkql4oknEokgEokgFosJP+6fSCSEfyKRgOM4wh/Nd8ZiMUQiEaRSKZ8FjD9bLBY9zxWLRfHuaDSKiYkJj//CwoLPAsp54IEHYFmWLy3bQZjMcPdDhw4JtwceeAAA8M033wi3IBqNBvL5PObn53H16lWPXz6fx9DQkMetFalUCpVKBf39/ejr60Mmkwm1ymQyGUQiESEjnMXFRei6jnq9LupSjsMtC7FYDDMzMz/Z+umEer2OTCaDPXv24L333pO9BfF4HI1GQ/Qt7TYcNRoNT5uLxWLCjz+3b98+0bYTiYTr6R/afSqVQjQaFXUtv29ubs5npXa3R16/vE1yuYhGo0J2o9EootGor13z8G5ZyWQynr5M7hMqlQruvfdeVKtVnDlzBpFIRITn+ZT7h4mJCZHHSLPvkQmTwyeffBIAcOXKFY87cXdByiXRM27fvg1d12VnT+cv41a2wuAD8Z/+9CfZC2gqDGtra3jppZdkL8HQ0BB+//vf4/z581haWsL4+DhM05SDAQAmJycxOzuL9fV1fP/998J9fn5eDPjnz5+HaZrQdR3//Oc/8e9//xuzs7NQFEVMNY+Pj2N4eFhM2wPAfffdBzTLZHR0FNFoFLZtw7ZtVKtVnD9/XrxvYmICZ8+exaVLl8AYg6IoWFtb8yhH5XIZkAb5YrGIo0eP4vjx42CM4erVqzhz5oxnoJqfn287/ceny7aTMJm5ffs20PwQkFlYWJCdBJlMBqqq4rnnnsPg4CDuueceOUhXvPnmm1hYWMD8/DxefPFFAMCJEyfkYACAsbEx6LruUwqvXr2KPXv2YHFxEaurqzBNEy+//LLwl2Xh+vXrmJ6exujoqCcemd1WP4ODg7hw4YJQWuLxuE8xQ1Pheeihh3D27Nm2Mjo1NYWLFy9idXUV8XgcL7zwQujHQ6PRgKZpopzL5TIURRH+i4uL0DQNp06dwtTUFHK5HC5fvuyJIx6PY3l5GdVqFYwxqKqK48ePC3/en7nTILfHS5cu4eWXX8ajjz4KNPub69evY319HRcvXsTGxgbW1tagqipu3rwp4kGIrIyPj3s+vOU+YWRkRHzI86VEfDkC/8vTgmYezpw5I/KYzWaFXxBBcnjw4EH897//lZ2JuwhSLu9iKpWK+DLlF1xfvvzqRAHcLO0G/0qlgtdeew0ff/xx4EAGAK+//jqSyWSoP+e7777DysoKotEoDhw4EPrufD6PjY0NwKUMommR4J3wW2+9haGhIayuror7gYEBrK2tiXTs2bMHjUZDKNerq6vCApROp6GqKqampjAwMICBgQHEYjGsrKwATevEmTNn8Nlnn2FkZAQA0N/f71Ncvv76a59V+Pnnn0c6nRZKEB/o3APV3NwcHnnkEXHfDdzawa/x8XGMj4973MKswr2gv79fdhKMjY2BMYZarYbl5WVomuazrHTLjRs34DgONE3zraWUWV1d9QzUAFAqlRCNRkV9yOWeTqdx8OBBjywA8Fm/OkVuv3Nzc3jsscc8bmHrnXtBWP1MTU1hdXUVa2trwuIVtJ5ydnYWjDF88MEHOHv2bMt1nN9//71oM0eOHEFfX58cRPDOO+94ynlkZATLy8vC/+rVq1hfX8ebb76JwcFBbGxseNZOzszMoFqtYnZ2VrTj/fv3o1qtijALCws+Zfz555+HaZqi/nm/w5U/d3+zd+9eEW59fd0nb2Gy4pa5oD5haWkJqqr6LNB8PbL745Snj1unT5w4gampKeHfKbxeiLsTUi7vYkZGRsTCen7hh0UxnosrN50QNJDv2bMn0B3SF75MvV7H6dOnUa1WQ8Pl83lYltXSaolmXP/5z3/wr3/9C7FYDC+99FLgVA/nypUr0DRNdMbc6ufuhBuNBizLwmuvvSbc3Lz33nvYs2cPVFX1DeaXL1/2WDzQVEwef/xxoKn8qarqKfsbN274LDnXrl3zKJyVSgXr6+v43e9+J9z4AMKVGv6x0E29uuGDP79M02y5UaMdQbLBFZQgPz792oqhoSG88cYbWF9fD7SsdMqpU6dw8uRJPPvsszh58iQ2NjZ8U9ocLg9uGanX60Jh4SwsLHgsZpcvXxZWUbimZ8Nkvh1y+9V1HeVy2eM2NjYmPxZKUB1stX4GBwfxxhtvwLKsQOslmtOryWQSc3NzspdgfHwcU1NTiMVi+NOf/oQPP/wwNL7p6WlPOcuUSiX89a9/FUrZ1atXPW3rk08+gWEYHqXtyy+/9CiT8/Pzog3D1R6feeYZ4cYtpLLypyiKCMdlSd7QFSYrbpmT+wQA+PTTT31uaL5XTsvQ0BAKhQJSqVTgtD9BdAIpl0TP2Lt3r+crnjM6OopqteoZiL788suWOyrr9Tqefvpp/Otf/xIKnrz+CQAuXbqEZDLp+yKXGRoaEl/fjuNgcHCwpQIkDyx/+9vffJ0wHxDClICBgQHk83nUajWMj48LBZV31u5BuFKpwLIsMVDNz8/71oNdvnzZZ/UqlUoeq8WtW7cASTH56KOPoCiKUCaDrCtByDvJt4MwmeFWu+vXrwu3r7/+GugiXa2sWJ0yNTWFkZERbGxs4LvvvsOJEydCp8WDBuogReLChQtifTBX9O+//37hPzk5eVfUz3fffSc7+QizgnK4FZH3LVNTU4HtMaic3fCPALe1uFQq4Ve/+pW4X1pa8nzcOY6DfD6P3/72t8JNnhHgSwR4/+Q4Dt5++208++yzIgxcCiGXEy5L7n4tKA/j4+M++ZL7BO52+PBhjxtCFFE0FXs+NS9/BMt0Wt/E3QUpl0TP4F/Z3FLGeeKJJ6AoCk6dOgU0O8l8Po833nhD3Lun6hzHwfHjxz1T4ZVKBd9++60r1h/c5ubmQgf7IAYGBjxTYWHwAdVxHExMTIiOv16vC8tVWMcMALFYTCiR8vT70NAQFEURA0+leQRLMpkUA+Pg4CBWV1fhOA4qlQoWFxeB5jT9xMQEHMcR1o2+vj6kUik4jiMGRz4QzczMCOsO58aNG4Br04TMwsICVFX1DFjbRZjMDA0NQdM0/O1vfxN5PXfuHNLptEgXn56HS4Z4vh3Hwbvvvuuz/nI2NjbEkoZOeOutt8RGhTCuXbsmZISXK59qdRwHTnMzCABhaefLLm7duiVkjU9fumXNzW6pH/fyiEql4mkTPLyu60Lm+eY1/q5Go4ELFy4gmUyK94axvLzcsjy4AsQ/vurNTTGcxcVFz4citzju27dPtLcjR46IJRKNRgPxeByqqgpro9vCx9sjn9au1+toNBo4f/48YrEY9u7di3w+L+S1VCp51th++umnvv4mSFb27Nnj2UAW1CegOcXe19cn0s0plUro7+9HPp9HsVhEPp/3lEs0GhX/y7STw1YbMIm7AHn7OHF3s1WRCDtupFAoMFVVGZqHW7uP6uC/asGfS6fTDAEHpMtHxxiG4XPrFfzYDF3XWa1WE4cSp9NpccSGnA837gOjFUXxlUkul/OUh+xfq9WYqqqeZzVNY6rrlzFs2xZh3EfCpNNpcZC4pmm+I3T4kTFhh15rmuZLTyu2chQRayEzQQfvu9PL/TjZbNZz8L5hGL5jcdLptCeMqqo9k6GgcuXH0vD6kH8MgLnkPUjWgui2frZyFBHbQv3wPDHXL7jwclAUxVcWtm2zdDot2kVQnFshl8uJ9/MjqTiGYXiOPLIsiymK4mlvtVpNyE5Y+uX2aNu2KCMenrdPfkA+P9bJnZ4gWWIBsqJKPxQQlAbmaiuqqnrc+YH1vH7d/ZL7PUG0kkME/AAAHUV0d0E1TXjYauMP+zUPYnfAPwK6qb+tKpckM52zmfrZqnJJ9fPjUygUfB8bQecH7xSt5JDOuSQYKZfEdrBdv0NMbC/cEhJmqdhOSGbaQ/Vz96JLh8lzhV9WOHeCdnKohfzUKSmXdxdU08S2UG7+RnBYB0TcWWSzWZZMJn3TyDsJyUw4VD93N+4peT69vZXZgs3STg7pt8UJToT9MBVKEARBEATRNY7j4NixY3jxxRd9p3rE43HPcVKkctwdkHJJEARBEARB9Aw6ioggCIIgCILoGaRcEgRBEARBED2DlEuCIAiCIAiiZ5BySRAEQRAEQfQMUi4JgiAIgiCInvH/ATjqaLsdQO8JAAAAAElFTkSuQmCC\"\u003e\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003eThe c-statistic in the test set was 0.92 (95%CI 0.90\u0026ndash;0.95). On internal validation, there was minimal model optimism (c-statistic in the test set 0.91 (95%CI 0.87\u0026ndash;0.95)). The optimal cutpoint identified by maximizing Youden\u0026rsquo;s index was 0.50 (i.e., the model estimating a 50% probability of a true PSC diagnosis). Using this dichotomized cutoff, the LASSO regression model had a sensitivity of 87.4% (195/223; 95%CI 82.4\u0026ndash;91.5%) and positive predictive value of 70.7% (195/276; 95%CI 64.9\u0026ndash;76.0). Figure 1 shows ROC curves for ICD-10 only and PSC prediction model.\u003c/p\u003e\n\u003cp\u003eLLM alone had a sensitivity of 91.7% (187/204; 95%CI 87.0-95.1) and a positive predictive value of 51.0% (187/367; 95%CI 45.7%-56.2). C-statistic for LLM alone was 0.77 (95%CI 0.75\u0026ndash;0.81). Table 2 shows test characteristics for the ICD-10 code alone, the LASSO regression model, and the LLM. \u0026nbsp;\u003c/p\u003e\n\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv\u003eTable 2\u003c/div\u003e\n \u003cdiv\u003e\n \u003cp\u003eTest Performance of ICD only, LASSO regression model, and large language model for identifying PSC\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e\n \u003cp\u003eC-statistic\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c2\"\u003e\n \u003cp\u003eICD-code only\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003eLASSO Regression Model\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003eLarge Language Model\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003cth align=\"left\" colname=\"c2\"\u003e\n \u003cp\u003e0.87 (0.84\u0026ndash;0.89)\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c3\"\u003e\n \u003cp\u003e0.92 (0.90\u0026ndash;0.95)\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003e0.77 (0.75\u0026ndash;0.81)\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eSensitivity\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e87.6 (82.6\u0026ndash;91.6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\n \u003cp\u003e87.4 (82.4\u0026ndash;91.5)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003e91.7 (87.0-95.1)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003eSpecificity\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e86.0 (83.0-88.6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\n \u003cp\u003e86.4 (83.4\u0026ndash;89.1)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003e63.8 (59.4\u0026ndash;68.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colname=\"c1\"\u003e\n \u003cp\u003ePositive Predictive Value\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\n \u003cp\u003e68.8 (63.1\u0026ndash;74.1)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\n \u003cp\u003e70.7 (64.9\u0026ndash;76.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colname=\"c4\"\u003e\n \u003cp\u003e51.0 (45.7\u0026ndash;56.2)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e"},{"header":"Discussion","content":"\u003cp\u003ePrior identification of patients with PSC in administrative datasets required the use of multiple ICD-9 codes, including a cholangitis-specific ICD code and an IBD-related ICD code.\u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e Despite the use of multiple diagnostic codes to create an optimal algorithm, this method only had a sensitivity of 56% and positive predictive value of 59%.\u003csup\u003e10\u003c/sup\u003e The ICD-9 code for cholangitis alone had a sensitivity of 83.7% and PPV of 7.2%.\u003csup\u003e10\u003c/sup\u003e These prior data suggested that with the ICD codes available, it was not possible to accurately identify patients with PSC.\u003c/p\u003e \u003cp\u003eWith the development of a new ICD-10 code specific to PSC, we observed markedly improved sensitivity and positive predictive value using the ICD-10 code alone as a predictor of PSC. We investigated whether we could optimize an algorithm for predicting PSC by including additional variables that we considered \u003cem\u003ea priori\u003c/em\u003e to be associated with a true PSC diagnosis (e.g., elevated alkaline phosphatase, ursodiol treatment, concurrent inflammatory bowel disease, etc.). Though we developed and internally validated a prediction model with excellent discrimination, the model\u0026rsquo;s performance was not meaningfully improved over the ICD-code alone (c-statistics of 0.91 vs 0.87, sensitivities of 87.9% vs 87.6%, and PPVs of 70.3% vs 68.8%, respectively. Based on our data, it seems clear that the ICD code alone is sufficient for use in database-related research.\u003c/p\u003e \u003cp\u003eNotably, the LLM yielded substantially lower sensitivity and discrimination. Compared to ICD-10 codes that are more inclusive for billing purposes, our prompting of the LLM required explicit documentation of PSC to generate a positive prediction. This likely explains GPT-4o\u0026rsquo;s conservative classification compared to ICD-10 code alone. In addition, in our implementation, the LLM was limited to evaluating unstructured text in clinic notes but did not have access to other structured (i.e. laboratory tests, diagnostic codes) and unstructured (i.e. MRCP reports, ERCP reports, pathology reports) that adjudicators in this study used to determine the presence/absence of PSC. Providing these data to LLM may improve LLM performance but given that ICD-10 code alone demonstrated strong discrimination and is easier to extract from the electronic health record, the added complexity of feeding multiple structured and unstructured data elements did not appear justified.\u003c/p\u003e \u003cp\u003eThe PSC prediction model included additional variables including alkaline phosphatase, ursodiol use, and associated diagnoses including IBD and acute cholangitis. The prevalence of IBD in the PSC patients included in this study was 76%, in line with the reported prevalence of 70\u0026ndash;80% in the general PSC population.\u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e In addition, acute bacterial cholangitis is a classic complication among patients with PSC.\u003csup\u003e\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e Ursodiol was also incorporated in the optimal PSC prediction model \u0026ndash; this is because while there are no current therapies to treat PSC, many patients with PSC are prescribed ursodiol.\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e Studies demonstrate that while ursodiol may improve liver tests, ursodiol does not improve survival and may lead to increased rates of adverse events.\u003csup\u003e\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e With new potential therapies for PSC on the horizon,\u003csup\u003e23\u003c/sup\u003e rates of ursodiol use among patients with PSC may decrease, making this variable less effective in a PSC prediction model. Finally, autoimmune hepatitis and PBC were eliminated by LASSO regression. The prevalence of autoimmune hepatitis overlap in our PSC cohort (15%) is similar to what has been reported in the general PSC population (7\u0026ndash;14%) though notably the \u0026ldquo;prevalence\u0026rdquo; of PBC overlap (58%) far exceeds case reports in the literature.\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e As these diagnoses were made based on ICD-10 codes, we strongly suspect that most PSC patients in our cohort experienced miscoding for one or more billable events (e.g., the PBC diagnosis code was inputted instead of the PSC code) leading to the very high proportion of PSC patients also carrying a PBC diagnosis code. Furthermore, our data showing expected prevalence of IBD and autoimmune hepatitis in our PSC cohort supports the validity of our approach to validate true PSC diagnoses.\u003c/p\u003e \u003cp\u003ePSC has been recognized as a spectrum of disease ranging from small duct PSC to classic, large duct PSC. While patients with large duct PSC have classic diagnostic features to assist with diagnosis, small duct PSC, a rarer form of PSC with better prognosis, can often be challenging to diagnosis.\u003csup\u003e\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e The definition of PSC in this population included classic diagnostic features of large duct PSC and also hepatologist/gastroenterologist statement of PSC even without known diagnostic features. This definition attempts to capture patients with PSC across the spectrum of disease. However, given diversity of PSC presentations, future studies may benefit from evaluating the validity of the ICD-10 code within a larger and more diverse administrative dataset.\u003c/p\u003e \u003cp\u003eThis study has several limitations. First, data were limited to a single center experience which does not capture different coding practices across systems. Second, variables used within the PSC optimization model were limited by variables available in the electronic health record. Finally, the study cohort included patients with positive ICD-10 code for PSC and patients identified in gastroenterology/hepatology and primary care notes or discharge summaries to have \u0026ldquo;PSC\u0026rdquo; or similar iterations in the notes by national language processing. Thus, negative predictive value could not be calculated within this study population and positive predictive value must be interpreted with caution as the population is enriched with patients likely to have PSC which would artificially inflate the positive predictive value.\u003c/p\u003e \u003cp\u003eIn conclusion, we validated the new PSC-specific ICD-10 code and demonstrated use of either the ICD-10 code alone or an optimized PSC prediction model including additional clinical variables are effective in identifying patients with PSC in administrative databases. Further, an LLM approach solely using unstructured data from notes was inferior. Future studies can safely utilize the ICD-10 code for investigating PSC.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003ePSC; primary sclerosing cholangitis\u003c/p\u003e\n\u003cp\u003eNLP; national language processing\u003c/p\u003e\n\u003cp\u003ePPV; positive predictive value\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAMA; anti-mitochondrial antibody\u003c/p\u003e\n\u003cp\u003eANA; anti-nuclear antibody\u003c/p\u003e\n\u003cp\u003eSMA; smooth muscle antibody\u003c/p\u003e\n\u003cp\u003eSLA; soluble liver antigen antibody\u003c/p\u003e\n\u003cp\u003eMRCP; magnetic resonance cholangiopancreatography\u003c/p\u003e\n\u003cp\u003eERCP; endoscopic retrograde cholangiopancreatography\u003cbr\u003e\u0026nbsp;IBD; inflammatory bowel disease\u0026nbsp;\u003c/p\u003e\n\u003cp\u003ePHI; patient health information\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eFunding:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors of this study were supported by a PSC Partners Young Investigator Award (Li), K23DK139455 (National Institute of Diabetes and Digestive and Kidney Diseases, Ge), P30DK026743 (UCSF Liver Center Grant, Ge and Lai), T32DK060414 (National Institute of Diabetes and Digestive and Kidney Diseases , Wang) and R01DK133527 (National Institute of Diabetes and Digestive and Kidney Diseases, Lai), K24AG080021 (National Institute on Aging, Lai). \u0026nbsp; The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or any other funding agencies. \u0026nbsp;The funding agencies played no role in the analysis of the data or the preparation of this manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting Interests:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors of this manuscript have the following potential conflicts of interest to disclose as described by:\u003c/p\u003e\n\u003cul type=\"disc\"\u003e\n \u003cli\u003eDr. Jin Ge previously received research support (to institution) from Merck and Co. \u0026nbsp;He previously served on an advisory board for Gilead Sciences and previously consulted for Madrigal Pharmaceuticals and Astellas Pharmaceuticals/Iota Biosciences.\u003c/li\u003e\n \u003cli\u003eDr. Jennifer C. Lai receives research support (to institution) from Lipocene and Gore Medical; serves on an advisory board for Boehringer Ingelheim, Cosmo; and consults for Novo Nordisk.\u003c/li\u003e\n \u003cli\u003eDr. Michael Li receives research support (to institution) from FUJIFILM Wako and previously consulted for Ono Pharmaceuticals.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eAuthor Contribution Statement:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003eMW: study design, data analysis, data interpretation, manuscript writing, funding\u003c/li\u003e\n \u003cli\u003eMD: data analysis, data interpretation, manuscript editing\u003c/li\u003e\n \u003cli\u003eMD: data analysis, data interpretation, manuscript editing\u003c/li\u003e\n \u003cli\u003eCF: data acquisition, data interpretation, manuscript editing\u003c/li\u003e\n \u003cli\u003eMYL: data analysis, data interpretation, manuscript editing\u003c/li\u003e\n \u003cli\u003eJRB: \u0026nbsp;data interpretation, manuscript editing\u003c/li\u003e\n \u003cli\u003eJG: study design, data interpretation, manuscript editing, funding\u003c/li\u003e\n \u003cli\u003eJCL: data interpretation, manuscript editing, funding\u003c/li\u003e\n \u003cli\u003eML: study design, data analysis, data interpretation, manuscript editing, funding\u003c/li\u003e\n\u003c/ul\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eBoonstra K, Beuers U, Ponsioen CY. Epidemiology of primary sclerosing cholangitis and primary biliary cirrhosis: a systematic review. Journal of hepatology. 2012;56(5):1181\u0026ndash;8.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDyson JK, Beuers U, Jones DE, Lohse AW, Hudson M. Primary sclerosing cholangitis. The Lancet. 2018;391(10139):2547\u0026ndash;59.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChapman R, Fevery J, Kalloo A, Nagorney DM, Boberg KM, Shneider B, et al. Diagnosis and management of primary sclerosing cholangitis. Hepatology. 2010;51(2):660\u0026ndash;78.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHaapam\u0026auml;ki J, Tenca A, Sintonen H, Barner-Rasmussen N, F\u0026auml;rkkil\u0026auml; MA. Health‐related quality of life among patients with primary sclerosing cholangitis. Liver international. 2015;35(9):2194\u0026ndash;201.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMol B, van Munster K, Bogaards J, Weersma R, Inderson A, de Groof J, et al. Health-related quality of life in patients with primary sclerosing cholangitis: a longitudinal population-based cohort study. Journal of Hepatology. 2022;77:S338-S9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTrivedi PJ, Bowlus CL, Yimam KK, Razavi H, Estes C. Epidemiology, natural history, and outcomes of primary sclerosing cholangitis: a systematic review of population-based studies. Clinical Gastroenterology and Hepatology. 2022;20(8):1687\u0026ndash;700. e4.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eToy E, Balasubramanian S, Selmi C, Li C-S, Bowlus CL. The prevalence, incidence and natural history of primary sclerosing cholangitis in an ethnically diverse population. BMC gastroenterology. 2011;11:1\u0026ndash;7.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang M, Auerbach A, Oreper SM, Hohmann SF, Lai JC, Rubin JB. Leveraging a new ICD-10 diagnosis code to characterize hospitalized patients with primary sclerosing cholangitis. Clinical Gastroenterology and Hepatology. 2022.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMehta TI, Weissman S, Fung BM, Sotiriadis J, Lindor KD, Tabibian JH. Global incidence, prevalence and features of primary sclerosing cholangitis: a systematic review and meta-analysis. Liver international. 2021;41(10):2418\u0026ndash;26.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMolodecky NA, Myers RP, Barkema HW, Quan H, Kaplan GG. Validity of administrative data for the diagnosis of primary sclerosing cholangitis: a population-based study. Liver International. 2011;31(5):712\u0026ndash;20.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePonsioen CY, Assis DN, Boberg KM, Bowlus CL, Deneau M, Thorburn D, et al. Defining primary sclerosing cholangitis: results from an international primary sclerosing cholangitis study group consensus process. Gastroenterology. 2021;161(6):1764\u0026ndash;75. e5.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBowlus CL, Li CS, Karlsen TH, Lie BA, Selmi C. Primary sclerosing cholangitis in genetically diverse populations listed for liver transplantation: unique clinical and human leukocyte antigen associations. Liver Transplantation. 2010;16(11):1324\u0026ndash;30.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAre VS, Vilar-Gomez E, Gromski MA, Akisik F, Lammert C, Ghabril M, et al. Racial differences in primary sclerosing cholangitis mortality is associated with community socioeconomic status. Liver International. 2021;41(11):2703\u0026ndash;11.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang M, Auerbach A, Oreper SM, Hohmann SF, Lai JC, Rubin JB. Leveraging a new ICD-10 diagnosis code to characterize hospitalized patients with primary sclerosing cholangitis. Clinical Gastroenterology and Hepatology. 2023;21(10):2692\u0026ndash;4.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang M, Harris A, McCulloch CE, Wadhwani SI, Lai JC, Rubin JB. Racial differences in primary sclerosing cholangitis: A retrospective cohort study leveraging a new ICD-10 code. Annals of Hepatology. 2025;30(1):101901.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFar AT, Bastani A, Lee A, Gologorskaya O, Huang CY, Pletcher MJ, et al. Evaluating the positive predictive value of code-based identification of cirrhosis and its complications utilizing GPT-4. Hepatology. 2025;81(6):1753\u0026ndash;63.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMcMurry AJ, Phelan D, Dixon BE, Geva A, Gottlieb D, Jones JR, et al. Large Language Model Symptom Identification From Clinical Text: Multicenter Study. J Med Internet Res. 2025;27:e72984.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFriedman JH, Hastie T, Tibshirani R. Regularization paths for generalized linear models via coordinate descent. Journal of statistical software. 2010;33:1\u0026ndash;22.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNguyen A, Torabi Sagvand B, Alizadeh M, Nguyen C, Scott W, von Rosenvinge EC. Primary sclerosing cholangitis and pancreatic cancer: a retrospective cohort study of United States veterans. Frontiers in Gastroenterology. 2023;1:1076788.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRossi RE, Conte D, Massironi S. Primary sclerosing cholangitis associated with inflammatory bowel disease: an update. European journal of gastroenterology \u0026amp; hepatology. 2016;28(2):123\u0026ndash;31.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLoftus E, Harewood G, Loftus C, Tremaine W, Harmsen W, Zinsmeister A, et al. PSC-IBD: a unique form of inflammatory bowel disease associated with primary sclerosing cholangitis. Gut. 2005;54(1):91\u0026ndash;6.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLindor KD, Kowdley KV, Luketic VA, Harrison ME, McCashland T, Befeler AS, et al. High-dose ursodeoxycholic acid for the treatment of primary sclerosing cholangitis. Hepatology. 2009;50(3):808\u0026ndash;14.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbbas N, Quraishi MN, Trivedi P. Emerging drugs for the treatment of primary sclerosing cholangitis. Current opinion in pharmacology. 2022;62:23\u0026ndash;35.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBoberg KM, Chapman RW, Hirschfield GM, Lohse AW, Manns MP, Schrumpf E. Overlap syndromes: The International Autoimmune Hepatitis Group (IAIHG) position statement on a controversial issue. Journal of Hepatology. 2011;54(2):374\u0026ndash;85.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDeliwala S, Sundus S, Haykal T, Elbedawi MM, Bachuwa G. Small duct primary sclerosing cholangitis: an underdiagnosed cause of chronic liver disease and cirrhosis. Cureus. 2020;12(3).\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"},{"header":"Table 1","content":"\u003cp\u003eTable 1 is available in the Supplementary Files section.\u003c/p\u003e\n"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"digestive-diseases-and-sciences","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ddsj","sideBox":"Learn more about [Digestive Diseases and Sciences](http://link.springer.com/journal/10620)","snPcode":"10620","submissionUrl":"https://submission.nature.com/new-submission/10620/3","title":"Digestive Diseases and Sciences","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"primary sclerosing cholangitis, algorithm, ICD, database, artificial intelligence","lastPublishedDoi":"10.21203/rs.3.rs-9152968/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9152968/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eRetrospective studies investigating primary sclerosing cholangitis (PSC) have been limited by the absence of a PSC-specific diagnostic code. In 2018, a new PSC-specific ICD-10 code was introduced.\u003c/p\u003e\u003ch2\u003eAims\u003c/h2\u003e \u003cp\u003eWe aimed to validate the new ICD-10 code and compare it to other methods of identifying patients with PSC.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eAll gastroenterology/hepatology and primary clinic notes and discharge summaries were extracted from UCSF Epic Clarity database and potential PSC patients were identified using natural language processing (NLP). PSC diagnosis was determined by physician adjudication through chart review. LASSO regression was used to develop and internally validate a PSC prediction model. Separately, we tested large language model\u0026rsquo;s (LLM) ability to distinguish PSC from non-PSC patients.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eAmong 867 patients identified using NLP, 226 (26%) patients were adjudicated to have a true PSC diagnosis. The LASSO model selected ICD-10 code, alkaline phosphatase\u0026thinsp;\u0026gt;\u0026thinsp;120 IU/L, ursodiol use, inflammatory bowel disease, and history of cholangitis. ICD-10 code alone had a c-statistic of 0.87, sensitivity 87.6%, and PPV 68.8%. The LASSO model had a c-statistic of 0.92, sensitivity 87.4%, and PPV 70.7%. LLM had a c-statistic 0.77, sensitivity 91.7%, and PPV 51.0%.\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eThe PSC-specific ICD-10 code had excellent discriminatory capacity for identifying patients with PSC. While an optimized PSC prediction algorithm had slightly improved test characteristics, ICD-10 code alone was sufficient in identifying patients with PSC, supporting the use of the ICD-10 code in future database studies of PSC. In contrast, LLM had inferior discrimination compared to either ICD-10 code or the prediction model.\u003c/p\u003e","manuscriptTitle":"Identification of primary sclerosing cholangitis: ICD-10 code validation and comparison with a large language model approach","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-01 10:17:13","doi":"10.21203/rs.3.rs-9152968/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-04-29T12:43:04+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-20T21:58:13+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-18T17:56:55+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"62314331347906050174537067245621603983","date":"2026-03-31T10:47:41+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"59981577494298039030248243600157961332","date":"2026-03-31T01:24:25+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-03-30T19:27:17+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-03-18T19:09:45+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-03-18T05:10:58+00:00","index":"","fulltext":""},{"type":"submitted","content":"Digestive Diseases and Sciences","date":"2026-03-17T23:41:53+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"digestive-diseases-and-sciences","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ddsj","sideBox":"Learn more about [Digestive Diseases and Sciences](http://link.springer.com/journal/10620)","snPcode":"10620","submissionUrl":"https://submission.nature.com/new-submission/10620/3","title":"Digestive Diseases and Sciences","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"ea1e911d-b6dc-45de-8e46-70c7d75df4a8","owner":[],"postedDate":"April 1st, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Revision requested","date":"2026-04-29T12:43:04+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-05-09T08:38:33+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-01 10:17:13","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9152968","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9152968","identity":"rs-9152968","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}