Medical pre-training and fine-tuning improve large-language-model prediction of rheumatoid-arthritis disease activity

Medical pre-training and fine-tuning improve large-language-model prediction of rheumatoid-arthritis disease activity | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Medical pre-training and fine-tuning improve large-language-model prediction of rheumatoid-arthritis disease activity Suguru Honda, Katsunori Ikari, Mayuko Fujisaki, Eiichi Tanaka, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7159212/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Large-language models (LLMs) already excel at extracting clinical facts from electronic health records and drafting differential diagnoses. However, when applied to clinical support for rheumatoid arthritis (RA), despite the fact that the accuracy of disease activity prediction is directly related to the adjustment of treatment intensity, the performance of LLMs and how much they can be improved by pre-training and fine-tuning for medical use have not been fully tested. We therefore trained privacy-preserving, on-premise Llama-2 models with medical domain pre-training (Meditron) and QLoRA fine-tuning, and compared their two-year predictions of RA activity and disability with logistic regression, random forest and XGBoost. The refined LLMs surpassed conventional models on most Disease Activity Score (DAS)-based outcomes, matched them on remission tasks, retained reliable calibration, and offered small but consistent net-benefit advantages where high-disease-activity was rare, while avoiding the clinical harm observed for tabular methods in high-disability prediction. These results show that endpoint-specific, locally deployable LLMs can complement or replace established tabular models in RA management without sacrificing data privacy. Biological sciences/Computational biology and bioinformatics Health sciences/Health care Physical sciences/Mathematics and computing Health sciences/Medical research Health sciences/Rheumatology large language models disease activity rheumatoid arthritis prediction pre-training fine-tuning Figures Figure 1 Figure 2 Figure 3 Figure 4 INTRODUCTION In the past few years, advancements in Large Language Models (LLMs) have shown substantial potential for improving the management and diagnosis of diseases, extracting clinical information from Electronic Medical Records (EMRs), and assisting in creating pathology and imaging reports 1 – 3 . Their emerging role as clinical decision-support tools has prompted active exploration of how they might be integrated into EMR systems to advance medical practice, research, and education. Yet several critical obstacles remain—most notably predictive accuracy on task-specific outcomes and practical issues of data security and privacy. Accuracy is central because treat-to-target care for rheumatoid arthritis (RA) depends on timely forecasts of disease-activity indices such as Disease Activity Score (DAS) 28- Erythrocyte sedimentation rate (ESR), DAS28- C-reactive protein (CRP), and Clinical Disease Activity Index (CDAI). Although proprietary models like GPT-4 already exceed the pass marks of medical board examinations 4 , their capacity to predict quantitative RA activity scores has not been rigorously tested. These scores guide clinicians when adjusting therapy, so evaluating how well LLMs can anticipate them is a prerequisite for clinical adoption. At the same time, privacy concerns limit the appeal of cloud-based systems. Models such as GPT-4 rely on opaque processing pipelines that can expose protected health information to third parties 5 , 6 . Open-source LLMs—for example Llama 2 7 ,—can run entirely on-premise and thus avoid external data transfer, but they typically trail proprietary models in medical accuracy. To close this gap, investigators have begun domain-specific pre-training and parameter-efficient fine-tuning of open-source models. Meditron, a 70-billion-parameter variant of Llama 2 further trained on medical literature and practice guidelines, raised its MedQA accuracy from 63.8 to 70.2 percent 8 . In addition, low-rank adaptation methods such as Quantized Low-Rank Adapter (QLoRA) enable task-specific fine-tuning that sometimes outperforms medical experts in text summarisation 1 . Despite these advances, there is still no comprehensive evaluation of pre-trained and fine-tuned LLMs for structured RA cohort prediction, nor any head-to-head comparison with strong tabular baselines such as logistic regression, random forest, or XGBoost 9 – 11 , The reliability and calibration of LLM outputs in rheumatology therefore remain uncertain. Accordingly, our study set out (i) to quantify the incremental benefit of medical-domain pre-training and QLoRA fine-tuning for Llama 2 when predicting two-year RA disease activity and disability, and (ii) to benchmark each learning stage against logistic regression, random forest, and XGBoost using data from 11,865 patients in the IORRA cohort (Fig. 1 ). By addressing both accuracy and privacy considerations, we aim to clarify whether locally deployable LLMs can deliver well-calibrated, clinically useful predictions for RA management. RESULTS Patients’ characteristics The IORRA cohort included 11,865 patients with RA, predominantly female (82.9%), with a mean age of 63.0 years and an average disease duration of 15.4 years. A significant proportion (33.6%) of the patients had a history of smoking (Supplemental Table 1). The average DAS28-ESR, DAS28-CRP is 2.3, and CDAI scores were 2.9, 2.3, and 6.0, respectively, all of which met the criteria for low disease activity. Methotrexate was the most used conventional synthetic disease-modifying antirheumatic drug (csDMARD) (64.1%), and 40.8% of the patients were on prednisolone. The cohort consisted of older patients with long-standing RA, many of whom were receiving csDMARDs or biologic DMARDs. Impact of Pre-training and Fine-Tuning on Model Performance First, to determine whether Meditron, the model in which Llama2 was pre-trained with medical knowledge, had acquired accurate knowledge of disease activity in RA, both models were asked the following question about the DAS28-ESR: “ How is the DAS28-ESR calculated in patients with rheumatoid arthritis? First, what variables are needed to calculate the DAS28-ESR? Second, please provide and explain the calculation formula. Finally, please give us your interpretation of the DAS28-ESR. ” The answer of Llama2 was incorrect, with the formula “DAS28-ESR = 0.56 × √(SJC + TJC) + 0.28 × PGA + 0.11 × ESR,” while Meditron provided the correct formula, “DAS28-ESR = 0.56 × √(tender28) + 0.28 × √(swollen28) + 0 .70 × ln(esr) + 0.014 × global." Furthermore, Meditron accurately defined the criteria for remission and high disease activity, whereas the explanation provided by Llama2 was inaccurate (Supplemental Table 2). The effect of pre-training on predicting the categories of disease activity or J-HAQ score was assessed by comparing the AUC of Meditron and Llama2 without fine-tuning using several indices. Meditrons without fine-tuning significantly outperformed Llama2 without fine-tuning for DAS28-ESR > 5.1, DAS28-ESR < 2.6, DAS28-CRP 2.5, and J-HAQ score < 0.5 (Meditron -fine-tuning (FT) vs Llama2 -FT in Fig. 2 , P < 0.05; Supplemental Table 3). Similarly, the effects of fine-tuning were assessed separately in the Llama2 and Meditron models. Fine-tuning of Llama2 resulted in a significant improvement in the AUC values across all indices (Llama2 -FT vs Llama2 + FT in Fig. 2 , P 22 ( P = 0.35). For Meditron, fine-tuning led to significant improvements in AUC (Meditron -FT vs Meditron + FT in Fig. 2 , P 2.5, and J-HAQ score < 0.5 ( P = 0.13 and P = 0.091, respectively). Calibration plots visualise the impact of medical pre-training and task-specific fine-tuning on probability reliability—effects that are not captured by discrimination metrics such as AUC alone. Calibration behaviour therefore differed by endpoint (Supplemental Fig. 1). DAS28-ESR remission represents a setting where medical pre-training alone markedly improved reliability: the Meditron base model already traced the identity line, whereas subsequent fine-tuning produced only a minor additional gain (Supplemental Fig. 1A). In contrast, J-HAQ remission required task-specific fine-tuning; the Meditron base model under-estimated mid-range risk (around 0.7), and the Llama 2 base produced a degenerate probability distribution that precluded a stable calibration curve. QLoRA-tuned variants, however, converged on near-perfect calibration (Supplemental Fig. 1B). Finally, CDAI remission illustrated that both steps are necessary: neither pre-training nor fine-tuning alone removed the systematic bias (Supplemental Fig. 1C). These three archetypes underscore that the optimal calibration strategy depends on the endpoint—pre-training suffices for some tasks, whereas others demand fine-tuning or the full two-step workflow. Comparison of performance using AUC and Brier score Across the eight prediction tasks, four different algorithms each emerged as the top-performer in exactly two endpoints: fine-tuned Llama 2, fine-tuned Meditron, XGBoost, and logistic regression (Fig. 2 A). Fine-tuned Llama 2 yielded the highest AUC for DAS28-CRP high disease activity (0.865, 95% CI 0.834–0.894) and for J-HAQ high disability (0.969, 95% CI 0.951–0.983). Fine-tuned Meditron led for DAS28-CRP remission (0.817, 95% CI 0.798–0.833) and DAS28-ESR high disease activity (0.867, 95% CI 0.832–0.899). XGBoost performed best in CDAI remission (0.821, 95% CI 0.804–0.838) and J-HAQ remission (0.935, 95% CI 0.924–0.944). Logistic regression achieved the top AUC for CDAI high disease activity (0.878, 95% CI 0.829–0.917) and DAS28-ESR remission (0.861, 95% CI 0.846–0.877). We then compared model performance using the Brier score (Fig. 2 B), which measures the mean squared difference between predicted probabilities and actual binary outcomes—lower values indicate better overall probability accuracy, encompassing both calibration and discrimination. Across the eight endpoints, four algorithms emerged as top-performers, with fine-tuned Llama 2 leading in three tasks, XGBoost and fine-tuned Meditron each in two, and logistic regression in one. Fine-tuned Llama 2 achieved the lowest Brier scores for DAS28-CRP high activity (0.051; 95% CI 0.044–0.059), DAS28-CRP remission (0.165; 95% CI 0.157–0.174) and DAS28-ESR remission (0.151; 95% CI 0.143–0.160). XGBoost performed best for CDAI high disease activity (0.022; 95% CI 0.017–0.028) and CDAI remission (0.171; 95% CI 0.162–0.178). Fine-tuned Meditron led on DAS28-ESR high activity (0.043; 95% CI 0.037–0.051) and J-HAQ high disability (0.017; 95% CI 0.013–0.021). Logistic regression was top for J-HAQ remission (0.098; 95% CI 0.091–0.107). Rank-based Comparison of Detailed Performance Metrics Next, we evaluated the models with five complementary metrics beyond AUC and the Brier score—sensitivity (Sn), specificity (Sp), positive-predictive value (PPV), negative-predictive value (NPV) and the F1-score—and visualised the within-indicator rankings with rank-based heat-maps (Fig. 3 A– 3 E). Across the eight clinical end-points the two fine-tuned LLMs secured first place (including tied ranks) in seven tasks for sensitivity (Fig. 3 A) and eight for NPV (Fig. 3 D), whereas the three conventional algorithms obtained five first ranks for specificity (Fig. 3 B). PPV (Fig. 3 C) was evenly split, with each model family topping four end-points, while the F1-score (Fig. 3 E) showed a task-dependent pattern: the LLMs ranked first for all four DAS28-derived outcomes, and logistic regression or XGBoost led in both CDAI and both J-HAQ outcomes (four versus four first ranks). These complementary strengths indicate that the optimal modelling approach depends not only on the clinical index but also on the evaluation metric applied. Clinical utility assessed with decision-curve analysis Decision-curve analysis (DCA) was performed to test whether adapting therapy on the basis of each model’s predicted probabilities confers a decision-analytic net benefit—the gain from correctly identifying patients who should receive the endpoint-specific intervention minus the harm from unnecessary intervention—across a plausible range of decision thresholds (pt). In this framework the grey dashed “intervene-all” line represents a policy that applies the relevant action to every patient (e.g., escalation when the target is high disease activity or tapering when the target is remission), whereas the black dashed “intervene-none” line withholds that action from all patients regardless of model output. For the remission endpoints, which have comparatively high event rates, the net-benefit curves of the fine-tuned Llama 2 and of the best conventional baseline were virtually superimposed (Fig. 4 A; Supplementary Fig. 2A–C), showing no material difference in decision-level utility across customary threshold regions. The picture diverged for high-disease-activity (HDA) tasks based on DAS28. In DAS28-CRP HDA (Fig. 4 B) and DAS28-ESR HDA (Supplementary Fig. 2D) the LLM provided a modest yet consistent advantage from pt ≈ 0 to 0.3–0.6, after which both models’ curves converged toward zero. For CDAI HDA the pattern was reversed: the conventional model initially overlapped the intervene-all line (pt < 0.1) and then exceeded the LLM between pt = 0.1 and 0.4 (Fig. 4 C); the absolute difference remained small (< 0.01) and vanished at higher thresholds. In J-HAQ HDA (Fig. 4 D) the conventional model’s curve lay below the intervene-none line throughout and became negative beyond pt ≈ 0.2, whereas the LLM stayed at—or marginally above—zero, suggesting that the conventional strategy could be harmful in this setting while the LLM would, at worst, be clinically neutral. In summary, net benefit did not differ between models for the high-prevalence remission outcomes, whereas for low-prevalence indices the LLMs generally maintained small but positive net benefit—although the magnitude and direction of the advantage remained endpoint-specific. DISCUSSION This work provides, to the best of our knowledge, the first head-to-head evaluation of open-source LLMs that can be run entirely on-premise versus three strong tabular baselines—logistic regression, random forest and XGBoost—across eight clinically relevant two-year outcomes in rheumatoid arthritis. Pre-training on medical corpora (Meditron) and parameter-efficient fine-tuning (QLoRA) both improved discrimination, calibration, and decision-analytic net benefit, yet their contributions were endpoint-specific. No single algorithm dominated: fine-tuned Llama 2 excelled in three tasks, fine-tuned Meditron, XGBoost and logistic regression each led in two by AUC, and decision-curve analysis showed clinically meaningful advantages that shifted with the underlying index. Even without fine-tuning, pre-training alone significantly improved the performance of Meditron when compared to that of Llama2. This improvement can be attributed to Meditron’s pre-training in the medical literature and guidelines, which enabled the model to acquire domain-specific knowledge about RA and related conditions. For instance, the model could explain complex clinical relationships, such as how IL-6 inhibitors can cause CRP levels to normalize or how comorbidities can lead to higher HAQ scores. Using this foundational knowledge, Meditron can make more accurate predictions without the need for additional fine-tuning. This nuanced understanding makes it challenging for traditional linear models to capture and highlights the unique capabilities of LLMs to comprehend and integrate broader clinical scenarios. Interestingly, despite the expected advantages of pre-training followed by fine-tuning, Meditron’s fine-tuned model showed only comparable performance to Llama2’s fine-tuned model. One explanation may be catastrophic forgetting, a phenomenon in which a model loses previously acquired knowledge during fine-tuning 12 . To explore this, we asked the fine-tuned Meditron model the same question regarding the DAS28 ESR calculation that was posed during the pre-training evaluation, and it still provided a correct and accurate response (see Supplemental Table 5). This suggests that catastrophic forgetting is not a primary issue. Instead, it is possible that Meditron’s pre-trained knowledge provided diminishing returns when applied to the structured cohort data used in this study. Previous research has shown that LLMs excel at interpreting unstructured data such as physician notes in electronic health records 2 , and their advantages may be less pronounced when working with highly structured clinical datasets, as is the case here. This may explain why the difference in performance between Meditron and Llama2 was modest after fine-tuning. This study had some limitations. First, the dataset used for training and testing was from a single-center cohort, which may have limited the generalizability of the findings to other populations. Additionally, although we used a comprehensive set of disease activity and physical function indices, the models were not tested on more complex, unstructured clinical data, such as physicians' notes or imaging data, where LLMs may demonstrate superior performance. In conclusion, medical-domain pre-training and QLoRA fine-tuning enable locally deployable LLMs to deliver calibrated, privacy-preserving predictions for rheumatoid-arthritis management that are at least competitive with, and sometimes superior to, state-of-the-art tabular models. This study lays the groundwork for future applications of LLMs in the secure and effective management of patients with RA. METHODS Study population Clinical data were obtained from the Institute of Rheumatology, Rheumatoid Arthritis (IORRA) cohort, which was initiated in 2000. The IORRA cohort was a large single-center observational study of Japanese patients with RA, comprising approximately 0.5% of the total Japanese population with RA 13 . Between 2000 and 2022, 15,004 patients were enrolled in this study. For this study, we selected cases in which at least one disease activity index, including DAS 28-ESR, DAS28-CRP, CDAI, or Japanese Health Assessment Questionnaire (J-HAQ) score, was followed for at least two years (n = 11,865) (Fig. 1 ). To prevent data leakage during model training, we first selected 102 cases for the validation set and 2,263 cases for the test set with the remaining cases used for training (n = 9,500). Data conversion Data extracted from the IORRA cohort were converted into text-based problem statements using in-house scripts. For example, if a patient with ID1 has the following characteristics: Age 72, Sex 1, Methotrexate 1, Prednisone 0, and Hypertension 1, the corresponding text would be: "A 72-year-old female rheumatoid arthritis patient treated with methotrexate ... The patient has a history of hypertension. What is the probability that this patient's CDAI will be remission at 2 years?" These text-based questions, paired with their respective answers, were stored in JSON format. Model Selection In this study, we utilized two LLMs: Llama2 (70B) and Meditron (70B). Both models are autoregressive and open source, which allows for local deployment, ensuring data privacy and security during model development. Llama2, developed by Meta, is a widely used open-source model known for its instruction-following capabilities and extended context length 7 . We selected the 70B version of Llama2 to benefit from its larger parameter size, as it provides improved performance on complex tasks. The Meditron is based on Llama2 and further pre-trained on a large corpus of medical literature, including PubMed articles, abstracts, and internationally recognized clinical practice guidelines 8 . This additional medical domain-specific pre-training allows Meditron to outperform standard LLMs, such as GPT-3.5, in medical reasoning tasks, making it a suitable candidate for predicting disease activity in patients with RA. Fine-tuning We employed the QLoRA fine-tuning technique to fine-tune the Llama2 and Meditron (70 B) models to predict disease activity in patients with RA. QLoRA is an efficient low-rank adaptation method that allows fine-tuning with minimal changes to the original model weights by inserting trainable matrices into the attention layers 14 . This method was chosen to minimize computational resources while maintaining the integrity of the base model. Fine-tuning was conducted with a learning rate of 3e-5, batch size of 2, and gradient accumulation steps of 8. The LoRA parameters included a rank of 8 and an alpha of 16. Based on the dataset and batch size, the number of epochs was approximately one. Decision-curve analysis For each of the eight binary outcomes, we calculated decision curves 15 on the independent test set to evaluate the clinical utility of every model. Net benefit was defined as $$\:Net\:Benenefit\left(pt\right)=\:\frac{True\:positive}{Number\:of\:sample}-\frac{False\:positive}{Number\:of\:sample}\frac{pt}{1-pt}$$ where pt is the threshold probability at which the predicted positive class would trigger the endpoint-specific intervention (treatment escalation for high-disease-activity outcomes and treatment tapering or maintenance for remission outcomes). We swept pt from 0.01 to 0.99 in 0.01 increments, plotting each model’s net-benefit curve alongside two reference strategies: intervene-all (apply the intervention to every patient) and intervene-none (withhold the intervention from all patients). Statistical analysis DAS28-ESR, DAS28-CRP, and CDAI scores were categorized as high disease activity or remission, and the J-HAQ score was categorized as high score or remission two years after the baseline assessment (Supplemental Table 6). The prevalence of each binary outcome in the held-out test set was as follows: DAS28-ESR remission 45.2%, DAS28-ESR high disease activity (HDA) 5.6%, DAS28-CRP remission 61.1%, DAS28-CRP HDA 6.9%, CDAI remission 41.4%, CDAI HDA 2.5%, J-HAQ remission 49.1% and J-HAQ high disability 4.5%. The corresponding continuous score distributions were: DAS28-ESR median 2.73 (IQR 1.99–3.60), DAS28-CRP median 1.98 (IQR 1.42–2.85), CDAI median 4.00 (IQR 1.10–8.40), and J-HAQ median 0.50 (IQR 0.00–1.25). Consequently, the datasets for high disease activity (HDA) and high disability outcomes were substantially imbalanced. Explanatory variables used in the model included demographics, disease activity, laboratory data, medication use, and medical history (details are provided in Supplemental Table 1). After excluding patients who lacked the 2-year outcome, all predictor variables with missing entries were imputed by the missForest algorithm. The imputation model was fitted on the training set only and subsequently applied to the validation and test sets to avoid information leakage. To benchmark LLM performance against conventional tabular methods, we fitted logistic regression, random forest and XGBoost classifiers. Hyper-parameters for all three models were optimised by exhaustive GridSearch CV and are listed in Supplemental Table 7. We primarily used the area under the curve (AUC) from the receiver operating characteristic (ROC) analysis to assess model performance. We applied the DeLong test to assess differences between pre-trained and fine-tuned LLMs. To complement AUC, probabilistic accuracy was evaluated with the Brier score 16 , 17 , scoring rule that combines calibration with predictive resolution, and calibration was further illustrated with reliability plots. To obtain 95% confidence intervals (CIs) for key metrics and for the calibration curves, we used non-parametric prediction bootstrapping (1000 resamples): for each bootstrap draw we resampled, with replacement, the prediction–label pairs of the held-out test set, recalculated ROC-AUC and Brier score, and re-estimated the fraction-of-positives for each calibration-plot bin. Statistical significance was set at P < 0.05. Other metrics, including sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1 score (the harmonic mean of precision and recall, providing a balance between the two) were calculated using Python's sklearn library. The development environment was Python 3.8.16, running on two parallel NVIDIA RTX 3090 24 GB GPUs and a 6000 Ada 48GB GPU to fine-tune the LLM. Declarations Competing Interests Competing interestsS.H. and M.F. declare no conflicts of interest. K.I. received speaker fees from Asahi Kasei Pharma Co., Astellas Pharma Inc., AbbVie Japan GK, Ayumi Pharmaceutical Corporation, Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Eisai Co., Ltd., Eli Lilly Japan K.K., Janssen Pharmaceutical K.K., Kaken Pharmaceutical Co. Ltd., Mitsubishi Tanabe Pharma Co., Pfizer Japan Inc., Takeda Pharmaceutical Co. Ltd., Teijin Pharma Ltd, and UCB Japan Co. Ltd. Division of Multidisciplinary Management of Rheumatic Diseases is an endowment department, supported by an unrestricted grant from Ayumi Pharmaceutical Corp., Chugai Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., and Nippon Kayaku Co., Ltd. Teijin Pharma Ltd. ET has received lecture fees or consulting fees from AbbVie Japan GK, Asahi Kasei Corp., Astellas Pharma Inc., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Daiichi-Sankyo, Inc., Eisai Co., Ltd., Eli Lilly Japan K.K., Gilead Sciences, Inc., Pfizer Japan Inc, Nichi-Iko Pharmaceutical Co., Ltd., Taisho Pharmaceutical Co., Ltd, Takeda Pharmaceutical Co., Ltd, Mitsubishi Tanabe Pharma Co., UCB Japan Co. Ltd. and Viatris Inc. ET received research funding from Pfizer Inc. and UCB Japan Co., Ltd. MH has received research grants from AbbVie Japan GK, Asahi Kasei Corp., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Eisai Co., Ltd., Eli Lilly Japan K.K., Kaken Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., Nippon Kayaku Co., Ltd., Nippon Shinyaku Co., Ltd., Pfizer Japan Inc., Taisho Pharmaceutical Co., Ltd., Teijin Pharma Ltd., UCB Japan Co., Ltd., and Viatris Japan. MH has received speaker’s fee from AbbVie Japan GK, Asahi Kasei Corp., Astra Zeneca K. K., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Eisai Co., Ltd., Eli Lilly Japan K.K., GlaxoSmithKline K.K., Gilead Sciences Inc., Janssen Pharmaceutical K.K., Kissei Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., Ono Pharmaceutical Co., Ltd., Pfizer Japan Inc., Taisho Pharmaceutical Co., Ltd., and Teijin Pharma Ltd. MH is a consultant for AbbVie, Boehringer Ingelheim, Bristol Myers Squibb Co., Kissei Pharmaceutical Co., Ltd., and Teijin Pharma. Ethics approval The IORRA cohort study (#2952-R and #2922-R16) was approved by the ethics committee of Tokyo Women’s Medical University, and informed consent was obtained from all patients before each survey. Patient and public involvement The patients and/or the public were not involved in the design, conduct, reporting, or dissemination of this study. Patient consent for publication Not required. Data sharing statement Data using analysis that supporting the findings of this study are available upon reasonable request from the authors. Funding This work was supported by JSPS KAKENHI (Grant Number JP23K24465) and a research grant from the Program for the Promotion of Precision Medicine in Rheumatoid Arthritis, Japan College of Rheumatology, 2023. Author Contribution M.H. and S.H. conceived the study design. S.H. conducted all analyses with the help of K.I., and S.H. drafted the manuscript. K.I., M.F., S.H., E.T., and M.H. collected samples and clinical information. All authors critically reviewed and approved the manuscript. Acknowledgement AcknowledgmentWe thank all patients in the IORRA database and all members of the Institute of Rheumatology, Tokyo Women’s Medical University Hospital, for the successful management of the IORRA study cohort. The IORRA cohort was supported by unrestricted grants from six pharmaceutical companies: AbbVie Japan GK, Asahi Kasei Pharma Co., Ayumi Pharmaceutical Co., Taisho Pharma Co., Ltd., Teijin Pharma Ltd., and Bristol-Myers Squibb Company. We used ChatGPT-4 minimally for grammar checking during the initial stages of preparing this manuscript. Following this, we utilized Editage's services for professional English language editing. We thank Editage [http://www.editage.com] for editing and reviewing this manuscript for English language. References Van Veen D, Van Uden C, Blankemeier L, et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat Med 2024; 30 : 1134–42. Jiang LY, Liu XC, Nejatian NP, et al. Health system-scale language models are all-purpose prediction engines. Nature 2023; 619 : 357–62. Tu T, Azizi S, Driess D, et al. Towards Generalist Biomedical AI. NEJM AI 2024; 1 . DOI:10.1056/AIoa2300138. Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of GPT-4 on Medical Challenge Problems. 2023. DOI:10.48550/ARXIV.2303.13375. Venerito V, Bilgin E, Iannone F, Kiraz S. AI am a rheumatologist: a practical primer to large language models for rheumatologists. Rheumatology (Oxford) 2023; 62 : 3256–60. Benavent D, Madrid-García A. Large language models and rheumatology: are we there yet? Rheumatology Advances in Practice 2024; : rkae119. Touvron H, Martin L, Stone K, et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023. DOI:10.48550/ARXIV.2307.09288. Chen Z, Cano AH, Romanou A, et al. MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. 2023. DOI:10.48550/ARXIV.2311.16079. Ye C, Zweck E, Ma Z, Smith J, Katz S. Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross-Sectional Study. Arthritis Rheumatol 2024; 76 : 479–84. Rusinovich Lovgach O, Calvo-Aranda E, Ramos-Lisbona AI, et al. POS0444 ARTIFICIAL INTELLIGENCE VERSUS RHEUMATOLOGIST IN DECISION MAKING IN THE TREATMENT OF RHEUMATOID ARTHRITIS. DO WE THINK ALIKE? In: Scientific Abstracts. BMJ Publishing Group Ltd and European League Against Rheumatism, 2024: 467.1-467. Coskun BN, Yagiz B, Ocakoglu G, Dalkilic E, Pehlivan Y. Assessing the accuracy and completeness of artificial intelligence language models in providing information on methotrexate use. Rheumatol Int 2024; 44 : 509–15. Luo Y, Yang Z, Meng F, Li Y, Zhou J, Zhang Y. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning. 2023. DOI:10.48550/ARXIV.2308.08747. Yamanaka H, Tanaka E, Nakajima A, et al. A large observational cohort study of rheumatoid arthritis, IORRA: Providing context for today’s treatment options. Mod Rheumatol 2020; 30 : 1–6. Dettmers T, Pagnoni A, Holtzman A, Zettlemoyer L. QLoRA: Efficient Finetuning of Quantized LLMs. 2023. DOI:10.48550/ARXIV.2305.14314. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 2006; 26 : 565–74. Rufibach K. Use of Brier score to assess binary predictions. J Clin Epidemiol 2010; 63 : 938–9; author reply 939. Brier GW. VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY. Mon Wea Rev 1950; 78 : 1–3. Additional Declarations Competing interest reported. Competing interests S.H. and M.F. declare no conflicts of interest. K.I. received speaker fees from Asahi Kasei Pharma Co., Astellas Pharma Inc., AbbVie Japan GK, Ayumi Pharmaceutical Corporation, Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Eisai Co., Ltd., Eli Lilly Japan K.K., Janssen Pharmaceutical K.K., Kaken Pharmaceutical Co. Ltd., Mitsubishi Tanabe Pharma Co., Pfizer Japan Inc., Takeda Pharmaceutical Co. Ltd., Teijin Pharma Ltd, and UCB Japan Co. Ltd. Division of Multidisciplinary Management of Rheumatic Diseases is an endowment department, supported by an unrestricted grant from Ayumi Pharmaceutical Corp., Chugai Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., and Nippon Kayaku Co., Ltd. Teijin Pharma Ltd. ET has received lecture fees or consulting fees from AbbVie Japan GK, Asahi Kasei Corp., Astellas Pharma Inc., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Daiichi-Sankyo, Inc., Eisai Co., Ltd., Eli Lilly Japan K.K., Gilead Sciences, Inc., Pfizer Japan Inc, Nichi-Iko Pharmaceutical Co., Ltd., Taisho Pharmaceutical Co., Ltd, Takeda Pharmaceutical Co., Ltd, Mitsubishi Tanabe Pharma Co., UCB Japan Co. Ltd. and Viatris Inc. ET received research funding from Pfizer Inc. and UCB Japan Co., Ltd. MH has received research grants from AbbVie Japan GK, Asahi Kasei Corp., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Eisai Co., Ltd., Eli Lilly Japan K.K., Kaken Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., Nippon Kayaku Co., Ltd., Nippon Shinyaku Co., Ltd., Pfizer Japan Inc., Taisho Pharmaceutical Co., Ltd., Teijin Pharma Ltd., UCB Japan Co., Ltd., and Viatris Japan. MH has received speaker’s fee from AbbVie Japan GK, Asahi Kasei Corp., Astra Zeneca K. K., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Eisai Co., Ltd., Eli Lilly Japan K.K., GlaxoSmithKline K.K., Gilead Sciences Inc., Janssen Pharmaceutical K.K., Kissei Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., Ono Pharmaceutical Co., Ltd., Pfizer Japan Inc., Taisho Pharmaceutical Co., Ltd., and Teijin Pharma Ltd. MH is a consultant for AbbVie, Boehringer Ingelheim, Bristol Myers Squibb Co., Kissei Pharmaceutical Co., Ltd., and Teijin Pharma. Supplementary Files LLMSupplementaryver3.docx SupplementalFigurewithlegendsver2.pptx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7159212","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":496630140,"identity":"3dcb0d47-7b66-44a5-a0ab-f7ec36a98d1d","order_by":0,"name":"Suguru Honda","email":"","orcid":"","institution":"Tokyo Women’s Medical University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Suguru","middleName":"","lastName":"Honda","suffix":""},{"id":496630144,"identity":"ffc4f6da-d4ca-440b-acaa-fc4666a46d14","order_by":1,"name":"Katsunori Ikari","email":"","orcid":"","institution":"Tokyo Women’s Medical University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Katsunori","middleName":"","lastName":"Ikari","suffix":""},{"id":496630146,"identity":"afebd834-5035-46a2-a1ee-20711aafe44a","order_by":2,"name":"Mayuko Fujisaki","email":"","orcid":"","institution":"Tokyo Women’s Medical University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Mayuko","middleName":"","lastName":"Fujisaki","suffix":""},{"id":496630147,"identity":"2e687c4e-4e90-4e6e-bb7e-6d3a69d8fb72","order_by":3,"name":"Eiichi Tanaka","email":"","orcid":"","institution":"Tokyo Women’s Medical University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Eiichi","middleName":"","lastName":"Tanaka","suffix":""},{"id":496630148,"identity":"eaad74bf-1e9e-428f-b1a7-6d77fb9341c3","order_by":4,"name":"Masayoshi Harigai","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAwklEQVRIiWNgGAWjYDACCSBOYGDg4WfmAXHZSNAi2UySFhAwOMBDpLt0Zzc/+/Dgl52M8XHeY5JfGPjyCGoxu3PMeEZiXzKP2WG+NGkZBrZiwlpuJBgzJPYwA7XwmN2WYGBLbCCsJf0zUEs9j3Ez8VpyjBkSfhzmMQBadPMDkVqKGRIbjvNIHOYx/81gQJRf0jcz/vhTbc/ff8bY8EfFMcIhBgaMbRCamcfgWAJxWhj+QLX+YKghVssoGAWjYBSMIAAAnnM5fuSeYFcAAAAASUVORK5CYII=","orcid":"","institution":"Tokyo Women’s Medical University School of Medicine","correspondingAuthor":true,"prefix":"","firstName":"Masayoshi","middleName":"","lastName":"Harigai","suffix":""}],"badges":[],"createdAt":"2025-07-18 15:53:13","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7159212/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7159212/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":88603086,"identity":"04576532-c1b8-4274-984c-c02e8497384e","added_by":"auto","created_at":"2025-08-08 08:14:56","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":342395,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eOverview of study design and workflow.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe IORRA cohort (n = 15,004) was used, excluding patients with less than two-year follow-up (n = 3,139). The data were divided into training (n = 9,500), validation (n = 102), and test (n = 2,263) sets. Llama2 and Meditron (pre-trained on medical literature) served as large language models (LLMs) to predict disease activity and physical function. Both LLMs were fine-tuned using the training data. Logistic regression, random forest and XGBoost was used for comparison. Predictive indicators included DAS28-ESR, DAS28-CRP, CDAI (high disease activity or remission), and J-HAQ score (high score or remission) after two years.\u003c/p\u003e\n\u003cp\u003eCDAI, Clinical Disease Activity Index; CRP, C-reactive protein; DAS, Disease Activity Score; ESR, Erythrocyte Sedimentation Rate; J-HAQ, Japanese Health Assessment Questionnaire.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7159212/v1/e8ac455a608eb2d5bd342cd3.png"},{"id":88603085,"identity":"c700c175-83f0-4be5-a747-6f2170900951","added_by":"auto","created_at":"2025-08-08 08:14:56","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":253143,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eComparative Performance of LLMs and Conventional ML Models for RA Disease Activity Classification\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e(A) ROC AUC: Each dot represents the mean area under the receiver operating characteristic curve (AUC) for each model and clinical outcome, with horizontal bars indicating 95% confidence intervals estimated by bootstrapping. (B) Brier Score: Each dot represents the mean Brier score for each model and clinical outcome, with horizontal bars indicating 95% confidence intervals estimated by bootstrapping. The Brier score quantifies the accuracy of probabilistic predictions, with lower values indicating better calibration and predictive accuracy.\u003c/p\u003e\n\u003cp\u003eFT, fine-tuning. Disease-activity abbreviations are defined in Figure 1.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7159212/v1/0aa16bf752df64a4956e860d.png"},{"id":88603091,"identity":"5a5dfdaf-cd81-4f21-aeea-c99a53e5e6ce","added_by":"auto","created_at":"2025-08-08 08:14:56","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":392645,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eRank-based heat-maps for five classification metrics.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eEach panel depicts one performance metric—panel A: sensitivity (Sn), B: specificity (Sp), C: positive-predictive value (PPV), D: negative-predictive value (NPV) or E: F1-score—across the eight clinical end-points evaluated in this study (DAS28-ESR, DAS28-CRP, CDAI and J-HAQ, each in high-activity and remission categories). Within every end-point the five models are ranked (1 = best, 5 = worst); the cell colour represents this rank, progressing from dark blue for rank 1 to white for rank 5, while the number printed inside the cell is the actual metric value rounded to two decimals. In cases where multiple models had identical metric values rounded to the four-decimal place, the same rank colour was assigned to all tied cells. The orange horizontal line separates the three conventional machine-learning approaches shown above it (logistic regression, random forest, and XGBoost) from the two large-language-model (LLM) approaches shown below it (Llama 2 +fine-tuning (FT) and Meditron +FT). The compact horizontal colour-bar at the lower right provides the common rank scale (ticks = 1–5) used by all panels.\u003c/p\u003e\n\u003cp\u003eDisease-activity abbreviations are defined in Figure 1.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-7159212/v1/b775e9ed60d6aa48987816d2.png"},{"id":88603093,"identity":"3e6b8313-bb1b-4c64-bc91-6d2451a0be20","added_by":"auto","created_at":"2025-08-08 08:14:56","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":314330,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eDecision-curve analysis for four representative rheumatoid arthritis outcomes\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFor each endpoint (panel A: DAS28-CRP remission, B: DAS28-CRP high disease activity, C: CDAI high disease activity, D: J-HAQ high disease activity) net benefit is plotted against the decision threshold probability pt. The solid blue curve shows the model that achieved the highest ROC-AUC among all LLM variants, whereas the solid orange curve represents the conventional baseline that obtained the highest ROC-AUC in the non-LLM group. Dashed grey and black lines depict the “treat-all’’ and “treat-none’’ strategies, respectively, providing clinical reference points for comparison.\u003c/p\u003e\n\u003cp\u003eDisease-activity abbreviations are defined in Figure 1.\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7159212/v1/74e76cbfdfea9b34a72993a5.png"},{"id":94189281,"identity":"f07d0f59-792d-43fd-9485-d0f8b4986e14","added_by":"auto","created_at":"2025-10-23 11:39:30","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2026722,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7159212/v1/34a947fb-b38d-467a-8a17-badc151c1cac.pdf"},{"id":88603084,"identity":"def74a6e-1f88-4a97-9acb-32ea4a462c9c","added_by":"auto","created_at":"2025-08-08 08:14:56","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":44063,"visible":true,"origin":"","legend":"","description":"","filename":"LLMSupplementaryver3.docx","url":"https://assets-eu.researchsquare.com/files/rs-7159212/v1/8dd2e96de74d4fd3e23ca7db.docx"},{"id":88603089,"identity":"dbac883d-7bf3-4120-9e1d-e42350efc433","added_by":"auto","created_at":"2025-08-08 08:14:56","extension":"pptx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":1831248,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementalFigurewithlegendsver2.pptx","url":"https://assets-eu.researchsquare.com/files/rs-7159212/v1/35b7177113bd8c2baf0f06e8.pptx"}],"financialInterests":"Competing interest reported. Competing interests\nS.H. and M.F. declare no conflicts of interest. K.I. received speaker fees from Asahi Kasei Pharma Co., Astellas Pharma Inc., AbbVie Japan GK, Ayumi Pharmaceutical Corporation, Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Eisai Co., Ltd., Eli Lilly Japan K.K., Janssen Pharmaceutical K.K., Kaken Pharmaceutical Co. Ltd., Mitsubishi Tanabe Pharma Co., Pfizer Japan Inc., Takeda Pharmaceutical Co. Ltd., Teijin Pharma Ltd, and UCB Japan Co. Ltd. Division of Multidisciplinary Management of Rheumatic Diseases is an endowment department, supported by an unrestricted grant from Ayumi Pharmaceutical Corp., Chugai Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., and Nippon Kayaku Co., Ltd. Teijin Pharma Ltd. ET has received lecture fees or consulting fees from AbbVie Japan GK, Asahi Kasei Corp., Astellas Pharma Inc., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Daiichi-Sankyo, Inc., Eisai Co., Ltd., Eli Lilly Japan K.K., Gilead Sciences, Inc., Pfizer Japan Inc, Nichi-Iko Pharmaceutical Co., Ltd., Taisho Pharmaceutical Co., Ltd, Takeda Pharmaceutical Co., Ltd, Mitsubishi Tanabe Pharma Co., UCB Japan Co. Ltd. and Viatris Inc. ET received research funding from Pfizer Inc. and UCB Japan Co., Ltd. MH has received research grants from AbbVie Japan GK, Asahi Kasei Corp., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Eisai Co., Ltd., Eli Lilly Japan K.K., Kaken Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., Nippon Kayaku Co., Ltd., Nippon Shinyaku Co., Ltd., Pfizer Japan Inc., Taisho Pharmaceutical Co., Ltd., Teijin Pharma Ltd., UCB Japan Co., Ltd., and Viatris Japan. MH has received speaker’s fee from AbbVie Japan GK, Asahi Kasei Corp., Astra Zeneca K. K., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Eisai Co., Ltd., Eli Lilly Japan K.K., GlaxoSmithKline K.K., Gilead Sciences Inc., Janssen Pharmaceutical K.K., Kissei Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., Ono Pharmaceutical Co., Ltd., Pfizer Japan Inc., Taisho Pharmaceutical Co., Ltd., and Teijin Pharma Ltd. MH is a consultant for AbbVie, Boehringer Ingelheim, Bristol Myers Squibb Co., Kissei Pharmaceutical Co., Ltd., and Teijin Pharma.","formattedTitle":"Medical pre-training and fine-tuning improve large-language-model prediction of rheumatoid-arthritis disease activity","fulltext":[{"header":"INTRODUCTION","content":"\u003cp\u003eIn the past few years, advancements in Large Language Models (LLMs) have shown substantial potential for improving the management and diagnosis of diseases, extracting clinical information from Electronic Medical Records (EMRs), and assisting in creating pathology and imaging reports \u003csup\u003e\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e. Their emerging role as clinical decision-support tools has prompted active exploration of how they might be integrated into EMR systems to advance medical practice, research, and education. Yet several critical obstacles remain\u0026mdash;most notably predictive accuracy on task-specific outcomes and practical issues of data security and privacy.\u003c/p\u003e\u003cp\u003eAccuracy is central because treat-to-target care for rheumatoid arthritis (RA) depends on timely forecasts of disease-activity indices such as Disease Activity Score (DAS) 28- Erythrocyte sedimentation rate (ESR), DAS28- C-reactive protein (CRP), and Clinical Disease Activity Index (CDAI). Although proprietary models like GPT-4 already exceed the pass marks of medical board examinations \u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e, their capacity to predict quantitative RA activity scores has not been rigorously tested. These scores guide clinicians when adjusting therapy, so evaluating how well LLMs can anticipate them is a prerequisite for clinical adoption.\u003c/p\u003e\u003cp\u003eAt the same time, privacy concerns limit the appeal of cloud-based systems. Models such as GPT-4 rely on opaque processing pipelines that can expose protected health information to third parties \u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e,\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e. Open-source LLMs\u0026mdash;for example Llama 2 \u003csup\u003e7\u003c/sup\u003e,\u0026mdash;can run entirely on-premise and thus avoid external data transfer, but they typically trail proprietary models in medical accuracy.\u003c/p\u003e\u003cp\u003eTo close this gap, investigators have begun domain-specific pre-training and parameter-efficient fine-tuning of open-source models. Meditron, a 70-billion-parameter variant of Llama 2 further trained on medical literature and practice guidelines, raised its MedQA accuracy from 63.8 to 70.2 percent \u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e. In addition, low-rank adaptation methods such as Quantized Low-Rank Adapter (QLoRA) enable task-specific fine-tuning that sometimes outperforms medical experts in text summarisation \u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003eDespite these advances, there is still no comprehensive evaluation of pre-trained and fine-tuned LLMs for structured RA cohort prediction, nor any head-to-head comparison with strong tabular baselines such as logistic regression, random forest, or XGBoost \u003csup\u003e\u003cspan additionalcitationids=\"CR10\" citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e, The reliability and calibration of LLM outputs in rheumatology therefore remain uncertain. Accordingly, our study set out (i) to quantify the incremental benefit of medical-domain pre-training and QLoRA fine-tuning for Llama 2 when predicting two-year RA disease activity and disability, and (ii) to benchmark each learning stage against logistic regression, random forest, and XGBoost using data from 11,865 patients in the IORRA cohort (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). By addressing both accuracy and privacy considerations, we aim to clarify whether locally deployable LLMs can deliver well-calibrated, clinically useful predictions for RA management.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e"},{"header":"RESULTS","content":"\u003cp\u003e\u003cb\u003ePatients\u0026rsquo; characteristics\u003c/b\u003e\u003c/p\u003e\u003cp\u003eThe IORRA cohort included 11,865 patients with RA, predominantly female (82.9%), with a mean age of 63.0 years and an average disease duration of 15.4 years. A significant proportion (33.6%) of the patients had a history of smoking (Supplemental Table\u0026nbsp;1). The average DAS28-ESR, DAS28-CRP is 2.3, and CDAI scores were 2.9, 2.3, and 6.0, respectively, all of which met the criteria for low disease activity. Methotrexate was the most used conventional synthetic disease-modifying antirheumatic drug (csDMARD) (64.1%), and 40.8% of the patients were on prednisolone. The cohort consisted of older patients with long-standing RA, many of whom were receiving csDMARDs or biologic DMARDs.\u003c/p\u003e\u003cp\u003e\u003cb\u003eImpact of Pre-training and Fine-Tuning on Model Performance\u003c/b\u003e\u003c/p\u003e\u003cp\u003eFirst, to determine whether Meditron, the model in which Llama2 was pre-trained with medical knowledge, had acquired accurate knowledge of disease activity in RA, both models were asked the following question about the DAS28-ESR: \u0026ldquo;\u003cem\u003eHow is the DAS28-ESR calculated in patients with rheumatoid arthritis? First, what variables are needed to calculate the DAS28-ESR? Second, please provide and explain the calculation formula. Finally, please give us your interpretation of the DAS28-ESR.\u003c/em\u003e\u0026rdquo; The answer of Llama2 was incorrect, with the formula \u0026ldquo;DAS28-ESR\u0026thinsp;=\u0026thinsp;0.56 \u0026times; \u0026radic;(SJC\u0026thinsp;+\u0026thinsp;TJC)\u0026thinsp;+\u0026thinsp;0.28 \u0026times; PGA\u0026thinsp;+\u0026thinsp;0.11 \u0026times; ESR,\u0026rdquo; while Meditron provided the correct formula, \u0026ldquo;DAS28-ESR\u0026thinsp;=\u0026thinsp;0.56 \u0026times; \u0026radic;(tender28)\u0026thinsp;+\u0026thinsp;0.28 \u0026times; \u0026radic;(swollen28)\u0026thinsp;+\u0026thinsp;0 .70 \u0026times; ln(esr)\u0026thinsp;+\u0026thinsp;0.014 \u0026times; global.\" Furthermore, Meditron accurately defined the criteria for remission and high disease activity, whereas the explanation provided by Llama2 was inaccurate (Supplemental Table\u0026nbsp;2).\u003c/p\u003e\u003cp\u003eThe effect of pre-training on predicting the categories of disease activity or J-HAQ score was assessed by comparing the AUC of Meditron and Llama2 without fine-tuning using several indices. Meditrons without fine-tuning significantly outperformed Llama2 without fine-tuning for DAS28-ESR\u0026thinsp;\u0026gt;\u0026thinsp;5.1, DAS28-ESR\u0026thinsp;\u0026lt;\u0026thinsp;2.6, DAS28-CRP\u0026thinsp;\u0026lt;\u0026thinsp;2.3, J-HAQ score\u0026thinsp;\u0026gt;\u0026thinsp;2.5, and J-HAQ score\u0026thinsp;\u0026lt;\u0026thinsp;0.5 (Meditron -fine-tuning (FT) vs Llama2 -FT in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, P\u0026thinsp;\u0026lt;\u0026thinsp;0.05; Supplemental Table\u0026nbsp;3).\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eSimilarly, the effects of fine-tuning were assessed separately in the Llama2 and Meditron models. Fine-tuning of Llama2 resulted in a significant improvement in the AUC values across all indices (Llama2 -FT vs Llama2\u0026thinsp;+\u0026thinsp;FT in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, P\u0026thinsp;\u0026lt;\u0026thinsp;0.05, Supplemental Table\u0026nbsp;4) except for the prediction of CDAI\u0026thinsp;\u0026gt;\u0026thinsp;22 (\u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.35). For Meditron, fine-tuning led to significant improvements in AUC (Meditron -FT vs Meditron\u0026thinsp;+\u0026thinsp;FT in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, P\u0026thinsp;\u0026lt;\u0026thinsp;0.05, Supplemental Table\u0026nbsp;4) except for J-HAQ score\u0026thinsp;\u0026gt;\u0026thinsp;2.5, and J-HAQ score\u0026thinsp;\u0026lt;\u0026thinsp;0.5 (\u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.13 and \u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.091, respectively).\u003c/p\u003e\u003cp\u003eCalibration plots visualise the impact of medical pre-training and task-specific fine-tuning on probability reliability\u0026mdash;effects that are not captured by discrimination metrics such as AUC alone. Calibration behaviour therefore differed by endpoint (Supplemental Fig.\u0026nbsp;1). DAS28-ESR remission represents a setting where medical pre-training alone markedly improved reliability: the Meditron base model already traced the identity line, whereas subsequent fine-tuning produced only a minor additional gain (Supplemental Fig.\u0026nbsp;1A). In contrast, J-HAQ remission required task-specific fine-tuning; the Meditron base model under-estimated mid-range risk (around 0.7), and the Llama 2 base produced a degenerate probability distribution that precluded a stable calibration curve. QLoRA-tuned variants, however, converged on near-perfect calibration (Supplemental Fig.\u0026nbsp;1B). Finally, CDAI remission illustrated that both steps are necessary: neither pre-training nor fine-tuning alone removed the systematic bias (Supplemental Fig.\u0026nbsp;1C). These three archetypes underscore that the optimal calibration strategy depends on the endpoint\u0026mdash;pre-training suffices for some tasks, whereas others demand fine-tuning or the full two-step workflow.\u003c/p\u003e\u003cp\u003e\u003cb\u003eComparison of performance using AUC and Brier score\u003c/b\u003e\u003c/p\u003e\u003cp\u003eAcross the eight prediction tasks, four different algorithms each emerged as the top-performer in exactly two endpoints: fine-tuned Llama 2, fine-tuned Meditron, XGBoost, and logistic regression (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eA). Fine-tuned Llama 2 yielded the highest AUC for DAS28-CRP high disease activity (0.865, 95% CI 0.834\u0026ndash;0.894) and for J-HAQ high disability (0.969, 95% CI 0.951\u0026ndash;0.983). Fine-tuned Meditron led for DAS28-CRP remission (0.817, 95% CI 0.798\u0026ndash;0.833) and DAS28-ESR high disease activity (0.867, 95% CI 0.832\u0026ndash;0.899). XGBoost performed best in CDAI remission (0.821, 95% CI 0.804\u0026ndash;0.838) and J-HAQ remission (0.935, 95% CI 0.924\u0026ndash;0.944). Logistic regression achieved the top AUC for CDAI high disease activity (0.878, 95% CI 0.829\u0026ndash;0.917) and DAS28-ESR remission (0.861, 95% CI 0.846\u0026ndash;0.877).\u003c/p\u003e\u003cp\u003eWe then compared model performance using the Brier score (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eB), which measures the mean squared difference between predicted probabilities and actual binary outcomes\u0026mdash;lower values indicate better overall probability accuracy, encompassing both calibration and discrimination. Across the eight endpoints, four algorithms emerged as top-performers, with fine-tuned Llama 2 leading in three tasks, XGBoost and fine-tuned Meditron each in two, and logistic regression in one. Fine-tuned Llama 2 achieved the lowest Brier scores for DAS28-CRP high activity (0.051; 95% CI 0.044\u0026ndash;0.059), DAS28-CRP remission (0.165; 95% CI 0.157\u0026ndash;0.174) and DAS28-ESR remission (0.151; 95% CI 0.143\u0026ndash;0.160). XGBoost performed best for CDAI high disease activity (0.022; 95% CI 0.017\u0026ndash;0.028) and CDAI remission (0.171; 95% CI 0.162\u0026ndash;0.178). Fine-tuned Meditron led on DAS28-ESR high activity (0.043; 95% CI 0.037\u0026ndash;0.051) and J-HAQ high disability (0.017; 95% CI 0.013\u0026ndash;0.021). Logistic regression was top for J-HAQ remission (0.098; 95% CI 0.091\u0026ndash;0.107).\u003c/p\u003e\u003cp\u003e\u003cb\u003eRank-based Comparison of Detailed Performance Metrics\u003c/b\u003e\u003c/p\u003e\u003cp\u003eNext, we evaluated the models with five complementary metrics beyond AUC and the Brier score\u0026mdash;sensitivity (Sn), specificity (Sp), positive-predictive value (PPV), negative-predictive value (NPV) and the F1-score\u0026mdash;and visualised the within-indicator rankings with rank-based heat-maps (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eA\u0026ndash;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eE). Across the eight clinical end-points the two fine-tuned LLMs secured first place (including tied ranks) in seven tasks for sensitivity (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eA) and eight for NPV (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eD), whereas the three conventional algorithms obtained five first ranks for specificity (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eB). PPV (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eC) was evenly split, with each model family topping four end-points, while the F1-score (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eE) showed a task-dependent pattern: the LLMs ranked first for all four DAS28-derived outcomes, and logistic regression or XGBoost led in both CDAI and both J-HAQ outcomes (four versus four first ranks). These complementary strengths indicate that the optimal modelling approach depends not only on the clinical index but also on the evaluation metric applied.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003cb\u003eClinical utility assessed with decision-curve analysis\u003c/b\u003e\u003c/p\u003e\u003cp\u003eDecision-curve analysis (DCA) was performed to test whether adapting therapy on the basis of each model\u0026rsquo;s predicted probabilities confers a decision-analytic net benefit\u0026mdash;the gain from correctly identifying patients who should receive the endpoint-specific intervention minus the harm from unnecessary intervention\u0026mdash;across a plausible range of decision thresholds (pt). In this framework the grey dashed \u0026ldquo;intervene-all\u0026rdquo; line represents a policy that applies the relevant action to every patient (e.g., escalation when the target is high disease activity or tapering when the target is remission), whereas the black dashed \u0026ldquo;intervene-none\u0026rdquo; line withholds that action from all patients regardless of model output.\u003c/p\u003e\u003cp\u003eFor the remission endpoints, which have comparatively high event rates, the net-benefit curves of the fine-tuned Llama 2 and of the best conventional baseline were virtually superimposed (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003eA; Supplementary Fig.\u0026nbsp;2A\u0026ndash;C), showing no material difference in decision-level utility across customary threshold regions. The picture diverged for high-disease-activity (HDA) tasks based on DAS28. In DAS28-CRP HDA (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003eB) and DAS28-ESR HDA (Supplementary Fig.\u0026nbsp;2D) the LLM provided a modest yet consistent advantage from pt\u0026thinsp;\u0026asymp;\u0026thinsp;0 to 0.3\u0026ndash;0.6, after which both models\u0026rsquo; curves converged toward zero. For CDAI HDA the pattern was reversed: the conventional model initially overlapped the intervene-all line (pt\u0026thinsp;\u0026lt;\u0026thinsp;0.1) and then exceeded the LLM between pt\u0026thinsp;=\u0026thinsp;0.1 and 0.4 (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003eC); the absolute difference remained small (\u0026lt;\u0026thinsp;0.01) and vanished at higher thresholds. In J-HAQ HDA (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003eD) the conventional model\u0026rsquo;s curve lay below the intervene-none line throughout and became negative beyond pt\u0026thinsp;\u0026asymp;\u0026thinsp;0.2, whereas the LLM stayed at\u0026mdash;or marginally above\u0026mdash;zero, suggesting that the conventional strategy could be harmful in this setting while the LLM would, at worst, be clinically neutral.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eIn summary, net benefit did not differ between models for the high-prevalence remission outcomes, whereas for low-prevalence indices the LLMs generally maintained small but positive net benefit\u0026mdash;although the magnitude and direction of the advantage remained endpoint-specific.\u003c/p\u003e"},{"header":"DISCUSSION","content":"\u003cp\u003eThis work provides, to the best of our knowledge, the first head-to-head evaluation of open-source LLMs that can be run entirely on-premise versus three strong tabular baselines—logistic regression, random forest and XGBoost—across eight clinically relevant two-year outcomes in rheumatoid arthritis. Pre-training on medical corpora (Meditron) and parameter-efficient fine-tuning (QLoRA) both improved discrimination, calibration, and decision-analytic net benefit, yet their contributions were endpoint-specific. No single algorithm dominated: fine-tuned Llama 2 excelled in three tasks, fine-tuned Meditron, XGBoost and logistic regression each led in two by AUC, and decision-curve analysis showed clinically meaningful advantages that shifted with the underlying index.\u003c/p\u003e\u003cp\u003eEven without fine-tuning, pre-training alone significantly improved the performance of Meditron when compared to that of Llama2. This improvement can be attributed to Meditron’s pre-training in the medical literature and guidelines, which enabled the model to acquire domain-specific knowledge about RA and related conditions. For instance, the model could explain complex clinical relationships, such as how IL-6 inhibitors can cause CRP levels to normalize or how comorbidities can lead to higher HAQ scores. Using this foundational knowledge, Meditron can make more accurate predictions without the need for additional fine-tuning. This nuanced understanding makes it challenging for traditional linear models to capture and highlights the unique capabilities of LLMs to comprehend and integrate broader clinical scenarios.\u003c/p\u003e\u003cp\u003eInterestingly, despite the expected advantages of pre-training followed by fine-tuning, Meditron’s fine-tuned model showed only comparable performance to Llama2’s fine-tuned model. One explanation may be catastrophic forgetting, a phenomenon in which a model loses previously acquired knowledge during fine-tuning \u003csup\u003e\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e. To explore this, we asked the fine-tuned Meditron model the same question regarding the DAS28 ESR calculation that was posed during the pre-training evaluation, and it still provided a correct and accurate response (see Supplemental Table\u0026nbsp;5). This suggests that catastrophic forgetting is not a primary issue. Instead, it is possible that Meditron’s pre-trained knowledge provided diminishing returns when applied to the structured cohort data used in this study. Previous research has shown that LLMs excel at interpreting unstructured data such as physician notes in electronic health records \u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e, and their advantages may be less pronounced when working with highly structured clinical datasets, as is the case here. This may explain why the difference in performance between Meditron and Llama2 was modest after fine-tuning.\u003c/p\u003e\u003cp\u003eThis study had some limitations. First, the dataset used for training and testing was from a single-center cohort, which may have limited the generalizability of the findings to other populations. Additionally, although we used a comprehensive set of disease activity and physical function indices, the models were not tested on more complex, unstructured clinical data, such as physicians' notes or imaging data, where LLMs may demonstrate superior performance.\u003c/p\u003e\u003cp\u003eIn conclusion, medical-domain pre-training and QLoRA fine-tuning enable locally deployable LLMs to deliver calibrated, privacy-preserving predictions for rheumatoid-arthritis management that are at least competitive with, and sometimes superior to, state-of-the-art tabular models. This study lays the groundwork for future applications of LLMs in the secure and effective management of patients with RA.\u003c/p\u003e"},{"header":"METHODS","content":"\u003cp\u003e\u003cb\u003eStudy population\u003c/b\u003e\u003c/p\u003e\u003cp\u003eClinical data were obtained from the Institute of Rheumatology, Rheumatoid Arthritis (IORRA) cohort, which was initiated in 2000. The IORRA cohort was a large single-center observational study of Japanese patients with RA, comprising approximately 0.5% of the total Japanese population with RA \u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. Between 2000 and 2022, 15,004 patients were enrolled in this study. For this study, we selected cases in which at least one disease activity index, including DAS 28-ESR, DAS28-CRP, CDAI, or Japanese Health Assessment Questionnaire (J-HAQ) score, was followed for at least two years (n = 11,865) (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). To prevent data leakage during model training, we first selected 102 cases for the validation set and 2,263 cases for the test set with the remaining cases used for training (n = 9,500).\u003c/p\u003e\u003cp\u003e\u003cb\u003eData conversion\u003c/b\u003e\u003c/p\u003e\u003cp\u003eData extracted from the IORRA cohort were converted into text-based problem statements using in-house scripts. For example, if a patient with ID1 has the following characteristics: Age 72, Sex 1, Methotrexate 1, Prednisone 0, and Hypertension 1, the corresponding text would be:\u003c/p\u003e\u003cp\u003e\u003cem\u003e\"A 72-year-old female rheumatoid arthritis patient treated with methotrexate ... The patient has a history of hypertension. What is the probability that this patient's CDAI will be remission at 2 years?\"\u003c/em\u003e\u003c/p\u003e\u003cp\u003eThese text-based questions, paired with their respective answers, were stored in JSON format.\u003c/p\u003e\u003cp\u003e\u003cb\u003eModel Selection\u003c/b\u003e\u003c/p\u003e\u003cp\u003eIn this study, we utilized two LLMs: Llama2 (70B) and Meditron (70B). Both models are autoregressive and open source, which allows for local deployment, ensuring data privacy and security during model development. Llama2, developed by Meta, is a widely used open-source model known for its instruction-following capabilities and extended context length \u003csup\u003e\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e. We selected the 70B version of Llama2 to benefit from its larger parameter size, as it provides improved performance on complex tasks.\u003c/p\u003e\u003cp\u003eThe Meditron is based on Llama2 and further pre-trained on a large corpus of medical literature, including PubMed articles, abstracts, and internationally recognized clinical practice guidelines \u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e. This additional medical domain-specific pre-training allows Meditron to outperform standard LLMs, such as GPT-3.5, in medical reasoning tasks, making it a suitable candidate for predicting disease activity in patients with RA.\u003c/p\u003e\u003cp\u003e\u003cb\u003eFine-tuning\u003c/b\u003e\u003c/p\u003e\u003cp\u003eWe employed the QLoRA fine-tuning technique to fine-tune the Llama2 and Meditron (70 B) models to predict disease activity in patients with RA. QLoRA is an efficient low-rank adaptation method that allows fine-tuning with minimal changes to the original model weights by inserting trainable matrices into the attention layers \u003csup\u003e\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e. This method was chosen to minimize computational resources while maintaining the integrity of the base model.\u003c/p\u003e\u003cp\u003eFine-tuning was conducted with a learning rate of 3e-5, batch size of 2, and gradient accumulation steps of 8. The LoRA parameters included a rank of 8 and an alpha of 16. Based on the dataset and batch size, the number of epochs was approximately one.\u003c/p\u003e\u003cp\u003e\u003cb\u003eDecision-curve analysis\u003c/b\u003e\u003c/p\u003e\u003cp\u003eFor each of the eight binary outcomes, we calculated decision curves \u003csup\u003e\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e on the independent test set to evaluate the clinical utility of every model. Net benefit was defined as\u003c/p\u003e\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e\n$$\\:Net\\:Benenefit\\left(pt\\right)=\\:\\frac{True\\:positive}{Number\\:of\\:sample}-\\frac{False\\:positive}{Number\\:of\\:sample}\\frac{pt}{1-pt}$$\u003c/div\u003e\u003c/div\u003e\u003cp\u003ewhere pt is the threshold probability at which the predicted positive class would trigger the endpoint-specific intervention (treatment escalation for high-disease-activity outcomes and treatment tapering or maintenance for remission outcomes). We swept pt from 0.01 to 0.99 in 0.01 increments, plotting each model’s net-benefit curve alongside two reference strategies: intervene-all (apply the intervention to every patient) and intervene-none (withhold the intervention from all patients).\u003c/p\u003e\u003ch2\u003eStatistical analysis\u003c/h2\u003e\u003cp\u003eDAS28-ESR, DAS28-CRP, and CDAI scores were categorized as high disease activity or remission, and the J-HAQ score was categorized as high score or remission two years after the baseline assessment (Supplemental Table\u0026nbsp;6). The prevalence of each binary outcome in the held-out test set was as follows: DAS28-ESR remission 45.2%, DAS28-ESR high disease activity (HDA) 5.6%, DAS28-CRP remission 61.1%, DAS28-CRP HDA 6.9%, CDAI remission 41.4%, CDAI HDA 2.5%, J-HAQ remission 49.1% and J-HAQ high disability 4.5%. The corresponding continuous score distributions were: DAS28-ESR median 2.73 (IQR 1.99–3.60), DAS28-CRP median 1.98 (IQR 1.42–2.85), CDAI median 4.00 (IQR 1.10–8.40), and J-HAQ median 0.50 (IQR 0.00–1.25). Consequently, the datasets for high disease activity (HDA) and high disability outcomes were substantially imbalanced. Explanatory variables used in the model included demographics, disease activity, laboratory data, medication use, and medical history (details are provided in Supplemental Table\u0026nbsp;1). After excluding patients who lacked the 2-year outcome, all predictor variables with missing entries were imputed by the missForest algorithm. The imputation model was fitted on the training set only and subsequently applied to the validation and test sets to avoid information leakage. To benchmark LLM performance against conventional tabular methods, we fitted logistic regression, random forest and XGBoost classifiers. Hyper-parameters for all three models were optimised by exhaustive GridSearch CV and are listed in Supplemental Table\u0026nbsp;7. We primarily used the area under the curve (AUC) from the receiver operating characteristic (ROC) analysis to assess model performance. We applied the DeLong test to assess differences between pre-trained and fine-tuned LLMs. To complement AUC, probabilistic accuracy was evaluated with the Brier score\u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e,\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e, scoring rule that combines calibration with predictive resolution, and calibration was further illustrated with reliability plots. To obtain 95% confidence intervals (CIs) for key metrics and for the calibration curves, we used non-parametric prediction bootstrapping (1000 resamples): for each bootstrap draw we resampled, with replacement, the prediction–label pairs of the held-out test set, recalculated ROC-AUC and Brier score, and re-estimated the fraction-of-positives for each calibration-plot bin. Statistical significance was set at \u003cem\u003eP\u003c/em\u003e \u0026lt; 0.05. Other metrics, including sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1 score (the harmonic mean of precision and recall, providing a balance between the two) were calculated using Python's sklearn library. The development environment was Python 3.8.16, running on two parallel NVIDIA RTX 3090 24 GB GPUs and a 6000 Ada 48GB GPU to fine-tune the LLM.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003ch2\u003eCompeting Interests\u003c/h2\u003e\u003cp\u003eCompeting interestsS.H. and M.F. declare no conflicts of interest. K.I. received speaker fees from Asahi Kasei Pharma Co., Astellas Pharma Inc., AbbVie Japan GK, Ayumi Pharmaceutical Corporation, Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Eisai Co., Ltd., Eli Lilly Japan K.K., Janssen Pharmaceutical K.K., Kaken Pharmaceutical Co. Ltd., Mitsubishi Tanabe Pharma Co., Pfizer Japan Inc., Takeda Pharmaceutical Co. Ltd., Teijin Pharma Ltd, and UCB Japan Co. Ltd. Division of Multidisciplinary Management of Rheumatic Diseases is an endowment department, supported by an unrestricted grant from Ayumi Pharmaceutical Corp., Chugai Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., and Nippon Kayaku Co., Ltd. Teijin Pharma Ltd. ET has received lecture fees or consulting fees from AbbVie Japan GK, Asahi Kasei Corp., Astellas Pharma Inc., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Daiichi-Sankyo, Inc., Eisai Co., Ltd., Eli Lilly Japan K.K., Gilead Sciences, Inc., Pfizer Japan Inc, Nichi-Iko Pharmaceutical Co., Ltd., Taisho Pharmaceutical Co., Ltd, Takeda Pharmaceutical Co., Ltd, Mitsubishi Tanabe Pharma Co., UCB Japan Co. Ltd. and Viatris Inc. ET received research funding from Pfizer Inc. and UCB Japan Co., Ltd. MH has received research grants from AbbVie Japan GK, Asahi Kasei Corp., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Eisai Co., Ltd., Eli Lilly Japan K.K., Kaken Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., Nippon Kayaku Co., Ltd., Nippon Shinyaku Co., Ltd., Pfizer Japan Inc., Taisho Pharmaceutical Co., Ltd., Teijin Pharma Ltd., UCB Japan Co., Ltd., and Viatris Japan. MH has received speaker\u0026rsquo;s fee from AbbVie Japan GK, Asahi Kasei Corp., Astra Zeneca K. K., Ayumi Pharmaceutical Co., Boehringer Ingelheim Japan, Inc., Bristol Myers Squibb Co., Ltd., Chugai Pharmaceutical Co., Ltd., Eisai Co., Ltd., Eli Lilly Japan K.K., GlaxoSmithKline K.K., Gilead Sciences Inc., Janssen Pharmaceutical K.K., Kissei Pharmaceutical Co., Ltd., Mitsubishi Tanabe Pharma Co., Mochida Pharmaceutical Co., Ltd., Ono Pharmaceutical Co., Ltd., Pfizer Japan Inc., Taisho Pharmaceutical Co., Ltd., and Teijin Pharma Ltd. MH is a consultant for AbbVie, Boehringer Ingelheim, Bristol Myers Squibb Co., Kissei Pharmaceutical Co., Ltd., and Teijin Pharma.\u003c/p\u003e\u003c/p\u003e\u003cp\u003e\u003ch2\u003eEthics approval\u003c/h2\u003e\u003cp\u003e The IORRA cohort study (#2952-R and #2922-R16) was approved by the ethics committee of Tokyo Women\u0026rsquo;s Medical University, and informed consent was obtained from all patients before each survey.\u003c/p\u003e\u003c/p\u003e\u003cp\u003e\u003ch2\u003ePatient and public involvement\u003c/h2\u003e\u003cp\u003eThe patients and/or the public were not involved in the design, conduct, reporting, or dissemination of this study.\u003c/p\u003e\u003c/p\u003e\u003cp\u003e\u003cstrong\u003ePatient consent for publication\u003c/strong\u003e\u003cp\u003eNot required.\u003c/p\u003e\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eData sharing statement\u003c/strong\u003e\u003cp\u003eData using analysis that supporting the findings of this study are available upon reasonable request from the authors.\u003c/p\u003e\u003c/p\u003e\u003ch2\u003eFunding\u003c/h2\u003e\u003cp\u003eThis work was supported by JSPS KAKENHI (Grant Number JP23K24465) and a research grant from the Program for the Promotion of Precision Medicine in Rheumatoid Arthritis, Japan College of Rheumatology, 2023.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eM.H. and S.H. conceived the study design. S.H. conducted all analyses with the help of K.I., and S.H. drafted the manuscript. K.I., M.F., S.H., E.T., and M.H. collected samples and clinical information. All authors critically reviewed and approved the manuscript.\u003c/p\u003e\u003ch2\u003eAcknowledgement\u003c/h2\u003e\u003cp\u003eAcknowledgmentWe thank all patients in the IORRA database and all members of the Institute of Rheumatology, Tokyo Women\u0026rsquo;s Medical University Hospital, for the successful management of the IORRA study cohort. The IORRA cohort was supported by unrestricted grants from six pharmaceutical companies: AbbVie Japan GK, Asahi Kasei Pharma Co., Ayumi Pharmaceutical Co., Taisho Pharma Co., Ltd., Teijin Pharma Ltd., and Bristol-Myers Squibb Company. We used ChatGPT-4 minimally for grammar checking during the initial stages of preparing this manuscript. Following this, we utilized Editage's services for professional English language editing. We thank Editage [http://www.editage.com] for editing and reviewing this manuscript for English language.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eVan Veen D, Van Uden C, Blankemeier L, \u003cem\u003eet al.\u003c/em\u003e Adapted large language models can outperform medical experts in clinical text summarization. \u003cem\u003eNat Med\u003c/em\u003e 2024; \u003cstrong\u003e30\u003c/strong\u003e: 1134\u0026ndash;42.\u003c/li\u003e\n\u003cli\u003eJiang LY, Liu XC, Nejatian NP, \u003cem\u003eet al.\u003c/em\u003e Health system-scale language models are all-purpose prediction engines. \u003cem\u003eNature\u003c/em\u003e 2023; \u003cstrong\u003e619\u003c/strong\u003e: 357\u0026ndash;62.\u003c/li\u003e\n\u003cli\u003eTu T, Azizi S, Driess D, \u003cem\u003eet al.\u003c/em\u003e Towards Generalist Biomedical AI. \u003cem\u003eNEJM AI\u003c/em\u003e 2024; \u003cstrong\u003e1\u003c/strong\u003e. DOI:10.1056/AIoa2300138.\u003c/li\u003e\n\u003cli\u003eNori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of GPT-4 on Medical Challenge Problems. 2023. DOI:10.48550/ARXIV.2303.13375.\u003c/li\u003e\n\u003cli\u003eVenerito V, Bilgin E, Iannone F, Kiraz S. AI am a rheumatologist: a practical primer to large language models for rheumatologists. \u003cem\u003eRheumatology (Oxford)\u003c/em\u003e 2023; \u003cstrong\u003e62\u003c/strong\u003e: 3256\u0026ndash;60.\u003c/li\u003e\n\u003cli\u003eBenavent D, Madrid-Garc\u0026iacute;a A. Large language models and rheumatology: are we there yet? \u003cem\u003eRheumatology Advances in Practice\u003c/em\u003e 2024; : rkae119.\u003c/li\u003e\n\u003cli\u003eTouvron H, Martin L, Stone K, \u003cem\u003eet al.\u003c/em\u003e Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023. DOI:10.48550/ARXIV.2307.09288.\u003c/li\u003e\n\u003cli\u003eChen Z, Cano AH, Romanou A, \u003cem\u003eet al.\u003c/em\u003e MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. 2023. DOI:10.48550/ARXIV.2311.16079.\u003c/li\u003e\n\u003cli\u003eYe C, Zweck E, Ma Z, Smith J, Katz S. Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross-Sectional Study. \u003cem\u003eArthritis Rheumatol\u003c/em\u003e 2024; \u003cstrong\u003e76\u003c/strong\u003e: 479\u0026ndash;84.\u003c/li\u003e\n\u003cli\u003eRusinovich Lovgach O, Calvo-Aranda E, Ramos-Lisbona AI, \u003cem\u003eet al.\u003c/em\u003e POS0444 ARTIFICIAL INTELLIGENCE VERSUS RHEUMATOLOGIST IN DECISION MAKING IN THE TREATMENT OF RHEUMATOID ARTHRITIS. DO WE THINK ALIKE? In: Scientific Abstracts. BMJ Publishing Group Ltd and European League Against Rheumatism, 2024: 467.1-467.\u003c/li\u003e\n\u003cli\u003eCoskun BN, Yagiz B, Ocakoglu G, Dalkilic E, Pehlivan Y. Assessing the accuracy and completeness of artificial intelligence language models in providing information on methotrexate use. \u003cem\u003eRheumatol Int\u003c/em\u003e 2024; \u003cstrong\u003e44\u003c/strong\u003e: 509\u0026ndash;15.\u003c/li\u003e\n\u003cli\u003eLuo Y, Yang Z, Meng F, Li Y, Zhou J, Zhang Y. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning. 2023. DOI:10.48550/ARXIV.2308.08747.\u003c/li\u003e\n\u003cli\u003eYamanaka H, Tanaka E, Nakajima A, \u003cem\u003eet al.\u003c/em\u003e A large observational cohort study of rheumatoid arthritis, IORRA: Providing context for today\u0026rsquo;s treatment options. \u003cem\u003eMod Rheumatol\u003c/em\u003e 2020; \u003cstrong\u003e30\u003c/strong\u003e: 1\u0026ndash;6.\u003c/li\u003e\n\u003cli\u003eDettmers T, Pagnoni A, Holtzman A, Zettlemoyer L. QLoRA: Efficient Finetuning of Quantized LLMs. 2023. DOI:10.48550/ARXIV.2305.14314.\u003c/li\u003e\n\u003cli\u003eVickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. \u003cem\u003eMed Decis Making\u003c/em\u003e 2006; \u003cstrong\u003e26\u003c/strong\u003e: 565\u0026ndash;74.\u003c/li\u003e\n\u003cli\u003eRufibach K. Use of Brier score to assess binary predictions. \u003cem\u003eJ Clin Epidemiol\u003c/em\u003e 2010; \u003cstrong\u003e63\u003c/strong\u003e: 938\u0026ndash;9; author reply 939.\u003c/li\u003e\n\u003cli\u003eBrier GW. VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY. \u003cem\u003eMon Wea Rev\u003c/em\u003e 1950; \u003cstrong\u003e78\u003c/strong\u003e: 1\u0026ndash;3.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"large language models, disease activity, rheumatoid arthritis, prediction, pre-training, fine-tuning","lastPublishedDoi":"10.21203/rs.3.rs-7159212/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7159212/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eLarge-language models (LLMs) already excel at extracting clinical facts from electronic health records and drafting differential diagnoses. However, when applied to clinical support for rheumatoid arthritis (RA), despite the fact that the accuracy of disease activity prediction is directly related to the adjustment of treatment intensity, the performance of LLMs and how much they can be improved by pre-training and fine-tuning for medical use have not been fully tested. We therefore trained privacy-preserving, on-premise Llama-2 models with medical domain pre-training (Meditron) and QLoRA fine-tuning, and compared their two-year predictions of RA activity and disability with logistic regression, random forest and XGBoost. The refined LLMs surpassed conventional models on most Disease Activity Score (DAS)-based outcomes, matched them on remission tasks, retained reliable calibration, and offered small but consistent net-benefit advantages where high-disease-activity was rare, while avoiding the clinical harm observed for tabular methods in high-disability prediction. These results show that endpoint-specific, locally deployable LLMs can complement or replace established tabular models in RA management without sacrificing data privacy.\u003c/p\u003e","manuscriptTitle":"Medical pre-training and fine-tuning improve large-language-model prediction of rheumatoid-arthritis disease activity","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-08-08 08:14:51","doi":"10.21203/rs.3.rs-7159212/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"b6425b55-82d9-4edf-a22b-5c9cae46196b","owner":[],"postedDate":"August 8th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":52753820,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":52753821,"name":"Health sciences/Health care"},{"id":52753822,"name":"Physical sciences/Mathematics and computing"},{"id":52753823,"name":"Health sciences/Medical research"},{"id":52753824,"name":"Health sciences/Rheumatology"}],"tags":[],"updatedAt":"2025-10-23T11:39:18+00:00","versionOfRecord":[],"versionCreatedAt":"2025-08-08 08:14:51","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7159212","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7159212","identity":"rs-7159212","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}