Large Language Models in Inflammatory Arthritis: A Systematic Review Across Clinical Tasks

Large Language Models in Inflammatory Arthritis: A Systematic Review Across Clinical Tasks | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Systematic Review Large Language Models in Inflammatory Arthritis: A Systematic Review Across Clinical Tasks Yosef Adiniaev, Mahmud Omar, Tohar M Timor, Yiftach Barash, Olga R Brook, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9529682/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background Managing inflammatory arthritis involves combining clinical, serological, and imaging data while following evolving treatment guidelines. Large language models (LLMs) are increasingly being evaluated for rheumatology tasks, but whether this promise holds in inflammatory arthritis remains unclear. We therefore systematically reviewed the performance of LLMs across clinical tasks in inflammatory arthritis. Methods We conducted a systematic review (PROSPERO: CRD420261359100) searching PubMed, Scopus, and PubMed Central (January 2022 to April 2026). Eligible studies evaluated LLM performance on clinical tasks in inflammatory arthritis. Two reviewers (Y.A., A.G.) screened 113 records. Risk of bias was assessed using an adapted QUADAS-2 framework with AI-specific modifications. Results Eighteen studies met inclusion criteria, covering rheumatoid arthritis (n=4), ankylosing spondylitis/axial spondyloarthritis (n=7), psoriatic arthritis (n=2), gout (n=1), juvenile idiopathic arthritis (n=1), and multiple diseases (n=3). Over 20 distinct LLMs were evaluated, including ChatGPT-3.5, ChatGPT-4, ChatGPT-4o, Gemini 2.0, DeepSeek-R1/V3, Claude, Perplexity, and Kimi. Findings were synthesized across four application domains: patient education (n=11), guideline adherence (n=6), clinical reasoning and real-world applications (n=3), and other (n=1). All studies assessing readability reported outputs above recommended thresholds; studies using the Flesch-Kincaid index reported Grade Levels above 15. Studies comparing multiple LLMs found a trade-off between readability and scientific reliability. Guideline concordance varied widely across models, from 48% to 96%. In one blinded comparison, patients preferred LLM responses over specialist-written answers. Lower accuracy was reported for case-based clinical scenarios (4.24/6) compared with FAQ and guideline-based questions (5.32–5.36/6; overall p=0.044). When LLM outputs were compared with clinical data from 116 patients with axial spondyloarthritis and psoriatic arthritis, agreement was low to moderate (Cohen and Fleiss kappa). Conclusions LLMs may support patient education, factual medication queries, and structured guideline questions when used under clinician review, but should not be used for case-based reasoning, treatment selection, or autonomous clinical decisions. None of the 18 included studies evaluated retrieval-augmented or agent-based systems, and none prospectively validated LLMs in clinical workflows. Safe integration in rheumatology will require purpose-built, knowledge-grounded systems and prospective evaluation before routine clinical use. Registration: PROSPERO CRD420261359100 Inflammatory arthritis Large language models Systematic review Patient education Figures Figure 1 Figure 2 Figure 3 1. Introduction Inflammatory arthritis, defined here as chronic immune-mediated joint diseases characterized by synovial inflammation and progressive structural damage, places a heavy burden on patients and healthcare systems [ 1 ]. Rheumatoid arthritis (RA) alone affects approximately 18 million people worldwide [ 2 ]. Management of these conditions is complex and involves integration of multiple data sources [ 3 ]. Clinicians must interpret labs, imaging, and clinical criteria, all while keeping up with frequently updated treatment guidelines from organizations like the American College of Rheumatology (ACR) and the European Alliance of Associations for Rheumatology (EULAR) [ 4 , 5 ]. Early initiation of treatment is critical. In RA and psoriatic arthritis (PsA), delays in disease-modifying antirheumatic drugs (DMARDs) can lead to permanent joint damage [ 3 , 6 ]. The same is true for biologic therapies in ankylosing spondylitis (AS) and urate-lowering therapy in gout [ 7 , 8 ]. Large language models (LLMs) such as ChatGPT have recently drawn attention as potential tools for clinical support. These AI systems can generate human-like text, making them appealing for tasks like answering patient questions, summarizing guidelines, and supporting diagnostic reasoning [ 9 ]. Recent studies have evaluated how well LLMs perform on tasks relevant to inflammatory arthritis. However, LLMs were not built for clinical decision-making. Although recent developments have enabled access to external tools and real-time data sources [ 10 ], LLMs still do not consistently follow current guidelines, and sometimes produce confident but incorrect answers, a phenomenon known as hallucination [ 11 , 12 ]. Their performance also varies considerably depending on model type and clinical task. Prior reviews have examined the role of natural language processing and LLMs in rheumatology broadly [ 13 ], while others have focused specifically on osteoarthritis [ 14 , 15 ]. However, no systematic review has specifically examined LLM use in inflammatory arthritis. We focused on RA, PsA, AS/axSpA, and gout because these are the most prevalent inflammatory arthritides with established ACR/EULAR diagnostic and treatment guidelines, and because a preliminary literature survey confirmed that published LLM evaluation studies in inflammatory arthritis were concentrated on these four conditions [ 13 , 14 ]. Other inflammatory joint diseases, including calcium pyrophosphate deposition disease, reactive arthritis, and autoinflammatory syndromes such as familial Mediterranean fever and Behcet's disease, were outside the predefined scope of this review but represent important targets for future evaluation. This systematic review aims to (1) evaluate the accuracy and clinical relevance of LLM-generated outputs across diagnostic, therapeutic, and educational tasks in RA, PsA, AS/axSpA, and gout; (2) compare performance across LLM models, diseases, and task types; and (3) identify methodological gaps and safety concerns relevant to clinical implementation. 2. Methods Protocol and registration We conducted this review following the PRISMA 2020 guidelines [ 16 ]. The protocol was registered on PROSPERO (CRD420261359100). No deviations from the registered protocol were identified. Data sources and searches We searched three databases: PubMed, Scopus, and PubMed Central (PMC). The search covered studies published from January 1, 2022, to April 3, 2026. We chose 2022 as the start date due to the release of ChatGPT-3.5 and the surge of LLM into clinical medicine by that year. The search combined two groups of terms. The first group captured LLM-related concepts: "large language model," "LLM," "ChatGPT," "GPT-4," "GPT-3," "GPT-4o," "generative AI," and "generative artificial intelligence." The second group captured inflammatory arthritis conditions: "rheumatoid arthritis," "psoriatic arthritis," "ankylosing spondylitis," "axial spondyloarthritis," "gout," "gouty arthritis," "spondyloarthritis," and "inflammatory arthritis." Full search strings for each database are provided in the Supplementary Materials . Only English language publications were included. We also performed forward and backward citation searching to identify additional relevant studies. Study screening We included studies that evaluated LLMs in the context of inflammatory arthritis. This covered any generative AI system capable of producing human-like text, including ChatGPT, GPT-4, GPT-3.5, Google Gemini, Anthropic Claude, and DeepSeek, among others. The diseases of interest were RA, PsA, AS/axial spondyloarthritis (axSpA), and gout. We accepted studies that assessed LLMs for clinical diagnosis, treatment recommendations, patient education, guideline adherence, or clinical decision support. Each study needed to report some form of performance data, whether quantitative (e.g., accuracy, concordance, DISCERN scores) or through systematic expert evaluation. Only peer-reviewed journal articles were included. We excluded studies that focused only on osteoarthritis, as this area has been covered by recent reviews [ 14 , 15 ]. We also excluded studies that evaluated traditional machine learning, deep learning, or NLP methods without an LLM component. Studies using AI only for medical imaging without involving an LLM were excluded as well. Editorials, commentaries, letters without original data, conference abstracts, preprints, and study protocols were also not eligible. Two reviewers (Y.A., A.G.) independently screened all titles and abstracts using Rayyan [ 17 ]. Articles flagged for potential inclusion by either reviewer moved to full-text review. Any disagreements were resolved through third reviewer (E.K.). Data extraction We created a standardized extraction form and tested it on three studies before applying it to all included articles. One reviewer (Y.A.) extracted the data, and a second reviewer (A.G.) verified all entries. Disagreements were resolved by discussion. For each study, we collected the first author, year, journal, and study design. We also recorded the disease(s) studied, the LLM(s) used (including model version and access date when available), and the type of clinical task assessed. To capture performance, we extracted the main outcome, the comparator, the sample size, key results, and reported limitations. Risk of bias assessment We adapted the QUADAS-2 framework with AI-specific modifications to assess risk of bias in the included studies. Given the absence of a validated risk-of-bias tool for LLM evaluation studies, QUADAS-2 was selected as the closest established framework and adapted to reflect key methodological features specific to AI-based evaluations. We modified it by redefining the index test domain to address LLM-specific factors (model version, access date, prompt design, and reproducibility) and by adjusting the reference standard domain to account for comparators used in LLM studies (expert panels, clinical guidelines, or validated scoring tools). The adapted instrument evaluates four domains: (1) data selection, (2) the index test (the LLM), (3) the reference standard, and (4) flow and timing. Each domain was rated as low risk, high risk, or unclear. We also assessed applicability concerns for the first three domains. The complete adapted checklist is provided in Supplementary Table S1. We considered a study to be at high risk of bias if it used a small or unrepresentative set of questions, relied on a single LLM version without reporting the access date or prompt design, lacked comparison with clinical experts or validated tools, used a single evaluator without checking inter-rater reliability, or selectively reported outcomes. One reviewer (Y.A.) assessed each study, and a second reviewer (A.G.) verified the ratings. Disagreements were resolved through discussion. Synthesis of findings Because the included studies varied widely in design, LLM models tested, clinical tasks, and reporting methods, a quantitative meta-analysis was not feasible. Instead, we performed a narrative synthesis. We first grouped studies by disease type: RA, PsA, AS, and gout. We then examined patterns across application domains, including patient education, guideline adherence, clinical reasoning, and clinical data extraction. Finally, we compared findings across LLM models, comparators, and outcome measures reported. 3. Results Study selection The search identified 159 records: 53 from PubMed, 99 from Scopus, and 7 from PubMed Central. After removing 46 duplicates, 113 records were screened by title and abstract. Eighty-five did not meet the inclusion criteria. Full-text assessment of the remaining 28 articles led to the exclusion of 10 that did not specifically evaluate LLMs in the context of inflammatory arthritis, lacked performance data, or focused on diseases outside the scope of this review. Eighteen studies were included in the final synthesis (Fig. 1 ). Study characteristics The 18 included studies were published between 2024 and 2026, with the majority (11 studies) appearing in 2025. Sixteen studies used cross-sectional designs, one was a retrospective observational study, and one was a pilot study. AS/axSpA was the most frequently studied condition (7 studies), followed by RA (4), PsA (2), gout (1), JIA (1), and three studies covering multiple diseases. ChatGPT and its variants (GPT-3.5, GPT-4, GPT-4o) were evaluated in 16 of 18 studies. Other models tested included Google Gemini (7 studies), DeepSeek (5), Claude (1), and Perplexity (2) (Table 1 , Fig. 2 ). Table 1 Characteristics and key findings of included studies, organized by disease. First Author Year LLM(s) Application Key Finding Sample Rheumatoid Arthritis (n = 4) Coskun BN 2024 GPT-3.5, GPT-4, BARD, Bing Patient info (MTX) GPT-4: 100% accuracy; BARD/Bing: 60.87% 23 Qs; 2 rheumatologists Cabuk Celik N 2025 ChatGPT-4 EULAR adherence 96% binary correct; self-corrected errors at day 14 100 Qs at 2 time-points Berghea F 2025 ChatGPT 4.5, Gemini, Qwen, DeepSeek, Perplexity Risk appetite ChatGPT least risk-averse; method-dependent profiles RA scenarios; 5 GenAIs Saji JG 2025 ChatGPT, Gemini, Copilot, Apple Intelligence, Meta AI Patient education Readability-reliability trade-off across platforms 5 conditions x 5 platforms Psoriatic Arthritis (n = 2) Forte G 2025 ChatGPT-4 Expert comparison Patients preferred ChatGPT (49.2%) over experts (34.4%) 32 Qs; 12 experts; 67 patients Atilan AU 2026 7 LLMs Patient ed. (Turkish) Claude/Gemini most reliable; ChatGPT most readable 7 LLMs; DISCERN & readability Ankylosing Spondylitis / axSpA (n = 7) Ren Y 2025 ChatGPT-4o, 3.5, Gemini, Kimi Patient education ChatGPT-4o/Kimi outperformed guidelines; high acceptance 182 volunteers; 15 Qs Usen A 2025 ChatGPT-3.5, 4o, Gemini 2.0 ASAS-EULAR adherence GPT-3.5 recommended contraindicated therapies 15 guideline Qs Bai J 2026 GPT-4.0, DeepSeek R1, Hunyuan, Kimi, Wenxin Health guidance GPT-4.0: 94% guideline agreement 84 patients, 26 MDs Kara M 2025 ChatGPT-4o, Gemini, Perplexity Patient info quality Perplexity highest quality; all above readability threshold 25 Google Trends Qs Altunel Kilinc E 2025 ChatGPT-4 Patient/physician ed. FAQs 5.32/6; complex cases 4.24/6 (p = 0.044) 75 Qs in 3 groups Cabuk Celik N 2025 ChatGPT-4o, 3.5, DeepSeek R1/V3 Patient ed. quality ChatGPT-4o highest DISCERN (72.38); all Flesch-Kincaid Grade Level (FKGL) > 15 10 Qs; 4 LLMs Sari F 2025 ChatGPT-4, DeepSeek-V3 Exercise/rehab DeepSeek-V3 > ChatGPT-4 (p < 0.001); both very difficult 50 Qs; 3 physiotherapists Gout (n = 1) Meral HB 2026 ChatGPT-4o, Gemini 2.0 EULAR adherence ChatGPT-4o: 76% vs Gemini 48%; Gemini 8% contradictory 25 EULAR Qs Juvenile Idiopathic Arthritis (n = 1) La Bella S 2025 ChatGPT 4o Guideline adherence 52–84% adherent to ACR; regional differences 10 PICOs x 5 countries Multiple / Mixed (n = 3) Kayacan-Erdogan E 2024 GPT-4 (trained/untrained) Clinical reasoning Trained GPT-4 scored higher than residents at all stages 10 cases; 10 residents Miao BY 2025 GPT-4 + 8 opensource Clinical NLP (EHR) GPT-4 F1 0.75–0.83; efficacy loss 56.9% 9,187 patients Polyzou M 2026 GPT-5, DeepSeek-R1 Dx/Tx vs real data Moderate agreement with clinical data from 116 patients 116 patients Risk of bias Risk of bias was assessed using an adapted QUADAS-2 framework across four domains. Of 126 individual domain assessments (18 studies across 7 domains including applicability), 105 (83.3%) were rated as low risk, 14 (11.1%) as high risk, and 7 (5.6%) as unclear (Figure S1). Most studies evaluated LLMs using synthetic clinical questions or multiple-choice formats rather than real patient data, which may inflate the proportion of low-risk ratings; the adapted QUADAS-2 ratings reflect the internal methodological quality of each study within its chosen design. In the Data Selection domain, 14 of 18 studies (77.8%) were rated as low risk, three as high risk because they relied on small question sets or non-clinically validated sources, and one as unclear. In the Index Test domain, 14 of 18 studies (77.8%) were low risk, three were high risk, and one was unclear, primarily due to evaluation of a single LLM model without reporting access dates. Reference Standard assessments were low risk in 16 of 18 studies (88.9%), with one rated high risk due to weak inter-rater agreement (kappa values mostly below 0.40) [ 19 ] and one rated unclear. Flow and Timing was low risk in 17 of 18 studies (94.4%), with one rated unclear. Applicability concerns were mostly low. The main exceptions involved studies conducted exclusively in Chinese or Turkish, where LLM performance may not generalize to other languages. The study by Berghea et al. [ 18 ] had high applicability concerns across multiple domains because its focus on AI risk appetite does not align with standard clinical evaluation frameworks. Synthesis of findings Findings were organized into four predefined application domains based on common evaluation categories reported in prior LLM medical literature [ 13 , 14 ]: patient education and information quality (11 studies), guideline adherence and clinical decision support (6 studies), clinical reasoning and real-world applications (3 studies), and other applications (1 study). Three studies were classified under more than one domain based on their stated objectives; the totals therefore exceed 18. The included studies used heterogeneous evaluation metrics, including percentage correct, Likert-scale accuracy scores, DISCERN scores, F1 scores, and guideline concordance ratings, which limits direct numerical comparison across studies. A summary of outcomes by application domain is presented in Table 2 . Patient education and information quality Eleven studies evaluated LLM-generated content for patient education across RA, PsA, AS, JIA, and gout [ 19 – 29 ]. Of these, 8 of 11 reported moderate to high accuracy for at least one LLM, as defined by study-specific scoring systems. Among the 7 studies that used DISCERN or modified DISCERN, scores ranged from 38.5 to 72.38. All 10 studies that assessed readability found outputs above the recommended sixth-grade reading level. Reported accuracy ranged from 60.87% (BARD and Bing in Coskun et al. [ 20 ]) to 100% (GPT-4 in the same study). In the largest patient education study, Forte et al. [ 22 ] collected 32 questions from 76 PsA patients and compared ChatGPT-4 responses against answers written by 12 specialist rheumatologists. Fourteen blinded experts found no significant difference in accuracy between the two sources. In a separate arm, 67 patients reviewed blinded answer pairs and selected the ChatGPT response 49.2% of the time, compared with 34.4% for the expert answer. One domain where ChatGPT scored significantly lower was pregnancy and fertility. Atilan et al. [ 23 ] compared seven LLMs generating Turkish patient brochures for PsA and psoriasis. Claude and Gemini achieved the highest DISCERN reliability scores (60.5 and 57.5), while ChatGPT produced the most readable output (Atessman score 80.2). Mistral scored lowest across all domains (total DISCERN 38.5). Saji et al. [ 21 ] observed a similar pattern across five AI platforms: Google Gemini was the most readable but scored lowest on reliability, while Microsoft Copilot was the most reliable but the least readable. All studies that assessed readability reported levels above recommended thresholds. Cabuk Celik et al. [ 28 ] found all four LLMs tested for AS education produced text requiring university-level reading (FKGL above 15). Sari et al. [ 29 ] reported Flesch Reading Ease scores of 28.77 to 30.4 for exercise information. La Bella et al. [ 19 ] reported a median Flesch Reading Ease Score (FRES) of 30 for JIA content across five countries. One exception was Ren et al. [ 24 ], who reported high patient acceptance of LLM-generated AS education in Chinese, with ChatGPT-4o and Kimi outperforming traditional guideline text. Guideline adherence and clinical decision support Six studies tested LLM concordance with established treatment guidelines [ 19 , 25 , 27 , 30 – 32 ]. Of these, 5 of 6 reported concordance above 75% for at least one GPT-4 variant, while concordance for non-GPT models was lower and more variable. Two of 6 studies documented clinically inappropriate recommendations produced by at least one model. Concordance rates ranged from 48% (Gemini for gout [ 32 ]) to 96% (ChatGPT-v4 for binary EULAR RA questions [ 30 ]). In 5 of 6 studies, GPT-4 variants achieved the highest concordance scores. Bai et al. [ 25 ] reported 94% overall agreement between GPT-4.0 and axSpA guidelines. Cabuk Celik et al. [ 30 ] found 96% accuracy on binary EULAR RA questions, and when the same questions were repeated 14 days later, the model corrected four of seven previous errors. For gout, Meral et al. [ 32 ] found ChatGPT-4o aligned with EULAR guidelines in 76% of responses, compared with 48% for Gemini. Gemini contradicted the guidelines in 8% of cases, while ChatGPT-4o had none (Fig. 3 ). In at least one study, older models generated recommendations inconsistent with current treatment guidelines. Usen et al. [ 31 ] found that ChatGPT-3.5 recommended IL-23 inhibitors for axSpA, which are not part of the approved treatment algorithm, and suggested DMARD combinations inconsistent with ASAS-EULAR guidelines. Lower accuracy scores were reported for case-based clinical scenarios compared with factual or guideline-based questions in studies that evaluated both formats. Altunel Kilinc et al. [ 27 ] tested ChatGPT-4 on three categories of AS questions: frequently asked patient questions (5.32/6), open-ended guideline questions (5.36/6), and case-based clinical scenarios (4.24/6, p = 0.044). Clinical reasoning and real-world applications Three studies evaluated LLMs beyond structured question-answering [ 33 – 35 ]. In a pilot study, Kayacan Erdogan et al. [ 33 ] compared a trained GPT-4 against 10 junior rheumatology residents on 10 inflammatory arthritis cases. The trained model scored higher than the residents at all stages of clinical reasoning. In a post-study survey, 60% of residents indicated they would welcome AI as a supportive tool. Miao et al. [ 34 ] applied GPT-4 and eight open-source LLMs to extract reasons for TNFi switching from the electronic health records of 9,187 patients at a single academic center. GPT-4 achieved micro-F1 scores of 0.75 (stopped drug), 0.80 (started drug), and 0.83 (switching reason). The most frequently identified reason was lack of efficacy (56.9%), followed by adverse events (13.5%) and insurance or cost issues (10.8%). Polyzou et al. [ 35 ] compared GPT-5 and DeepSeek-R1 against clinical findings from 116 patients with axSpA and PsA at a tertiary center in Germany. Cohen and Fleiss Kappa coefficients showed no to low agreement between LLM predictions and clinical data. Heterogeneity across studies There was substantial heterogeneity across the included studies. Models ranged from GPT-3.5 (released 2022) to GPT-5 (2026). Prompting strategies varied from standardized zero-shot prompts to trained models. Languages included English, Chinese, and Turkish. Evaluation metrics were not standardized. Sample sizes ranged from 10 questions to 9,187 patient records. Given this heterogeneity, a formal meta-analysis was not feasible. Across studies, LLMs achieved higher accuracy on structured, guideline-based tasks and lower performance on case-based or real-world clinical scenarios. This pattern was observed across diseases and model types, suggesting a general limitation in clinical reasoning rather than disease-specific variability. 4. Discussion This review identified 18 studies evaluating LLMs across RA, PsA, AS/axSpA, gout, and JIA. LLMs achieved concordance rates of 76–96% with EULAR and ASAS-EULAR guidelines on structured questions [ 25 , 30 , 32 ], and generated patient education content that patients with PsA preferred over specialist-written answers in one blinded comparison [ 22 ]. However, when tested on real clinical data from 116 patients with axSpA and PsA, agreement dropped to low-moderate levels [ 35 ]. Disease-specific performance in rheumatology LLM performance differed across inflammatory arthritis subtypes. For RA, LLMs performed well on methotrexate-related questions and binary EULAR criteria [ 20 , 30 ], but showed inconsistent risk tolerance across treatment scenarios, with variation between models and assessment methods [ 18 ]. For AS/axSpA, where seven studies were available, LLMs handled structured questions adequately (Group 1: 5.32/6, Group 2: 5.36/6) but achieved lower scores on case-based clinical scenarios (4.24/6, p = 0.044) [ 27 ]. In axSpA, treatment decisions require sequencing between NSAIDs, TNF inhibitors, and IL-17 inhibitors, and this difference between structured and case-based performance may limit clinical applicability. For gout, ChatGPT-4o aligned with EULAR recommendations in 76% of responses, but Gemini contradicted them in 8% of cases [ 32 ], a finding relevant to clinical practice given the role of comorbidity assessment in urate-lowering therapy decisions. For PsA, the only study with real patient involvement showed that while patients preferred AI-generated content, ChatGPT performed poorly on pregnancy and fertility counselling [ 22 ]. These results indicate that LLM evaluation in rheumatology should include clinically relevant decision-making scenarios such as DMARD escalation, biologic switching, treatment in pregnancy, and management of comorbidities. Beyond standalone LLMs: AI agents and retrieval-augmented generation In this review, LLMs performed well on guideline-based MCQs but lacked the clinical reasoning needed for real patient management, which requires external data and tool integration [ 12 ]. All 18 included studies evaluated standalone LLMs without access to external data sources, real-time guidelines, or iterative reasoning capabilities. Recent studies report that AI agent systems, which extend LLMs with tool integration, iterative planning, and retrieval-augmented generation (RAG), outperform base LLMs on clinical tasks [ 36 ]. Independent evaluations report similar results: a RAG-augmented system improved hepatology guideline accuracy from 43% to 99% compared with standalone GPT-4 [ 37 ], and retrieval-augmented models outperformed base LLMs across nine medical specialties in a blinded clinician evaluation [ 38 ]. A recent meta-analysis of 20 studies confirmed that RAG significantly improves LLM performance in biomedical applications (pooled OR 1.35, 95% CI 1.19–1.53) [ 39 ]. In rheumatology, where treatment guidelines are updated frequently (e.g., 2022 ASAS-EULAR update [ 5 ], 2021 ACR RA guideline [ 4 ]) and where biologic selection depends on real-time laboratory and imaging data, an agent-based approach grounded in current guidelines through RAG could address several limitations observed in this review. Tool-augmented LLMs that retrieve current ACR/EULAR recommendations before generating responses could reduce the risk of outdated or contraindicated treatment suggestions, such as the IL-23 inhibitor error observed by Usen et al. [ 31 ]. Knowledge-grounded systems have shown improved accuracy and reduced hallucination in other clinical domains [ 37 – 39 ]. Even when augmented with external sources, LLMs may still generate or misinterpret clinical information, and human verification remains essential [ 40 ]. Safety and reliability The safety profile of LLMs in rheumatology raises several concerns beyond simple inaccuracy. LLMs are vulnerable to adversarial hallucination attacks: when presented with fabricated clinical details, models elaborated on false information in up to 83% of cases [ 40 ]. This is particularly relevant in rheumatology, where clinical notes may contain ambiguous findings, and an LLM that confidently elaborates on incorrect serological results or imaging interpretations could mislead clinicians. Additionally, LLM outputs are sensitive to how clinical questions are framed. Recent work has shown that patient communication style alone, such as urgent or demanding language, can shift AI-generated triage decisions and sick-leave recommendations [ 41 , 42 ]. In rheumatology practice, where patients often present with subjective complaints like morning stiffness or fatigue, this framing sensitivity means that the same clinical situation could generate different AI recommendations depending on how the question is phrased. None of the studies in this review tested prompt sensitivity, leaving the robustness of reported accuracy rates uncertain. Structured prompting strategies may partially mitigate these effects, but remain untested in rheumatology [ 42 ]. Clinical implications for rheumatology practice The evidence from this review does not support using LLMs for direct treatment decisions in inflammatory arthritis without specialist involvement. In RA, DMARD escalation from methotrexate to biologics depends on disease activity scores, comorbidity profiles, and prior treatment response, none of which current LLMs can access or integrate. In axSpA, the choice between TNF and IL-17 inhibitors requires consideration of extra-articular manifestations such as uveitis and inflammatory bowel disease. In gout, urate-lowering therapy must be adjusted for renal function. These decisions require more than guideline recall, and the errors observed in this review, including recommendation of contraindicated IL-23 inhibitors for axSpA [ 31 ] and guideline-contradictory responses for gout [ 32 ], confirm that unsupervised use carries real clinical risk. Regulatory and ethical considerations None of the included studies addressed liability when LLM-generated recommendations lead to patient harm. This gap is particularly relevant in rheumatology, where treatment errors can result in irreversible joint destruction if DMARDs are delayed, teratogenic exposure if methotrexate is not stopped before conception, or renal complications if gout therapies are prescribed without monitoring. As LLMs become more accessible to patients and non-specialist clinicians, the question of who bears responsibility for AI-generated rheumatology advice, whether the developer, the clinician who shares it, or the institution that deploys it, remains unresolved. Regulatory bodies have not yet established frameworks for LLM use in subspecialty medicine, and the rheumatology community should engage with this process before these tools enter routine clinical workflows. Practical guidance for rheumatologists Based on the 18 studies reviewed, LLMs may reasonably support three tasks in rheumatology practice today, always under clinician review. First, drafting patient education content, where accuracy was generally acceptable although readability consistently exceeded recommended levels and required editing. Second, answering factual medication questions, as shown by GPT-4 achieving 100% accuracy on methotrexate-related queries in RA [ 20 ]. Third, responding to structured binary guideline questions, where ChatGPT-4 reached 96% concordance with EULAR RA recommendations [ 30 ]. LLMs should not be used without clinician verification for several tasks. Case-based clinical reasoning showed consistent performance drops across studies, with scores falling from 5.32/6 on structured questions to 4.24/6 on complex cases [ 27 ]. Treatment selection and sequencing carry documented safety risks, including the recommendation of contraindicated IL-23 inhibitors for axSpA [ 31 ]. Guideline-dependent decisions such as gout management showed model-specific errors, with Gemini contradicting EULAR recommendations in 8% of responses [ 32 ]. Autonomous clinical decision-making is not supported by current evidence, as no study prospectively validated LLM outputs in clinical workflows. Rheumatology-specific challenges for LLM integration Several features of rheumatology practice make it a particularly demanding setting for LLM deployment. Treatment guidelines are updated frequently (2022 ASAS-EULAR update [ 5 ], 2021 ACR RA guideline [ 4 ]), which means models trained on older data will generate outdated recommendations unless connected to current sources. Biologic and targeted synthetic DMARD selection depends on multiple interacting factors including comorbidities, pregnancy status, prior treatment failures, and emerging safety signals such as the 2022 FDA boxed warning for JAK inhibitors, which standalone LLMs cannot reliably integrate. Inflammatory arthritides share overlapping clinical features, complicating differential diagnosis in ways that generic LLMs tend to oversimplify. Finally, 10 of 18 included studies were conducted in Turkish and reported variable model performance, indicating that LLM utility in lower-resource languages remains unclear. These characteristics explain why LLM integration in rheumatology will likely require purpose-built, retrieval-augmented systems rather than general-purpose chatbots. Limitations This review has limitations. The published literature to date is largely based on earlier-generation models: BARD has been discontinued, GPT-3.5 has been superseded, and GPT-5 represents the current generation. This reflects the pace of LLM development rather than a limitation of the review itself, but it means that the accuracy estimates reported here likely represent a floor rather than a ceiling of current capability. Evaluations of newer models are beginning to appear but remain limited. Because most evaluations relied on synthetic prompts or predefined questions rather than real patient encounters, current evidence may overestimate LLM performance in clinical practice. The concentration of studies in Turkey (10 of 18) limits geographic generalizability. This overrepresentation likely reflects the early adoption of LLM evaluation research by Turkish rheumatology groups rather than a higher clinical need. Most Turkish studies evaluated LLM responses in Turkish, and model performance in lower-resource languages may not generalize to English or other languages. The limited representation of North America, East Asia, and sub-Saharan Africa means that findings may not reflect LLM performance in healthcare systems with different practice patterns and guideline frameworks. Heterogeneous evaluation metrics and clinical tasks precluded meta-analysis. Gout and JIA were each represented by a single study. All studies were conducted in research settings with no prospective clinical validation, and publication bias likely favours positive results. The wide range of application domains assessed (patient education, clinical reasoning, guideline adherence, and data extraction) increased study heterogeneity and precluded quantitative pooling. Future reviews focusing on a single domain across a broader spectrum of inflammatory joint diseases may enable meta-analysis. Conclusions For the practicing rheumatologist, current evidence supports LLM use under clinician review for patient education drafting, factual medication queries, and structured guideline questions. LLMs should not be used for complex case-based reasoning, treatment selection, or autonomous clinical decisions. Across 18 studies, no evaluation tested retrieval-augmented or agent-based systems, and none prospectively validated LLM outputs in clinical workflows. Rheumatology poses particular challenges for LLM integration, including frequently updated guidelines, complex treatment algorithms with multiple interacting factors, and limited evidence in non-English clinical settings. These gaps may be more important than accuracy estimates in guiding future research and in determining when LLMs are ready for clinical deployment. Declarations Author Contributions: Conceptualization, Y.A., A.G., E.K.; Methodology, Y.A., A.G., E.K.; Formal Analysis, Y.A., A.G.; Data Curation, Y.A., A.G.; Writing-Original Draft Preparation, Y.A., A.G.; Writing-Review & Editing, Y.A., A.G., M.O., E.K., Y.B., O.R.B., T.M.T.; Supervision: E.K., A.G. Funding. None Competing interests. The authors declare that they have no competing interests. References Bournia V-K, Fragoulis GE, Mitrou P et al (2024) Increased prevalence of inflammatory arthritis, systemic lupus erythematosus and systemic sclerosis, during 2020–2023 versus 2016–2019 in a Nation-Wide Cohort Study. Rheumatol Int 44:2837–2846. https://doi.org/10.1007/s00296-024-05733-y GBD 2021 Rheumatoid Arthritis Collaborators (2023) Global, regional, and national burden of rheumatoid arthritis, 1990–2020, and projections to 2050: a systematic analysis of the Global Burden of Disease Study 2021. Lancet Rheumatol 5:e594–610. https://doi.org/10.1016/S2665-9913(23)00211-4 Aletaha D, Smolen JS (2018) Diagnosis and Management of Rheumatoid Arthritis: A Review. JAMA 320:1360–1372. https://doi.org/10.1001/jama.2018.13103 Fraenkel L, Bathon JM, England BR et al (2021) 2021 American College of Rheumatology Guideline for the Treatment of Rheumatoid Arthritis. Arthritis Care Res (Hoboken) 73:924–939. https://doi.org/10.1002/acr.24596 Ramiro S, Nikiphorou E, Sepriano A et al (2023) ASAS-EULAR recommendations for the management of axial spondyloarthritis: 2022 update. Ann Rheum Dis 82:19–34. https://doi.org/10.1136/ard-2022-223296 Combe B, Landewe R, Daien CI et al (2017) 2016 update of the EULAR recommendations for the management of early arthritis. Ann Rheum Dis 76:948–959. https://doi.org/10.1136/annrheumdis-2016-210602 Ward MM, Deodhar A, Gensler LS et al (2019) 2019 Update of the American College of Rheumatology/Spondylitis Association of America/Spondyloarthritis Research and Treatment Network Recommendations for the Treatment of Ankylosing Spondylitis and Non-radiographic Axial Spondyloarthritis. Arthritis Care Res (Hoboken) 71:1285–1299. https://doi.org/10.1002/acr.24025 FitzGerald JD, Dalbeth N, Mikuls T et al (2020) 2020 American College of Rheumatology Guideline for the Management of Gout. Arthritis Care Res (Hoboken) 72:744–760. https://doi.org/10.1002/acr.24180 Thirunavukarasu AJ, Ting DSJ, Elangovan K et al (2023) Large language models in medicine. Nat Med 29:1930–1940. https://doi.org/10.1038/s41591-023-02448-8 Bedi S, Liu Y, Orr-Ewing L et al (2025) Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA 333:319–328. https://doi.org/10.1001/jama.2024.21700 Ji Z, Lee N, Frieske R et al (2023) Survey of Hallucination in Natural Language Generation. ACM Comput Surv 55:248:1–248. https://doi.org/10.1145/3571730 Hager P, Jungmann F, Holland R et al (2024) Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med 30:2613–2622. https://doi.org/10.1038/s41591-024-03097-1 Omar M, Naffaa ME, Glicksberg BS et al (2024) Advancing rheumatology with natural language processing: insights and prospects from a systematic review. Rheumatol Adv Pract 8:rkae120. https://doi.org/10.1093/rap/rkae120 Fairley JL, Kapoor M, Sharma D (2026) Generative artificial intelligence in osteoarthritis: A systematic scoping review of current applications and future directions. Osteoarthritis Cartilage. https://doi.org/10.1016/j.joca.2026.03.001 . S1063-4584(26)00685-0 Ma Z, Liu Y, Zhang Z et al (2025) Clinical applications of large language models in knee osteoarthritis: a systematic review. Front Med (Lausanne) 12:1670824. https://doi.org/10.3389/fmed.2025.1670824 Page MJ, McKenzie JE, Bossuyt PM et al (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372:n71. https://doi.org/10.1136/bmj.n71 Ouzzani M, Hammady H, Fedorowicz Z et al (2016) Rayyan-a web and mobile app for systematic reviews. Syst Rev 5:210. https://doi.org/10.1186/s13643-016-0384-4 Berghea F, Andras D, Berghea EC (2025) Generative Artificial Intelligence and Risk Appetite in Medical Decisions in Rheumatoid Arthritis. Appl Sci 15:5700. https://doi.org/10.3390/app15105700 La Bella S, Bayraktar D, Porreca A et al (2025) Global variations in artificial intelligence-generated information on juvenile idiopathic arthritis. Rheumatology (Oxford) 64:5687–5697. https://doi.org/10.1093/rheumatology/keaf329 Coskun BN, Yagiz B, Ocakoglu G et al (2024) Assessing the accuracy and completeness of artificial intelligence language models in providing information on methotrexate use. Rheumatol Int 44:509–515. https://doi.org/10.1007/s00296-023-05473-5 Saji JG, Jabeen J, Antony JT (2025) Evaluating readability, reliability, and originality of artificial intelligence-generated patient education guides for common rheumatological conditions. J Public Health (Berl). https://doi.org/10.1007/s10389-025-02628-5 Forte G, Mauro D, Raimondi M et al ChatGPT vs rheumatologists: cross-sectional study on accuracy and patient perception of AI-generated information for psoriatic arthritis. Ann Rheum Dis 2025:S0003-4967(25)04538-8. https://doi.org/10.1016/j.ard.2025.11.012 Atilan AU, Cetin N (2026) An old disease, a new linguistic challenge for large language models: patient education on psoriasis and psoriatic arthritis in an underrepresented medical language. Int J Med Inf 209:106246. https://doi.org/10.1016/j.ijmedinf.2025.106246 Ren Y, Kang Y, Cao S et al (2025) Evaluating the performance of large language models in health education for patients with ankylosing spondylitis/spondyloarthritis: a cross-sectional, single-blind study in China. BMJ Open 15:e097528. https://doi.org/10.1136/bmjopen-2024-097528 Bai J, Ji X, Yu J et al (2026) Assessing the Quality of AI Responses to Patient Concerns About Axial Spondyloarthritis: Delphi-Based Evaluation. JMIR AI 5:e79153. https://doi.org/10.2196/79153 Kara M, Ozduran E, Kara MM et al (2025) Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about Ankylosing spondylitis. PLoS ONE 20:e0326351. https://doi.org/10.1371/journal.pone.0326351 Altunel Kılınç E, Çabuk Çelik N (2025) Evaluation of artificial ıntelligence use in ankylosing spondylitis with ChatGPT-4: patient and physician perspectives. Clin Rheumatol 44:4015–4023. https://doi.org/10.1007/s10067-025-07648-w Çabuk Çelik N, Altunel Kılınç E (2026) AI-generated patient education for ankylosing spondylitis: a comparative study of readability and quality. Clin Rheumatol 45:2003–2008. https://doi.org/10.1007/s10067-025-07771-8 Sari F, Çelik Z, Mirza Y (2026) ChatGPT-4 vs. DeepSeek-V3: a comparative study of response quality, reliability, usefulness, and readability for exercise and rehabilitation strategies in patients with ankylosing spondylitis. Clin Rheumatol 45:187–195. https://doi.org/10.1007/s10067-025-07789-y Çelik NÇ, Kılınç EA (2025) Assessment of ChatGPT’s adherence to EULAR diagnostic criteria and therapeutic protocols for rheumatoid arthritis at two distinct time points, 14 days apart, utilizing binary and multiple-choice inquiries. Clin Rheumatol 44:2233–2239. https://doi.org/10.1007/s10067-025-07417-9 Usen A, Kuculmez O (2025) Evaluation of the Performance of Large Language Models in the Management of Axial Spondyloarthropathy: Analysis of EULAR 2022 Recommendations. Diagnostics 15:1455. https://doi.org/10.3390/diagnostics15121455 Meral HB, Kolak E (2026) Evaluation of ChatGPT-4o and Gemini for gout management: a comparative analysis based on EULAR guidelines. Sci Rep 16:4831. https://doi.org/10.1038/s41598-026-35166-5 Kayacan Erdoğan E, Babaoğlu H (2024) Clinical Reasoning and Knowledge Assessment of Rheumatology Residents Compared to AI Models: A Pilot Study. J Clin Med 13:7405. https://doi.org/10.3390/jcm13237405 Miao BY, Binvignat M, Garcia-Agundez A et al (2025) JAMIA Open 8:ooaf132. https://doi.org/10.1093/jamiaopen/ooaf132 . Extracting TNFi switching reasons and trajectories from real-world data using large language models Polyzou M, Baraliakos X (2026) Artificial Intelligence (AI) in rheumatology: a comparative evaluation of the ChatGPT and DeepSeek application. BMC Rheumatol 10:13. https://doi.org/10.1186/s41927-026-00618-y Gorenshtein A, Omar M, Glicksberg BS et al AI Agents in Clinical Medicine: A Systematic Review 2025:2025.08.22.25334232. https://doi.org/10.1101/2025.08.22.25334232 Kresevic S, Giuffrè M, Ajcevic M et al (2024) Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework. Npj Digit Med 7:102. https://doi.org/10.1038/s41746-024-01091-y Zakka C, Shad R, Chaurasia A et al (2024) Almanac — Retrieval-Augmented Language Models for Clinical Medicine. NEJM AI 1:AIoa2300068. https://doi.org/10.1056/AIoa2300068 Liu S, McCoy AB, Wright A (2025) Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. J Am Med Inf Assoc 32:605–615. https://doi.org/10.1093/jamia/ocaf008 Omar M, Sorin V, Collins JD et al (2025) Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support. Commun Med (Lond) 5:330. https://doi.org/10.1038/s43856-025-01021-3 Klang E, Glicksberg BS, Gorenshtein A et al Clinical Agents Don’t Care 2025:2025.10.17.25338226. https://doi.org/10.1101/2025.10.17.25338226 Omar M, Gorenshtein A, Agbareia R et al Impact of Patient Communication Style on Agentic AI-Generated Clinical Advice in E-Medicine 2025:2025.12.02.25341475. https://doi.org/10.64898/2025.12.02.25341475 Table 2 Table 2 is available in the Supplementary Files section. Additional Declarations The authors declare no competing interests. Supplementary Files APPENDIXFinalV.pdf APPENDIX Table2.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9529682","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Systematic Review","associatedPublications":[],"authors":[{"id":629652398,"identity":"37fbd291-11b2-47cd-ac95-c0221873db1b","order_by":0,"name":"Yosef Adiniaev","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA80lEQVRIiWNgGAWjYLCCCgZmHnlm/ocPgGwePqK0nGFgljFs72E2AGlhI1aLDcOZM2wSIA5BLQbH2x8+OFBjzcM4I/dY5dccOxk2BuaHj27g03LmjLHBgWPpPOwSeWm3ZbclAx3GZmycg0/LjRw26Q9sh4G2JJjdltzGDNTCwyaNX0v68x8H/h3mYbiRYFYsua2eGC0JZgwH24BazpwxY/y47TBhLZJAv0gc7EvnMWxvS5Zm3Hach42ZgF/4gCH24cA3a3t5ZuaDH39uq7bnZ29++BifFoUDSBxmHjCJRzkIyDcgcRh/EFA9CkbBKBgFIxMAACecS0o8LmguAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0009-0002-7678-633X","institution":"Faculty of Medicine, University of Debrecen, Debrecen, Hungary; BRIDGE GenAI Lab, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA","correspondingAuthor":true,"prefix":"","firstName":"Yosef","middleName":"","lastName":"Adiniaev","suffix":""},{"id":629652399,"identity":"efcda320-4e9d-4527-a7d2-ac360386a8b4","order_by":1,"name":"Mahmud Omar","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5klEQVRIiWNgGAWjYFACHiBmAzH4Pxz4AGKzE6+FwfDhDBCbmQQtxsYgNgMhLbrtZ49u5imzieaffSBN2ubXNnk+ZgbGDx9zcGsxO5OXdpvnXFrujHMJx6Rz+24btjEzMEvO3IZHy4Ecs9u8bYdzG84wtknn9txmBGphY+bFp+X8G4iW+WeY2aQte27bE9ZyA2rLhjNszMYMP24nEqHljdnNOUC/bDzDw/iwt+F2chszYzN+v5zPAeoqs8mdd4aH4cCPP7dt57c3H/zwEY8WVMDYBiYbiFUPAn9IUTwKRsEoGAUjBQAA+6dUKUzbOUYAAAAASUVORK5CYII=","orcid":"https://orcid.org/0009-0001-0438-0827","institution":"BRIDGE GenAI Lab, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA","correspondingAuthor":true,"prefix":"","firstName":"Mahmud","middleName":"","lastName":"Omar","suffix":""},{"id":629652400,"identity":"d13a4822-c3a5-4475-9c1f-6bd4b99c41a8","order_by":2,"name":"Tohar M Timor","email":"","orcid":"https://orcid.org/0009-0000-9842-1636","institution":"Faculty of Medicine, University of Debrecen, Debrecen, Hungary","correspondingAuthor":false,"prefix":"","firstName":"Tohar","middleName":"M","lastName":"Timor","suffix":""},{"id":629652401,"identity":"36fe89a4-6cce-474f-80cd-666b21e15a93","order_by":3,"name":"Yiftach Barash","email":"","orcid":"https://orcid.org/0000-0002-7242-1328","institution":"BRIDGE GenAI Lab, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA; Department of Radiology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA","correspondingAuthor":false,"prefix":"","firstName":"Yiftach","middleName":"","lastName":"Barash","suffix":""},{"id":629652402,"identity":"282f2f21-696b-486d-aaa8-8f19197d155b","order_by":4,"name":"Olga R Brook","email":"","orcid":"https://orcid.org/0000-0002-7074-8909","institution":"Department of Radiology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA","correspondingAuthor":false,"prefix":"","firstName":"Olga","middleName":"R","lastName":"Brook","suffix":""},{"id":629652403,"identity":"45d58b9f-b413-4910-8fe4-2ce32274bba1","order_by":5,"name":"Mohammad E Naffaa","email":"","orcid":"https://orcid.org/0000-0001-9777-2864","institution":"Rheumatology Unit, Galilee Medical Center, Nahariya, Israel; Azrieli Faculty of Medicine, Bar-Ilan University, Safed, Israel","correspondingAuthor":false,"prefix":"","firstName":"Mohammad","middleName":"E","lastName":"Naffaa","suffix":""},{"id":629652404,"identity":"20dbe11a-4ece-4580-a258-04e9af02a52f","order_by":6,"name":"Alon Gorenshtein","email":"","orcid":"https://orcid.org/0009-0000-7542-8608","institution":"BRIDGE GenAI Lab, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA; Department of Neurology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA","correspondingAuthor":false,"prefix":"","firstName":"Alon","middleName":"","lastName":"Gorenshtein","suffix":""},{"id":629652405,"identity":"f3cc1fcc-9fc7-4766-b569-7e948be95a50","order_by":7,"name":"Eyal Klang","email":"","orcid":"https://orcid.org/0000-0002-4567-3108","institution":"BRIDGE GenAI Lab, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA; Department of Radiology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA","correspondingAuthor":false,"prefix":"","firstName":"Eyal","middleName":"","lastName":"Klang","suffix":""}],"badges":[],"createdAt":"2026-04-26 06:10:32","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-9529682/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9529682/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108011344,"identity":"dfe7e548-b1c7-4738-ba59-f1f11a43c760","added_by":"auto","created_at":"2026-04-28 13:14:45","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":581434,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003ePRISMA 2020 flow diagram of the study selection process.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-9529682/v1/625c7f2a80ef22e3ca446965.png"},{"id":108011416,"identity":"bf67463e-ede0-40b2-b86a-c1d481d2be0d","added_by":"auto","created_at":"2026-04-28 13:14:49","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":97905,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eResearch landscape: LLMs evaluated across inflammatory arthritis conditions.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"Figure1.png","url":"https://assets-eu.researchsquare.com/files/rs-9529682/v1/0f49d30dfbc2edf8472bbae4.png"},{"id":108011552,"identity":"323b8dc0-39a5-4f72-b155-d24973897905","added_by":"auto","created_at":"2026-04-28 13:15:04","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":125760,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eLLM accuracy and guideline concordance across inflammatory arthritis studies.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-9529682/v1/48f8c3ef731fc020299fb9bf.png"},{"id":108181633,"identity":"d8a263c5-b321-4def-9f56-5d5f1cab7ae4","added_by":"auto","created_at":"2026-04-30 08:58:48","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1143947,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9529682/v1/ebbe0b63-a179-4481-b209-b0b0e81bc1a0.pdf"},{"id":108011655,"identity":"6bf22cd3-3e26-48c9-8631-ea605df68d5c","added_by":"auto","created_at":"2026-04-28 13:15:19","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":502902,"visible":true,"origin":"","legend":"\u003cp\u003eAPPENDIX\u003c/p\u003e","description":"","filename":"APPENDIXFinalV.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9529682/v1/5f440945b6a822e8f7829b2f.pdf"},{"id":108011408,"identity":"03a45f7c-b32b-4215-b18f-c789b2ce8eae","added_by":"auto","created_at":"2026-04-28 13:14:48","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":16513,"visible":true,"origin":"","legend":"","description":"","filename":"Table2.docx","url":"https://assets-eu.researchsquare.com/files/rs-9529682/v1/5e3c6d68f324d8aa2cd9e2da.docx"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003e\u003cstrong\u003eLarge Language Models in Inflammatory Arthritis: A Systematic Review Across Clinical Tasks\u003c/strong\u003e\u003c/p\u003e","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eInflammatory arthritis, defined here as chronic immune-mediated joint diseases characterized by synovial inflammation and progressive structural damage, places a heavy burden on patients and healthcare systems [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. Rheumatoid arthritis (RA) alone affects approximately 18\u0026nbsp;million people worldwide [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eManagement of these conditions is complex and involves integration of multiple data sources [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. Clinicians must interpret labs, imaging, and clinical criteria, all while keeping up with frequently updated treatment guidelines from organizations like the American College of Rheumatology (ACR) and the European Alliance of Associations for Rheumatology (EULAR) [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. Early initiation of treatment is critical. In RA and psoriatic arthritis (PsA), delays in disease-modifying antirheumatic drugs (DMARDs) can lead to permanent joint damage [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. The same is true for biologic therapies in ankylosing spondylitis (AS) and urate-lowering therapy in gout [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eLarge language models (LLMs) such as ChatGPT have recently drawn attention as potential tools for clinical support. These AI systems can generate human-like text, making them appealing for tasks like answering patient questions, summarizing guidelines, and supporting diagnostic reasoning [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. Recent studies have evaluated how well LLMs perform on tasks relevant to inflammatory arthritis.\u003c/p\u003e \u003cp\u003eHowever, LLMs were not built for clinical decision-making. Although recent developments have enabled access to external tools and real-time data sources [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e], LLMs still do not consistently follow current guidelines, and sometimes produce confident but incorrect answers, a phenomenon known as hallucination [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. Their performance also varies considerably depending on model type and clinical task.\u003c/p\u003e \u003cp\u003ePrior reviews have examined the role of natural language processing and LLMs in rheumatology broadly [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e], while others have focused specifically on osteoarthritis [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. However, no systematic review has specifically examined LLM use in inflammatory arthritis.\u003c/p\u003e \u003cp\u003eWe focused on RA, PsA, AS/axSpA, and gout because these are the most prevalent inflammatory arthritides with established ACR/EULAR diagnostic and treatment guidelines, and because a preliminary literature survey confirmed that published LLM evaluation studies in inflammatory arthritis were concentrated on these four conditions [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. Other inflammatory joint diseases, including calcium pyrophosphate deposition disease, reactive arthritis, and autoinflammatory syndromes such as familial Mediterranean fever and Behcet's disease, were outside the predefined scope of this review but represent important targets for future evaluation.\u003c/p\u003e \u003cp\u003eThis systematic review aims to (1) evaluate the accuracy and clinical relevance of LLM-generated outputs across diagnostic, therapeutic, and educational tasks in RA, PsA, AS/axSpA, and gout; (2) compare performance across LLM models, diseases, and task types; and (3) identify methodological gaps and safety concerns relevant to clinical implementation.\u003c/p\u003e"},{"header":"2. Methods","content":"\u003cp\u003e \u003cb\u003eProtocol and registration\u003c/b\u003e \u003c/p\u003e \u003cp\u003eWe conducted this review following the PRISMA 2020 guidelines [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. The protocol was registered on PROSPERO (CRD420261359100). No deviations from the registered protocol were identified.\u003c/p\u003e \u003cp\u003e \u003cb\u003eData sources and searches\u003c/b\u003e \u003c/p\u003e \u003cp\u003eWe searched three databases: PubMed, Scopus, and PubMed Central (PMC). The search covered studies published from January 1, 2022, to April 3, 2026. We chose 2022 as the start date due to the release of ChatGPT-3.5 and the surge of LLM into clinical medicine by that year.\u003c/p\u003e \u003cp\u003eThe search combined two groups of terms. The first group captured LLM-related concepts: \"large language model,\" \"LLM,\" \"ChatGPT,\" \"GPT-4,\" \"GPT-3,\" \"GPT-4o,\" \"generative AI,\" and \"generative artificial intelligence.\" The second group captured inflammatory arthritis conditions: \"rheumatoid arthritis,\" \"psoriatic arthritis,\" \"ankylosing spondylitis,\" \"axial spondyloarthritis,\" \"gout,\" \"gouty arthritis,\" \"spondyloarthritis,\" and \"inflammatory arthritis.\"\u003c/p\u003e \u003cp\u003eFull search strings for each database are provided in the \u003cb\u003eSupplementary Materials\u003c/b\u003e. Only English language publications were included. We also performed forward and backward citation searching to identify additional relevant studies.\u003c/p\u003e \u003cp\u003e \u003cb\u003eStudy screening\u003c/b\u003e \u003c/p\u003e \u003cp\u003eWe included studies that evaluated LLMs in the context of inflammatory arthritis. This covered any generative AI system capable of producing human-like text, including ChatGPT, GPT-4, GPT-3.5, Google Gemini, Anthropic Claude, and DeepSeek, among others. The diseases of interest were RA, PsA, AS/axial spondyloarthritis (axSpA), and gout.\u003c/p\u003e \u003cp\u003e We accepted studies that assessed LLMs for clinical diagnosis, treatment recommendations, patient education, guideline adherence, or clinical decision support. Each study needed to report some form of performance data, whether quantitative (e.g., accuracy, concordance, DISCERN scores) or through systematic expert evaluation. Only peer-reviewed journal articles were included.\u003c/p\u003e \u003cp\u003eWe excluded studies that focused only on osteoarthritis, as this area has been covered by recent reviews [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. We also excluded studies that evaluated traditional machine learning, deep learning, or NLP methods without an LLM component. Studies using AI only for medical imaging without involving an LLM were excluded as well. Editorials, commentaries, letters without original data, conference abstracts, preprints, and study protocols were also not eligible.\u003c/p\u003e \u003cp\u003eTwo reviewers (Y.A., A.G.) independently screened all titles and abstracts using Rayyan [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Articles flagged for potential inclusion by either reviewer moved to full-text review. Any disagreements were resolved through third reviewer (E.K.).\u003c/p\u003e \u003cp\u003e \u003cb\u003eData extraction\u003c/b\u003e \u003c/p\u003e \u003cp\u003eWe created a standardized extraction form and tested it on three studies before applying it to all included articles. One reviewer (Y.A.) extracted the data, and a second reviewer (A.G.) verified all entries. Disagreements were resolved by discussion.\u003c/p\u003e \u003cp\u003eFor each study, we collected the first author, year, journal, and study design. We also recorded the disease(s) studied, the LLM(s) used (including model version and access date when available), and the type of clinical task assessed. To capture performance, we extracted the main outcome, the comparator, the sample size, key results, and reported limitations.\u003c/p\u003e \u003cp\u003e \u003cb\u003eRisk of bias assessment\u003c/b\u003e \u003c/p\u003e \u003cp\u003eWe adapted the QUADAS-2 framework with AI-specific modifications to assess risk of bias in the included studies. Given the absence of a validated risk-of-bias tool for LLM evaluation studies, QUADAS-2 was selected as the closest established framework and adapted to reflect key methodological features specific to AI-based evaluations. We modified it by redefining the index test domain to address LLM-specific factors (model version, access date, prompt design, and reproducibility) and by adjusting the reference standard domain to account for comparators used in LLM studies (expert panels, clinical guidelines, or validated scoring tools). The adapted instrument evaluates four domains: (1) data selection, (2) the index test (the LLM), (3) the reference standard, and (4) flow and timing. Each domain was rated as low risk, high risk, or unclear. We also assessed applicability concerns for the first three domains. The complete adapted checklist is provided in Supplementary Table S1.\u003c/p\u003e \u003cp\u003eWe considered a study to be at high risk of bias if it used a small or unrepresentative set of questions, relied on a single LLM version without reporting the access date or prompt design, lacked comparison with clinical experts or validated tools, used a single evaluator without checking inter-rater reliability, or selectively reported outcomes. One reviewer (Y.A.) assessed each study, and a second reviewer (A.G.) verified the ratings. Disagreements were resolved through discussion.\u003c/p\u003e \u003cp\u003e \u003cb\u003eSynthesis of findings\u003c/b\u003e \u003c/p\u003e \u003cp\u003eBecause the included studies varied widely in design, LLM models tested, clinical tasks, and reporting methods, a quantitative meta-analysis was not feasible. Instead, we performed a narrative synthesis. We first grouped studies by disease type: RA, PsA, AS, and gout. We then examined patterns across application domains, including patient education, guideline adherence, clinical reasoning, and clinical data extraction. Finally, we compared findings across LLM models, comparators, and outcome measures reported.\u003c/p\u003e"},{"header":"3. Results","content":"\u003cp\u003e \u003cb\u003eStudy selection\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe search identified 159 records: 53 from PubMed, 99 from Scopus, and 7 from PubMed Central. After removing 46 duplicates, 113 records were screened by title and abstract. Eighty-five did not meet the inclusion criteria. Full-text assessment of the remaining 28 articles led to the exclusion of 10 that did not specifically evaluate LLMs in the context of inflammatory arthritis, lacked performance data, or focused on diseases outside the scope of this review. Eighteen studies were included in the final synthesis (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cb\u003eStudy characteristics\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe 18 included studies were published between 2024 and 2026, with the majority (11 studies) appearing in 2025. Sixteen studies used cross-sectional designs, one was a retrospective observational study, and one was a pilot study. AS/axSpA was the most frequently studied condition (7 studies), followed by RA (4), PsA (2), gout (1), JIA (1), and three studies covering multiple diseases.\u003c/p\u003e \u003cp\u003eChatGPT and its variants (GPT-3.5, GPT-4, GPT-4o) were evaluated in 16 of 18 studies. Other models tested included Google Gemini (7 studies), DeepSeek (5), Claude (1), and Perplexity (2) (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCharacteristics and key findings of included studies, organized by disease.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFirst Author\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eYear\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLLM(s)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eApplication\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eKey Finding\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eSample\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"6\" nameend=\"c6\" namest=\"c1\"\u003e \u003cp\u003eRheumatoid Arthritis (n\u0026thinsp;=\u0026thinsp;4)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCoskun BN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2024\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGPT-3.5, GPT-4, BARD, Bing\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePatient info (MTX)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eGPT-4: 100% accuracy; BARD/Bing: 60.87%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e23 Qs; 2 rheumatologists\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCabuk Celik N\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2025\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChatGPT-4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eEULAR adherence\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e96% binary correct; self-corrected errors at day 14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e100 Qs at 2 time-points\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBerghea F\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2025\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChatGPT 4.5,\u003c/p\u003e \u003cp\u003eGemini, Qwen,\u003c/p\u003e \u003cp\u003eDeepSeek,\u003c/p\u003e \u003cp\u003ePerplexity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eRisk appetite\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eChatGPT least risk-averse; method-dependent profiles\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eRA scenarios; 5 GenAIs\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSaji JG\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2025\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChatGPT,\u003c/p\u003e \u003cp\u003eGemini, Copilot,\u003c/p\u003e \u003cp\u003eApple\u003c/p\u003e \u003cp\u003eIntelligence, Meta\u003c/p\u003e \u003cp\u003eAI\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePatient education\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eReadability-reliability trade-off across platforms\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e5 conditions x 5 platforms\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"6\" nameend=\"c6\" namest=\"c1\"\u003e \u003cp\u003ePsoriatic Arthritis (n\u0026thinsp;=\u0026thinsp;2)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eForte G\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2025\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChatGPT-4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eExpert comparison\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003ePatients preferred ChatGPT\u003c/p\u003e \u003cp\u003e(49.2%) over experts (34.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e32 Qs; 12 experts; 67 patients\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAtilan AU\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2026\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e7 LLMs\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePatient ed. (Turkish)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eClaude/Gemini most reliable;\u003c/p\u003e \u003cp\u003eChatGPT most readable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e7 LLMs; DISCERN \u0026amp; readability\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"6\" nameend=\"c6\" namest=\"c1\"\u003e \u003cp\u003eAnkylosing Spondylitis / axSpA (n\u0026thinsp;=\u0026thinsp;7)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRen Y\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2025\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChatGPT-4o, 3.5,\u003c/p\u003e \u003cp\u003eGemini, Kimi\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePatient education\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eChatGPT-4o/Kimi outperformed guidelines; high acceptance\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e182 volunteers; 15 Qs\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eUsen A\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2025\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChatGPT-3.5, 4o, Gemini 2.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eASAS-EULAR\u003c/p\u003e \u003cp\u003eadherence\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eGPT-3.5 recommended\u003c/p\u003e \u003cp\u003econtraindicated therapies\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e15 guideline Qs\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBai J\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2026\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGPT-4.0,\u003c/p\u003e \u003cp\u003eDeepSeek R1,\u003c/p\u003e \u003cp\u003eHunyuan, Kimi,\u003c/p\u003e \u003cp\u003eWenxin\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eHealth guidance\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eGPT-4.0: 94% guideline agreement\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e84 patients, 26 MDs\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eKara M\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2025\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChatGPT-4o,\u003c/p\u003e \u003cp\u003eGemini,\u003c/p\u003e \u003cp\u003ePerplexity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePatient info quality\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003ePerplexity highest quality; all above readability threshold\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e25 Google Trends Qs\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAltunel Kilinc E\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2025\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChatGPT-4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePatient/physician ed.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eFAQs 5.32/6; complex cases 4.24/6 (p\u0026thinsp;=\u0026thinsp;0.044)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e75 Qs in 3 groups\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCabuk Celik\u003c/p\u003e \u003cp\u003eN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2025\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChatGPT-4o, 3.5,\u003c/p\u003e \u003cp\u003eDeepSeek R1/V3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePatient ed. quality\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eChatGPT-4o highest DISCERN\u003c/p\u003e \u003cp\u003e(72.38); all Flesch-Kincaid Grade Level (FKGL)\u0026thinsp;\u0026gt;\u0026thinsp;15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e10 Qs; 4 LLMs\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSari F\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2025\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChatGPT-4,\u003c/p\u003e \u003cp\u003eDeepSeek-V3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eExercise/rehab\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eDeepSeek-V3\u0026thinsp;\u0026gt;\u0026thinsp;ChatGPT-4\u003c/p\u003e \u003cp\u003e(p\u0026thinsp;\u0026lt;\u0026thinsp;0.001); both very difficult\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e50 Qs; 3 physiotherapists\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"6\" nameend=\"c6\" namest=\"c1\"\u003e \u003cp\u003eGout (n\u0026thinsp;=\u0026thinsp;1)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMeral HB\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2026\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChatGPT-4o, Gemini 2.0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eEULAR adherence\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eChatGPT-4o: 76% vs Gemini\u003c/p\u003e \u003cp\u003e48%; Gemini 8% contradictory\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e25 EULAR Qs\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"6\" nameend=\"c6\" namest=\"c1\"\u003e \u003cp\u003eJuvenile Idiopathic Arthritis (n\u0026thinsp;=\u0026thinsp;1)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLa Bella S\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2025\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChatGPT 4o\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eGuideline adherence\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e52\u0026ndash;84% adherent to ACR; regional differences\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e10 PICOs x 5 countries\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"6\" nameend=\"c6\" namest=\"c1\"\u003e \u003cp\u003eMultiple / Mixed (n\u0026thinsp;=\u0026thinsp;3)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eKayacan-Erdogan E\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2024\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGPT-4\u003c/p\u003e \u003cp\u003e(trained/untrained)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eClinical reasoning\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eTrained GPT-4 scored higher than residents at all stages\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e10 cases; 10 residents\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMiao BY\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2025\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGPT-4\u0026thinsp;+\u0026thinsp;8 opensource\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eClinical NLP (EHR)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eGPT-4 F1 0.75\u0026ndash;0.83; efficacy loss\u003c/p\u003e \u003cp\u003e56.9%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e9,187 patients\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePolyzou M\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2026\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eGPT-5,\u003c/p\u003e \u003cp\u003eDeepSeek-R1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eDx/Tx vs real data\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eModerate agreement with clinical data from 116 patients\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e116 patients\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eRisk of bias\u003c/b\u003e \u003c/p\u003e \u003cp\u003eRisk of bias was assessed using an adapted QUADAS-2 framework across four domains. Of 126 individual domain assessments (18 studies across 7 domains including applicability), 105 (83.3%) were rated as low risk, 14 (11.1%) as high risk, and 7 (5.6%) as unclear (Figure S1). Most studies evaluated LLMs using synthetic clinical questions or multiple-choice formats rather than real patient data, which may inflate the proportion of low-risk ratings; the adapted QUADAS-2 ratings reflect the internal methodological quality of each study within its chosen design.\u003c/p\u003e \u003cp\u003eIn the Data Selection domain, 14 of 18 studies (77.8%) were rated as low risk, three as high risk because they relied on small question sets or non-clinically validated sources, and one as unclear. In the Index Test domain, 14 of 18 studies (77.8%) were low risk, three were high risk, and one was unclear, primarily due to evaluation of a single LLM model without reporting access dates. Reference Standard assessments were low risk in 16 of 18 studies (88.9%), with one rated high risk due to weak inter-rater agreement (kappa values mostly below 0.40) [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e] and one rated unclear. Flow and Timing was low risk in 17 of 18 studies (94.4%), with one rated unclear.\u003c/p\u003e \u003cp\u003eApplicability concerns were mostly low. The main exceptions involved studies conducted exclusively in Chinese or Turkish, where LLM performance may not generalize to other languages. The study by Berghea et al. [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e] had high applicability concerns across multiple domains because its focus on AI risk appetite does not align with standard clinical evaluation frameworks.\u003c/p\u003e \u003cp\u003e \u003cb\u003eSynthesis of findings\u003c/b\u003e \u003c/p\u003e \u003cp\u003eFindings were organized into four predefined application domains based on common evaluation categories reported in prior LLM medical literature [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]: patient education and information quality (11 studies), guideline adherence and clinical decision support (6 studies), clinical reasoning and real-world applications (3 studies), and other applications (1 study). Three studies were classified under more than one domain based on their stated objectives; the totals therefore exceed 18. The included studies used heterogeneous evaluation metrics, including percentage correct, Likert-scale accuracy scores, DISCERN scores, F1 scores, and guideline concordance ratings, which limits direct numerical comparison across studies. A summary of outcomes by application domain is presented in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePatient education and information quality\u003c/b\u003e \u003c/p\u003e \u003cp\u003eEleven studies evaluated LLM-generated content for patient education across RA, PsA, AS, JIA, and gout [\u003cspan additionalcitationids=\"CR20 CR21 CR22 CR23 CR24 CR25 CR26 CR27 CR28\" citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e]. Of these, 8 of 11 reported moderate to high accuracy for at least one LLM, as defined by study-specific scoring systems. Among the 7 studies that used DISCERN or modified DISCERN, scores ranged from 38.5 to 72.38. All 10 studies that assessed readability found outputs above the recommended sixth-grade reading level. Reported accuracy ranged from 60.87% (BARD and Bing in Coskun et al. [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]) to 100% (GPT-4 in the same study).\u003c/p\u003e \u003cp\u003eIn the largest patient education study, Forte et al. [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e] collected 32 questions from 76 PsA patients and compared ChatGPT-4 responses against answers written by 12 specialist rheumatologists. Fourteen blinded experts found no significant difference in accuracy between the two sources. In a separate arm, 67 patients reviewed blinded answer pairs and selected the ChatGPT response 49.2% of the time, compared with 34.4% for the expert answer. One domain where ChatGPT scored significantly lower was pregnancy and fertility.\u003c/p\u003e \u003cp\u003eAtilan et al. [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e] compared seven LLMs generating Turkish patient brochures for PsA and psoriasis. Claude and Gemini achieved the highest DISCERN reliability scores (60.5 and 57.5), while ChatGPT produced the most readable output (Atessman score 80.2). Mistral scored lowest across all domains (total DISCERN 38.5). Saji et al. [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e] observed a similar pattern across five AI platforms: Google Gemini was the most readable but scored lowest on reliability, while Microsoft Copilot was the most reliable but the least readable.\u003c/p\u003e \u003cp\u003eAll studies that assessed readability reported levels above recommended thresholds. Cabuk Celik et al. [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e] found all four LLMs tested for AS education produced text requiring university-level reading (FKGL above 15). Sari et al. [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e] reported Flesch Reading Ease scores of 28.77 to 30.4 for exercise information. La Bella et al. [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e] reported a median Flesch Reading Ease Score (FRES) of 30 for JIA content across five countries. One exception was Ren et al. [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e], who reported high patient acceptance of LLM-generated AS education in Chinese, with ChatGPT-4o and Kimi outperforming traditional guideline text.\u003c/p\u003e \u003cp\u003e \u003cb\u003eGuideline adherence and clinical decision support\u003c/b\u003e \u003c/p\u003e \u003cp\u003eSix studies tested LLM concordance with established treatment guidelines [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan additionalcitationids=\"CR31\" citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]. Of these, 5 of 6 reported concordance above 75% for at least one GPT-4 variant, while concordance for non-GPT models was lower and more variable. Two of 6 studies documented clinically inappropriate\u003c/p\u003e \u003cp\u003erecommendations produced by at least one model. Concordance rates ranged from 48% (Gemini for gout [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]) to 96% (ChatGPT-v4 for binary EULAR RA questions [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]).\u003c/p\u003e \u003cp\u003eIn 5 of 6 studies, GPT-4 variants achieved the highest concordance scores. Bai et al. [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e] reported 94% overall agreement between GPT-4.0 and axSpA guidelines. Cabuk Celik et al. [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e] found 96% accuracy on binary EULAR RA questions, and when the same questions were repeated 14 days later, the model corrected four of seven previous errors. For gout, Meral et al. [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e] found ChatGPT-4o aligned with EULAR guidelines in 76% of responses, compared with 48% for Gemini. Gemini contradicted the guidelines in 8% of cases, while ChatGPT-4o had none (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e In at least one study, older models generated recommendations inconsistent with current treatment guidelines. Usen et al. [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e] found that ChatGPT-3.5 recommended IL-23 inhibitors for axSpA, which are not part of the approved treatment algorithm, and suggested DMARD combinations inconsistent with ASAS-EULAR guidelines.\u003c/p\u003e \u003cp\u003e Lower accuracy scores were reported for case-based clinical scenarios compared with factual or guideline-based questions in studies that evaluated both formats. Altunel Kilinc et al. [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e] tested ChatGPT-4 on three categories of AS questions: frequently asked patient questions (5.32/6), open-ended guideline questions (5.36/6), and case-based clinical scenarios (4.24/6, p\u0026thinsp;=\u0026thinsp;0.044).\u003c/p\u003e \u003cp\u003e \u003cb\u003eClinical reasoning and real-world applications\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThree studies evaluated LLMs beyond structured question-answering [\u003cspan additionalcitationids=\"CR34\" citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e]. In a pilot study, Kayacan Erdogan et al. [\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e] compared a trained GPT-4 against 10 junior rheumatology residents on 10 inflammatory arthritis cases. The trained model scored higher than the residents at all stages of clinical reasoning. In a post-study survey, 60% of residents indicated they would welcome AI as a supportive tool.\u003c/p\u003e \u003cp\u003eMiao et al. [\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e] applied GPT-4 and eight open-source LLMs to extract reasons for TNFi switching from the electronic health records of 9,187 patients at a single academic center. GPT-4 achieved micro-F1 scores of 0.75 (stopped drug), 0.80 (started drug), and 0.83 (switching reason). The most frequently identified reason was lack of efficacy (56.9%), followed by adverse events (13.5%) and insurance or cost issues (10.8%).\u003c/p\u003e \u003cp\u003ePolyzou et al. [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e] compared GPT-5 and DeepSeek-R1 against clinical findings from 116 patients with axSpA and PsA at a tertiary center in Germany. Cohen and Fleiss Kappa coefficients showed no to low agreement between LLM predictions and clinical data.\u003c/p\u003e \u003cp\u003e \u003cb\u003eHeterogeneity across studies\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThere was substantial heterogeneity across the included studies. Models ranged from GPT-3.5 (released 2022) to GPT-5 (2026). Prompting strategies varied from standardized zero-shot prompts to trained models. Languages included English, Chinese, and Turkish. Evaluation metrics were not standardized. Sample sizes ranged from 10 questions to 9,187 patient records. Given this heterogeneity, a formal meta-analysis was not feasible. Across studies, LLMs achieved higher accuracy on structured, guideline-based tasks and lower performance on case-based or real-world clinical scenarios. This pattern was observed across diseases and model types, suggesting a general limitation in clinical reasoning rather than disease-specific variability.\u003c/p\u003e "},{"header":"4. Discussion","content":"\u003cp\u003eThis review identified 18 studies evaluating LLMs across RA, PsA, AS/axSpA, gout, and JIA. LLMs achieved concordance rates of 76\u0026ndash;96% with EULAR and ASAS-EULAR guidelines on structured questions [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e], and generated patient education content that patients with PsA preferred over specialist-written answers in one blinded comparison [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. However, when tested on real clinical data from 116 patients with axSpA and PsA, agreement dropped to low-moderate levels [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e].\u003c/p\u003e \u003cp\u003e \u003cb\u003eDisease-specific performance in rheumatology\u003c/b\u003e \u003c/p\u003e \u003cp\u003eLLM performance differed across inflammatory arthritis subtypes. For RA, LLMs performed well on methotrexate-related questions and binary EULAR criteria [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e], but showed inconsistent risk tolerance across treatment scenarios, with variation between models and assessment methods [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. For AS/axSpA, where seven studies were available, LLMs handled structured questions adequately (Group 1: 5.32/6, Group 2: 5.36/6) but achieved lower scores on case-based clinical scenarios (4.24/6, p\u0026thinsp;=\u0026thinsp;0.044) [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. In axSpA, treatment decisions require sequencing between NSAIDs, TNF inhibitors, and IL-17 inhibitors, and this difference between structured and case-based performance may limit clinical applicability. For gout, ChatGPT-4o aligned with EULAR recommendations in 76% of responses, but Gemini contradicted them in 8% of cases [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e], a finding relevant to clinical practice given the role of comorbidity assessment in urate-lowering therapy decisions. For PsA, the only study with real patient involvement showed that while patients preferred AI-generated content, ChatGPT performed poorly on pregnancy and fertility counselling [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. These results indicate that LLM evaluation in rheumatology should include clinically relevant decision-making scenarios such as DMARD escalation, biologic switching, treatment in pregnancy, and management of comorbidities.\u003c/p\u003e \u003cp\u003e \u003cb\u003eBeyond standalone LLMs: AI agents and retrieval-augmented generation\u003c/b\u003e \u003c/p\u003e \u003cp\u003eIn this review, LLMs performed well on guideline-based MCQs but lacked the clinical reasoning needed for real patient management, which requires external data and tool integration [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. All 18 included studies evaluated standalone LLMs without access to external data sources, real-time guidelines, or iterative reasoning capabilities. Recent studies report that AI agent systems, which extend LLMs with tool integration, iterative planning, and retrieval-augmented generation (RAG), outperform base LLMs on clinical tasks [\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e]. Independent evaluations report similar results: a RAG-augmented system improved hepatology guideline accuracy from 43% to 99% compared with standalone GPT-4 [\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e], and retrieval-augmented models outperformed base LLMs across nine medical specialties in a blinded clinician evaluation [\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e]. A recent meta-analysis of 20 studies confirmed that RAG significantly improves LLM performance in biomedical applications (pooled OR 1.35, 95% CI 1.19\u0026ndash;1.53) [\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e]. In rheumatology, where treatment guidelines are updated frequently (e.g., 2022 ASAS-EULAR update [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], 2021 ACR RA guideline [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]) and where biologic selection depends on real-time laboratory and imaging data, an agent-based approach grounded in current guidelines through RAG could address several limitations observed in this review. Tool-augmented LLMs that retrieve current ACR/EULAR recommendations before generating responses could reduce the risk of outdated or contraindicated treatment suggestions, such as the IL-23 inhibitor error observed by Usen et al. [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]. Knowledge-grounded systems have shown improved accuracy and reduced hallucination in other clinical domains [\u003cspan additionalcitationids=\"CR38\" citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e]. Even when augmented with external sources, LLMs may still generate or misinterpret clinical information, and human verification remains essential [\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e].\u003c/p\u003e \u003cp\u003e \u003cb\u003eSafety and reliability\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe safety profile of LLMs in rheumatology raises several concerns beyond simple inaccuracy. LLMs are vulnerable to adversarial hallucination attacks: when presented with fabricated clinical details, models elaborated on false information in up to 83% of cases [\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e]. This is particularly relevant in rheumatology, where clinical notes may contain ambiguous findings, and an LLM that confidently elaborates on incorrect serological results or imaging interpretations could mislead clinicians. Additionally, LLM outputs are sensitive to how clinical questions are framed. Recent work has shown that patient communication style alone, such as urgent or demanding language, can shift AI-generated triage decisions and sick-leave recommendations [\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e, \u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e]. In rheumatology practice, where patients often present with subjective complaints like morning stiffness or fatigue, this framing sensitivity means that the same clinical situation could generate different AI recommendations depending on how the question is phrased. None of the studies in this review tested prompt sensitivity, leaving the robustness of reported accuracy rates uncertain. Structured prompting strategies may partially mitigate these effects, but remain untested in rheumatology [\u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e].\u003c/p\u003e \u003cp\u003e \u003cb\u003eClinical implications for rheumatology practice\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe evidence from this review does not support using LLMs for direct treatment decisions in inflammatory arthritis without specialist involvement. In RA, DMARD escalation from methotrexate to biologics depends on disease activity scores, comorbidity profiles, and prior treatment response, none of which current LLMs can access or integrate. In axSpA, the choice between TNF and IL-17 inhibitors requires consideration of extra-articular manifestations such as uveitis and inflammatory bowel disease. In gout, urate-lowering therapy must be adjusted for renal function. These decisions require more than guideline recall, and the errors observed in this review, including recommendation of contraindicated IL-23 inhibitors for axSpA [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e] and guideline-contradictory responses for gout [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e], confirm that unsupervised use carries real clinical risk.\u003c/p\u003e \u003cp\u003e \u003cb\u003eRegulatory and ethical considerations\u003c/b\u003e \u003c/p\u003e \u003cp\u003eNone of the included studies addressed liability when LLM-generated recommendations lead to patient harm. This gap is particularly relevant in rheumatology, where treatment errors can result in irreversible joint destruction if DMARDs are delayed, teratogenic exposure if methotrexate is not stopped before conception, or renal complications if gout therapies are prescribed without monitoring. As LLMs become more accessible to patients and non-specialist clinicians, the question of who bears responsibility for AI-generated rheumatology advice, whether the developer, the clinician who shares it, or the institution that deploys it, remains unresolved. Regulatory bodies have not yet established frameworks for LLM use in subspecialty medicine, and the rheumatology community should engage with this process before these tools enter routine clinical workflows.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePractical guidance for rheumatologists\u003c/b\u003e \u003c/p\u003e \u003cp\u003eBased on the 18 studies reviewed, LLMs may reasonably support three tasks in rheumatology practice today, always under clinician review. First, drafting patient education content, where accuracy was generally acceptable although readability consistently exceeded recommended levels and required editing. Second, answering factual medication questions, as shown by GPT-4 achieving 100% accuracy on methotrexate-related queries in RA [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. Third, responding to structured binary guideline questions, where ChatGPT-4 reached 96% concordance with EULAR RA recommendations [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eLLMs should not be used without clinician verification for several tasks. Case-based clinical reasoning showed consistent performance drops across studies, with scores falling from 5.32/6 on structured questions to 4.24/6 on complex cases [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. Treatment selection and sequencing carry documented safety risks, including the recommendation of contraindicated IL-23 inhibitors for axSpA [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]. Guideline-dependent decisions such as gout management showed model-specific errors, with Gemini contradicting EULAR recommendations in 8% of responses [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]. Autonomous clinical decision-making is not supported by current evidence, as no study prospectively validated LLM outputs in clinical workflows.\u003c/p\u003e \u003cp\u003e \u003cb\u003eRheumatology-specific challenges for LLM integration\u003c/b\u003e \u003c/p\u003e \u003cp\u003eSeveral features of rheumatology practice make it a particularly demanding setting for LLM deployment. Treatment guidelines are updated frequently (2022 ASAS-EULAR update [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], 2021 ACR RA guideline [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]), which means models trained on older data will generate outdated recommendations unless connected to current sources. Biologic and targeted synthetic DMARD selection depends on multiple interacting factors including comorbidities, pregnancy status, prior treatment failures, and emerging safety signals such as the 2022 FDA boxed warning for JAK inhibitors, which standalone LLMs cannot reliably integrate. Inflammatory arthritides share overlapping clinical features, complicating differential diagnosis in ways that generic LLMs tend to oversimplify. Finally, 10 of 18 included studies were conducted in Turkish and reported variable model performance, indicating that LLM utility in lower-resource languages remains unclear. These characteristics explain why LLM integration in rheumatology will likely require purpose-built, retrieval-augmented systems rather than general-purpose chatbots.\u003c/p\u003e \u003cp\u003e \u003cb\u003eLimitations\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThis review has limitations. The published literature to date is largely based on earlier-generation models: BARD has been discontinued, GPT-3.5 has been superseded, and GPT-5 represents the current generation. This reflects the pace of LLM development rather than a limitation of the review itself, but it means that the accuracy estimates reported here likely represent a floor rather than a ceiling of current capability. Evaluations of newer models are beginning to appear but remain limited. Because most evaluations relied on synthetic prompts or predefined questions rather than real patient encounters, current evidence may overestimate LLM performance in clinical practice. The concentration of studies in Turkey (10 of 18) limits geographic generalizability. This overrepresentation likely reflects the early adoption of LLM evaluation research by Turkish rheumatology groups rather than a higher clinical need. Most Turkish studies evaluated LLM responses in Turkish, and model performance in lower-resource languages may not generalize to English or other languages. The limited representation of North America, East Asia, and sub-Saharan Africa means that findings may not reflect LLM performance in healthcare systems with different practice patterns and guideline frameworks. Heterogeneous evaluation metrics and clinical tasks precluded meta-analysis. Gout and JIA were each represented by a single study. All studies were conducted in research settings with no prospective clinical validation, and publication bias likely favours positive results. The wide range of application domains assessed (patient education, clinical reasoning, guideline adherence, and data extraction) increased study heterogeneity and precluded quantitative pooling. Future reviews focusing on a single domain across a broader spectrum of inflammatory joint diseases may enable meta-analysis.\u003c/p\u003e"},{"header":"Conclusions","content":"\u003cp\u003e For the practicing rheumatologist, current evidence supports LLM use under clinician review for patient education drafting, factual medication queries, and structured guideline questions. LLMs should not be used for complex case-based reasoning, treatment selection, or autonomous clinical decisions. Across 18 studies, no evaluation tested retrieval-augmented or agent-based systems, and none prospectively validated LLM outputs in clinical workflows. Rheumatology poses particular challenges for LLM integration, including frequently updated guidelines, complex treatment algorithms with multiple interacting factors, and limited evidence in non-English clinical settings. These gaps may be more important than accuracy estimates in guiding future research and in determining when LLMs are ready for clinical deployment.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAuthor Contributions:\u003c/strong\u003e Conceptualization, Y.A., A.G., E.K.; Methodology, Y.A., A.G., E.K.; Formal Analysis, Y.A., A.G.; Data Curation, Y.A., A.G.; Writing-Original Draft Preparation, Y.A., A.G.; Writing-Review \u0026amp; Editing, Y.A., A.G., M.O., E.K., Y.B., O.R.B., T.M.T.; Supervision: E.K., A.G.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding.\u0026nbsp;\u003c/strong\u003eNone\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests.\u003c/strong\u003e The authors declare that they have no competing interests.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eBournia V-K, Fragoulis GE, Mitrou P et al (2024) Increased prevalence of inflammatory arthritis, systemic lupus erythematosus and systemic sclerosis, during 2020\u0026ndash;2023 versus 2016\u0026ndash;2019 in a Nation-Wide Cohort Study. Rheumatol Int 44:2837\u0026ndash;2846. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s00296-024-05733-y\u003c/span\u003e\u003cspan address=\"10.1007/s00296-024-05733-y\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGBD 2021 Rheumatoid Arthritis Collaborators (2023) Global, regional, and national burden of rheumatoid arthritis, 1990\u0026ndash;2020, and projections to 2050: a systematic analysis of the Global Burden of Disease Study 2021. Lancet Rheumatol 5:e594\u0026ndash;610. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/S2665-9913(23)00211-4\u003c/span\u003e\u003cspan address=\"10.1016/S2665-9913(23)00211-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAletaha D, Smolen JS (2018) Diagnosis and Management of Rheumatoid Arthritis: A Review. JAMA 320:1360\u0026ndash;1372. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1001/jama.2018.13103\u003c/span\u003e\u003cspan address=\"10.1001/jama.2018.13103\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFraenkel L, Bathon JM, England BR et al (2021) 2021 American College of Rheumatology Guideline for the Treatment of Rheumatoid Arthritis. Arthritis Care Res (Hoboken) 73:924\u0026ndash;939. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/acr.24596\u003c/span\u003e\u003cspan address=\"10.1002/acr.24596\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRamiro S, Nikiphorou E, Sepriano A et al (2023) ASAS-EULAR recommendations for the management of axial spondyloarthritis: 2022 update. Ann Rheum Dis 82:19\u0026ndash;34. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1136/ard-2022-223296\u003c/span\u003e\u003cspan address=\"10.1136/ard-2022-223296\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCombe B, Landewe R, Daien CI et al (2017) 2016 update of the EULAR recommendations for the management of early arthritis. Ann Rheum Dis 76:948\u0026ndash;959. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1136/annrheumdis-2016-210602\u003c/span\u003e\u003cspan address=\"10.1136/annrheumdis-2016-210602\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWard MM, Deodhar A, Gensler LS et al (2019) 2019 Update of the American College of Rheumatology/Spondylitis Association of America/Spondyloarthritis Research and Treatment Network Recommendations for the Treatment of Ankylosing Spondylitis and Non-radiographic Axial Spondyloarthritis. Arthritis Care Res (Hoboken) 71:1285\u0026ndash;1299. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/acr.24025\u003c/span\u003e\u003cspan address=\"10.1002/acr.24025\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFitzGerald JD, Dalbeth N, Mikuls T et al (2020) 2020 American College of Rheumatology Guideline for the Management of Gout. Arthritis Care Res (Hoboken) 72:744\u0026ndash;760. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1002/acr.24180\u003c/span\u003e\u003cspan address=\"10.1002/acr.24180\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eThirunavukarasu AJ, Ting DSJ, Elangovan K et al (2023) Large language models in medicine. Nat Med 29:1930\u0026ndash;1940. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41591-023-02448-8\u003c/span\u003e\u003cspan address=\"10.1038/s41591-023-02448-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBedi S, Liu Y, Orr-Ewing L et al (2025) Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA 333:319\u0026ndash;328. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1001/jama.2024.21700\u003c/span\u003e\u003cspan address=\"10.1001/jama.2024.21700\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJi Z, Lee N, Frieske R et al (2023) Survey of Hallucination in Natural Language Generation. ACM Comput Surv 55:248:1\u0026ndash;248. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1145/3571730\u003c/span\u003e\u003cspan address=\"10.1145/3571730\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHager P, Jungmann F, Holland R et al (2024) Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med 30:2613\u0026ndash;2622. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41591-024-03097-1\u003c/span\u003e\u003cspan address=\"10.1038/s41591-024-03097-1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOmar M, Naffaa ME, Glicksberg BS et al (2024) Advancing rheumatology with natural language processing: insights and prospects from a systematic review. Rheumatol Adv Pract 8:rkae120. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/rap/rkae120\u003c/span\u003e\u003cspan address=\"10.1093/rap/rkae120\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFairley JL, Kapoor M, Sharma D (2026) Generative artificial intelligence in osteoarthritis: A systematic scoping review of current applications and future directions. Osteoarthritis Cartilage. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.joca.2026.03.001\u003c/span\u003e\u003cspan address=\"10.1016/j.joca.2026.03.001\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. S1063-4584(26)00685-0\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMa Z, Liu Y, Zhang Z et al (2025) Clinical applications of large language models in knee osteoarthritis: a systematic review. Front Med (Lausanne) 12:1670824. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3389/fmed.2025.1670824\u003c/span\u003e\u003cspan address=\"10.3389/fmed.2025.1670824\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePage MJ, McKenzie JE, Bossuyt PM et al (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372:n71. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1136/bmj.n71\u003c/span\u003e\u003cspan address=\"10.1136/bmj.n71\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOuzzani M, Hammady H, Fedorowicz Z et al (2016) Rayyan-a web and mobile app for systematic reviews. Syst Rev 5:210. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s13643-016-0384-4\u003c/span\u003e\u003cspan address=\"10.1186/s13643-016-0384-4\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBerghea F, Andras D, Berghea EC (2025) Generative Artificial Intelligence and Risk Appetite in Medical Decisions in Rheumatoid Arthritis. Appl Sci 15:5700. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/app15105700\u003c/span\u003e\u003cspan address=\"10.3390/app15105700\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLa Bella S, Bayraktar D, Porreca A et al (2025) Global variations in artificial intelligence-generated information on juvenile idiopathic arthritis. Rheumatology (Oxford) 64:5687\u0026ndash;5697. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/rheumatology/keaf329\u003c/span\u003e\u003cspan address=\"10.1093/rheumatology/keaf329\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCoskun BN, Yagiz B, Ocakoglu G et al (2024) Assessing the accuracy and completeness of artificial intelligence language models in providing information on methotrexate use. Rheumatol Int 44:509\u0026ndash;515. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s00296-023-05473-5\u003c/span\u003e\u003cspan address=\"10.1007/s00296-023-05473-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSaji JG, Jabeen J, Antony JT (2025) Evaluating readability, reliability, and originality of artificial intelligence-generated patient education guides for common rheumatological conditions. J Public Health (Berl). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10389-025-02628-5\u003c/span\u003e\u003cspan address=\"10.1007/s10389-025-02628-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eForte G, Mauro D, Raimondi M et al ChatGPT vs rheumatologists: cross-sectional study on accuracy and patient perception of AI-generated information for psoriatic arthritis. Ann Rheum Dis 2025:S0003-4967(25)04538-8. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.ard.2025.11.012\u003c/span\u003e\u003cspan address=\"10.1016/j.ard.2025.11.012\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAtilan AU, Cetin N (2026) An old disease, a new linguistic challenge for large language models: patient education on psoriasis and psoriatic arthritis in an underrepresented medical language. Int J Med Inf 209:106246. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.ijmedinf.2025.106246\u003c/span\u003e\u003cspan address=\"10.1016/j.ijmedinf.2025.106246\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRen Y, Kang Y, Cao S et al (2025) Evaluating the performance of large language models in health education for patients with ankylosing spondylitis/spondyloarthritis: a cross-sectional, single-blind study in China. BMJ Open 15:e097528. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1136/bmjopen-2024-097528\u003c/span\u003e\u003cspan address=\"10.1136/bmjopen-2024-097528\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBai J, Ji X, Yu J et al (2026) Assessing the Quality of AI Responses to Patient Concerns About Axial Spondyloarthritis: Delphi-Based Evaluation. JMIR AI 5:e79153. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2196/79153\u003c/span\u003e\u003cspan address=\"10.2196/79153\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKara M, Ozduran E, Kara MM et al (2025) Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about Ankylosing spondylitis. PLoS ONE 20:e0326351. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1371/journal.pone.0326351\u003c/span\u003e\u003cspan address=\"10.1371/journal.pone.0326351\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAltunel Kılın\u0026ccedil; E, \u0026Ccedil;abuk \u0026Ccedil;elik N (2025) Evaluation of artificial ıntelligence use in ankylosing spondylitis with ChatGPT-4: patient and physician perspectives. Clin Rheumatol 44:4015\u0026ndash;4023. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10067-025-07648-w\u003c/span\u003e\u003cspan address=\"10.1007/s10067-025-07648-w\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u0026Ccedil;abuk \u0026Ccedil;elik N, Altunel Kılın\u0026ccedil; E (2026) AI-generated patient education for ankylosing spondylitis: a comparative study of readability and quality. Clin Rheumatol 45:2003\u0026ndash;2008. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10067-025-07771-8\u003c/span\u003e\u003cspan address=\"10.1007/s10067-025-07771-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSari F, \u0026Ccedil;elik Z, Mirza Y (2026) ChatGPT-4 vs. DeepSeek-V3: a comparative study of response quality, reliability, usefulness, and readability for exercise and rehabilitation strategies in patients with ankylosing spondylitis. Clin Rheumatol 45:187\u0026ndash;195. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10067-025-07789-y\u003c/span\u003e\u003cspan address=\"10.1007/s10067-025-07789-y\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u0026Ccedil;elik N\u0026Ccedil;, Kılın\u0026ccedil; EA (2025) Assessment of ChatGPT\u0026rsquo;s adherence to EULAR diagnostic criteria and therapeutic protocols for rheumatoid arthritis at two distinct time points, 14 days apart, utilizing binary and multiple-choice inquiries. Clin Rheumatol 44:2233\u0026ndash;2239. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10067-025-07417-9\u003c/span\u003e\u003cspan address=\"10.1007/s10067-025-07417-9\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eUsen A, Kuculmez O (2025) Evaluation of the Performance of Large Language Models in the Management of Axial Spondyloarthropathy: Analysis of EULAR 2022 Recommendations. Diagnostics 15:1455. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/diagnostics15121455\u003c/span\u003e\u003cspan address=\"10.3390/diagnostics15121455\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMeral HB, Kolak E (2026) Evaluation of ChatGPT-4o and Gemini for gout management: a comparative analysis based on EULAR guidelines. Sci Rep 16:4831. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41598-026-35166-5\u003c/span\u003e\u003cspan address=\"10.1038/s41598-026-35166-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKayacan Erdoğan E, Babaoğlu H (2024) Clinical Reasoning and Knowledge Assessment of Rheumatology Residents Compared to AI Models: A Pilot Study. J Clin Med 13:7405. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3390/jcm13237405\u003c/span\u003e\u003cspan address=\"10.3390/jcm13237405\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMiao BY, Binvignat M, Garcia-Agundez A et al (2025) JAMIA Open 8:ooaf132. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/jamiaopen/ooaf132\u003c/span\u003e\u003cspan address=\"10.1093/jamiaopen/ooaf132\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Extracting TNFi switching reasons and trajectories from real-world data using large language models\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePolyzou M, Baraliakos X (2026) Artificial Intelligence (AI) in rheumatology: a comparative evaluation of the ChatGPT and DeepSeek application. BMC Rheumatol 10:13. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1186/s41927-026-00618-y\u003c/span\u003e\u003cspan address=\"10.1186/s41927-026-00618-y\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGorenshtein A, Omar M, Glicksberg BS et al AI Agents in Clinical Medicine: A Systematic Review 2025:2025.08.22.25334232. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1101/2025.08.22.25334232\u003c/span\u003e\u003cspan address=\"10.1101/2025.08.22.25334232\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKresevic S, Giuffr\u0026egrave; M, Ajcevic M et al (2024) Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework. Npj Digit Med 7:102. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41746-024-01091-y\u003c/span\u003e\u003cspan address=\"10.1038/s41746-024-01091-y\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZakka C, Shad R, Chaurasia A et al (2024) Almanac \u0026mdash; Retrieval-Augmented Language Models for Clinical Medicine. NEJM AI 1:AIoa2300068. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1056/AIoa2300068\u003c/span\u003e\u003cspan address=\"10.1056/AIoa2300068\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu S, McCoy AB, Wright A (2025) Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines. J Am Med Inf Assoc 32:605\u0026ndash;615. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/jamia/ocaf008\u003c/span\u003e\u003cspan address=\"10.1093/jamia/ocaf008\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOmar M, Sorin V, Collins JD et al (2025) Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support. Commun Med (Lond) 5:330. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s43856-025-01021-3\u003c/span\u003e\u003cspan address=\"10.1038/s43856-025-01021-3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKlang E, Glicksberg BS, Gorenshtein A et al Clinical Agents Don\u0026rsquo;t Care 2025:2025.10.17.25338226. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1101/2025.10.17.25338226\u003c/span\u003e\u003cspan address=\"10.1101/2025.10.17.25338226\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOmar M, Gorenshtein A, Agbareia R et al Impact of Patient Communication Style on Agentic AI-Generated Clinical Advice in E-Medicine 2025:2025.12.02.25341475. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.64898/2025.12.02.25341475\u003c/span\u003e\u003cspan address=\"10.64898/2025.12.02.25341475\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"},{"header":"Table 2","content":"\u003cp\u003eTable 2 is available in the Supplementary Files section.\u003c/p\u003e\n"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Beth Israel Deaconess Medical Center","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Inflammatory arthritis, Large language models, Systematic review, Patient education","lastPublishedDoi":"10.21203/rs.3.rs-9529682/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9529682/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBackground\u003c/p\u003e\n\u003cp\u003eManaging inflammatory arthritis involves combining clinical, serological, and imaging data while following evolving treatment guidelines. Large language models (LLMs) are increasingly being evaluated for rheumatology tasks, but whether this promise holds in inflammatory arthritis remains unclear. We therefore systematically reviewed the performance of LLMs across clinical tasks in inflammatory arthritis.\u003c/p\u003e\n\u003cp\u003eMethods\u003c/p\u003e\n\u003cp\u003eWe conducted a systematic review (PROSPERO: CRD420261359100) searching PubMed, Scopus, and PubMed Central (January 2022 to April 2026). Eligible studies evaluated LLM performance on clinical tasks in inflammatory arthritis. Two reviewers (Y.A., A.G.) screened 113 records. Risk of bias was assessed using an adapted QUADAS-2 framework with AI-specific modifications.\u003c/p\u003e\n\u003cp\u003eResults\u003c/p\u003e\n\u003cp\u003eEighteen studies met inclusion criteria, covering rheumatoid arthritis (n=4), ankylosing spondylitis/axial spondyloarthritis (n=7), psoriatic arthritis (n=2), gout (n=1), juvenile idiopathic arthritis (n=1), and multiple diseases (n=3).\u003c/p\u003e\n\u003cp\u003eOver 20 distinct LLMs were evaluated, including ChatGPT-3.5, ChatGPT-4, ChatGPT-4o, Gemini 2.0, DeepSeek-R1/V3, Claude, Perplexity, and Kimi. Findings were synthesized across four application domains: patient education (n=11), guideline adherence (n=6), clinical reasoning and real-world applications (n=3), and other (n=1). All studies assessing readability reported outputs above recommended thresholds; studies using the Flesch-Kincaid index reported Grade Levels above 15. Studies comparing multiple LLMs found a trade-off between readability and scientific reliability.\u003c/p\u003e\n\u003cp\u003eGuideline concordance varied widely across models, from 48% to 96%. In one blinded comparison, patients preferred LLM responses over specialist-written answers. Lower accuracy was reported for case-based clinical scenarios (4.24/6) compared with FAQ and guideline-based questions (5.32–5.36/6; overall p=0.044). When LLM outputs were compared with clinical data from 116 patients with axial spondyloarthritis and psoriatic arthritis, agreement was low to moderate (Cohen and Fleiss kappa).\u003c/p\u003e\n\u003cp\u003eConclusions\u003c/p\u003e\n\u003cp\u003eLLMs may support patient education, factual medication queries, and structured guideline questions when used under clinician review, but should not be used for case-based reasoning, treatment selection, or autonomous clinical decisions. None of the 18 included studies evaluated retrieval-augmented or agent-based systems, and none prospectively validated LLMs in clinical workflows. Safe integration in rheumatology will require purpose-built, knowledge-grounded systems and prospective evaluation before routine clinical use.\u003c/p\u003e\n\u003cp\u003eRegistration: PROSPERO CRD420261359100\u003c/p\u003e","manuscriptTitle":"Large Language Models in Inflammatory Arthritis: A Systematic Review Across Clinical Tasks","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-28 13:10:30","doi":"10.21203/rs.3.rs-9529682/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"bfa76557-031a-4df2-8934-ee2b83c31521","owner":[],"postedDate":"April 28th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-04-28T13:10:30+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-28 13:10:30","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9529682","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9529682","identity":"rs-9529682","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}