Comparative Analysis of Large Language Models in Dermatological Diagnosis: An Evaluation of Diagnostic Accuracy.

The field of dermatology, characterized by its reliance on visual pattern recognition and extensive clinical experience, is increasingly being influenced by the advent of artificial intelligence (AI), specifically large language models (LLMs) [ 1 ]. The application of these models holds promise for augmenting diagnostic capabilities, streamlining clinical workflows, and ultimately improving patient care [ 2 - 5 ]. AI has already started transforming dermatology by improving diagnostic accuracy, treatment planning, and surgical support [ 4 ]. The integration of machine learning in healthcare offers possibilities for enhancing care across a spectrum of applications [ 6 ]. In recent years, machine learning has demonstrated semantic understanding and information extraction, sometimes detecting abstract patterns with greater accuracy than human experts. This study delves into the diagnostic performance of three prominent LLMs - ChatGPT-4o, Claude 3.7 Sonnet, and Gemini 2.0 Flash - in the context of rare dermatological conditions. We present a comparative analysis of their ability to generate accurate diagnoses and comprehensive differential diagnoses based solely on the clinical features extracted from a collection of case reports, which are the cornerstone for clinical acumen in rare dermatological diseases. The clinical integration of AI in dermatology workflow remains challenging despite its potential in medical applications [ 7 ]. The rationale behind this research stems from the growing recognition of the potential of AI to assist dermatologists in diagnostic decision-making [ 8 ]. However, the deployment of AI in daily clinical workflows is still a problem. The primary objective of this study was to evaluate and compare the diagnostic accuracy of three LLMs: ChatGPT-4o, Claude 3.7 Sonnet, and Gemini 2.0 Flash, when applied to rare dermatological cases. For clarity, diagnostic accuracy was defined as both correct top-choice predictions and inclusion of the correct diagnosis within differential diagnosis lists. Secondary objectives included assessing their ability to generate clinically relevant differential diagnoses and analyzing case-level failures to identify contexts in which these tools may be less reliable. We focused on rare dermatological conditions because they are diagnostically challenging, often underrepresented in training corpora, and carry significant clinical implications when misdiagnosed. We hypothesized that while all three models would demonstrate some degree of diagnostic accuracy, important differences would emerge in their ability to handle rare and diagnostically complex conditions. By addressing this gap, our study provides early insights into the potential and limitations of LLMs for dermatology and highlights areas for future refinement. By addressing this gap, our study contributes to the growing body of work comparing LLMs in clinical decision-making [ 9 ] and provides early insights into their potential and limitations for dermatology. Furthermore, in line with prior work exploring AI-based tools in dermatology [ 10 ], we emphasize the importance of developing a fundamental understanding of AI within the medical community to ensure safe and effective integration into clinical workflows.

Table 1 presents the actual diagnoses of the 15 dermatological cases and the most probable diagnoses generated by the three LLMs under investigation. A crucial performance indicator was the Top Matches  metric, which quantified the number of cases in which the model correctly identified the most probable diagnosis. In addition to evaluating the models’ ability to pinpoint the correct diagnosis, the research also delved into their capacity to generate comprehensive and accurate differential diagnosis lists. The performance summary table below summarizes an evaluation of the diagnostic capabilities of three LLMs in the context of 15 dermatological cases (Table 2 ). The analysis examined whether the models’ most probable diagnoses aligned with the actual case diagnoses, and whether the correct diagnosis was included in the models’ differential diagnosis lists. LLM, large language model The heatmap below provides a visual representation of the models’ performance across individual cases (Figure 1 ). A value of 1 signifies that the model correctly identified the diagnosis as the top choice, while a value of 0.5 indicates that the correct diagnosis was included in the list of differential diagnoses. Conversely, a value of 0 denotes that the model failed to accurately identify the correct diagnosis. In terms of diagnostic accuracy, ChatGPT-4o and Claude 3.7 Sonnet each correctly identified the most probable diagnosis in 10 (66.7%) of 15 cases, while Gemini 2.0 Flash achieved a lower top-match rate of 8 out of 15 (53.3%). When including cases where the correct diagnosis was not the top prediction but appeared in the list of differentials, both ChatGPT-4o and Claude 3.7 Sonnet demonstrated a total coverage rate of 86.7%, reflecting a broader diagnostic awareness. In contrast, Gemini 2.0 Flash showed limited differential utility, reaching a total coverage of 60.0%, with only 1 case out of 15 (6.7%) containing the correct diagnosis in the differential list without it being the top choice. These results highlight that while Gemini demonstrated some diagnostic capability, ChatGPT-4o and Claude 3.7 provided substantially more reliable outputs, both in primary and supporting diagnoses. The bar chart below presented compares the overall performance of the three language models in terms of the total number of top matches, differential-only matches, and overall coverage (Figure 2 ). The results indicate that ChatGPT-4o and Claude 3.7 Sonnet demonstrate comparable total coverage, although their individual strengths may differ. LLM, large language model Noteworthy findings include five cases in which all three models unanimously succeeded in identifying the correct diagnosis. In contrast, Case 3, involving Blastic Plasmacytoid Dendritic Cell Neoplasm (BPDCN), was a notable failure, with all models inaccurately predicting the diagnosis. Neither ChatGPT-4o nor Gemini 2.0 Flash demonstrated any exclusive instances of outperforming both of the other models. However, Claude 3.7 Sonnet showed one exclusive instance of superior performance in Case 12, correctly identifying Granulosis Rubra Nasi when the other models did not. The small sample size and paired nature of the data limited the reliability of a chi-square test. Given these constraints, descriptive statistics and visual comparisons were prioritized over inferential tests. Future studies with larger case sets may permit more robust statistical comparisons using paired testing methods such as McNemar’s test. The results revealed that ChatGPT-4o and Claude 3.7 Sonnet exhibited similar levels of overall accuracy, outperforming Gemini 2.0 Flash in terms of both top matches and differential diagnoses. The analysis indicated that while all three models demonstrated proficiency in certain cases, there were also instances where they struggled to accurately identify the correct diagnosis.

The integration of LLMs into dermatological diagnostics presents an exciting advancement in AI-assisted healthcare. In this study, we evaluated the performance of three advanced LLMs - ChatGPT-4o, Claude 3.7 Sonnet, and Gemini 2.0 Flash across 15 rare dermatological cases, assessing their ability to generate accurate primary diagnoses and plausible differential lists based solely on clinical findings. Overall, ChatGPT-4o and Claude 3.7 demonstrated strong diagnostic capabilities, correctly identifying the top diagnosis in 10 of 15 cases (66.7%) and achieving a total diagnostic coverage of 13 of 15 (86.7%) cases when differentials were included. Gemini 2.0 Flash, while still competent, lagged slightly behind with a top match accuracy of 8 (53.3%) of 15 cases and total coverage of 9 (60.0%) of 15 cases. These results highlight the potential utility of certain LLMs in clinical decision support, particularly in settings where immediate dermatological expertise may not be available. However, aggregate accuracy does not tell the full story. Specific case-level failures illuminate important limitations. In Case 3, for example, BPDCN - all three models failed to identify the diagnosis, either as a top choice or within their differential lists. This suggests that LLMs, while broadly informed, may struggle with rare hematologic-dermatologic interface disorders that require domain-specific training. In another critical failure (Case 15), the correct diagnosis of amelanotic melanoma was missed by all models, with plausible but incorrect suggestions such as squamous cell carcinoma or benign inflammatory conditions offered instead. Both conditions are rare and likely underrepresented in training corpora. In addition, BPDCN requires recognition of complex hematologic-dermatologic overlap, while amelanotic melanoma depends heavily on visual cues such as dermoscopic features, which are difficult to capture in text-only prompts. These errors underscore the risk that confident, medically reasonable but inaccurate outputs may create a false sense of diagnostic certainty for clinicians [ 27 ]. Descriptive statistics and visual comparisons (e.g., heatmaps, bar charts) provided valuable insight into performance differences between models. While ChatGPT-4o and Claude 3.7 Sonnet showed superior diagnostic alignment, model performance varied across cases, emphasizing the importance of case complexity and presentation in influencing outcomes. The inclusion of differential diagnoses proved to be a useful metric of diagnostic awareness, particularly in scenarios where the top diagnosis was incorrect. Importantly, the clinical relevance of these findings extends into both general and aesthetic dermatology. In cosmetic practice, where patients frequently present with pigmentary or textural concerns, distinguishing benign aesthetic issues from serious pathologies is critical. LLMs may support practitioners by flagging uncommon but clinically significant conditions early in the diagnostic process. Their ability to synthesize differential diagnoses also supports patient triage, referral decisions, and pre-procedural screening, enhancing safety and clinical efficiency. Despite promising results, careful consideration must be given to the risks of over-reliance on LLMs. These tools should be viewed as augmented intelligence, not replacements for clinical judgment. Misleading confidence, knowledge gaps, and hallucinated details remain nontrivial risks. To safely implement LLMs in dermatology, future efforts must include robust validation, transparent uncertainty estimation, and training on diverse, representative datasets [ 27 ][ 1 ]. Additionally, prompt engineering and model fine-tuning tailored to dermatological use cases could significantly improve reliability [ 27 ]. The performance of large language models in this study highlights their potential as diagnostic aids, while also underscoring the importance of careful evaluation and validation. The performance of these models is influenced by the quality and format of the input data. These models may be particularly helpful in rare and difficult-to-diagnose cases, where clinical experience and expert knowledge are critical. In summary, while ChatGPT-4o and Claude 3.7 Sonnet show strong potential as clinical diagnostic assistants, critical evaluation of their outputs and caution in their application remains essential. With proper safeguards, LLMs may evolve into powerful tools that support, rather than supplant, the clinical expertise at the heart of dermatologic care. The study underscores the importance of validating LLMs on diverse and representative datasets, as well as establishing clear guidelines for their responsible and ethical implementation in healthcare settings [ 28 - 29 ]. Limitations This study has several limitations. The sample size was small, comprising only 15 cases, which restricts the generalizability of the findings. Because the cases were drawn from published reports of rare dermatological conditions, selection bias is possible, as such reports may overrepresent diagnostically complex or unusual presentations. The rarity of the included conditions also limits their representativeness for routine dermatology practice. In addition, only clinical features were provided to the models, without histopathological or dermoscopic data. This introduces a fundamental visual-textual disconnect, given that dermatology is primarily a visual discipline, and limits the clinical applicability of our findings, particularly for conditions such as melanoma. Diagnostic outputs were further influenced by the specific prompt structure, as prompt engineering can significantly affect performance. Moreover, the LLMs assessed were trained primarily on general-domain data rather than dermatology-specific corpora, which may constrain their accuracy in highly specialized cases. As LLMs are non-static systems that evolve with ongoing updates, strict reproducibility of results cannot be guaranteed even when identical prompts are used. Finally, because this was a retrospective analysis of published cases, it cannot account for real-time clinical variability. Prospective studies incorporating larger, more representative datasets and multimodal inputs will be essential to establish the clinical utility of these tools.

This study directly compared the diagnostic accuracy of three large language models, namely, ChatGPT-4o, Claude 3.7 Sonnet, and Gemini 2.0 Flash in rare dermatological cases. ChatGPT-4o and Claude 3.7 Sonnet outperformed Gemini 2.0 Flash in both top-match accuracy and total diagnostic coverage, although performance varied across cases depending on complexity and clinical presentation. Because dermatology is fundamentally a visual specialty, the reliance on text-based case descriptions represents a key limitation that constrains clinical applicability. Model predictions were compared against gold-standard diagnoses established in the published case reports, rather than through direct benchmarking against clinician performance, and the findings should therefore be interpreted as preliminary and exploratory. While these results suggest that certain LLMs hold promise as diagnostic aids, careful evaluation is essential for responsible implementation. Future research including multimodal approaches that incorporate visual data and prospective benchmarking against dermatologist performance will be critical to establish the true clinical utility of these tools in dermatology and other medical specialties.

Data acquisition and preparation A retrospective collection of 15 case reports, each representing a distinct and relatively rare dermatological condition, served as the foundation for this study [ 11 - 25 ]. We selected these 15 cases because they represent rare dermatological conditions documented in recent open-access case reports that provided sufficient clinical detail for standardized extraction. Many other published reports were excluded because they lacked adequate clinical descriptions to be used in our prompt template. Clinical features were transcribed verbatim from the published case reports, restricted to sections describing patient demographics (age, sex), lesion morphology (size, shape, color, distribution), and associated symptoms. Clinical features were independently extracted by two clinically trained reviewers. Discrepancies were first resolved through discussion and consensus; if consensus could not be reached, a third reviewer adjudicated to ensure accuracy and minimize bias in the data abstraction process. The small sample size reflects the relative scarcity of such published cases, but it allowed for focused evaluation of LLM performance across diverse, diagnostically challenging conditions. Ethical considerations were prioritized by ensuring that all patient identifiers were removed from the case reports, safeguarding patient confidentiality, and adhering to established ethical guidelines for medical research. To emulate the diagnostic process initiated during a patient encounter, data extraction was performed to isolate key clinical features detailed within each case report. The extracted features encompassed a spectrum of pertinent information, including the patient’s age, sex, detailed descriptions of lesion morphology (e.g., size, shape, color, distribution), and associated symptoms. A standardized prompt structure was used across all models. A representative example of the prompt, together with the corresponding model output, is provided to enhance reproducibility and transparency (Appendix). All models were queried between May and June 2025 using their official web-based interfaces, ensuring consistency across outputs. LLM prompting and output collection The prepared clinical features for each case were then systematically input into the three LLMs under investigation: ChatGPT-4o, Claude 3.7 Sonnet, and Gemini 2.0 Flash. To ensure consistency and minimize bias, a standardized prompt structure was employed across all models. The prompts were carefully designed to elicit two distinct outputs from each model: the most probable diagnosis based on the provided clinical features, and a list of differential diagnoses. The models’ outputs, comprising the most probable diagnosis and the list of up to five differential diagnoses for each case, were recorded and organized into a structured format suitable for subsequent analysis. This process ensured that all relevant information was captured accurately and consistently, enabling a robust comparison of the models’ diagnostic capabilities. Evaluation metrics and statistical analysis The accuracy of the models’ most probable diagnoses was assessed by comparing them to the confirmed diagnoses documented in the original case reports. In instances where the most probable diagnosis provided by a model did not align with the actual diagnosis, the list of differential diagnoses generated by that model was examined to determine if the correct diagnosis was included among the proposed differentials. The quality of the differential diagnosis lists was evaluated using the approach described by McDuff et al. [ 26 ], which defines diagnostic coverage as the inclusion of the correct diagnosis either as the top choice or within the list of differentials.