Performance of the ChatGPT-5 Language Model in Solving a Specialty Examination in Balneology and Physical Medicine.

ChatGPT, built on the GPT architecture, was launched in 2018 and rapidly gained widespread attention within both the scientific community and the general public. Its release was described as a milestone, as it markedly transformed the way researchers communicate and conduct scientific work. Within two years of its debut, the tool began to be applied to multiple aspects of academic activity, ranging from drafting and editing scientific texts, through literature searches, to the automation of code analysis. While this has contributed to increased productivity, concerns have also been raised regarding the quality and reliability of AI-generated content [ 1 ]. Modern language models are rooted in a seminal publication from 2017 [ 2 ], which proposed the use of attention as the core architectural principle. This laid the groundwork for successive generations of models (GPT-1, GPT-2, and GPT-3), each differing in scale and performance in natural language processing. Notably, GPT-3, released in 2020, with its 175 billion parameters, demonstrated the ability to perform tasks in both few-shot and zero-shot settings, enabling smoother human-machine interaction. GPT-4, announced in March 2023, further enhanced the architecture with multimodal capabilities, allowing image analysis and broadening practical applications, including in medicine [ 1 , 3 ]. Currently, GPT-5, officially released by OpenAI in August 2025, has replaced earlier versions such as GPT-4o and GPT-4.5 [ 1 ]. According to a study published by Emory University, GPT-5 demonstrated higher effectiveness in medical reasoning and multimodal diagnostics tasks than both the previous GPT-4o version and pre-licensed doctors. The model achieved a score of 95.84% on the MedQA test, representing an increase of 4.8 percentage points compared to GPT-4o. In tasks requiring the integration of patient history, medical images, and laboratory results, it reached 70% accuracy, which is nearly a 30-point improvement over its predecessor. Additionally, GPT-5 outperformed pre-licensed doctors by 24.23% in reasoning and by 29.40% in comprehension [ 4 ]. The development of AI has yielded tangible benefits in healthcare and biomedical sciences, including supporting faster diagnostic processes, enabling large-scale molecular data analysis, improving point-of-care diagnostic tools, and refining patient risk stratification. Clinical research published in high-impact journals, such as Tikhomirov et al. in 2024, confirms that AI improves diagnostic quality, particularly in radiology and other medical fields. However, the need for standardized research methodologies, multicenter analyses, and the inclusion of population-based data remains emphasized. Furthermore, the rapid implementation of AI in medical practice requires caution: it is essential to distinguish between experimental simulations and clinical validation while also considering cognitive factors influencing physicians’ decision-making [ 5 ]. The present study demonstrates that ChatGPT-5 is capable of passing the PES in Balneology and Physical Medicine, marking a significant advancement in handling complex medical questions. This example illustrates the rapid pace of AI development and the evolution of OpenAI’s models. Tracking AI progress requires systematic research documenting real-time changes. The objective of this study was to evaluate the effectiveness of the ChatGPT-5 language model in answering National Specialty Examination (PES) questions in Balneology and Physical Medicine. Particular attention was paid to the accuracy of responses and the model’s self-assessed confidence. The analysis was based on a comparison of ChatGPT’s answers with the official answer key provided by the Center for Medical Examinations (CEM).

The ChatGPT-5 model was presented with 120 questions from the spring 2024 specialization exam in balneology and physical medicine (Table 1 ). The model answered 83 questions correctly (70.34%) and 35 incorrectly (29.66%) (Table 2 , Figure 1 ). Two questions were invalidated by the Medical Education Center. No significant differences were observed in the performance on clinical versus theoretical questions (p = 0.98309; χ² = 0.00045) (Table 3 ). Confidence levels were found to correlate with answer accuracy (p = 0.02903; χ² = 4.76605) (Table 4 ). Moreover, the probability of obtaining a correct response was significantly higher when the model reported a higher confidence level (p = 0.0329) (Table 4 ). The Mann-Whitney U test result (p = 0.07137) suggests there is no statistically significant difference between clinical and theoretical questions. The table presents the ChatGPT-5 model’s responses in relation to the correct answers provided in the answer key obtained from the CEM in Łódź. Each question includes the confidence level reported by the model, rated on a scale from 1 to 5 (1 - no confidence; 2 - low confidence; 3 - moderate confidence; 4 - high confidence; and 5 - complete confidence). * The full test contained 120 questions; two questions inconsistent with current medical knowledge were excluded from the analysis. Based on the analysis of the stated confidence levels in the responses provided by the ChatGPT-5 model, a significant difference was observed between correct and incorrect responses. A significant difference was identified when the confidence distribution was analyzed; correct responses were more likely to result in higher confidence levels, although incorrect responses were more likely to receive lower ratings (Figure 2 ).

PES in balneology and physical medicine represents a crucial stage in the process of obtaining specialization in this field. It requires candidates to demonstrate comprehensive theoretical knowledge and practical understanding of balneology and physical medicine, covering a wide range of conditions, diagnostic methods, and therapeutic procedures. The complexity and breadth of the topics make success in this exam a significant indicator of a specialist’s competence while also demanding integration of knowledge from various areas of medicine to support sound clinical decision-making. This study evaluates the performance of the latest publicly available version of the ChatGPT-5 language model on examination tasks intended for future specialists in balneology and physical medicine. The model achieved a score of 70.34%, thereby meeting the passing threshold. Comparisons with earlier versions of OpenAI models in other medical domains demonstrate a clear trend of improving accuracy in addressing medical knowledge tasks. AI is growing increasingly significant in the fields of physical medicine and balneology, where it aids in assessing the qualities of therapeutic mud and mineral water and helps to customize treatment [ 6 ]. Additionally, AI models enable the prediction of treatment results, therapeutic protocol optimization, and patient progress tracking, offering significant support for specialist training and clinical decision-making [ 7 ]. These technologies also offer remote patient state assessment and effective administration of balneological facilities [ 8 ]. Balneotherapy has been shown in prior research to be beneficial for rheumatic, dermatological, and mental health issues. AI integration can improve the accuracy of medical treatment and the efficacy of the educational process [ 6 , 7 ]. Hsieh et al. in 2024 demonstrated that the ChatGPT-3.5 version did not achieve the minimum score required to pass the Taiwanese Board of Plastic Surgery examination [ 9 ]. In a similar vein, a Polish study by Kufel et al. in 2023 on PES solving in radiology and imaging medicine revealed that ChatGPT-3.5 failed the examination [ 10 ]. In 2024, Yudovich et al. tested ChatGPT-3.5 and ChatGPT-4 on standardized urology knowledge assessments in the United States and demonstrated improved accuracy of responses with ChatGPT-4 compared to the earlier version. Nevertheless, neither model was able to meet the passing test requirements [ 11 ]. A study by Goodings et al. demonstrated ChatGPT-4’s promising potential in this field by demonstrating that it can manage the demands of complex medical examinations, especially when properly customized and taught in a specialized environment [ 12 ]. These findings, however, contrast with results from a study on the dermatology board examination [ 13 ]. Using 120 questions from the national specialty exam, the study assessed the performance of ChatGPT-3.5 and ChatGPT-4.0. In the Polish and English versions, ChatGPT-4.0 achieved minimum scores of 70% and 80%, respectively. Importantly, even in this narrowly focused specialty exam, the model demonstrated its potential as an educational tool, surpassing the passing threshold of 60% in both settings. Our study’s findings emphasize ChatGPT-5’s adaptability and potential in medical education by showing that it performs similarly on clinical and theoretical questions (p = 0.983). A substantial association (p < 0.05) was found when the relationship between self-reported confidence and answer correctness was examined. Furthermore, the Mann-Whitney U test confirmed that correct answers had much greater confidence levels. When combined, these results suggest that confidence ratings could be a helpful assessment of how reliable model-generated responses are. These observations align with previous studies on GPT-4, which demonstrated that higher confidence levels were associated with better quality responses and explanations. The literature also indicates that large language models effectively encode clinical knowledge [ 14 ] and perform well in solving educational and clinical tasks in both national and international settings [ 15 , 16 ]. In light of these findings, ChatGPT-5 may represent a valuable tool for supporting medical education and clinical decision-making, provided that its limitations are acknowledged and responses are subject to expert verification. The low GPT-5 score in balneology and physical medicine (70.34%) may be related to the fact that these are niche fields with relatively little available data. The model relies primarily on publicly available sources, while practical knowledge and detailed procedures in these areas are rarely described in detail. Consequently, in tests requiring precise answers, it may perform worse than in better-documented medical fields. The differences in performance achieved by subsequent versions of ChatGPT models across different medical disciplines suggest a need for further research into the use of AI in specialty testing. While ChatGPT-3.5 failed to pass exams in plastic surgery, radiology, and urology [ 9 - 11 ], newer versions, such as ChatGPT-4.0, achieved scores above the pass threshold in some areas, including dermatology [ 13 ] and nuclear medicine [ 17 ], as well as in our study using the ChatGPT-5 model. This variation suggests that the performance of AI models depends on both the software version and the specific exam material. Further research is essential to better understand both the capabilities and limitations of AI across different fields of medicine. It is necessary to develop methods that enhance the accuracy of generated responses and to evaluate the extent to which AI models can support learning and preparation for specialty examinations. Systematic validation in diverse clinical contexts will enable the safe and effective use of AI in medical education and practice, while minimizing the risk of erroneous outputs that could lead to serious consequences. The study also highlights directions for future research, including the validation of models on real patient data, support for integrating AI into the educational process, and the development of clinical decision support systems, which may contribute to improved diagnostics and therapy selection. Furthermore, the use of AI in medical education could reduce barriers and training costs, increasing access to high-quality educational tools regardless of location or institutional resources. Our study has several significant limitations. First, the use of Polish questions in an analysis conducted in a tool primarily in English could have introduced translational distortions and affected the accuracy of the model’s responses. Linguistic differences and nuances in terminology could have hindered the interpretation of the question content and, to some extent, reduced the effectiveness of the obtained results. Another limitation is the narrow scope of the topics; the questions concerned only balneology and physical medicine, which limits the transferability of the findings to other fields of medicine. Furthermore, only responses from one model (ChatGPT-5) were analyzed, without comparison to other systems (e.g., GPT-4, Gemini, and Claude), which prevents us from assessing the relative effectiveness of different large language models. Finally, confidence levels were assessed solely based on the model’s self-reported outputs, without independent validation in a clinical setting. Future studies should incorporate a broader range of questions, cross-model comparisons, and standardized translation procedures to enhance the reproducibility and reliability of results. Accordingly, AI should be regarded as a tool to support the learning process, rather than as a substitute for the expertise and experience of medical professionals.

The study results indicate that reasoning-based models are capable of effectively analyzing complex cases in balneology and physical medicine and may, in the future, find practical applications in medical education and clinical practice, particularly in the context of technical and cost-related limitations. With the growing integration of AI into medical training, generative AI systems may enhance learning efficiency and preparation for specialization examinations. Future research should focus on verifying the effectiveness of such models using real patient data, evaluating their impact on student performance, and developing strategies for integrating AI tools into both teaching and healthcare practice.

This study was conducted using GPT-5 on a single, randomly selected State Specialization Examination (PES) in Balneology and Physical Medicine (Spring 2024), obtained from the archives of the CEM in Łódź, Poland. The exam consisted of 120 multiple-choice questions, each with five options, only one of which was correct. Additionally, CEM excluded two questions as invalid, and their results were not included in the analysis. Appendix A presents the content of the questions translated into English along with the official CEM answers, GPT-5 chat answers, and the confidence level of the model’s answers. The questions were divided into two categories: clinical questions, which consisted of patient-based scenarios requiring symptom analysis, interpretation of test results, and diagnostic or therapeutic decision-making; and theoretical questions, assessing general knowledge and familiarity with treatment standards. Before the examination, the model was introduced to the rules, including the number of questions, response options, and the rule of one correct answer per question. Each answer generated by GPT-5 was compared with the official CEM key and recorded. After every response, the model was asked to rate its confidence on a 5-point scale: 1 - no confidence; 2 - low; 3 - moderate; 4 - high; and 5 - full confidence. All questions were provided in Polish to ensure alignment with the original exam materials. Statistical analysis was conducted using Microsoft Excel (Microsoft Corporation, Redmond, WA, USA) and GraphPad Prism (GraphPad Software, San Diego, CA, USA). The chi-square test was applied to compare the distribution of correct and incorrect answers between categories, while the Mann-Whitney U test was used to assess differences in confidence levels between correct and incorrect responses. A p-value < 0.05 was considered statistically significant. Statistical analyses included the chi-square test and the Mann-Whitney U test, both performed in Microsoft Excel. Because the investigators are located in different cities, collaboration between the research centers took place via online communication and document-sharing platforms, including Microsoft Teams (Microsoft Corporation), Facebook Messenger (Meta Platforms, Inc., Menlo Park, CA, USA), email, and Google Docs/Google Drive (Google, Inc., Mountain View, CA, USA). These tools supported real-time information exchange, collaborative document editing, and centralized storage of study materials. All manuscript sections prepared by different teams were peer-reviewed using the same platforms, ensuring consistency across contributions.