The Effectiveness of the Multimodal Language Model, Google Gemini 2.5 Pro, in Solving the Specialization Exam in Gynecology and Obstetrics.

Everyone knows ChatGPT (OpenAI, Inc., San Francisco, California, United States) [ 1 ], but it is not the only large language model on the artificial intelligence (AI) market. Alphabet Inc. (Mountain View, California, United States), which owns Google products, among others, is also active in this area. Gemini AI is an advanced artificial intelligence model developed by Google DeepMind and Google Research in response to the emergence of other AI models [ 2 ]. It is a versatile system designed for natural language processing, image analysis, coding, and many other tasks based on AI. Its main goal is to provide intelligent, precise, and contextual answers, as well as to support users in tasks requiring a creative and analytical approach. The first version of Gemini, available under the name Bard, was released in February 2023 [ 3 ]. After a short time, Google began the process of phasing out the name Bard in favor of Gemini, which was intended to emphasize a new direction in the development of AI technology, placing greater emphasis on multimodality and the ability to process different types of data [ 4 ]. Google Gemini is available in four versions: Ultra, Pro, Flash, and Nano [ 5 ]. Each of these versions has been designed for different requirements and applications, adapting to the needs of users depending on their goals and available resources. Gemini Pro, which is discussed in this study, seems to be the best model for scaling across a wide range of applications. Gemini 2.5 Pro was released on March 25, 2025. It is the latest flagship model from Google DeepMind, which combines advanced natural language processing capabilities with improved analytical abilities [ 6 ]. Unlike traditional chatbots, which work by predicting subsequent words, Gemini 2.5 actually "thinks" before answering, analyzes the context, draws conclusions, and checks its own reasoning, minimizing the risk of errors [ 7 ]. The use of advanced language models in medical education is a subject of growing interest among researchers, especially in terms of their potential to solve examination tests, such as the Państwowy Egzamin Specjalizacyjny (PES) or the State Specialization Examination [ 8 ]. The research published so far has pointed to significant achievements of AI models in this area. Błecha et al. documented that ChatGPT-4o successfully passed the Polish final exams, obtaining 71.43% on the PES in infectious diseases [ 9 ]. Furthermore, Sławińska B et al. achieved a score of 78.3% on the PES in ophthalmology in their study [ 10 ]. The AI model has not proven to be sufficiently reliable in all fields. Suwała et al. in their 2023 paper proved that the performance of ChatGPT in the internal medicine exam was insufficient and the required 60% threshold was not achieved [ 11 ]. AI is bringing revolutionary effects in medicine, changing every aspect of it, from diagnosis and treatment to the management of medical facilities. Its impact is already significant, and it will be even more transformative in the future. AI algorithms can analyze medical images, process vast amounts of laboratory and genetic data, identify disease patterns and markers, and even analyze images of tissue biopsies. Considering the above, the aim of this study was to verify the possibility of the multimodal Gemini pro passing the PES in obstetrics and gynecology. Furthermore, the study aimed to assess whether the model's confidence level corresponded with the actual accuracy of its answers. The analysis took into account both the substantive correctness of the answers provided and the subjective assessment of the confidence with which the model gave the answers. The success criterion was achieving a threshold of 60% correct answers. The data for the analysis came from the examination questions and the answer key, published by the Center for Medical Examinations (CEM) in Łódź [ 8 , 12 ].

According to the observations, Gemini 2.5 PRO passed the exam with a score of 96.63%, achieving 115 correct and four incorrect answers (Figure 1 , Table 1 ). Based on the data summarized in Table 2 , which compares the model's effectiveness, Gemini Pro correctly answered 91.67% of questions pertaining to clinical cases and 97.19% of theoretical questions. It was concluded that the difference between these outcomes did not reach statistical significance. (p=0.313; x²=1,015). The study showed that the effectiveness of the answer is not correlated with the model's level of confidence (p=0.0639; x²=3.431) (Table 3 ). Based on additional data, an analysis was performed comparing the model's confidence level (on a 1-5 scale) depending on whether the question concerned a clinical case or theoretical knowledge (Figure 2 ). The Gemini Pro model exhibited a similar level of confidence, regardless of whether it was solving complex clinical problems or answering a theoretical question (U = 305; p = 0.119). Statistical significance was not reached. *State Specialization Examination CEM: Center for Medical Examinations Data are presented as absolute numbers (n) and percentages (%). No statistically significant differences in response accuracy were observed between clinical and theoretical/other questions (χ² = 1.015; p = 0.313). A significance level of p< 0,05 was adopted for the analysis. No statistically significant differences in response accuracy were observed between clinical and theoretical/other questions (χ² = 3.431; p = 0.0639). A significance level of p< 0,05 was adopted for the analysis. Based on the data from Table 4 , it can be concluded that the score of 115 points (96.63%) achieved by the Gemini 2.5 PRO model not only significantly exceeds the passing threshold of 72 points, but also surpasses the mean score (88.3), the median (91.0), and even the maximum score (109 points) achieved by physicians taking the exam for the first time. This indicates that within the conducted simulation, the AI model demonstrated higher effectiveness than the top-performing of the 111 physicians who took the exam. The table presents the model's answers in relation to the correct answers, according to the official answer key published by the CEM in Łódź. For each question, the confidence level declared by the model was included. The model assessed each question on a 5-point scale. This scale is defined as follows: 1 - no confidence, 2 - low confidence, 3 - moderate confidence, 4 - high confidence, and 5 - full confidence. *State Specialization Examination The results of 111 physicians were analyzed. The mean test score was 88.3 with a standard deviation of 12.28. The median score (91.0) was higher than the mean, indicating a slightly left-skewed distribution of scores, with the majority of examinees achieving results above the average. The high reliability of the test, estimated using the Kuder-Richardson 20 formula (KR20 = 0.898), demonstrates the strong internal consistency of the measurement tool. The passing threshold was set at 72 points, and the percentage of individuals who did not meet this threshold was 9.01%.

In the present study, the model achieved a score of 96.63% (115 correct answers, four incorrect), thus significantly exceeding the passing threshold. Additionally, it demonstrated an effectiveness surpassing the average results of physicians taking the same exam for the first time. Statistical analysis revealed no significant difference in performance between clinical and theoretical questions (χ²=1.015; p=0.313). The model performed comparably well regardless of the question type, on both clinical case-based tasks and questions testing purely theoretical knowledge. This result represents significant progress compared to previous studies on the effectiveness of AI models on Polish and international medical exams. As recently as 2023, Suwała et al. demonstrated that the ChatGPT model was unable to achieve the required 60% threshold on the PES in Internal Medicine from the years 2013-2017 [ 11 ]. A 2022 study by Hanna et al. evaluated the performance of ChatGPT 4.0, ChatGPT 3.5, and Bard (an earlier version of Gemini) on the Family Medicine specialty certification exam [ 13 ]. The highest performance was demonstrated by ChatGPT 4.0, which achieved a score of 86.5%. Lower scores were recorded for ChatGPT 3.5 and Bard, both at 64.2%. All three AI models successfully passed the exam. In March 2024, a study published by Botross et al. reported that Bard obtained a score of 62.4% on the ophthalmology specialty certification exam [ 14 ]. The presented data unequivocally indicate a rapid and significant advancement in the capacity of AI to address complex medical problems. A key point where our results differ from previous studies is the issue of self-assessment, or confidence. A statistical analysis was conducted using the Mann-Whitney U test to examine how the model self-assessed the confidence of its answers on a scale from 1 (no confidence) to 5 (full confidence). As the calculated p-value is greater than the standard significance threshold (p<0.05), there is no statistically significant difference in the confidence levels between correct and incorrect answers (U=305, p=0.199). This means that Gemini Pro did not demonstrate a capacity for "metacognition"; it was unaware of when it was providing a correct answer. This is a significant difference compared to analyses of the GPT series models. For example, in the study by Suwała et al. evaluating GPT-3.5 on the Medical Final Examination (LEK), it was observed that the model showed higher confidence when providing correct answers [ 15 ]. Similarly, in the work by Bielówka et al., where ChatGPT 3.5 scored 45.38% on the specialization exam in pathomorphology, a correlation was shown between the model's confidence and the correctness of its answers [ 16 ]. The absence of this correlation in the case of Gemini Pro is a significant finding that requires further investigation, especially in the context of potential clinical use, where a model that is "confident in its errors" could pose a threat. Discrepancies in the results obtained in the cited works may stem from differences in the datasets on which the individual models were trained, as well as their processing capabilities. In a study conducted by Hen et al. comparing the performance of the Gemini, ChatGPT-4, and ChatGPT-3.5 models on the Taiwanese pulmonology exam from 2013-2023, the Gemini model performed slightly worse than ChatGPT-4 in terms of answer accuracy [ 17 ]. This may suggest differences in the knowledge levels between these models in specific domains. These observations are consistent with the results from a study on the German medical exam, where Geissler et al. also noted a slight advantage of GPT-4 over Gemini Pro [ 18 ]. This may suggest differences in the knowledge levels between these models in specific, narrow domains. Nevertheless, the overall trend is clear: newer versions of AI models are characterized by higher pass rates and better results than older models, as seen in the study by Humar et al., in which they evaluated the performance of ChatGPT on the Plastic Surgery In-Service Examination and compared it with the results of residents on a national scale [ 19 ]. In total, 1129 questions were included in the final analysis, and ChatGPT answered 630 (55.8%) of them correctly. To conduct a more thorough analysis, further observations and studies of these models across various medical fields are necessary. A critical evaluation of this study's methodology reveals several limitations that affect the interpretation of the results. First and foremost, a static evaluation method was used, with each question being posed only once. It was not investigated how multiple attempts or changes in the phrasing of the questions could have influenced the results, which is significant in the context of the probabilistic nature of the model's responses. The analysis also did not include a qualitative assessment of the errors, leaving it unknown whether they resulted from a lack of knowledge or a misinterpretation of the question. Moreover, the scope of the study was limited to a single Polish specialization exam from one session and a single language model, Gemini 2.5 Pro. This specificity narrows the generalizability of the findings to other examination systems, countries, or AI models. The potential risk that the training data may have included material overlapping with the exam content must also be considered, as this could have given the model an unfair advantage and inflated its scores. Finally, the observed lack of correlation between the model's confidence and the correctness of its answers is in itself a limitation for practical applications, casting doubt on its reliability.

The Gemini 2.5 PRO model achieved an encouraging result of 96.63% in this study, exceeding the passing threshold for the specialization exam in obstetrics and gynecology (spring 2025). The obtained result should be assessed as very good. Gemini not only outperformed the average physician taking the exam for the first time, but its score also surpassed that of the top-performing human candidate. Despite its impressive effectiveness, there are discrepancies between the model's confidence and its reliability. Further work is needed to correlate the model's confidence level with the answers given and thus minimize the risk of error. This is crucial for building trust in the model and its usefulness in all fields of medicine.

This study was designed to empirically verify the capability of the multimodal language model Gemini 2.5 PRO in solving the PES in gynecology and obstetrics. The official PES paper from the spring 2025 session, which is publicly available on the website of the CEM in Łódź [ 8 ], was used for the study. The examination initially consisted of 120 multiple-choice questions; however, one question was invalidated by the examination committee due to ambiguity, which resulted in a final set of 119 questions for the analysis. To prepare the model for the task, it was first instructed on the rules of the examination before the simulation began. These rules consist of solving a specific pool of multiple-choice questions. This means that the respondent is tasked with selecting the correct or best answer from a given set of options (distractors). Each question consists of a so-called "stem" (a question, sentence, or phrase) and a set of possible answers. The questions were then presented to the model sequentially by pasting them one by one into the chatbot. In the prompt for the question, we included an instruction for Gemini, which read as follows: "Based on your knowledge, answer the above question and rate your confidence level on a scale of 1 to 5, where 1 is no confidence and 5 means full confidence." We evaluated the model's results by comparing its answers, which we collected in a text file, with the official answer key available on the CEM website, and the primary success criterion was defined as achieving at least 60% (72 points) correct answers (see Appendices). Every interaction during this simulation was meticulously documented to ensure the integrity of the collected data. For the purpose of a more granular analysis, the 119 exam questions were categorized into two distinct groups: clinical questions and theoretical questions. The clinical group was comprised of 12 questions that required the model to interpret specific patient cases to make diagnostic or therapeutic decisions. The theoretical group consisted of the remaining 107 questions, which tested general and foundational knowledge in the field. In addition to assessing the correctness of each answer, a subjective measure of the model's confidence was recorded. After providing an answer, the model was prompted to rate its confidence on a five-point scale, where 1 indicated no confidence, 2 represented low confidence, 3 signified moderate confidence, 4 corresponded to high confidence, and 5 denoted full confidence. This process allowed for a secondary analysis of the correlation between the model's self-assessed confidence and the actual accuracy of its responses. All collected data were statistically analyzed using the Statistica software program (TIBCO Software Inc., Palo Alto, California, United States). The chi-squared test was specifically employed to compare the model's effectiveness between the clinical and theoretical question categories. To assess the relationship and differences in confidence levels between correct and incorrect answers, the Mann-Whitney U test was utilized. A p-value of less than 0.05 was established as the threshold for statistical significance in all tests. The research was conducted remotely by authors in different locations, who utilized collaborative tools such as Microsoft Teams (Microsoft Corporation, Redmond, Washington, United States), Zoom (Zoom Communications, Inc., San Jose, California, United States), Facebook Messenger (Meta Platforms, Inc., Menlo Park, California, United States), email, and Google Docs (Google LLC, Mountain View, California, United States) to ensure a comprehensive and peer-reviewed approach to the study.