Evaluating a Large Reasoning Model’s Performance on Open-Ended Medical Scenarios

Abstract Large language models (LLMs) have emerged as a dominant form of generative artificial intelligence (GenAI) in multiple domains. In early 2025, DeepSeek R1 was released, which is a new large reasoning model (LRM) that includes CoT (CoT) reasoning, Mixture of Experts (MoE), and reinforcement learning. As these technologies continue to improve, evaluating the accuracy and reliability of LLMs and LRMs in medicine remains a crucial challenge. This paper reports on a follow-up study using DeepSeek R1 to evaluate medical scenarios contained in the MMLU-Pro benchmark, an enhanced benchmark designed to evaluate language understanding models across broader and more challenging tasks. In the previously reported study, the accuracy rate was 96% when multiple-choice MMLU-Pro answers were provided. In the current study, we evaluated DeepSeek R1 on 162 medical scenarios, but without multiple-choice answers provided. The overall accuracy was 92%. This approach mirrors a more realistic clinical scenario where the clinician must decide on the most likely diagnosis and differential diagnoses without any clues. Further research is necessary to determine how to deploy LRMs in clinical medicine, given their high accuracy rate, both with and without answers provided. Competing Interest Statement The authors have declared no competing interest. Funding Statement No funding Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes