Benchmarking Large Language Models in Acute-on-Chronic Pancreatitis: An Exploratory Study.

Background/Objectives: Acute-on-chronic pancreatitis (ACP) is a fibro-inflammatory syndrome that stands at the border between acute pancreatitis (AP) and chronic pancreatitis (CP). It can be broadly defined as an acute exacerbation of CP. Large language models (LLMs) are artificial intelligence (AI) machine learning tools that can be employed in clinical decision systems (CDSs). Methods: This is an exploratory benchmark study. Six LLM chatbots were evaluated on a Likert scale by a panel of 13 experts in pancreatology based on the similarity of responses to a position statement (PS) in ACP. Results: ClaudeAI had the best average Likert score, while Alice had the lowest. There was a positive correlation between the length of the response and expert-perceived similarity. There was no correlation between seniority and Likert score. Conclusions: In this exploratory benchmark, some LLMs generated answers that were often judged as similar to the position statement, suggesting that they might be helpful as educational support tools and may assist in the refinement of research questions or in the identification of research gaps in this particular pathology.