A Study on the Accuracy of Pre-Treatment Consultation Responses for Adult Orthodontic Patients Based on Large Language Models

Abstract This study compiled the 50 most common preoperative consultation questions from adult orthodontic patients through clinical observation. Responses were generated in new dialogue sessions using three large language models: Ernie Bot, ChatGPT, and Gemini. The answers were assessed across five dimensions: professionalism and accuracy, clarity and comprehensibility of language, personalization and specificity, completeness and thoroughness of information, and empathy and humanistic care. The results demonstrated that Technical Accuracy(TA) was rated as reliable (44%, 78%, 74%); Clarity and Comprehensibility (CC) was also found reliable (62%, 44%, 46%); Personalization and Relevance (PR) and Information Completeness (IC) were reliable as well (58%, 70%, 70%) and (74%, 82%, 66%) respectively; Empathy and Human-Centeredness (EHC) was considered moderately reliable (64%, 54%, 46%). The three AI language models showed moderate to reliable performance in terms of clarity, personalization, and completeness. However, they fell short in the dimension of humanistic care. Therefore, it can be concluded that AI language models present potential benefits for preoperative consultations. Nonetheless, given the complex individual needs of patients in clinical settings, further optimization of AI models is essential, and clinical consultations should be prioritized when necessary.