Evaluation of ChatGPT’s Performance in Making-Decision of Dialysis in Acute Kidney Injury

Abstract Purpose To evaluate the performance of ChatGPT in identifying patients in need of dialysis. Methods A total of 100 patients who presented with acute kidney injury and were treated with dialysis and without dialysis at the internal medicine clinic were retrospectively reviewed. Patient histories that included age, gender, comorbidities, blood pressure, body volume status, uremic symptom status, urine output status, and blood gas analysis, as well as measurements of urea and creatinine levels were created. These patient histories were inputted into the free version of ChatGPT 3.5, and we requested a clinical evaluation of the patients along with recommendations categorizing them as low, medium, or high risk for dialysis treatment. The responses from ChatGPT were compared with the actual dialysis status of the patients. Additionally, ChatGPT responses were evaluated and scored by two nephrologists who were unaware of the dialysis status. Results The sensitivity of ChatGPT in recommending patients' need for dialysis was calculated as 94%, 97% and 97% for ChatGPT 1, 2 and 3 answers, respectively. Specificity for ChatGPT responses 1, 2, and 3 was calculated as 81%, 76%, and 78%, respectively. (p<0.001) The mean clinical evaluation scores were 4.71 ± 0.4 and 4.67 ± 0.4, and treatment recommendation scores were 4.45 ± 0.7 and 4.39 ± 0.7 for Nephrologist 1 and Nephrologist 2. (p=0.002) (p<0.001) (respectively) Conclusion ChatGPT can be used as a decision support tool to identify patients who may need dialysis. Nevertheless, healthcare professionals should remain a step of the decision-making process at present.