A conversational agent framework for mental health screening: Design, implementation, and usability

The study introduces a chatbot system, entitled aiCARE, whose framework supports Likert-based screening for mental disorders. The goals were to accurately interpret users' free-text responses and correctly identify the corresponding Likert answer, reduce the frequency of sub-dialogues to clarify responses, and evaluate the chatbot's usability. The chatbot was trained on data collected in a non-AI phase (Phase 1; N = 274; Mage = 21.86, SD = 5.50, 67.8% women) and later tested in an AI-assisted data phase (Phase 2; N = 587; Mage = 21.56, SD = 5.56, 67.8% women). The performance of the AI models was high in all three surveys on which it was tested (i.e., The Patient Health Questionnaire - 9, General Anxiety Disorder - 7, The Posttraumatic Stress Disorder Checklist), F1 ranged from 0.70 to 0.73. The interaction between the conversational agent and the user was smooth at Phase 2, only in 4.64% of the cases the chatbot asked for clarifications. Moreover, the AI understood the responses that it received in 85.63% of the cases and was able to interpret free-text responses similar to human annotators with a Cohen's k varying from .76 to .85. The usability of the chatbot, measured with the Chatbot Usability Questionnaire, was optimal. In contrast to the training phase, the chatbot in Phase 2 was perceived as more engaging, less robotic, friendlier, easier to use, and as handling errors well, which could be indirectly due to the higher autonomy provided by the AI component. Limitations and future directions are discussed.