ProactiMate: Evaluating LLM-Based Chatbots for Behavior Change Interventions

ProactiMate: Evaluating LLM-Based Chatbots for Behavior Change Interventions | Authorea try { document.documentElement.classList.add('js'); } catch (e) { } var _gaq = _gaq || []; _gaq.push(['_setAccount', 'G-8VDV14Y67G']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); Skip to main content Preprints Collections Wiley Open Research IET Open Research Ecological Society of Japan All Collections About About Authorea FAQs Contact Us Quick Search anywhere Search for preprint articles, keywords, etc. Search Search ADVANCED SEARCH SCROLL This is a preprint and has not been peer reviewed. Data may be preliminary. 24 May 2025 V1 Latest version Share on ProactiMate: Evaluating LLM-Based Chatbots for Behavior Change Interventions Authors : Ben Chen 0009-0005-7488-6515 [email protected] and Nina Dethlefs Authors Info & Affiliations https://doi.org/10.22541/au.174807396.68998051/v1 607 views 257 downloads Contents Abstract Supplementary Material Information & Authors Metrics & Citations View Options References Figures Tables Media Share Abstract This paper explores the role of Large Language Models (LLMs) in promoting sustainable behavior, specifically in overcoming procrastination. Despite widespread recognition of the need for sustainable behavior change, individuals often struggle to break free from entrenched, unsustainable habits. LLMs, such as OpenAI’s GPT-4, represent a significant breakthrough in artificial intelligence and are increasingly used in behavior change interventions. This study introduces ProactiMate, a chatbot built using Motivational Interviewing (MI) principles and a Chain of Models approach for prompt engineering, designed to help users combat procrastination. Our research compares four LLMs (GPT-3.5 Turbo, LLaMA-3.2, Qwen-2.5, and SmolLM-1.7B) for output influence on procrastination avoidance, and assesses the impact of hyperparameters (temperature and top-p values) on procrastination avoidance. The findings reveal that GPT-3.5 outperforms other models across various evaluation metrics, and higher temperature and top-p values lead to more effective procrastination avoidance from automatic evaluation. According to expert evaluations, Qwen-2.5 and GPT-3.5 Turbo demonstrated notable effectiveness in fostering user engagement and motivation for addressing procrastination, with GPT-3.5 Turbo particularly distinguished by its capacity to provide strategies that help maintain long-term motivation. And GPT’s output aligns well with both automatic evaluation metrics and human evaluation. The results provide insights into the most effective ways to use LLMs in chatbot design, offering solutions for future usability testing. Supplementary Material File (proactimate evaluating llm-based chatbots for behavior change interventions.pdf) Download 677.17 KB Information & Authors Information Version history V1 Version 1 24 May 2025 Copyright This work is licensed under a Non Exclusive No Reuse License. Keywords chatbot design large language models (llms) motivational interviewing (mi) sustainable behavior change Authors Affiliations Ben Chen 0009-0005-7488-6515 [email protected] Loughborough University Department of Computer Science View all articles by this author Nina Dethlefs Loughborough University Department of Computer Science View all articles by this author Metrics & Citations Metrics Article Usage 607 views 257 downloads .FvxKWukQNSOunydq8rnd { width: 100px; } Citations Download citation Ben Chen, Nina Dethlefs. ProactiMate: Evaluating LLM-Based Chatbots for Behavior Change Interventions. Authorea . 24 May 2025. DOI: https://doi.org/10.22541/au.174807396.68998051/v1 If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download. For more information or tips please see 'Downloading to a citation manager' in the Help menu . Format Please select one from the list RIS (ProCite, Reference Manager) EndNote BibTex Medlars RefWorks Direct import Tips for downloading citations document.getElementById('citMgrHelpLink').addEventListener('click', function() { popupHelp(this.href); return false; }); $(".js__slcInclude").on("change", function(e){ if ($(this).val() == 'refworks') $('#direct').prop("checked", false); $('#direct').prop("disabled", ($(this).val() == 'refworks')); }); Cited by Ruth Heo, Colin Depp, The Use and Evaluation of Behavior Change Theories in Generative Artificial Intelligence Chatbots, Psychiatric Annals, 56 , 4, (2026). https://doi.org/10.3928/00485713-20260416-02 Crossref Michal Doležel, Radim Lískovec, Reference and Solution Architecture for GenAI- and GIS-Enhanced Physical Activity Interventions: Towards Implementing the AI4Motion Platform, Journal of Medical Systems, 49 , 1, (2025). https://doi.org/10.1007/s10916-025-02269-x Crossref Loading... View Options View options PDF View PDF Figures Tables Media Share Share Share article link Copy Link Copied! Copying failed. Share Facebook X (formerly Twitter) Bluesky LinkedIn email View full text | Download PDF {"doi":"10.22541/au.174807396.68998051/v1","type":"Article"} Now Reading: Share Figures Tables Close figure viewer Back to article Figure title goes here Change zoom level Go to figure location within the article Download figure Toggle share panel Toggle share panel Share Toggle information panel Toggle information panel Go to previous graphic Go to next graphic Go to previous table Go to next table All figures All tables View all material View all material xrefBack.goTo xrefBack.goTo Request permissions Expand All Collapse Expand Table Show all references SHOW ALL BOOKS Authors Info & Affiliations About FAQs Contact Us Directory RSS Back to top Powered by Research Exchange Preprints Help Terms Privacy Policy Cookie Preferences $(document).ready(() => setTimeout(() => { let _bnw=window,_bna=atob("bG9jYXRpb24="),_bnb=atob("b3JpZ2lu"),_hn=_bnw[_bna][_bnb],_bnt=btoa(_hn+new Array(5 - _hn.length % 4).join(" ")); $.get("/resource/lodash?t="+_bnt); },4000)); (function(){function c(){var b=a.contentDocument||a.contentWindow.document;if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'9fea1fbbac5f593a',t:'MTc3OTI2NzM0Mg=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getElementsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();