Using ChatGPT to Develop the Statistical Analysis Plan for a Randomized Controlled Trial: A Case Report

Abstract Background Statistical analysis plans (SAPs) outline the way data will be collected, preprocessed and analyzed, and should be developed during the design phase of the study. However, veterinary researchers often have limited access to biostatistical resources to develop statistical analysis plans. That access can be improved using natural language artificial intelligence chat services (chatbots), such as ChatGPT, which provide ways to access biostatistical expertise that can be understood by nonstatistician researchers. The aim of this study was to demonstrate a ChatGPT query protocol that produced a statistical analysis plan for randomized controlled trial comparing orthopedic surgical methods. Implementation Our overall approach was to interact with the chatbot ChapGPT as we would with a biostatistician planning a clinical trial’s data preprocessing and statistical analysis. Collaborations with biostatisticians are iterative, where the investigator initiates the statistical conversation with an overview of the project, and then the biostatistician and investigator go back and forth to refine the SAP with a series of more detailed questions and responses. Similarly, we iteratively queried ChatGPT after reviewing its responses. Results A good SAP was developed after four queries, which took 15 minutes. Researchers with less statistical experience may require more queries and time. ChatGPT produced an acceptable SAP that could be understood by nonstatistician researchers. Discussion ChatGPT 3.5 is free, fast, and interactive. Chatbot-generated SAPs might improve the reproducibility of veterinary research because they could be included as supplementary material. Additionally, SAPs are useful for responding to statistical reviewers because SAPs anticipate reviewer questions. Moreover, key analysis elements are often missing from study reports, and SAPs could serve as checklists to ensure that all statistical steps are performed. Finally, even when biostatistical support is available, a chatbot-generated SAP could provide a framework for a biostatistician, and speed up the research process by weeks. Conclusions When used iteratively, ChatGPT provides a sound framework for the steps in the statistical analysis of randomized controlled trials. We anticipate that it would be useful for other common study designs, such as risk factor studies, survival analyses, systematic reviews, and observational cohort studies.