Heart Fire: A Smart Watch-Based Musician-Listener Interaction System for Online Live-Streamed Concerts: A Pilot Study

Online live-streaming has become the new status quo in live music performances in the post-COVID-19 era. However, methods to enhance interaction between musicians and listeners at online concerts are yet to be adequately researched. In this pilot study, we propose Heart Fire, a system to promote musician-listener interaction that visualizes the listeners’mental states using a smart watch-based architecture. Accordingly, the listeners' heart rates are first measured using a Galaxy  smart watch and then processed into a real-time animation of a burning flame, whose intensity is dependent on the heart rate, using Azure Kinect and TouchDesigner. The feasibility of the proposed system was confirmed using an experiment involving ten subjects. Each subject selected two types of music—cheerful and relaxing. The BPM and energy of each song were measured, and each subject was asked to answer a questionnaire about the emotions they experienced before, during, and after listening to the songs. The results demonstrated that the proposed system is capable of visualizing audience response to music in real-time.