A Unified Platform for At-Home Post-Stroke Rehabilitation Enabled by Wearable Technologies and Artificial Intelligence

A Unified Platform for At-Home Post-Stroke Rehabilitation Enabled by Wearable Technologies and Artificial Intelligence | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article A Unified Platform for At-Home Post-Stroke Rehabilitation Enabled by Wearable Technologies and Artificial Intelligence Luigi Occhipinti, Chenyu Tang, Ruizhi Zhang, Shuo Gao, Zihe Zhao, and 19 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5538299/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract At-home rehabilitation for post-stroke patients presents significant challenges, as continuous, personalized care is often limited outside clinical settings. Additionally, the absence of comprehensive solutions addressing diverse rehabilitation needs in home environments complicates recovery efforts. Here, we introduce a smart home platform that integrates wearable sensors, ambient monitoring, and large language model (LLM)-powered assistance to provide seamless health monitoring and intelligent support. The system leverages machine learning enabled plantar pressure arrays for motor recovery assessment (94% classification accuracy), a wearable eye-tracking module for cognitive evaluation, and ambient sensors for precise smart home control (100% operational success, < 1 s latency). Additionally, the LLM-powered agent, Auto-Care, offers real-time interventions, such as health reminders and environmental adjustments, enhancing user satisfaction by 29%. This work establishes a fully integrated platform for long-term, personalized rehabilitation, offering new possibilities for managing chronic conditions and supporting aging populations. Physical sciences/Engineering/Biomedical engineering Physical sciences/Engineering/Electrical and electronic engineering Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 I. Main Stroke is the third leading cause of disability worldwide, affecting more than 101 million people [1, 2]. Survivors often experience motor impairments (60–80%), cognitive deficits (20–30%), and speech difficulties (30–50%), which significantly compromise their independence and quality of life [3, 4]. Post-stroke recovery is not only a prolonged process but also a resource-intensive one, imposing significant economic and caregiving burdens on families and healthcare systems—a challenge exacerbated by global aging [5]. For many patients, the home becomes a critical environment for rehabilitation, as opportunities for continuous and personalized care are limited outside of clinical settings [6]. An ideal solution would resemble the intelligent, adaptive environments often envisioned in science fiction: a system that uses multi-modal, multi-dimensional sensing to continuously monitor patients’ physical and physiological states, while leveraging artificial intelligence to provide real-time health assessments and assistance [7, 8]. Recent advancements in wearable sensors and artificial intelligence (AI) technologies have brought this vision within reach [9, 10, 11, 12]. Combined with AI methods, wearable devices such as force sensors, accelerometers, EMG sensors, and eye trackers have enabled tracking of motor recovery and provided valuable insights into cognitive function [13-23]. Assistive tools, including robotic aids and smart home devices, have been developed to address specific daily needs [24, 25, 26]. However, the comprehensive solution envisioned is still absent. To unlock the full potential of at-home rehabilitation, we step further to a unified, patient-centered system capable of navigating the complexities of real-world scenarios. Here, we report a smart home system specifically designed for long-term, at-home rehabilitation of post-stroke patients, integrating health monitoring and assistive functionalities into a single platform (Fig. 1). By leveraging multi-sensor fusion, the system comprehensively addresses the needs of patients with post-stroke impairments. For rehabilitation monitoring, our plantar pressure array, coupled with a machine learning model, evaluates motor recovery, achieving a classification accuracy of 94.1% across three rehabilitation states. A wearable eye-tracking module extracts key indicators of cognitive function, while ambient sensors such as cameras and microphones, in collaboration with the eye-tracking module, enable seamless and precise smart home control (100% operational success rate with a latency of <1 s). This multi-sensor collaborative design ensures accessibility for a diverse range of users, allowing them to choose the most suitable interaction modality based on their specific needs. Additionally, we introduce an autonomous assistive agent, Auto-Care, powered by a large language model (LLM), which analyzes multi-modal data to provide timely interventions such as health reminders, environmental adjustments, or caregiver notifications, increasing user satisfaction by 29% compared to scenarios without the agent. This system represents the first fully integrated solution for simultaneous health monitoring and intelligent assistance in post-stroke home rehabilitation, offering a pathway toward comprehensive, patient-centered management. Furthermore, it holds potential for broader applications in other chronic conditions, such as amyotrophic lateral sclerosis (ALS) and Parkinson’s disease, and aging populations. II. Results The multimodal rehabilitation system As shown in Fig. 1a, the platform integrates wearable devices, including plantar pressure insoles (SFig. 1-4), wristband module (SFig. 5, 6), and eye-tracking module (SFig. 7), alongside ambient sensors such as cameras and microphones, to enable comprehensive, round-the-clock monitoring of patients [27, 28, 29]. This multimodal sensing system collects a full spectrum of patient information, providing intelligent assessments of rehabilitation progress and offering daily assistance to support independent living. The core concept of the platform is realized through the IoT architecture depicted in Fig. 1b (SFig. 11), which consolidates data from all sensing modalities into a local host server at home. Wearable devices and ambient sensors transmit data locally via Bluetooth Low Energy (BLE) and WiFi protocols, ensuring seamless communication, minimal latency, protection of users’ data. The host server processes the aggregated data in real time, transforming it into actionable outputs for health monitoring and assistive decision-making. To evaluate the platform’s capability in tracking motor recovery, 20 post-stroke patients with varying degrees of motor impairments and diverse post-stroke complications including hemiplegia, knee valgus, and foot inversion were recruited. Patients were stratified based on their motor function scores obtained by using the Fugl-Meyer Assessment (FMA) scale [30] into three rehabilitation levels: mild, moderate, and severe. Plantar pressure data from a 48-channel sensor matrix were recorded during routine walking tasks, capturing detailed gait dynamics across different recovery stages. As shown in Fig. 2a, patients in the mild rehabilitation level exhibited gait signals resembling those of healthy individuals, characterized by consistent amplitude and symmetry, indicating effective weight distribution and propulsion [29, 31]. In contrast, patients in the moderate rehabilitation level showed irregular oscillations and reduced symmetry, reflecting instability during gait phases. Those in the severe rehabilitation level, such as individuals with left-sided hemiplegia, exhibited diminished signals on the affected side and exaggerated signals on the unaffected side, indicative of compensatory mechanisms [32]. These distinctive signal patterns across recovery states underpin the machine learning model’s ability to objectively decode and monitor rehabilitation progress. Fig. 2b shows the comparison of eye-tracking patterns between subjects with and without cognitive impairments during interactions with a smart light. Among the 20 participants, 4 were identified by clinicians as exhibiting cognitive impairments based on professional evaluations of their behavior and neurological assessments. While the limited sample size of cognitive impairment cases does not allow for the development of a statistically robust tracking model, the collected data still provide valuable insights for clinicians via the IoT system. For subjects without cognitive impairment, gaze trajectories during interaction were precise and efficient, with fixations rapidly converging on the target smart light, as reflected by the compact heatmap. In contrast, the subject with cognitive impairment demonstrated dispersed and irregular gaze patterns, with frequent distractions and prolonged fixations on irrelevant objects before locating the target. The corresponding heatmap exhibits a broad, scattered distribution, consistent with delayed visual attention and impaired decision-making, hallmark traits often associated with cognitive deficits. Additionally, the platform recorded other statistical metrics, such as blink frequency and duration patterns, providing further indicators of cognitive function (SFig. 8, 9). These data, aggregated through the IoT system, can be securely shared with clinicians to analyze, offering valuable supplementary information for understanding the patient’s cognitive condition and guiding personalized interventions. Motor rehabilitation states monitoring via plantar pressure insoles and deep learning To accurately track motor impairment recovery stages, we collected walking data from the subjects and segmented it into 5-second samples to construct the dataset. Among the 20 participants, 6 were annotated as being in the severe, 7 in the moderate, and 7 in the mild rehabilitation level. Fig. 3a and Fig. 3b visualize key statistical characteristics of the gait data. Specifically, Fig. 3a displays the coefficient of variation (CV) of plantar pressure, which quantifies the variability in foot pressure relative to the mean pressure, with higher CV values reflecting greater instability in walking. Patients in the severe stage exhibit significantly higher CV compared to those in the moderate and mild stages, indicating less consistent gait dynamics. Fig. 3b highlights the asymmetry in pressure distribution and stance phase ratio between the left and right feet, with more severe cases showing pronounced imbalances. This reflects the underlying biomechanical challenges, where motor impairments disrupt load distribution and timing between the two legs. Fig. 3c outlines our deep learning pipeline for decoding rehabilitation states. The 48-channel plantar pressure signals from each foot are converted into 224 × 224 two-dimensional heatmaps, which are then fed into a convolutional neural network (CNN) to encode spatiotemporal gait features. This 2D transformation allows the model to capture the spatial relationships between channels while preserving temporal dynamics, a critical aspect for distinguishing gait patterns across recovery stages. The encoded features from both feet are subsequently processed by a multi-layer perceptron (MLP) classifier to decode the patient's rehabilitation status. Performance comparisons of various baseline models as gait feature encoders are shown in Fig. 3d, where ResNet-101 outperformed alternatives with the highest accuracy of 94.1%, supporting its selection as the optimal encoder (optimal hyperparameters detailed in STable 1). The confusion matrix in Fig. 3e demonstrates robust classification across all rehabilitation states, with minimal misclassification errors. Furthermore, the encoder’s output feature representations, visualized using UMAP in Fig. 3f, reveal clear clustering of the three rehabilitation states, underscoring the model's ability to differentiate between mild, moderate, and severe motor impairment stages effectively. This pipeline provides a robust, end-to-end, and data-driven approach to monitor recovery progress in post-stroke patients. Smart home control based on multi-sensor fusion To address the challenges of monitoring stroke patients in home settings, we developed a real-time scene detection and human action recognition system based on video streams (Fig. 4a). This system employs a lightweight neural network architecture to achieve efficient local computation while maintaining patient privacy, balancing accuracy and computational efficiency. Scene images are extracted from video streams captured by home cameras and processed using a fine-tuned YOLOv8n model [33]. The model identifies the patient and household objects with corresponding confidence scores. Objects with scores exceeding a predefined threshold are retained, and their spatial relationships are analyzed to infer the current home environment. For human action recognition, patient images are processed using the open-source MediaPipe model to extract pose landmarks, which are then converted into normalized 3D coordinates. These coordinates serve as input for an MLP action classification model, enabling the detection of typical human actions with an inference latency of less than 50 ms. On a self-collected test dataset, the model achieved an accuracy of 99.3% (Fig. 4b, SFig. 10). By combining scene and action data, the system provides real-time video feedback to a host device while storing data for retrospective analysis, facilitating long-term rehabilitation monitoring. For patients with speech impairments, the system extends its functionality to support multimodal interaction by integrating audio and eye-tracking data. Patients with partial speech capability can issue voice commands through a microphone, while those with severe speech impairments can rely on gaze direction and blinking patterns captured by the eye tracker (Fig. 4c). These signals are processed to generate control commands for smart home devices. This approach provides a seamless and intuitive interface for patients with diverse disabilities, fostering independence and improving their quality of life during at-home rehabilitation. The IoT-based architecture ensures reliable communication with smart devices, enabling adaptive control of the home environment to meet individual needs (SVideo). LLM agent for autonomous rehabilitation management To overcome the limitations of patients interacting with the platform solely based on subjective needs, we embedded an autonomous health management agent, Auto-Care, powered by GPT-4o Mini API. This agent operates continuously, analyzing multimodal data streams 24/7 to intelligently detect and address various patient needs (SVideo). For example, as shown in Fig. 5a, during gait training (point 1), the agent detected rising heart rate and temperature coupled with decreasing heart rate variability (HRV), prompting it to recommend hydration, pause training, and activate air conditioning to maintain comfort. At point 2, when a fall was detected, the agent used a microphone to check the patient’s status and, upon confirming the need for assistance or receiving no response, alerted a caregiver. At point 3, as ambient light levels decreased, the agent adjusted the smart lighting in the dining room based on the patient’s location to ensure adequate illumination. The effectiveness of Auto-Care was further enhanced through prompt optimization, as shown in Fig. 5b. Adding Chain-of-Thought reasoning and pre-defined intervention demos to the prompts significantly improved user satisfaction [34]. Multimodal data were downsampled to 1-minute intervals before being input into the agent, ensuring computational efficiency without compromising real-time responsiveness. Fig. 5c shows that a six-minute data context provided the optimal balance for decision-making accuracy and computational efficiency. Overall, as demonstrated in Fig. 5d, the integration of the Auto-Care Agent into the platform significantly enhanced patient outcomes across multiple dimensions, including reduced psychological burden, improved operational efficiency, and increased overall satisfaction (evaluation criteria detailed in STable 2). Compared to scenarios where the platform was not used, users reported a 67% improvement in overall satisfaction, with an additional 29% increase upon the integration of the Auto-Care Agent. These results highlight the potential of the platform and agent to transform at-home rehabilitation, providing continuous, intelligent support tailored to individual patient needs. III. Discussion In this work, we present a smart home system specifically designed to address the multifaceted rehabilitation needs of post-stroke patients in a home environment. The system’s integrated platform combines advanced health monitoring and assistive functionalities, demonstrating exceptional performance in real-world scenarios. Comprehensive evaluations confirm its efficacy: the plantar pressure array and machine learning model achieved 94.1% accuracy in classifying three rehabilitation states, while the wearable eye-tracking module and ambient sensors enabled seamless smart home control with a 100% success rate and latency of < 1 s. The introduction of Auto-Care, an LLM-powered autonomous agent, further enhances the system’s functionality, improving user satisfaction by 29% through timely and intelligent interventions. These features collectively position the system as a groundbreaking solution for post-stroke rehabilitation, offering continuous, personalized care and assistance. Future directions will focus on advancing the system’s adaptability, scalability, and inclusivity. First, expanding its application to other chronic conditions and diverse user populations, including aging individuals and those with neurodegenerative diseases, will enhance its broader relevance. Second, improving the system’s robustness and interoperability with external assistive devices, such as robotic aids and exoskeletons, will further enrich its capabilities. Lastly, optimizing its computational efficiency through edge computing will reduce power consumption and latency, enabling seamless operation and ensuring privacy in at-home environments. Looking ahead, the system has the potential to transform long-term post-stroke care by improving both physical and psychological well-being. Continuous, personalized monitoring and intelligent assistance empower patients to regain independence, enhance social interaction, and improve quality of life. Additionally, the system’s ability to collect and analyze long-term, multi-modal data opens new possibilities for predicting stroke progression, recovery trajectories, and even the risk of secondary strokes. By identifying subtle patterns in motor performance, cognitive behaviors, physiological signals, and environmental factors, the system could enable proactive, personalized interventions to mitigate future risks and optimize recovery strategies. These capabilities position the system not only as a rehabilitation tool but as a predictive and comprehensive health management platform for post-stroke care. IV. Methods Fabrication of the plantar pressure insole To detect plantar pressure, we developed a custom insole equipped with a 4 × 12 resistive pressure sensor array, comprising 48 sensing points with an average sensor density of 0.23 sensors/cm². The structural details and sensor dimensions are reported in the figure. The topmost layer of the insole consists of a polyethylene terephthalate (PET) protective film, beneath which lies a layer of copper row and column electrodes etched onto a polyimide (PI) substrate. An FSR (force-sensitive resistor) graphite layer is placed between the electrode layers to form the resistive sensing elements. This flexible design ensures that the insole can withstand frequent bending during walking without losing functionality. At just 100 µm thick, the insole provides a comfortable wearing experience without causing discomfort. To process the pressure data, we designed a custom resistive array detection circuit with the HC32F460 microcontroller, based on the ARM Cortex-M4 architecture operating at 200 MHz. This MCU offers robust computational capability for processing large volumes of pressure data. A low-dropout regulator (LDO) reduces the input voltage from 5.6 V to 5 V, ensuring a stable power supply to the ADC for precise signal conversion. The circuit also includes an integrated TP4054 lithium battery charging chip, enabling recharging via a Micro-USB interface. For high-speed data communication, all modules utilize fast GPIO and protocols such as SPI and I2C. To enable wireless data transmission, the circuit integrates a CH9141 Bluetooth module that communicates with the MCU via the USART protocol. This efficient and flexible design supports real-time plantar pressure monitoring, with data transmission capabilities that are robust and optimized for rehabilitation scenarios. Fabrication of the wristband A custom-designed wireless wristband was developed to enable efficient, continuous acquisition and transmission of physiological and environmental data within the IoT system, ensuring seamless integration and modularity for the specific multimodal sensing needs of the platform. Unlike commercial solutions, the custom design ensures full compatibility with the IoT architecture and allows precise customization of sensing modalities to guarantee integrated functionality. The wristband integrates six functional modules: an STM32L412 microcontroller for system control and coordination of the overall sensing and data management functions, a CH9141 BLE module for bidirectional data and instruction transmission, a power management module, and three sensing modules—MAX30101 for PPG signal acquisition, AS7341 for environmental light detection, and MTS4B for temperature measurement. Each sensing module operates independently, collecting data at predefined sampling rates and temporarily storing raw signals in internal registers. The STM32L412 microcontroller controls system functionality, polling each module at regular intervals via the I2C protocol and transmitting aggregated data to the host computer via the CH9141 Bluetooth module. The wristband is powered by a 4.2V rechargeable lithium battery, charged through a TP4054 linear charger and protected against overcharging, over-discharging, and overcurrent using a DW01 chip. Real-time battery status is monitored by a BQ27220 coulometer, while voltage regulation is managed by XC6206P332MR and XC6206P182MR LDOs (3.3V and 1.8V, respectively) and a ME2188A50XG linear regulator delivering 5V for the MAX30101’s integrated LED. To ensure reliability and user comfort during daily wear, the wristband features a compact six-layer PCB with double-sided component mounting, measuring 40 × 30 × 1.6 mm. This design achieves a balance between unobtrusiveness and robust functionality, facilitating comfortable long-term use while maintaining consistent performance. Fabrication of the wearable eye tracker A custom head-mounted wireless eye tracker was developed for real-time gaze tracking and environmental scene analysis, tailored for applications in rehabilitation and smart home interaction. The system integrates two near-infrared (IR) cameras, each equipped with four edge-mounted IR LEDs, for precise pupil and corner-of-eye detection, and a forward-facing visible-light camera for capturing the wearer's environmental context. All cameras utilize the IMX258 image sensor with an 80-degree fixed-focus lens, providing high-resolution (12 MP) imagery at 30 FPS. Centralized processing is performed by an OrangePi CM5 module, which incorporates the RK3588S SoC (quad-core Cortex-A76 at 2.4 GHz and quad-core Cortex-A55), ensuring efficient real-time data handling. The module is powered by an RK806-1 power management IC, supporting stable operations for computationally intensive tasks. Wireless data transmission is facilitated by a CDW-20U5622 WiFi module, enabling seamless integration with the IoT system. The device is powered by a compact 5V 4A lithium battery, ensuring portability and extended use. Key eye features, such as pupil center and eye corner coordinates, are extracted in real time by the processing unit, which computes gaze coordinates using calibration data. The visible-light camera data is synchronized with gaze coordinates, providing a robust mechanism for environmental interaction. The system achieves high accuracy and reliability in dynamic scenarios, supporting diverse applications such as rehabilitation monitoring, assistive device control, and cognitive assessments. IoT Framework for Multimodal Data Integration To enable seamless integration and processing of multimodal data in the home environment, we designed a hierarchical IoT architecture comprising the following layers (SFig. 11): Sensor Layer This layer includes all data collection devices responsible for sensing user states and environmental conditions. Key components are wearable eye-tracking devices, wristbands, plantar pressure insoles, and Hikvision DS-2SC2Q133MW cameras (integrated with microphones and speakers). Data Transmission Layer : Communication across devices utilizes three main protocols: BLE, HTTP, and MiIO. The wristbands and insoles communicate with the gateway using BLE 4.2 via Bluetooth modules. The eye-tracking devices and cameras connect to the gateway over a WiFi network using the HTTP protocol. Smart home devices, such as lights and air conditioners, use the MiIO protocol over WiFi for communication. Data Processing Layer A custom gateway software is deployed on a host device equipped with both WiFi and BLE modules. This software aggregates data streams from all sensors, processes multimodal data fusion, and distributes control commands to connected devices. The host device handles real-time data synchronization and processing to ensure consistent and actionable outputs across modalities. Endpoint Layer This layer consists of smart home devices such as smart TVs, air conditioners, and table lamps, which are controlled via the MiIO protocol. Time Synchronization Server A local network time protocol (NTP) server is set up on the host device to ensure precise time synchronization for all collected data. During data processing at the gateway, timestamps generated from the NTP-synchronized server are embedded in the frame headers to maintain temporal alignment across all modalities. This framework ensures efficient, synchronized communication and integration of data from diverse sensing devices, enabling robust multimodal monitoring and interaction within the smart home rehabilitation system. Plantar pressure data acquisition We conducted a motor impairment study involving 20 stroke patients (mean age: 51.4 ± 9.8 years; 14 males, 6 females) who were recruited in compliance with the Ethics Committee approval by the Committee for Medical Research Ethics at the First Hospital of Shijiazhuang City, China (assigned project number of 2020036). All participants provided written informed consent prior to enrollment. Patients were instructed to walk naturally on a flat surface while wearing plantar pressure insoles under the supervision of medical professionals. Continuous plantar pressure signals were recorded during the walking sessions. Following the data collection, the patients’ motor recovery status was assessed using the Fugl-Meyer Assessment (FMA) scale, a clinically validated tool for evaluating motor impairment recovery. Based on their FMA scores, patients were categorized into three levels of motor impairment recovery: mild (FMA score ≥ 85), moderate (FMA score 50–84), and severe (FMA score < 50). The plantar pressure insoles captured data at a sampling frequency of 200 Hz. For analysis, continuous signals from both feet were segmented into five-second intervals, with each interval constituting a single sample. A total of 1,543 gait samples were collected across the 20 participants (80% were selected as the training set, while 20% formed the test set), providing a robust dataset for subsequent analysis of motor recovery patterns. Software environment for motor impairment monitoring model training Signal preprocessing was performed on a MacBook Pro equipped with an M1 Max CPU. Network training was conducted using Python 3.8.13, Miniconda 3, and PyTorch 2.0.1 in a performance-optimized environment. Training acceleration was enabled by CUDA on NVIDIA 4090 GPU. The smart home control system The smart home control system combines lightweight neural networks and wearable eye-tracking technology to achieve efficient, privacy-preserving, real-time interaction in home environments. For scene detection and action recognition, a fine-tuned YOLOv8n model was employed to analyze video streams from a Hikvision DS-2SC2Q133MW camera. This lightweight model, with 3.2M parameters, achieves near 100% accuracy for detecting human actions (e.g., walking, sitting, standing, and falling) and household objects (e.g., sofas, lights, and TVs) while maintaining an inference time of less than 100 ms on a CPU. Action classification leverages MediaPipe for extracting 3D normalized pose coordinates from images and a compact 3-layer MLP with 0.7M parameters for action recognition, achieving 99.3% accuracy with an inference time below 50 ms. Wearable eye trackers further enable intuitive control of smart home devices. These devices capture infrared pupil images and field-of-view (FoV) images to compute real-time gaze coordinates. After a nine-point calibration process using least-squares fitting, gaze points are mapped onto detected household objects. A decision is made if the gaze is continuously fixed on an object across five consecutive frames, prompting audio feedback like “TV detected, please issue a command.” For patients with speech capabilities, commands are captured via a microphone, processed through Whisper-tiny (39M parameters) [ 35 ], and translated into device actions via MiIO protocol. For patients with severe speech impairments, eye gestures and blinks are detected to enable interaction. Gaze direction is classified by analyzing real-time pupil positions, while blinks are identified by tracking closed-eye intervals using statistical thresholds derived from calibration data. These signals are translated into control commands, allowing comprehensive interaction with smart home devices through a combination of gaze, speech, and physical gestures, optimized for accessibility and efficiency. Design of the Auto-Care agent Auto-Care utilizes the GPT-4o Mini API to analyze multimodal patient data streams and deliver real-time, context-aware interventions. Prompts are dynamically generated to include a six-minute context window, summarizing recent trends and events. CoT reasoning is embedded to enable stepwise analysis of patient status and recommended actions, such as pausing rehabilitation, suggesting hydration, or adjusting environmental conditions. The API is configured with a response token limit of 200 to ensure concise outputs, and a temperature setting of 0.7 balances variability and reliability. Predefined templates and multi-step reasoning ensure robust and contextually relevant decisions across diverse rehabilitation scenarios. Feedback from real-world applications continuously refines prompt structures, enhancing precision and system performance. Declarations Data availability The datasets supporting this study will be available from the GitHub repository before publication. Code availability The code supporting this study will be available from the GitHub repository before publication. In relation to the data and code supporting the study, these are available and accessible from the following repository: https://github.com/tcy21414/Smart-Home-Platform-for-Stroke-Rehabilitation Acknowledgments S.G. acknowledges funding from the National Natural Science Foundation of China (62171014), and Beihang Ganwei Project (JKF-20240590). H.Z. acknowledges funding from The Royal Society Research Grant (RGS\R2\222333), the Engineering and Physical Sciences Research Council (EPSRC) Grant (13171178 R00287), and the European Innovation Council (EIC) under the European Union’s Horizon Europe research and innovation program (grant agreement No. 101099093). L.G.O. acknowledges funding from The British Council (contract No. 45371261) and the EPSRC (grants No. EP/K03099X/1, EP/L016087/1, EP/W024284/1, EP/P027628/1). C.T. was supported by Endoenergy Systems (grant No. G119004) and Haleon (CAPE partnership grant No. G110480), W.Y. was supported by Pragmatic Semiconductor (grant No. G117793) and Haleon (CAPE partnership grant No. G110480). References Prust ML et al (2024) Addressing disparities in the global epidemiology of stroke. Nat Reviews Neurol 20:207–221 Feigin VL et al (2023) Pragmatic solutions to reduce the global burden of stroke: a World Stroke Organization–Lancet Neurology Commission. Lancet Neurol 22:1160–1206 Tsao CW et al (2022) Heart disease and stroke statistics—2022 update: a report from the American Heart Association. Circulation 145:e153–e639 Lo JW et al (2022) Long-term cognitive decline after stroke: an individual participant data meta-analysis. Stroke 53:1318–1327 Scott CA et al (2022) Diverging temporal trends in stroke incidence in younger vs older people: a systematic review and meta-analysis. JAMA Neurol 79:1036–1048 Cramer SC et al (2019) Efficacy of home-based telerehabilitation vs in-clinic therapy for adults after stroke: a randomized clinical trial. JAMA Neurol 76:1079–1087 Bivard A et al (2020) Artificial intelligence for decision support in acute stroke—current roles and potential. Nat Reviews Neurol 16:575–585 Mouridsen K et al (2020) Artificial intelligence applications in stroke. Stroke 51:2573–2579 Tang C et al (2024) A roadmap for the development of human body digital twins. Nat Reviews Electr Eng 1:199–207 Vaghasiya JV et al (2023) Wearable sensors for telehealth based on emerging materials and nanoarchitectonics. npj Flex Electron 7:26 Chen C et al (2023) Digital health for aging populations. Nat Med 29:1623–1630 Tang C et al (2022) Wearables-based multi-column neural network for human activity recognition. IEEE J Biomedical Health Inf 27:339–350 Lin F et al (2016) Smart insole: A wearable sensor device for unobtrusive gait monitoring in daily life. IEEE Trans Industr Inf 12:2281–2291 Chen J et al (2022) Plantar pressure-based insole gait monitoring techniques for diseases monitoring and analysis: a review. Adv Mater Technol 7:2100566 Rusu L et al (2021) Plantar pressure and contact area measurement of foot abnormalities in stroke rehabilitation. Brain Sci 11:1213 Pradhan GB et al (2024) A wearable, self-sustainable, and wireless plantar pressure and temperature monitoring system for foot ulceration prognosis and rehabilitation. Sens Actuators A: Phys 379:115985 Adans-Dester C et al (2020) Enabling precision rehabilitation interventions using wearable sensors and machine learning to track motor recovery. npj Digit Med 3:121 Patel S et al (2010) A novel approach to monitor rehabilitation outcomes in stroke survivors using wearable technology. Proceedings of the IEEE 98, 450–461 Bisio I et al (2019) When eHealth meets IoT: A smart wireless system for post-stroke home rehabilitation. IEEE Wirel Commun 26:24–29 Zadka A et al (2024) A wearable sensor and machine learning estimate step length in older adults and patients with neurological disorders. npj Digit Med 7:142 Forssman L et al (2017) Eye-tracking-based assessment of cognitive function in low-resource settings. Arch Dis Child 102:301–302 Tao L et al (2020) Eye tracking metrics to screen and assess cognitive impairment in patients with neurological disorders. Neurol Sci 41:1697–1704 Skaramagkas V et al (2021) Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Rev Biomed Eng 16:260–277 Abedi A et al (2024) Artificial intelligence-driven virtual rehabilitation for people living in the community: A scoping review. npj Digit Med 7:25 Chen X et al (2022) An IoT and wearables-based smart home for ALS patients. IEEE Internet Things J 9:20945–20956 Arntz A et al (2023) Technologies in home-based digital rehabilitation: scoping review. JMIR rehabilitation Assist Technol 10:e43615 Zhao Z et al (2024) A Strabismus Widespread Screening Method Based on Wearable Eye Tracker. IEEE BioSensors Conference (BioSensors). IEEE, 2024 Wang Z et al (2022) A visual feedback supported intelligent assistive technique for amyotrophic lateral sclerosis patients. Adv Intell Syst 4:2100097 Chen J et al (2021) A flexible insole gait monitoring technique for the internet of health things. IEEE Sens J 21:26397–26405 Gladstone DJ et al (2022) The Fugl-Meyer assessment of motor recovery after stroke: a critical review of its measurement properties. Neurorehabilit Neural Repair 16:232–240 Zhang M et al (2022) Gait pattern recognition based on plantar pressure signals and acceleration signals. IEEE Trans Instrum Meas 71:1–15 Jones TA (2017) Motor compensation and its effects on neural reorganization after stroke. Nat Rev Neurosci 18:267–280 Terven J et al (2023) A comprehensive review of yolo architectures in computer vision: From yolov1 to yolov8 and yolo-nas. Mach Learn Knowl Extr 5:1680–1716 Wei J et al (2022) Chain-of-thought prompting elicits reasoning in large language models. Adv Neural Inf Process Syst 35:24824–24837 Radford A et al (2023) Robust speech recognition via large-scale weak supervision. International conference on machine learning. PMLR Additional Declarations There is NO Competing Interest. Supplementary Files SIforsubmission.docx Supplementary Information SvideoRev480P.mp4 Supplementary Movie 1 Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5538299","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":385204607,"identity":"fe4a0e86-d3d5-45c6-a1cb-9285c7ec8c3a","order_by":0,"name":"Luigi Occhipinti","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAvUlEQVRIiWNgGAWjYBACPgYGxocfKuD8BMJa2BgYmI0lzpCohU2Ct40kLfyLD0hIzrOT121gfviBsS2NCC0SzxIMCrclG247wGYswdiWQ4yWMwYJktsOMG47wGDGwNhWQZyWA7xzDthvO8D+jUgt/D2GDbwNBxK3HeAB2UKUw9iSmSWOJSdvO8xTLJFwjgjv8/MfPv7zQ42d7bbj7Rs/fChLJqyFQSIBymBmICpWQNYcIErZKBgFo2AUjGQAAFgCNTFpaMgHAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-9067-2534","institution":"University of Cambridge","correspondingAuthor":true,"prefix":"","firstName":"Luigi","middleName":"","lastName":"Occhipinti","suffix":""},{"id":385204608,"identity":"fc430f3e-a1b4-4cd2-9f17-ab4408703ec6","order_by":1,"name":"Chenyu Tang","email":"","orcid":"https://orcid.org/0000-0002-6368-5639","institution":"University of Cambridge","correspondingAuthor":false,"prefix":"","firstName":"Chenyu","middleName":"","lastName":"Tang","suffix":""},{"id":385204609,"identity":"50801f01-1c5a-4c64-81da-179e03637962","order_by":2,"name":"Ruizhi Zhang","email":"","orcid":"","institution":"School of Instrumentation and Optoelectronic Engineering, Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Ruizhi","middleName":"","lastName":"Zhang","suffix":""},{"id":385204610,"identity":"59b42548-b811-4e01-a098-2076a4c5dba7","order_by":3,"name":"Shuo Gao","email":"","orcid":"https://orcid.org/0000-0003-3096-4700","institution":"Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Shuo","middleName":"","lastName":"Gao","suffix":""},{"id":385204611,"identity":"5d9a5325-f1fa-4ac2-8294-3de63f834dca","order_by":4,"name":"Zihe Zhao","email":"","orcid":"","institution":"School of Instrumentation and Optoelectronic Engineering, Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Zihe","middleName":"","lastName":"Zhao","suffix":""},{"id":385204612,"identity":"1c6770af-9078-4bb5-81ca-5ddc58c842c9","order_by":5,"name":"Zibo Zhang","email":"","orcid":"","institution":"University of Cambridge","correspondingAuthor":false,"prefix":"","firstName":"Zibo","middleName":"","lastName":"Zhang","suffix":""},{"id":385204613,"identity":"21ba1fef-ea64-485c-9be8-473d26c70335","order_by":6,"name":"Jiaqi Wang","email":"","orcid":"","institution":"School of Instrumentation and Optoelectronic Engineering, Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Jiaqi","middleName":"","lastName":"Wang","suffix":""},{"id":385204614,"identity":"8ac9a779-5824-4849-a312-e6c3148c5ec4","order_by":7,"name":"Cong Li","email":"","orcid":"","institution":"School of Instrumentation and Optoelectronic Engineering, Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Cong","middleName":"","lastName":"Li","suffix":""},{"id":385204615,"identity":"be6c66c1-c79c-4245-8aea-0478a6be51d5","order_by":8,"name":"Junliang Chen","email":"","orcid":"","institution":"School of Instrumentation and Optoelectronic Engineering, Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Junliang","middleName":"","lastName":"Chen","suffix":""},{"id":385204616,"identity":"91690151-8ac7-4058-a8a7-c887b4ca1684","order_by":9,"name":"Yanning Dai","email":"","orcid":"","institution":"King Abdullah University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Yanning","middleName":"","lastName":"Dai","suffix":""},{"id":385204617,"identity":"0c651445-d1c8-41d7-8d7e-f0972e8eb214","order_by":10,"name":"Shengbo Wang","email":"","orcid":"https://orcid.org/0000-0003-1212-138X","institution":"School of Instrumentation and Optoelectronic Engineering, Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Shengbo","middleName":"","lastName":"Wang","suffix":""},{"id":385204618,"identity":"11221799-6829-46e6-af3d-80f589dfab5f","order_by":11,"name":"Ruoyu Juan","email":"","orcid":"","institution":"Beijing New Guoxin Software Evaluation Technology Co ltd","correspondingAuthor":false,"prefix":"","firstName":"Ruoyu","middleName":"","lastName":"Juan","suffix":""},{"id":385204619,"identity":"d65494df-fc6d-4fc6-a212-1010d59516df","order_by":12,"name":"Qiaoying Li","email":"","orcid":"","institution":"Stomatology Department, Shijiazhuang People's Hospital","correspondingAuthor":false,"prefix":"","firstName":"Qiaoying","middleName":"","lastName":"Li","suffix":""},{"id":385204620,"identity":"fb004746-da10-4456-8d36-bc98254dc3a0","order_by":13,"name":"Ruimou Xie","email":"","orcid":"","institution":"Department of Rehabilitation Medicine, Beijing Tsinghua Changgung Hospital, Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Ruimou","middleName":"","lastName":"Xie","suffix":""},{"id":385204621,"identity":"d03e2ea9-8b8d-4b27-afe9-70b157f5ed3a","order_by":14,"name":"Xuhang Chen","email":"","orcid":"https://orcid.org/0009-0003-1757-9303","institution":"University of Cambridge","correspondingAuthor":false,"prefix":"","firstName":"Xuhang","middleName":"","lastName":"Chen","suffix":""},{"id":385204622,"identity":"04e473a7-7780-4e04-b8c4-1f278171c900","order_by":15,"name":"Xinkai Zhou","email":"","orcid":"","institution":"University College London","correspondingAuthor":false,"prefix":"","firstName":"Xinkai","middleName":"","lastName":"Zhou","suffix":""},{"id":385204623,"identity":"bf5618b4-31ed-4827-ba95-318b11dca361","order_by":16,"name":"Yunjia Xia","email":"","orcid":"","institution":"University College London","correspondingAuthor":false,"prefix":"","firstName":"Yunjia","middleName":"","lastName":"Xia","suffix":""},{"id":385204624,"identity":"7f5de098-d163-4208-9e55-c81eed12d500","order_by":17,"name":"Jianan Chen","email":"","orcid":"","institution":"University College London","correspondingAuthor":false,"prefix":"","firstName":"Jianan","middleName":"","lastName":"Chen","suffix":""},{"id":385204625,"identity":"b59c5978-af02-4995-a7d8-451371e961ea","order_by":18,"name":"Fanghao Lu","email":"","orcid":"","institution":"Hangzhou International Innovation Institute, Beihang University","correspondingAuthor":false,"prefix":"","firstName":"Fanghao","middleName":"","lastName":"Lu","suffix":""},{"id":385204626,"identity":"5785bef4-36cf-49e9-99b7-ddf815bcaabd","order_by":19,"name":"Xin Li","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Xin","middleName":"","lastName":"Li","suffix":""},{"id":385204627,"identity":"b41e5207-0349-46cd-8fdf-8325a1785c06","order_by":20,"name":"Ningli Wang","email":"","orcid":"https://orcid.org/0000-0002-8933-4482","institution":"Beijing Institute of Ophthalmology, Beijing Tongren Eye Center, Beijing Key Laboratory of Ophthalmology and Visual Sciences, Beijing Tongren Hospital, Capital Medical University","correspondingAuthor":false,"prefix":"","firstName":"Ningli","middleName":"","lastName":"Wang","suffix":""},{"id":385204628,"identity":"7957b013-8a16-4072-9f6a-67b0cb148f1a","order_by":21,"name":"Peter Smielewski","email":"","orcid":"https://orcid.org/0000-0001-5096-3938","institution":"University of Cambridge","correspondingAuthor":false,"prefix":"","firstName":"Peter","middleName":"","lastName":"Smielewski","suffix":""},{"id":385204629,"identity":"cbdd3fc2-7d01-4676-9e89-9d9d23e0fa65","order_by":22,"name":"Yu Pan","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Yu","middleName":"","lastName":"Pan","suffix":""},{"id":385204630,"identity":"423dfbf5-965d-452f-b8ff-13ac1fdde3e7","order_by":23,"name":"Hubin Zhao","email":"","orcid":"https://orcid.org/0000-0001-9408-4724","institution":"University College London, HUB of Intelligent Neuro-engineering (HUBIN), CREATe, Division of Surgery and Interventional Science","correspondingAuthor":false,"prefix":"","firstName":"Hubin","middleName":"","lastName":"Zhao","suffix":""}],"badges":[],"createdAt":"2024-11-28 00:25:16","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5538299/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5538299/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":78883537,"identity":"9d586be3-e3f8-4a5e-a30f-4834f7f1745e","added_by":"auto","created_at":"2025-03-20 09:09:17","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":794833,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eOverview of the platform developed for at-home rehabilitation of post-stroke patients. a, \u003c/strong\u003eSmart home setup. The system integrates wearable and ambient sensors, enabling real-time health monitoring and seamless interaction with smart appliances to support post-stroke recovery at home. \u003cstrong\u003eb, \u003c/strong\u003eSystem architecture and module design for multi-modal sensing. The platform comprises wearable modules for eye tracking, plantar pressure sensing, and physiological monitoring, along with a host server and ambient sensors to collect, process, and analyze multi-modal data for personalized rehabilitation and assistance.\u003c/p\u003e","description":"","filename":"image1.png","url":"https://assets-eu.researchsquare.com/files/rs-5538299/v1/6eea5aa214e1ce7b53099255.png"},{"id":78883539,"identity":"caf1622e-a194-4724-8dc2-74725e592b53","added_by":"auto","created_at":"2025-03-20 09:09:18","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":1488129,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eVisualization of signals for motor and cognitive impairments. a, \u003c/strong\u003eTime-frequency spectrums of plantar pressure signals for left (L) and right (R) feet across mild, moderate, and severe rehabilitation levels. \u003cstrong\u003eb,\u003c/strong\u003e Comparison of gaze trajectories and heatmaps during interaction with a target smart light of the user with and without cognitive impairment.\u003c/p\u003e","description":"","filename":"image2.png","url":"https://assets-eu.researchsquare.com/files/rs-5538299/v1/64609962b1de6343f8ccd626.png"},{"id":78883540,"identity":"ba72148e-ed2f-43c4-93f2-3360046eee36","added_by":"auto","created_at":"2025-03-20 09:09:18","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":709558,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eMotor rehabilitation monitoring framework and performance evaluation. a\u003c/strong\u003e, Coefficient of variation across rehabilitation stages. The variability in plantar pressure measurements increases with rehabilitation severity, highlighting distinct patterns among mild, moderate, and severe patients. \u003cstrong\u003eb\u003c/strong\u003e, Plantar pressure and stance phase ratio asymmetry. Left and right foot comparisons of pressure and stance phase ratio across rehabilitation stages, showing asymmetrical trends associated with severity. \u003cstrong\u003ec\u003c/strong\u003e, Deep learning framework for motor rehabilitation status classification. \u003cstrong\u003ed\u003c/strong\u003e, Classification accuracy of various deep learning models. \u003cstrong\u003ee\u003c/strong\u003e, Confusion matrix of classification results. \u003cstrong\u003ef\u003c/strong\u003e, UMAP visualization of latent features.\u003c/p\u003e","description":"","filename":"image3.png","url":"https://assets-eu.researchsquare.com/files/rs-5538299/v1/2424935524b8f5ebda27cd4e.png"},{"id":78883550,"identity":"5e137008-a664-4e2c-a26c-51209b46feee","added_by":"auto","created_at":"2025-03-20 09:09:18","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":1450646,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eReal-time scene detection, action recognition, and multimodal interaction system for stroke patient monitoring and smart home control. a, \u003c/strong\u003eScene images are analyzed using a fine-tuned YOLOv8n model for object detection and scene recognition, while human actions (e.g., sitting, walking, falling) are identified using a MediaPipe-based MLP classifier with 99.3% accuracy and \u0026lt;50 ms latency. Integrated results provide real-time feedback. \u003cstrong\u003eb,\u003c/strong\u003e Training curves and inference time confirm model accuracy and efficiency.\u003cstrong\u003e c,\u003c/strong\u003e Multimodal interaction combines gaze detection, blink recognition, and speech inputs to generate control commands for smart home devices, supporting diverse patient needs and enabling adaptive rehabilitation.\u003c/p\u003e","description":"","filename":"image4.png","url":"https://assets-eu.researchsquare.com/files/rs-5538299/v1/a0bd98d9064adec42112cd31.png"},{"id":78883553,"identity":"b50f28ee-b514-4b7c-a53a-d8b4621ac2b1","added_by":"auto","created_at":"2025-03-20 09:09:18","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":464489,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eLLM agent (Auto-Care) for autonomous rehabilitation management. a\u003c/strong\u003e, Daily monitoring and intervention by Auto-Care. Physiological and environmental signals, including heart rate (HR), heart rate variability (HRV), temperature, light intensity, and user state are continuously monitored. Auto-Care provides context-aware interventions such as safety checks, health advice, and environmental adjustments based on comprehensive analysis. \u003cstrong\u003eb\u003c/strong\u003e, Prompt design (basic, chain-of-thought (CoT), and CoT with demo-based prompts) and its impact on satisfaction. \u003cstrong\u003ec\u003c/strong\u003e, Effect of context length on agent performance. \u003cstrong\u003ed\u003c/strong\u003e, Radar plot comparing user satisfaction across various configurations (With Auto-Care, Without Auto-Care, and Without the platform).\u003c/p\u003e","description":"","filename":"image5.png","url":"https://assets-eu.researchsquare.com/files/rs-5538299/v1/28733b53c54c504993496d8e.png"},{"id":78886366,"identity":"50c976a5-b0a6-4718-a4c0-0ca5e135adfc","added_by":"auto","created_at":"2025-03-20 09:41:20","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":7005025,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5538299/v1/ebf634a4-4747-4ca2-96ca-7e5a935e36ab.pdf"},{"id":78883554,"identity":"1e420acf-6c3f-4008-b1e6-3974ed4e6b33","added_by":"auto","created_at":"2025-03-20 09:09:18","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":6474608,"visible":true,"origin":"","legend":"Supplementary Information","description":"","filename":"SIforsubmission.docx","url":"https://assets-eu.researchsquare.com/files/rs-5538299/v1/c7a529a1c96eec4ce12f6510.docx"},{"id":78884030,"identity":"ee0cd63e-4182-4608-8606-ffc6eb0f21ab","added_by":"auto","created_at":"2025-03-20 09:17:18","extension":"mp4","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":27455218,"visible":true,"origin":"","legend":"Supplementary Movie 1","description":"","filename":"SvideoRev480P.mp4","url":"https://assets-eu.researchsquare.com/files/rs-5538299/v1/de8959d77b76597742abe5b9.mp4"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"A Unified Platform for At-Home Post-Stroke Rehabilitation Enabled by Wearable Technologies and Artificial Intelligence","fulltext":[{"header":"I. Main","content":"\u003cp\u003eStroke is the third leading cause of disability worldwide, affecting more than 101 million people [1, 2]. Survivors often experience motor impairments (60–80%), cognitive deficits (20–30%), and speech difficulties (30–50%), which significantly compromise their independence and quality of life [3, 4]. Post-stroke recovery is not only a prolonged process but also a resource-intensive one, imposing significant economic and caregiving burdens on families and healthcare systems—a challenge exacerbated by global aging [5]. For many patients, the home becomes a critical environment for rehabilitation, as opportunities for continuous and personalized care are limited outside of clinical settings [6]. An ideal solution would resemble the intelligent, adaptive environments often envisioned in science fiction: a system that uses multi-modal, multi-dimensional sensing to continuously monitor patients’ physical and physiological states, while leveraging artificial intelligence to provide real-time health assessments and assistance [7, 8].\u003c/p\u003e\n\u003cp\u003eRecent advancements in wearable sensors and artificial intelligence (AI) technologies have brought this vision within reach [9, 10, 11, 12]. Combined with AI methods, wearable devices such as force sensors, accelerometers, EMG sensors, and eye trackers have enabled tracking of motor recovery and provided valuable insights into cognitive function [13-23]. Assistive tools, including robotic aids and smart home devices, have been developed to address specific daily needs [24, 25, 26]. However, the comprehensive solution envisioned is still absent. To unlock the full potential of at-home rehabilitation, we step further to a unified, patient-centered system capable of navigating the complexities of real-world scenarios.\u003c/p\u003e\n\u003cp\u003eHere, we report a smart home system specifically designed for long-term, at-home rehabilitation of post-stroke patients, integrating health monitoring and assistive functionalities into a single platform (Fig. 1). By leveraging multi-sensor fusion, the system comprehensively addresses the needs of patients with post-stroke impairments. For rehabilitation monitoring, our plantar pressure array, coupled with a machine learning model, evaluates motor recovery, achieving a classification accuracy of 94.1% across three rehabilitation states. A wearable eye-tracking module extracts key indicators of cognitive function, while ambient sensors such as cameras and microphones, in collaboration with the eye-tracking module, enable seamless and precise smart home control (100% operational success rate with a latency of \u0026lt;1 s). This multi-sensor collaborative design ensures accessibility for a diverse range of users, allowing them to choose the most suitable interaction modality based on their specific needs. Additionally, we introduce an autonomous assistive agent, Auto-Care, powered by a large language model (LLM), which analyzes multi-modal data to provide timely interventions such as health reminders, environmental adjustments, or caregiver notifications, increasing user satisfaction by 29% compared to scenarios without the agent. This system represents the first fully integrated solution for simultaneous health monitoring and intelligent assistance in post-stroke home rehabilitation, offering a pathway toward comprehensive, patient-centered management. Furthermore, it holds potential for broader applications in other chronic conditions, such as amyotrophic lateral sclerosis (ALS) and Parkinson’s disease, and aging populations.\u003c/p\u003e"},{"header":"II. Results","content":"\u003cp\u003e\u003cstrong\u003eThe multimodal rehabilitation system\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAs shown in Fig. 1a, the platform integrates wearable devices, including plantar pressure insoles (SFig. 1-4), wristband module (SFig. 5, 6), and eye-tracking module (SFig. 7), alongside ambient sensors such as cameras and microphones, to enable comprehensive, round-the-clock monitoring of patients [27, 28, 29]. This multimodal sensing system collects a full spectrum of patient information, providing intelligent assessments of rehabilitation progress and offering daily assistance to support independent living. The core concept of the platform is realized through the IoT architecture depicted in Fig. 1b (SFig. 11), which consolidates data from all sensing modalities into a local host server at home. Wearable devices and ambient sensors transmit data locally via Bluetooth Low Energy (BLE) and WiFi protocols, ensuring seamless communication, minimal latency, protection of users\u0026rsquo; data. The host server processes the aggregated data in real time, transforming it into actionable outputs for health monitoring and assistive decision-making.\u003c/p\u003e\n\u003cp\u003eTo evaluate the platform\u0026rsquo;s capability in tracking motor recovery, 20 post-stroke patients with varying degrees of motor impairments and diverse post-stroke complications including hemiplegia, knee valgus, and foot inversion were recruited. Patients were stratified based on their motor function scores obtained by using the Fugl-Meyer Assessment (FMA) scale [30] into three rehabilitation levels: mild, moderate, and severe. Plantar pressure data from a 48-channel sensor matrix were recorded during routine walking tasks, capturing detailed gait dynamics across different recovery stages. As shown in Fig. 2a, patients in the mild rehabilitation level exhibited gait signals resembling those of healthy individuals, characterized by consistent amplitude and symmetry, indicating effective weight distribution and propulsion [29, 31]. In contrast, patients in the moderate rehabilitation level showed irregular oscillations and reduced symmetry, reflecting instability during gait phases. Those in the severe rehabilitation level, such as individuals with left-sided hemiplegia, exhibited diminished signals on the affected side and exaggerated signals on the unaffected side, indicative of compensatory mechanisms [32]. These distinctive signal patterns across recovery states underpin the machine learning model\u0026rsquo;s ability to objectively decode and monitor rehabilitation progress.\u003c/p\u003e\n\u003cp\u003eFig. 2b shows the comparison of eye-tracking patterns between subjects with and without cognitive impairments during interactions with a smart light. Among the 20 participants, 4 were identified by clinicians as exhibiting cognitive impairments based on professional evaluations of their behavior and neurological assessments. While the limited sample size of cognitive impairment cases does not allow for the development of a statistically robust tracking model, the collected data still provide valuable insights for clinicians via the IoT system. For subjects without cognitive impairment, gaze trajectories during interaction were precise and efficient, with fixations rapidly converging on the target smart light, as reflected by the compact heatmap. In contrast, the subject with cognitive impairment demonstrated dispersed and irregular gaze patterns, with frequent distractions and prolonged fixations on irrelevant objects before locating the target. The corresponding heatmap exhibits a broad, scattered distribution, consistent with delayed visual attention and impaired decision-making, hallmark traits often associated with cognitive deficits. Additionally, the platform recorded other statistical metrics, such as blink frequency and duration patterns, providing further indicators of cognitive function (SFig. 8, 9). These data, aggregated through the IoT system, can be securely shared with clinicians to analyze, offering valuable supplementary information for understanding the patient\u0026rsquo;s cognitive condition and guiding personalized interventions.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMotor rehabilitation states monitoring via plantar pressure insoles and deep learning\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo accurately track motor impairment recovery stages, we collected walking data from the subjects and segmented it into 5-second samples to construct the dataset. Among the 20 participants, 6 were annotated as being in the severe, 7 in the moderate, and 7 in the mild rehabilitation level. Fig. 3a and Fig. 3b visualize key statistical characteristics of the gait data. Specifically, Fig. 3a displays the coefficient of variation (CV) of plantar pressure, which quantifies the variability in foot pressure relative to the mean pressure, with higher CV values reflecting greater instability in walking. Patients in the severe stage exhibit significantly higher CV compared to those in the moderate and mild stages, indicating less consistent gait dynamics. Fig. 3b highlights the asymmetry in pressure distribution and stance phase ratio between the left and right feet, with more severe cases showing pronounced imbalances. This reflects the underlying biomechanical challenges, where motor impairments disrupt load distribution and timing between the two legs.\u003c/p\u003e\n\u003cp\u003eFig. 3c outlines our deep learning pipeline for decoding rehabilitation states. The 48-channel plantar pressure signals from each foot are converted into 224 \u0026times; 224 two-dimensional heatmaps, which are then fed into a convolutional neural network (CNN) to encode spatiotemporal gait features. This 2D transformation allows the model to capture the spatial relationships between channels while preserving temporal dynamics, a critical aspect for distinguishing gait patterns across recovery stages. The encoded features from both feet are subsequently processed by a multi-layer perceptron (MLP) classifier to decode the patient\u0026apos;s rehabilitation status. Performance comparisons of various baseline models as gait feature encoders are shown in Fig. 3d, where ResNet-101 outperformed alternatives with the highest accuracy of 94.1%, supporting its selection as the optimal encoder (optimal hyperparameters detailed in STable 1). The confusion matrix in Fig. 3e demonstrates robust classification across all rehabilitation states, with minimal misclassification errors. Furthermore, the encoder\u0026rsquo;s output feature representations, visualized using UMAP in Fig. 3f, reveal clear clustering of the three rehabilitation states, underscoring the model\u0026apos;s ability to differentiate between mild, moderate, and severe motor impairment stages effectively. This pipeline provides a robust, end-to-end, and data-driven approach to monitor recovery progress in post-stroke patients.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSmart home control based on multi-sensor fusion\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo address the challenges of monitoring stroke patients in home settings, we developed a real-time scene detection and human action recognition system based on video streams (Fig. 4a). This system employs a lightweight neural network architecture to achieve efficient local computation while maintaining patient privacy, balancing accuracy and computational efficiency. Scene images are extracted from video streams captured by home cameras and processed using a fine-tuned YOLOv8n model [33]. The model identifies the patient and household objects with corresponding confidence scores. Objects with scores exceeding a predefined threshold are retained, and their spatial relationships are analyzed to infer the current home environment. For human action recognition, patient images are processed using the open-source MediaPipe model to extract pose landmarks, which are then converted into normalized 3D coordinates. These coordinates serve as input for an MLP action classification model, enabling the detection of typical human actions with an inference latency of less than 50 ms. On a self-collected test dataset, the model achieved an accuracy of 99.3% (Fig. 4b, SFig. 10). By combining scene and action data, the system provides real-time video feedback to a host device while storing data for retrospective analysis, facilitating long-term rehabilitation monitoring.\u003c/p\u003e\n\u003cp\u003eFor patients with speech impairments, the system extends its functionality to support multimodal interaction by integrating audio and eye-tracking data. Patients with partial speech capability can issue voice commands through a microphone, while those with severe speech impairments can rely on gaze direction and blinking patterns captured by the eye tracker (Fig. 4c). These signals are processed to generate control commands for smart home devices. This approach provides a seamless and intuitive interface for patients with diverse disabilities, fostering independence and improving their quality of life during at-home rehabilitation. The IoT-based architecture ensures reliable communication with smart devices, enabling adaptive control of the home environment to meet individual needs (SVideo).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLLM agent for autonomous rehabilitation management\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo overcome the limitations of patients interacting with the platform solely based on subjective needs, we embedded an autonomous health management agent, Auto-Care, powered by GPT-4o Mini API. This agent operates continuously, analyzing multimodal data streams 24/7 to intelligently detect and address various patient needs (SVideo). For example, as shown in Fig. 5a, during gait training (point 1), the agent detected rising heart rate and temperature coupled with decreasing heart rate variability (HRV), prompting it to recommend hydration, pause training, and activate air conditioning to maintain comfort. At point 2, when a fall was detected, the agent used a microphone to check the patient\u0026rsquo;s status and, upon confirming the need for assistance or receiving no response, alerted a caregiver. At point 3, as ambient light levels decreased, the agent adjusted the smart lighting in the dining room based on the patient\u0026rsquo;s location to ensure adequate illumination.\u003c/p\u003e\n\u003cp\u003eThe effectiveness of Auto-Care was further enhanced through prompt optimization, as shown in Fig. 5b. Adding Chain-of-Thought reasoning and pre-defined intervention demos to the prompts significantly improved user satisfaction [34]. Multimodal data were downsampled to 1-minute intervals before being input into the agent, ensuring computational efficiency without compromising real-time responsiveness. Fig. 5c shows that a six-minute data context provided the optimal balance for decision-making accuracy and computational efficiency. Overall, as demonstrated in Fig. 5d, the integration of the Auto-Care Agent into the platform significantly enhanced patient outcomes across multiple dimensions, including reduced psychological burden, improved operational efficiency, and increased overall satisfaction (evaluation criteria detailed in STable 2). Compared to scenarios where the platform was not used, users reported a 67% improvement in overall satisfaction, with an additional 29% increase upon the integration of the Auto-Care Agent. These results highlight the potential of the platform and agent to transform at-home rehabilitation, providing continuous, intelligent support tailored to individual patient needs.\u003c/p\u003e"},{"header":"III. Discussion","content":"\u003cp\u003eIn this work, we present a smart home system specifically designed to address the multifaceted rehabilitation needs of post-stroke patients in a home environment. The system\u0026rsquo;s integrated platform combines advanced health monitoring and assistive functionalities, demonstrating exceptional performance in real-world scenarios. Comprehensive evaluations confirm its efficacy: the plantar pressure array and machine learning model achieved 94.1% accuracy in classifying three rehabilitation states, while the wearable eye-tracking module and ambient sensors enabled seamless smart home control with a 100% success rate and latency of \u0026lt;\u0026thinsp;1 s. The introduction of Auto-Care, an LLM-powered autonomous agent, further enhances the system\u0026rsquo;s functionality, improving user satisfaction by 29% through timely and intelligent interventions. These features collectively position the system as a groundbreaking solution for post-stroke rehabilitation, offering continuous, personalized care and assistance.\u003c/p\u003e \u003cp\u003eFuture directions will focus on advancing the system\u0026rsquo;s adaptability, scalability, and inclusivity. First, expanding its application to other chronic conditions and diverse user populations, including aging individuals and those with neurodegenerative diseases, will enhance its broader relevance. Second, improving the system\u0026rsquo;s robustness and interoperability with external assistive devices, such as robotic aids and exoskeletons, will further enrich its capabilities. Lastly, optimizing its computational efficiency through edge computing will reduce power consumption and latency, enabling seamless operation and ensuring privacy in at-home environments.\u003c/p\u003e \u003cp\u003eLooking ahead, the system has the potential to transform long-term post-stroke care by improving both physical and psychological well-being. Continuous, personalized monitoring and intelligent assistance empower patients to regain independence, enhance social interaction, and improve quality of life. Additionally, the system\u0026rsquo;s ability to collect and analyze long-term, multi-modal data opens new possibilities for predicting stroke progression, recovery trajectories, and even the risk of secondary strokes. By identifying subtle patterns in motor performance, cognitive behaviors, physiological signals, and environmental factors, the system could enable proactive, personalized interventions to mitigate future risks and optimize recovery strategies. These capabilities position the system not only as a rehabilitation tool but as a predictive and comprehensive health management platform for post-stroke care.\u003c/p\u003e"},{"header":"IV. Methods","content":"\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003eFabrication of the plantar pressure insole\u003c/h2\u003e \u003cp\u003eTo detect plantar pressure, we developed a custom insole equipped with a 4 \u0026times; 12 resistive pressure sensor array, comprising 48 sensing points with an average sensor density of 0.23 sensors/cm\u0026sup2;. The structural details and sensor dimensions are reported in the figure. The topmost layer of the insole consists of a polyethylene terephthalate (PET) protective film, beneath which lies a layer of copper row and column electrodes etched onto a polyimide (PI) substrate. An FSR (force-sensitive resistor) graphite layer is placed between the electrode layers to form the resistive sensing elements. This flexible design ensures that the insole can withstand frequent bending during walking without losing functionality. At just 100 \u0026micro;m thick, the insole provides a comfortable wearing experience without causing discomfort.\u003c/p\u003e \u003cp\u003eTo process the pressure data, we designed a custom resistive array detection circuit with the HC32F460 microcontroller, based on the ARM Cortex-M4 architecture operating at 200 MHz. This MCU offers robust computational capability for processing large volumes of pressure data. A low-dropout regulator (LDO) reduces the input voltage from 5.6 V to 5 V, ensuring a stable power supply to the ADC for precise signal conversion. The circuit also includes an integrated TP4054 lithium battery charging chip, enabling recharging via a Micro-USB interface.\u003c/p\u003e \u003cp\u003eFor high-speed data communication, all modules utilize fast GPIO and protocols such as SPI and I2C. To enable wireless data transmission, the circuit integrates a CH9141 Bluetooth module that communicates with the MCU via the USART protocol. This efficient and flexible design supports real-time plantar pressure monitoring, with data transmission capabilities that are robust and optimized for rehabilitation scenarios.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eFabrication of the wristband\u003c/h2\u003e \u003cp\u003eA custom-designed wireless wristband was developed to enable efficient, continuous acquisition and transmission of physiological and environmental data within the IoT system, ensuring seamless integration and modularity for the specific multimodal sensing needs of the platform. Unlike commercial solutions, the custom design ensures full compatibility with the IoT architecture and allows precise customization of sensing modalities to guarantee integrated functionality. The wristband integrates six functional modules: an STM32L412 microcontroller for system control and coordination of the overall sensing and data management functions, a CH9141 BLE module for bidirectional data and instruction transmission, a power management module, and three sensing modules\u0026mdash;MAX30101 for PPG signal acquisition, AS7341 for environmental light detection, and MTS4B for temperature measurement.\u003c/p\u003e \u003cp\u003eEach sensing module operates independently, collecting data at predefined sampling rates and temporarily storing raw signals in internal registers. The STM32L412 microcontroller controls system functionality, polling each module at regular intervals via the I2C protocol and transmitting aggregated data to the host computer via the CH9141 Bluetooth module. The wristband is powered by a 4.2V rechargeable lithium battery, charged through a TP4054 linear charger and protected against overcharging, over-discharging, and overcurrent using a DW01 chip. Real-time battery status is monitored by a BQ27220 coulometer, while voltage regulation is managed by XC6206P332MR and XC6206P182MR LDOs (3.3V and 1.8V, respectively) and a ME2188A50XG linear regulator delivering 5V for the MAX30101\u0026rsquo;s integrated LED.\u003c/p\u003e \u003cp\u003eTo ensure reliability and user comfort during daily wear, the wristband features a compact six-layer PCB with double-sided component mounting, measuring 40 \u0026times; 30 \u0026times; 1.6 mm. This design achieves a balance between unobtrusiveness and robust functionality, facilitating comfortable long-term use while maintaining consistent performance.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eFabrication of the wearable eye tracker\u003c/h3\u003e\n\u003cp\u003eA custom head-mounted wireless eye tracker was developed for real-time gaze tracking and environmental scene analysis, tailored for applications in rehabilitation and smart home interaction. The system integrates two near-infrared (IR) cameras, each equipped with four edge-mounted IR LEDs, for precise pupil and corner-of-eye detection, and a forward-facing visible-light camera for capturing the wearer's environmental context. All cameras utilize the IMX258 image sensor with an 80-degree fixed-focus lens, providing high-resolution (12 MP) imagery at 30 FPS.\u003c/p\u003e \u003cp\u003eCentralized processing is performed by an OrangePi CM5 module, which incorporates the RK3588S SoC (quad-core Cortex-A76 at 2.4 GHz and quad-core Cortex-A55), ensuring efficient real-time data handling. The module is powered by an RK806-1 power management IC, supporting stable operations for computationally intensive tasks. Wireless data transmission is facilitated by a CDW-20U5622 WiFi module, enabling seamless integration with the IoT system. The device is powered by a compact 5V 4A lithium battery, ensuring portability and extended use.\u003c/p\u003e \u003cp\u003eKey eye features, such as pupil center and eye corner coordinates, are extracted in real time by the processing unit, which computes gaze coordinates using calibration data. The visible-light camera data is synchronized with gaze coordinates, providing a robust mechanism for environmental interaction. The system achieves high accuracy and reliability in dynamic scenarios, supporting diverse applications such as rehabilitation monitoring, assistive device control, and cognitive assessments.\u003c/p\u003e\n\u003ch3\u003eIoT Framework for Multimodal Data Integration\u003c/h3\u003e\n\u003cp\u003eTo enable seamless integration and processing of multimodal data in the home environment, we designed a hierarchical IoT architecture comprising the following layers (SFig. 11):\u003c/p\u003e \u003cp\u003e \u003cstrong\u003eSensor Layer\u003c/strong\u003e \u003cp\u003eThis layer includes all data collection devices responsible for sensing user states and environmental conditions. Key components are wearable eye-tracking devices, wristbands, plantar pressure insoles, and Hikvision DS-2SC2Q133MW cameras (integrated with microphones and speakers).\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eData Transmission Layer\u003c/b\u003e: Communication across devices utilizes three main protocols: BLE, HTTP, and MiIO. The wristbands and insoles communicate with the gateway using BLE 4.2 via Bluetooth modules. The eye-tracking devices and cameras connect to the gateway over a WiFi network using the HTTP protocol. Smart home devices, such as lights and air conditioners, use the MiIO protocol over WiFi for communication.\u003c/p\u003e \u003cp\u003e \u003cstrong\u003eData Processing Layer\u003c/strong\u003e \u003cp\u003eA custom gateway software is deployed on a host device equipped with both WiFi and BLE modules. This software aggregates data streams from all sensors, processes multimodal data fusion, and distributes control commands to connected devices. The host device handles real-time data synchronization and processing to ensure consistent and actionable outputs across modalities.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eEndpoint Layer\u003c/strong\u003e \u003cp\u003eThis layer consists of smart home devices such as smart TVs, air conditioners, and table lamps, which are controlled via the MiIO protocol.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eTime Synchronization Server\u003c/strong\u003e \u003cp\u003eA local network time protocol (NTP) server is set up on the host device to ensure precise time synchronization for all collected data. During data processing at the gateway, timestamps generated from the NTP-synchronized server are embedded in the frame headers to maintain temporal alignment across all modalities.\u003c/p\u003e \u003c/p\u003e \u003cp\u003eThis framework ensures efficient, synchronized communication and integration of data from diverse sensing devices, enabling robust multimodal monitoring and interaction within the smart home rehabilitation system.\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003ePlantar pressure data acquisition\u003c/h2\u003e \u003cp\u003eWe conducted a motor impairment study involving 20 stroke patients (mean age: 51.4\u0026thinsp;\u0026plusmn;\u0026thinsp;9.8 years; 14 males, 6 females) who were recruited in compliance with the Ethics Committee approval by the Committee for Medical Research Ethics at the First Hospital of Shijiazhuang City, China (assigned project number of 2020036). All participants provided written informed consent prior to enrollment. Patients were instructed to walk naturally on a flat surface while wearing plantar pressure insoles under the supervision of medical professionals. Continuous plantar pressure signals were recorded during the walking sessions. Following the data collection, the patients\u0026rsquo; motor recovery status was assessed using the Fugl-Meyer Assessment (FMA) scale, a clinically validated tool for evaluating motor impairment recovery. Based on their FMA scores, patients were categorized into three levels of motor impairment recovery: mild (FMA score\u0026thinsp;\u0026ge;\u0026thinsp;85), moderate (FMA score 50\u0026ndash;84), and severe (FMA score\u0026thinsp;\u0026lt;\u0026thinsp;50).\u003c/p\u003e \u003cp\u003eThe plantar pressure insoles captured data at a sampling frequency of 200 Hz. For analysis, continuous signals from both feet were segmented into five-second intervals, with each interval constituting a single sample. A total of 1,543 gait samples were collected across the 20 participants (80% were selected as the training set, while 20% formed the test set), providing a robust dataset for subsequent analysis of motor recovery patterns.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eSoftware environment for motor impairment monitoring model training\u003c/h2\u003e \u003cp\u003eSignal preprocessing was performed on a MacBook Pro equipped with an M1 Max CPU. Network training was conducted using Python 3.8.13, Miniconda 3, and PyTorch 2.0.1 in a performance-optimized environment. Training acceleration was enabled by CUDA on NVIDIA 4090 GPU.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eThe smart home control system\u003c/h2\u003e \u003cp\u003eThe smart home control system combines lightweight neural networks and wearable eye-tracking technology to achieve efficient, privacy-preserving, real-time interaction in home environments. For scene detection and action recognition, a fine-tuned YOLOv8n model was employed to analyze video streams from a Hikvision DS-2SC2Q133MW camera. This lightweight model, with 3.2M parameters, achieves near 100% accuracy for detecting human actions (e.g., walking, sitting, standing, and falling) and household objects (e.g., sofas, lights, and TVs) while maintaining an inference time of less than 100 ms on a CPU. Action classification leverages MediaPipe for extracting 3D normalized pose coordinates from images and a compact 3-layer MLP with 0.7M parameters for action recognition, achieving 99.3% accuracy with an inference time below 50 ms.\u003c/p\u003e \u003cp\u003eWearable eye trackers further enable intuitive control of smart home devices. These devices capture infrared pupil images and field-of-view (FoV) images to compute real-time gaze coordinates. After a nine-point calibration process using least-squares fitting, gaze points are mapped onto detected household objects. A decision is made if the gaze is continuously fixed on an object across five consecutive frames, prompting audio feedback like \u0026ldquo;TV detected, please issue a command.\u0026rdquo; For patients with speech capabilities, commands are captured via a microphone, processed through Whisper-tiny (39M parameters) [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e], and translated into device actions via MiIO protocol.\u003c/p\u003e \u003cp\u003eFor patients with severe speech impairments, eye gestures and blinks are detected to enable interaction. Gaze direction is classified by analyzing real-time pupil positions, while blinks are identified by tracking closed-eye intervals using statistical thresholds derived from calibration data. These signals are translated into control commands, allowing comprehensive interaction with smart home devices through a combination of gaze, speech, and physical gestures, optimized for accessibility and efficiency.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eDesign of the Auto-Care agent\u003c/h2\u003e \u003cp\u003eAuto-Care utilizes the GPT-4o Mini API to analyze multimodal patient data streams and deliver real-time, context-aware interventions. Prompts are dynamically generated to include a six-minute context window, summarizing recent trends and events. CoT reasoning is embedded to enable stepwise analysis of patient status and recommended actions, such as pausing rehabilitation, suggesting hydration, or adjusting environmental conditions. The API is configured with a response token limit of 200 to ensure concise outputs, and a temperature setting of 0.7 balances variability and reliability. Predefined templates and multi-step reasoning ensure robust and contextually relevant decisions across diverse rehabilitation scenarios. Feedback from real-world applications continuously refines prompt structures, enhancing precision and system performance.\u003c/p\u003e \u003c/div\u003e "},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eData availability \u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe datasets supporting this study will be available from the GitHub repository before publication.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCode availability \u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe code supporting this study will be available from the GitHub repository before publication.\u003c/p\u003e\n\u003cp\u003eIn relation to the data and code supporting the study, these are available and accessible from the following repository: https://github.com/tcy21414/Smart-Home-Platform-for-Stroke-Rehabilitation\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgments\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eS.G. acknowledges funding from the National Natural Science Foundation of China (62171014), and Beihang Ganwei Project (JKF-20240590). H.Z. acknowledges funding from The Royal Society Research Grant (RGS\\R2\\222333), the Engineering and Physical Sciences Research Council (EPSRC) Grant (13171178 R00287), and the European Innovation Council (EIC) under the European Union\u0026rsquo;s Horizon Europe research and innovation program (grant agreement No. 101099093). L.G.O. acknowledges funding from The British Council (contract No. 45371261) and the EPSRC (grants No.\u0026nbsp;EP/K03099X/1, EP/L016087/1, EP/W024284/1, EP/P027628/1). C.T. was supported by Endoenergy Systems (grant No. G119004) and Haleon (CAPE partnership grant No. G110480), W.Y. was supported by Pragmatic Semiconductor (grant No. G117793) and Haleon (CAPE partnership grant No. G110480).\u0026nbsp;\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003ePrust ML et al (2024) Addressing disparities in the global epidemiology of stroke. Nat Reviews Neurol 20:207\u0026ndash;221\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFeigin VL et al (2023) Pragmatic solutions to reduce the global burden of stroke: a World Stroke Organization\u0026ndash;Lancet Neurology Commission. Lancet Neurol 22:1160\u0026ndash;1206\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTsao CW et al (2022) Heart disease and stroke statistics\u0026mdash;2022 update: a report from the American Heart Association. Circulation 145:e153\u0026ndash;e639\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLo JW et al (2022) Long-term cognitive decline after stroke: an individual participant data meta-analysis. Stroke 53:1318\u0026ndash;1327\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eScott CA et al (2022) Diverging temporal trends in stroke incidence in younger vs older people: a systematic review and meta-analysis. JAMA Neurol 79:1036\u0026ndash;1048\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCramer SC et al (2019) Efficacy of home-based telerehabilitation vs in-clinic therapy for adults after stroke: a randomized clinical trial. JAMA Neurol 76:1079\u0026ndash;1087\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBivard A et al (2020) Artificial intelligence for decision support in acute stroke\u0026mdash;current roles and potential. Nat Reviews Neurol 16:575\u0026ndash;585\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMouridsen K et al (2020) Artificial intelligence applications in stroke. Stroke 51:2573\u0026ndash;2579\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTang C et al (2024) A roadmap for the development of human body digital twins. Nat Reviews Electr Eng 1:199\u0026ndash;207\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVaghasiya JV et al (2023) Wearable sensors for telehealth based on emerging materials and nanoarchitectonics. npj Flex Electron 7:26\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen C et al (2023) Digital health for aging populations. Nat Med 29:1623\u0026ndash;1630\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTang C et al (2022) Wearables-based multi-column neural network for human activity recognition. IEEE J Biomedical Health Inf 27:339\u0026ndash;350\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLin F et al (2016) Smart insole: A wearable sensor device for unobtrusive gait monitoring in daily life. IEEE Trans Industr Inf 12:2281\u0026ndash;2291\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen J et al (2022) Plantar pressure-based insole gait monitoring techniques for diseases monitoring and analysis: a review. Adv Mater Technol 7:2100566\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRusu L et al (2021) Plantar pressure and contact area measurement of foot abnormalities in stroke rehabilitation. Brain Sci 11:1213\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePradhan GB et al (2024) A wearable, self-sustainable, and wireless plantar pressure and temperature monitoring system for foot ulceration prognosis and rehabilitation. Sens Actuators A: Phys 379:115985\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAdans-Dester C et al (2020) Enabling precision rehabilitation interventions using wearable sensors and machine learning to track motor recovery. npj Digit Med 3:121\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePatel S et al (2010) A novel approach to monitor rehabilitation outcomes in stroke survivors using wearable technology. \u003cem\u003eProceedings of the IEEE\u003c/em\u003e 98, 450\u0026ndash;461\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBisio I et al (2019) When eHealth meets IoT: A smart wireless system for post-stroke home rehabilitation. IEEE Wirel Commun 26:24\u0026ndash;29\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZadka A et al (2024) A wearable sensor and machine learning estimate step length in older adults and patients with neurological disorders. npj Digit Med 7:142\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eForssman L et al (2017) Eye-tracking-based assessment of cognitive function in low-resource settings. Arch Dis Child 102:301\u0026ndash;302\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTao L et al (2020) Eye tracking metrics to screen and assess cognitive impairment in patients with neurological disorders. Neurol Sci 41:1697\u0026ndash;1704\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSkaramagkas V et al (2021) Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Rev Biomed Eng 16:260\u0026ndash;277\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbedi A et al (2024) Artificial intelligence-driven virtual rehabilitation for people living in the community: A scoping review. npj Digit Med 7:25\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen X et al (2022) An IoT and wearables-based smart home for ALS patients. IEEE Internet Things J 9:20945\u0026ndash;20956\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eArntz A et al (2023) Technologies in home-based digital rehabilitation: scoping review. JMIR rehabilitation Assist Technol 10:e43615\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhao Z et al (2024) A Strabismus Widespread Screening Method Based on Wearable Eye Tracker. \u003cem\u003eIEEE BioSensors Conference (BioSensors).\u003c/em\u003e IEEE, 2024\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang Z et al (2022) A visual feedback supported intelligent assistive technique for amyotrophic lateral sclerosis patients. Adv Intell Syst 4:2100097\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen J et al (2021) A flexible insole gait monitoring technique for the internet of health things. IEEE Sens J 21:26397\u0026ndash;26405\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGladstone DJ et al (2022) The Fugl-Meyer assessment of motor recovery after stroke: a critical review of its measurement properties. Neurorehabilit Neural Repair 16:232\u0026ndash;240\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang M et al (2022) Gait pattern recognition based on plantar pressure signals and acceleration signals. IEEE Trans Instrum Meas 71:1\u0026ndash;15\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJones TA (2017) Motor compensation and its effects on neural reorganization after stroke. Nat Rev Neurosci 18:267\u0026ndash;280\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTerven J et al (2023) A comprehensive review of yolo architectures in computer vision: From yolov1 to yolov8 and yolo-nas. Mach Learn Knowl Extr 5:1680\u0026ndash;1716\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWei J et al (2022) Chain-of-thought prompting elicits reasoning in large language models. Adv Neural Inf Process Syst 35:24824\u0026ndash;24837\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRadford A et al (2023) Robust speech recognition via large-scale weak supervision. \u003cem\u003eInternational conference on machine learning. PMLR\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-5538299/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5538299/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAt-home rehabilitation for post-stroke patients presents significant challenges, as continuous, personalized care is often limited outside clinical settings. Additionally, the absence of comprehensive solutions addressing diverse rehabilitation needs in home environments complicates recovery efforts. Here, we introduce a smart home platform that integrates wearable sensors, ambient monitoring, and large language model (LLM)-powered assistance to provide seamless health monitoring and intelligent support. The system leverages machine learning enabled plantar pressure arrays for motor recovery assessment (94% classification accuracy), a wearable eye-tracking module for cognitive evaluation, and ambient sensors for precise smart home control (100% operational success, \u0026lt; 1 s latency). Additionally, the LLM-powered agent, Auto-Care, offers real-time interventions, such as health reminders and environmental adjustments, enhancing user satisfaction by 29%. This work establishes a fully integrated platform for long-term, personalized rehabilitation, offering new possibilities for managing chronic conditions and supporting aging populations.\u003c/p\u003e","manuscriptTitle":"A Unified Platform for At-Home Post-Stroke Rehabilitation Enabled by Wearable Technologies and Artificial Intelligence","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-03-20 09:09:12","doi":"10.21203/rs.3.rs-5538299/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"35ebf0d7-f3aa-46e2-b4f8-a3bf154913eb","owner":[],"postedDate":"March 20th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":41033441,"name":"Physical sciences/Engineering/Biomedical engineering"},{"id":41033442,"name":"Physical sciences/Engineering/Electrical and electronic engineering"}],"tags":[],"updatedAt":"2025-03-20T09:09:14+00:00","versionOfRecord":[],"versionCreatedAt":"2025-03-20 09:09:12","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5538299","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5538299","identity":"rs-5538299","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}