Multimodal Sensor Fusion Deep Learning Model for Early Prediction of Freezing of Gait in Parkinson's Disease

Background : Freezing of gait (FoG) is a common and debilitating symptom in individuals with advanced Parkinson’s disease (PD), significantly increasing the risk of falls. Wearable devices have facilitated the detection of FoG and falls, but early prediction remains underexplored. This study investigates the use of multimodal sensor fusion and deep learning for the early prediction of FoG events in PD patients. Research Question : Can a multimodal sensor fusion deep learning model accurately predict FoG events well before time in Parkinson’s disease patients, and how robust is the model to noise and inter-subject variability? Methods : The proposed study utilized Inertial Measurement Unit (IMU), Electromyography (EMG), and Electroencephalography (EEG) signals from PD patients to develop and evaluate deep learning models. The CNN+LSTM architecture was employed and compared with other classifiers. Stratified ten-fold cross-validation was used to assess model accuracy. The robustness of IMU+EMG and IMU+EMG+EEG configurations to noise was tested, and inter-subject performance evaluation was conducted. Pre-FOG detection capabilities were also analyzed to emphasize the importance of temporal dynamics in the multimodal approach. Results : The CNN+LSTM model achieved a high accuracy of 94.45% in predicting FoG events. The IMU+EMG and IMU+EMG+EEG configurations demonstrated robust performance across inter-subject evaluations. The models showed resilience to noise, with the CNN+LSTM and IMU+EMG+EEG configurations maintaining high accuracy. Pre-FOG detection achieved 94.20% accuracy, highlighting the model’s effectiveness in capturing temporal dynamics. Significance : The CNN+LSTM model, particularly in the IMU+EMG+EEG configuration, proves to be a robust and accurate predictor of FoG events in PD patients. The study’s findings underscore the potential clinical impact of multimodal sensor fusion and deep learning in reducing false positives and negatives and enhancing precision, sensitivity, and specificity. These insights are crucial for deploying reliable FoG prediction systems in real-world settings and advancing PD management. Future research should explore additional sensor modalities, transferability to different PD cohorts, longitudinal data, and real-time deployment in clinical environments. Title: Multimodal Sensor Fusion Deep Learning Model for Early Prediction of Freezing of Gait in Parkinson’s Disease 1. Rohit Gupta Dept. of Biomedical Engineering SRM IST, Kattankulathur, Tamil Nadu, India, [email protected] 2. Amit Bhongade Department of Electrical Engineering Indian Institute of Technology Delhi New Delhi, India [email protected] 3. Tapan Kumar Gandhi Department of Electrical Engineering Indian Institute of Technology Delhi New Delhi, India [email protected]

Background : Freezing of gait (FoG) is a common and debilitating symptom in individuals with advanced Parkinson’s disease (PD), significantly increasing the risk of falls. Wearable devices have facilitated the detection of FoG and falls, but early prediction remains underexplored. This study investigates the use of multimodal sensor fusion and deep learning for the early prediction of FoG events in PD patients. Research Question : Can a multimodal sensor fusion deep learning model accurately predict FoG events well before time in Parkinson’s disease patients, and how robust is the model to noise and inter-subject variability?

The proposed study utilized Inertial Measurement Unit (IMU), Electromyography (EMG), and Electroencephalography (EEG) signals from PD patients to develop and evaluate deep learning models. The CNN+LSTM architecture was employed and compared with other classifiers. Stratified ten-fold cross-validation was used to assess model accuracy. The robustness of IMU+EMG and IMU+EMG+EEG configurations to noise was tested, and inter-subject performance evaluation was conducted. Pre-FOG detection capabilities were also analyzed to emphasize the importance of temporal dynamics in the multimodal approach.

The CNN+LSTM model achieved a high accuracy of 94.45% in predicting FoG events. The IMU+EMG and IMU+EMG+EEG configurations demonstrated robust performance across inter-subject evaluations. The models showed resilience to noise, with the CNN+LSTM and IMU+EMG+EEG configurations maintaining high accuracy. Pre-FOG detection achieved 94.20% accuracy, highlighting the model’s effectiveness in capturing temporal dynamics. Significance : The CNN+LSTM model, particularly in the IMU+EMG+EEG configuration, proves to be a robust and accurate predictor of FoG events in PD patients. The study’s findings underscore the potential clinical impact of multimodal sensor fusion and deep learning in reducing false positives and negatives and enhancing precision, sensitivity, and specificity. These insights are crucial for deploying reliable FoG prediction systems in real-world settings and advancing PD management. Future research should explore additional sensor modalities, transferability to different PD cohorts, longitudinal data, and real-time deployment in clinical environments.

Parkinson’s disease, Freezing of Gait, deep learning model, early detection, multi-model approach.

Parkinson’s disease (PD) is the second most prevalent neurodegenerative condition related to aging [1] . The severity and duration of the illness are determined by the degeneration of dopaminergic and other sub-cortical neurons [2]. The presence of both motor and non-motor symptoms clinically distinguishes PD. The motor characteristics include bradykinesia (slowness of movement), cogwheel rigidity (muscular stiffness), resting tremor, festination (hastening of the gait), akinesia (lack of spontaneous movements), diminished arm swing, and poor postural stability. These symptoms harm one’s ability to move and be self-reliant, leading to emotional strain and a decrease in overall well-being [3][4][5]. Freezing of Gait (FoG) is a severe gait dysrhythmic characterized by a sudden and temporary inability to lift the feet off the ground [6] consciously. The measurement of FoG is challenging due to its susceptibility to ambient cues, cognitive input, medicine, and particularly anxiety [6], [7]. Consequently, FoG is more common in-home environments than clinical settings, especially in situations with little light and higher cognitive demands [8]. Schaafsma et al. identified and classified five distinct patterns of freezing in different conditions: when starting to walk, when turning, when navigating narrow passageways, when stopping walking, and when hesitating in broad spaces [9]. The severity of FoG is often evaluated using measures such as the Unified Parkinson’s Disease Rating Scale (UPDRS) and the FoG questionnaire (FoG-Q) [10]. Nevertheless, these subjective scales do not have validation for the frequency (i.e., the number of occurrences each day), the time at which the events start, and the length of the FoG events. Furthermore, FoG mainly manifests during certain ambulatory activities that are only sometimes easily replicated in clinical settings. Several automated techniques have been proposed for the detection of FoG. These approaches include analyzing data collected by EMG systems[11], [12] 3D motion systems [13], foot pressure sensors [14], and accelerometers and gyroscopes [2], [15]–[19] IMU is one of the most widely used sensing modality for gait-related analysis [20]–[23]. FoG detection systems primarily use the spectrum properties of signals obtained from sensors positioned on the body’s lower extremities (ankle and knee). Mazilu et al. used three-dimensional acceleration data obtained from the ankle, knee, and hip of individuals diagnosed with PD to detect episodes of FoG [24]. The authors used two methodologies to train and evaluate the models: ”patient-dependent,” which included using data from each participant for both training and testing, and ”patient-independent,” which utilized leave-one-out cross-validation (LOOCV). The Random Forest (RF) classifiers achieved an average sensitivity of 99.54% and a specificity of 99.96% in their patient-dependent models. The patient-independent models had an average sensitivity of 66.25% and an average specificity of 95.38%, which were considerably lower. Xia et al. used a deep convolutional neural network to autonomously acquire knowledge about characteristics and identify FoG occurrences in 4-second intervals of acceleration data [25]. The patient-dependent models achieved an average sensitivity and specificity of 99.64% and 99.99%, respectively. On the other hand, the patient-independent models acquired an average sensitivity and specificity of 74.43% and 90.60%, respectively. Camps et al. used a deep convolutional neural network consisting of eight layers. They adopted spectral window stacking to merge data from previous and current signal windows obtained from a single IMU placed on the waist of thirteen participants [26]. The network identified episodes of FoG in four previously unseen patients (not part of the training set) with a sensitivity of 91.9% and a specificity of 89.5%. Sigcha et al. conducted a study to examine the performance of several machine-learning techniques for detecting FoG. They attained an average sensitivity and specificity of 87.1% among 21 individuals using the LOOCV method and a network of convolutional and recurrent layers [23]. In addition, Mikos et al. introduced a technique that can dynamically adjust and update a model without relying on patient-specific data. This approach attained a sensitivity of 95.90% and a specificity of 93.05% [18]. While the external cues provided after detection of FoG may assist patients in overcoming freezing, the ability to anticipate FoG before it happens allows for proactive cueing. It may decrease the probability of FoG [27]. Naghavi et al. examined the capacity of ensemble classifiers to detect episodes of FoG both before and after their occurrence [19]. To enhance the accuracy of FoG recognition, they used an oversampling technique and cost-sensitive classification since there were fewer FoG samples compared to normal gait. The ensemble model successfully detected 97.4% of the occurrences, with 66.7% being accurately predicted within a 2-second timeframe before the FoG event. Palmerini et al. eliminated samples identified as FoG and created a dataset consisting of 2-second pre-FoG and regular gait segments, resulting in a binary dataset [28]. The linear discriminant analysis (LDA) classifier achieved an average identification rate of 83% for pre-FoG events in patient-dependent models while accurately identifying only 67% of average gait data. Torvi et al. examined the efficacy of a profound domain adaptation. The PD FoG system was designed to undergo testing in the participants’ homes as a future endeavor. The acceleration and rotational velocity are obtained using wearable IMUs on the ankles. Shimmer Sensing provides these IMUs. The mobile application that has been created will evaluate the data in real-time to detect episodes of FoG and activate time-based signals on a wristwatch. The method aims to handle the diversity in gait data and create a prediction model for a specific patient using data from several individuals [29]. Their algorithm achieved an 88% prediction accuracy within one second before the beginning of FoG in patient-specific models. Using transfer learning methods resulted in an enhanced prediction accuracy of 93%. Pardoel et al. used IMUs and insole plantar pressure sensors to extract features for training and testing classifiers. They employed a LOOCV technique [21]. Using decision-tree ensemble classifiers, 55.2% of FOG episodes were accurately predicted, whereas 13.8% of regular gait intervals were erroneously labelled as pre-FoG. Apart from the IMU, EMG and EEG signals have also been utilized widely for gait analysis and detecting neurodevelopmental disorders [30]–[39] In terms of stability, noise tolerance, and signal quality EEG and EMG signals are complimentary to IMU signals. Hence combing the EMG and EEG signal with IMU may results in more stable and better performance. In the presented research work, a comprehensive study focused on the development and evaluation of a novel FoG event prediction model for individuals with PD has been developed. A significant contribution of this study lies in the meticulous exploration and integration of multimodal sensor data, specifically incorporating IMU, EMG, and EEG signals. The key innovation involves the application of a deep learning architecture that intertwines Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) layers to effectively capture the intricate temporal and spatial patterns intrinsic to FoG dynamics. The research probes the model’s robustness under diverse conditions, including inter-subject variations, noise scenarios, and pre-FOG detection, shedding light on its adaptability and potential real-world utility. This work will establish a foundational framework that advances our comprehension of FoG in PD, paving the way for the development of assistive technologies and intervention strategies to address this debilitating motor symptom.

2.1 Dataset Detail: The main objective of this research is to predict FoG episodes in PD patients. Therefore, selecting a dataset with an adequate number of FoG episodes was crucial. The selected dataset was acquired from publicly accessible sources [40].The dataset comprises EEG, EMG, Skin conductance (SC), and IMU recordings from 12 individuals (6 males and six females) diagnosed with PD. The positions of the sensors are illustrated in Figure 1(a) [40]. The 32-channel wireless MOVE system (MOVE, Brain Products GmbH, Gilching, Germany) collected EEG and EMG data. The device operated at a sampling rate of 1000 Hz. To specifically address brain activity related to FoG, 25 EEG signals were recorded, emphasizing the frontal, parietal, and occipital lobes. The mastoid process of the temporal bone was captured by channels TP9 and TP10, which were used as reference data after preprocessing. The electrooculogram was recorded via the 28th Channel (IO channel), shown in figure 1(b) [40]. The remaining three channels of the MOVE system recorded EMG data from the gastrocnemius (GS) muscle of the right leg and tibia anterior (TA) muscles of both legs, as shown in figure 1(c) [40]. The TDK MPU6050 6-DoF accelerometer (ACC), gyro, and STMicroelectronics STM32 CPU were used to capture IMU and SC data using specialized hardware subsystems. Four inertial sensors were strategically positioned at the lateral tibia of both legs, the fifth lumbar spine (L5) of the waist, and the left arm. The SC acquisition was incorporated into the inertial sensor on the left arm. The sampling rate for SC and ACCs was set at 500 Hz, and the data was saved on a TF memory card. The SC measurement was taken at the left index and middle finger’s distal phalanges. Episodes of FoG, frequently triggered by environmental variables and emotional states, commonly arise in everyday situations. According to the research, common triggers for FoG include manoeuvring in tight places, encountering barriers, and making turns, among other factors. The experimental protocol was carefully crafted to recreate these situations with specific tasks intended to provoke FoG. Crucially, the participants independently dealt with FoG instances without external interference. Throughout the experiment, data were consistently recorded from the beginning to the occurrence of FOG, the participant’s self-management of FoG, and finally, the experiment’s conclusion. Before the investigation commenced, the participants were provided with detailed information about the study, and their written agreement was collected. The experimental methodology has been validated and endorsed by the ethics committee of Xuanwu Hospital at Capital Medical University in Beijing, China (No. 2019-014). The complete experimental procedure was recorded on video for later examination by physicians to identify and categorize periods of FoG and periods without FoG. Upon the experiment’s conclusion, the recorded data were thoroughly examined to verify the absence of any procedural errors. If any problems arise, the designated job is repeated following a 2-minute break. The dataset collection experiment involved a series of actions performed in a specific order. These activities included walking in an arena with quarter turns and U-turns, initiating and concluding a gait, and navigating around obstacles, as depicted in Figure 1(d) [40]. The present study utilizes EEG, EMG, and IMU sensor data from ten participants who have experienced an adequate number of FoG episodes for further analysis. | (c) | (d) | | Figure 1: Experimental setup and instrumentation, (a) Fully instrumented subject, (b) EEG electrode locations and respective labels, (c) Muscles considered for the EMG signal, (d) Experimental pathway with obstacles and turns | 2.2 Signal Processing: Figure 2 shows the signal-processing pipeline utilized for the FoG event prediction. The process integrates information from three primary signals: IMU, EMG, and EEG. Specifically, signals from two IMU sensors (left and right legs), EMG signals from three muscles, and EEG signals from 21 channels were employed. Each signal modality has different characteristics and sampling frequencies. Hence, all the signals are preprocessed separately. The first step involved filtering the IMU signal using a zero-lag, 2 nd -order Butterworth low-pass filter with a cutoff frequency of 10Hz [20], [38]. A zero-lag, 4th-order bandpass Butterworth filter was applied for the EMG signal, with a frequency band spanning from 10 to 450 Hz [35], [37], [38], [41]. The EEG signal has been preprocessed using the EEGLab toolbox [30], [32]. In the first place, the EEG signal was re-referenced with the averaged values of reference electrodes (TP9 and TP10). Further, independent component analysis was performed to filter out the eye blink artifacts from the re-referenced signal. Afterward, a zero-lag, 4 th -order Butterworth bandpass filter with a frequency range of 0.5 to 60 Hz has been applied to filter out the environment noise [36]. Figure 2: Signal-processing pipeline utilized for the FoG event prediction. The IMU signal was acquired at 500 Hz, whereas the EMG and EEG signals were acquired at 1000 Hz. Hence, in the subsequent step, the EMG and EEG signals were down-sampled at 500Hz, equal to the sampling frequency of IMU. Afterward, the initial windowing operation was performed on all the signs to segment the call for further analysis. The previous study [39] highlighted the three main factors to be considered for the FoG cueing system: 1) the processing time, 2) pre-prediction time, and 3) prediction accuracy. The processing time depends on the window size used for FoG prediction and the processing time required by the prediction model. If the window size has been kept too high, the processing time will be higher. However, if it has been kept low, there may not be sufficient information for the prediction model to classify the windowed sample. Hence, the window size should be kept moderate. For the current research, the window size has been held at 0.128 sec (64 samples) for all the signals. Further, the study also suggested the existence of a temporal pattern in the EEG signal even before the five seconds of the FoG event. This indicates that the pre-prediction time up to 5 seconds from the FoG event can be exploited. Hence, the continuous windows of 0.128 sec have been extracted from the five seconds before the FoG event and annotated as FoG event (class 1), i.e., let the FoG event was at time t sec, then the windowing will be done from t-5 sec, and all the windows will be annotated as FoG event (class 1). Figure 3 illustrates the windowing methodology opted for the current research work. Figure 3: Windowing illustration and annotation The presented study compares the performance of various machine learning models with deep learning models to identify the suitable choice for the time ahead of FoG event prediction. To apply the machine learning model, input signal features were extracted for each window, and a feature vector was constituted. for each window of the IMU sensor, six time-domain features were estimated. The estimated features were kurtosis, root mean square, variance, skewness for the individual degree of freedom (DoF), and correlation between each pair of DoF of accelerometer and gyroscope[20]. It resulted in a total of 36 features for each IMU sensor; hence, a total of (36x2)72 features were in the feature vector constituted for the two IMU sensors. Similarly, a total of 23 time-domain features were extracted for each EMG signal [34][33]. Hence, a total of (23x3) 69 features were in the feature vector constructed for the three EMG sensors. Whereas, for the individual window of each EEG sensor twelve time-domain features have been extracted. The extracted features are mean absolute value, window length, root mean square, zero crossing, slope sign change, variance, mean of absolute difference, skewness, kurtosis, minimum, maximum, and entropy. These features are widely accepted and recommended in EEG signal analysis and classification [42]–[46]. Hence, the feature vector for EEG sensors has a total of 300 (25x12) features. The presented study compares the performance of individual sensing modalities along with their combination to find the most suitable modality for FoG predictions. Further, the performance of the five classifiers has also been estimated and compared to select the suitable classifier for the FoG prediction. Moreover, the performance of the deep learning algorithm was also evaluated and compared with the machine learning algorithm. 2.3 Learning Models: The current research work compares the performance of five machine-learning algorithms along with the deep-learning algorithm. The performance of the proposed FoG event prediction algorithm has been estimated for five different classifiers (linear and non-linear). The compared classifiers were support vector machine (SVM), LDA, decision tree (DT), RF, and NN. Table 1 depicts the parameters of all the classifiers [37], [47], [48]. Further, a hybrid 1D CNN-LSTM model has been developed for FoG event prediction. Table 2 and figure 4 depict the detailed information of the hybrid deep neural network (CNN+LSTM) along with its architecture, respectively. Adam optimizer with a learning rate of 0.001, beta 1 of 0.9, and beta 2 of 0.999 was used on the binary cross-entropy loss during the training [39]. A 10-fold cross-validation strategy has been used to validate the performance of prediction models. Table 1: Parameters of various classifiers | Support vector machine | One vs. one and quadratic polynomial kernel | | Linear discriminant analysis | Linear discriminator with zero threshold on linear coefficients | | Decision tree | Minimum number of instances per leaf: 2, Confidence factor 0.25 | | Random forest | Number of decision trees:20 Maximum Depth:05 | | Neural Network | Number of hidden layers: 01, Number of neurons=10 Training algorithm: Backpropagation, Performance measure: Mean square error | Table 2: Parametric information of Hybrid deep neural network (CNN+LSTM) | 1 | Conv1D | ReLu | 3x1 | 32 | 1 | 32x62 | 128 | | 2 | BatchNormalization | 32x62 | 128 | |||| | 3 | Activation | ReLu | 32x62 | 0 | ||| | 4 | MaxPooling1D | 2x1 | 32x31 | 0 | ||| | 5 | Conv1D_1 | ReLu | 3x1 | 64 | 1 | 32x29 | 6208 | | 6 | BatchNormalization_1 | 32x29 | 256 | |||| | 7 | Activation_1 | ReLu | 32x29 | 0 | ||| | 8 | MaxPooling1D_1 | 2x1 | 32x14 | 0 | ||| | 9 | LSTM | ReLu | 50 | 50x14 | 23000 | || | 10 | LSTM_1 | ReLu | 50 | 50 | 20200 | || | 11 | Flatten | 50 | 0 | |||| | 12 | Dense | ReLu | 64 | 64 | 3264 | || | 13 | BatchNormalization_2 | 64 | 256 | |||| | 14 | Activation_2 | ReLu | 64 | 0 | ||| | 15 | Dense_1 | Sigmoide | 1 | 1 | 65 | || | 16 | Activation_3 | Sigmoide | 1 | 0 | Figure 4: Hybrid deep neural network (CNN+LSTM) architecture 2.4 Performance measures: The performance of the proposed FoG prediction models has been compared using five performance measures, namely, % classification accuracy, precision, sensitivity, specificity, and F1-score. % classification accuracy represents the proportion of correctly classified instances out of the total instances. High accuracy is desired, but it may not be sufficient for imbalanced datasets. It does not provide insights into the distribution of correctly and incorrectly classified instances across classes. Precision is the ratio of true positive predictions to the total positive predictions. It is crucial when the cost of false positives is high. Precision helps to assess the model’s ability to avoid false positives. Whereas sensitivity is the ratio of true positive predictions to the total actual positive instances. It is vital when the cost of false negatives is high. Sensitivity helps to evaluate how well the model identifies positive instances. Specificity is the ratio of true negative predictions to the total actual negative instances. It assesses the model’s ability to correctly identify negative instances. The F1-score is the harmonic mean of precision and sensitivity. It is a useful index for an uneven class distribution. It provides a balance between precision and sensitivity [32]. All the performance measures have been derived from the confusion matrix (Eq 1-5). | \(Precision\ =\ \frac{T_{P}}{T_{P}+F_{P}}\) | (2) | | \(Sensitivity\ =\ \frac{T_{P}}{T_{P}+F_{N}}\) | (3) | | \(Specificity\ \ =\ \frac{T_{N}}{T_{N}+F_{P}}\) | (4) | | \(F1\ score\ \ =\ \frac{2\times Precision\ \times Recall}{Precision+Recall}\) | (5) | Here, \(T_{P},\ T_{N},F_{P},\ \text{and\ }F_{N}\) are true positive, true negative, false positive and false negative, respectively.

and Discussion 3.1 Stratified Ten-fold Cross-Validation: Table 3 depicts the % classification accuracy for different learning models and sensing modalities with Stratified Ten-fold Cross-Validation. The comprehensive evaluation of FoG event prediction models across various input modalities and classifiers reveals intriguing patterns and provides valuable insights for advancing our understanding and management of PD. The %classification accuracy serves as a quantitative measure, reflecting the model’s overall performance. Notably, the multimodal configurations, especially IMU+EMG+EEG, consistently exhibit superior accuracy, with an impressive 94.45±1.02%. This underscores the importance of leveraging the complementary information encoded in IMU, EMG, and EEG signals for a more holistic understanding of FoG dynamics. Table 3: % classification accuracy for different learning models and sensing modalities with Stratified Ten-fold Cross-Validation | IMU | 60.76±5.01 | 61.23±6.02 | 62.40±5.10 | 63.51±4.44 | 64.02±4.19 | 65.13±4.31 | | EMG | 49.64±3.06 | 50.24±4.55 | 52.11±3.10 | 54.63±3.80 | 54.87±3.50 | 61.26±5.02 | | EEG | 51.59±3.01 | 53.63±4.72 | 56.57±2.41 | 58.39±3.49 | 60.31±3.20 | 62.15±4.46 | | IMU+EMG | 76.64±2.95 | 78.10±5.70 | 79.15±2.48 | 79.46±3.74 | 83.44±3.40 | 84.96±5.19 | | IMU+EEG | 71.37±2.07 | 71.56±4.18 | 71.81±1.87 | 71.82±3.04 | 73.49±2.35 | 76.75±4.08 | | EEG+EMG | 60.16±1.60 | 61.59±2.12 | 61.62±1.00 | 63.09±1.13 | 64.74±1.45 | 64.99±2.11 | | IMU+EMG+EEG | 83.47±2.10 | 87.33±1.94 | 87.82±1.20 | 88.20±1.83 | 89.36±1.30 | 94.45±1.02 | Analyzing individual modalities, EEG stands out as a crucial contributor, achieving accuracy scores ranging from 51.59% to 62.15%. This reinforces the notion that capturing neural activity provides valuable insights into FoG episodes. On the other hand, the combination of IMU and EMG (IMU+EMG) demonstrates remarkable accuracy, ranging from 76.64% to 84.96%. This suggests that the fusion of kinetic and muscle activity information is particularly effective in predicting FoG. Interestingly, when all three modalities are combined (IMU+EMG+EEG), a notable increase in accuracy is observed across all classifiers, reaching an outstanding 94.45%. The choice of classifiers significantly influences the prediction outcomes. The CNN+LSTM architecture consistently outperforms other classifiers across different modalities. For instance, in the IMU+EMG+EEG configuration, CNN+LSTM achieves the highest accuracy at 94.45%, showcasing the robustness of this deep learning approach in capturing complex patterns within multimodal data. Across various input modalities, it is evident that the CNN+LSTM architecture consistently outperforms other classifiers. This deep learning model, known for its ability to capture complex temporal and spatial patterns in sequential data, demonstrates superior adaptability to the multimodal nature of IMU, EMG, and EEG signals. The CNN+LSTM’s outstanding accuracy in the IMU+EMG+EEG configuration, reaching 94.45%, underscores its efficacy in discerning intricate FoG patterns from the amalgamation of sensor data. Considering individual modalities, the RF classifier emerges as a strong performer, particularly in the IMU, EMG, and EEG configurations. RF’s ensemble learning approach, which aggregates multiple DT, appears effective in handling the diverse and dynamic nature of FoG events. Its performance improvement with multimodal inputs suggests its capability to extract complementary information from different sensor sources. In contrast, traditional machine learning classifiers such as DT and (LDA exhibit lower accuracy compared to their deep learning counterparts. This highlights the challenge of FoG prediction as a complex, nonlinear problem that benefits from the hierarchical feature learning capacity of deep neural networks. SVM, despite being a powerful classifier in various applications, shows modest performance in FoG prediction across different input modalities. This emphasizes the need for algorithms that can robustly capture the intricate patterns specific to PD-induced FoG events. Table 4 presents the other performance measures for different learning models and sensing modalities with Stratified Ten-fold Cross-Validation. The precision values of the prediction models for FoG event prediction provide a nuanced perspective on the classifiers’ ability to correctly identify true positive instances while minimizing false positives. For the DT classifier, varying precision across sensor configurations has been observed. For instance, in the IMU+EMG+EEG setup, DT achieves a precision of 83.9%, indicating its capability to discern FoG events accurately. However, this precision is comparatively lower than the CNN+LSTM model in the same configuration, emphasizing the model’s superiority in capturing complex patterns. LDA consistently exhibits lower precision values across all sensor configurations. For instance, in the IMU configuration, LDA achieves a precision of 65.5%, indicating a relatively higher rate of false positives. This highlights the limitations of LDA in capturing the intricate dynamics associated with FoG. For the SVM classifier, more competitive precision values have been observed, with the IMU+EMG+EEG setup reaching 87.8%. SVM, known for its effectiveness in high-dimensional spaces, benefits from the multimodal fusion of IMU, EMG, and EEG signals, resulting in improved FoG detection accuracy. RF consistently demonstrates high precision across different sensor configurations. In the IMU+EMG setup, RF achieves a precision of 77.9%, showcasing its robustness in minimizing false positives. RF’s ensemble learning approach proves beneficial in capturing the diverse patterns associated with FoG. The NN classifier consistently outperforms traditional machine learning models. In the IMU+EMG+EEG setup, NN achieves a precision of 89.4%, showcasing its ability to learn intricate patterns from multimodal sensor data. The CNN+LSTM model consistently exhibits superior precision across all sensor configurations. In the IMU+EMG setup, CNN+LSTM achieves a precision of 71.2%, surpassing other models. This result emphasizes the effectiveness of deep learning techniques, particularly in capturing hierarchical features from diverse sensor inputs. Table 4: Various performance measures for different learning models and sensing modalities with Stratified Ten-fold Cross-Validation | DT | LDA | SVM | RF | NN | CNN+LSTM | | | IMU | 0.699±0.054 | 0.655±0.063 | 0.611±0.042 | 0.673±0.057 | 0.712±0.065 | 0.701±0.055 | | EMG | 0.477±0.021 | 0.519±0.049 | 0.611±0.041 | 0.523±0.041 | 0.677±0.041 | 0.632±0.049 | | EEG | 0.517±0.022 | 0.604±0.053 | 0.550±0.037 | 0.563±0.038 | 0.642±0.030 | 0.664±0.048 | | IMU+EMG | 0.804±0.023 | 0.856±0.055 | 0.778±0.022 | 0.779±0.052 | 0.858±0.040 | 0.892±0.048 | | IMU+EEG | 0.780±0.021 | 0.752±0.040 | 0.707±0.018 | 0.752±0.026 | 0.798±0.034 | 0.780±0.042 | | EEG+EMG | 0.592±0.009 | 0.689±0.036 | 0.598±0.014 | 0.650±0.015 | 0.730±0.028 | 0.685±0.012 | | IMU+EMG+EEG | 0.839±0.008 | 0.904±0.041 | 0.930±0.006 | 0.874±0.003 | 0.932±0.001 | 0.959±0.003 | | Sensitivity | |||||| | IMU | 0.474±0.035 | 0.497±0.046 | 0.585±0.040 | 0.540±0.051 | 0.539±0.045 | 0.536±0.047 | | EMG | 0.461±0.028 | 0.404±0.031 | 0.395±0.027 | 0.474±0.034 | 0.400±0.031 | 0.539±0.045 | | EEG | 0.432±0.020 | 0.429±0.025 | 0.490±0.032 | 0.544±0.033 | 0.504±0.033 | 0.516±0.032 | | IMU+EMG | 0.683±0.023 | 0.668±0.019 | 0.738±0.048 | 0.766±0.034 | 0.774±0.051 | 0.774±0.045 | | IMU+EEG | 0.619±0.015 | 0.616±0.017 | 0.685±0.044 | 0.619±0.015 | 0.628±0.029 | 0.695±0.038 | | EEG+EMG | 0.542±0.010 | 0.515±0.010 | 0.565±0.031 | 0.552±0.012 | 0.521±0.022 | 0.548±0.015 | | IMU+EMG+EEG | 0.786±0.012 | 0.822±0.007 | 0.809±0.034 | 0.863±0.010 | 0.826±0.029 | 0.916±0.023 | | Specificity | |||||| | IMU | 0.763±0.072 | 0.730±0.073 | 0.659±0.050 | 0.733±0.071 | 0.754±0.065 | 0.768±0.058 | | EMG | 0.529±0.034 | 0.606±0.044 | 0.681±0.043 | 0.611±0.057 | 0.746±0.055 | 0.686±0.040 | | EEG | 0.599±0.038 | 0.665±0.043 | 0.634±0.028 | 0.620±0.048 | 0.706±0.047 | 0.730±0.038 | | IMU+EMG | 0.844±0.046 | 0.891±0.048 | 0.834±0.034 | 0.819±0.057 | 0.888±0.046 | 0.917±0.032 | | IMU+EEG | 0.814±0.039 | 0.809±0.042 | 0.747±0.027 | 0.811±0.016 | 0.841±0.044 | 0.830±0.022 | | EEG+EMG | 0.657±0.021 | 0.732±0.016 | 0.662±0.007 | 0.709±0.010 | 0.788±0.038 | 0.751±0.011 | | IMU+EMG+EEG | 0.875±0.010 | 0.920±0.010 | 0.943±0.006 | 0.897±0.008 | 0.950±0.037 | 0.968±0.004 | | F1 Score | |||||| | IMU | 0.565±0.042 | 0.565±0.040 | 0.598±0.053 | 0.599±0.057 | 0.613±0.061 | 0.608±0.054 | | EMG | 0.469±0.033 | 0.454±0.020 | 0.480±0.039 | 0.497±0.039 | 0.503±0.043 | 0.582±0.048 | | EEG | 0.471±0.032 | 0.502±0.021 | 0.518±0.042 | 0.553±0.042 | 0.565±0.047 | 0.581±0.043 | | IMU+EMG | 0.739±0.033 | 0.750±0.025 | 0.757±0.058 | 0.773±0.052 | 0.814±0.063 | 0.829±0.049 | | IMU+EEG | 0.690±0.027 | 0.677±0.018 | 0.696±0.030 | 0.679±0.026 | 0.703±0.036 | 0.735±0.030 | | EEG+EMG | 0.566±0.019 | 0.589±0.012 | 0.581±0.023 | 0.597±0.020 | 0.608±0.020 | 0.609±0.012 | | IMU+EMG+EEG | 0.811±0.020 | 0.861±0.002 | 0.865±0.034 | 0.869±0.019 | 0.876±0.015 | 0.937±0.015 | The IMU consistently demonstrates strong precision values, underscoring its importance in capturing limb movement dynamics. The combination of IMU with EMG further enhances precision, leveraging both inertial and muscle activity information. The inclusion of EEG in multimodal setups consistently refines precision, providing insights into neural activity. The precision values obtained from the FoG prediction model have significant practical implications, especially in the context of PD management. The high precision values observed in the IMU+EMG+EEG configuration, particularly with the NN classifier and CNN+LSTM models, suggest that these models are adept at accurately identifying genuine FoG occurrences. This precision is of paramount importance in real-world scenarios, as it implies a reduced likelihood of false alarms or unnecessary interventions. In the clinical context, false positives could lead to unnecessary interventions, causing anxiety and inconvenience to patients. Therefore, a high precision rate ensures that interventions, such as cueing mechanisms or medication adjustments, are implemented judiciously when truly needed. On the other hand, lower precision values, as seen with the LDA classifier, highlight the challenges associated with certain modeling approaches. In practical terms, a lower precision implies a higher risk of misclassifying non-FoG instances as FoG, potentially leading to unwarranted interventions. This could be concerning for both patients and healthcare providers, as it may result in unnecessary adjustments to treatment plans or lifestyle modifications. The results highlight the complementary nature of IMU and EMG signals. The IMU, capturing inertial information from limb movements, and EMG, measuring muscle activity, together contribute to a robust FoG detection system. The added inclusion of EEG signals further refines the precision, as it provides insights into the brain’s electrical activity, enhancing the overall accuracy of the prediction models. The Sensitivity results reveal interesting patterns across different classifiers and input modalities. For individual sensor inputs, IMU exhibits varied sensitivities with SVM demonstrating the highest at 58.5% and LDA the lowest at 47.4%. EMG, on the other hand, shows diverse performance with SVM reaching 47.4%, while CNN+LSTM achieves the highest at 53.9%. EEG presents relatively consistent sensitivities across classifiers, ranging from 43.2% to 54.4%. Combining sensors in IMU+EMG and IMU+EEG configurations generally improves sensitivity, with IMU+EMG+EEG consistently yielding the highest values, reaching 78.6% with Decision Trees and 91.6% with CNN+LSTM. The specificity values for the prediction model, as outlined in the table, shed light on the classifiers’ ability to accurately identify true negatives, i.e., instances where Freezing of Gait (FoG) is correctly recognized as absent. Across various sensor combinations, the results indicate nuanced patterns. The IMU+EMG+EEG combination consistently stands out with high specificity values, suggesting that the fusion of data from IMU, EMG, and EEG contributes to effective discrimination between FoG and non-FoG states. Interestingly, while certain classes of SVM and CNN+LSTM exhibit higher specificities, the overall performance also depends on the sensor modality. For instance, the EMG sensor, capturing muscle activity, appears to play a crucial role in enhancing specificity. Across various sensor combinations and classifiers, the IMU+EMG+EEG combination consistently demonstrates superior specificity, showcasing its effectiveness in distinguishing FoG from non-FoG states. Notably, SVM and CNN+LSTM classifiers exhibit robust specificities compared to other algorithms. The numerical values underline the high specificity achieved by the IMU+EMG+EEG combination, reaching up to 97% with CNN+LSTM, emphasizing the significance of multimodal sensor fusion for accurate FoG prediction. The F1 score serves as a comprehensive metric that balances the trade-off between precision and sensitivity. It is particularly crucial in scenarios like FoG prediction, where both false positives and false negatives can have significant consequences. Across different sensor combinations and classifiers, the IMU+EMG+EEG configuration consistently demonstrates high F1 scores, reflecting its effectiveness in achieving a harmonious blend of precision and sensitivity. Notably, the CNN+LSTM classifier consistently outperforms others in terms of F1 score, indicating its suitability for FoG detection. The numerical values highlight the robustness of the IMU+EMG+EEG combination, achieving F1 scores of around 93.7% with CNN+LSTM. The comprehensive evaluation of the FoG prediction models, considering % classification accuracy, precision, sensitivity, specificity, and F1-score, reveals valuable insights. The IMU+EMG+EEG configuration consistently outperforms other sensor combinations, demonstrating high accuracy, precision, sensitivity, and F1-score, coupled with remarkable specificity. Specifically, the CNN+LSTM classifier stands out as the most effective model, consistently achieving superior performance across various metrics. This highlights the importance of leveraging multimodal sensor data, as the fusion of IMU, EMG, and EEG signals significantly enhances the predictive capabilities of the model. The CNN+LSTM classifier emerges as the optimal choice, showcasing its robustness and ability to capture complex temporal dependencies within the data. The IMU+EMG+EEG combination proves to be a potent sensor configuration, emphasizing the importance of leveraging multiple modalities for a more comprehensive understanding of FoG events. 3.2 Inter-subject performance: Figure 5 shows the methodology utilized to estimate the inter-subject performance of the FoG event prediction model. The inter-subject performance has been quantified for individual subjects, by considering their data as test data and the data of the rest of the subjects as training data. The inter-subject performance has been estimated for all sensor modalities and learning model combinations. Figure 5: Methodology utilized to estimate the inter-subject performance of the FoG event prediction model Figure 6 shows the inter-subject performance, %classification accuracy, of all the prediction model and sensor modalities along with the % reduction in performance as compared to the performance of stratified ten-fold cross-validation. The investigation into inter-subject performance provides a detailed understanding of the classification accuracy and robustness across diverse sensor configurations and classifiers. Examining the numerical results for each combination reveals distinctive patterns. In the case of IMU, the accuracy ranged from 57.88% to 63.89%, showcasing stable performance. EMG demonstrated accuracy between 47.21% and 60.29%, with minimal reductions, emphasizing its reliability. EEG, with accuracy from 49.24% to 60.35%, displayed modest reductions, indicating reasonable stability. IMU+EMG exhibited high accuracy (73.26% to 83.43%) and minor reductions, underscoring its robustness. IMU+EEG, ranging from 68.28% to 75.42%, displayed minimal reductions, highlighting stability. EEG+EMG, with accuracy from 57.23% to 63.90%, demonstrated moderate reductions, suggesting variability. IMU+EMG+EEG achieved remarkable accuracy (up to 93.70%) with extremely low reductions, showcasing robustness. Overall, certain combinations, particularly IMU+EMG and IMU+EMG+EEG, stood out for their superior accuracy and minimal reduction in performance, affirming the potential of multimodal sensor fusion in capturing diverse inter-subject patterns effectively. Figure 6: Inter-subject performance of the various learning models with % reduction in accuracy as compared to the stratified ten-fold cross-validation (black bar depicting the % reduction in classification accuracy) 3.3 Robustness towards noise: Noise tolerance is one of the most important and desired characteristics of any physiological signal-based prediction model. This can be achieved at the hardware level as well as the software level. However, at the software level, it is very convenient and efficient as it does not require additional electronic components. Among the various types of noises, the power lines interface (50 Hz) is one of the most common types of noise. To quantify the performance of the proposed FOG event prediction model towards noise tolerance, two scenarios have been considered here. Scenario 1, figure 7 (a), the performance of the prediction model has been estimated for the unfiltered signal, i.e. the testing of the prediction model has been performed with an unfiltered signal, whereas the training process utilized the filtered signal. In scenario 2, figure 7 (b), the prediction model is trained using the filtered signal. However, during the testing phase, an additional noise of 50Hz has been added to the signals. A sine wave with a random phase, without harmonics, has been added to the signals to the required SNR level with an arbitrary frequency between 49.5-50.5Hz and has been utilized to simulate the power line noise [49]. (a) (b) Figure 7: Methodology to estimate the noise tolerance ability of the FoG prediction model (a) scenario 1: internal noise, (b) scenario 2: 50 Hz power line interference. The investigation into noise tolerance, particularly in Scenario 1, reveals intriguing insights into the performance of the FOG event prediction model when subjected to unfiltered signals during testing. Figure 8 shows the % classification accuracy of the FoG prediction model in scenario 1 noise. The classification accuracy across different sensor configurations and classifiers shows notable variations. For instance, using only the IMU signals, the accuracy ranges from 55.25% to 61.94%, while for the combination of IMU, EMG, and EEG signals, the accuracy spans from 78.14% to 92.22%. Interestingly, the RF and hybrid deep neural network (CNN+LSTM) consistently demonstrate robust performance across sensor configurations. The % reduction in classification accuracy compared to stratified ten-fold cross-validation highlights the impact of introducing unfiltered signals during testing. The reductions vary across sensors and classifiers, with certain combinations experiencing more significant decreases than others. Notably, IMU+EMG+EEG with CNN+LSTM stands out as the most resilient combination, showcasing only a 2.36% reduction. Figure 8: % classification accuracy of the FoG prediction model in scenario 1 noise (inner black bar depicts the % reduction in accuracy as compared to the stratified ten-fold cross-validation) In Scenario 2, where noise tolerance is evaluated by adding a simulated 50Hz power line noise during testing, the classification accuracy of the FOG event prediction model demonstrates intriguing patterns across different sensor configurations and classifiers. Figure 9 shows the % classification accuracy of the FoG prediction model in scenario 2 noise. The accuracy varies from 43.68% to 91.22%, showcasing the resilience of the model to the introduced noise. Notably, the hybrid deep neural network (CNN+LSTM) consistently exhibits high accuracy across sensor combinations, with IMU+EMG+EEG reaching an impressive 91.22%. Analyzing the % reduction in classification accuracy compared to stratified ten-fold cross-validation reveals valuable insights. Some combinations, such as IMU and EMG, experience notable reductions ranging from 10.42% to 14.27%, highlighting the sensitivity of these configurations to the added noise. However, the IMU+EMG+EEG combination stands out with only a 3.41% reduction, emphasizing its robustness in noise-affected scenarios. Figure 9: % classification accuracy of the FoG prediction model in scenario 2 noise (inner black bar depicts the % reduction in accuracy as compared to the stratified ten-fold cross-validation) Comparing the results between Scenario 1 and Scenario 2 sheds light on the robustness and adaptability of the FOG event prediction model in the presence of noise. In Scenario 1, where the model is tested with an unfiltered signal but trained on a filtered one, the classification accuracy ranges from 44.91% to 79.79%. Interestingly, the CNN+LSTM consistently outperformed other classifiers across sensor combinations. However, when assessing the % reduction in accuracy compared to stratified ten-fold cross-validation, some configurations exhibited significant sensitivity to unfiltered signals, with reductions ranging from 5.18% to 9.85%. In contrast, Scenario 2, where a simulated 50Hz power line noise was added during testing, presented a different perspective. The classification accuracy varied from 43.68% to 91.22%, with the CNN+LSTM again demonstrating resilience to added noise. Remarkably, the IMU+EMG+EEG configuration maintained high accuracy (91.22%) even with the introduced noise, showcasing its robust performance. The % reduction in accuracy, while still observable, showed intriguing patterns, with some configurations experiencing less impact from the added noise compared to Scenario 1. This comparison underscores the model’s ability to adapt and mitigate the effects of noise, especially in Scenario 2, where it was directly exposed to the simulated interference. The variations in accuracy and % reduction emphasize the importance of considering noise scenarios during model development and testing. The findings provide valuable insights into the model’s performance under different noise conditions, offering guidance for its deployment in real-world applications where signal quality may be compromised. The robustness demonstrated in Scenario 2, particularly by the CNN+LSTM and IMU+EMG+EEG configurations, underscores their potential for reliable FOG prediction in environments prone to external interferences. 3.4 Pre-FoG detection performance: The pre-FoG detection is one of the most essential features of the FoG event prediction module. It signifies the ability of time ahead prediction of FoG events. If the FoG event can be predicted well on time, the cueing mechanism can be initiated, and a seamless gait transition should be achieved for the PD patient. To quantify the performance of the FoG prediction module for the time ahead prediction, five different pre-FoG time stamps, -1, -2, -3, -4, and -5 sec, have been considered. Figure 10 shows the segmentation, and annotation methodology for the dataset preparation. A total of five separate datasets have been prepared for time ahead of FoG event prediction for t=-1, -2, -3, -4, and -5 sec, respectively. For t=-1 sec dataset, all W1 widows, of length 0.125sec, prior to the FoG event have been segmented for all available FoG events and annotated as FoG events. Similarly, the t=-2, -3, -4, and -5sec datasets have been prepared and utilized for performance estimation of the developed module. Figure 10: Segmentation, and annotation methodology for the dataset preparation to estimate the Pre-FoG detection performance Table 5 illustrates the % classification accuracy of FoG prediction model for different pre-FoG time stamp. The results show that the IMU consistently showcased commendable predictive capabilities, maintaining an accuracy range of 63.97% to 64.69% across different temporal contexts. This stability suggests that IMU data remains a reliable source for capturing temporal patterns associated with FoG events. EMG signal demonstrated robustness, achieving a noteworthy accuracy of 64.54%, emphasizing its effectiveness in predicting FoG events. Despite a slight reduction to 63.73%, EMG maintained competitive predictive capabilities, reinforcing its reliability over time. EEG signal displayed competitive accuracy, reaching 61.81% with CNN+LSTM at t = -1 sec. This stability persisted across different temporal contexts, with EEG maintaining an accuracy range of 61.02% to 61.24%. These results emphasize the significance of EEG in the multimodal approach for FoG event prediction in PD. The multimodal combination of IMU, EMG, and EEG emerged as a powerful predictor, achieving an impressive accuracy of 94.20% at t = -1sec. This superiority persisted over time, with accuracies of 93.22% and 92.87% at t = -2sec and t = -3sec, respectively. Moreover, all sensing modalities and learning model combinations have shown a reduction in % classification accuracy as the pre-FoG detection time increases. The CNN+LSTM model consistently outperformed other prediction models, highlighting its efficacy in capturing temporal dependencies and complex patterns in the data. Overall, these findings emphasize the importance of considering both sensor modalities and temporal dynamics in developing robust FoG prediction models for PD. The multimodal approach, particularly leveraging CNN+LSTM, stands out as a promising avenue for enhancing predictive capabilities and advancing the understanding of FoG events in PD. Table 5: % classification accuracy of FoG prediction model for different pre-FoG time stamps. | DT | LDA | SVM | RF | NN | CNN+LSTM | | | IMU | 60.12 | 60.75 | 61.75 | 63.33 | 63.45 | 64.69 | | EMG | 49.07 | 49.65 | 51.90 | 54.03 | 54.64 | 60.86 | | EEG | 51.39 | 52.99 | 56.19 | 57.71 | 59.74 | 61.81 | | IMU+EMG | 75.77 | 77.79 | 78.24 | 78.73 | 83.20 | 84.67 | | IMU+EEG | 70.74 | 70.70 | 71.07 | 71.11 | 73.10 | 76.47 | | EEG+EMG | 59.95 | 60.86 | 60.88 | 62.47 | 64.55 | 64.54 | | IMU+EMG+EEG | 83.03 | 86.70 | 87.03 | 87.64 | 89.05 | 94.20 | | Pre-FOG time=-2sec | |||||| | IMU | 59.14 | 59.62 | 60.51 | 61.90 | 62.22 | 64.11 | | EMG | 48.29 | 49.37 | 51.14 | 53.04 | 53.93 | 60.41 | | EEG | 50.78 | 52.29 | 55.25 | 56.63 | 58.65 | 61.24 | | IMU+EMG | 75.00 | 76.19 | 77.85 | 77.67 | 82.03 | 83.89 | | IMU+EEG | 69.89 | 70.46 | 69.92 | 70.75 | 71.30 | 75.42 | | EEG+EMG | 58.70 | 60.70 | 60.57 | 62.14 | 63.52 | 64.13 | | IMU+EMG+EEG | 81.34 | 85.45 | 85.91 | 86.69 | 87.93 | 93.22 | | Pre-FOG time=-3sec | |||||| | IMU | 57.87 | 59.71 | 60.91 | 61.02 | 62.13 | 63.97 | | EMG | 48.10 | 48.36 | 50.42 | 52.97 | 52.26 | 60.26 | | EEG | 49.35 | 51.27 | 55.00 | 56.02 | 58.65 | 61.02 | | IMU+EMG | 73.33 | 74.36 | 75.65 | 76.16 | 79.67 | 83.38 | | IMU+EEG | 68.98 | 69.63 | 69.53 | 68.73 | 70.72 | 75.53 | | EEG+EMG | 57.86 | 59.23 | 59.32 | 60.86 | 61.53 | 63.73 | | IMU+EMG+EEG | 81.35 | 84.21 | 85.37 | 85.94 | 87.08 | 92.87 | | Pre-FOG time=-4sec | |||||| | IMU | 57.79 | 57.92 | 57.96 | 60.34 | 60.14 | 63.10 | | EMG | 46.76 | 47.70 | 48.75 | 51.80 | 51.54 | 59.61 | | EEG | 48.81 | 50.93 | 53.37 | 55.28 | 56.92 | 60.31 | | IMU+EMG | 71.31 | 72.51 | 74.30 | 74.82 | 77.39 | 82.46 | | IMU+EEG | 67.16 | 67.04 | 67.64 | 68.39 | 69.29 | 74.52 | | EEG+EMG | 55.79 | 57.75 | 58.62 | 58.51 | 61.53 | 62.95 | | IMU+EMG+EEG | 79.16 | 82.91 | 83.13 | 81.70 | 83.20 | 91.90 | | Pre-FOG time=-5sec | |||||| | IMU | 56.22 | 57.88 | 57.22 | 60.31 | 59.45 | 62.60 | | EMG | 47.19 | 46.31 | 49.21 | 49.59 | 50.94 | 58.96 | | EEG | 48.60 | 49.21 | 52.03 | 53.46 | 55.08 | 59.90 | | IMU+EMG | 71.76 | 71.99 | 74.77 | 73.87 | 76.31 | 81.61 | | IMU+EEG | 65.17 | 66.65 | 67.17 | 66.95 | 67.75 | 73.92 | | EEG+EMG | 57.33 | 57.08 | 56.86 | 58.77 | 60.53 | 62.70 | | IMU+EMG+EEG | 79.42 | 82.01 | 80.37 | 82.78 | 81.65 | 91.61 | The performance of the proposed FoG events prediction model was rigorously benchmarked against the existing state-of-the-art models, revealing compelling insights into its superiority. Table 6 provides a comparative overview, showcasing the proposed model’s efficacy across diverse modalities, classifiers, and pre-FOG prediction times. Notably, the proposed model, employing a hybrid Deep Neural Network architecture (CNN+LSTM), achieved an outstanding accuracy of 91.61% in the IMU+EMG+EEG multimodal configuration with a pre-FOG prediction time of 5 seconds. This surpasses the performance of existing models, such as those relying on EEG modalities with MLP and KNN classifiers (71%), ECG+SC with Multivariate Gaussian Distribution (71.3%), IMU with ML algorithms (78.8%), and EEG+IMU with Deep Neural Network (multi-model) (86.2%). The results emphasize the superior predictive capabilities of proposed model, positioning it as a frontrunner in FoG event prediction, thus offering a substantial advancement in the field. Table 6: Comparison of the performance of the proposed FoG events prediction model with the existing models | [50] | EEG | MLP and KNN | 5 | 71.00% | | [17] | ECG +SC | Multivariate Gaussian Distribution | 4.2 | 71.30% | | [51] | IMU | ML algorithm | 2 | 78.80% | | [39] | EEG+IMU | Deep NN (multi-model) | 5 | 86.20% | | Proposed | IMU+EMG+EEG | Hybrid Deep Neural Network (CNN+LSTM) | 5 | 91.61% |

The study presents a holistic evaluation of FoG event prediction models, emphasizing the significance of multimodal sensor fusion and deep learning techniques. CNN+LSTM emerges as a robust classifier, consistently achieving superior performance across various metrics, particularly in the IMU+EMG+EEG configuration with 94.45% accuracy. The novel contributions lie in the exploration of inter-subject performance, noise tolerance, and pre-FOG detection, revealing the adaptability of the proposed models to diverse scenarios. The study underscores the potential clinical impact, reducing false positives and negatives, and enhancing precision, sensitivity, and specificity. The findings offer valuable guidance for deploying reliable FoG prediction systems in real-world settings, contributing to the advancement of PD management. There are several avenues for future research in the domain of FoG event prediction. Further exploration of additional sensor modalities and their potential contributions could enhance the overall understanding of FoG dynamics. Investigating the transferability of the proposed models to different PD cohorts and incorporating longitudinal data could provide insights into the models’ robustness over time. Continuous refinement of noise-tolerant models and real-time deployment in clinical settings may be crucial for practical applications.

[1] T. R. Mhyre, R. Nw, J. T. Boyd, G. Hall, and C. Room, Protein Aggregation and Fibrillogenesis in Cerebral and Systemic Amyloid Disease, vol. 65. 2012. doi: 10.1007/978-94-007-5416-4.[2] A. M. Maitin, J. P. Romero Muñoz, and Á. J. García-Tejedor, “Survey of Machine Learning Techniques in the Analysis of EEG Signals for Parkinson’s Disease: A Systematic Review,” Appl. Sci., vol. 12, no. 14, 2022, doi: 10.3390/app12146967.[3] R. N. Rees, A. J. Noyce, and A. Schrag, “The prodromes of Parkinson’s disease,” Eur. J. Neurosci., vol. 49, no. 3, pp. 320–327, 2019, doi: 10.1111/ejn.14269.[4] R. C. Hughes, “Parkinson’s Disease and its Management,” Bmj, vol. 308, no. 6923, p. 281, 1994, doi: 10.1136/bmj.308.6923.281.[5] M. A. Puhan et al., “Arm Swing as a Potential New Prodromal Marker of Parkinson’s Disease,” Mov. Disord., vol. 37, no. 4, pp. 784–790, 2017, doi: 10.1002/mds.26720.Arm.[6] K. A. Ehgoetz Martens, C. G. Ellard, and Q. J. Almeida, “Does anxiety cause freezing of gait in Parkinson’s disease?,” PLoS One, vol. 9, no. 9, 2014, doi: 10.1371/journal.pone.0106561.[7] J. Spildooren, S. Vercruysse, K. Desloovere, W. Vandenberghe, E. Kerckhofs, and A. Nieuwboer, “Freezing of gait in Parkinson’s disease: The impact of dual-tasking and turning,” Mov. Disord., vol. 25, no. 10, pp. 2563–2570, 2010.[8] K. A. Ehgoetz Martens, F. Pieruccini-Faria, and Q. J. Almeida, “Could Sensory Mechanisms Be a Core Factor That Underlies Freezing of Gait in Parkinson’s Disease?,” PLoS One, vol. 8, no. 5, pp. 6–13, 2013, doi: 10.1371/journal.pone.0062602.[9] J. D. Schaafsma, Y. Balash, T. Gurevich, A. L. Bartels, J. M. Hausdorff, and N. Giladi, “Characterization of freezing of gait subtypes and the response of each to levodopa in Parkinson’s disease,” Eur. J. Neurosci., vol. 10, no. 4, pp. 391–398, 2003.[10] T. Ellis, J. T. Cavanaugh, G. M. Earhart, M. P. Ford, K. B. Foreman, and L. E. Dibble, “Which measures of physical function and motor impairment best predict quality of life in Parkinson’s disease?,” Park. Relat. Disord., vol. 17, no. 9, pp. 693–697, 2011, doi: 10.1016/j.parkreldis.2011.07.004.[11] A. Nieuwboer, R. Dom, W. De Weerdt, K. Desloovere, L. Janssens, and V. Stijn, “Electromyographic profiles of gait prior to onset of freezing episodes in patients with Parkinson’s disease,” Brain, vol. 127, no. 7, pp. 1650–1660, 2004, doi: 10.1093/brain/awh189.[12] B. T. Cole, S. H. Roy, and S. H. Nawab, “Detecting freezing-of-gait during unscripted and unconstrained activity,” Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBS, pp. 5649–5652, 2011, doi: 10.1109/IEMBS.2011.6091367.[13] S.-B. Koh, K.-W. Park, D.-H. Lee, S. J. Kim, and J.-S. Yoon, “Gait Analysis in Patients With Parkinson’s Disease: Relationship to Clinical Features and Freezing,” J. Mov. Disord., vol. 1, no. 2, pp. 59–64, 2008, doi: 10.14802/jmd.08011.[14] J. M. Hausdorff, J. D. Schaafsma, Y. Balash, A. L. Bartels, T. Gurevich, and N. Giladi, “Impaired regulation of stride variability in Parkinson’s disease subjects with freezing of gait,” Exp. Brain Res., vol. 149, no. 2, pp. 187–194, 2003, doi: 10.1007/s00221-002-1354-8.[15] S. T. Moore, H. G. MacDougall, and W. G. Ondo, “Ambulatory monitoring of freezing of gait in Parkinson’s disease,” J. Neurosci. Methods, vol. 167, no. 2, pp. 340–348, 2008, doi: 10.1016/j.jneumeth.2007.08.023.[16] E. E. Tripoliti et al., “Automatic detection of freezing of gait events in patients with Parkinson’s disease,” Comput. Methods Programs Biomed., vol. 110, no. 1, pp. 12–26, 2013, doi: 10.1016/j.cmpb.2012.10.016.[17] L. Borzì, I. Mazzetta, A. Zampogna, A. Suppa, G. Olmo, and F. Irrera, “Wearables and Machine Learning,” Sensors, pp. 1–19, 2021.[18] V. Mikos et al., “Real-time patient adaptivity for freezing of gait classification through semi-supervised neural networks,” Proc. - 16th IEEE Int. Conf. Mach. Learn. Appl. ICMLA 2017, vol. 2017-Decem, pp. 871–876, 2017, doi: 10.1109/ICMLA.2017.00-46.[19] N. Naghavi, A. Miller, and E. Wade, “Towards real-time prediction of freezing of gait in patients with parkinson’s disease: Addressing the class imbalance problem,” Sensors (Switzerland), vol. 19, no. 18, pp. 1–17, 2019, doi: 10.3390/s19183898.[20] A. Bhongade, R. Gupta, and T. K. Gandhi, “Machine Learning-Based Gait Characterization Using Single IMU Sensor,” 3rd IEEE 2022 Int. Conf. Comput. Commun. Intell. Syst. ICCCIS 2022, pp. 263–266, 2022, doi: 10.1109/ICCCIS56430.2022.10037621.[21] S. Pardoel, G. Shalin, J. Nantel, E. D. Lemaire, and J. Kofman, “Early detection of freezing of gait during walking using inertial measurement unit and plantar pressure distribution data,” Sensors, vol. 21, no. 6, pp. 1–14, 2021, doi: 10.3390/s21062246.[22] G. Shalin, S. Pardoel, E. D. Lemaire, J. Nantel, and J. Kofman, “Prediction and detection of freezing of gait in Parkinson’s disease from plantar pressure data using long short-term memory neural-networks,” J. Neuroeng. Rehabil., vol. 18, no. 1, pp. 1–15, 2021, doi: 10.1186/s12984-021-00958-5.[23] L. Sigcha et al., “Deep learning approaches for detecting freezing of gait in parkinson’s disease patients through on-body acceleration sensors,” Sensors (Switzerland), vol. 20, no. 7, 2020, doi: 10.3390/s20071895.[24] S. Mazilu et al., “Online detection of freezing of gait with smartphones and machine learning techniques,” 2012 6th Int. Conf. Pervasive Comput. Technol. Healthc. Work. PervasiveHealth 2012, no. 3, pp. 123–130, 2012, doi: 10.4108/icst.pervasivehealth.2012.248680.[25] Y. Xia, J. Zhang, Q. Ye, N. Cheng, Y. Lu, and D. Zhang, “Evaluation of deep convolutional neural networks for detection of freezing of gait in Parkinson’s disease patients,” Biomed. Signal Process. Control, vol. 46, pp. 221–230, 2018, doi: 10.1016/j.bspc.2018.07.015.[26] J. Camps et al., “Deep learning for freezing of gait detection in Parkinson’s disease patients in their homes using a waist-worn inertial measurement unit,” Knowledge-Based Syst., vol. 139, pp. 119–131, 2018, doi: 10.1016/j.knosys.2017.10.017.[27] A. Nieuwboer et al., “Cueing training in the home improves gait-related mobility in Parkinson’s disease: The RESCUE trial,” J. Neurol. Neurosurg. Psychiatry, vol. 78, no. 2, pp. 134–140, 2007, doi: 10.1136/jnnp.200X.097923.[28] L. Palmerini, L. Rocchi, S. Mazilu, E. Gazit, J. M. Hausdorff, and L. Chiari, “Identification of characteristic motor patterns preceding freezing of gait in Parkinson’s disease using wearable sensors,” Front. Neurol., vol. 8, no. AUG, pp. 1–12, 2017, doi: 10.3389/fneur.2017.00394.[29] V. G. Torvi, A. Bhattacharya, and S. Chakraborty, “Deep Domain Adaptation to Predict Freezing of Gait in Patients with Parkinson’s Disease,” Proc. - 17th IEEE Int. Conf. Mach. Learn. Appl. ICMLA 2018, pp. 1001–1006, 2018, doi: 10.1109/ICMLA.2018.00163.[30] T. Wadhera and M. Mahmud, “Brain Functional Network Topology in Autism Spectrum Disorder: A Novel Weighted Hierarchical Complexity Metric for Electroencephalogram,” IEEE J. Biomed. Heal. Informatics, vol. 27, no. 4, pp. 1718–1725, 2023, doi: 10.1109/JBHI.2022.3232550.[31] T. Wadhera, J. Bedi, and S. Sharma, “Autism spectrum disorder prediction using bidirectional stacked gated recurrent unit with time-distributor wrapper: an EEG study,” Neural Comput. Appl., vol. 35, no. 13, pp. 9803–9818, 2023, doi: 10.1007/s00521-023-08218-4.[32] T. Wadhera and M. Mahmud, “Computing Hierarchical Complexity of the Brain from Electroencephalogram Signals: A Graph Convolutional Network-based Approach,” Proc. Int. Jt. Conf. Neural Networks, vol. 2022-July, pp. 1–6, 2022, doi: 10.1109/IJCNN55064.2022.9892799.[33] R. Gupta and R. Agarwal, “Continuous human locomotion identification for lower limb prosthesis control,” CSI Trans. ICT, vol. 6, no. 1, pp. 17–31, Mar. 2018, doi: 10.1007/s40012-017-0178-4.[34] R. Gupta and R. Agarwal, “Electromyographic Signal-Driven Continuous Locomotion Mode Identification Module Design for Lower Limb Prosthesis Control,” Arab. J. Sci. Eng., vol. 43, no. 12, pp. 7817–7835, Dec. 2018, doi: 10.1007/s13369-018-3193-3.[35] R. Gupta and R. Agarwal, “Human Muscle Synergy Recruitment and Variability Assessment for Walking Speed Prediction Module Design,” IETE J. Res., 2022, doi: 10.1080/03772063.2022.2101555.[36] R. Gupta, A. Bhongade, and T. K. Gandhi, “EEG and EMG fusion-based hand 3D Trajectory Estimation using deep learning model : A preliminary study,” 2023 14th Int. Conf. Comput. Commun. Netw. Technol., pp. 1–6, 2023, doi: 10.1109/ICCCNT56998.2023.10306915.[37] R. Gupta and R. Agarwal, “Single channel EMG-based continuous terrain identification with simple classifier for lower limb prosthesis,” Biocybern. Biomed. Eng., vol. 39, no. 3, pp. 775–788, 2019, doi: 10.1016/j.bbe.2019.07.002.[38] R. Gupta, I. S. Dhindsa, and R. Agarwal, “Continuous angular position estimation of human ankle during unconstrained locomotion,” Biomed. Signal Process. Control, vol. 60, p. 101968, 2020, doi: 10.1016/j.bspc.2020.101968.[39] R. Bajpai and D. Joshi, “A multimodal model-fusion approach for improved prediction of Freezing of Gait in Parkinson’s disease,” IEEE Sens. J., vol. 23, no. 4, pp. 16168–16175, 2023.[40] W. Zhang et al., “Multimodal Data for the Detection of Freezing of Gait in Parkinson’s Disease,” Sci. Data, vol. 9, no. 1, pp. 1–10, 2022, doi: 10.1038/s41597-022-01713-8.[41] R. Gupta, I. S. Dhindsa, and R. Agarwal, “Development and Uncertainty Assessment of Low-Cost Portable EMG Acquisition Module,” Mapan - J. Metrol. Soc. India, 2023, doi: 10.1007/s12647-023-00706-1.[42] R. Panda, P. S. Khobragade, P. D. Jambhule, S. N. Jengthe, P. R. Pal, and T. K. Gandhi, “Classification of EEG signal using wavelet transform and support vector machine for epileptic seizure diction,” Int. Conf. Syst. Med. Biol. ICSMB 2010 - Proc., no. December, pp. 405–408, 2010, doi: 10.1109/ICSMB.2010.5735413.[43] T. K. Gandhi, P. Chakraborty, G. G. Roy, and B. K. Panigrahi, “Discrete harmony search based expert model for epileptic seizure detection in electroencephalography,” Expert Syst. Appl., vol. 39, no. 4, pp. 4055–4062, 2012, doi: 10.1016/j.eswa.2011.09.093.[44] T. Gandhi, B. K. Panigrahi, and S. Anand, “A comparative study of wavelet families for EEG signal classification,” Neurocomputing, vol. 74, no. 17, pp. 3051–3057, 2011, doi: 10.1016/j.neucom.2011.04.029.[45] P. Ofner, A. Schwarz, J. Pereira, D. Wyss, R. Wildburger, and G. R. Müller-Putz, “Attempted Arm and Hand Movements can be Decoded from Low-Frequency EEG from Persons with Spinal Cord Injury,” Sci. Rep., vol. 9, no. 1, pp. 1–15, 2019, doi: 10.1038/s41598-019-43594-9.[46] K. Ren, Z. Chen, Y. Ling, and J. Zhao, “Recognition of freezing of gait in Parkinson’s disease based on combined wearable sensors,” BMC Neurol., vol. 22, no. 1, pp. 1–13, 2022, doi: 10.1186/s12883-022-02732-z.[47] R. Gupta, I. S. Dhindsa, and R. Agarwal, “Surface Electromyogram Feature Set Optimization for Lower Limb Activity Classification,” IETE J. Res., vol. 69, no. 8, pp. 5000–5014, 2023, doi: 10.1080/03772063.2021.1973589.[48] R. Gupta and R. Agarwal, “Single Muscle Surface EMGs Locomotion Identification Module for Prosthesis Control,” Neurophysiology, vol. 51, no. 3, pp. 191–208, May 2019, doi: 10.1007/s11062-019-09812-w.[49] P. McCool, G. D. Fraser, A. D. C. Chan, L. Petropoulakis, and J. J. Soraghan, “Identification of contaminant type in surface electromyography (EMG) signals,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 22, no. 4, pp. 774–783, 2014, doi: 10.1109/TNSRE.2014.2299573.[50] A. M. A. Handojoseno, J. M. Shine, T. N. Nguyen, Y. Tran, S. J. G. Lewis, and H. T. Nguyen, “Using EEG spatial correlation, cross frequency energy, and wavelet coefficients for the prediction of Freezing of Gait in Parkinson’s Disease patients,” Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBS, pp. 4263–4266, 2013, doi: 10.1109/EMBC.2013.6610487.[51] N. Kleanthous, A. J. Hussain, W. Khan, and P. Liatsis, “A new machine learning based approach to predict Freezing of Gait,” Pattern Recognit. Lett., vol. 140, pp. 119–126, 2020, doi: 10.1016/j.patrec.2020.09.011. Information & Authors Information Version history Copyright This work is licensed under a Non Exclusive No Reuse License.

Authors Metrics & Citations Metrics Article Usage 881views 415downloads Citations Download citation Rohit Gupta, Amit Bhongade, Tapan Kumar Gandhi. Multimodal Sensor Fusion Deep Learning Model for Early Prediction of Freezing of Gait in Parkinson's Disease. Authorea. 25 November 2024. DOI: https://doi.org/10.22541/au.173254915.59934298/v1 DOI: https://doi.org/10.22541/au.173254915.59934298/v1 If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download. For more information or tips please see 'Downloading to a citation manager' in the Help menu. Cited by - Surface Electromyography for Parkinson’s Disease Monitoring: A Review of Machine and Deep Learning Techniques, Sensors, 26, 10, (2927), (2026).https://doi.org/10.3390/s26102927 Loading...