Enhancing COVID-19 Forecasts Through Multivariate Deep Learning Models

Enhancing COVID-19 Forecasts Through Multivariate Deep Learning Models | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Enhancing COVID-19 Forecasts Through Multivariate Deep Learning Models Jooha Oh, Zhe Liu, Kyulhee Han, Taewan Goo, Hanbyul Song, Jiwon Park, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5400759/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background It is well known that deep learning (DL) models often struggle with low prediction performance due to data scarcity. This scarcity hampers the effectiveness of DL methods that typically require large datasets to generate reliable forecasts. Recently, several DL models have been proposed for predicting the spread of COVID-19. These models are country specific models and thus use the COVID-19 data only from the target country. To improve COVID-19 forecasting using DL models, we propose multivariate DL models using the additional data available from other countries. Methods Based on the rankings determined by Dynamic Time Warping (DTW) distance, which calculates the similarity of infection trends across countries, univariate DL models using only the target country data were extended to multivariate models which integrated data from the top-ranked countries to optimize performance. We considered seven DL models including the Transformer model, TCN, CNN-LSTM, BiLSTM, GRU, RNN, and LSTM. Results Our results showed that the multivariate transformer model achieved the most significant improvements in forecasting accuracy, with an average reduction of 60.15% in mean root mean square error (RMSE) across the five target countries and five time periods when integrating data from additional countries, compared to univariate models using only the target country data. Additionally, multivariate transformer models consistently demonstrated significant improvements over univariate models in terms of mean RMSE, as evidenced by the Diebold-Mariano test. Other multivariate DL models also showed performance gains, with the TCN model achieving an average reduction in RMSE of 36.28%, followed by CNN-LSTM at 29.47%, BiLSTM at 21.07%, GRU at 21.43%, RNN at 17.46%, and LSTM at 16.38%. Conclusions These findings indicate that leveraging similar infection patterns from data of other countries can provide valuable information for predicting the COVID-19 spread in the target country, especially when data is scarce, thereby enhancing forecasting performance. COVID-19 Forecasting Deep Learning models Transformer model Time series prediction Dynamic Time Warping Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 1. Introduction The COVID-19 pandemic, caused by the SARS-CoV-2 virus and emerging in late 2019, has presented unprecedented challenges to public health systems worldwide. This situation highlights the urgent need for accurate models to predict infection cases, which is crucial for timely public health interventions and effective resource allocation. While various predictive models have been developed, including traditional statistical models such as AutoRegressive Integrated Moving Average (ARIMA) and time series generalized linear models (tsglm), as well as machine learning (ML) models like LightGBM [ 1 , 2 ]. However, these models often show lower performance than deep learning (DL) models. DL models, known for their ability to capture complex temporal patterns, have the potential to outperform traditional models in COVID-19 forecasting tasks. However, DL models typically require large amounts of data to achieve high prediction accuracy, and if data is scarce, it significantly limits their performance. Compared to classical DL models like Long Short-Term Memory (LSTM) networks, the state-of-the-art Transformer model, introduced by Vaswani et al. (2017), requires significantly larger datasets due to its more complex architecture and higher number of parameters than other DL models [ 3 ]. Despite these challenges, Transformers have advanced the field significantly by using self-attention mechanisms, which model dependencies without the sequential limitations of Recurrent Neural Networks (RNNs). This allows for more efficient training and better performance on large datasets [ 4 ]. To integrate more information and further enhance the model performance, recent studies have employed the Temporal Fusion Transformer (TFT), improving forecasting effectiveness by integrating both temporal and static covariates [ 5 ]. Chimmula and Zhang (2020) applied LSTM networks to predict the spread of COVID-19 in Canada, demonstrating the ability of DL models to handle sequential data and learn from multiple sources simultaneously [ 6 ]. To mitigate data limitations, some studies have also used data augmentation techniques and transfer learning, enabling models to leverage additional information and improve forecasting accuracy [ 7 ]. However, most research predicts COVID-19 spread using only historical data from the target country, which may limit the potential of DL models, particularly Transformers. This issue becomes more serious during the early stages of the pandemic or when countries collect data on a weekly basis, as limited datasets restrict the available information for DL model training, leading to suboptimal predictions. To enhance performance of DL models, we proposed a multivariate method that integrates data from other countries to improve predictions by leveraging similar patterns, rather than relying on the univariate model, which uses only data from the target country. Multivariate method can be achieved through two main approaches based on the nature of the models: use data in a multivariate way and in a multiple way. Multivariate models in a multivariate way involves forecasting not just a specific country but multiple countries simultaneously as a vector by integrating multiple time series. This approach is particularly well-suited for DL models, as they can expand the input matrix to capture intricate relationships between variables across different time series. By leveraging their capacity to learn from large, complex datasets and capture long-term dependencies, DL models effectively utilize the additional information from multiple countries. Alternatively, using data in a multiple way involves including data from other countries as additional predictive variables to improve forecasts for a specific country. This approach is particularly meaningful for some traditional statistical models and ML models that cannot use data in a multivariate way. These models can be extended to handle multiple regressors, allowing them to predict a single target variable more effectively. To effectively integrate data via multivariate methods, identifying the most informative countries is crucial, as irrelevant data may introduce noise and reduce performance. To determine these key countries, we use Dynamic Time Warping (DTW) distance to rank and select the most relevant ones, thereby enhancing forecasting performance [ 8 ]. We compared the performance of the various univariate models with multivariate models. We first compared univariate models with multivariate models using data in a multivariate way, including deep learning models such as Transformer, RNN, LSTM, BiLSTM, GRU, CNN-LSTM, and TCN. We further compared univariate models with a multivariate model using data in a multiple way, including LightGBM. Our results demonstrate that multivariate models can improve forecasting performance by integrating information from other sources. Furthermore, comparing benchmark multivariate DL models with multivariate LightGBM, which has shown strong performance in COVID-19 forecasting, highlights the advantages of DL in handling large datasets and capturing complex dependencies. In summary, our study demonstrates that integrating data from multiple countries can significantly enhance the performance of DL models in forecasting COVID-19. By leveraging similar infection patterns across countries, multivariate Transformer models, in particular, show strong potential for improving prediction accuracy. This approach offers a valuable strategy for enhancing COVID-19 forecasting by utilizing data from diverse countries, allowing DL models to better capture complex dependencies and trends. 2. Methods 2.1. Data acquisition and preprocessing We obtained daily COVID-19 case data from five countries—South Korea, Japan, Russia, Italy, and the United States —from the Our World in Data (OWID) repository [ 9 ]. These five countries, including those from Asia, Europe, and the Americas, reflect diverse epidemic response strategies and public health conditions, enhancing the generalizability and applicability of the findings. Additionally, the data quality and completeness from these countries are relatively high, ensuring the credibility of the research results. The raw OWID dataset required preprocessing to handle missing values and correct entries that were recorded weekly instead of daily. Specifically, missing or weekly data points were replaced by a 7-day simple moving average (SMA) [ 10 ]. Since the multivariate models in the multiple way cannot simultaneously predict values for additional countries in the test set, we filled in the data for other countries in the test set with the last observation from the training set. To ensure that the selected countries had populations comparable to those of the five target countries, we excluded countries with populations under 10 million from our analysis, resulting in a total of 90 countries. To determine which countries’ data to integrate with the target country’s data, we ranked the 90 countries based on their similarity to the target country using DTW distance. DTW is a technique used to find the optimal alignment between two time-dependent sequences. The general equation for DTW distance between two sequences \(\:{X}_{1}={\left\{{X}_{11},\:{X}_{12,\:}\dots\:,{X}_{1n}\right\}}^{T}\) and \(\:{X}_{2}={\left\{{X}_{21},\:{X}_{22,\:}\dots\:,{X}_{2n}\right\}}^{T}\) is as following where \(\:P\) is a warping path, which is a sequence of index pairs that define the alignment between \(\:{X}_{1}\) and \(\:{X}_{2}\) , and \(\:d\left({X}_{1i},\:{X}_{2j}\right)\) denotes the Euclidean distance between the points \(\:{X}_{1i}\) and \(\:{X}_{2j}\) . $$\:{\text{D}\text{T}\text{W}(X}_{1},\:{X}_{2})=\text{m}\text{i}\text{n}\left(\sum\:_{(i,\:j)\in\:P}d({X}_{1i},\:{X}_{2j})\right)$$ In our study, it measures the similarity between the time series of daily COVID-19 cases in the target country and those in other countries. DTW is particularly effective for this purpose because it accounts for temporal misalignments and varying speeds in time series data, providing a robust measure of similarity between pairs of time series. Based on the ranking, we treated the number of additional countries in the multivariate model as a parameter, exploring values from 1 to 90, specifically, we explored configurations with 1, 3, 5, 10, 15, 20, 30, 45, 60, and 90 countries, where 90 represents the inclusion of all countries in our dataset. By varying the number of included countries, we aim to explore how the inclusion of additional countries affects the model's performance in capturing the complexities of the pandemic spread. Moreover, we used five distinct intervals to ensure that our approach is robust across varying training and testing periods. The training periods were January 1, 2022, to July 31, 2022; January 1, 2022, to August 31, 2022; January 1, 2022, to September 30, 2022; January 1, 2022, to October 31, 2022; and January 1, 2022, to November 30, 2022. For each training period, the corresponding testing period extended up to 28 days beyond the end of the training period. 2.2. Comparative models and hyperparameters tuning Figure 1 illustrates the differences between the univariate model and the multivariate model. The univariate models are country specific models. In this approach, the time series data from the specific country, denoted as \(\:{X}_{i}\) = \(\:\left\{{x}_{i}^{1},{x}_{i}^{2},\dots\:,{x}_{i}^{t}\right\}\) , where t represents the length of the training period, is used to forecast for that specific country. The predicted values are represented as \(\:\widehat{{Y}_{i}}=\{\widehat{{Y}_{i}^{t+1}},\) \(\:\widehat{{Y}_{i}^{t+2}},\dots\:,\widehat{{Y}_{i}^{t+m}}\}\) , where \(\:m\) represents the length of the test period (Fig. 1 A). This model focuses only on the historical data of the target but does not leverage potentially valuable information from other sources that could enhance the accuracy and robustness of the forecasts. In contrast, multivariate models integrate time series data from multiple sources. As shown in Fig. 1 (B), in the multivariate model using data in a multivariate way, data from both the target country and additional sources are integrated into a matrix format This matrix is then fed into the models to predict both the target country’s values and those of the additional sources simultaneously. Specifically, if \(\:X={\left\{{X}_{1},\:{X}_{2,\:}\dots\:,{X}_{n}\right\}}^{T}\) represents the matrix of time series data from n countries, where \(\:{X}_{i,\:}\) corresponds to the time series data from each different source, the forecasted values are denoted by: \(\:\widehat{Y}=\{\widehat{{Y}_{1}},\widehat{{\:Y}_{2}}\) , … , \(\:{\widehat{{Y}_{n}}\}}^{T}\) ,where \(\:\widehat{{Y}_{i}}=\{\widehat{{Y}_{i}^{t+1}}\) , \(\:\widehat{{Y}_{i}^{t+2}},\dots\:,\widehat{{Y}_{i}^{t+m}}\}\) , with \(\:\widehat{{Y}_{i}^{t+k}}\) representing the forecasted values for country i at time step t+k and m being the length of the test period. Multivariate DL models are particularly well-suited for handling data in a multivariate way due to their flexibility in adjusting input and output dimensions, and they can effectively capture interactions between time series from different countries. Thus, in our study, all multivariate DL models were implemented using a multivariate way. The multivariate models using data in a multiple way, referred to as the “multiple way” in the following sections, uses data from both the target country and other countries as predictors within a single model (Fig. 1 C). This method is applicable to situations where models cannot simultaneously predict multiple targets. For example, tree-based models like LightGBM are optimized for predicting error of a single target variable and produce a single output per input sample, making them inherently designed for single-target scenarios. Therefore, we extend LightGBM to accommodate multiple predictors, adapting it to a multivariate model in a multiple way. (A) Univariate model using only data from the target country (B) Multivariate model using data from both target country and other countries in a multivariate way (C) Multivariate model using data from multiple countries in a multiple way. The target country data is represented by blocks with orange dashed lines. In the DL model, the green circles represent the hidden layers within the DL framework, while the red circles denote the fully connected layer. In the tree-based models, the green circles denote initial node or primary split points, and the red color indicates the leaf node. The superscript denotes the time step, and the subscript indicates the index of the country. We compared the performance of different DL models in both univariate and multivariate methods. The Transformer model used only the encoder part to capture temporal dependencies, with its output fed into a fully connected layer to predict future infection cases [ 4 ]. Recurrent Neural Network (RNN) handles sequential data by maintaining hidden states that capture temporal dependencies [ 11 , 12 ]. Long Short-Term Memory (LSTM) networks build on RNNs by mitigating the vanishing gradient problem through their memory cell structures [ 11 , 13 ]. Bidirectional LSTM (BiLSTM) models enhance temporal capture by processing input sequence in both forward and backward directions, allowing for more comprehensive temporal dependencies [ 14 ]. Gated Recurrent Unit (GRU) offers an alternative to LSTMs with a similar ability to capture dependencies, often providing comparable performance with faster training times [ 15 ]. Convolutional LSTM (CNN-LSTM) combines convolutional layers for feature extraction with LSTM layers for sequential modeling, enabling the model to capture both spatial and temporal dependencies [ 16 , 17 ]. Temporal Convolutional Network (TCN) use causal convolutions and dilation within convolutional layers to capture long-range dependencies without relying on recurrent structure [ 18 ]. Additionally, we implemented a sliding window approach, with an input window of 14 days and an output window of one day. This setup uses data from the past 28 days to forecast for the next single day. To ensure a fair comparison, each model was trained for 300 epochs under the same computational constraints. Hyperparameter tuning was performed using the Ray library in Python, which provides a robust and efficient framework for distributed hyperparameter search [ 19 ]. The Adam optimizer was used across all DL models. Moreover, to validate the effectiveness of multivariate models using data in a multiple way, we extended our multivariate idea to LightGBM, a gradient boosting framework that builds an ensemble of decision trees [ 20 ]. The parameter ranges for the Transformer model are detailed in Supplementary Table 1, for RNNs, LSTMs, BiLSTM, and GRU in Supplementary Table 2, for CNN-LSTM and TCN in Supplementary Tables 3 and 4, and for LightGBM in Supplementary Table 5, respectively. 2.3. Model evaluation and statistical testing Model performance was evaluated using the Root Mean Squared Error (RMSE) on the test period as the primary metric. RMSE was selected for its effectiveness in quantifying the average magnitude of prediction errors and providing a clear measure of. To further examine the differences in forecast accuracy between the models, we employed the two-sided Diebold-Mariano test [ 21 ]. The DM test evaluates whether the forecast errors from two competing models are statistically significantly different. Specifically, we calculated the loss differential \(\:{d}_{t}=L\left({e}_{1,t}\right)-L({e}_{2,t}\) ), where \(\:{e}_{1,t}\) and \(\:{e}_{2,t}\) represent the forecast errors of the two models at time \(\:t\) , and \(\:L\) is the loss function. The test statistic is given by: $$\:\:DM=\stackrel{-}{d}/\sqrt{\frac{1}{T{\sum\:}_{t=1}^{T}{d}_{t}^{2}}}$$ where \(\:\stackrel{-}{d}\) is the sample mean of the loss differentials \(\:{d}_{t}\) , and \(\:T\) is the number of forecast errors. The p-values from the DM tests, conducted across five distinct test periods, were aggregated using Kost's method. This advanced statistical technique integrates results across multiple test periods to provide a comprehensive analysis [ 22 ]. The combined p-value using Kost’s method is calculated as follows: $$\:{p}_{combined}=\:{\Phi\:}\left(\frac{1}{\sqrt{n}}\sum\:_{i=1}^{n}{{\Phi\:}}^{-1}\left({p}_{i}\right)\right)$$ where \(\:{p}_{combined}\) is the aggregated p-value, \(\:n\) is the number of p-values being combined, \(\:{p}_{i}\) represents each individual p-value, and \(\:{\Phi\:}\) and \(\:{{\Phi\:}}^{-1}\:\) are the cumulative distribution function and quantile function of the standard normal distribution, respectively. Kost’s method enhances the reliability and robustness of our findings. By aggregating p-values, this method accounts for potential variations in model performance across different time frames, ensuring that the overall assessment accurately reflects the models' forecasting abilities of the models throughout entire evaluation period. 3. Results To identify the most informative countries for each target country—South Korea, Japan, Russia, Italy, and the USA, we rank countries based on the DTW distance. Figure 2 presents the ranking results based on DTW distance for each target country, and Supplementary Table 6 lists the 10 most informative countries for each target country. Based on the rankings, we developed multivariate models, including Transformer, RNNs, LSTMs, BiLSTM, GRU, CNN-LSTM, and TCN, using data in a multivariate way, as well as lightGBM, using data in a multiple way, to forecast daily COVID-19 cases across five different test periods for each target country. Additionally, univariate versions of the above DL model and traditional ML model were developed, using only data from the target country. The mean RMSE result for South Korea, comparing both the univariate model and multivariate models, are displayed in Fig. 3 . For all DL models except the Transformer, adding more countries had minimal improvements in RMSE. The multivariate Transformer model showed significant improvement using data from 10 additional countries, balancing computational efficiency and accuracy. However, this approach negatively affected the TCN model, causing extreme RMSE values. LightGBM benefited from additional country data but still underperformed compared to the univariate DL model. This highlights the superiority of DL in handling large-scale datasets and capturing complex temporal patterns, and as a result, LightGBM was excluded from further analysis. Except for Italy, multivariate models demonstrated varying degrees of performance improvement in the other four target countries (Supplementary Figs. 1–4). A smaller distance indicates that the time series pattern of a specific country is more similar to that of the target country than others. A distance of 0 means that two time series are exactly the same in terms of time warping. Compared to others, the USA displays a larger distribution of distances, indicating that the time series pattern of the USA is more dissimilar from those of other countries. We further analyzed the improvement in multivariate DL models. Figure 4 illustrates the percentage difference in mean RMSE between the univariate model and the optimal multivariate model, identified by the minimum RMSE, for each DL model across five target countries and five time periods. The percentage difference is calculated as the RMSE of the multivariate model minus the RMSE of the univariate model, divided by the RMSE of the univariate model. Additionally, to provide a more comprehensive comparison, we calculated the average percentage difference in mean RMSE for each model across all target countries. In the results of the average case, all multivariate models outperformed their univariate counterparts. The Transformer model showed the most significant improvement, with an average reduction of 60.15% in RMSE. Other DL models also exhibited varying degrees of enhancement: TCN reduced the mean RMSE by 36.28%, CNN-LSTM by 29.47%, BiLSTM by 21.07%, GRU by 21.43%, RNN by 17.46%, and LSTM by 16.38%. Notably, in four out of the five target countries, excluding Italy, the multivariate models consistently outperformed the univariate models across all DL models. This robust performance across diverse countries with varying epidemic trends highlights the advantage of multivariate models in handling complex, multi-source data. Additionally, even in Italy, the Transformer model achieved a 56.71% reduction in mean RMSE. This finding suggests that the Transformer model significantly benefits from integrating large datasets, likely due to its complex architecture, which requires more data to effectively train its parameters. To assess whether the improvements in forecasting accuracy between univariate and multivariate models were statistically significant, we conducted the Diebold-Mariano test using Korea as an example. The null hypothesis for this test was there was no significant difference between the forecasts produced by univariate and multivariate models. Using Kost’s method, we combined the five p-values from the five different test periods, where the multivariate models integrated data from 1, 3, 5, 10, 15, 20, 30, 45, 60, and 90 other countries. This approach allowed us to examine how varying the amount of additional data affected the performance of the models across a range of complexities. Figure 5 reveals that multivariate Transformer models consistently achieved significant improvements over univariate models in terms of mean RMSE, except for one case. However, other DL models showed smaller mean RMSE differences, with p-values greater than 0.05, indicating non-significant difference. Table 1 summarizes how often the multivariate model outperformed the univariate model in terms of mean RMSE, and whether these improvements were statistically significant. The Transformer model had the highest proportion of cases with significant improvements, followed by the BiLSTM, though BiLSTM’s RMSE reduction was less pronounced compared to the Transformer model. Table 1 Count and proportion of cases with negative mean RMSE difference and combined p-value lower than 0.05. Total cases for each cell are 9 (including 1, 3, 5, 10, 15, 20, 30, 45, 60, and 90 additional countries). Transformer RNN LSTM BiLSTM GRU CNN-LSTM TCN Korea 8 2 2 5 1 3 4 Japan 7 4 8 6 9 1 7 Russia 9 0 1 7 3 3 4 Italy 7 0 0 3 0 0 3 USA 2 0 3 9 4 5 3 Sum 33 6 13 30 17 12 21 Proportion 0.733 0.133 0.289 0.667 0.378 0.267 0.467 4. Conclusion In this study, we examined whether integrating data from other countries could improve the forecasting performance of DL models for COVID-19 predictions in a target country. We proposed two multivariate methods for integrating additional data: using data in a multivariate way and using data in a multiple way. Our results revealed that, for four out of five target countries, compared to univariate model, multivariate DL models consistently demonstrated performance improvements. This suggests that leveraging additional data from other countries can provide more comprehensive information, significantly enhancing forecasting accuracy. However, in Italy case, some univariate DL models performed better. According to the DTW results, this may be due to the minimal pattern differences between the Italian data and those of other countries. In such cases, a complex multivariate model might add unnecessary complexity, and a univariate model could more effectively capture these stable patterns. Nevertheless, for the other four target countries, all multivariate DL models showed significant improvements when integrating additional countries’ data within a multivariate framework, compared to univariate models using only the target country’s data. This indicates that relying only on target country data may not provide sufficient information, while multivariate models can effectively address this limitation, making them a powerful tool for predicting COVID-19 trends. Among the seven DL models, the Transformer exhibited the most significant performance improvements, likely due to its complex architecture, which includes a large parameter space and self-attention mechanisms that efficiently leverage increased data volume. These improvements were further validated by the DM test to providing more reliable results. In contract, other DL models—such as RNN, LSTM, BiLSTM, GRU, CNN-LSTM, and TCN— showed more modest improvements and often experienced instability with larger datasets. For instance, the TCN model demonstrated irregular fluctuations in RMSE when integrating data from too many countries, indicating challenges in generalizing with larger, more complex datasets. These observations suggest that such models may be better suited for focused analyses with fewer data sources, particularly under conditions of limited computational resources. We also explored multivariate traditional ML models, specifically LightGBM, to investigate whether using data in a multiple way could enhance performance. Our experimental results indicated that multivariate LightGBM demonstrated a notable performance improvement in predicting COVID-19 in South Korea compared to univariate model. However, after reaching a performance peak, further increasing the number of countries did not yield additional improvements. This may be due to the model’s limitations in handling excessively large datasets or the substantial differences between the infection trends in the additional countries and Korea, which may not have contributed useful information. Despite these promising results, our study faced several limitations that warrant further investigation. Firstly, due to computational constraints, the analysis was limited to a discrete set of country combinations (1, 3, 5, 10, 15, 30, 45, 60, 90). As a result, the truly optimal number of countries might not have been identified. In our case, the bisection method wasn't used because the minimum was found between 5 and 10 countries, but it would be applied if the minimum occurred over a broader interval. Intermediate values may offer improved performance, and future research should explore a broader range of combinations to refine these findings. Additionally, while the Transformer's attention mechanism seems particularly well-suited to handling multivariate data, the underlying reasons for its superior performance remain challenging to fully understand [ 23 – 27 ]. The complex nature of the attention mechanism makes it difficult to mathematically validate why the Transformer outperforms other DL models in this context. Future research should delve deeper into this area, possibly by developing new interpretability methods tailored to the Transformer's architecture. Currently, our study utilized basic multivariate Transformer models for COVID-19 forecasting. However, there is potential for further improvement by experimenting with more advanced architectures. Models such as BERT (Bidirectional Encoder Representations from Transformers) or TFT (Temporal Fusion Transformers) could offer enhanced predictive accuracy due to their sophisticated mechanisms for handling large-scale datasets and complex temporal relationships [ 5 , 28 , 29 ]. In addition to national COVID-19 case data, incorporating additional features such as mobility data, vaccination rates, and socio-economic factors can enhance the model's ability to capture the dynamics of disease transmission. These variables can be introduced as time-varying covariates, serving as additional regressors in forecasting models. This approach is exemplified in TFT, which can effectively integrate multiple covariates into time series predictions. In practice, mobility data may serve as a leading indicator for changes in transmission, with spikes in mobility potentially preceding increases in case numbers [ 30 , 31 ]. Similarly, vaccination rates can be modeled as factors that mitigate disease spread, while socio-economic factors such as population density and income levels might modulate both susceptibility to infection and response to public health interventions [ 32 , 33 ]. The integration of these features requires careful preprocessing, including normalization and temporal alignment with the case data, to ensure their effective inclusion in the model. Moreover, the methodology developed in this study for COVID-19 forecasting could be adapted to other diseases or domains where data limitations hinder predictive performance. Our approach could be extended to financial and environmental sectors, with careful hyperparameter tuning to accommodate the unique characteristics of these domains. [ 34 , 35 ]. In conclusion, our study introduces multivariate models that integrate time series data from multiple sources. This approach effectively leverages COVID-19 data from other countries to capture trends similar to those in the target country, thereby improving forecasting performance. This demonstrates the potential of multivariate models to enhance forecasting in scenarios where data may be insufficient for DL models, making them a promising tool for time series forecasting in such contexts. Declarations Ethics approval and consent to participate: Not applicable Competing interests : The authors declare no competing interests. Clinical trial number Not applicable Funding: The authors declare that there is no funding source for this study. Author Contribution T.P. led the conceptualization and supervision of the study; J.O., Z.L., K.H., T.G., H.S., and J.P. contributed to methodology; J.O. and Z.L. performed the formal analysis and data curation; investigation was carried out by J.O., Z.L., K.H., T.G., H.S., and J.P.; J.O. and Z.L. prepared the original draft, while T.P., J.O., and Z.L. reviewed and edited it; visualization and software were handled by J.O. Project administration was overseen by T.P. All authors have reviewed and approved the final published version. Acknowledgments: The authors express their sincere gratitude to all participants involved in this study. Data availability: Publicly available data were used in this study and can be found in the references provided in the text. References Han K, Lee B, Lee D, Heo G, Oh J, Lee S, Park T. Forecasting the spread of COVID-19 based on policy, vaccination, and Omicron data. Sci Rep. 2024;14(1):9962. Lee B, Song H, Apio C, Han K, Park J, Liu Z, Park T. (2023). An analysis of the waning effect of COVID-19 vaccinations. Genomics Inf, 21 (4). Hyndman RJ, Athanasopoulos G. Forecasting: principles and practice. OTexts; 2018. Vaswani A et al. (2017). Attention is all you need. Adv Neural Inf Process Syst, 30. Lim W et al. (2021). Temporal fusion transformers for interpretable multi-horizon time series forecasting. arXiv preprint arXiv:1912.09363. Chimmula VKR, Zhang L. Time series forecasting of COVID-19 transmission in Canada using LSTM networks. Chaos Solitons Fractals. 2020;135:109864. Wang L, et al. COVID-Net: A tailored deep convolutional neural network design for detection of COVID-19 cases from chest X-ray images. Sci Rep. 2020;10(1):19549. Berndt DJ, Clifford J. (1994). Using dynamic time warping to find patterns in time series. In Proceedings of the 3rd international conference on knowledge discovery and data mining ,359–370. Mathieu E, Ritchie H, Ortiz-Ospina E, et al. A global database of COVID-19 vaccinations. Nat Hum Behav. 2021;5:947–53. Chatfield C, Xing H. (2019). The analysis of time series: an introduction with R . Chapman and hall/CRC. Sherstinsky A. (2020). Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Physica D: Nonlinear Phenomena , 404 , 132306. Faria, D. R.,. (2021). Forecasting the number of COVID-19 cases in the world. Chaos, Solitons & Fractals , 142, 110497. Cho K et al. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735–80. Graves A, Schmidhuber J. (2005). Framewise phoneme classification with bidirectional LSTM networks. Proceedings of the IEEE International Joint Conference on Neural Networks, 2047–2052. Chung J, Gulcehre C, Cho K, Bengio Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:14123555. Shi X et al. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Adv Neural Inf Process Syst, 28. Bai S, Kolter JZ, Koltun V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:180301271. Lea C, Flynn MD, Vidal R, Reiter A, Hager GD. (2017). Temporal convolutional networks for action segmentation and detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 156–165. Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., … Stoica, I.(2018). Ray: A distributed framework for emerging {AI} applications. In 13th USENIX symposium on operating systems design and implementation (OSDI 18) (pp. 561–577). Ke G, Meng Q, Finley T, Wang et al. (2017). Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems , 30 . Diebold FX, Mariano RS. Comparing Predictive Accuracy. J Bus Economic Stat. 1995;13(3):253–63. Kost JT, McDermott MP. Combining dependent P-values. Stat Probab Lett. 2002;60(2):183–90. Zhang C, Bengio S, Hardt M, Recht B, Vinyals O. (2017). Understanding deep learning requires rethinking generalization. International Conference on Learning Representations (ICLR) . Belkin M, Hsu D, Ma S, Mandal S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences , 116(32), 15849–15854. Neyshabur B, Tomioka R, Srebro N. (2015). In search of the real inductive bias: On the role of implicit regularization in deep learning. International Conference on Learning Representations (ICLR) . Arora S, Cohen N, Hu W, Luo Y. (2019). Implicit regularization in deep matrix factorization. Adv Neural Inf Process Syst, 32. Bartlett PL, Montanari A, Rakhlin A. Deep learning: a statistical viewpoint. Acta Numerica. 2021;30:87–201. Devlin J, Chang MW, Lee K, Toutanova K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 . Brown TB, Mann B, Ryder N et al. (2020). Language Models are Few-Shot Learners. Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020) . Kraemer, M. U. G., Yang, C.-H., Gutierrez, B., Wu, C.-H., Klein, B., Pigott, D. M.,… Scarpino, S. V. (2020). The effect of human mobility and control measures on the COVID-19 epidemic in China. Science , 368(6490), 493–497. Jia JS, Lu X, Yuan Y, Xu G, Jia J, Christakis NA. Population flow drives spatio-temporal distribution of COVID-19 in China. Nature. 2020;582(7812):389–94. Cheng, C., Jiang, W. M., Fan, B., Cheng, Y. C., Hsu, Y. T., Wu, H. Y., … Tsou, H.H. (2023). Real-time forecasting of COVID-19 spread according to protective behavior and vaccination: autoregressive integrated moving average models. BMC public health , 23 (1), 1500. Borchering RK. (2021). Modeling of future COVID-19 cases, hospitalizations, and deaths, by vaccination rates and nonpharmaceutical intervention scenarios—United States, April–September 2021. MMWR. Morbidity and Mortality Weekly Report , 70 . Sezer OB, Gudelek MU, Ozbayoglu AM. Financial time series forecasting with deep learning: A systematic literature review: 2005–2019. Appl Soft Comput. 2020;90:106181. Lin WH, Wang P, Chao KM, Lin HC, Yang ZY, Lai YH. Wind power forecasting with deep learning networks: Time-series forecasting. Appl Sci. 2021;11(21):10335. Additional Declarations No competing interests reported. Supplementary Files SupplementaryMaterial.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5400759","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":378079732,"identity":"5ccf7e37-2794-4463-acdf-1cd103168db9","order_by":0,"name":"Jooha Oh","email":"","orcid":"","institution":"Seoul National University","correspondingAuthor":false,"prefix":"","firstName":"Jooha","middleName":"","lastName":"Oh","suffix":""},{"id":378079733,"identity":"0c54d761-343d-402d-bad3-5fa1c82c6f84","order_by":1,"name":"Zhe Liu","email":"","orcid":"","institution":"Seoul National University","correspondingAuthor":false,"prefix":"","firstName":"Zhe","middleName":"","lastName":"Liu","suffix":""},{"id":378079734,"identity":"af506cbd-a708-4ab7-9d04-dfab0216023c","order_by":2,"name":"Kyulhee Han","email":"","orcid":"","institution":"Seoul National University","correspondingAuthor":false,"prefix":"","firstName":"Kyulhee","middleName":"","lastName":"Han","suffix":""},{"id":378079735,"identity":"470799a6-6e7d-4911-a1c3-6f3081ab8fb2","order_by":3,"name":"Taewan Goo","email":"","orcid":"","institution":"Seoul National University","correspondingAuthor":false,"prefix":"","firstName":"Taewan","middleName":"","lastName":"Goo","suffix":""},{"id":378079736,"identity":"41d07061-b176-410a-9f47-1bd005779d96","order_by":4,"name":"Hanbyul Song","email":"","orcid":"","institution":"Seoul National University","correspondingAuthor":false,"prefix":"","firstName":"Hanbyul","middleName":"","lastName":"Song","suffix":""},{"id":378079737,"identity":"1b68c05e-4a52-4cdd-9cf8-f20e5f771aa6","order_by":5,"name":"Jiwon Park","email":"","orcid":"","institution":"Seoul National University","correspondingAuthor":false,"prefix":"","firstName":"Jiwon","middleName":"","lastName":"Park","suffix":""},{"id":378079738,"identity":"6cada001-01c6-4a5f-aa56-c7c57a26c219","order_by":6,"name":"Taesung Park","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6klEQVRIie3RMQrCMBSA4ReEdglkTQfvUBAEQcxVDIFO4uz4pk6FrnoLD+DQEnCqulZ0UAqdHAJOgoMBcRAk4uaQHwIJ5CMPAuDz/W3F8P1M0HGZPkliN53fiP6BCLYpr7NqJ0S40wbuR4mhPpHFyvEKn6qoqg8yoyrgJG0l0iQmy9ZFaByhOYwpqMAOpCXCBMipcBBW9W5otoKypmMHs4RdvhCY9COsC5JxBRwCS7h9Zeki9TQZYKVkVjd9LlPdS3kbl3MHCfON3uN6JMJcNsbcdTdn6nzOHOT1M8/GdgV2OcE78fl8Pt+nHmjyU5oGgboqAAAAAElFTkSuQmCC","orcid":"","institution":"Seoul National University","correspondingAuthor":true,"prefix":"","firstName":"Taesung","middleName":"","lastName":"Park","suffix":""}],"badges":[],"createdAt":"2024-11-06 08:08:52","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5400759/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5400759/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":70098037,"identity":"82c9cc97-bdf2-4329-8ed2-77a41d713694","added_by":"auto","created_at":"2024-11-28 10:04:43","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":590321,"visible":true,"origin":"","legend":"\u003cp\u003eComparison of univariate model and multivariate model\u003c/p\u003e\n\u003cp\u003e(A) Univariate model using only data from the target country (B) Multivariate model using data from both target country and other countries in a multivariate way (C) Multivariate model using data from multiple countries in a multiple way. The target country data is represented by blocks with orange dashed lines. In the DL model, the green circles represent the hidden layers within the DL framework, while the red circles denote the fully connected layer. In the tree-based models, the green circles denote initial node or primary split points, and the red color indicates the leaf node. The superscript denotes the time step, and the subscript indicates the index of the country.\u003c/p\u003e","description":"","filename":"Figure1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-5400759/v1/6757b28fa8c3d5d60ff57990.jpeg"},{"id":70097731,"identity":"3b0ab59b-19ca-49a2-bf80-63f0e072bd82","added_by":"auto","created_at":"2024-11-28 09:56:43","extension":"jpeg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":248534,"visible":true,"origin":"","legend":"\u003cp\u003eDynamic Time Warping distance of each target country.\u003c/p\u003e","description":"","filename":"Figure2.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-5400759/v1/c5f22151ddf6c3174db61e73.jpeg"},{"id":70098036,"identity":"39bc52da-be21-4113-a1b2-33a2040959d4","added_by":"auto","created_at":"2024-11-28 10:04:43","extension":"jpeg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":194064,"visible":true,"origin":"","legend":"\u003cp\u003eMean RMSE over five periods in Korea for eight different models.\u003c/p\u003e\n\u003cp\u003eThe blue triangle indicates the minimum mean RMSE, achieved with the Transformer model using data from 10 countries. Mean RMSE values over 120000 were illustrated as cap value.\u003c/p\u003e","description":"","filename":"Figure3.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-5400759/v1/59760f19591c9ffab9e65b65.jpeg"},{"id":70097732,"identity":"be096d60-3e5b-4315-9d90-be9bfb01835f","added_by":"auto","created_at":"2024-11-28 09:56:43","extension":"jpeg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":235243,"visible":true,"origin":"","legend":"\u003cp\u003ePercentage difference of mean RMSE between univariate and optimal multivariate model for each deep learning model. Blue bars indicate cases where the multivariate model outperforms the univariate model, while orange bars represent cases where the univariate model performs better. The “Average” represents the average percentage difference across the five target countries.\u003c/p\u003e","description":"","filename":"Figure4.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-5400759/v1/ce6a51ab0f374e4b5edcedda.jpeg"},{"id":70097734,"identity":"9f2e04c3-8688-43f1-b553-b11d790d2d8b","added_by":"auto","created_at":"2024-11-28 09:56:43","extension":"jpeg","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":97157,"visible":true,"origin":"","legend":"\u003cp\u003eMean RMSE difference between univariate and multivariate models vs. negative log transformation of combined p-value from the DM test in South Korea. Two horizontal red dashed lines indicate significance level of 0.05 and 0.1.\u003c/p\u003e","description":"","filename":"Figure5.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-5400759/v1/8e8a264f5dde6be37d7fd04c.jpeg"},{"id":79270972,"identity":"dd3019a2-e0ce-4862-8d8c-ab5e6e82de5e","added_by":"auto","created_at":"2025-03-26 11:16:45","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2006689,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5400759/v1/d2d53c51-07aa-4e7e-b3ac-5a83930290c9.pdf"},{"id":70097736,"identity":"81743ff7-a12a-42b0-889c-8365abf0f1b9","added_by":"auto","created_at":"2024-11-28 09:56:43","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":423753,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryMaterial.docx","url":"https://assets-eu.researchsquare.com/files/rs-5400759/v1/dfa87042ca691d78f46ed1bd.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Enhancing COVID-19 Forecasts Through Multivariate Deep Learning Models","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eThe COVID-19 pandemic, caused by the SARS-CoV-2 virus and emerging in late 2019, has presented unprecedented challenges to public health systems worldwide. This situation highlights the urgent need for accurate models to predict infection cases, which is crucial for timely public health interventions and effective resource allocation. While various predictive models have been developed, including traditional statistical models such as AutoRegressive Integrated Moving Average (ARIMA) and time series generalized linear models (tsglm), as well as machine learning (ML) models like LightGBM [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. However, these models often show lower performance than deep learning (DL) models. DL models, known for their ability to capture complex temporal patterns, have the potential to outperform traditional models in COVID-19 forecasting tasks. However, DL models typically require large amounts of data to achieve high prediction accuracy, and if data is scarce, it significantly limits their performance. Compared to classical DL models like Long Short-Term Memory (LSTM) networks, the state-of-the-art Transformer model, introduced by Vaswani et al. (2017), requires significantly larger datasets due to its more complex architecture and higher number of parameters than other DL models [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. Despite these challenges, Transformers have advanced the field significantly by using self-attention mechanisms, which model dependencies without the sequential limitations of Recurrent Neural Networks (RNNs). This allows for more efficient training and better performance on large datasets [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. To integrate more information and further enhance the model performance, recent studies have employed the Temporal Fusion Transformer (TFT), improving forecasting effectiveness by integrating both temporal and static covariates [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. Chimmula and Zhang (2020) applied LSTM networks to predict the spread of COVID-19 in Canada, demonstrating the ability of DL models to handle sequential data and learn from multiple sources simultaneously [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. To mitigate data limitations, some studies have also used data augmentation techniques and transfer learning, enabling models to leverage additional information and improve forecasting accuracy [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. However, most research predicts COVID-19 spread using only historical data from the target country, which may limit the potential of DL models, particularly Transformers. This issue becomes more serious during the early stages of the pandemic or when countries collect data on a weekly basis, as limited datasets restrict the available information for DL model training, leading to suboptimal predictions.\u003c/p\u003e \u003cp\u003eTo enhance performance of DL models, we proposed a multivariate method that integrates data from other countries to improve predictions by leveraging similar patterns, rather than relying on the univariate model, which uses only data from the target country. Multivariate method can be achieved through two main approaches based on the nature of the models: use data in a multivariate way and in a multiple way. Multivariate models in a multivariate way involves forecasting not just a specific country but multiple countries simultaneously as a vector by integrating multiple time series. This approach is particularly well-suited for DL models, as they can expand the input matrix to capture intricate relationships between variables across different time series. By leveraging their capacity to learn from large, complex datasets and capture long-term dependencies, DL models effectively utilize the additional information from multiple countries. Alternatively, using data in a multiple way involves including data from other countries as additional predictive variables to improve forecasts for a specific country. This approach is particularly meaningful for some traditional statistical models and ML models that cannot use data in a multivariate way. These models can be extended to handle multiple regressors, allowing them to predict a single target variable more effectively. To effectively integrate data via multivariate methods, identifying the most informative countries is crucial, as irrelevant data may introduce noise and reduce performance. To determine these key countries, we use Dynamic Time Warping (DTW) distance to rank and select the most relevant ones, thereby enhancing forecasting performance [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eWe compared the performance of the various univariate models with multivariate models. We first compared univariate models with multivariate models using data in a multivariate way, including deep learning models such as Transformer, RNN, LSTM, BiLSTM, GRU, CNN-LSTM, and TCN. We further compared univariate models with a multivariate model using data in a multiple way, including LightGBM. Our results demonstrate that multivariate models can improve forecasting performance by integrating information from other sources. Furthermore, comparing benchmark multivariate DL models with multivariate LightGBM, which has shown strong performance in COVID-19 forecasting, highlights the advantages of DL in handling large datasets and capturing complex dependencies.\u003c/p\u003e \u003cp\u003eIn summary, our study demonstrates that integrating data from multiple countries can significantly enhance the performance of DL models in forecasting COVID-19. By leveraging similar infection patterns across countries, multivariate Transformer models, in particular, show strong potential for improving prediction accuracy. This approach offers a valuable strategy for enhancing COVID-19 forecasting by utilizing data from diverse countries, allowing DL models to better capture complex dependencies and trends.\u003c/p\u003e"},{"header":"2. Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1. Data acquisition and preprocessing\u003c/h2\u003e \u003cp\u003eWe obtained daily COVID-19 case data from five countries\u0026mdash;South Korea, Japan, Russia, Italy, and the United States \u0026mdash;from the Our World in Data (OWID) repository [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. These five countries, including those from Asia, Europe, and the Americas, reflect diverse epidemic response strategies and public health conditions, enhancing the generalizability and applicability of the findings. Additionally, the data quality and completeness from these countries are relatively high, ensuring the credibility of the research results. The raw OWID dataset required preprocessing to handle missing values and correct entries that were recorded weekly instead of daily. Specifically, missing or weekly data points were replaced by a 7-day simple moving average (SMA) [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. Since the multivariate models in the multiple way cannot simultaneously predict values for additional countries in the test set, we filled in the data for other countries in the test set with the last observation from the training set. To ensure that the selected countries had populations comparable to those of the five target countries, we excluded countries with populations under 10\u0026nbsp;million from our analysis, resulting in a total of 90 countries. To determine which countries\u0026rsquo; data to integrate with the target country\u0026rsquo;s data, we ranked the 90 countries based on their similarity to the target country using DTW distance. DTW is a technique used to find the optimal alignment between two time-dependent sequences. The general equation for DTW distance between two sequences \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{X}_{1}={\\left\\{{X}_{11},\\:{X}_{12,\\:}\\dots\\:,{X}_{1n}\\right\\}}^{T}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{X}_{2}={\\left\\{{X}_{21},\\:{X}_{22,\\:}\\dots\\:,{X}_{2n}\\right\\}}^{T}\\)\u003c/span\u003e\u003c/span\u003eis as following where \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:P\\)\u003c/span\u003e\u003c/span\u003e is a warping path, which is a sequence of index pairs that define the alignment between \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{X}_{1}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{X}_{2}\\)\u003c/span\u003e\u003c/span\u003e, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:d\\left({X}_{1i},\\:{X}_{2j}\\right)\\)\u003c/span\u003e\u003c/span\u003e denotes the Euclidean distance between the points \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{X}_{1i}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{X}_{2j}\\)\u003c/span\u003e\u003c/span\u003e.\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e\n$$\\:{\\text{D}\\text{T}\\text{W}(X}_{1},\\:{X}_{2})=\\text{m}\\text{i}\\text{n}\\left(\\sum\\:_{(i,\\:j)\\in\\:P}d({X}_{1i},\\:{X}_{2j})\\right)$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eIn our study, it measures the similarity between the time series of daily COVID-19 cases in the target country and those in other countries. DTW is particularly effective for this purpose because it accounts for temporal misalignments and varying speeds in time series data, providing a robust measure of similarity between pairs of time series. Based on the ranking, we treated the number of additional countries in the multivariate model as a parameter, exploring values from 1 to 90, specifically, we explored configurations with 1, 3, 5, 10, 15, 20, 30, 45, 60, and 90 countries, where 90 represents the inclusion of all countries in our dataset. By varying the number of included countries, we aim to explore how the inclusion of additional countries affects the model's performance in capturing the complexities of the pandemic spread. Moreover, we used five distinct intervals to ensure that our approach is robust across varying training and testing periods. The training periods were January 1, 2022, to July 31, 2022; January 1, 2022, to August 31, 2022; January 1, 2022, to September 30, 2022; January 1, 2022, to October 31, 2022; and January 1, 2022, to November 30, 2022. For each training period, the corresponding testing period extended up to 28 days beyond the end of the training period.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2. Comparative models and hyperparameters tuning\u003c/h2\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e illustrates the differences between the univariate model and the multivariate model. The univariate models are country specific models. In this approach, the time series data from the specific country, denoted as \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{X}_{i}\\)\u003c/span\u003e\u003c/span\u003e=\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\left\\{{x}_{i}^{1},{x}_{i}^{2},\\dots\\:,{x}_{i}^{t}\\right\\}\\)\u003c/span\u003e\u003c/span\u003e, where \u003cem\u003et\u003c/em\u003e represents the length of the training period, is used to forecast for that specific country. The predicted values are represented as \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\widehat{{Y}_{i}}=\\{\\widehat{{Y}_{i}^{t+1}},\\)\u003c/span\u003e\u003c/span\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\widehat{{Y}_{i}^{t+2}},\\dots\\:,\\widehat{{Y}_{i}^{t+m}}\\}\\)\u003c/span\u003e\u003c/span\u003e, where \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:m\\)\u003c/span\u003e\u003c/span\u003e represents the length of the test period (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eA). This model focuses only on the historical data of the target but does not leverage potentially valuable information from other sources that could enhance the accuracy and robustness of the forecasts. In contrast, multivariate models integrate time series data from multiple sources. As shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e(B), in the multivariate model using data in a multivariate way, data from both the target country and additional sources are integrated into a matrix format This matrix is then fed into the models to predict both the target country\u0026rsquo;s values and those of the additional sources simultaneously. Specifically, if \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:X={\\left\\{{X}_{1},\\:{X}_{2,\\:}\\dots\\:,{X}_{n}\\right\\}}^{T}\\)\u003c/span\u003e\u003c/span\u003e represents the matrix of time series data from n countries, where \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{X}_{i,\\:}\\)\u003c/span\u003e\u003c/span\u003e corresponds to the time series data from each different source, the forecasted values are denoted by: \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\widehat{Y}=\\{\\widehat{{Y}_{1}},\\widehat{{\\:Y}_{2}}\\)\u003c/span\u003e\u003c/span\u003e, \u003cem\u003e\u0026hellip;\u003c/em\u003e ,\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\widehat{{Y}_{n}}\\}}^{T}\\)\u003c/span\u003e\u003c/span\u003e ,where \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\widehat{{Y}_{i}}=\\{\\widehat{{Y}_{i}^{t+1}}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\widehat{{Y}_{i}^{t+2}},\\dots\\:,\\widehat{{Y}_{i}^{t+m}}\\}\\)\u003c/span\u003e\u003c/span\u003e, with \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\widehat{{Y}_{i}^{t+k}}\\)\u003c/span\u003e\u003c/span\u003e representing the forecasted values for country \u003cem\u003ei\u003c/em\u003e at time step t+k and m being the length of the test period. Multivariate DL models are particularly well-suited for handling data in a multivariate way due to their flexibility in adjusting input and output dimensions, and they can effectively capture interactions between time series from different countries. Thus, in our study, all multivariate DL models were implemented using a multivariate way. The multivariate models using data in a multiple way, referred to as the \u0026ldquo;multiple way\u0026rdquo; in the following sections, uses data from both the target country and other countries as predictors within a single model (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eC). This method is applicable to situations where models cannot simultaneously predict multiple targets. For example, tree-based models like LightGBM are optimized for predicting error of a single target variable and produce a single output per input sample, making them inherently designed for single-target scenarios. Therefore, we extend LightGBM to accommodate multiple predictors, adapting it to a multivariate model in a multiple way.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e(A) Univariate model using only data from the target country (B) Multivariate model using data from both target country and other countries in a multivariate way (C) Multivariate model using data from multiple countries in a multiple way. The target country data is represented by blocks with orange dashed lines. In the DL model, the green circles represent the hidden layers within the DL framework, while the red circles denote the fully connected layer. In the tree-based models, the green circles denote initial node or primary split points, and the red color indicates the leaf node. The superscript denotes the time step, and the subscript indicates the index of the country.\u003c/p\u003e \u003cp\u003eWe compared the performance of different DL models in both univariate and multivariate methods. The Transformer model used only the encoder part to capture temporal dependencies, with its output fed into a fully connected layer to predict future infection cases [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. Recurrent Neural Network (RNN) handles sequential data by maintaining hidden states that capture temporal dependencies [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. Long Short-Term Memory (LSTM) networks build on RNNs by mitigating the vanishing gradient problem through their memory cell structures [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. Bidirectional LSTM (BiLSTM) models enhance temporal capture by processing input sequence in both forward and backward directions, allowing for more comprehensive temporal dependencies [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. Gated Recurrent Unit (GRU) offers an alternative to LSTMs with a similar ability to capture dependencies, often providing comparable performance with faster training times [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Convolutional LSTM (CNN-LSTM) combines convolutional layers for feature extraction with LSTM layers for sequential modeling, enabling the model to capture both spatial and temporal dependencies [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Temporal Convolutional Network (TCN) use causal convolutions and dilation within convolutional layers to capture long-range dependencies without relying on recurrent structure [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. Additionally, we implemented a sliding window approach, with an input window of 14 days and an output window of one day. This setup uses data from the past 28 days to forecast for the next single day. To ensure a fair comparison, each model was trained for 300 epochs under the same computational constraints. Hyperparameter tuning was performed using the Ray library in Python, which provides a robust and efficient framework for distributed hyperparameter search [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. The Adam optimizer was used across all DL models. Moreover, to validate the effectiveness of multivariate models using data in a multiple way, we extended our multivariate idea to LightGBM, a gradient boosting framework that builds an ensemble of decision trees [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. The parameter ranges for the Transformer model are detailed in Supplementary Table\u0026nbsp;1, for RNNs, LSTMs, BiLSTM, and GRU in Supplementary Table\u0026nbsp;2, for CNN-LSTM and TCN in Supplementary Tables\u0026nbsp;3 and 4, and for LightGBM in Supplementary Table\u0026nbsp;5, respectively.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3. Model evaluation and statistical testing\u003c/h2\u003e \u003cp\u003eModel performance was evaluated using the Root Mean Squared Error (RMSE) on the test period as the primary metric. RMSE was selected for its effectiveness in quantifying the average magnitude of prediction errors and providing a clear measure of. To further examine the differences in forecast accuracy between the models, we employed the two-sided Diebold-Mariano test [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. The DM test evaluates whether the forecast errors from two competing models are statistically significantly different. Specifically, we calculated the loss differential \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{d}_{t}=L\\left({e}_{1,t}\\right)-L({e}_{2,t}\\)\u003c/span\u003e\u003c/span\u003e), where \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{e}_{1,t}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{e}_{2,t}\\)\u003c/span\u003e\u003c/span\u003erepresent the forecast errors of the two models at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:t\\)\u003c/span\u003e\u003c/span\u003e, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:L\\)\u003c/span\u003e\u003c/span\u003e is the loss function. The test statistic is given by:\u003cdiv id=\"Equb\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equb\" name=\"EquationSource\"\u003e\n$$\\:\\:DM=\\stackrel{-}{d}/\\sqrt{\\frac{1}{T{\\sum\\:}_{t=1}^{T}{d}_{t}^{2}}}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\stackrel{-}{d}\\)\u003c/span\u003e\u003c/span\u003e is the sample mean of the loss differentials \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{d}_{t}\\)\u003c/span\u003e\u003c/span\u003e, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:T\\)\u003c/span\u003e\u003c/span\u003e is the number of forecast errors. The p-values from the DM tests, conducted across five distinct test periods, were aggregated using Kost's method. This advanced statistical technique integrates results across multiple test periods to provide a comprehensive analysis [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. The combined p-value using Kost\u0026rsquo;s method is calculated as follows:\u003cdiv id=\"Equc\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equc\" name=\"EquationSource\"\u003e\n$$\\:{p}_{combined}=\\:{\\Phi\\:}\\left(\\frac{1}{\\sqrt{n}}\\sum\\:_{i=1}^{n}{{\\Phi\\:}}^{-1}\\left({p}_{i}\\right)\\right)$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{p}_{combined}\\)\u003c/span\u003e\u003c/span\u003e is the aggregated p-value, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:n\\)\u003c/span\u003e\u003c/span\u003e is the number of p-values being combined, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{p}_{i}\\)\u003c/span\u003e\u003c/span\u003e represents each individual p-value, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{\\Phi\\:}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{{\\Phi\\:}}^{-1}\\:\\)\u003c/span\u003e\u003c/span\u003eare the cumulative distribution function and quantile function of the standard normal distribution, respectively. Kost\u0026rsquo;s method enhances the reliability and robustness of our findings. By aggregating p-values, this method accounts for potential variations in model performance across different time frames, ensuring that the overall assessment accurately reflects the models' forecasting abilities of the models throughout entire evaluation period.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. Results","content":"\u003cp\u003eTo identify the most informative countries for each target country\u0026mdash;South Korea, Japan, Russia, Italy, and the USA, we rank countries based on the DTW distance. Figure\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e presents the ranking results based on DTW distance for each target country, and Supplementary Table\u0026nbsp;6 lists the 10 most informative countries for each target country. Based on the rankings, we developed multivariate models, including Transformer, RNNs, LSTMs, BiLSTM, GRU, CNN-LSTM, and TCN, using data in a multivariate way, as well as lightGBM, using data in a multiple way, to forecast daily COVID-19 cases across five different test periods for each target country. Additionally, univariate versions of the above DL model and traditional ML model were developed, using only data from the target country. The mean RMSE result for South Korea, comparing both the univariate model and multivariate models, are displayed in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e. For all DL models except the Transformer, adding more countries had minimal improvements in RMSE. The multivariate Transformer model showed significant improvement using data from 10 additional countries, balancing computational efficiency and accuracy. However, this approach negatively affected the TCN model, causing extreme RMSE values. LightGBM benefited from additional country data but still underperformed compared to the univariate DL model. This highlights the superiority of DL in handling large-scale datasets and capturing complex temporal patterns, and as a result, LightGBM was excluded from further analysis. Except for Italy, multivariate models demonstrated varying degrees of performance improvement in the other four target countries (Supplementary Figs.\u0026nbsp;1\u0026ndash;4).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eA smaller distance indicates that the time series pattern of a specific country is more similar to that of the target country than others. A distance of 0 means that two time series are exactly the same in terms of time warping. Compared to others, the USA displays a larger distribution of distances, indicating that the time series pattern of the USA is more dissimilar from those of other countries.\u003c/p\u003e\u003cp\u003eWe further analyzed the improvement in multivariate DL models. Figure\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e illustrates the percentage difference in mean RMSE between the univariate model and the optimal multivariate model, identified by the minimum RMSE, for each DL model across five target countries and five time periods. The percentage difference is calculated as the RMSE of the multivariate model minus the RMSE of the univariate model, divided by the RMSE of the univariate model. Additionally, to provide a more comprehensive comparison, we calculated the average percentage difference in mean RMSE for each model across all target countries. In the results of the average case, all multivariate models outperformed their univariate counterparts. The Transformer model showed the most significant improvement, with an average reduction of 60.15% in RMSE. Other DL models also exhibited varying degrees of enhancement: TCN reduced the mean RMSE by 36.28%, CNN-LSTM by 29.47%, BiLSTM by 21.07%, GRU by 21.43%, RNN by 17.46%, and LSTM by 16.38%. Notably, in four out of the five target countries, excluding Italy, the multivariate models consistently outperformed the univariate models across all DL models. This robust performance across diverse countries with varying epidemic trends highlights the advantage of multivariate models in handling complex, multi-source data. Additionally, even in Italy, the Transformer model achieved a 56.71% reduction in mean RMSE. This finding suggests that the Transformer model significantly benefits from integrating large datasets, likely due to its complex architecture, which requires more data to effectively train its parameters.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eTo assess whether the improvements in forecasting accuracy between univariate and multivariate models were statistically significant, we conducted the Diebold-Mariano test using Korea as an example. The null hypothesis for this test was there was no significant difference between the forecasts produced by univariate and multivariate models. Using Kost\u0026rsquo;s method, we combined the five p-values from the five different test periods, where the multivariate models integrated data from 1, 3, 5, 10, 15, 20, 30, 45, 60, and 90 other countries. This approach allowed us to examine how varying the amount of additional data affected the performance of the models across a range of complexities. Figure\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e reveals that multivariate Transformer models consistently achieved significant improvements over univariate models in terms of mean RMSE, except for one case. However, other DL models showed smaller mean RMSE differences, with p-values greater than 0.05, indicating non-significant difference. Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e summarizes how often the multivariate model outperformed the univariate model in terms of mean RMSE, and whether these improvements were statistically significant. The Transformer model had the highest proportion of cases with significant improvements, followed by the BiLSTM, though BiLSTM\u0026rsquo;s RMSE reduction was less pronounced compared to the Transformer model.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cb\u003eCount and proportion of cases with negative mean RMSE difference and combined p-value lower than 0.05.\u003c/b\u003e Total cases for each cell are 9 (including 1, 3, 5, 10, 15, 20, 30, 45, 60, and 90 additional countries).\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"9\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTransformer\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRNN\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e \u003cp\u003eLSTM\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eBiLSTM\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eGRU\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eCNN-LSTM\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c9\"\u003e \u003cp\u003eTCN\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eKorea\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eJapan\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRussia\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eItaly\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eUSA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSum\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e33\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e30\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e21\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eProportion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e0.733\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003e0.133\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.289\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.667\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.378\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c8\"\u003e \u003cp\u003e0.267\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c9\"\u003e \u003cp\u003e0.467\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e"},{"header":"4. Conclusion","content":"\u003cp\u003eIn this study, we examined whether integrating data from other countries could improve the forecasting performance of DL models for COVID-19 predictions in a target country. We proposed two multivariate methods for integrating additional data: using data in a multivariate way and using data in a multiple way. Our results revealed that, for four out of five target countries, compared to univariate model, multivariate DL models consistently demonstrated performance improvements. This suggests that leveraging additional data from other countries can provide more comprehensive information, significantly enhancing forecasting accuracy. However, in Italy case, some univariate DL models performed better. According to the DTW results, this may be due to the minimal pattern differences between the Italian data and those of other countries. In such cases, a complex multivariate model might add unnecessary complexity, and a univariate model could more effectively capture these stable patterns. Nevertheless, for the other four target countries, all multivariate DL models showed significant improvements when integrating additional countries\u0026rsquo; data within a multivariate framework, compared to univariate models using only the target country\u0026rsquo;s data. This indicates that relying only on target country data may not provide sufficient information, while multivariate models can effectively address this limitation, making them a powerful tool for predicting COVID-19 trends.\u003c/p\u003e \u003cp\u003eAmong the seven DL models, the Transformer exhibited the most significant performance improvements, likely due to its complex architecture, which includes a large parameter space and self-attention mechanisms that efficiently leverage increased data volume. These improvements were further validated by the DM test to providing more reliable results. In contract, other DL models\u0026mdash;such as RNN, LSTM, BiLSTM, GRU, CNN-LSTM, and TCN\u0026mdash; showed more modest improvements and often experienced instability with larger datasets. For instance, the TCN model demonstrated irregular fluctuations in RMSE when integrating data from too many countries, indicating challenges in generalizing with larger, more complex datasets. These observations suggest that such models may be better suited for focused analyses with fewer data sources, particularly under conditions of limited computational resources. We also explored multivariate traditional ML models, specifically LightGBM, to investigate whether using data in a multiple way could enhance performance. Our experimental results indicated that multivariate LightGBM demonstrated a notable performance improvement in predicting COVID-19 in South Korea compared to univariate model. However, after reaching a performance peak, further increasing the number of countries did not yield additional improvements. This may be due to the model\u0026rsquo;s limitations in handling excessively large datasets or the substantial differences between the infection trends in the additional countries and Korea, which may not have contributed useful information.\u003c/p\u003e \u003cp\u003eDespite these promising results, our study faced several limitations that warrant further investigation. Firstly, due to computational constraints, the analysis was limited to a discrete set of country combinations (1, 3, 5, 10, 15, 30, 45, 60, 90). As a result, the truly optimal number of countries might not have been identified. In our case, the bisection method wasn't used because the minimum was found between 5 and 10 countries, but it would be applied if the minimum occurred over a broader interval. Intermediate values may offer improved performance, and future research should explore a broader range of combinations to refine these findings. Additionally, while the Transformer's attention mechanism seems particularly well-suited to handling multivariate data, the underlying reasons for its superior performance remain challenging to fully understand [\u003cspan additionalcitationids=\"CR24 CR25 CR26\" citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. The complex nature of the attention mechanism makes it difficult to mathematically validate why the Transformer outperforms other DL models in this context. Future research should delve deeper into this area, possibly by developing new interpretability methods tailored to the Transformer's architecture.\u003c/p\u003e \u003cp\u003eCurrently, our study utilized basic multivariate Transformer models for COVID-19 forecasting. However, there is potential for further improvement by experimenting with more advanced architectures. Models such as BERT (Bidirectional Encoder Representations from Transformers) or TFT (Temporal Fusion Transformers) could offer enhanced predictive accuracy due to their sophisticated mechanisms for handling large-scale datasets and complex temporal relationships [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e, \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e]. In addition to national COVID-19 case data, incorporating additional features such as mobility data, vaccination rates, and socio-economic factors can enhance the model's ability to capture the dynamics of disease transmission. These variables can be introduced as time-varying covariates, serving as additional regressors in forecasting models. This approach is exemplified in TFT, which can effectively integrate multiple covariates into time series predictions. In practice, mobility data may serve as a leading indicator for changes in transmission, with spikes in mobility potentially preceding increases in case numbers [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]. Similarly, vaccination rates can be modeled as factors that mitigate disease spread, while socio-economic factors such as population density and income levels might modulate both susceptibility to infection and response to public health interventions [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e, \u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e]. The integration of these features requires careful preprocessing, including normalization and temporal alignment with the case data, to ensure their effective inclusion in the model. Moreover, the methodology developed in this study for COVID-19 forecasting could be adapted to other diseases or domains where data limitations hinder predictive performance. Our approach could be extended to financial and environmental sectors, with careful hyperparameter tuning to accommodate the unique characteristics of these domains. [\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e, \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn conclusion, our study introduces multivariate models that integrate time series data from multiple sources. This approach effectively leverages COVID-19 data from other countries to capture trends similar to those in the target country, thereby improving forecasting performance. This demonstrates the potential of multivariate models to enhance forecasting in scenarios where data may be insufficient for DL models, making them a promising tool for time series forecasting in such contexts.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e \u003ch2\u003eEthics approval and consent to participate:\u003c/h2\u003e \u003cp\u003eNot applicable\u003c/p\u003e \u003c/p\u003e\u003cp\u003e \u003ch2\u003e \u003cb\u003eCompeting interests\u003c/b\u003e:\u003c/h2\u003e \u003cp\u003eThe authors declare no competing interests.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eClinical trial number\u003c/strong\u003e \u003cp\u003eNot applicable\u003c/p\u003e \u003c/p\u003e\u003ch2\u003eFunding:\u003c/h2\u003e \u003cp\u003eThe authors declare that there is no funding source for this study.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eT.P. led the conceptualization and supervision of the study; J.O., Z.L., K.H., T.G., H.S., and J.P. contributed to methodology; J.O. and Z.L. performed the formal analysis and data curation; investigation was carried out by J.O., Z.L., K.H., T.G., H.S., and J.P.; J.O. and Z.L. prepared the original draft, while T.P., J.O., and Z.L. reviewed and edited it; visualization and software were handled by J.O. Project administration was overseen by T.P. All authors have reviewed and approved the final published version.\u003c/p\u003e\u003ch2\u003eAcknowledgments:\u003c/h2\u003e \u003cp\u003eThe authors express their sincere gratitude to all participants involved in this study.\u003c/p\u003e\u003ch2\u003eData availability:\u003c/h2\u003e \u003cp\u003ePublicly available data were used in this study and can be found in the references provided in the text.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eHan K, Lee B, Lee D, Heo G, Oh J, Lee S, Park T. Forecasting the spread of COVID-19 based on policy, vaccination, and Omicron data. Sci Rep. 2024;14(1):9962.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLee B, Song H, Apio C, Han K, Park J, Liu Z, Park T. (2023). An analysis of the waning effect of COVID-19 vaccinations. Genomics Inf, \u003cem\u003e21\u003c/em\u003e(4).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHyndman RJ, Athanasopoulos G. Forecasting: principles and practice. OTexts; 2018.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVaswani A et al. (2017). Attention is all you need. Adv Neural Inf Process Syst, 30.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLim W et al. (2021). Temporal fusion transformers for interpretable multi-horizon time series forecasting. \u003cem\u003earXiv preprint arXiv:1912.09363.\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChimmula VKR, Zhang L. Time series forecasting of COVID-19 transmission in Canada using LSTM networks. Chaos Solitons Fractals. 2020;135:109864.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang L, et al. COVID-Net: A tailored deep convolutional neural network design for detection of COVID-19 cases from chest X-ray images. Sci Rep. 2020;10(1):19549.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBerndt DJ, Clifford J. (1994). Using dynamic time warping to find patterns in time series. In \u003cem\u003eProceedings of the 3rd international conference on knowledge discovery and data mining\u003c/em\u003e,359\u0026ndash;370.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMathieu E, Ritchie H, Ortiz-Ospina E, et al. A global database of COVID-19 vaccinations. Nat Hum Behav. 2021;5:947\u0026ndash;53.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChatfield C, Xing H. (2019). \u003cem\u003eThe analysis of time series: an introduction with R\u003c/em\u003e. Chapman and hall/CRC.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSherstinsky A. (2020). Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. \u003cem\u003ePhysica D: Nonlinear Phenomena\u003c/em\u003e, \u003cem\u003e404\u003c/em\u003e, 132306. Faria, D. R.,. (2021). Forecasting the number of COVID-19 cases in the world. \u003cem\u003eChaos, Solitons \u0026amp; Fractals\u003c/em\u003e, 142, 110497.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCho K et al. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735\u0026ndash;80.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGraves A, Schmidhuber J. (2005). Framewise phoneme classification with bidirectional LSTM networks. Proceedings of the IEEE International Joint Conference on Neural Networks, 2047\u0026ndash;2052.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChung J, Gulcehre C, Cho K, Bengio Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:14123555.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShi X et al. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Adv Neural Inf Process Syst, 28.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBai S, Kolter JZ, Koltun V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:180301271.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLea C, Flynn MD, Vidal R, Reiter A, Hager GD. (2017). Temporal convolutional networks for action segmentation and detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 156\u0026ndash;165.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMoritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., \u0026hellip; Stoica, I.(2018). Ray: A distributed framework for emerging {AI} applications. In \u003cem\u003e13th USENIX symposium on operating systems design and implementation (OSDI 18)\u003c/em\u003e (pp. 561\u0026ndash;577).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKe G, Meng Q, Finley T, Wang et al. (2017). Lightgbm: A highly efficient gradient boosting decision tree. \u003cem\u003eAdvances in neural information processing systems\u003c/em\u003e, \u003cem\u003e30\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDiebold FX, Mariano RS. Comparing Predictive Accuracy. J Bus Economic Stat. 1995;13(3):253\u0026ndash;63.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKost JT, McDermott MP. Combining dependent P-values. Stat Probab Lett. 2002;60(2):183\u0026ndash;90.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang C, Bengio S, Hardt M, Recht B, Vinyals O. (2017). Understanding deep learning requires rethinking generalization. \u003cem\u003eInternational Conference on Learning Representations (ICLR)\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBelkin M, Hsu D, Ma S, Mandal S. (2019). Reconciling modern machine-learning practice and the classical bias\u0026ndash;variance trade-off. \u003cem\u003eProceedings of the National Academy of Sciences\u003c/em\u003e, 116(32), 15849\u0026ndash;15854.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNeyshabur B, Tomioka R, Srebro N. (2015). In search of the real inductive bias: On the role of implicit regularization in deep learning. \u003cem\u003eInternational Conference on Learning Representations (ICLR)\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eArora S, Cohen N, Hu W, Luo Y. (2019). Implicit regularization in deep matrix factorization. Adv Neural Inf Process Syst, 32.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBartlett PL, Montanari A, Rakhlin A. Deep learning: a statistical viewpoint. Acta Numerica. 2021;30:87\u0026ndash;201.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDevlin J, Chang MW, Lee K, Toutanova K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. \u003cem\u003earXiv preprint arXiv:1810.04805\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBrown TB, Mann B, Ryder N et al. (2020). Language Models are Few-Shot Learners. \u003cem\u003eProceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020)\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKraemer, M. U. G., Yang, C.-H., Gutierrez, B., Wu, C.-H., Klein, B., Pigott, D. M.,\u0026hellip; Scarpino, S. V. (2020). The effect of human mobility and control measures on the COVID-19 epidemic in China. \u003cem\u003eScience\u003c/em\u003e, 368(6490), 493\u0026ndash;497.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJia JS, Lu X, Yuan Y, Xu G, Jia J, Christakis NA. Population flow drives spatio-temporal distribution of COVID-19 in China. Nature. 2020;582(7812):389\u0026ndash;94.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCheng, C., Jiang, W. M., Fan, B., Cheng, Y. C., Hsu, Y. T., Wu, H. Y., \u0026hellip; Tsou, H.H. (2023). Real-time forecasting of COVID-19 spread according to protective behavior and vaccination: autoregressive integrated moving average models. \u003cem\u003eBMC public health\u003c/em\u003e, \u003cem\u003e23\u003c/em\u003e(1), 1500.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBorchering RK. (2021). Modeling of future COVID-19 cases, hospitalizations, and deaths, by vaccination rates and nonpharmaceutical intervention scenarios\u0026mdash;United States, April\u0026ndash;September 2021. \u003cem\u003eMMWR. Morbidity and Mortality Weekly Report\u003c/em\u003e, \u003cem\u003e70\u003c/em\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSezer OB, Gudelek MU, Ozbayoglu AM. Financial time series forecasting with deep learning: A systematic literature review: 2005\u0026ndash;2019. Appl Soft Comput. 2020;90:106181.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLin WH, Wang P, Chao KM, Lin HC, Yang ZY, Lai YH. Wind power forecasting with deep learning networks: Time-series forecasting. Appl Sci. 2021;11(21):10335.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"COVID-19 Forecasting, Deep Learning models, Transformer model, Time series prediction, Dynamic Time Warping","lastPublishedDoi":"10.21203/rs.3.rs-5400759/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5400759/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eIt is well known that deep learning (DL) models often struggle with low prediction performance due to data scarcity. This scarcity hampers the effectiveness of DL methods that typically require large datasets to generate reliable forecasts. Recently, several DL models have been proposed for predicting the spread of COVID-19. These models are country specific models and thus use the COVID-19 data only from the target country. To improve COVID-19 forecasting using DL models, we propose multivariate DL models using the additional data available from other countries.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eBased on the rankings determined by Dynamic Time Warping (DTW) distance, which calculates the similarity of infection trends across countries, univariate DL models using only the target country data were extended to multivariate models which integrated data from the top-ranked countries to optimize performance. We considered seven DL models including the Transformer model, TCN, CNN-LSTM, BiLSTM, GRU, RNN, and LSTM.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eOur results showed that the multivariate transformer model achieved the most significant improvements in forecasting accuracy, with an average reduction of 60.15% in mean root mean square error (RMSE) across the five target countries and five time periods when integrating data from additional countries, compared to univariate models using only the target country data. Additionally, multivariate transformer models consistently demonstrated significant improvements over univariate models in terms of mean RMSE, as evidenced by the Diebold-Mariano test. Other multivariate DL models also showed performance gains, with the TCN model achieving an average reduction in RMSE of 36.28%, followed by CNN-LSTM at 29.47%, BiLSTM at 21.07%, GRU at 21.43%, RNN at 17.46%, and LSTM at 16.38%.\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eThese findings indicate that leveraging similar infection patterns from data of other countries can provide valuable information for predicting the COVID-19 spread in the target country, especially when data is scarce, thereby enhancing forecasting performance.\u003c/p\u003e","manuscriptTitle":"Enhancing COVID-19 Forecasts Through Multivariate Deep Learning Models","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-11-28 09:56:38","doi":"10.21203/rs.3.rs-5400759/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"c12f165e-8e42-42ba-8a92-21d7894ea774","owner":[],"postedDate":"November 28th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-03-26T11:08:37+00:00","versionOfRecord":[],"versionCreatedAt":"2024-11-28 09:56:38","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5400759","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5400759","identity":"rs-5400759","version":["v1"]},"buildId":"_2-kVJe1T_tPrBINL-cwx","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}