Attention-based generative adversarial networks for aquaponics environment time series data imputation

Attention-based generative adversarial networks for aquaponics environment time series data imputation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Attention-based generative adversarial networks for aquaponics environment time series data imputation Keyang Zhong, Xueqian Sun, Gedi Liu, Yifeng Jiang, Yi Ouyang, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3209861/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Environmental parameter data collected by sensors for monitoring the environment of aquaponics facility operations are usually incomplete due to external environmental disturbances and device failures. And the missing of collected data is completely at random. In practice, missing data could create biased estimations and make multivariate time series predictions of environmental parameters difficult, leading to imprecise environmental control. A multivariate time series imputation model based on generative adversarial networks and multi-head attention (ATTN-GAN) is proposed in this work to reducing the negative consequence of missing data. ATTN-GAN can capture the temporal and spatial correlation of time series, and has a good capacity to learn data distribution. In the downstream experiments, we used ATTN-GAN and baseline models for data imputation, and predicted the imputed data, respectively. Experiments reveal that ATTN-GAN outperforms all baseline models in terms of data imputation accuracy. The data processed by ATTN-GAN is the best for time series prediction. Attention-based Generative Adversarial Networks Aquaponics Greenhouse Missing Data Data Imputation Multivariate Time Series Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 1. Introduction Prediction and classification of multivariate time series are tasks that are commonly utilized in finance (Che et al., 2018 ; Hsieh et al., 2011 ), transportation(Yu et al., 2016 ), healthcare (Choi et al., 2018 ), and agricultural facilities (Boomgard-Zagrodnik and Brown, 2022 ). In agricultural applications, sensors are required to collect environmental data to monitor the production environment. Complete data in agricultural production are crucial for the analysis and control of environment and will affect agricultural production (Boomgard-Zagrodnik and Brown, 2022 ; Huan et al., 2020 ; Zhang and Thorburn, 2021 ). For example: during crop growth, sensors need to monitor data on soil moisture, salinity and other parameters; in fish farming, sensors need to collect water quality data and meteorological data. Then during the high-frequency collection of environmental data, it is inevitable that some data will be lost due to interruptions and failures in the data collection and transmission process. The prediction and classification of time series requires a complete data set and missing values in the time series data can lead to biased estimations. Therefore, how to deal with missing values in multivariate time series becomes a key topic. The methods for dealing with missing values can be divided into two categories: one is the case deletion method, which removes missing elements directly (Kaiser, 2014 ; Silva and Zárate, 2014 ) and ignores certain data information and impose bias, making further analysis challenging. The second is to impute missing values that is superior to case deletion because imputation can make use of information from some observations. Existing missing value imputation methods can be classified into five categories, including simple imputation methods, statistical analysis methods, machine learning-based methods, attention mechanisms, and generative adversarial network. The simple imputation methods such as mean imputation, and last observation imputation, make use of local information but ignore significant long-term temporal trends. Statistical analysis methods like autoregressive method (Box and Jenkins, 1968 ) and smooth interpolation utilizes the historical information of the time series and reconstruct missing values by fitting a smooth curve (Fung, 2006 ; Hamzacebi, 2008 ; Kreindler and Lumsden, 2006 ; Wilson, 2016). However, these methods demand the time series remain stationary and loss sight of the spatial connection between multivariate time series. Machine learning-based imputation methods are useful for multivariate time series imputation. K-nearest neighbors (KNN) (Hudak et al., 2008 ; Tak et al., 2016 ) uses the mean value of k nearest neighbors to fill missing values. Mmatrix factorization (Acar et al., 2010 ; Hastie et al., 2014 ; Mazumder et al., 2010 ) decompose the incomplete data set into low-rank matrices and use the product of these two matrices to imputation the missing components. Multilayer perceptron (MLP) (Chen et al., 2001 ) estimates missing values by stacking multiple layers of neurons. Nevertheless, these methods neither capture temporal patterns in time series datasets nor take into account the link between missing values and observations. Cao et al. ( 2018 ) and Yoon et al. ( 2019 ) used a bidirectional recurrent neural network to impute missing values in a healthcare dataset. Che et al. ( 2018 ) proposed a variant of Gated Recurrent Unit (GRU-D) which takes the advantage of last observed value and mean value to handle missing data in time series classification problems. However, recurrent neural networks are difficult to capture spatial correlations in multivariate time series, and the imputation performance is not very good. Attention mechanisms (Vaswani et al., 2017 ) are believed to be able to capture both temporal and spatial correlations in multivariate time series. Ma et al.(2019) jointly applied cross-dimensional self-attention from three dimensions (time, location, and measurement) to fill missing values in geotagged data. Zhang and Thorburn ( 2021 ) proposed a dual-head sequence-to-sequence imputation model, which includes an encoder with two gated recurrent units and an attention mechanism in the encoder's output layer and achieved good performance in a water quality information system dataset. Du et al. ( 2022 ) proposed a method (SAITS) for multivariate time series missing value imputation based on the self-attention mechanism. SAITS learns the distribution of data from a weighted combination of two diagonally masked self-attention blocks through joint optimization to complete missing values imputation. Many researchers in the field of multivariate time series processing have also developed methods to impute missing data based on generative adversarial networks (Goodfellow et al., 2014 ). The first to propose a GAN-based multivariate time series missing value imputation approach is Luo et al. ( 2018 ). They adopted a modified gate recursive unit (GRUI) in GAN to model the irregularity of incomplete time series. GRUI takes into account non-fixed time lags and attenuates the influence of past observations determined by the time lags. Following that, they suggested an E2GAN (end-to-end generative model) (Luo et al., 2019 ). E2GAN inputs random noise to the generator, learns the mapping of random noise to the distribution of the training data set, and simplifies the imputation process of the time series. Yoon et al. ( 2018 ) improves upon the standard GAN architecture by feeding the discriminator with generated data and providing hinting mechanism information, which assures the it generates samples based on the actual underlying distribution. Gupta and Beheshti ( 2020 ) proposed model (Bi-GAN) uses a bidirectional recurrent network in a generative adversarial setting which makes use of RNNs' strength in capturing temporal correlations in time series and GAN designs' superiority in fitting data distributions. The purpose of this study was to explore the best model for missing values imputation on aquaponics greenhouse environment data. We explore the effectiveness of the proposed model in capturing temporal and spatial correlations in multivariate time series by conducting comparative experiments. This paper provides a model for missing data imputation based on a generative adversarial network architecture that employs a multi-head attention mechanism in both the generator and the discriminator. It combines the superior performance of attention mechanisms in capturing spatiotemporal correlations and the superior ability of GANs to fit data distributions. The generator and the discriminator are not identical. The generator imputes missing values using two weighted link multi-head attention modules, whereas the discriminator employs a single multi-head attention module to discriminate between imputed and observed values. Overall, the following are the study's major contributions: A multi-head attention-based generative adversarial network imputation algorithm was proposed, which can effectively learn the distribution of data to achieve quite accurate imputation. The proposed model combines an attention mechanism with a GAN architecture to capture the temporal and spatial correlation of multivariate time series to improve the estimation of missing values. The model was tested on datasets gathered from aquaponics greenhouse environments and results show that it outperforms state-of-the-art methods in both data imputation accuracy and downstream prediction tasks. 2. Materials and methods 2.1 Dataset A real-world dataset was collected to validate this model. The dataset was collected from an aquaponics greenhouse practicing base in Tongzhou, Beijing. The base's sensors recorded eight characteristics, including soil moisture, water temperature, and air humidity. The sensors capture data every two hours. From July 3, 2020, to January 2, 2021, a total of 2208 pieces of data were gathered. To validate the suggested model, the first 80% of the data was separated into a training set and the remaining 20% was separated into a test set. The data includes around 8% missing values due to sensor failure and other factors. 2.2 Problem Formulation Firstly, the problem formulation and some necessary preliminaries were presented. A collection of multivariate time series with \(D\) dimensions and \(T\) steps was given: $$X =\left\{{x}_{1},{x}_{2},\dots ,{x}_{t},\dots {x}_{T}\right\}\in {\mathbb{R}}^{T\times D}$$ where \({x}_{t}\) is the \(t\) -th observation of \(X\) , \({x}_{t}=\left\{{x}_{t}^{1},{x}_{t}^{2},\dots ,{x}_{t}^{d}, \dots ,{x}_{t}^{D}\right\}\in {\mathbb{R}}^{D}\) . Except some missing values, some components were artificially masked randomly. And \({X}_{init}\) is used to represent raw data that initially has missing values. The location matrix \(M\) of missing or masked values as well as the time delays matrix \(\delta\) of two adjacent values were recorded: $${M}_{t}^{d}=\left\{\begin{array}{c}1, if {x}_{t}^{d} is observation\\ 0, if {x}_{t}^{d} is missing or masked\end{array}\right.$$ $${\delta }_{t}^{d}=\left\{\begin{array}{c}0,\\ {t}_{i}-{t}_{i-1},\\ {\delta }_{{t}_{i-1}}^{d}+{t}_{i}-{t}_{i-1},\end{array}\begin{array}{c}i==0\\ { M}_{{t}_{i-1}}^{d}==1\&i>0\\ { M}_{{t}_{i-1}}^{d}==0\&i>0\end{array}\right.$$ where \({x}_{t}^{d}\) is the \(t\) -th step observation of \(d\) -th dimension variable in \(X\) , \(M{\in \mathbb{R}}^{T\times D}\) , and \(\delta {\in \mathbb{R}}^{T\times D}\) . To distinguish artificially masked values and originally missing values, the indicating matrix \(I\) and \({M}^{{\prime }}\) are introduced. $${I}_{t}^{d}=\left\{\begin{array}{c}1, if {X}_{t}^{d} is masked\\ 0, otherwise\end{array}\right.$$ $${{M}^{{\prime }}}_{t}^{d}=\left\{\begin{array}{c}1, if {x}_{t}^{d} is observation or masked\\ 0, if {x}_{t}^{d} is missing\end{array}\right.$$ $$H=h{M}^{{\prime }}$$ where \(H\) is Hint Matrix which generated from \({M}^{{\prime }}\) , and \(h\) is a random number. 2.3 Approach 2.3.1 General Framework Figure 1 illustrates the general architecture of ATTN-GAN. A generator (G) and a discriminator (D) comprise the generative adversarial network (GAN). The generator which was feed with incomplete data and some other information imputed the missing values of multivariate time series. And the discriminator aims to predict the matrix \(M\) which indicates whether the element is generated or observed. Next, the details of the multi-head self-attention blocks was described, and generator and discriminator that consist mainly of multi-head self-attention blocks are also presented detailly. 2.3.2 Multi-head Self-attention Blocks Vaswani et al. ( 2017 ) proposed a traditional self-attention mechanism to solve the language translation task. It is now widely used in tasks such as sequence modeling and computer vision. The attention function can be described as the mapping of a query and a set of key-value pairs to an output, where the query and key are of dimension \({d}_{k}\) and the value is of dimension \({d}_{v}\) . In practice, the attention function is computed simultaneously on a set of queries packed together into a matrix \(Q\) . The keys and values are also packed together into matrix \(K\) and \(V\) . The output matrix of the attention function is computed as follows: $$\begin{array}{c}Attention\left(Q,K,V\right)=softmax\left(\frac{Q{K}^{T}}{\sqrt{{d}_{k}}}\right)V\#\left(1\right)\end{array}$$ Multi-head self-attention allows the model to jointly focus information from different representations subspaces at different locations. The multi-head self-attention means that \(h\) different output matrixes of the attention function are mapped from the \(Q,K,V\) groups. $$\begin{array}{c}MultiHead\left(Q,K,V\right)=Concat\left(hea{d}_{1},\dots , hea{d}_{h}\right){W}^{o}\#\left(2\right)\end{array}$$ $$\begin{array}{c}Where hea{d}_{i}=Attention\left(Q{W}_{i}^{Q},K{W}_{i }^{K},V{W}_{i}^{V}\right)\#\left(3\right)\end{array}$$ where \(h\) is the number of heads, \({W}_{i}^{Q}{\in \mathbb{R}}^{{d}_{model}\times {d}_{k}}, { W}_{i }^{K}{\in \mathbb{R}}^{{d}_{model}\times {d}_{k}},{ W}_{i}^{V}{\in \mathbb{R}}^{{d}_{model}\times {d}_{v}}, {W}^{o}{\in \mathbb{R}}^{h{d}_{v}\times {d}_{model}}\) . In this work, \(h=3,{d}_{k}={d}_{v}={d}_{model}=64\) were employed in discriminator; \(h=3,{d}_{k}={d}_{v}=64,{d}_{model}=128\) were employed in generator. Multi-head self-attention blocks include residual mode, which is followed by feed Forward and Layer Norm module activities. Here, these are generalized with the symbol \(MultiHeadBlocks\left(\right)\) . 2.3.3 Generator The Generator Network architecture is shown in Fig. 2 . The generator consists of two combined Multi-head Self-attention blocks and a weighted combine block. The goal of the generator is to fit the distribution of the complete data, which in turns generates the completed multivariate time series. To utilize the limited information as much as possible, incomplete time series, \(M\) matrix, and δ matrix were fed into the generator. Thus, the generator was described as a mapping: $$\begin{array}{c}G\left(X,M,\delta \right)=\stackrel{\sim}{X}\#\left(4\right)\end{array}$$ Then the missing values of \(X\) was replaced with the following equation: $$\begin{array}{c}\widehat{X}=X\odot {M}^{{\prime }}+\left(1-{M}^{{\prime }}\right)\odot \stackrel{\sim}{X}\#\left(5\right)\end{array}$$ where \(\odot\) is element-wise multiplication; \(\stackrel{\sim}{X}\) is the generated data vector of G; \(\widehat{X}\) is the imputed data vector, that is, the matrix obtained by replace the missing values of the original sequence with the corresponding components generated by G. The first Multi-head Self-attention blocks take the time lag vector \(\delta\) and \(X\) as inputs. Firstly, the concatenated \(\delta\) and \(X\) pass through the encoding layer, adding the positional encoding as the input of Multi-head Self-attention blocks. Then \({\stackrel{\sim}{X}}_{1}\) was got from the first Multi-head Self-attention blocks following a fully connected layer. The following equations show the data transfer process: $$\begin{array}{c}c=\left(concat\left(X,\delta \right){W}_{c}+{b}_{c}\right)+p\#\left(6\right)\end{array}$$ $$\begin{array}{c}z={\left\{MultiHeadBlocks\left(c\right)\right\}}^{N}\#\left(7\right)\end{array}$$ $$\begin{array}{c}{\stackrel{\sim}{X}}_{1}=z*{W}_{z}+{b}_{z}\#\left(8\right)\end{array}$$ where \({W}_{c}{\in \mathbb{R}}^{\left(2*{d}_{k}\right)\times {d}_{model}}\) , \({W}_{z}{\in \mathbb{R}}^{{d}_{model}\times {d}_{k}}\) , \({b}_{c}{\in \mathbb{R}}^{2*{d}_{k}}\) , \({b}_{z}{\in \mathbb{R}}^{{d}_{k}}\) are parameters, \({\left\{\right\}}^{N}\) is stacking \(N\) Multi-head Self-attention blocks, and \(p\) is positional encoding: $$\begin{array}{c}\left\{\begin{array}{c}P{E}_{\left(pos,2i\right)}=\text{sin}\left(\frac{pos}{1000{0}^{2i/{d}_{model}}}\right)\\ P{E}_{\left(pos,2i+1\right)}=\text{cos}\left(\frac{pos}{1000{0}^{2i/{d}_{model}}}\right)\end{array}\right.\#\left(9\right)\end{array}$$ where \(pos\) is the time-step position, and \(i\) is the dimension. The second Multi-head Self-attention blocks takes the \(M\) and \({\stackrel{\sim}{X}}_{1}\) as input. Similar to the first Multi-head Self-attention blocks, the concatenated \(M\) and \({\stackrel{\sim}{X}}_{1}\) pass through the encoding layer, adding the positional encoding as the input of Multi-head Self-attention blocks: $$\begin{array}{c}\alpha =\left(concat\left({\stackrel{\sim}{X}}_{1},M\right){W}_{\alpha }+{b}_{\alpha }\right)+p\#\left(10\right)\end{array}$$ $$\begin{array}{c}\beta ={\left\{MultiHeadBlocks\left(\alpha \right)\right\}}^{N}\#\left(11\right)\end{array}$$ $$\begin{array}{c}{\stackrel{\sim}{X}}_{2}=\beta *{W}_{\beta }+{b}_{\beta }\#\left(12\right)\end{array}$$ To obtain a better learned representation \(\stackrel{\sim}{X}\) , a weighted combination block is designed to dynamically weight \({\stackrel{\sim}{X}}_{1}\) and \({\stackrel{\sim}{X}}_{2}\) according to temporal correlation and missing information: $$\begin{array}{c}\widehat{A}=\frac{1}{h}\sum _{i}^{h}{A}_{i}\#\left(13\right)\end{array}$$ $$\begin{array}{c}\gamma =sigmoid\left(\widehat{A}{W}_{\gamma }+{b}_{\gamma }\right)\#\left(14\right)\end{array}$$ $$\begin{array}{c}\widehat{X}=\left(1-\gamma \right)\odot {\stackrel{\sim}{X}}_{1}+ \gamma \odot {\stackrel{\sim}{X}}_{2}\#\left(15\right)\end{array}$$ where \({W}_{\alpha }{\in \mathbb{R}}^{\left(2*{d}_{k}\right)\times {d}_{model}}\) , \({W}_{\beta }{\in \mathbb{R}}^{{d}_{model}\times {d}_{k}}\) , \({W}_{\gamma }{\in \mathbb{R}}^{T\times {d}_{k}}\) , \({b}_{\alpha }{\in \mathbb{R}}^{2*{d}_{k}}\) , \({b}_{\beta }{\in \mathbb{R}}^{{d}_{k}}\) , \({b}_{\gamma }{\in \mathbb{R}}^{{d}_{k}}\) are parameters, and \(\widehat{A}{\in \mathbb{R}}^{T\times T}\) is averaged from attention weights \(A\) output by the last layer of the second Multi-head Self-attention block. Since the purpose of the generator is to learn the distribution of complete multivariate time series, a squared error loss was added to the loss function of the generator. The loss function is defined as follows: $$\begin{array}{c}{L}_{G}=\lambda \left|\right|X\odot {M}^{{\prime }}-\widehat{X}\odot {M}^{{\prime }}|{|}_{2}+\eta ||X\odot I-\widehat{X}\odot I|{|}_{2}-{L}_{D}\#\left(16\right)\end{array}$$ where \(\lambda\) and \(\eta\) are hyper-parameters that controls the weight of the discriminator’s loss and the squared error loss between generated \(\widehat{X}\) and original \(X\) , and \({L}_{D}\) is the loss of discriminator. 2.3.4 Discriminator As in regular GAN framework, a discriminator D used to train G was introduced in this method. The Discriminator Network architecture of proposed method is shown at Fig. 3. Unlike the standard GAN whose output is either to judge the input is complete real or complete fake (generated by G), in the proposed model the discriminator attempts to distinguish which components are real (observed) or fake (imputed by G) -this amounts to predicting the matrix \(M\) . In simple terms, the output of D is the probability that each component is real. A hint mechanism was introduced (Yoon et al., 2018 ). The hint mechanism is a random variable, \(H\) ,which depend on each sample \((X,M)\) . The function of D is as follows: $$\begin{array}{c}D\left(\widehat{X},H\right)=P\#\left(17\right)\end{array}$$ where each component in \(P\) represents the probability of the component is real (not generated by G). The Multi-head Self-attention blocks take the Hint Matrix and \(\widehat{X}\) , which consists of observed components and generated components. The concatenated Hint Matrix \(H\) and \(\widehat{X}\) pass through the encoding layer, adding the position encoding as the input of Multi-head Self-attention blocks. Multi-head self-attention blocks are followed by linear layers and activation functions, and then the probability matrix \(P\) is got. The following equations show the data transfer process: $$\begin{array}{c}e=\left(concat\left(\widehat{X},H\right){W}_{e}+{b}_{e}\right)+p\#\left(18\right)\end{array}$$ $$\begin{array}{c}g=\{MultiHeadBlocks\left(e\right){\}}^{N}\#\left(19\right)\end{array}$$ $$\begin{array}{c}P=Sigmoid\left(\left(g*{W}_{g}+{b}_{g}\right)\right)\#\left(20\right)\end{array}$$ where the \({W}_{e}{\in \mathbb{R}}^{\left(2*{d}_{k}\right)\times {d}_{model}}\) , \({W}_{g}{\in \mathbb{R}}^{{d}_{model}\times {d}_{k}}\) , \({b}_{e}{\in \mathbb{R}}^{2*{d}_{k}}\) , \({b}_{g}{\in \mathbb{R}}^{{d}_{k}}\) are parameters. The loss function of the discriminator is as follows: $$\begin{array}{c}{L}_{D}=|\left|P-M\right|{|}_{2}\#\left(21\right)\end{array}$$ 2.4 Evaluation and baseline 2.4.1 Metrics To demonstrate the effectiveness of proposed methods for multivariate time series missing values imputation, we employ mean absolute error (MAE) root mean squared error (RMSE), and mean absolute percentage error (MAPE) to evaluate our methods. Since the missing values of the original missing data are not known, the imputation performance of the artificially masked parts is used to evaluate each model. Therefore the evaluation metrics are defined as follows: $$MAE= \frac{{\sum }_{d}^{D}{\sum }_{t}^{T}|\left(\widehat{{x}_{t}^{d}}-{x}_{ini{t}_{t}}^{d}\right)\odot {I}_{t}^{d}|}{{\sum }_{d}^{D}{\sum }_{t}^{T}{I}_{t}^{d}}$$ $$RMSE= \sqrt{\frac{{\left(\left(\widehat{{x}_{t}^{d}}-{x}_{ini{t}_{t}}^{d}\right)\odot {I}_{t}^{d}\right)}^{2}}{{\sum }_{d}^{D}{\sum }_{t}^{T}{I}_{t}^{d}}}$$ $$MRE= \frac{{\sum }_{d}^{D}{\sum }_{t}^{T}|\left(\widehat{{x}_{t}^{d}}-{x}_{ini{t}_{t}}^{d}\right)\odot {I}_{t}^{d}|}{{\sum }_{d}^{D}{\sum }_{t}^{T}|\widehat{{x}_{t}^{d}} \odot {I}_{t}^{d}|}$$ 2.4.2 Baseline The baseline model compared with the proposed model is as follows: MEAN: For the corresponding feature, missing data was replaced with the weekly mean. LAST: Missing values are replaced with the last observation ARIMA(Box and Jenkins, 1968 ): The imputation of missing values was implement by using an autoregressive moving average model for time series prediction KNN(Hudak et al., 2008 ): The missing values are imputed based on the k nearest neighbor samples. Transformer: The basic component of the proposed model is the multi-head self-attention proposed by Vaswani et al. ( 2017 ). Here the basic Transformer model was used for comparison. SAITS(Du et al., 2022 ): It proposed two diagonally-masked self-attention(DMSA) blocks to learn the temporal and feature correlations between missing and observed values, resulting in accurate imputation of missing values. 3. Results 3.1 Model comparison of imputation results Table 1 Imputation performance of the proposed method and baseline model in (MAE / RMSE / MRE). Bold indicates best performance. Missing rate MEAN LSAT KNN ARIMA Transformer SIATS ATTM-GAN 10% 0.4807/0.6573/0.4583 0.4319/0.4959/0.4694 0.2626/0.4125/0.3050 0.2794/0.3858/0.3084 0.2274/0.3085/0.2112 0.1221/ 0.1476 /0.1201 0.1024 /0.1532/ 0.1092 20% 0.567/0.696/0.4944 0.4853/0.5014/0.4743 0.3297/0.4738/0.3843 0.2817/0.3002/0.3118 0.2317/0.3302/0.2213 0.1301/0.1603/0.1314 0.1115 / 0.1593 / 0.1144 30% 0.5819/0.6924/0.5475 0.5522/0.5958/0.5323 0.4189/0.5683/0.4932 0.3141/0.4554/0.3521 0.2413/0.3534/0.2345 0.1383/ 0.1637 /0.142 0.1189 /0.1736/ 0.1237 40% 0.6073/0.7308/0.6611 0.6077/0.6366/0.6618 0.4828/0.6486/0.5675 0.3388/0.4987/0.3806 0.2688/0.3897/0.2587 0.1458/0.2508/0.1503 0.1236 / 0.1908 / 0.1359 50% 0.6898/0.7733/0.6753 0.6445/0.6955/0.7046 0.5421/0.6903/0.6368 0.3730/0.5509/0.4206 0.3030/0.4039/0.2816 0.1703/0.2720/0.1609 0.1424/0.2012/0.1443 60% 0.7432/0.8248/0.7816 0.69690.7711/0.7680 0.5897/0.7364/0.6954 0.4094/0.5941/0.4648 0.3490/0.4441/0.3048 0.1955/0.3016/0.1701 0.1579 / 0.2314 / 0.1492 70% 0.8093/0.8696/0.8494 0.73710.8266/0.8144 0.6097/0.7864/0.7454 0.4287/0.6061/0.4868 0.3871/0.5036/0.3268 0.2017/0.3214/0.1820 0.1708 / 0.2539 / 0.1515 80% 0.822/0.9072/0.9713 0.7466/0.8737/0.9483 0.6801/0.8545/0.7986 0.4927/0.6949/0.5610 0.4237/0.5514/0.3532 0.2137/0.3398/0.2041 0.2047 / 0.2688 / 0.1790 Table 1 provides the global experimental results of ATTN-GAN and all baseline methods. When the evaluation index is RMSE and the data missing rate is 10% and 30%, the imputation performance of ATTN-GAN is slightly worse than that of SAITS, but obtain excellent imputation results. What’s more, as Table 1 shows, ATTN-GAN achieves the best results under any other evaluation metrics. In order to illustrate the imputation effect of the proposed model in the cases of low, medium, and high data missing rate, we only briefly describe the cases of missing rate of 20%, 50%, and 80%. Compared to the optimal baseline results, it lowers down RMSE by 0.623%, 26.02%, 20.89% over the 20%, 50% and 80% missing rate. Specifically, the performance of MEAN is worst upon most instances, also the LAST has a very large error. This is because that MEAN and LAST is simple imputation methods which Simply take advantage of information from the observed data to impute missing values in time series. Machine learning-based KNN and statistics based ARIMA show a big improvement over the simple methods, especially when the missing rate is small. Transformer, SIATS, and ATTN-GAN based on attention mechanism further optimize the performance of missing values imputation. Obviously, compared to Transformer, ATTN-GAN reduces the RMSE from 0.3302, 0.4039, 0.5514 to 0.1593, 0.2012, 0.2688 over the 20%, 50% and 80% missing rate. Therefore, our proposed model, ATTN-GAN, consisting of two combined multi-headed self-attentive blocks and a weighted combined block in the generator is beneficial for the imputation of missing data. In order to more intuitively reveal the impact of the increase in the missing rate on the performance of the model, Fig. 4 shows the changes in the imputation results of the model. The simple model (i.e., MEAN and LAST) performs very poorly across all missing rates. When the data missing rate gradually increases, the performance of the statistics-based imputation model and machine learning-based KNN method drop sharply, and the performance of Transformer based on the attention mechanism also drops significantly. Surprisingly, the bias of the proposed ATTN-GAN's and SAITS’s estimation of missing values increases only slightly as the missing rate increases. However, when the missing rate is high, ATTN-GAN has a smaller imputation error in comparison. This is because the ATTN-GAN takes advantage of both generative adversarial structure and attention mechanism, which means it has a strong ability to capture data distribution and spatiotemporal correlation of data. To compare the actual imputation results more intuitively, the detailed imputation results of the baseline model and the proposed model for four consecutive missing values are plotted. Figure 5(a) shows the implementation in the absence of four consecutive air humidity measurements, and Fig. 5(b) depicts the implementation in the absence of four consecutive water temperature measurements. It is intuitively clear from Fig. 5. that the simple method imputation results are very poor and cannot fit the trends of time series, which can only simply use the contextual information. KNN can simply fit the trend of the time series, leading to a relatively large error. In general, SAITS, Transformer, and ATTN-GAN can more accurately use data information to fit missing data, and outperform statistics-based and machine-learning-based models in missing value imputation. What’s more, as shown in Fig. 5, ATTN-GAN estimate missing values better than Transformer and SAITS, especially when values change rapidly. 3.2 Comparison of prediction results Due to equipment or transportation reasons, we do not have access to the original complete data, so we cannot directly compare the advantages and disadvantages of each imputation model. A good approach is to use the performance of downstream tasks to compare the true imputation performance of each model’s imputation for raw missing values indirectly. For downstream tasks, we first perform missing value imputation on multivariate time series with a missing rate of 40% by the proposed model and the baseline model. We then predict water temperature using the DA-RNN proposed by (Qin et al., 2017 ) and two simple forecasting models (MLP and LSTM). The MLP was set with a single hidden layer consisted of 128 neurons for comparison. Similarly, we apply LSTM as the middle unit of RNN and set the hidden size to 64. All three prediction models predict only one time step into the future for comparison. We can measure the accuracy of imputation by comparing the prediction performance of different models after imputation of missing value sequences indirectly. Figure 6 clearly shows that the sequence processed by the proposed model significantly outperforms other baseline models in downstream prediction tasks. Table 2 shows the specific results of the prediction, from which, using the sequences imputed by the ATTN-GAN model as the input dataset achieves the best performance on all three prediction methods. Specifically, data processed with ATTN-GAN compared to SAITS as input to the prediction model reduces the MSE from 0.7552, 0.9038, 0.3832 to 0.6816, 0.8375, 0.3736 over MLP, LSTM, DA-RNN prediction methods. Table 2 MSE of water temperature prediction results by prediction models (MLP, LSTM, DA-RNN) trained on datasets imputed by different methods. MEAN LSAT KNN ARIMA Transformer SIATS ATTM-GAN MLP 0.9993 0.9838 1.1827 0.9703 0.7831 0.7552 0.6816 LSTM 1.1796 1.0528 1.2562 0.9896 0.9410 0.9038 0.8375 DA-RNN 0.7697 0.6701 0.5003 0.5333 0.5445 0.3832 0.3736 4. Discussion 4.1 Comparison of imputation models Due to the extreme instability of simple models and simply utilization of observed data, MEAN and LAST value imputation methods have very large errors in the imputation of missing values, which is very detrimental to downstream tasks such as time series forecasting. Statistical-based ARIMA(Box and Jenkins, 1968 ) simply memorizes the temporal trends of the sequence, which leads to large errors when there is a amount of missing data. Machine-learning-based KNN(Hudak et al., 2008 ) which uses the mean value of k nearest neighbors to fill missing values neither captures the temporal pattern in the time-series dataset nor considers the connection between missing values and observations, which leads to their imputation effects decreasing with the increase in the missing rate is rapidly becoming worse. Taking advantage of attention mechanism, the Transformer (Vaswani et al., 2017 ) is able to capture both temporal and spatial correlations in multivariate time series. However, As the rate of missing data increases, the amount of contextual information available to the model decreases, and the data interpolation performance decreases significantly. Similar to Transformer, SAITS (Du et al., 2022 ) methods based on attention mechanism can learn the temporal and spatial correlations in multivariate time series. The imputation performance is better than the Transformer models, because it learns the distribution of data from a weighted combination of two diagonally masked self-attention blocks through joint optimization to complete missing values imputation. The proposed model can better learn the distribution of data from the remaining small amount of data, leading to best imputation performance in general. This is because the generator of the Generative Adversarial Network may capture the distribution of data, and the discriminator forces the data generated by the generator to more closely approximate the distribution of the real data (Goodfellow et al., 2014 ). Furthermore, attention mechanisms in the generator and discriminator enable them to learn temporal and spatial correlations from multivariate time series, making easier for the generator to impute the missing values. It is worth noting that the architecture of the generator is very similar to the SAITS model, but the details are slightly different. This model utilizes the time lag information between observations as part of the input network and simplifies the weighted combinatorial block Although the combined attention module is simplified, due to the extra discriminator, the model is too large and require more time and memory to train than the baseline models. More fatally, as the time window lengthens and the number of variables increases, the volume of the model will explode, which causes great inconvenience to both training and the practical use of the model. The problem of how to produce lighter weight and higher performance models must be addressed. What’s more, fine-tuning the parameters takes a long time because the models contain a large number of hyperparameters compared to the baseline model. For various multivariate time series types, different parameters have highly different performance. Finally, the proposed model still does not fit ideally for time series with large data fluctuations. However, fitting the dramatic fluctuations of environmental data is important for agricultural production 4.2 Ablation study Table 3 performance comparison of ATTN-GAN (single), generator, ATTN-GAN in (MAE / RMSE / MRE%). Bold indicates best performance. Missing rate ATTN-GAN (single) generator ATTM-GAN 10% 0.2548/0.1604/0.1311 0.1292/0.1654/0.1298 0.1024 /0.1532/ 0.1092 20% 0.1941/0.1942/0.1402 0.1353/0.1763/0.1305 0.1115 / 0.1593 / 0.1144 30% 0.1868/0.2131/0.1329 0.1283/ 0.1569 / 0.1182 0.1189 /0.1736/ 0.1237 40% 0.2176/0.2347/0.1587 0.1425/0.2113/0.1403 0.1236 / 0.1908 / 0.1359 50% 0.2275/0.2439/0.1816 0.1508/0.2262/ 0.1399 0.1424/0.2012/0.1443 60% 0.2629/0.2721/0.1948 0.16070.2487/ 0.1461 0.1579 / 0.2314 / 0.1492 70% 0.2861/0.3036/0.2268 0.1877/0.2618/ 0.1484 0.1708 / 0.2539 / 0.1515 80% 0.3048/0.3214/0.2432 0.2127/0.2828/ 0.1727 0.2047 / 0.2688 / 0.1790 To further justify the proposed model, i.e., the effectiveness of attention blocks and adversarial generative architecture on time series missing value imputation, we conduct an ablation study for the proposed model. Table 3 shows the performance comparison of three models: ATTN-GAN (single) which is the proposed ATTN-GAN with single multi-head self-attention blocks, generator which comes from the proposed model and ATTN-GAN, and proposed ATTN-GAN. In comparison, the RMSE of generator decreases from 0.1942,0.2439,0.3214 to 0.1763, 0.2262, 0.2828 than ATTN-GAN (single) at missing rates of 20%, 50%, 80%, which proves that the weighted combination of two multi-head self-attentive blocks can capture the temporal and spatial pattern of multivariate time series much better than the single multi-head self-attentive blocks. Compared to the generator, ATTN-GAN which has an additional discriminator module lowers down RMSE by 9.66%, 11.1%, 4.95%. According to the results of ablation experience, it can be concluded that two combined multi-head self-attention blocks and one weighted combination block are beneficial to extract temporal and spatial patterns in time series, and the adversarial generative structure helps the model to fit the distribution of the original data. 5. Conclusion In this work, we propose an adversarial generative network-based model for imputation of missing values in multivariate time series. Compared with simple GAN, the generator and discriminator of the proposed ATTN-GAN model consist of multi-head attention blocks. According to the experimental results, the generative adversarial network architecture combined with the multi-head attention mechanism can learn the distribution of data and capture the spatio-temporal correlation well, thus successfully imputing the missing values of multivariate time series. Overall, ATTN-GAN outperforms the baseline in missing value input accuracy. For the forecasting of the downstream tasks, multivariate time series with a missing rate of 40% which imputed by baseline methods and ATTN-GAN were input to prediction models. Comparing the prediction results, the proposed model performs best on downstream prediction tasks. This also suggests that imputing missing values in multivariate time series before performing downstream tasks such as multivariate time forecasting may significantly improve forecast accuracy. However, the experiments were only carried out in the data aquaponics greenhouse environment, and it can only be concluded that the proposed model performed well on the missing data of the aquaponics greenhouse environment. For environmental data from other agricultural domains, such as fish farming, open fields, etc., the effect of the proposed model is unknown. And the proposed model is larger and takes longer to train than the baseline model. In future work, shrinking the model on the basis of ensuring the best input to reduce the training time is a worthwhile research direction. It will also be important to find out how applicable the model is to other areas of agriculture such as fish farming. Declarations Acknowledgement This research was financially supported by the 2115 Talent Development Program of China Agricultural University, National Key Research and Development Program of China: Sino-Malta Fund 2019: “Research and Demonstration of Real-time Accurate Monitoring System for Early-stage Fish in Recirculating Aquaculture System” (Aqua-Detector, Grant No. 2019YFE0103700), Overseas High-level Youth Talents Program (China Agricultural University, China, Grant No. 62339001), National Innovation Center for Digital Fishery, and Beijing Engineering and Technology Research Center for Internet of Things in Agriculture. The authors also appreciate constructive and valuable comments provided by reviewers. Ethical Approval Not applicable. Competing interests The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Authors’ contributions Keyang Zhong: Writing – Original Draft, Methodology, Model; Xueqian Sun : Writing – Original Draft, Methodology, Model; Gedi Liu : Writing-Reviewing & Editing; Yifeng Jiang: Writing-Reviewing & Editing; Yi Ouyang: Writing-Reviewing & Editing; Yang Wang : Writing-Reviewing & Editing, Supervising, Conceptualization. Funding This research was financially supported by the 2115 Talent Development Program of China Agricultural University, National Key Research and Development Program of China: Sino-Malta Fund 2019: “Research and Demonstration of Real-time Accurate Monitoring System for Early-stage Fish in Recirculating Aquaculture System” (Aqua-Detector, Grant No. 2019YFE0103700), Overseas High-level Youth Talents Program (China Agricultural University, China, Grant No. 62339001) Availability of data and materials The data and materials used in this article can be accessed after contacting authors. References Acar E, Dunlavy DM, Kolda TG, Mørup M (2010) Scalable Tensor Factorizations with Missing Data, in: Proceedings of the 2010 SIAM International Conference on Data Mining. Presented at the Proceedings of the 2010 SIAM International Conference on Data Mining, Society for Industrial and Applied Mathematics, pp. 701–712. https://doi.org/10.1137/1.9781611972801.61 Boomgard-Zagrodnik JP, Brown DJ (2022) Machine learning imputation of missing Mesonet temperature observations. Comput Electron Agric 192:106580. https://doi.org/10.1016/j.compag.2021.106580 Box GEP, Jenkins GM (1968) Some Recent Advances in Forecasting and Control. J Royal Stat Soc Ser C (Applied Statistics) 17:91–109. https://doi.org/10.2307/2985674 Cao W, Wang D, Li J, Zhou H, Li L, Li Y (2018) BRITS: Bidirectional Recurrent Imputation for Time Series. https://doi.org/10.48550/arXiv.1805.10572 Che Z, Purushotham S, Cho K, Sontag D, Liu Y (2018) Recurrent Neural Networks for Multivariate Time Series with Missing Values. Sci Rep 8:6085. https://doi.org/10.1038/s41598-018-24271-9 Chen H, Grant-Muller S, Mussone L, Montgomery F (2001) A Study of Hybrid Neural Network Approaches and the Effects of Missing Data on Traffic Forecasting. Neural Comput Appl 10:277–286. https://doi.org/10.1007/s521-001-8054-3 Choi E, Xiao C, Stewart WF, Sun J (2018) MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare Du W, Cote D, Liu Y (2022) SAITS: Self-Attention-based Imputation for Time Series. https://doi.org/10.48550/arXiv.2202.08516 Fung D (2006) Methods for the estimation of missing values in time series. Doctorates and Masters, Theses Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative Adversarial Networks. arXiv:1406.2661 [cs, stat]. Gupta M, Beheshti R (2020) Time-series Imputation and Prediction with Bi-Directional. https://doi.org/10.48550/arXiv.2009.08900 . Generative Adversarial Networks Hamzacebi C (2008) Improving artificial neural networks’ performance in seasonal time series forecasting. Inf Sci 178:4550–4559. https://doi.org/10.1016/j.ins.2008.07.024 Hastie T, Mazumder R, Lee J, Zadeh R (2014) Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares Hsieh T-J, Hsiao H-F, Yeh W-C (2011) Forecasting stock markets using wavelet transforms and recurrent neural networks: An integrated system based on artificial bee colony algorithm. Appl Soft Comput 11:2510–2525. https://doi.org/10.1016/j.asoc.2010.09.007 Huan J, Li H, Wu F, Cao W (2020) Design of water quality monitoring system for aquaculture ponds based on NB-IoT. Aquacult Eng 90:102088. https://doi.org/10.1016/j.aquaeng.2020.102088 Hudak AT, Crookston NL, Evans JS, Hall DE, Falkowski MJ (2008) Nearest neighbor imputation of species-level, plot-scale forest structure attributes from LiDAR data. Remote Sens Environ 112:2232–2245. https://doi.org/10.1016/j.rse.2007.10.009 Kaiser J (2014) Dealing with Missing Values in Data. J Syst Integr 5:42–51. https://doi.org/10.20470/jsi.v5i1.178 Kreindler D, Lumsden C (2006) Effects of the irregular sample and missing data in time series analysis. Nonlinear dynamics, psychology, and life sciences 10, 187–214 Luo Y, Cai X, Zhang Y, Xu J, Yuan X (2018) Multivariate time series imputation with generative adversarial networks, in: Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18. Curran Associates Inc., Red Hook, NY, USA, pp. 1603–1614 Luo Y, Zhang Y, Cai X, Yuan X (2019) E 2 GAN: End-to-End Generative Adversarial Network for Multivariate Time Series Imputation, in: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. Presented at the Twenty-Eighth International Joint Conference on Artificial Intelligence {IJCAI-19}, International Joint Conferences on Artificial Intelligence Organization, Macao, China, pp. 3094–3100. https://doi.org/10.24963/ijcai.2019/429 Ma J, Shou Z, Zareian A, Mansour H, Vetro A, Chang S-F (2019) CDSA: Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation. https://doi.org/10.48550/arXiv.1905.09904 Mazumder R, Hastie T, Tibshirani R (2010) Spectral Regularization Algorithms for Learning Large Incomplete Matrices 36 Qin Y, Song D, Chen H, Cheng W, Jiang G, Cottrell GW (2017) A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction, in: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Presented at the Twenty-Sixth International Joint Conference on Artificial Intelligence, International Joint Conferences on Artificial Intelligence Organization, Melbourne, Australia, pp. 2627–2633. https://doi.org/10.24963/ijcai.2017/366 Silva L, Zárate L (2014) A brief review of the main approaches for treatment of missing data. Intell Data Anal 18:1177–1198. https://doi.org/10.3233/IDA-140690 Tak S, Woo S, Yeo H (2016) IEEE Trans Intell Transport Syst 17:1762–1771. https://doi.org/10.1109/TITS.2016.2530312 . Data-Driven Imputation Method for Traffic Data in Sectional Units of Road Links Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention Is All You Need Wilson GT, Time Series Analysis: Forecasting and Control, 5th Edition, by, George EP, Box, Gwilym M, Jenkins GC, Reinsel, Ljung GM, 2015. Published by John Wiley and Sons Inc., Hoboken, New Jersey, pp. 712. ISBN: 978-1-118-67502-1: TIME SERIES ANALYSIS: FORECASTING AND CONTROL,5TH EDITION, by George E., Box P, Jenkins GM, Reinsel GC, Ljung GM (2016) 2015. Publis. J. Time Ser. Anal. 37, 709–711. https://doi.org/10.1111/jtsa.12194 Yoon J, Jordon J, van der Schaar M (2018) GAIN: Missing Data Imputation using Generative Adversarial Nets. arXiv:1806.02920 [cs, stat]. Yoon J, Zame WR, van der Schaar M (2019) IEEE Trans Biomed Eng 66:1477–1490. https://doi.org/10.1109/TBME.2018.2874712 . Estimating Missing Data in Temporal Data Streams Using Multi-Directional Recurrent Neural Networks Yu H-F, Rao N, Dhillon IS (2016) Temporal regularized matrix factorization for high-dimensional time series prediction, in: Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16. Curran Associates Inc., Red Hook, NY, USA, pp. 847–855 Zhang Y, Thorburn PJ (2021) A dual-head attention model for time series data imputation. Comput Electron Agric 189:106377. https://doi.org/10.1016/j.compag.2021.106377 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3209861","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":223173366,"identity":"1816b634-ff44-4059-a19d-f4962874d122","order_by":0,"name":"Keyang Zhong","email":"","orcid":"","institution":"National Innovation Center for Digital Fishery, China Agricultural University","correspondingAuthor":false,"prefix":"","firstName":"Keyang","middleName":"","lastName":"Zhong","suffix":""},{"id":223173367,"identity":"495108b2-2450-4820-990c-eea900d77560","order_by":1,"name":"Xueqian Sun","email":"","orcid":"","institution":"National Innovation Center for Digital Fishery, China Agricultural University","correspondingAuthor":false,"prefix":"","firstName":"Xueqian","middleName":"","lastName":"Sun","suffix":""},{"id":223173368,"identity":"72b415a5-01fd-4600-906f-43c1558d28dd","order_by":2,"name":"Gedi Liu","email":"","orcid":"","institution":"National Innovation Center for Digital Fishery, China Agricultural University","correspondingAuthor":false,"prefix":"","firstName":"Gedi","middleName":"","lastName":"Liu","suffix":""},{"id":223173369,"identity":"23417ba5-1bc6-4a36-a51b-3c4d791a73f3","order_by":3,"name":"Yifeng Jiang","email":"","orcid":"","institution":"National Innovation Center for Digital Fishery, China Agricultural University","correspondingAuthor":false,"prefix":"","firstName":"Yifeng","middleName":"","lastName":"Jiang","suffix":""},{"id":223173370,"identity":"5c5926e5-a38a-4b3f-9256-5e570c76803a","order_by":4,"name":"Yi Ouyang","email":"","orcid":"","institution":"National Innovation Center for Digital Fishery, China Agricultural University","correspondingAuthor":false,"prefix":"","firstName":"Yi","middleName":"","lastName":"Ouyang","suffix":""},{"id":223173371,"identity":"a2f9150d-f51d-457d-9b06-6cb91a85abb4","order_by":5,"name":"Yang Wang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABBklEQVRIiWNgGAWjYHACAyR2BYSSIEHLGZK1MLYRoYV/dvM2CcY2uzz5iORnD7/Os5M3OMB88DYPg10eLi0Sd46VAbUkFxveSDM3lt2WbLjhAFuyNQ9DcjFOa27kmAG1MCdunJ1gJi257QDjhgM8ZtI8DAcSG3DokIdoqQdqSf8mLTnngP2GA/zf8GoxgGg5nDhfOsdM8mPDgUSgLWx4tQC9UGzBcO544gb5N2XSDMeSk2ceZjO2nGOQjFOL3I3kjTcYyqoT5/cc3yb5o8bOtu9488MbbyrscGoBAhbpv2xAFx5gYGDmAfGZwQ7GrR6k5APDH2A4AA1l/IFX4SgYBaNgFIxUAABtN1gt+SSWzgAAAABJRU5ErkJggg==","orcid":"","institution":"National Innovation Center for Digital Fishery, China Agricultural University","correspondingAuthor":true,"prefix":"","firstName":"Yang","middleName":"","lastName":"Wang","suffix":""}],"badges":[],"createdAt":"2023-07-27 12:59:22","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3209861/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3209861/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":41191692,"identity":"c7409f8d-cacd-4b55-ab8d-98c4e9dac8ea","added_by":"auto","created_at":"2023-08-07 15:59:57","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":60966,"visible":true,"origin":"","legend":"\u003cp\u003eThe architecture of ATTN-GAN.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-3209861/v1/5a524651917cdadf6b9bea95.png"},{"id":41191693,"identity":"d9385f58-75c2-4c46-9c60-c72bd27e90a0","added_by":"auto","created_at":"2023-08-07 15:59:57","extension":"jpeg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":162705,"visible":true,"origin":"","legend":"\u003cp\u003eThe architecture of generator.\u003c/p\u003e","description":"","filename":"floatimage2.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-3209861/v1/3274decb191c9d54936984ff.jpeg"},{"id":41191691,"identity":"db8a7ad3-eba4-4164-ad37-ae3deca36101","added_by":"auto","created_at":"2023-08-07 15:59:57","extension":"jpeg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":64009,"visible":true,"origin":"","legend":"\u003cp\u003eThe architecture of discriminator.\u003c/p\u003e","description":"","filename":"floatimage3.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-3209861/v1/92c3b2e921ca9389a734e402.jpeg"},{"id":41191695,"identity":"95edcc77-85fc-4ab7-9388-1e643d8d11a8","added_by":"auto","created_at":"2023-08-07 15:59:57","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":1017529,"visible":true,"origin":"","legend":"\u003cp\u003eThe performance in（MAE/RMSE/MSE）of the baseline model and the proposed model with increasing data missing rate.\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-3209861/v1/03830385f76685bf32a042c4.png"},{"id":41191696,"identity":"05606eff-39a4-4282-bec1-dac702108e10","added_by":"auto","created_at":"2023-08-07 15:59:58","extension":"jpeg","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":246762,"visible":true,"origin":"","legend":"\u003cp\u003ePerformance of the baseline model and ATTN-GAN on imputation of four consecutive missing values.\u003c/p\u003e","description":"","filename":"floatimage6.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-3209861/v1/709b0466292ed390cabe1e70.jpeg"},{"id":41191694,"identity":"f1d79c4d-30ec-4f4c-be7e-6097ef37c223","added_by":"auto","created_at":"2023-08-07 15:59:57","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":35275,"visible":true,"origin":"","legend":"\u003cp\u003eMSE of water temperature prediction results by different prediction models trained on datasets imputed by different methods.\u003c/p\u003e","description":"","filename":"floatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-3209861/v1/9817a63873d3e0c39f2c140b.png"},{"id":41349284,"identity":"c8051a32-8e14-4559-b694-27f779bd3724","added_by":"auto","created_at":"2023-08-10 07:37:32","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":943056,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3209861/v1/09e81a16-89bc-452b-8fd6-0e69b303f068.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Attention-based generative adversarial networks for aquaponics environment time series data imputation","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003ePrediction and classification of multivariate time series are tasks that are commonly utilized in finance (Che et al., \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2018\u003c/span\u003e; Hsieh et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2011\u003c/span\u003e), transportation(Yu et al., \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2016\u003c/span\u003e), healthcare (Choi et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2018\u003c/span\u003e), and agricultural facilities (Boomgard-Zagrodnik and Brown, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). In agricultural applications, sensors are required to collect environmental data to monitor the production environment. Complete data in agricultural production are crucial for the analysis and control of environment and will affect agricultural production (Boomgard-Zagrodnik and Brown, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Huan et al., \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2020\u003c/span\u003e; Zhang and Thorburn, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). For example: during crop growth, sensors need to monitor data on soil moisture, salinity and other parameters; in fish farming, sensors need to collect water quality data and meteorological data. Then during the high-frequency collection of environmental data, it is inevitable that some data will be lost due to interruptions and failures in the data collection and transmission process. The prediction and classification of time series requires a complete data set and missing values in the time series data can lead to biased estimations. Therefore, how to deal with missing values in multivariate time series becomes a key topic.\u003c/p\u003e \u003cp\u003eThe methods for dealing with missing values can be divided into two categories: one is the case deletion method, which removes missing elements directly (Kaiser, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2014\u003c/span\u003e; Silva and Z\u0026aacute;rate, \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2014\u003c/span\u003e) and ignores certain data information and impose bias, making further analysis challenging. The second is to impute missing values that is superior to case deletion because imputation can make use of information from some observations. Existing missing value imputation methods can be classified into five categories, including simple imputation methods, statistical analysis methods, machine learning-based methods, attention mechanisms, and generative adversarial network.\u003c/p\u003e \u003cp\u003eThe simple imputation methods such as mean imputation, and last observation imputation, make use of local information but ignore significant long-term temporal trends. Statistical analysis methods like autoregressive method (Box and Jenkins, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e1968\u003c/span\u003e) and smooth interpolation utilizes the historical information of the time series and reconstruct missing values by fitting a smooth curve (Fung, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2006\u003c/span\u003e; Hamzacebi, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2008\u003c/span\u003e; Kreindler and Lumsden, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2006\u003c/span\u003e; Wilson, 2016). However, these methods demand the time series remain stationary and loss sight of the spatial connection between multivariate time series. Machine learning-based imputation methods are useful for multivariate time series imputation. K-nearest neighbors (KNN) (Hudak et al., \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2008\u003c/span\u003e; Tak et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2016\u003c/span\u003e) uses the mean value of k nearest neighbors to fill missing values. Mmatrix factorization (Acar et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2010\u003c/span\u003e; Hastie et al., \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2014\u003c/span\u003e; Mazumder et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2010\u003c/span\u003e) decompose the incomplete data set into low-rank matrices and use the product of these two matrices to imputation the missing components. Multilayer perceptron (MLP) (Chen et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2001\u003c/span\u003e) estimates missing values by stacking multiple layers of neurons. Nevertheless, these methods neither capture temporal patterns in time series datasets nor take into account the link between missing values and observations. Cao et al. (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2018\u003c/span\u003e) and Yoon et al. (\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e2019\u003c/span\u003e) used a bidirectional recurrent neural network to impute missing values in a healthcare dataset. Che et al. (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2018\u003c/span\u003e) proposed a variant of Gated Recurrent Unit (GRU-D) which takes the advantage of last observed value and mean value to handle missing data in time series classification problems. However, recurrent neural networks are difficult to capture spatial correlations in multivariate time series, and the imputation performance is not very good.\u003c/p\u003e \u003cp\u003eAttention mechanisms (Vaswani et al., \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2017\u003c/span\u003e) are believed to be able to capture both temporal and spatial correlations in multivariate time series. Ma et al.(2019) jointly applied cross-dimensional self-attention from three dimensions (time, location, and measurement) to fill missing values in geotagged data. Zhang and Thorburn (\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) proposed a dual-head sequence-to-sequence imputation model, which includes an encoder with two gated recurrent units and an attention mechanism in the encoder's output layer and achieved good performance in a water quality information system dataset. Du et al. (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2022\u003c/span\u003e) proposed a method (SAITS) for multivariate time series missing value imputation based on the self-attention mechanism. SAITS learns the distribution of data from a weighted combination of two diagonally masked self-attention blocks through joint optimization to complete missing values imputation.\u003c/p\u003e \u003cp\u003eMany researchers in the field of multivariate time series processing have also developed methods to impute missing data based on generative adversarial networks (Goodfellow et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2014\u003c/span\u003e). The first to propose a GAN-based multivariate time series missing value imputation approach is Luo et al. (\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). They adopted a modified gate recursive unit (GRUI) in GAN to model the irregularity of incomplete time series. GRUI takes into account non-fixed time lags and attenuates the influence of past observations determined by the time lags. Following that, they suggested an E2GAN (end-to-end generative model) (Luo et al., \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). E2GAN inputs random noise to the generator, learns the mapping of random noise to the distribution of the training data set, and simplifies the imputation process of the time series. Yoon et al. (\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2018\u003c/span\u003e) improves upon the standard GAN architecture by feeding the discriminator with generated data and providing hinting mechanism information, which assures the it generates samples based on the actual underlying distribution. Gupta and Beheshti (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2020\u003c/span\u003e) proposed model (Bi-GAN) uses a bidirectional recurrent network in a generative adversarial setting which makes use of RNNs' strength in capturing temporal correlations in time series and GAN designs' superiority in fitting data distributions.\u003c/p\u003e \u003cp\u003eThe purpose of this study was to explore the best model for missing values imputation on aquaponics greenhouse environment data. We explore the effectiveness of the proposed model in capturing temporal and spatial correlations in multivariate time series by conducting comparative experiments. This paper provides a model for missing data imputation based on a generative adversarial network architecture that employs a multi-head attention mechanism in both the generator and the discriminator. It combines the superior performance of attention mechanisms in capturing spatiotemporal correlations and the superior ability of GANs to fit data distributions. The generator and the discriminator are not identical. The generator imputes missing values using two weighted link multi-head attention modules, whereas the discriminator employs a single multi-head attention module to discriminate between imputed and observed values.\u003c/p\u003e \u003cp\u003eOverall, the following are the study's major contributions:\u003c/p\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eA multi-head attention-based generative adversarial network imputation algorithm was proposed, which can effectively learn the distribution of data to achieve quite accurate imputation.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eThe proposed model combines an attention mechanism with a GAN architecture to capture the temporal and spatial correlation of multivariate time series to improve the estimation of missing values.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eThe model was tested on datasets gathered from aquaponics greenhouse environments and results show that it outperforms state-of-the-art methods in both data imputation accuracy and downstream prediction tasks.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e"},{"header":"2. Materials and methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Dataset\u003c/h2\u003e \u003cp\u003eA real-world dataset was collected to validate this model. The dataset was collected from an aquaponics greenhouse practicing base in Tongzhou, Beijing. The base's sensors recorded eight characteristics, including soil moisture, water temperature, and air humidity. The sensors capture data every two hours. From July 3, 2020, to January 2, 2021, a total of 2208 pieces of data were gathered. To validate the suggested model, the first 80% of the data was separated into a training set and the remaining 20% was separated into a test set. The data includes around 8% missing values due to sensor failure and other factors.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Problem Formulation\u003c/h2\u003e \u003cp\u003eFirstly, the problem formulation and some necessary preliminaries were presented. A collection of multivariate time series with \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(D\\)\u003c/span\u003e\u003c/span\u003e dimensions and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(T\\)\u003c/span\u003e\u003c/span\u003e steps was given:\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e\n$$X =\\left\\{{x}_{1},{x}_{2},\\dots ,{x}_{t},\\dots {x}_{T}\\right\\}\\in {\\mathbb{R}}^{T\\times D}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x}_{t}\\)\u003c/span\u003e\u003c/span\u003e is the \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e-th observation of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(X\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x}_{t}=\\left\\{{x}_{t}^{1},{x}_{t}^{2},\\dots ,{x}_{t}^{d}, \\dots ,{x}_{t}^{D}\\right\\}\\in {\\mathbb{R}}^{D}\\)\u003c/span\u003e\u003c/span\u003e. Except some missing values, some components were artificially masked randomly. And \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({X}_{init}\\)\u003c/span\u003e\u003c/span\u003e is used to represent raw data that initially has missing values.\u003c/p\u003e \u003cp\u003eThe location matrix \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(M\\)\u003c/span\u003e\u003c/span\u003e of missing or masked values as well as the time delays matrix \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\delta\\)\u003c/span\u003e\u003c/span\u003e of two adjacent values were recorded:\u003cdiv id=\"Equb\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equb\" name=\"EquationSource\"\u003e\n$${M}_{t}^{d}=\\left\\{\\begin{array}{c}1, if {x}_{t}^{d} is observation\\\\ 0, if {x}_{t}^{d} is missing or masked\\end{array}\\right.$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equc\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equc\" name=\"EquationSource\"\u003e\n$${\\delta }_{t}^{d}=\\left\\{\\begin{array}{c}0,\\\\ {t}_{i}-{t}_{i-1},\\\\ {\\delta }_{{t}_{i-1}}^{d}+{t}_{i}-{t}_{i-1},\\end{array}\\begin{array}{c}i==0\\\\ { M}_{{t}_{i-1}}^{d}==1\\\u0026amp;i\u0026gt;0\\\\ { M}_{{t}_{i-1}}^{d}==0\\\u0026amp;i\u0026gt;0\\end{array}\\right.$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x}_{t}^{d}\\)\u003c/span\u003e\u003c/span\u003e is the \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e-th step observation of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(d\\)\u003c/span\u003e\u003c/span\u003e-th dimension variable in \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(X\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(M{\\in \\mathbb{R}}^{T\\times D}\\)\u003c/span\u003e\u003c/span\u003e, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\delta {\\in \\mathbb{R}}^{T\\times D}\\)\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eTo distinguish artificially masked values and originally missing values, the indicating matrix \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(I\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({M}^{{\\prime }}\\)\u003c/span\u003e\u003c/span\u003e are introduced.\u003cdiv id=\"Equd\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equd\" name=\"EquationSource\"\u003e\n$${I}_{t}^{d}=\\left\\{\\begin{array}{c}1, if {X}_{t}^{d} is masked\\\\ 0, otherwise\\end{array}\\right.$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Eque\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Eque\" name=\"EquationSource\"\u003e\n$${{M}^{{\\prime }}}_{t}^{d}=\\left\\{\\begin{array}{c}1, if {x}_{t}^{d} is observation or masked\\\\ 0, if {x}_{t}^{d} is missing\\end{array}\\right.$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equf\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equf\" name=\"EquationSource\"\u003e\n$$H=h{M}^{{\\prime }}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(H\\)\u003c/span\u003e\u003c/span\u003eis Hint Matrix which generated from \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({M}^{{\\prime }}\\)\u003c/span\u003e\u003c/span\u003e, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(h\\)\u003c/span\u003e\u003c/span\u003e is a random number.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Approach\u003c/h2\u003e \u003cdiv id=\"Sec6\" class=\"Section3\"\u003e \u003ch2\u003e2.3.1 General Framework\u003c/h2\u003e \u003cp\u003eFigure\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e illustrates the general architecture of ATTN-GAN. A generator (G) and a discriminator (D) comprise the generative adversarial network (GAN). The generator which was feed with incomplete data and some other information imputed the missing values of multivariate time series. And the discriminator aims to predict the matrix \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(M\\)\u003c/span\u003e\u003c/span\u003e which indicates whether the element is generated or observed. Next, the details of the multi-head self-attention blocks was described, and generator and discriminator that consist mainly of multi-head self-attention blocks are also presented detailly.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section3\"\u003e \u003ch2\u003e2.3.2 Multi-head Self-attention Blocks\u003c/h2\u003e \u003cp\u003eVaswani et al. (\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2017\u003c/span\u003e) proposed a traditional self-attention mechanism to solve the language translation task. It is now widely used in tasks such as sequence modeling and computer vision. The attention function can be described as the mapping of a query and a set of key-value pairs to an output, where the query and key are of dimension \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({d}_{k}\\)\u003c/span\u003e\u003c/span\u003e and the value is of dimension \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({d}_{v}\\)\u003c/span\u003e\u003c/span\u003e. In practice, the attention function is computed simultaneously on a set of queries packed together into a matrix \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(Q\\)\u003c/span\u003e\u003c/span\u003e. The keys and values are also packed together into matrix \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(K\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(V\\)\u003c/span\u003e\u003c/span\u003e. The output matrix of the attention function is computed as follows:\u003cdiv id=\"Equg\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equg\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}Attention\\left(Q,K,V\\right)=softmax\\left(\\frac{Q{K}^{T}}{\\sqrt{{d}_{k}}}\\right)V\\#\\left(1\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eMulti-head self-attention allows the model to jointly focus information from different representations subspaces at different locations. The multi-head self-attention means that \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(h\\)\u003c/span\u003e\u003c/span\u003e different output matrixes of the attention function are mapped from the \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(Q,K,V\\)\u003c/span\u003e\u003c/span\u003egroups.\u003cdiv id=\"Equh\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equh\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}MultiHead\\left(Q,K,V\\right)=Concat\\left(hea{d}_{1},\\dots , hea{d}_{h}\\right){W}^{o}\\#\\left(2\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equi\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equi\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}Where hea{d}_{i}=Attention\\left(Q{W}_{i}^{Q},K{W}_{i }^{K},V{W}_{i}^{V}\\right)\\#\\left(3\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(h\\)\u003c/span\u003e\u003c/span\u003e is the number of heads, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({W}_{i}^{Q}{\\in \\mathbb{R}}^{{d}_{model}\\times {d}_{k}}, { W}_{i }^{K}{\\in \\mathbb{R}}^{{d}_{model}\\times {d}_{k}},{ W}_{i}^{V}{\\in \\mathbb{R}}^{{d}_{model}\\times {d}_{v}}, {W}^{o}{\\in \\mathbb{R}}^{h{d}_{v}\\times {d}_{model}}\\)\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eIn this work, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(h=3,{d}_{k}={d}_{v}={d}_{model}=64\\)\u003c/span\u003e\u003c/span\u003e were employed in discriminator; \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(h=3,{d}_{k}={d}_{v}=64,{d}_{model}=128\\)\u003c/span\u003e\u003c/span\u003e were employed in generator. Multi-head self-attention blocks include residual mode, which is followed by feed Forward and Layer Norm module activities. Here, these are generalized with the symbol \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(MultiHeadBlocks\\left(\\right)\\)\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section3\"\u003e \u003ch2\u003e2.3.3 Generator\u003c/h2\u003e \u003cp\u003eThe Generator Network architecture is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. The generator consists of two combined Multi-head Self-attention blocks and a weighted combine block. The goal of the generator is to fit the distribution of the complete data, which in turns generates the completed multivariate time series. To utilize the limited information as much as possible, incomplete time series, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(M\\)\u003c/span\u003e\u003c/span\u003e matrix, and δ matrix were fed into the generator. Thus, the generator was described as a mapping:\u003cdiv id=\"Equj\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equj\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}G\\left(X,M,\\delta \\right)=\\stackrel{\\sim}{X}\\#\\left(4\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eThen the missing values of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(X\\)\u003c/span\u003e\u003c/span\u003e was replaced with the following equation:\u003cdiv id=\"Equk\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equk\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}\\widehat{X}=X\\odot {M}^{{\\prime }}+\\left(1-{M}^{{\\prime }}\\right)\\odot \\stackrel{\\sim}{X}\\#\\left(5\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\odot\\)\u003c/span\u003e\u003c/span\u003e is element-wise multiplication; \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\stackrel{\\sim}{X}\\)\u003c/span\u003e\u003c/span\u003e is the generated data vector of G; \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{X}\\)\u003c/span\u003e\u003c/span\u003e is the imputed data vector, that is, the matrix obtained by replace the missing values of the original sequence with the corresponding components generated by G.\u003c/p\u003e \u003cp\u003eThe first Multi-head Self-attention blocks take the time lag vector \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\delta\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(X\\)\u003c/span\u003e\u003c/span\u003e as inputs. Firstly, the concatenated \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\delta\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(X\\)\u003c/span\u003e\u003c/span\u003e pass through the encoding layer, adding the positional encoding as the input of Multi-head Self-attention blocks. Then \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\stackrel{\\sim}{X}}_{1}\\)\u003c/span\u003e\u003c/span\u003e was got from the first Multi-head Self-attention blocks following a fully connected layer. The following equations show the data transfer process:\u003cdiv id=\"Equl\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equl\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}c=\\left(concat\\left(X,\\delta \\right){W}_{c}+{b}_{c}\\right)+p\\#\\left(6\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equm\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equm\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}z={\\left\\{MultiHeadBlocks\\left(c\\right)\\right\\}}^{N}\\#\\left(7\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equn\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equn\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}{\\stackrel{\\sim}{X}}_{1}=z*{W}_{z}+{b}_{z}\\#\\left(8\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({W}_{c}{\\in \\mathbb{R}}^{\\left(2*{d}_{k}\\right)\\times {d}_{model}}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({W}_{z}{\\in \\mathbb{R}}^{{d}_{model}\\times {d}_{k}}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({b}_{c}{\\in \\mathbb{R}}^{2*{d}_{k}}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({b}_{z}{\\in \\mathbb{R}}^{{d}_{k}}\\)\u003c/span\u003e\u003c/span\u003e are parameters, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\left\\{\\right\\}}^{N}\\)\u003c/span\u003e\u003c/span\u003e is stacking \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(N\\)\u003c/span\u003e\u003c/span\u003e Multi-head Self-attention blocks, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(p\\)\u003c/span\u003e\u003c/span\u003e is positional encoding:\u003cdiv id=\"Equo\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equo\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}\\left\\{\\begin{array}{c}P{E}_{\\left(pos,2i\\right)}=\\text{sin}\\left(\\frac{pos}{1000{0}^{2i/{d}_{model}}}\\right)\\\\ P{E}_{\\left(pos,2i+1\\right)}=\\text{cos}\\left(\\frac{pos}{1000{0}^{2i/{d}_{model}}}\\right)\\end{array}\\right.\\#\\left(9\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(pos\\)\u003c/span\u003e\u003c/span\u003e is the time-step position, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e is the dimension.\u003c/p\u003e \u003cp\u003eThe second Multi-head Self-attention blocks takes the \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(M\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\stackrel{\\sim}{X}}_{1}\\)\u003c/span\u003e\u003c/span\u003e as input. Similar to the first Multi-head Self-attention blocks, the concatenated \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(M\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\stackrel{\\sim}{X}}_{1}\\)\u003c/span\u003e\u003c/span\u003e pass through the encoding layer, adding the positional encoding as the input of Multi-head Self-attention blocks:\u003cdiv id=\"Equp\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equp\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}\\alpha =\\left(concat\\left({\\stackrel{\\sim}{X}}_{1},M\\right){W}_{\\alpha }+{b}_{\\alpha }\\right)+p\\#\\left(10\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equq\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equq\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}\\beta ={\\left\\{MultiHeadBlocks\\left(\\alpha \\right)\\right\\}}^{N}\\#\\left(11\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equr\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equr\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}{\\stackrel{\\sim}{X}}_{2}=\\beta *{W}_{\\beta }+{b}_{\\beta }\\#\\left(12\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eTo obtain a better learned representation \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\stackrel{\\sim}{X}\\)\u003c/span\u003e\u003c/span\u003e, a weighted combination block is designed to dynamically weight \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\stackrel{\\sim}{X}}_{1}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\stackrel{\\sim}{X}}_{2}\\)\u003c/span\u003e\u003c/span\u003e according to temporal correlation and missing information:\u003cdiv id=\"Equs\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equs\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}\\widehat{A}=\\frac{1}{h}\\sum _{i}^{h}{A}_{i}\\#\\left(13\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equt\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equt\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}\\gamma =sigmoid\\left(\\widehat{A}{W}_{\\gamma }+{b}_{\\gamma }\\right)\\#\\left(14\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equu\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equu\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}\\widehat{X}=\\left(1-\\gamma \\right)\\odot {\\stackrel{\\sim}{X}}_{1}+ \\gamma \\odot {\\stackrel{\\sim}{X}}_{2}\\#\\left(15\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({W}_{\\alpha }{\\in \\mathbb{R}}^{\\left(2*{d}_{k}\\right)\\times {d}_{model}}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({W}_{\\beta }{\\in \\mathbb{R}}^{{d}_{model}\\times {d}_{k}}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({W}_{\\gamma }{\\in \\mathbb{R}}^{T\\times {d}_{k}}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({b}_{\\alpha }{\\in \\mathbb{R}}^{2*{d}_{k}}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({b}_{\\beta }{\\in \\mathbb{R}}^{{d}_{k}}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({b}_{\\gamma }{\\in \\mathbb{R}}^{{d}_{k}}\\)\u003c/span\u003e\u003c/span\u003e are parameters, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{A}{\\in \\mathbb{R}}^{T\\times T}\\)\u003c/span\u003e\u003c/span\u003e is averaged from attention weights \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(A\\)\u003c/span\u003e\u003c/span\u003e output by the last layer of the second Multi-head Self-attention block.\u003c/p\u003e \u003cp\u003eSince the purpose of the generator is to learn the distribution of complete multivariate time series, a squared error loss was added to the loss function of the generator. The loss function is defined as follows:\u003cdiv id=\"Equv\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equv\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}{L}_{G}=\\lambda \\left|\\right|X\\odot {M}^{{\\prime }}-\\widehat{X}\\odot {M}^{{\\prime }}|{|}_{2}+\\eta ||X\\odot I-\\widehat{X}\\odot I|{|}_{2}-{L}_{D}\\#\\left(16\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\lambda\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\eta\\)\u003c/span\u003e\u003c/span\u003e are hyper-parameters that controls the weight of the discriminator\u0026rsquo;s loss and the squared error loss between generated \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{X}\\)\u003c/span\u003e\u003c/span\u003e and original \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(X\\)\u003c/span\u003e\u003c/span\u003e, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({L}_{D}\\)\u003c/span\u003e\u003c/span\u003e is the loss of discriminator.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section3\"\u003e \u003ch2\u003e2.3.4 Discriminator\u003c/h2\u003e \u003cp\u003eAs in regular GAN framework, a discriminator D used to train G was introduced in this method. The Discriminator Network architecture of proposed method is shown at Fig.\u0026nbsp;3. Unlike the standard GAN whose output is either to judge the input is complete real or complete fake (generated by G), in the proposed model the discriminator attempts to distinguish which components are real (observed) or fake (imputed by G) -this amounts to predicting the matrix \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(M\\)\u003c/span\u003e\u003c/span\u003e. In simple terms, the output of D is the probability that each component is real. A hint mechanism was introduced (Yoon et al., \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). The hint mechanism is a random variable, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(H\\)\u003c/span\u003e\u003c/span\u003e ,which depend on each sample\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\((X,M)\\)\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eThe function of D is as follows:\u003cdiv id=\"Equw\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equw\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}D\\left(\\widehat{X},H\\right)=P\\#\\left(17\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere each component in \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(P\\)\u003c/span\u003e\u003c/span\u003e represents the probability of the component is real (not generated by G).\u003c/p\u003e \u003cp\u003eThe Multi-head Self-attention blocks take the Hint Matrix and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{X}\\)\u003c/span\u003e\u003c/span\u003e, which consists of observed components and generated components. The concatenated Hint Matrix \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(H\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{X}\\)\u003c/span\u003e\u003c/span\u003e pass through the encoding layer, adding the position encoding as the input of Multi-head Self-attention blocks. Multi-head self-attention blocks are followed by linear layers and activation functions, and then the probability matrix \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(P\\)\u003c/span\u003e\u003c/span\u003e is got. The following equations show the data transfer process:\u003cdiv id=\"Equx\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equx\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}e=\\left(concat\\left(\\widehat{X},H\\right){W}_{e}+{b}_{e}\\right)+p\\#\\left(18\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equy\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equy\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}g=\\{MultiHeadBlocks\\left(e\\right){\\}}^{N}\\#\\left(19\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equz\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equz\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}P=Sigmoid\\left(\\left(g*{W}_{g}+{b}_{g}\\right)\\right)\\#\\left(20\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere the \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({W}_{e}{\\in \\mathbb{R}}^{\\left(2*{d}_{k}\\right)\\times {d}_{model}}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({W}_{g}{\\in \\mathbb{R}}^{{d}_{model}\\times {d}_{k}}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({b}_{e}{\\in \\mathbb{R}}^{2*{d}_{k}}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({b}_{g}{\\in \\mathbb{R}}^{{d}_{k}}\\)\u003c/span\u003e\u003c/span\u003e are parameters.\u003c/p\u003e \u003cp\u003eThe loss function of the discriminator is as follows:\u003cdiv id=\"Equaa\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equaa\" name=\"EquationSource\"\u003e\n$$\\begin{array}{c}{L}_{D}=|\\left|P-M\\right|{|}_{2}\\#\\left(21\\right)\\end{array}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e2.4 Evaluation and baseline\u003c/h2\u003e \u003cdiv id=\"Sec11\" class=\"Section3\"\u003e \u003ch2\u003e2.4.1 Metrics\u003c/h2\u003e \u003cp\u003eTo demonstrate the effectiveness of proposed methods for multivariate time series missing values imputation, we employ mean absolute error (MAE) root mean squared error (RMSE), and mean absolute percentage error (MAPE) to evaluate our methods. Since the missing values of the original missing data are not known, the imputation performance of the artificially masked parts is used to evaluate each model. Therefore the evaluation metrics are defined as follows:\u003cdiv id=\"Equab\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equab\" name=\"EquationSource\"\u003e\n$$MAE= \\frac{{\\sum }_{d}^{D}{\\sum }_{t}^{T}|\\left(\\widehat{{x}_{t}^{d}}-{x}_{ini{t}_{t}}^{d}\\right)\\odot {I}_{t}^{d}|}{{\\sum }_{d}^{D}{\\sum }_{t}^{T}{I}_{t}^{d}}$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equac\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equac\" name=\"EquationSource\"\u003e\n$$RMSE= \\sqrt{\\frac{{\\left(\\left(\\widehat{{x}_{t}^{d}}-{x}_{ini{t}_{t}}^{d}\\right)\\odot {I}_{t}^{d}\\right)}^{2}}{{\\sum }_{d}^{D}{\\sum }_{t}^{T}{I}_{t}^{d}}}$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equad\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equad\" name=\"EquationSource\"\u003e\n$$MRE= \\frac{{\\sum }_{d}^{D}{\\sum }_{t}^{T}|\\left(\\widehat{{x}_{t}^{d}}-{x}_{ini{t}_{t}}^{d}\\right)\\odot {I}_{t}^{d}|}{{\\sum }_{d}^{D}{\\sum }_{t}^{T}|\\widehat{{x}_{t}^{d}} \\odot {I}_{t}^{d}|}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section3\"\u003e \u003ch2\u003e2.4.2 Baseline\u003c/h2\u003e \u003cp\u003eThe baseline model compared with the proposed model is as follows:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eMEAN: For the corresponding feature, missing data was replaced with the weekly mean.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eLAST: Missing values are replaced with the last observation\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eARIMA(Box and Jenkins, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e1968\u003c/span\u003e): The imputation of missing values was implement by using an autoregressive moving average model for time series prediction\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eKNN(Hudak et al., \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2008\u003c/span\u003e): The missing values are imputed based on the k nearest neighbor samples.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eTransformer: The basic component of the proposed model is the multi-head self-attention proposed by Vaswani et al. (\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2017\u003c/span\u003e). Here the basic Transformer model was used for comparison.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eSAITS(Du et al., \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2022\u003c/span\u003e): It proposed two diagonally-masked self-attention(DMSA) blocks to learn the temporal and feature correlations between missing and observed values, resulting in accurate imputation of missing values.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"3. Results","content":"\u003cdiv id=\"Sec14\" class=\"Section2\"\u003e\n \u003ch2\u003e3.1 Model comparison of imputation results\u003c/h2\u003e\n \u003cdiv class=\"gridtable\"\u003e\u0026nbsp;\u0026nbsp;\u003ctable id=\"Tab1\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eImputation performance of the proposed method and baseline model in (MAE / RMSE / MRE). Bold indicates best performance.\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMissing rate\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMEAN\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eLSAT\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eKNN\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eARIMA\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eTransformer\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSIATS\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eATTM-GAN\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e10%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.4807/0.6573/0.4583\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.4319/0.4959/0.4694\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.2626/0.4125/0.3050\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.2794/0.3858/0.3084\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.2274/0.3085/0.2112\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.1221/\u003cstrong\u003e0.1476\u003c/strong\u003e/0.1201\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.1024\u003c/strong\u003e/0.1532/\u003cstrong\u003e0.1092\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e20%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.567/0.696/0.4944\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.4853/0.5014/0.4743\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.3297/0.4738/0.3843\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.2817/0.3002/0.3118\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.2317/0.3302/0.2213\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.1301/0.1603/0.1314\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.1115\u003c/strong\u003e/\u003cstrong\u003e0.1593\u003c/strong\u003e/\u003cstrong\u003e0.1144\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e30%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.5819/0.6924/0.5475\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.5522/0.5958/0.5323\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.4189/0.5683/0.4932\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.3141/0.4554/0.3521\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.2413/0.3534/0.2345\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.1383/\u003cstrong\u003e0.1637\u003c/strong\u003e/0.142\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.1189\u003c/strong\u003e/0.1736/\u003cstrong\u003e0.1237\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e40%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.6073/0.7308/0.6611\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.6077/0.6366/0.6618\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.4828/0.6486/0.5675\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.3388/0.4987/0.3806\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.2688/0.3897/0.2587\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.1458/0.2508/0.1503\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.1236\u003c/strong\u003e/\u003cstrong\u003e0.1908\u003c/strong\u003e/\u003cstrong\u003e0.1359\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e50%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.6898/0.7733/0.6753\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.6445/0.6955/0.7046\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.5421/0.6903/0.6368\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.3730/0.5509/0.4206\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.3030/0.4039/0.2816\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.1703/0.2720/0.1609\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.1424/0.2012/0.1443\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e60%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.7432/0.8248/0.7816\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.69690.7711/0.7680\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.5897/0.7364/0.6954\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.4094/0.5941/0.4648\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.3490/0.4441/0.3048\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.1955/0.3016/0.1701\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.1579\u003c/strong\u003e/\u003cstrong\u003e0.2314\u003c/strong\u003e/\u003cstrong\u003e0.1492\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e70%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.8093/0.8696/0.8494\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.73710.8266/0.8144\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.6097/0.7864/0.7454\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.4287/0.6061/0.4868\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.3871/0.5036/0.3268\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.2017/0.3214/0.1820\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.1708\u003c/strong\u003e/\u003cstrong\u003e0.2539\u003c/strong\u003e/\u003cstrong\u003e0.1515\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e80%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.822/0.9072/0.9713\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.7466/0.8737/0.9483\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.6801/0.8545/0.7986\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.4927/0.6949/0.5610\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.4237/0.5514/0.3532\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.2137/0.3398/0.2041\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.2047\u003c/strong\u003e/\u003cstrong\u003e0.2688\u003c/strong\u003e/\u003cstrong\u003e0.1790\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003c/div\u003e\n \u003cp\u003eTable 1 provides the global experimental results of ATTN-GAN and all baseline methods. When the evaluation index is RMSE and the data missing rate is 10% and 30%, the imputation performance of ATTN-GAN is slightly worse than that of SAITS, but obtain excellent imputation results. What\u0026rsquo;s more, as Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e shows, ATTN-GAN achieves the best results under any other evaluation metrics. In order to illustrate the imputation effect of the proposed model in the cases of low, medium, and high data missing rate, we only briefly describe the cases of missing rate of 20%, 50%, and 80%. Compared to the optimal baseline results, it lowers down RMSE by 0.623%, 26.02%, 20.89% over the 20%, 50% and 80% missing rate. Specifically, the performance of MEAN is worst upon most instances, also the LAST has a very large error. This is because that MEAN and LAST is simple imputation methods which Simply take advantage of information from the observed data to impute missing values in time series. Machine learning-based KNN and statistics based ARIMA show a big improvement over the simple methods, especially when the missing rate is small. Transformer, SIATS, and ATTN-GAN based on attention mechanism further optimize the performance of missing values imputation. Obviously, compared to Transformer, ATTN-GAN reduces the RMSE from 0.3302, 0.4039, 0.5514 to 0.1593, 0.2012, 0.2688 over the 20%, 50% and 80% missing rate. Therefore, our proposed model, ATTN-GAN, consisting of two combined multi-headed self-attentive blocks and a weighted combined block in the generator is beneficial for the imputation of missing data.\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003eIn order to more intuitively reveal the impact of the increase in the missing rate on the performance of the model, Fig. \u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e shows the changes in the imputation results of the model. The simple model (i.e., MEAN and LAST) performs very poorly across all missing rates. When the data missing rate gradually increases, the performance of the statistics-based imputation model and machine learning-based KNN method drop sharply, and the performance of Transformer based on the attention mechanism also drops significantly. Surprisingly, the bias of the proposed ATTN-GAN\u0026apos;s and SAITS\u0026rsquo;s estimation of missing values increases only slightly as the missing rate increases. However, when the missing rate is high, ATTN-GAN has a smaller imputation error in comparison. This is because the ATTN-GAN takes advantage of both generative adversarial structure and attention mechanism, which means it has a strong ability to capture data distribution and spatiotemporal correlation of data.\u003c/p\u003e\n\u003cp\u003eTo compare the actual imputation results more intuitively, the detailed imputation results of the baseline model and the proposed model for four consecutive missing values are plotted. Figure\u0026nbsp;5(a) shows the implementation in the absence of four consecutive air humidity measurements, and Fig.\u0026nbsp;5(b) depicts the implementation in the absence of four consecutive water temperature measurements. It is intuitively clear from Fig.\u0026nbsp;5. that the simple method imputation results are very poor and cannot fit the trends of time series, which can only simply use the contextual information. KNN can simply fit the trend of the time series, leading to a relatively large error. In general, SAITS, Transformer, and ATTN-GAN can more accurately use data information to fit missing data, and outperform statistics-based and machine-learning-based models in missing value imputation. What\u0026rsquo;s more, as shown in Fig.\u0026nbsp;5, ATTN-GAN estimate missing values better than Transformer and SAITS, especially when values change rapidly.\u003c/p\u003e\n\u003cdiv id=\"Sec15\" class=\"Section2\"\u003e\n \u003ch2\u003e3.2 Comparison of prediction results\u003c/h2\u003e\n \u003cp\u003eDue to equipment or transportation reasons, we do not have access to the original complete data, so we cannot directly compare the advantages and disadvantages of each imputation model. A good approach is to use the performance of downstream tasks to compare the true imputation performance of each model\u0026rsquo;s imputation for raw missing values indirectly. For downstream tasks, we first perform missing value imputation on multivariate time series with a missing rate of 40% by the proposed model and the baseline model. We then predict water temperature using the DA-RNN proposed by (Qin et al., \u003cspan class=\"CitationRef\"\u003e2017\u003c/span\u003e) and two simple forecasting models (MLP and LSTM). The MLP was set with a single hidden layer consisted of 128 neurons for comparison. Similarly, we apply LSTM as the middle unit of RNN and set the hidden size to 64. All three prediction models predict only one time step into the future for comparison. We can measure the accuracy of imputation by comparing the prediction performance of different models after imputation of missing value sequences indirectly. Figure \u003cspan class=\"InternalRef\"\u003e6\u003c/span\u003e clearly shows that the sequence processed by the proposed model significantly outperforms other baseline models in downstream prediction tasks. Table \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e shows the specific results of the prediction, from which, using the sequences imputed by the ATTN-GAN model as the input dataset achieves the best performance on all three prediction methods. Specifically, data processed with ATTN-GAN compared to SAITS as input to the prediction model reduces the MSE from 0.7552, 0.9038, 0.3832 to 0.6816, 0.8375, 0.3736 over MLP, LSTM, DA-RNN prediction methods.\u003c/p\u003e\n \u003cdiv class=\"gridtable\"\u003e\u0026nbsp;\u003ctable id=\"Tab3\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eMSE of water temperature prediction results by prediction models (MLP, LSTM, DA-RNN) trained on datasets imputed by different methods.\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMEAN\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eLSAT\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eKNN\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eARIMA\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eTransformer\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSIATS\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eATTM-GAN\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eMLP\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.9993\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.9838\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e1.1827\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.9703\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.7831\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.7552\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.6816\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eLSTM\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e1.1796\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e1.0528\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e1.2562\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.9896\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.9410\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.9038\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.8375\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eDA-RNN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.7697\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.6701\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.5003\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.5333\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.5445\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e0.3832\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.3736\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003c/div\u003e\n\u003c/div\u003e"},{"header":"4. Discussion","content":"\u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Comparison of imputation models\u003c/h2\u003e \u003cp\u003eDue to the extreme instability of simple models and simply utilization of observed data, MEAN and LAST value imputation methods have very large errors in the imputation of missing values, which is very detrimental to downstream tasks such as time series forecasting. Statistical-based ARIMA(Box and Jenkins, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e1968\u003c/span\u003e) simply memorizes the temporal trends of the sequence, which leads to large errors when there is a amount of missing data. Machine-learning-based KNN(Hudak et al., \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2008\u003c/span\u003e) which uses the mean value of k nearest neighbors to fill missing values neither captures the temporal pattern in the time-series dataset nor considers the connection between missing values and observations, which leads to their imputation effects decreasing with the increase in the missing rate is rapidly becoming worse. Taking advantage of attention mechanism, the Transformer (Vaswani et al., \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2017\u003c/span\u003e) is able to capture both temporal and spatial correlations in multivariate time series. However, As the rate of missing data increases, the amount of contextual information available to the model decreases, and the data interpolation performance decreases significantly. Similar to Transformer, SAITS (Du et al., \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2022\u003c/span\u003e) methods based on attention mechanism can learn the temporal and spatial correlations in multivariate time series. The imputation performance is better than the Transformer models, because it learns the distribution of data from a weighted combination of two diagonally masked self-attention blocks through joint optimization to complete missing values imputation. The proposed model can better learn the distribution of data from the remaining small amount of data, leading to best imputation performance in general. This is because the generator of the Generative Adversarial Network may capture the distribution of data, and the discriminator forces the data generated by the generator to more closely approximate the distribution of the real data (Goodfellow et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2014\u003c/span\u003e). Furthermore, attention mechanisms in the generator and discriminator enable them to learn temporal and spatial correlations from multivariate time series, making easier for the generator to impute the missing values. It is worth noting that the architecture of the generator is very similar to the SAITS model, but the details are slightly different. This model utilizes the time lag information between observations as part of the input network and simplifies the weighted combinatorial block\u003c/p\u003e \u003cp\u003eAlthough the combined attention module is simplified, due to the extra discriminator, the model is too large and require more time and memory to train than the baseline models. More fatally, as the time window lengthens and the number of variables increases, the volume of the model will explode, which causes great inconvenience to both training and the practical use of the model. The problem of how to produce lighter weight and higher performance models must be addressed. What\u0026rsquo;s more, fine-tuning the parameters takes a long time because the models contain a large number of hyperparameters compared to the baseline model. For various multivariate time series types, different parameters have highly different performance. Finally, the proposed model still does not fit ideally for time series with large data fluctuations. However, fitting the dramatic fluctuations of environmental data is important for agricultural production\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Ablation study\u003c/h2\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eperformance comparison of ATTN-GAN (single), generator, ATTN-GAN in (MAE / RMSE / MRE%). Bold indicates best performance.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMissing rate\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eATTN-GAN (single)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003egenerator\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eATTM-GAN\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e10%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.2548/0.1604/0.1311\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.1292/0.1654/0.1298\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.1024\u003c/b\u003e/0.1532/\u003cb\u003e0.1092\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e20%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.1941/0.1942/0.1402\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.1353/0.1763/0.1305\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.1115\u003c/b\u003e/\u003cb\u003e0.1593\u003c/b\u003e/\u003cb\u003e0.1144\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e30%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.1868/0.2131/0.1329\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.1283/\u003cb\u003e0.1569\u003c/b\u003e/\u003cb\u003e0.1182\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.1189\u003c/b\u003e/0.1736/\u003cb\u003e0.1237\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e40%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.2176/0.2347/0.1587\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.1425/0.2113/0.1403\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.1236\u003c/b\u003e/\u003cb\u003e0.1908\u003c/b\u003e/\u003cb\u003e0.1359\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e50%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.2275/0.2439/0.1816\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.1508/0.2262/\u003cb\u003e0.1399\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.1424/0.2012/0.1443\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e60%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.2629/0.2721/0.1948\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.16070.2487/\u003cb\u003e0.1461\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.1579\u003c/b\u003e/\u003cb\u003e0.2314\u003c/b\u003e/\u003cb\u003e0.1492\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e70%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.2861/0.3036/0.2268\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.1877/0.2618/\u003cb\u003e0.1484\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.1708\u003c/b\u003e/\u003cb\u003e0.2539\u003c/b\u003e/\u003cb\u003e0.1515\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e80%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.3048/0.3214/0.2432\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.2127/0.2828/\u003cb\u003e0.1727\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e0.2047\u003c/b\u003e/\u003cb\u003e0.2688\u003c/b\u003e/\u003cb\u003e0.1790\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eTo further justify the proposed model, i.e., the effectiveness of attention blocks and adversarial generative architecture on time series missing value imputation, we conduct an ablation study for the proposed model. Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e3\u003c/span\u003e shows the performance comparison of three models: ATTN-GAN (single) which is the proposed ATTN-GAN with single multi-head self-attention blocks, generator which comes from the proposed model and ATTN-GAN, and proposed ATTN-GAN. In comparison, the RMSE of generator decreases from 0.1942,0.2439,0.3214 to 0.1763, 0.2262, 0.2828 than ATTN-GAN (single) at missing rates of 20%, 50%, 80%, which proves that the weighted combination of two multi-head self-attentive blocks can capture the temporal and spatial pattern of multivariate time series much better than the single multi-head self-attentive blocks. Compared to the generator, ATTN-GAN which has an additional discriminator module lowers down RMSE by 9.66%, 11.1%, 4.95%. According to the results of ablation experience, it can be concluded that two combined multi-head self-attention blocks and one weighted combination block are beneficial to extract temporal and spatial patterns in time series, and the adversarial generative structure helps the model to fit the distribution of the original data.\u003c/p\u003e \u003c/div\u003e"},{"header":"5. Conclusion","content":"\u003cp\u003eIn this work, we propose an adversarial generative network-based model for imputation of missing values in multivariate time series. Compared with simple GAN, the generator and discriminator of the proposed ATTN-GAN model consist of multi-head attention blocks. According to the experimental results, the generative adversarial network architecture combined with the multi-head attention mechanism can learn the distribution of data and capture the spatio-temporal correlation well, thus successfully imputing the missing values of multivariate time series. Overall, ATTN-GAN outperforms the baseline in missing value input accuracy. For the forecasting of the downstream tasks, multivariate time series with a missing rate of 40% which imputed by baseline methods and ATTN-GAN were input to prediction models. Comparing the prediction results, the proposed model performs best on downstream prediction tasks. This also suggests that imputing missing values in multivariate time series before performing downstream tasks such as multivariate time forecasting may significantly improve forecast accuracy. However, the experiments were only carried out in the data aquaponics greenhouse environment, and it can only be concluded that the proposed model performed well on the missing data of the aquaponics greenhouse environment. For environmental data from other agricultural domains, such as fish farming, open fields, etc., the effect of the proposed model is unknown. And the proposed model is larger and takes longer to train than the baseline model. In future work, shrinking the model on the basis of ensuring the best input to reduce the training time is a worthwhile research direction. It will also be important to find out how applicable the model is to other areas of agriculture such as fish farming.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAcknowledgement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis research was financially supported by the 2115 Talent Development Program of China Agricultural University, National Key Research and Development Program of China: Sino-Malta Fund 2019: \u0026ldquo;Research and Demonstration of Real-time Accurate Monitoring System for Early-stage Fish in Recirculating Aquaculture System\u0026rdquo; (Aqua-Detector, Grant No. \u0026nbsp; 2019YFE0103700), Overseas High-level Youth Talents Program (China Agricultural University, China, Grant No. 62339001), National Innovation Center for Digital Fishery, and Beijing Engineering and Technology Research Center for Internet of Things in Agriculture. The authors also appreciate constructive and valuable comments provided by reviewers.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEthical Approval\u003c/strong\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026rsquo; contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eKeyang Zhong:\u0026nbsp;\u003c/strong\u003eWriting \u0026ndash; Original Draft, Methodology, Model; \u003cstrong\u003eXueqian Sun\u003c/strong\u003e: Writing \u0026ndash; Original Draft, Methodology, Model; \u003cstrong\u003eGedi Liu\u003c/strong\u003e: Writing-Reviewing \u0026amp; Editing;\u003cstrong\u003e\u0026nbsp;Yifeng Jiang:\u0026nbsp;\u003c/strong\u003eWriting-Reviewing \u0026amp; Editing;\u003cstrong\u003e\u0026nbsp;Yi Ouyang:\u0026nbsp;\u003c/strong\u003eWriting-Reviewing \u0026amp; Editing;\u003cstrong\u003e\u0026nbsp;Yang Wang\u003c/strong\u003e: Writing-Reviewing \u0026amp; Editing, Supervising, Conceptualization.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding \u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis research was financially supported by the 2115 Talent Development Program of China Agricultural University, National Key Research and Development Program of China: Sino-Malta Fund 2019: \u0026ldquo;Research and Demonstration of Real-time Accurate Monitoring System for Early-stage Fish in Recirculating Aquaculture System\u0026rdquo; (Aqua-Detector, Grant No. 2019YFE0103700), Overseas High-level Youth Talents Program (China Agricultural University, China, Grant No. 62339001)\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and materials\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe data and materials used in this article can be accessed after contacting authors.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eAcar E, Dunlavy DM, Kolda TG, M\u0026oslash;rup M (2010) Scalable Tensor Factorizations with Missing Data, in: Proceedings of the 2010 SIAM International Conference on Data Mining. Presented at the Proceedings of the 2010 SIAM International Conference on Data Mining, Society for Industrial and Applied Mathematics, pp.\u0026nbsp;701\u0026ndash;712. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1137/1.9781611972801.61\u003c/span\u003e\u003cspan address=\"10.1137/1.9781611972801.61\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBoomgard-Zagrodnik JP, Brown DJ (2022) Machine learning imputation of missing Mesonet temperature observations. Comput Electron Agric 192:106580. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.compag.2021.106580\u003c/span\u003e\u003cspan address=\"10.1016/j.compag.2021.106580\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBox GEP, Jenkins GM (1968) Some Recent Advances in Forecasting and Control. J Royal Stat Soc Ser C (Applied Statistics) 17:91\u0026ndash;109. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.2307/2985674\u003c/span\u003e\u003cspan address=\"10.2307/2985674\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCao W, Wang D, Li J, Zhou H, Li L, Li Y (2018) BRITS: Bidirectional Recurrent Imputation for Time Series. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.1805.10572\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.1805.10572\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChe Z, Purushotham S, Cho K, Sontag D, Liu Y (2018) Recurrent Neural Networks for Multivariate Time Series with Missing Values. Sci Rep 8:6085. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41598-018-24271-9\u003c/span\u003e\u003cspan address=\"10.1038/s41598-018-24271-9\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen H, Grant-Muller S, Mussone L, Montgomery F (2001) A Study of Hybrid Neural Network Approaches and the Effects of Missing Data on Traffic Forecasting. Neural Comput Appl 10:277\u0026ndash;286. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s521-001-8054-3\u003c/span\u003e\u003cspan address=\"10.1007/s521-001-8054-3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChoi E, Xiao C, Stewart WF, Sun J (2018) MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDu W, Cote D, Liu Y (2022) SAITS: Self-Attention-based Imputation for Time Series. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2202.08516\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2202.08516\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFung D (2006) Methods for the estimation of missing values in time series. Doctorates and Masters, Theses\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGoodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative Adversarial Networks. arXiv:1406.2661 [cs, stat].\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGupta M, Beheshti R (2020) Time-series Imputation and Prediction with Bi-Directional. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2009.08900\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2009.08900\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Generative Adversarial Networks\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHamzacebi C (2008) Improving artificial neural networks\u0026rsquo; performance in seasonal time series forecasting. Inf Sci 178:4550\u0026ndash;4559. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.ins.2008.07.024\u003c/span\u003e\u003cspan address=\"10.1016/j.ins.2008.07.024\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHastie T, Mazumder R, Lee J, Zadeh R (2014) Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHsieh T-J, Hsiao H-F, Yeh W-C (2011) Forecasting stock markets using wavelet transforms and recurrent neural networks: An integrated system based on artificial bee colony algorithm. Appl Soft Comput 11:2510\u0026ndash;2525. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.asoc.2010.09.007\u003c/span\u003e\u003cspan address=\"10.1016/j.asoc.2010.09.007\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuan J, Li H, Wu F, Cao W (2020) Design of water quality monitoring system for aquaculture ponds based on NB-IoT. Aquacult Eng 90:102088. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.aquaeng.2020.102088\u003c/span\u003e\u003cspan address=\"10.1016/j.aquaeng.2020.102088\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHudak AT, Crookston NL, Evans JS, Hall DE, Falkowski MJ (2008) Nearest neighbor imputation of species-level, plot-scale forest structure attributes from LiDAR data. Remote Sens Environ 112:2232\u0026ndash;2245. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.rse.2007.10.009\u003c/span\u003e\u003cspan address=\"10.1016/j.rse.2007.10.009\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKaiser J (2014) Dealing with Missing Values in Data. J Syst Integr 5:42\u0026ndash;51. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.20470/jsi.v5i1.178\u003c/span\u003e\u003cspan address=\"10.20470/jsi.v5i1.178\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKreindler D, Lumsden C (2006) Effects of the irregular sample and missing data in time series analysis. Nonlinear dynamics, psychology, and life sciences 10, 187\u0026ndash;214\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLuo Y, Cai X, Zhang Y, Xu J, Yuan X (2018) Multivariate time series imputation with generative adversarial networks, in: Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS\u0026rsquo;18. Curran Associates Inc., Red Hook, NY, USA, pp.\u0026nbsp;1603\u0026ndash;1614\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLuo Y, Zhang Y, Cai X, Yuan X (2019) E\u003csup\u003e2\u003c/sup\u003eGAN: End-to-End Generative Adversarial Network for Multivariate Time Series Imputation, in: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. Presented at the Twenty-Eighth International Joint Conference on Artificial Intelligence {IJCAI-19}, International Joint Conferences on Artificial Intelligence Organization, Macao, China, pp.\u0026nbsp;3094\u0026ndash;3100. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.24963/ijcai.2019/429\u003c/span\u003e\u003cspan address=\"10.24963/ijcai.2019/429\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMa J, Shou Z, Zareian A, Mansour H, Vetro A, Chang S-F (2019) CDSA: Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.1905.09904\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.1905.09904\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMazumder R, Hastie T, Tibshirani R (2010) Spectral Regularization Algorithms for Learning Large Incomplete Matrices 36\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eQin Y, Song D, Chen H, Cheng W, Jiang G, Cottrell GW (2017) A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction, in: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Presented at the Twenty-Sixth International Joint Conference on Artificial Intelligence, International Joint Conferences on Artificial Intelligence Organization, Melbourne, Australia, pp.\u0026nbsp;2627\u0026ndash;2633. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.24963/ijcai.2017/366\u003c/span\u003e\u003cspan address=\"10.24963/ijcai.2017/366\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSilva L, Z\u0026aacute;rate L (2014) A brief review of the main approaches for treatment of missing data. Intell Data Anal 18:1177\u0026ndash;1198. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.3233/IDA-140690\u003c/span\u003e\u003cspan address=\"10.3233/IDA-140690\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTak S, Woo S, Yeo H (2016) IEEE Trans Intell Transport Syst 17:1762\u0026ndash;1771. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/TITS.2016.2530312\u003c/span\u003e\u003cspan address=\"10.1109/TITS.2016.2530312\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Data-Driven Imputation Method for Traffic Data in Sectional Units of Road Links\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention Is All You Need\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWilson GT, Time Series Analysis: Forecasting and Control, 5th Edition, by, George EP, Box, Gwilym M, Jenkins GC, Reinsel, Ljung GM, 2015. Published by John Wiley and Sons Inc., Hoboken, New Jersey, pp.\u0026nbsp;712. ISBN: 978-1-118-67502-1: TIME SERIES ANALYSIS: FORECASTING AND CONTROL,5TH EDITION, by George E., Box P, Jenkins GM, Reinsel GC, Ljung GM (2016) 2015. Publis. J. Time Ser. Anal. 37, 709\u0026ndash;711. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/jtsa.12194\u003c/span\u003e\u003cspan address=\"10.1111/jtsa.12194\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYoon J, Jordon J, van der Schaar M (2018) GAIN: Missing Data Imputation using Generative Adversarial Nets. arXiv:1806.02920 [cs, stat].\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYoon J, Zame WR, van der Schaar M (2019) IEEE Trans Biomed Eng 66:1477\u0026ndash;1490. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/TBME.2018.2874712\u003c/span\u003e\u003cspan address=\"10.1109/TBME.2018.2874712\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Estimating Missing Data in Temporal Data Streams Using Multi-Directional Recurrent Neural Networks\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYu H-F, Rao N, Dhillon IS (2016) Temporal regularized matrix factorization for high-dimensional time series prediction, in: Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS\u0026rsquo;16. Curran Associates Inc., Red Hook, NY, USA, pp.\u0026nbsp;847\u0026ndash;855\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang Y, Thorburn PJ (2021) A dual-head attention model for time series data imputation. Comput Electron Agric 189:106377. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.compag.2021.106377\u003c/span\u003e\u003cspan address=\"10.1016/j.compag.2021.106377\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Attention-based Generative Adversarial Networks, Aquaponics Greenhouse, Missing Data, Data Imputation, Multivariate Time Series","lastPublishedDoi":"10.21203/rs.3.rs-3209861/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3209861/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eEnvironmental parameter data collected by sensors for monitoring the environment of aquaponics facility operations are usually incomplete due to external environmental disturbances and device failures. And the missing of collected data is completely at random. In practice, missing data could create biased estimations and make multivariate time series predictions of environmental parameters difficult, leading to imprecise environmental control. A multivariate time series imputation model based on generative adversarial networks and multi-head attention (ATTN-GAN) is proposed in this work to reducing the negative consequence of missing data. ATTN-GAN can capture the temporal and spatial correlation of time series, and has a good capacity to learn data distribution. In the downstream experiments, we used ATTN-GAN and baseline models for data imputation, and predicted the imputed data, respectively. Experiments reveal that ATTN-GAN outperforms all baseline models in terms of data imputation accuracy. The data processed by ATTN-GAN is the best for time series prediction.\u003c/p\u003e","manuscriptTitle":"Attention-based generative adversarial networks for aquaponics environment time series data imputation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2023-08-07 15:59:53","doi":"10.21203/rs.3.rs-3209861/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e2c49632-a5b5-41ed-8190-4077b26852b7","owner":[],"postedDate":"August 7th, 2023","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2023-08-10T07:29:21+00:00","versionOfRecord":[],"versionCreatedAt":"2023-08-07 15:59:53","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3209861","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3209861","identity":"rs-3209861","version":["v1"]},"buildId":"_2-kVJe1T_tPrBINL-cwx","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}