Reconstructing Computational Spectra Using Deep Learning's Self-Attention Method

Reconstructing Computational Spectra Using Deep Learning's Self-Attention Method | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Reconstructing Computational Spectra Using Deep Learning's Self-Attention Method Hao Wu, Hui Wu, xingyu su, Jingjun Wu, shuangli liu This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4074358/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Miniaturized computational spectrometers have become a new research hotspot due to their portability and miniaturization. However, there are several issues, like low precision and poor stability. Because the problem of spectrum reconstruction accuracy is very evident, we suggested a novel approach to raise the reconstruction accuracy. Using the time-domain finite-difference (FDTD) method, a library of transmittance functions was acquired. A cross-correlation algorithm was then used to choose 100 sparse transmittance functions, which were then built as an encoding matrix. Then, based on the encoding matrix, a self-attention mechanism algorithm was used to improve the accuracy of the reconstructed spectrum by improving the autocorrelation characteristics between input signals. The mean square error (MSE) of the reconstructed spectrum is 0.0019, and its similarity coefficient (R 2 ) is 0.9780. This self-attention mechanism spectral reconstruction technique will open up new possibilities for high-accuracy reconstruction for a variety of computational spectrometer types. Spectral Reconstruction Self-Attention Encoding Matrix Cross-Correlation Figures Figure 1 Figure 2 Figure 3 Figure 4 1. Introduction The spectrometer is widely acknowledged as a vital tool in industry and scientific research. Miniaturized spectrometer is emerging as a key platform for many developing applications, including consumer electronics, hyperspectral imaging, and in-situ sensing[ 1 ]. Depending on the intricacy of the necessary algorithms, there are two primary types of spectrometer miniaturization techniques: traditional and computational[ 2 ]. A new paradigm in spectrometer miniaturization, the computational microspectrometer (CS) is based on computational spectroscopy and is anticipated to solve the limitations of conventional microspectroscopy techniques, including their inability to achieve extreme miniaturization, lack of stability, and limited resolution[ 3 ]. The CS relies on computational techniques to approximate or “reconstruct” an incident light spectrum from precalibrated information encoded within a set of detectors[ 4 ]. The process of reconstruction is crucial. Spectral reconstruction techniques have advanced significantly in recent years due to the growth of CS[ 5 ]. In current computational reconstruction algorithms, Chang et al. use the Gaussian kernel template denoising method and parameter minimization method using the l1 paradigm for reconstruction [ 7 ]. The results were compared with the l1-paradigm approach using the Tikhonov Regularized Nonnegative Least Squares (TNNLS) technique in both simulations and real data. However, the measurements largely depend on early training and Gaussian basis selection, which limits the application prospects. Conversely, Oliver et al. showed that the sparsity of the signal spectrum can be used to further improve the resolution[ 8 ][ 9 ]. Zhang et al. presented a reconstruction approach based on dictionary learning and sparse optimization. The experimental results imply that dictionary learning can greatly increase the sparsity of general spectra, as l1-paradigm minimization performs well for both direct sparse spectra and general spectra that need to be translated into dictionaries[ 10 ]. Yang et al. constructed the smallest microcomputing spectrometer in the world at the time by combining recovered spectral data fitted with a Gaussian basis function[ 11 ]. Huang et al. used a sparse encoding method and combined it with a compressed sensing algorithm to ensure the accuracy of spectral reconstruction, but the spectral resolution was not high enough [ 12 ]. Huang et al. were able to increase the spectral resolution by four times by employing this property of the codec model of the spectral features fed into the network to successfully reconstruct the spectrum information of 100 bands from 25 feature points[ 13 ][ 14 ]. The CS is unable to obtain accurate readings in real-time, and the reconstruction accuracy of these methods is subpar. Recent research has demonstrated that employing deep learning algorithms for spectrum reconstruction of computational spectroscopy systems produces good results. However, because just the linear transformation of the completely linked layer itself is used and the correlation between the spectra and the spectra itself is ignored, the accuracy of the spectral reconstruction is not very great[ 15 – 17 ]. On the other hand, it demonstrates that deep learning may be highly useful in spectrum reconstruction and that deep neural networks can learn the mapping between input and target output distributions by training a large number of samples. CNNs are more suited for processing data with local features and spatial information, like photos and videos, while the self-attention mechanism works well for processing sequential data, particularly for jobs with global dependencies. Currently, the self-attention mechanism has been widely used in language recognition [ 18 ], text categorization [ 19 ][ 20 ], speech translation [ 21 ], natural language generation [ 22 ], image recognition[ 23 ][ 24 ], image generation[ 25 ][ 26 ], and so on. The self-attention mechanism accomplishes adaptive weight allocation—which determines the relative importance of different bands—by computing attention weights. Certain spectral data bands may contain information that is redundant or noisy, while other bands may have more relevant information. The self-attention method improves reconstruction accuracy by allowing the network to autonomously assess the importance of various bands and assign them greater weights during the process. To capture global context dependence, the self-attention mechanism also evaluates the correlation between each element in the input sequence and other elements [ 27 ] [ 28 ]. Consequently, it is anticipated that this technique will raise the spectral reconstruction's accuracy even further. In this work, we address the issue of inadequate accuracy in reconstructing spectra through miniaturized computing by designing the encoding matrix using cross-correlation algorithms and combining a novel deep-learning self-attention mechanism to optimize the final reconstruction results. The spectral reconstruction accuracy of computational spectrometers will be much enhanced by this method, which will also be a significant step toward the practical use of computational spectrometers. 2. Method Principle of the data acquisition system This section describes and demonstrates how to measure the reflection spectrum of a target sample under sunlight, and the simplified schematic is shown in Fig. 1. When light is shone on the target picture, a transformation occurs as follows: $$\int L\left(\lambda \right)S\left(\lambda ,X,Y\right)T\left(\lambda \right)D\left(\lambda \right)d\lambda =I\left(X,Y\right)$$ 1 Makes the conversion into coded input data that can be fed into the algorithm model. In the formula, \(L\left(\lambda \right)\) represents daylight light information, \(S\left(\lambda ,X,Y\right)\) is the detection target, where \(\left(X,Y\right)\) is the two-dimensional position coordinates of the single-point spectrum, \(D\left(\lambda \right)\) is the detection camera's response function, \(T\left(\lambda \right)\) is the designed encoding matrix, and \(I\left(X,Y\right)\) is the encoding intensity of the corresponding position coordinates. This conversion process requires first pasting the designed encoding matrix in front of the camera, which is used to test the spectral information of the reflected light of the sample \(S\left(\lambda ,X,Y\right)\) , so the spectral intensity is \(L\left(\lambda \right)S\left(\lambda ,X,Y\right)\) before entering the camera conversion. Then the encoded intensity \(I\left(X,Y\right)\) is finally obtained by the camera's encoding matrix \(T\left(\lambda \right)\) and response function \(D\left(\lambda \right)\) . The specified encoding matrix can be turned into a chip that fits exactly with the camera to make spectral measurement more convenient. Filter function design and construction of data set The data construction in this study is divided into two parts. To construct an accurate spectral reconstruction model, suitable input and output data needs to be collected. The first part of the input data is a complete library of filtering functions simulated by FDTD and set by the expectation to 0.85 as a threshold condition as an initial sieve, and then utilized the cross-correlation function: $$\rho \left(X,Y\right)=\frac{Cov(X,Y)}{\sqrt{D\left(X\right)D\left(Y\right)}}=\frac{E\left(XY\right)-E\left(X\right)E\left(Y\right)}{\sqrt{E\left({X}^{2}\right)-{E}^{2}\left(X\right)}\sqrt{E\left({Y}^{2}\right)-{E}^{2}\left(Y\right)}}$$ 2 \(X,Y\) denote different curves respectively, the value of \(Cov\) is the covariance of \(X,Y\) curves, and its calculation result ρ is a scalar and the range \(\rho\) ∈[-1,1]; when \(\rho\) >0, it means positive correlation, and the closer to 1 means the stronger correlation; \(\rho\) <0, it means negative correlation and the closer to -1 means the stronger correlation; when the correlation coefficient is closer to 0, the correlation is weaker, and equals to 0 when two variables are not Correlation[ 33 ]. Using this formula, each of the filter function curves is used to calculate the correlation between the filter functions screened out at the beginning. Finally, the 100 filter functions whose absolute value is closest to 1 are sorted out and arranged into a 10x10 coded array. In the second part of the experiment, spectral simulations were first performed using advanced basis functions to generate a series of spectral curves. These simulated spectral curves introduced varying degrees of Gaussian noise on top of the actual spectral data, including noise levels of 5%, 10%, 15%, and 20%. In total, 20,000 such simulated spectral data were generated to ensure broad coverage of noise levels and spectral features. Next, these simulated spectral data were coded and transformed into uncoded input data using a pre-constructed filter function coding matrix. Then use the following formula to convert the simulated spectrum into decoded data: $${I}_{i}={\int }_{{\lambda }_{1}}^{{\lambda }_{2}}{T}_{i}\left(\lambda \right)(f\left(\lambda \right)+{e}_{i})d\lambda +\approx \sum _{j=1}^{M}{T}_{i}\left({\lambda }_{i}\right)(f\left({\lambda }_{i}\right)+{e}_{i}),i=1,\cdots ,N$$ 3 The appropriate input-encoded data were obtained in this manner, where \({T}_{i}\left(\lambda \right)\) is the encoding matrix, \(f\left(\lambda \right)\) is the spectral data, and \({e}_{i}\) is the noise level. Its integration can be decomposed into the sum of multiple points, where M is the number of measurements within the measurement band and N is the number of encodings. This step was designed to simulate the process of encoding and reconstruction of spectral data in real applications. Through this process, a total of 100,000 simulated data was obtained, which included both training and validation sets. To ensure the accuracy and robustness of the model, these 100,000 simulated data were divided into training and validation sets in a ratio of 8:2. It is worth noting that the selection of the validation set was done randomly to ensure extensive testing of different spectral data. Finally, an additional 1000 simulated spectral data that were not in the training and validation sets were transformed to serve as an independent test set to evaluate the accuracy and performance of the model on unseen data. This step is important to test the performance of our model in real-world applications. The framework of the neural network model The neural network architecture for spectrum reconstruction can be expressed easily as \(FC\left(100\right)\to LR\to Self-Attention\left(100\right)\to FC\left(120\right)\to LR\to FC\left(150\right)\to LR\to FC\left(200\right)\to LR\) . Each digit represents the number of cells in the relevant layer. The ReLU function is denoted by the symbol LR. fc means a fully connected layer, 100 random spectral filters correspond to the number of input cells, and 200 denotes the reconstructed spectral channels (8um-12um, 0.02um steps). In this paradigm, the self-attention mechanism and residuals are used for training. To get more relevant features, the input incoming data is first Processed to a self-attention operation, and then residuals are incorporated in each neural layer to minimize parameter explosion and overfitting during training. Deionizing in normal compressed sensing algorithms, on the other hand, heavily relies on prior knowledge, and parameters are typically updated manually throughout the iterative process to offset the bias generated by noise, which is effective but does not provide convincing results when the noise level varies. As a result, regularization parameters are introduced to the training model to improve its robustness [ 31 ][ 32 ]. In which Self-Attention is computed in the following steps: 1. Generate Query, Key and Value: First, the query vector (Q), key vector (K), and value vector (V) are generated by a linear transformation of the input data (usually using a weight matrix). Generating these vectors usually involves the multiplication of weight matrices to map the input data into a low-dimensional representation space for subsequent computation. 2. Compute the attention scores: Next, the attention scores between each query (Q) and each key (K) are computed, usually using the dot product (Dot Product) method. For a single query vector, the dot product is computed with all key vectors, and then a scaling factor is applied to control the range of scores. The formula is as follows: $$Attention\left(Q,K\right)=Softmax\left(\right(Q*{K}^{\wedge }T)/sqrt({d}_{k})$$ 4 Where Q is the query vector, K is the key vector and \({d}_{k}\) is the dimension of the key vector. The scores can be converted to probability distributions by applying the Softmax function. 3. Weighted Summation: The computed attention scores perform a weighted summation on the value vector (V) to generate the final self-attention representation: $$Self-Attention\left(Q,K,V\right)=Attention\left(Q,K\right)*V$$ 5 This step involves a weighted summation of the value vectors based on the attention scores to produce a self-attention representation. Finally, the reconstructed spectral data is derived from the self-attention values in the subsequent full connectivity. The residual mechanism is added at each fully connected layer to prevent the gradient from disappearing during training. The result is shown in Fig. 2 B. Training and validation In our model training process, we chose the mean square error (MSE) as the loss function, which is a loss function commonly used in regression problems. The MSE measures the mean squared error between the model predictions and the actual observations. By minimizing the MSE, our model can better fit the spectral data and ensure that the resulting reconstruction is as close as possible to the real spectrum. To improve the stability and speed of training, the batch normalization technique is applied. Batch normalization helps to normalize the inputs of each layer in the neural network, ensuring that the distribution of the data is stable and reducing the problem of gradient explosion or gradient vanishing. This is very useful when training deep neural networks and helps the model to converge faster. The Adam optimization algorithm was chosen to tune the model parameters, which combines the characteristics of the momentum method and the adaptive learning rate and is usually able to find the global minimum faster. In addition, the appropriate learning rate for model training was set to ensure that the weights were updated appropriately during the training process to avoid the model falling into local minima or divergence. Meanwhile, to avoid the overfitting problem, two regularization methods are used, namely L1 regularization and drops out. L1 regularization drives the model to be sparser and reduces the possibility of overfitting by applying penalties to the weight parameters of the model. Dropout, however, randomly discards some neurons during training to prevent the model from relying too much on the training data and improve the generalization performance. To verify the performance and practical application value of the model, we used actual spectral data measured by an optical spectrometer. These data have actual physical significance and reflect the nature of the actual spectral signals, thus ensuring that the model can be accurately reconstructed in the face of spectral data in real environments, thus demonstrating the practical usability and effectiveness of the model. This experimental design helps to generalize the research results to practical applications and provides solutions to practical problems in related fields. Spectral reconstruction performance indicators Two related indicators were used as reconstruction indicators. The R 2 similarity function is used to evaluate the following metrics: $${R}^{2}=1-\frac{{{{\Sigma }}_{i=1}^{n}\left({y}_{i}-{Y}_{i}\right)}^{2}}{{{{\Sigma }}_{i=1}^{n}\left({Y}_{i}-\stackrel{-}{Y}\right)}^{2}}$$ 6 where \({y}_{i}\) is the reconstructed spectrum's intensity value, \({Y}_{i}\) is the simulated spectrum's intensity value, and \(\stackrel{-}{Y}\) is the simulated spectrum's average intensity value; as well as MSE (mean square error function): $$MSE=\frac{1}{n}\sum _{i=1}^{n}{\left({y}_{i}-{Y}_{i}\right)}^{2}$$ 7 Where \({y}_{i}\) is the intensity value of the reconstructed spectrum and \({Y}_{i}\) is the intensity value of the simulated spectrum. 3. Result Design of data collection system This section presents a spectral acquisition system we have designed that can collect spectral information about objects illuminated by light. It mainly involves reflecting the information of the illuminated object onto the designed spectral chip through a Formula 2 conversion and then collecting undeciphered data through a detector. Spectral reconstruction of the self-attention mechanism framework Compared with traditional compressed sensing algorithms, deep learning neural networks have significant advantages in dealing with noise. In traditional compressed sensing algorithms, denoising often relies on a priori information, and some parameters usually need to be set manually to counteract the noise-induced bias. This method can be effective to some extent but has some limitations. When the noise level changes or the nature of the noise changes, the performance of traditional compressed sensing algorithms may suffer and fail to provide consistent and credible results. Deep learning neural networks excel at noise processing because they are adaptive. This means that neural networks can learn parameters and features from the data without much manual tuning. In the presence of noise, neural networks can automatically adjust their weights and structures to better fit and model noisy data. This adaptivity allows the neural network to maintain a better fit when dealing with data with different noise levels and properties. Deep learning neural networks are usually trained with large-scale data to give them strong generalization capabilities. This means that neural networks can process and reduce high-quality data even in the presence of noise not covered during training. This is important for spectral reconstruction in real-world applications, where the type and noise level can change at any time in a real environment. Therefore, to achieve better reconstruction results, the self-attention mechanism and residual in deep learning were used for reconstruction. The algorithm model is depicted in Fig. 2 b. To extract more valuable features from this randomly encoded input for better reconstruction, the model uses a self-attention strategy to improve the relationship between attributes. The self-attention mechanism computes the similarity between each site and other places. It uses it as a weight to weigh the data together, focusing attention on the relevant features to increase result accuracy. The residual mechanism adds cross-layer connections to the neural network layers, avoiding gradient disappearance and gradient explosion while also boosting model training [ 29 ][ 30 ]. Coding matrix design and dataset construction The FDTD simulation-generated filter function library is shown in Fig. 3a, and the initial screening-generated filter function matrix is presented in Fig. 3b. The initial screening calculates the expectation and variance of each filter function to verify that each filter function has an encoding effect because there would be no light intensity change when detecting the target, preventing the spectrum from being decoded. Finally, the filter matrix is filtered again using the intercorrelation function to yield the results shown in Fig. 3c. The architecture of the generated spectral data is also depicted in Fig. 3, where varying amounts of Gaussian noise are included to imitate genuine environmental conditions to improve the training model's robustness. Reconstruction results of self-attention mechanism algorithm Fig. 4. a-f demonstrates the outcomes of the attention-based algorithm model reconstruction, with the corresponding metric values annotated. The algorithm's reconstruction achieved good results with R 2 =0.9780 and MSE=0.0019, this is the overall average result metric for the entire test set. The results show that using the mutual correlation algorithm to filter the filter function and combining it with the self-attention mechanism network for spectral reconstruction can produce better reconstruction results, which is an important reference value for practical spectral reconstruction applications. We also tested the reconstruction results of different algorithms using our dataset, and the test results are shown in Table. 1. Table 1 Comparison of Results of Different Algorithms Different algorithms R 2 MSE Least Squares 0.87 0.12 compressed Sensing 0.89 0.10 DNN 0.91 0.07 seq2seq 0.94 0.04 Self-Attention 0.9780 0.0019 4. Discussion In this work, we propose a new method to construct a filtering function encoding matrix and integrate the self-attention mechanism in deep learning for spectral reconstruction, to improve the accuracy and stability of the final computed spectral reconstruction. In previous coding matrix designs, there was not much emphasis on the correlation between different data. However, a filtering method based on cross-correlation functions can ensure that the constructed coding matrix is sparse enough to better express features and facilitate data processing in subsequent spectral reconstruction. In addition, in previous algorithms, spectral reconstruction was mostly carried out by directly fitting the linear data to obtain the spectral information of the object. However, in the real environment, there is some nonlinear information that can make reconstruction difficult when mixed. Our spectral reconstruction method using a self-attention mechanism not only strengthens the self-correlation between input encoded data but also fits the nonlinear data. Reduce the influence of environmental factors, thereby further improving the accuracy and stability of the spectral reconstruction. Our research first uses the filtering method of cross-correlation function to filter the filtering function library simulated by FDTD, to screen out the filtering functions with cross-correlation characteristics, to efficiently construct the sparse encoding matrix of the filtering function. The sparse nature of data can make the encoding matrix more expressive; help extract effective information from spectral data and provide a better data foundation for subsequent spectral reconstruction. This filtering method also helps to reduce computational complexity and improve computational efficiency. Secondly, we adopted the popular self-attention mechanism in the field of deep learning to enhance the feature correlation between input-encoded data, enabling the model to better capture complex nonlinear data relationships. This plays an important role in spectral reconstruction, as spectral data typically contains complex correlations between multiple bands. The self-attention mechanism effectively improves the accuracy of spectral reconstruction by adaptively learning the correlation between bands. In addition, a residual network was introduced during training to prevent overfitting issues during the training process. This mechanism can enable the model to better record and preserve the good results of the previous step, especially in deep networks, which helps to improve the model's generalization ability; that is, it can still obtain good reconstruction results in complex environments. The experimental results show that the algorithm combining the self-attention mechanism has higher accuracy and better performance compared to other algorithms and can improve the accuracy and stability of the spectral reconstruction. This method is not only suitable for simulating data but also provides strong support for its application in real-world environments. In the future, we will consider expanding this research with more diverse application scenarios and datasets to further improve stability and accuracy. 5. Conclusion In this work, we propose a new method to construct a filtering function encoding matrix and integrate the self-attention mechanism in deep learning for spectral reconstruction, ultimately achieving good accuracy and stability in the reconstruction. Our work first utilizes the method of cross-correlation function to ensure that the constructed encoding matrix can better express features and improve the accuracy and performance of subsequent reconstruction results. Then, a combination of self-attention mechanism and residual is used to enhance the self-correlation between input encoded data, and non-linear data is fitted to reduce the influence of environmental factors, thereby further improving the accuracy and stability of spectral reconstruction. The results show that the reconstruction algorithm is significantly superior to other algorithms, achieving good results with R 2 = 0.9780 and MSE = 0.0019. To further validate and enhance the approach for the development of spectrum reconstruction approaches in the future, more experiments and practical application testing are needed. This work offers novel insights into how deep learning and self-attention mechanisms might be used to enhance the accuracy of spectral reconstruction. These findings could have a broad variety of implications for future investigations and real-world applications in spectroscopy and related domains. Declarations Conflicts of Interest: All authors disclosed no relevant relationships. Funding Declaration: This is an extended study conducted independently with no funding support. Author Contribution H.W. wrote and created the associated algorithms and gathered data. H. W. supplied the FDTD simulation dataset. X.S. processed and annotated certain pertinent data. S. L. and J. W. provide suggestions and directions for manuscript revisions. Data Availability Statement: The dataset is confidential and therefore not easy to disclose. References Yang Z et al (2019) Single-nanowire spectrometers. Science 365, 1017–1020 10.1126/science.aax8814 ; pmid: 31488686 Schuler LP, Milne JS, Dell JM, Faraone L (2009) MEMS-based microspectrometer technologies for NIR and MIR wavelengths. J Phys D 42:133001. 10.1088/ 0022-3727/42/13/133001 Malinen J et al (2014) Advances in miniature spectrometer and sensor development. Proc. SPIE 9101, 91010C 10.1117/12.2053567 M.Ebermann et al., Tunable MEMS Fabry-Pérot filters for infrared microspectrometers: A review. Proc. SPIE 9760, 97600H (2016) 10.1117/12.2209288 Crocombe RA (2018) Portable Spectrosc Appl Spectrosc 72:1701–1751 doi:.1177/0003702818809719; pmid: 30335465 Wolffenbuttel RF (2005) MEMS-based optical mini- and microspectrometers for the visible and infrared spectral range. J Micromech Microeng 15:S145–S152. 10.1088/0960-1317/15/7/021 Kurokawa U, Choi BI, Chang CC (2011) Filter-based miniature spectrometers: spectrum reconstruction using adaptive regularization. IEEE Sens J 11(7):1556–1563. https://doi.org/10.1109/jsen.2010.2103054 Oliver J, Lee W, Park S et al (2012) Improving resolution of miniature spectrometers by exploiting sparse nature of signals. Opt Expr 20(3):2613–2625 Oliver J, Lee WB, Lee HN (2013) Filters with random transmittance for improving resolution in filter-array-based spectrometers. Opt Expr 21(4):3969–3989. https://doi.org/10.1364/oe.21.003969 A Spectral Reconstruction Algorithm of Miniature (2018) Spectrometer Based on Sparse Optimization and Dictionary Learning. Shang Zhang, Yuhan Dong Miniaturization of optical spectrometers ZongYin Yang SCIENCE 29 Jan 2021 Vol 371, Issue 6528. 10.1126/science.abe0722 One-shot ultraspectral imaging with reconfigurable metasurfaces (2019) Xusheng Cai, Jian Xiong, Kaiyu Cui Yang J, Cui K, Cai X et al (2022) Ultraspectral Imaging Based on Metasurfaces with Freeform Shaped Meta-Atoms[J]. Laser & Photonics Reviews, p 16. 7 Baraniuk RG (2007) Compressive sensing. IEEE Signal Process Mag 24:118–121 Goodfellow I, Bengio Y, Courville A (2016) Deep Learning 1. MIT Press Song HY et al (2021) Deep-learned broadband encoding stochastic fifilters for computational spectroscopic instruments. Adv Theory Simul 4:2000299 Nie SJ et al (2018) IEEE, Salt Lake City, UT, USA,. Deeply learned fifilter response functions for hyperspectral reconstruction. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 18–23 June 2018 Huang J, Zhou W, Zhang Q, Li H, Li W (2018) Video-based sign language recognition without temporal segmentation. In Thirty-Second AAAI Conference on Artifificial Intelligence Gang Liu and Jiabao Guo (2019) Bidirectional lstm with attention mechanism and convolutional layer for text classifification. Neurocomputing 337:325–338 Karim, Ahmed (2017) Nitish Shirish Keskar, and Richard Socher. Weighted transformer network for machine translation. arXiv preprint arXiv:1711.02132 Sperber M, Neubig G, Niehues J, Waibel A (2019) Attention-passing models for robust and data-effificient end-to-end speech translation. Trans Association Comput Linguistics 7:313–325 Xu K, Wu L, Wang Z, Feng Y, Witbrock M, Sheinin V (2018) Graph2seq: Graph to sequence learning with attention-based neural networks. arXiv preprint arXiv:1804.00823 Fu J, Zheng H, Mei T Look closer to see better: Recurrent attention convolutional neural network for fifine-grained image recognition. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4476–4484, Honolulu, HI, July 2017. IEEE Han K, Guo J, Zhang C, Zhu M (2019) January. Attribute-aware attention model for fifine-grained representation learning. arXiv:1901.00392 [cs], arXiv:1901.00392 Dimitris Kastaniotis I, Ntinou D, Tsourounis G, Economou, Fotopoulos S (2018) Attention aware generative adversarial networks (ata-gans). In 2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), pages 1–5. IEEE Jiahui Yu Z, Lin J, Yang X, Shen X, Lu, Thomas S Huang. Generative image inpainting with contextual attention. arXiv:1801.07892 [cs], January 2018. arXiv: 1801.07892 Simone Frintrop (2006) VOCUS: A visual attention system for object detection and goal-directed search, vol 3899. Springer Nabil Ouerhani (2003) Visual attention: from bio-inspired modeling to real-time implementation. PhD thesis, Université de Neuchâtel Leibe B, Matas J, Sebe N et al (2016) [Lecture Notes in Computer Science] Computer Vision – ECCV 2016 9908 || Identity Mappings in Deep Residual Networks[J]. Chapter 38630–645. 10.1007/978-3-319-46493-0 He K, Zhang X, Ren S et al (2016) Deep Residual Learning for Image Recognition[C]// IEEE Conference on Computer Vision and Pattern Recognition. IEEE Kukaka J, Golkov V, Cremers D (2017). Regularization for Deep Learning: A Taxonomy[J] Wen F, Chu L, Liu P et al (2018) A Survey on Nonconvex Regularization Based Sparse and Low-Rank Recovery in Signal Processing, Statistics, and Machine Learning[J] Engle R (2002) Dynamic Conditional Correlation[J]. J Bus Economic Stat 20(3):339–350 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4074358","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":278936405,"identity":"77c8e421-e7a0-4df0-bab8-bc94139cdb31","order_by":0,"name":"Hao Wu","email":"","orcid":"","institution":"Southwest University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Hao","middleName":"","lastName":"Wu","suffix":""},{"id":278936406,"identity":"5db1a533-f62b-43b0-ae9e-5b736c489225","order_by":1,"name":"Hui Wu","email":"","orcid":"","institution":"Southwest University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Hui","middleName":"","lastName":"Wu","suffix":""},{"id":278936408,"identity":"27f58692-5888-4970-83ac-14310718450a","order_by":2,"name":"xingyu su","email":"","orcid":"","institution":"Southwest University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"xingyu","middleName":"","lastName":"su","suffix":""},{"id":278936410,"identity":"64e79734-6192-4718-b33f-c09113e594a5","order_by":3,"name":"Jingjun Wu","email":"","orcid":"","institution":"Nanjing University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Jingjun","middleName":"","lastName":"Wu","suffix":""},{"id":278936412,"identity":"0712ebc5-2f18-4ff3-9c7a-e8bc1798f4e3","order_by":4,"name":"shuangli liu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABDElEQVRIiWNgGAWjYBACPmYGBmYwi52x8eEHAwk5EPvAAzxa2OBamBmbjSUqLIzBWhLwaWGAa2Fgk+A5U5HYAOLg1cLOY/y5sM0uT96ZuUFCsk0ifX7Y4YdAW+zkdBtwOYzHTHpmW3Kx4WHGBoPCNoncjbfTDIBako3NDuDWwszbxpy4sZmxIUESpGV2AkjLgcRtuLUYf+ZtqwdrOcALdJjh7PQPhLQYSPO2HU6cz8zY2MBzRiJBXjqHkC1sZdI8544nbgAGMrNEhYThBumcggMJBrj9ws9/ePNnnrLqxPnt7c9/fjCok5efnb75w4cKOzlcWuDA4AAKw4CAchCQb0BnjIJRMApGwSiAAgC+2FkFYBtoAgAAAABJRU5ErkJggg==","orcid":"","institution":"Southwest University of Science and Technology","correspondingAuthor":true,"prefix":"","firstName":"shuangli","middleName":"","lastName":"liu","suffix":""}],"badges":[],"createdAt":"2024-03-11 12:51:19","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4074358/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4074358/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":52781153,"identity":"b410f216-eab3-4a99-80ad-f12a507877d1","added_by":"auto","created_at":"2024-03-15 17:10:03","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":214889,"visible":true,"origin":"","legend":"\u003cp\u003eTarget spectrum acquisition system. When the spectral signal of the target passes through the T-encoding matrix and is converted by the detector to obtain an decoded signal.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-4074358/v1/1a21cc226d2971068ca9dfa7.png"},{"id":52780594,"identity":"2050c882-11d2-44d8-8eef-118fee6c9c1a","added_by":"auto","created_at":"2024-03-15 17:02:03","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":82212,"visible":true,"origin":"","legend":"\u003cp\u003ePrinciple of the spectral reconstruction algorithm based on the self-attention mechanism. \u003cstrong\u003ea\u003c/strong\u003e is the encoded signal received by the camera; \u003cstrong\u003eb\u003c/strong\u003e is the principle of the algorithm framework with the mathematical principle labeled next to it; \u003cstrong\u003ec\u003c/strong\u003e is the spectral signal of the final reconstructed target sample.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-4074358/v1/2da7b0ee4c9b6cc8bec603dd.png"},{"id":52780593,"identity":"9b7a395b-d0eb-45d4-bd75-c079b7e8667a","added_by":"auto","created_at":"2024-03-15 17:02:03","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":265724,"visible":true,"origin":"","legend":"\u003cp\u003ecoding matrix and simulated spectral data design. \u003cstrong\u003ea\u003c/strong\u003eis a library of filter functions obtained from FDTD simulation with 1um height and 100um-500um length and width uniform silicon material; \u003cstrong\u003eb\u003c/strong\u003e is the result obtained after preliminary filtering of the library of filter functions; \u003cstrong\u003ec\u003c/strong\u003e is the result obtained by mutual correlation function, and \u003cstrong\u003ed\u003c/strong\u003e is one of the filter functions in \u003cstrong\u003ec\u003c/strong\u003e; \u003cstrong\u003ee\u003c/strong\u003e is a smooth spectral curve simulated by Gaussian basis function. \u003cstrong\u003ef\u003c/strong\u003e is a spectral curve with 5%, 10%, 15%, 20% Gaussian noise added respectively.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-4074358/v1/d94e4d6d7a51b70d034f5421.png"},{"id":52780591,"identity":"0a905f29-01e5-4334-81cd-01808c235532","added_by":"auto","created_at":"2024-03-15 17:02:03","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":306356,"visible":true,"origin":"","legend":"\u003cp\u003eReconstruction results.\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-4074358/v1/6b2b91df98fb7f88239edcbd.png"},{"id":53093879,"identity":"4639d30b-5912-4195-b03b-1bac894f52ef","added_by":"auto","created_at":"2024-03-20 13:29:47","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1195927,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4074358/v1/504a781f-fa08-4fb7-b1f2-c52a06ff3ebe.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Reconstructing Computational Spectra Using Deep Learning's Self-Attention Method","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eThe spectrometer is widely acknowledged as a vital tool in industry and scientific research. Miniaturized spectrometer is emerging as a key platform for many developing applications, including consumer electronics, hyperspectral imaging, and in-situ sensing[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. Depending on the intricacy of the necessary algorithms, there are two primary types of spectrometer miniaturization techniques: traditional and computational[\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. A new paradigm in spectrometer miniaturization, the computational microspectrometer (CS) is based on computational spectroscopy and is anticipated to solve the limitations of conventional microspectroscopy techniques, including their inability to achieve extreme miniaturization, lack of stability, and limited resolution[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe CS relies on computational techniques to approximate or \u0026ldquo;reconstruct\u0026rdquo; an incident light spectrum from precalibrated information encoded within a set of detectors[\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. The process of reconstruction is crucial. Spectral reconstruction techniques have advanced significantly in recent years due to the growth of CS[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. In current computational reconstruction algorithms, Chang et al. use the Gaussian kernel template denoising method and parameter minimization method using the l1 paradigm for reconstruction [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. The results were compared with the l1-paradigm approach using the Tikhonov Regularized Nonnegative Least Squares (TNNLS) technique in both simulations and real data. However, the measurements largely depend on early training and Gaussian basis selection, which limits the application prospects. Conversely, Oliver et al. showed that the sparsity of the signal spectrum can be used to further improve the resolution[\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e][\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. Zhang et al. presented a reconstruction approach based on dictionary learning and sparse optimization. The experimental results imply that dictionary learning can greatly increase the sparsity of general spectra, as l1-paradigm minimization performs well for both direct sparse spectra and general spectra that need to be translated into dictionaries[\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. Yang et al. constructed the smallest microcomputing spectrometer in the world at the time by combining recovered spectral data fitted with a Gaussian basis function[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Huang et al. used a sparse encoding method and combined it with a compressed sensing algorithm to ensure the accuracy of spectral reconstruction, but the spectral resolution was not high enough [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. Huang et al. were able to increase the spectral resolution by four times by employing this property of the codec model of the spectral features fed into the network to successfully reconstruct the spectrum information of 100 bands from 25 feature points[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e][\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe CS is unable to obtain accurate readings in real-time, and the reconstruction accuracy of these methods is subpar. Recent research has demonstrated that employing deep learning algorithms for spectrum reconstruction of computational spectroscopy systems produces good results. However, because just the linear transformation of the completely linked layer itself is used and the correlation between the spectra and the spectra itself is ignored, the accuracy of the spectral reconstruction is not very great[\u003cspan additionalcitationids=\"CR16\" citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. On the other hand, it demonstrates that deep learning may be highly useful in spectrum reconstruction and that deep neural networks can learn the mapping between input and target output distributions by training a large number of samples. CNNs are more suited for processing data with local features and spatial information, like photos and videos, while the self-attention mechanism works well for processing sequential data, particularly for jobs with global dependencies. Currently, the self-attention mechanism has been widely used in language recognition [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e], text categorization [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e][\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e], speech translation [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e], natural language generation [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e], image recognition[\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e][\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e], image generation[\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e][\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e], and so on.\u003c/p\u003e \u003cp\u003eThe self-attention mechanism accomplishes adaptive weight allocation\u0026mdash;which determines the relative importance of different bands\u0026mdash;by computing attention weights. Certain spectral data bands may contain information that is redundant or noisy, while other bands may have more relevant information. The self-attention method improves reconstruction accuracy by allowing the network to autonomously assess the importance of various bands and assign them greater weights during the process. To capture global context dependence, the self-attention mechanism also evaluates the correlation between each element in the input sequence and other elements [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e] [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. Consequently, it is anticipated that this technique will raise the spectral reconstruction's accuracy even further.\u003c/p\u003e \u003cp\u003eIn this work, we address the issue of inadequate accuracy in reconstructing spectra through miniaturized computing by designing the encoding matrix using cross-correlation algorithms and combining a novel deep-learning self-attention mechanism to optimize the final reconstruction results. The spectral reconstruction accuracy of computational spectrometers will be much enhanced by this method, which will also be a significant step toward the practical use of computational spectrometers.\u003c/p\u003e"},{"header":"2. Method","content":"\u003cp\u003e \u003cb\u003ePrinciple of the data acquisition system\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThis section describes and demonstrates how to measure the reflection spectrum of a target sample under sunlight, and the simplified schematic is shown in Fig.\u0026nbsp;1. When light is shone on the target picture, a transformation occurs as follows:\u003cdiv id=\"Equ1\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ1\" name=\"EquationSource\"\u003e\n$$\\int L\\left(\\lambda \\right)S\\left(\\lambda ,X,Y\\right)T\\left(\\lambda \\right)D\\left(\\lambda \\right)d\\lambda =I\\left(X,Y\\right)$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e1\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eMakes the conversion into coded input data that can be fed into the algorithm model. In the formula, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(L\\left(\\lambda \\right)\\)\u003c/span\u003e\u003c/span\u003e represents daylight light information, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(S\\left(\\lambda ,X,Y\\right)\\)\u003c/span\u003e\u003c/span\u003eis the detection target, where \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\left(X,Y\\right)\\)\u003c/span\u003e\u003c/span\u003e is the two-dimensional position coordinates of the single-point spectrum, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(D\\left(\\lambda \\right)\\)\u003c/span\u003e\u003c/span\u003e is the detection camera's response function, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(T\\left(\\lambda \\right)\\)\u003c/span\u003e\u003c/span\u003e is the designed encoding matrix, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(I\\left(X,Y\\right)\\)\u003c/span\u003e\u003c/span\u003e is the encoding intensity of the corresponding position coordinates. This conversion process requires first pasting the designed encoding matrix in front of the camera, which is used to test the spectral information of the reflected light of the sample \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(S\\left(\\lambda ,X,Y\\right)\\)\u003c/span\u003e\u003c/span\u003e, so the spectral intensity is \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(L\\left(\\lambda \\right)S\\left(\\lambda ,X,Y\\right)\\)\u003c/span\u003e\u003c/span\u003e before entering the camera conversion. Then the encoded intensity \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(I\\left(X,Y\\right)\\)\u003c/span\u003e\u003c/span\u003e is finally obtained by the camera's encoding matrix \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(T\\left(\\lambda \\right)\\)\u003c/span\u003e\u003c/span\u003e and response function \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(D\\left(\\lambda \\right)\\)\u003c/span\u003e\u003c/span\u003e. The specified encoding matrix can be turned into a chip that fits exactly with the camera to make spectral measurement more convenient.\u003c/p\u003e \u003cp\u003e \u003cb\u003eFilter function design and construction of data set\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe data construction in this study is divided into two parts. To construct an accurate spectral reconstruction model, suitable input and output data needs to be collected. The first part of the input data is a complete library of filtering functions simulated by FDTD and set by the expectation to 0.85 as a threshold condition as an initial sieve, and then utilized the cross-correlation function:\u003cdiv id=\"Equ2\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ2\" name=\"EquationSource\"\u003e\n$$\\rho \\left(X,Y\\right)=\\frac{Cov(X,Y)}{\\sqrt{D\\left(X\\right)D\\left(Y\\right)}}=\\frac{E\\left(XY\\right)-E\\left(X\\right)E\\left(Y\\right)}{\\sqrt{E\\left({X}^{2}\\right)-{E}^{2}\\left(X\\right)}\\sqrt{E\\left({Y}^{2}\\right)-{E}^{2}\\left(Y\\right)}}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e2\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003e \u003cspan class=\"InlineEquation\"\u003e \u003cspan class=\"mathinline\"\u003e\\(X,Y\\)\u003c/span\u003e \u003c/span\u003e denote different curves respectively, the value of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(Cov\\)\u003c/span\u003e\u003c/span\u003e is the covariance of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(X,Y\\)\u003c/span\u003e\u003c/span\u003e curves, and its calculation result ρ is a scalar and the range \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\rho\\)\u003c/span\u003e\u003c/span\u003e\u0026isin;[-1,1]; when \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\rho\\)\u003c/span\u003e\u003c/span\u003e \u0026gt;0, it means positive correlation, and the closer to 1 means the stronger correlation; \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\rho\\)\u003c/span\u003e\u003c/span\u003e \u0026lt;0, it means negative correlation and the closer to -1 means the stronger correlation; when the correlation coefficient is closer to 0, the correlation is weaker, and equals to 0 when two variables are not Correlation[\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e]. Using this formula, each of the filter function curves is used to calculate the correlation between the filter functions screened out at the beginning. Finally, the 100 filter functions whose absolute value is closest to 1 are sorted out and arranged into a 10x10 coded array.\u003c/p\u003e \u003cp\u003eIn the second part of the experiment, spectral simulations were first performed using advanced basis functions to generate a series of spectral curves. These simulated spectral curves introduced varying degrees of Gaussian noise on top of the actual spectral data, including noise levels of 5%, 10%, 15%, and 20%. In total, 20,000 such simulated spectral data were generated to ensure broad coverage of noise levels and spectral features.\u003c/p\u003e \u003cp\u003eNext, these simulated spectral data were coded and transformed into uncoded input data using a pre-constructed filter function coding matrix. Then use the following formula to convert the simulated spectrum into decoded data:\u003cdiv id=\"Equ3\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ3\" name=\"EquationSource\"\u003e\n$${I}_{i}={\\int }_{{\\lambda }_{1}}^{{\\lambda }_{2}}{T}_{i}\\left(\\lambda \\right)(f\\left(\\lambda \\right)+{e}_{i})d\\lambda +\\approx \\sum _{j=1}^{M}{T}_{i}\\left({\\lambda }_{i}\\right)(f\\left({\\lambda }_{i}\\right)+{e}_{i}),i=1,\\cdots ,N$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e3\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eThe appropriate input-encoded data were obtained in this manner, where \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({T}_{i}\\left(\\lambda \\right)\\)\u003c/span\u003e\u003c/span\u003e is the encoding matrix, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(f\\left(\\lambda \\right)\\)\u003c/span\u003e\u003c/span\u003e is the spectral data, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({e}_{i}\\)\u003c/span\u003e\u003c/span\u003e is the noise level. Its integration can be decomposed into the sum of multiple points, where M is the number of measurements within the measurement band and N is the number of encodings. This step was designed to simulate the process of encoding and reconstruction of spectral data in real applications. Through this process, a total of 100,000 simulated data was obtained, which included both training and validation sets.\u003c/p\u003e \u003cp\u003eTo ensure the accuracy and robustness of the model, these 100,000 simulated data were divided into training and validation sets in a ratio of 8:2. It is worth noting that the selection of the validation set was done randomly to ensure extensive testing of different spectral data.\u003c/p\u003e \u003cp\u003eFinally, an additional 1000 simulated spectral data that were not in the training and validation sets were transformed to serve as an independent test set to evaluate the accuracy and performance of the model on unseen data. This step is important to test the performance of our model in real-world applications.\u003c/p\u003e \u003cp\u003e \u003cb\u003eThe framework of the neural network model\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe neural network architecture for spectrum reconstruction can be expressed easily as\u003c/p\u003e \u003cp\u003e \u003cspan class=\"InlineEquation\"\u003e \u003cspan class=\"mathinline\"\u003e\\(FC\\left(100\\right)\\to LR\\to Self-Attention\\left(100\\right)\\to FC\\left(120\\right)\\to LR\\to FC\\left(150\\right)\\to LR\\to FC\\left(200\\right)\\to LR\\)\u003c/span\u003e \u003c/span\u003e. Each digit represents the number of cells in the relevant layer. The ReLU function is denoted by the symbol LR. fc means a fully connected layer, 100 random spectral filters correspond to the number of input cells, and 200 denotes the reconstructed spectral channels (8um-12um, 0.02um steps). In this paradigm, the self-attention mechanism and residuals are used for training. To get more relevant features, the input incoming data is first Processed to a self-attention operation, and then residuals are incorporated in each neural layer to minimize parameter explosion and overfitting during training. Deionizing in normal compressed sensing algorithms, on the other hand, heavily relies on prior knowledge, and parameters are typically updated manually throughout the iterative process to offset the bias generated by noise, which is effective but does not provide convincing results when the noise level varies. As a result, regularization parameters are introduced to the training model to improve its robustness [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e][\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn which Self-Attention is computed in the following steps:\u003c/p\u003e\n\u003ch3\u003e1. Generate Query, Key and Value:\u003c/h3\u003e\n\u003cp\u003eFirst, the query vector (Q), key vector (K), and value vector (V) are generated by a linear transformation of the input data (usually using a weight matrix).\u003c/p\u003e \u003cp\u003eGenerating these vectors usually involves the multiplication of weight matrices to map the input data into a low-dimensional representation space for subsequent computation.\u003c/p\u003e\n\u003ch3\u003e2. Compute the attention scores:\u003c/h3\u003e\n\u003cp\u003eNext, the attention scores between each query (Q) and each key (K) are computed, usually using the dot product (Dot Product) method. For a single query vector, the dot product is computed with all key vectors, and then a scaling factor is applied to control the range of scores. The formula is as follows:\u003cdiv id=\"Equ4\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ4\" name=\"EquationSource\"\u003e\n$$Attention\\left(Q,K\\right)=Softmax\\left(\\right(Q*{K}^{\\wedge }T)/sqrt({d}_{k})$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e4\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eWhere Q is the query vector, K is the key vector and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({d}_{k}\\)\u003c/span\u003e\u003c/span\u003e is the dimension of the key vector. The scores can be converted to probability distributions by applying the Softmax function.\u003c/p\u003e\n\u003ch3\u003e3. Weighted Summation:\u003c/h3\u003e\n\u003cp\u003eThe computed attention scores perform a weighted summation on the value vector (V) to generate the final self-attention representation:\u003cdiv id=\"Equ5\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ5\" name=\"EquationSource\"\u003e\n$$Self-Attention\\left(Q,K,V\\right)=Attention\\left(Q,K\\right)*V$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e5\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eThis step involves a weighted summation of the value vectors based on the attention scores to produce a self-attention representation. Finally, the reconstructed spectral data is derived from the self-attention values in the subsequent full connectivity. The residual mechanism is added at each fully connected layer to prevent the gradient from disappearing during training. The result is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e2\u003c/span\u003eB.\u003c/p\u003e \u003cp\u003e \u003cb\u003eTraining and validation\u003c/b\u003e \u003c/p\u003e \u003cp\u003eIn our model training process, we chose the mean square error (MSE) as the loss function, which is a loss function commonly used in regression problems. The MSE measures the mean squared error between the model predictions and the actual observations. By minimizing the MSE, our model can better fit the spectral data and ensure that the resulting reconstruction is as close as possible to the real spectrum.\u003c/p\u003e \u003cp\u003eTo improve the stability and speed of training, the batch normalization technique is applied. Batch normalization helps to normalize the inputs of each layer in the neural network, ensuring that the distribution of the data is stable and reducing the problem of gradient explosion or gradient vanishing. This is very useful when training deep neural networks and helps the model to converge faster. The Adam optimization algorithm was chosen to tune the model parameters, which combines the characteristics of the momentum method and the adaptive learning rate and is usually able to find the global minimum faster.\u003c/p\u003e \u003cp\u003eIn addition, the appropriate learning rate for model training was set to ensure that the weights were updated appropriately during the training process to avoid the model falling into local minima or divergence. Meanwhile, to avoid the overfitting problem, two regularization methods are used, namely L1 regularization and drops out. L1 regularization drives the model to be sparser and reduces the possibility of overfitting by applying penalties to the weight parameters of the model. Dropout, however, randomly discards some neurons during training to prevent the model from relying too much on the training data and improve the generalization performance.\u003c/p\u003e \u003cp\u003eTo verify the performance and practical application value of the model, we used actual spectral data measured by an optical spectrometer. These data have actual physical significance and reflect the nature of the actual spectral signals, thus ensuring that the model can be accurately reconstructed in the face of spectral data in real environments, thus demonstrating the practical usability and effectiveness of the model. This experimental design helps to generalize the research results to practical applications and provides solutions to practical problems in related fields.\u003c/p\u003e \u003cp\u003e \u003cb\u003eSpectral reconstruction performance indicators\u003c/b\u003e \u003c/p\u003e \u003cp\u003eTwo related indicators were used as reconstruction indicators. The R\u003csup\u003e2\u003c/sup\u003e similarity function is used to evaluate the following metrics:\u003cdiv id=\"Equ6\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ6\" name=\"EquationSource\"\u003e\n$${R}^{2}=1-\\frac{{{{\\Sigma }}_{i=1}^{n}\\left({y}_{i}-{Y}_{i}\\right)}^{2}}{{{{\\Sigma }}_{i=1}^{n}\\left({Y}_{i}-\\stackrel{-}{Y}\\right)}^{2}}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e6\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({y}_{i}\\)\u003c/span\u003e\u003c/span\u003e is the reconstructed spectrum's intensity value, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({Y}_{i}\\)\u003c/span\u003e\u003c/span\u003e is the simulated spectrum's intensity value, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\stackrel{-}{Y}\\)\u003c/span\u003e\u003c/span\u003e is the simulated spectrum's average intensity value; as well as MSE (mean square error function):\u003cdiv id=\"Equ7\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ7\" name=\"EquationSource\"\u003e\n$$MSE=\\frac{1}{n}\\sum _{i=1}^{n}{\\left({y}_{i}-{Y}_{i}\\right)}^{2}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e7\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eWhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({y}_{i}\\)\u003c/span\u003e\u003c/span\u003e is the intensity value of the reconstructed spectrum and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({Y}_{i}\\)\u003c/span\u003e\u003c/span\u003e is the intensity value of the simulated spectrum.\u003c/p\u003e"},{"header":"3. Result","content":"\u003cp\u003e \u003cb\u003eDesign of data collection system\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThis section presents a spectral acquisition system we have designed that can collect spectral information about objects illuminated by light. It mainly involves reflecting the information of the illuminated object onto the designed spectral chip through a Formula 2 conversion and then collecting undeciphered data through a detector.\u003c/p\u003e \u003cp\u003e \u003cb\u003eSpectral reconstruction of the self-attention mechanism framework\u003c/b\u003e \u003c/p\u003e \u003cp\u003eCompared with traditional compressed sensing algorithms, deep learning neural networks have significant advantages in dealing with noise. In traditional compressed sensing algorithms, denoising often relies on a priori information, and some parameters usually need to be set manually to counteract the noise-induced bias. This method can be effective to some extent but has some limitations. When the noise level changes or the nature of the noise changes, the performance of traditional compressed sensing algorithms may suffer and fail to provide consistent and credible results.\u003c/p\u003e \u003cp\u003eDeep learning neural networks excel at noise processing because they are adaptive. This means that neural networks can learn parameters and features from the data without much manual tuning. In the presence of noise, neural networks can automatically adjust their weights and structures to better fit and model noisy data. This adaptivity allows the neural network to maintain a better fit when dealing with data with different noise levels and properties.\u003c/p\u003e \u003cp\u003eDeep learning neural networks are usually trained with large-scale data to give them strong generalization capabilities. This means that neural networks can process and reduce high-quality data even in the presence of noise not covered during training. This is important for spectral reconstruction in real-world applications, where the type and noise level can change at any time in a real environment.\u003c/p\u003e \u003cp\u003eTherefore, to achieve better reconstruction results, the self-attention mechanism and residual in deep learning were used for reconstruction. The algorithm model is depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e2\u003c/span\u003eb. To extract more valuable features from this randomly encoded input for better reconstruction, the model uses a self-attention strategy to improve the relationship between attributes. The self-attention mechanism computes the similarity between each site and other places. It uses it as a weight to weigh the data together, focusing attention on the relevant features to increase result accuracy. The residual mechanism adds cross-layer connections to the neural network layers, avoiding gradient disappearance and gradient explosion while also boosting model training [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e][\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e].\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eCoding matrix design and dataset construction\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe FDTD simulation-generated filter function library is shown in Fig.\u0026nbsp;3a, and the initial screening-generated filter function matrix is presented in Fig.\u0026nbsp;3b. The initial screening calculates the expectation and variance of each filter function to verify that each filter function has an encoding effect because there would be no light intensity change when detecting the target, preventing the spectrum from being decoded. Finally, the filter matrix is filtered again using the intercorrelation function to yield the results shown in Fig.\u0026nbsp;3c. The architecture of the generated spectral data is also depicted in Fig.\u0026nbsp;3, where varying amounts of Gaussian noise are included to imitate genuine environmental conditions to improve the training model's robustness.\u003c/p\u003e \u003cp\u003e \u003cb\u003eReconstruction results of self-attention mechanism algorithm\u003c/b\u003e \u003c/p\u003e \u003cp\u003eFig. 4. a-f demonstrates the outcomes of the attention-based algorithm model reconstruction, with the corresponding metric values annotated.\u0026nbsp;The algorithm\u0026apos;s reconstruction achieved good results with R\u003csup\u003e2\u003c/sup\u003e=0.9780 and MSE=0.0019, this is the overall average result metric for the entire test set. The results show that using the mutual correlation algorithm to filter the filter function and combining it with the self-attention mechanism network for spectral reconstruction can produce better reconstruction results, which is an important reference value for practical spectral reconstruction applications. We also tested the reconstruction results of different algorithms using our dataset, and the test results are shown in Table. 1.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison of Results of Different Algorithms\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"3\" nameend=\"c3\" namest=\"c1\"\u003e \u003cp\u003eDifferent algorithms R\u003csup\u003e2\u003c/sup\u003e MSE\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLeast Squares\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.12\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ecompressed Sensing\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.89\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.10\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDNN\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.91\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.07\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eseq2seq\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.94\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.04\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSelf-Attention\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.9780\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.0019\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e"},{"header":"4. Discussion","content":"\u003cp\u003eIn this work, we propose a new method to construct a filtering function encoding matrix and integrate the self-attention mechanism in deep learning for spectral reconstruction, to improve the accuracy and stability of the final computed spectral reconstruction. In previous coding matrix designs, there was not much emphasis on the correlation between different data. However, a filtering method based on cross-correlation functions can ensure that the constructed coding matrix is sparse enough to better express features and facilitate data processing in subsequent spectral reconstruction. In addition, in previous algorithms, spectral reconstruction was mostly carried out by directly fitting the linear data to obtain the spectral information of the object. However, in the real environment, there is some nonlinear information that can make reconstruction difficult when mixed. Our spectral reconstruction method using a self-attention mechanism not only strengthens the self-correlation between input encoded data but also fits the nonlinear data. Reduce the influence of environmental factors, thereby further improving the accuracy and stability of the spectral reconstruction.\u003c/p\u003e \u003cp\u003eOur research first uses the filtering method of cross-correlation function to filter the filtering function library simulated by FDTD, to screen out the filtering functions with cross-correlation characteristics, to efficiently construct the sparse encoding matrix of the filtering function. The sparse nature of data can make the encoding matrix more expressive; help extract effective information from spectral data and provide a better data foundation for subsequent spectral reconstruction. This filtering method also helps to reduce computational complexity and improve computational efficiency.\u003c/p\u003e \u003cp\u003eSecondly, we adopted the popular self-attention mechanism in the field of deep learning to enhance the feature correlation between input-encoded data, enabling the model to better capture complex nonlinear data relationships. This plays an important role in spectral reconstruction, as spectral data typically contains complex correlations between multiple bands. The self-attention mechanism effectively improves the accuracy of spectral reconstruction by adaptively learning the correlation between bands.\u003c/p\u003e \u003cp\u003eIn addition, a residual network was introduced during training to prevent overfitting issues during the training process. This mechanism can enable the model to better record and preserve the good results of the previous step, especially in deep networks, which helps to improve the model's generalization ability; that is, it can still obtain good reconstruction results in complex environments.\u003c/p\u003e \u003cp\u003eThe experimental results show that the algorithm combining the self-attention mechanism has higher accuracy and better performance compared to other algorithms and can improve the accuracy and stability of the spectral reconstruction. This method is not only suitable for simulating data but also provides strong support for its application in real-world environments. In the future, we will consider expanding this research with more diverse application scenarios and datasets to further improve stability and accuracy.\u003c/p\u003e"},{"header":"5. Conclusion","content":" \u003cp\u003eIn this work, we propose a new method to construct a filtering function encoding matrix and integrate the self-attention mechanism in deep learning for spectral reconstruction, ultimately achieving good accuracy and stability in the reconstruction. Our work first utilizes the method of cross-correlation function to ensure that the constructed encoding matrix can better express features and improve the accuracy and performance of subsequent reconstruction results. Then, a combination of self-attention mechanism and residual is used to enhance the self-correlation between input encoded data, and non-linear data is fitted to reduce the influence of environmental factors, thereby further improving the accuracy and stability of spectral reconstruction. The results show that the reconstruction algorithm is significantly superior to other algorithms, achieving good results with R\u003csup\u003e2\u003c/sup\u003e\u0026thinsp;=\u0026thinsp;0.9780 and MSE\u0026thinsp;=\u0026thinsp;0.0019. To further validate and enhance the approach for the development of spectrum reconstruction approaches in the future, more experiments and practical application testing are needed. This work offers novel insights into how deep learning and self-attention mechanisms might be used to enhance the accuracy of spectral reconstruction. These findings could have a broad variety of implications for future investigations and real-world applications in spectroscopy and related domains.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e \u003ch2\u003eConflicts of Interest:\u003c/h2\u003e \u003cp\u003eAll authors disclosed no relevant relationships.\u003c/p\u003e \u003c/p\u003eFunding Declaration: This is an extended study conducted independently with no funding support.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eH.W. wrote and created the associated algorithms and gathered data. H. W. supplied the FDTD simulation dataset. X.S. processed and annotated certain pertinent data. S. L. and J. W. provide suggestions and directions for manuscript revisions.\u003c/p\u003e\u003ch2\u003eData Availability Statement:\u003c/h2\u003e \u003cp\u003eThe dataset is confidential and therefore not easy to disclose.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eYang Z et al (2019) Single-nanowire spectrometers. Science 365, 1017\u0026ndash;1020 \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1126/science.aax8814\u003c/span\u003e\u003cspan address=\"10.1126/science.aax8814\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e; pmid: 31488686\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSchuler LP, Milne JS, Dell JM, Faraone L (2009) MEMS-based microspectrometer technologies for NIR and MIR wavelengths. J Phys D 42:133001. 10.1088/ 0022-3727/42/13/133001\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMalinen J et al (2014) Advances in miniature spectrometer and sensor development. Proc. SPIE 9101, 91010C \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1117/12.2053567\u003c/span\u003e\u003cspan address=\"10.1117/12.2053567\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eM.Ebermann et al., Tunable MEMS Fabry-P\u0026eacute;rot filters for infrared microspectrometers: A review. Proc. SPIE 9760, 97600H (2016) \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1117/12.2209288\u003c/span\u003e\u003cspan address=\"10.1117/12.2209288\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCrocombe RA (2018) Portable Spectrosc Appl Spectrosc 72:1701\u0026ndash;1751 doi:.1177/0003702818809719; pmid: 30335465\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWolffenbuttel RF (2005) MEMS-based optical mini- and microspectrometers for the visible and infrared spectral range. J Micromech Microeng 15:S145\u0026ndash;S152. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1088/0960-1317/15/7/021\u003c/span\u003e\u003cspan address=\"10.1088/0960-1317/15/7/021\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKurokawa U, Choi BI, Chang CC (2011) Filter-based miniature spectrometers: spectrum reconstruction using adaptive regularization. IEEE Sens J 11(7):1556\u0026ndash;1563. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1109/jsen.2010.2103054\u003c/span\u003e\u003cspan address=\"10.1109/jsen.2010.2103054\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOliver J, Lee W, Park S et al (2012) Improving resolution of miniature spectrometers by exploiting sparse nature of signals. Opt Expr 20(3):2613\u0026ndash;2625\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOliver J, Lee WB, Lee HN (2013) Filters with random transmittance for improving resolution in filter-array-based spectrometers. Opt Expr 21(4):3969\u0026ndash;3989. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1364/oe.21.003969\u003c/span\u003e\u003cspan address=\"10.1364/oe.21.003969\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eA Spectral Reconstruction Algorithm of Miniature (2018) Spectrometer Based on Sparse Optimization and Dictionary Learning. Shang Zhang, Yuhan Dong\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMiniaturization of optical spectrometers ZongYin Yang SCIENCE 29 Jan 2021 Vol 371, Issue 6528.\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1126/science.abe0722\u003c/span\u003e\u003cspan address=\"10.1126/science.abe0722\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOne-shot ultraspectral imaging with reconfigurable metasurfaces (2019) Xusheng Cai, Jian Xiong, Kaiyu Cui\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang J, Cui K, Cai X et al (2022) Ultraspectral Imaging Based on Metasurfaces with Freeform Shaped Meta-Atoms[J]. Laser \u0026amp; Photonics Reviews, p 16. 7\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBaraniuk RG (2007) Compressive sensing. IEEE Signal Process Mag 24:118\u0026ndash;121\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGoodfellow I, Bengio Y, Courville A (2016) Deep Learning 1. MIT Press\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSong HY et al (2021) Deep-learned broadband encoding stochastic fifilters for computational spectroscopic instruments. Adv Theory Simul 4:2000299\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNie SJ et al (2018) IEEE, Salt Lake City, UT, USA,. Deeply learned fifilter response functions for hyperspectral reconstruction. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 18\u0026ndash;23 June 2018\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang J, Zhou W, Zhang Q, Li H, Li W (2018) Video-based sign language recognition without temporal segmentation. In Thirty-Second AAAI Conference on Artifificial Intelligence\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGang Liu and Jiabao Guo (2019) Bidirectional lstm with attention mechanism and convolutional layer for text classifification. Neurocomputing 337:325\u0026ndash;338\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKarim, Ahmed (2017) Nitish Shirish Keskar, and Richard Socher. Weighted transformer network for machine translation. arXiv preprint arXiv:1711.02132\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSperber M, Neubig G, Niehues J, Waibel A (2019) Attention-passing models for robust and data-effificient end-to-end speech translation. Trans Association Comput Linguistics 7:313\u0026ndash;325\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eXu K, Wu L, Wang Z, Feng Y, Witbrock M, Sheinin V (2018) Graph2seq: Graph to sequence learning with attention-based neural networks. arXiv preprint arXiv:1804.00823\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFu J, Zheng H, Mei T Look closer to see better: Recurrent attention convolutional neural network for fifine-grained image recognition. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4476\u0026ndash;4484, Honolulu, HI, July 2017. IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHan K, Guo J, Zhang C, Zhu M (2019) January. Attribute-aware attention model for fifine-grained representation learning. arXiv:1901.00392 [cs], arXiv:1901.00392\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDimitris Kastaniotis I, Ntinou D, Tsourounis G, Economou, Fotopoulos S (2018) Attention aware generative adversarial networks (ata-gans). In 2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), pages 1\u0026ndash;5. IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJiahui Yu Z, Lin J, Yang X, Shen X, Lu, Thomas S Huang. Generative image inpainting with contextual attention. arXiv:1801.07892 [cs], January 2018. arXiv: 1801.07892\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSimone Frintrop (2006) VOCUS: A visual attention system for object detection and goal-directed search, vol 3899. Springer\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNabil Ouerhani (2003) Visual attention: from bio-inspired modeling to real-time implementation. PhD thesis, Universit\u0026eacute; de Neuch\u0026acirc;tel\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLeibe B, Matas J, Sebe N et al (2016) [Lecture Notes in Computer Science] Computer Vision \u0026ndash; ECCV 2016 9908 || Identity Mappings in Deep Residual Networks[J]. Chapter 38630\u0026ndash;645. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/978-3-319-46493-0\u003c/span\u003e\u003cspan address=\"10.1007/978-3-319-46493-0\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHe K, Zhang X, Ren S et al (2016) Deep Residual Learning for Image Recognition[C]// IEEE Conference on Computer Vision and Pattern Recognition. IEEE\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKukaka J, Golkov V, Cremers D (2017). Regularization for Deep Learning: A Taxonomy[J]\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWen F, Chu L, Liu P et al (2018) A Survey on Nonconvex Regularization Based Sparse and Low-Rank Recovery in Signal Processing, Statistics, and Machine Learning[J]\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEngle R (2002) Dynamic Conditional Correlation[J]. J Bus Economic Stat 20(3):339\u0026ndash;350\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Spectral Reconstruction, Self-Attention, Encoding Matrix, Cross-Correlation","lastPublishedDoi":"10.21203/rs.3.rs-4074358/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4074358/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eMiniaturized computational spectrometers have become a new research hotspot due to their portability and miniaturization. However, there are several issues, like low precision and poor stability. Because the problem of spectrum reconstruction accuracy is very evident, we suggested a novel approach to raise the reconstruction accuracy. Using the time-domain finite-difference (FDTD) method, a library of transmittance functions was acquired. A cross-correlation algorithm was then used to choose 100 sparse transmittance functions, which were then built as an encoding matrix. Then, based on the encoding matrix, a self-attention mechanism algorithm was used to improve the accuracy of the reconstructed spectrum by improving the autocorrelation characteristics between input signals. The mean square error (MSE) of the reconstructed spectrum is 0.0019, and its similarity coefficient (R\u003csup\u003e2\u003c/sup\u003e) is 0.9780. This self-attention mechanism spectral reconstruction technique will open up new possibilities for high-accuracy reconstruction for a variety of computational spectrometer types.\u003c/p\u003e","manuscriptTitle":"Reconstructing Computational Spectra Using Deep Learning's Self-Attention Method","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-03-15 17:01:58","doi":"10.21203/rs.3.rs-4074358/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"678edde7-8fa3-42ef-a30b-8538e20e3f18","owner":[],"postedDate":"March 15th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-03-20T13:21:39+00:00","versionOfRecord":[],"versionCreatedAt":"2024-03-15 17:01:58","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4074358","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4074358","identity":"rs-4074358","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}