Fe-NODE: Feature extension Neural Ordinary Differential Equations for Modeling Black-box Control Systems

Fe-NODE: Feature extension Neural Ordinary Differential Equations for Modeling Black-box Control Systems | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Fe-NODE: Feature extension Neural Ordinary Differential Equations for Modeling Black-box Control Systems Zhehao Xia, Yizhong Wu, Haoyu Xu, Yihang Xing This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5030219/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 3 You are reading this latest preprint version Abstract This paper introduces a novel Feature-Extended Neural Ordinary Differential Equation (Fe-NODE) method for modeling black-box control systems by incorporating control parameters into the ordinary differential equation. Since the input to the model should consider both the system state and the control parameters, this is a challenge to the modeling process. The primary contribution of the proposed method is employing feature expansion to add information to the model input and enhance the perception ability of the model while utilizing intermediate state supervision and teacher forcing to optimize the training process, and effectively identify potential models for black-box control systems. The experimental results demonstrate that the Fe-NODE method outperforms the baseline in prediction accuracy and modeling efficiency. Additionally, we investigate the impact of different feature expansion coefficients on the model’s prediction accuracy, identifying the optimal coefficient through comparative analysis. This approach provides a novel methodology for modeling black-box control systems and lays the foundation for model-based reinforcement learning to achieve optimal control. Neural ODE Feature extension Black-box control systems System identification. Figures Figure 1 Figure 2 Figure 3 Figure 4 1. Introduction The state-space model [ 1 , 2 ] can describe the evolution of dynamic systems. This model is widely used in engineering, physics, system analysis, and control design. A crucial component of the state-space model is the state equation, which provides a mathematical description of the time evolution of system states. Based on an accurate state equation, dynamic programming [ 1 ], model predictive control [ 2 ], or model-based reinforcement learning [ 3 ] can achieve optimal control. The state equation is typically expressed as differential equations in continuous-time dynamic systems. We can relatively easily construct the state equation in linear systems based on physical principles or empirical dynamic models [ 4 – 6 ]. However, these methods can be restrictive when applied to system identification of complex nonlinear systems. Recent research has made significant strides in machine learning (ML) approaches for solving differential equations [7], which has made nonlinear ML models a potential candidate for solving system identification tasks. Raissi [ 8 ] introduces a physics-informed neural network (PINN) that is trained to solve supervised learning tasks by incorporating underlying physics principles. Brunton [ 9 ] proposed the sparse identification of a nonlinear dynamics system (SINDy), which uses sparse regression techniques to identify nonlinear differential equations of a dynamic system from measured data. Neural state-space models (NSSMs) [10–12] have demonstrated flexibility, allowing varying degrees of domain knowledge to be embedded in the model formulation. Deep learning models have also been developed to obtain differential equations with unknown derivative values [ 13 , 14 ]. Chen [15] proposed the neural ordinary differential equation (NODE) method based on a residual network, and constructed the state equation by sequence states, providing a way to obtain the state equation of uncontrolled dynamic systems. Some researchers have also begun to work on variants of the NODE method, such as Park [ 16 ] made an in-depth analysis of NODE and applied it to clarify the influence of several design choices on the underlying dynamics. Tzen [ 17 ] adds an independent Gaussian perturbation into NODE to solve a stochastic differential equation of the diffusion process. Other studies have also explored the potential of neural networks to solve the differential equation of dynamic systems. For example, Chen [ 18 ] proposed the symplectic recurrent neural network (SRNN) to capture the dynamics of physical systems from regularly observed data. Long [ 19 ] proposed a new feed-forward deep network, called partial differential equations networks (PDENet), to fulfill two objectives at the same time: to accurately predict the dynamics of complex systems and to uncover the underlying hidden PDE models. PDE-Net has the potential to uncover the hidden PDE of the observed dynamics and predict the dynamical behavior for a relatively long time. Sorourifar [ 20 ] introduced a framework of physics-enhanced NODE that enables a combination of partially known mechanistic models with universal surrogate approximators, which enables scientific-based learning from limited datasets. Li [ 21 ] proposed a hybrid structure modeling strategy combined with a neural network model is proposed. A continuous time state-space neural network (CSNN) model is used to represent the model, and the physical model is coupled by the state-space substructure method. Gorodetskiy [ 22 ] used NODE to offer a continuous-time formulation that captures the temporal dependencies. The NODE model is trained to predict the movement of the arm and is used to generate trajectories for model-based policy optimization. However, for a black-box dynamic system containing control variables, solving its state equation is a challenge, some researchers have conducted studies in this area [ 23 ]. Alvarez [ 24 ] proposed the Dy-NODE method which integrated the control variables into the input of the model through the concatenation method and then used the generated state equation optimal control. Li [ 25 ] proposes a novel approach by combining a symbolic neural network (S-Net) with NODE to solve differential equations containing exogenous inputs. To better integrate control parameters into the state equation of black-box control systems, we were inspired by the feature pyramid network (FPN) in the field of image processing [26] and proposed the feature extension neural ordinary differential equation (Fe-NODE) method, which is used to construct more accurate state equation of control system by adding richer features to the model. The innovation of this paper mainly lies in the following aspects: (1) Based on NODE, the "Intermediate State Supervision" and "Teacher Forcing" methods are adopted to train the model, which is suitable for the training of the equation of state model. (2) Combining polynomial extension with the NODE method, the Fe-NODE method is proposed to enrich the representation ability of input features (3) Through multiple tests of numerical examples and black-box examples, the proposed method effectively improves the accuracy of the state equation, the model accuracy is higher, and the optimal feature expansion coefficient is obtained by comparison. The remainder of this paper is organized as follows: Sect. 2 introduces the related work. In the Sect. 3, we explain our proposed Fe-NODE method. Sect. 4 contrasts the proposed approach with the baseline approach in four examples, and in Sect. 5 we summarize our findings and discuss potential avenues for future research. 2. Related work The dynamic system satisfies the Markov property [ 27 , 28 ], the value of the current state is related only to the previous state, not to the previous sequence of states. The dynamic behavior can usually be described by ordinary differential equations, for a continuous dynamic system without control, the state equation can generally be expressed as: $${\dot {x}_t}=f\left( {{x_t},\theta } \right)$$ 1 Where \({x_t}\) denotes the system state and \({\dot {x}_t}\) is the derivative of the system state, is a nonlinear function that describes the dynamic behavior, and \(\theta\) is the parameter corresponding to the nonlinear function. According to the characteristics of ordinary differential equations, we can use the integral method to obtain the system state of the next moment. $${\hat {x}_{t+1}}={x_t}+\int_{t}^{{t+1}} {f({x_t},\theta )} dt$$ 2 The above formula can obtain the time series data of system states. When the exact value of derivative \({\dot {x}_t}\) is unknown, it is difficult to train the parameters \(\theta\) of nonlinear functions. The NODE method can deal with this issue, the specific implementation method is to obtain the predicted value of the last moment state by integration, and also obtain the last moment state by black-box dynamic system simulation. The error is obtained by comparing the above two state values, and then adjusting the parameters \(\theta\) of the nonlinear function to reduce the error. The NODE method uses the adjoint method to train function parameters, so that function can be generated faster. The expression of error is as follows: $$Loss=\frac{1}{m}\sum\limits_{{j=1}}^{m} {({x_T} - {{\dot {x}}_T})}$$ 3 Where represents the dimension of the state, \({x_T}\) is the state obtained by simulation at the last moment, and \({\dot {x}_T}\) is the predicted state obtained by the NODE method. The NODE method provides an approach to obtain the state equation without control when the derivative value is unknown. Some researchers have adopted the concatenation method to add control parameters into the model input and obtain the state equation containing control. However, only the method of concatenation does not fully consider the complex underlying relationship between the control and the state in the model inputs, resulting in less information contained in the model input, which affects the estimation accuracy of the state equation. 3. Method In this section, we provided a detailed description of the proposed Fe-NODE method. The state equation of the black-box control system contains the system state and control, which can be expressed as: $${\dot {x}_t}=f\left( {{x_t},{u_t},\theta } \right)$$ 4 Where the system state \({x_t}\) and control \({u_t}\) are contained in the input of the state equation, in the case that the corresponding derivative value is unknown, our goal is to extract information from the sequence system states to obtain the state equation. Firstly, we have the initial system state \({x_0}\) and the neural network that replaces the state equation, and then we can obtain the next system state by using the integral method: $${\hat {x}_{t+1}}={x_t}+\int_{t}^{{t+1}} {f({x_t},{u_t},\theta )} dt$$ 5 An important assumption in this numerical evaluation is that, in the control systems, the control parameters remain constant for a small interval of time (between the current system state and the next system state). We use the numerical evaluation to collect data as shown in Fig. 1 . Then, the loss is obtained by comparing the predicted state and the actual state, the parameters of the neural network are adjusted to reduce the loss, and the parameters of the neural network are determined by a large number of iterations. The conventional NODE method obtains loss by comparing the real state and the predicted state at the last moment and adopts the adjoint method to update the model parameters. However, this approach ignores the state of the intermediate moment, which may have an important impact on improving the accuracy and stability of the model. To solve this problem, we made the following improvements. Firstly, we adopted the "Intermediate State Supervision" method [ 29 ] to compare the real state and the predicted state at multiple moments during the training process. Intermediate state monitoring by directly monitoring multiple intermediate states of the model during training, the problem of gradient disappearing or exploding can be effectively reduced. This approach allows the model to receive feedback at each time step, helping the model better learn the characteristics of the dynamic system. Its loss function is the average value of the absolute difference between the predicted state and the real state in the whole rolling process, which can be expressed as: $$Loss=\frac{1}{m}\sum\limits_{{j=1}}^{m} {(\frac{1}{n}\sum\limits_{{i=1}}^{n} {\left| {{x_i} - {{\hat {x}}_i}} \right|} )}$$ 6 Where the represents the length of the sequence and the represents the dimension of the system state. Secondly, we use the "Teacher Forcing" strategy [ 30 ] in the model training process, instead of using the model’s previous output as input to the next time step, the truth output from the training data is provided as input. This strategy is intended to help the model learn the correct sequence of output tokens more efficiently and stabilize the training process, which is often used in recurrent neural networks [ 31 ]. The use of the above two methods can effectively optimize the training process and improve the accuracy of the proxy model. To further improve the accuracy of the agent model, we are inspired by the FPN in the field of image processing and propose the Fe-NODE method for data feature expansion, which is the main research innovation made in this paper. For two-dimensional image data, FPN uses a special convolution kernel (such as VGG) to extract features. The data to be processed in this paper is one-dimensional data, which is not suitable for extracting features by convolution kernels, and the polynomial extension is more suitable for processing one-dimensional data. The new features are generated by a polynomial combination of the input, including the power series of the input and the interaction terms between the input. By setting the coefficient of polynomial expansion, we can control the highest order of the model input. In this paper, we compared the model estimation accuracy corresponding to three different polynomial coefficients, and the training process of the proposed Fe-NODE is shown in algorithm 1. Algorithm 1: Fe-NODE 1: Gather data set of random trajectories 2: Normalize dynamics model 3: for iter = 1 to max_iter do 4: for t = 0 to n do 5: x = PolynomialFeatures(x,degree = 2,3,4) 6: x = ODESolver( f, x, u) 7: Append x to X 8: end for 9: loss = MSEloss(X,X’) 10: Train f by performing gradient descent 11: end for In summary, the proposed Fe-NODE algorithm consists of three parts: feature extension, state equation neural network, and integrator process. We use Fe-NODE to obtain an estimate of the state sequence. Then the black box control system is simulated and the true value of the state sequence is obtained. The loss value is obtained by comparing the true value and the estimated value, and the state equation neural network is iteratively updated. The training process of the Fe-NODE parameters is shown in Fig. 2 . 4. Result In this section, we conducted a series of experiments on modeling continuous control systems in four environments to test the performance of Fe-NODE. Two of the environments are numerical examples with control parameters. The state equation for the first numerical example is as follows: $$\begin{gathered} {{\dot {x}}_1}={x_2} \hfill \\ {{\dot {x}}_2}={\text{g}} \cdot \sin (u) \hfill \\ \end{gathered}$$ 7 It contains two state parameters and one control parameter. The state equation for the second numerical example is as follows: $$\begin{gathered} {{\dot {x}}_1}={x_3} \cdot \sin (u) \hfill \\ {{\dot {x}}_2}= - {x_3} \cdot \cos (u) \hfill \\ {{\dot {x}}_3}={\text{g}} \cdot \cos (u) \hfill \\ \end{gathered}$$ 8 It contains three state parameters and one control parameter. The other two examples come from a black-box dynamic model with controls in OpenAI Gym [ 32 ]. For numerical examples, we use the integral method to obtain the sequence system state, and the black-box model uses the physical system simulator to obtain the data. We take a data-driven approach to obtaining the model, without combining physical or empirical models. Therefore, we compared the similar Dy-NODE method as the baseline algorithm, and different polynomial parameters were used to expand the input features and compare the influence of different polynomial parameters on the accuracy of the model. 4.1 Experimental setup We used a uniform random policy to collect samples from the environment for training and evaluation. As mentioned in section 3, we adopted the “Teacher Forcing” strategy during training, which uses the real value as the input of the model. In the testing process, we used the predicted value as the input of the model at the next moment, which is a more satisfying practical application for control problems which is difficult to obtain real-time feedback. In the model training process, each sample contains 40 to 200 sequences of 100 steps. For closed-loop dynamics prediction, 10 sequences of 100 steps were collected and 5 random numbers to construct multiple validation sets to comprehensively compare the prediction effect of the model. We use Torch to train the model, each method using the same number of neural network layers and neurons, training 100 epochs to obtain the model. We use the Polynomialfeatures function in Sklearn to implement the input feature extension. The influence of different polynomial parameters on the prediction accuracy of the model is compared. 4.2 Test result In Fig. 3 , we show the overall dependency of the root mean square error (RMSE) concerning the number of samples collected from the environment. The RMSE was computed over all state dimensions, across the entire length of the sequence. It is shown from the figure that the estimation error of the test set decreases with the increase of the training set, indicating that more training sets will effectively improve the estimation accuracy of the model. Several examples show that when the training set reaches a certain scale, the prediction accuracy of the model presents a relatively stable state. As can be seen from Fig. 3 , the proposed Fe-NODE method outperforms the baseline in terms of prediction accuracy. The proposed method uses fewer training sets to obtain an accurate model and has higher modeling efficiency than the baseline method. We compare the effect of three different feature extensive coefficients on the model and find that the model has the highest prediction accuracy when the feature extensive coefficient is four. To further demonstrate the advantages of the algorithm, we use the same number of training sets (60 samples) for each environment to construct the model and obtain the change in the cumulative prediction error as the sequence length increases. Figure 4 shows that the Fe-NODE method with a feature extension coefficient is four has high accuracy, the results are consistent with those in Fig. 3 . Indicating that the method reduces the influence of error accumulation on sequence length. Table 1 compares the accuracy of the models under different training sets of different sizes, and the results also prove that Fe-NODE is better than the baseline method, and in most cases, the prediction effect of the model is better when the feature coefficient is four, which is consistent with the results shown in Fig. 3 – 4 . Table 1 RMSE for the four environments of the Fe-NODE No.Samples Methods Example 1 Example 2 Pendulum Acrobot 60 Dy-NODE 1.30E-03 7.06E-06 4.30E-03 1.50E-04 Fe-NODE,2 2.69E-04 3.37E-06 4.34E-04 1.34E-04 Fe-NODE,3 1.30e-04 5.57E-06 4.43E-04 9.97E-05 Fe-NODE,4 1.11e-04 2.42E-06 3.08E-04 1.09E-04 120 Dy-NODE 1.54E-04 2.93E-06 1.81E-04 1.39E-04 Fe-NODE,2 2.64E-05 8.31E-07 4.32E-05 5.52E-05 Fe-NODE,3 1.39E-05 4.10E-06 4.48E-05 2.91E-05 Fe-NODE,4 5.55E-06 4.07E-06 2.47E-05 3.30E-05 200 Dy-NODE 1.58e-05 4.42e-06 5.97e-05 1.05e-04 Fe-NODE,2 7.80e-06 1.20e-06 2.59e-05 2.49e-05 Fe-NODE,3 3.83e-06 4.77e-07 2.15e-05 1.35e-05 Fe-NODE,4 6.11e-06 2.73e-06 7.34e-06 9.98e-06 The above results prove that the Fe-NODE method proposed in this paper is superior to the benchmark method in terms of model accuracy and training efficiency. We compare the effect of different feature parameters on the model through several examples, and the model has high accuracy when the feature extension coefficient is four. 5. Conclusion We propose a novel Fe-NODE method designed to effectively model black-box control systems. The primary innovation of this paper lies in extending the characteristics of the model inputs, enabling it to capture complex dynamic behaviors more efficiently and enhance the prediction accuracy of the model. During model training, we employ the strategies of intermediate state supervision and teacher forcing. Intermediate state supervision offers multilevel feedback, enhancing the model’s learning ability, while teacher coercion reduces error propagation during time series generation. These methods collectively increase the model’s efficiency and stability during training. We validate the effectiveness of the algorithm through four environments, including numerical examples and black-box control system examples. Results indicate that the proposed method achieves higher prediction accuracy than the baseline, allowing for more accurate models to be obtained with a smaller training set and enhanced modeling efficiency. Furthermore, we compare the effects of various feature extensive parameters on the model prediction accuracy, revealing that the model prediction accuracy is highest when the feature extension coefficient is four. The polynomial extension method increases the model input dimension with rising coefficients, consequently extending the model training time. Therefore, future research will focus on optimizing this method to reduce model training time while maintaining accuracy. Declarations Conflict of interest The authors declare that they have no conflict of interest. Funding This work was supported by the Equipment Pre-Research Common Technology Area Foundation of China (Grant No. 50923010101). Author Contribution Xia wrote the main manumanuscript text, Wu guided the reasearch of this paper, Xu and Xing prepared figures 1-2 . All authors reviewed the manuscript. Data Availability The Python codes and the datasets generated during the current studies are available from the corresponding author upon reasonable request. References Ghahramani, Z., Hinton, G.E.: Variational Learning for Switching State-Space Models. Neural. Comput. 12, 831-864 (2004). https://doi.org/10.1162/089976600300015619 Kantas, N., Doucet, A., Singh, S.S.: On Particle Methods for Parameter Estimation in State Space Models. Stat. Sci. 30, 328-351 (2014). https://doi.org/10.1214/14-STS511 Dadebo, S., Luus, R.: Optimal control of time‐delay systems by dynamic programming. Optim. Control Appl. Methods. 13, 29-41 (1992). https://doi.org/10.1002/oca.4660130103 Zanon, M., Lars, G., Diehl, M.: Periodic Optimal Control, Dissipativity and MPC. IEEE Trans. on Autom. Control (2017). https://doi.org/10.1109/TAC.2016.2601881 Kamalapurkar, R., Walters, P., Dixon, W.E.: Model-based reinforcement learning for approximate optimal regulation. Control Complex Syst. 64, 247-273 (2016). https://doi.org/10.1016/j.automatica.2015.10.039 Jarzebowska, E., Augustynek, K., Urbas, A.: Dynamics modeling method dedicated to system models with open- and closed-loop structures subjected to kinematic and task based constraints. Nonlinear Dyn. 111, 12053–12080 (2023). https://doi.org/10.1007/s11071-023-08471-1 Park, J., Pao, G.M., Sugihara, G.: Empirical mode modeling A data-driven approach to recover and forecast nonlinear dynamics from noisy data. Nonlinear Dyn. 108:2147–2160 (2022). https://doi.org/10.1007/s11071-022-07311-y Chang, C.W., Ushio, M., Hsieh, C.H.: Empirical dynamic modeling for beginners.Ecol. Res. 32, 1-12 (2017). https://doi.org/10.1007/s11284-017-1469-9 Kidger, P., Morrill, J., Foster, J.: Neural Controlled Differential Equations for Irregular Time Series. arXiv (2020). https://doi.org/10.48550/arXiv.2005.08926 Raissi, M., Perdikaris, P., Karniadakis G.E.: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686-707 (2019). https://doi.org/10.1016/j.jcp.2018.10.045 Brunton, S.L., Proctor, J.L., Kutz, J.N.: Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl. Acad. Sci. 113, 3932-3937 (2016). https://doi.org/10.1073/pnas.1517384113 Rangapuram, S.S., Seeger. M.W., Gasthaus, J.: Deep State Space Models for Time Series Forecasting. Neural Information Processing Systems (2018) Masti, D., Bemporad, A.: Learning Nonlinear State-Space Models Using Deep Autoencoders. IEEE Conference on Decision and Control (2018). https://doi.org/10.1109/CDC.2018.8619475 Skomski, E., Ján, D., Tuor A.: Automating Discovery of Physics-Informed Neural State Space Models via Learning and Evolution. Proceedings of the 3rd Conference on Learning for Dynamics and Control. 144, 980-991 (2021) Rahman, A., Drgoňa, J., Tuor, A.: Neural Ordinary Differential Equations for Nonlinear System Identification. American Control Conference (2022). https://doi.org/10.23919/ACC53348.2022.9867586 Herrera, C., Krach, F., Teichmann, J.: Neural jump ordinary differential equations: Consistent continuous-time prediction and filtering. International Conference on Learning Representation (2021) Chen, R.T.Q., Rubanova, Y., Bettencourt, J.: Neural ordinary differential equations. Advances in neural information processing systems. (2018) Massaroli, S., Poli, M., Park, J.: Dissecting neural odes. Advances in Neural Information Processing Systems. 33, 3952-3963 (2020) Tzen, B., Raginsky, M.: Neural stochastic differential equations: Deep latent Gaussian models in the diffusion limit. arXiv (2019). https://doi.org/10.48550/arXiv.1905.09883 Chen, Z., Zhang, J., Arjovsky, M.: Symplectic recurrent neural networks. arXiv (2019). https://doi.org/arXiv:1909.13334, 2019 Long, Z., Lu, Y., Ma, X.: Pde-net: Learning pdes from data. International conference on machine learning. 3208-3216 (2018) Sorourifar, F., Peng, Y., Castillo, I.: Physics-Enhanced neural ordinary differential equations: Application to industrial chemical reaction systems. Ind. Eng. Chem. Res. 62, 15563-15577(2023) Li, H.W., Hao, S., Ni, Y.Q.: Hybrid structural analysis integrating physical model and continuous time state‐space neural network model. Comput.-Aided Civ. Infrastruct. Eng. (2024) Gorodetskiy, A., Mironov, K., Panov A.: Model-Based Policy Optimization with Neural Differential Equations for Robotic Arm Control. International Conference on Interactive Collaborative Robotics. 258-266 (2023) Hashimoto, T., Matsui, N., Kamiura, N.: Exploring Model Structures to Reduce Data Requirements for Neural ODE Learning in Control Systems. J. Adv. Comput. Intell. Intell. Inform. 27, 537-542 (2023) Alvarez, V.M.M., Roşca, R., Fălcuţescu, C.G.: Dynode: Neural ordinary differential equations for dynamics modeling in continuous control. arXiv (2020). https://doi.org/10.48550/arXiv.2009.04278 Li, Q., Evje, S., Geng, J.: Learning parameterized odes from data. IEEE Access. 11, 54897-54909 (2023) Lin, T.Y., Dollár, P., Girshick, R.: Feature pyramid networks for object detection. Proceedings of the IEEE conference on computer vision and pattern recognition. 2117-2125 (2017) Chen, C., Hong, J., Lu, Y.: Stochastic differential equation with piecewise continuous arguments: Markov property, invariant measure and numerical approximation. Discreet. Contin. Dyn. Syst. Ser. B. 28, 765–807 (2023). https://doi.org/10.3934/dcdsb.2022098 Wu, F., Yin, G., Mei, H.: Stochastic functional differential equations with infinite delay: Existence and uniqueness of solutions, solution maps, Markov properties, and ergodicity. J. Differ. Equ. 262, 1226-1252 (2017) Li, R., Wang, X., Huang, G.: A comprehensive review on deep supervision: Theories and applications. arXiv (2022). https://arxiv.org/pdf/2207.02376 Feng, Y., Gu, S., Guo, D.: Guiding teacher forcing with seer forcing for neural machine translation. arXiv (2021) Lamb, A., Anirudh, G.: Professor forcing: A new algorithm for training recurrent networks. Advances in neural information processing systems, 29 (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym. arXiv (2016). https://arxiv.org/abs/1606. 01540. cite arxiv:1606.01540 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editor assigned by journal 05 Sep, 2024 Submission checks completed at journal 05 Sep, 2024 First submitted to journal 04 Sep, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5030219","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":349951760,"identity":"03bb111e-4390-40df-a063-b0c284302292","order_by":0,"name":"Zhehao Xia","email":"","orcid":"","institution":"Huazhong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Zhehao","middleName":"","lastName":"Xia","suffix":""},{"id":349951761,"identity":"c6bec753-ff8a-4804-b99f-9b5f6785f4ab","order_by":1,"name":"Yizhong Wu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA1UlEQVRIiWNgGAWjYHACAxAhZwDhMBOvxRhIMjaQpCVxA9Fa+NsPb2DmbatL3y6R/vwBQ4V1YgP72QN4tUicSSsAamHL3Tkjx7CB4Ux6YgNPXgIBV+UYMOe28eRuuJHD2MDYdjixQYLHAL8W/jcgLRLpBjfSHzYw/iNGiwTYFoMEgxsJhg2MDURokbjxrODwn3MJhhvOvDGckXAs3biNJwe/Fv7+5I0PZ5TVyRscT3/w4UONtWw/+xn8WkDgACMblJUAxGx4VCKBP8QpGwWjYBSMghEKABTURFY6qz4DAAAAAElFTkSuQmCC","orcid":"","institution":"Huazhong University of Science and Technology","correspondingAuthor":true,"prefix":"","firstName":"Yizhong","middleName":"","lastName":"Wu","suffix":""},{"id":349951762,"identity":"0de44eca-09fa-4675-92bd-1e17d5bdea51","order_by":2,"name":"Haoyu Xu","email":"","orcid":"","institution":"Huazhong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Haoyu","middleName":"","lastName":"Xu","suffix":""},{"id":349951763,"identity":"bd8f5674-5004-4ced-9e72-5c1116edcf15","order_by":3,"name":"Yihang Xing","email":"","orcid":"","institution":"Huazhong University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Yihang","middleName":"","lastName":"Xing","suffix":""}],"badges":[],"createdAt":"2024-09-04 09:05:21","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5030219/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5030219/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":66175242,"identity":"8b312d61-1061-428e-9c51-0f9a41b0c92d","added_by":"auto","created_at":"2024-10-08 11:24:35","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":10019,"visible":true,"origin":"","legend":"\u003cp\u003eSchematic diagram illustrating the process of data collection\u003c/p\u003e","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-5030219/v1/587a551ec88dce42d869a9a6.png"},{"id":66175243,"identity":"aee27111-2bbd-450d-b3db-348d93308ab1","added_by":"auto","created_at":"2024-10-08 11:24:35","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":19160,"visible":true,"origin":"","legend":"\u003cp\u003eSchematic diagram of the training process of the Fe-NODE parameters\u003c/p\u003e","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-5030219/v1/6e0835a3cd108088b53b0b9c.png"},{"id":66174345,"identity":"42a3df17-b50f-410a-a869-31dce5a2976a","added_by":"auto","created_at":"2024-10-08 11:16:35","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":53531,"visible":true,"origin":"","legend":"\u003cp\u003eRMSE of baseline (Dy-NODE) and Fe-NODE with different feature extensive coefficients. RMSE is calculated for 10 evaluation sequences of 100 steps, and over 5 random seeds.\u003c/p\u003e","description":"","filename":"Onlinefloatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-5030219/v1/ef65845acb1c90c28daa81e1.png"},{"id":66174347,"identity":"c68c23c4-6065-406d-b496-cfddebfaf1ab","added_by":"auto","created_at":"2024-10-08 11:16:35","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":54629,"visible":true,"origin":"","legend":"\u003cp\u003eCumulative prediction error for four environments. The horizontal axis shows the horizon length of the action sequences. The cumulative error is averaged over 10 evaluation sequences of 100 steps and 5 random seeds, collected with random policies.\u003c/p\u003e","description":"","filename":"Onlinefloatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-5030219/v1/1625cdbe505c84afd69b1bd7.png"},{"id":66175615,"identity":"5b7d28fb-5610-4fdc-b2b9-54d61207637a","added_by":"auto","created_at":"2024-10-08 11:32:36","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":580493,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5030219/v1/8bc4f0a9-a866-4842-bdea-0ac6547d0013.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Fe-NODE: Feature extension Neural Ordinary Differential Equations for Modeling Black-box Control Systems","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eThe state-space model [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e] can describe the evolution of dynamic systems. This model is widely used in engineering, physics, system analysis, and control design. A crucial component of the state-space model is the state equation, which provides a mathematical description of the time evolution of system states. Based on an accurate state equation, dynamic programming [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e], model predictive control [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e], or model-based reinforcement learning [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e] can achieve optimal control.\u003c/p\u003e \u003cp\u003eThe state equation is typically expressed as differential equations in continuous-time dynamic systems. We can relatively easily construct the state equation in linear systems based on physical principles or empirical dynamic models [\u003cspan additionalcitationids=\"CR5\" citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. However, these methods can be restrictive when applied to system identification of complex nonlinear systems. Recent research has made significant strides in machine learning (ML) approaches for solving differential equations [7], which has made nonlinear ML models a potential candidate for solving system identification tasks. Raissi [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e8\u003c/span\u003e] introduces a physics-informed neural network (PINN) that is trained to solve supervised learning tasks by incorporating underlying physics principles. Brunton [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e9\u003c/span\u003e] proposed the sparse identification of a nonlinear dynamics system (SINDy), which uses sparse regression techniques to identify nonlinear differential equations of a dynamic system from measured data. Neural state-space models (NSSMs) [10\u0026ndash;12] have demonstrated flexibility, allowing varying degrees of domain knowledge to be embedded in the model formulation.\u003c/p\u003e \u003cp\u003eDeep learning models have also been developed to obtain differential equations with unknown derivative values [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. Chen [15] proposed the neural ordinary differential equation (NODE) method based on a residual network, and constructed the state equation by sequence states, providing a way to obtain the state equation of uncontrolled dynamic systems. Some researchers have also begun to work on variants of the NODE method, such as Park [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e16\u003c/span\u003e] made an in-depth analysis of NODE and applied it to clarify the influence of several design choices on the underlying dynamics. Tzen [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e17\u003c/span\u003e] adds an independent Gaussian perturbation into NODE to solve a stochastic differential equation of the diffusion process.\u003c/p\u003e \u003cp\u003eOther studies have also explored the potential of neural networks to solve the differential equation of dynamic systems. For example, Chen [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e18\u003c/span\u003e] proposed the symplectic recurrent neural network (SRNN) to capture the dynamics of physical systems from regularly observed data. Long [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e19\u003c/span\u003e] proposed a new feed-forward deep network, called partial differential equations networks (PDENet), to fulfill two objectives at the same time: to accurately predict the dynamics of complex systems and to uncover the underlying hidden PDE models. PDE-Net has the potential to uncover the hidden PDE of the observed dynamics and predict the dynamical behavior for a relatively long time. Sorourifar [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e20\u003c/span\u003e] introduced a framework of physics-enhanced NODE that enables a combination of partially known mechanistic models with universal surrogate approximators, which enables scientific-based learning from limited datasets. Li [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e21\u003c/span\u003e] proposed a hybrid structure modeling strategy combined with a neural network model is proposed. A continuous time state-space neural network (CSNN) model is used to represent the model, and the physical model is coupled by the state-space substructure method. Gorodetskiy [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e22\u003c/span\u003e] used NODE to offer a continuous-time formulation that captures the temporal dependencies. The NODE model is trained to predict the movement of the arm and is used to generate trajectories for model-based policy optimization.\u003c/p\u003e \u003cp\u003eHowever, for a black-box dynamic system containing control variables, solving its state equation is a challenge, some researchers have conducted studies in this area [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. Alvarez [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e24\u003c/span\u003e] proposed the Dy-NODE method which integrated the control variables into the input of the model through the concatenation method and then used the generated state equation optimal control. Li [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e25\u003c/span\u003e] proposes a novel approach by combining a symbolic neural network (S-Net) with NODE to solve differential equations containing exogenous inputs. To better integrate control parameters into the state equation of black-box control systems, we were inspired by the feature pyramid network (FPN) in the field of image processing [26] and proposed the feature extension neural ordinary differential equation (Fe-NODE) method, which is used to construct more accurate state equation of control system by adding richer features to the model. The innovation of this paper mainly lies in the following aspects:\u003c/p\u003e \u003cp\u003e(1) Based on NODE, the \"Intermediate State Supervision\" and \"Teacher Forcing\" methods are adopted to train the model, which is suitable for the training of the equation of state model.\u003c/p\u003e \u003cp\u003e(2) Combining polynomial extension with the NODE method, the Fe-NODE method is proposed to enrich the representation ability of input features\u003c/p\u003e \u003cp\u003e(3) Through multiple tests of numerical examples and black-box examples, the proposed method effectively improves the accuracy of the state equation, the model accuracy is higher, and the optimal feature expansion coefficient is obtained by comparison.\u003c/p\u003e \u003cp\u003eThe remainder of this paper is organized as follows: Sect. 2 introduces the related work. In the Sect. 3, we explain our proposed Fe-NODE method. Sect. 4 contrasts the proposed approach with the baseline approach in four examples, and in Sect. 5 we summarize our findings and discuss potential avenues for future research.\u003c/p\u003e"},{"header":"2. Related work","content":"\u003cp\u003eThe dynamic system satisfies the Markov property [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e28\u003c/span\u003e], the value of the current state is related only to the previous state, not to the previous sequence of states. The dynamic behavior can usually be described by ordinary differential equations, for a continuous dynamic system without control, the state equation can generally be expressed as:\u003cdiv id=\"Equ1\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ1\" name=\"EquationSource\"\u003e\n$${\\dot {x}_t}=f\\left( {{x_t},\\theta } \\right)$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e1\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eWhere\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x_t}\\)\u003c/span\u003e\u003c/span\u003edenotes the system state and\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\dot {x}_t}\\)\u003c/span\u003e\u003c/span\u003eis the derivative of the system state, \u003cspan class=\"InlineEquation\"\u003e\u003c/span\u003eis a nonlinear function that describes the dynamic behavior, and\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\theta\\)\u003c/span\u003e\u003c/span\u003eis the parameter corresponding to the nonlinear function. According to the characteristics of ordinary differential equations, we can use the integral method to obtain the system state of the next moment.\u003cdiv id=\"Equ2\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ2\" name=\"EquationSource\"\u003e\n$${\\hat {x}_{t+1}}={x_t}+\\int_{t}^{{t+1}} {f({x_t},\\theta )} dt$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e2\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eThe above formula can obtain the time series data of system states. When the exact value of derivative\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\dot {x}_t}\\)\u003c/span\u003e\u003c/span\u003eis unknown, it is difficult to train the parameters\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\theta\\)\u003c/span\u003e\u003c/span\u003eof nonlinear functions. The NODE method can deal with this issue, the specific implementation method is to obtain the predicted value of the last moment state by integration, and also obtain the last moment state by black-box dynamic system simulation. The error is obtained by comparing the above two state values, and then adjusting the parameters\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\theta\\)\u003c/span\u003e\u003c/span\u003eof the nonlinear function to reduce the error. The NODE method uses the adjoint method to train function parameters, so that function can be generated faster. The expression of error is as follows:\u003cdiv id=\"Equ3\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ3\" name=\"EquationSource\"\u003e\n$$Loss=\\frac{1}{m}\\sum\\limits_{{j=1}}^{m} {({x_T} - {{\\dot {x}}_T})}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e3\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eWhere\u003cspan class=\"InlineEquation\"\u003e\u003c/span\u003erepresents the dimension of the state,\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x_T}\\)\u003c/span\u003e\u003c/span\u003eis the state obtained by simulation at the last moment, and\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\dot {x}_T}\\)\u003c/span\u003e\u003c/span\u003eis the predicted state obtained by the NODE method.\u003c/p\u003e \u003cp\u003eThe NODE method provides an approach to obtain the state equation without control when the derivative value is unknown. Some researchers have adopted the concatenation method to add control parameters into the model input and obtain the state equation containing control. However, only the method of concatenation does not fully consider the complex underlying relationship between the control and the state in the model inputs, resulting in less information contained in the model input, which affects the estimation accuracy of the state equation.\u003c/p\u003e"},{"header":"3. Method","content":"\u003cp\u003eIn this section, we provided a detailed description of the proposed Fe-NODE method. The state equation of the black-box control system contains the system state and control, which can be expressed as:\u003cdiv id=\"Equ4\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ4\" name=\"EquationSource\"\u003e\n$${\\dot {x}_t}=f\\left( {{x_t},{u_t},\\theta } \\right)$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e4\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eWhere the system state\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x_t}\\)\u003c/span\u003e\u003c/span\u003eand control\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({u_t}\\)\u003c/span\u003e\u003c/span\u003eare contained in the input of the state equation, in the case that the corresponding derivative value is unknown, our goal is to extract information from the sequence system states to obtain the state equation. Firstly, we have the initial system state\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x_0}\\)\u003c/span\u003e\u003c/span\u003eand the neural network\u003cspan class=\"InlineEquation\"\u003e\u003c/span\u003ethat replaces the state equation, and then we can obtain the next system state by using the integral method:\u003cdiv id=\"Equ5\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ5\" name=\"EquationSource\"\u003e\n$${\\hat {x}_{t+1}}={x_t}+\\int_{t}^{{t+1}} {f({x_t},{u_t},\\theta )} dt$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e5\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eAn important assumption in this numerical evaluation is that, in the control systems, the control parameters remain constant for a small interval of time (between the current system state and the next system state). We use the numerical evaluation to collect data as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThen, the loss is obtained by comparing the predicted state and the actual state, the parameters of the neural network are adjusted to reduce the loss, and the parameters of the neural network are determined by a large number of iterations. The conventional NODE method obtains loss by comparing the real state and the predicted state at the last moment and adopts the adjoint method to update the model parameters.\u003c/p\u003e \u003cp\u003eHowever, this approach ignores the state of the intermediate moment, which may have an important impact on improving the accuracy and stability of the model. To solve this problem, we made the following improvements. Firstly, we adopted the \"Intermediate State Supervision\" method [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e29\u003c/span\u003e] to compare the real state and the predicted state at multiple moments during the training process. Intermediate state monitoring by directly monitoring multiple intermediate states of the model during training, the problem of gradient disappearing or exploding can be effectively reduced. This approach allows the model to receive feedback at each time step, helping the model better learn the characteristics of the dynamic system. Its loss function is the average value of the absolute difference between the predicted state and the real state in the whole rolling process, which can be expressed as:\u003cdiv id=\"Equ6\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ6\" name=\"EquationSource\"\u003e\n$$Loss=\\frac{1}{m}\\sum\\limits_{{j=1}}^{m} {(\\frac{1}{n}\\sum\\limits_{{i=1}}^{n} {\\left| {{x_i} - {{\\hat {x}}_i}} \\right|} )}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e6\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eWhere the\u003cspan class=\"InlineEquation\"\u003e\u003c/span\u003erepresents the length of the sequence and the\u003cspan class=\"InlineEquation\"\u003e\u003c/span\u003erepresents the dimension of the system state. Secondly, we use the \"Teacher Forcing\" strategy [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e30\u003c/span\u003e] in the model training process, instead of using the model\u0026rsquo;s previous output as input to the next time step, the truth output from the training data is provided as input. This strategy is intended to help the model learn the correct sequence of output tokens more efficiently and stabilize the training process, which is often used in recurrent neural networks [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e31\u003c/span\u003e]. The use of the above two methods can effectively optimize the training process and improve the accuracy of the proxy model.\u003c/p\u003e \u003cp\u003eTo further improve the accuracy of the agent model, we are inspired by the FPN in the field of image processing and propose the Fe-NODE method for data feature expansion, which is the main research innovation made in this paper. For two-dimensional image data, FPN uses a special convolution kernel (such as VGG) to extract features. The data to be processed in this paper is one-dimensional data, which is not suitable for extracting features by convolution kernels, and the polynomial extension is more suitable for processing one-dimensional data. The new features are generated by a polynomial combination of the input, including the power series of the input and the interaction terms between the input. By setting the coefficient of polynomial expansion, we can control the highest order of the model input. In this paper, we compared the model estimation accuracy corresponding to three different polynomial coefficients, and the training process of the proposed Fe-NODE is shown in algorithm 1.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Taba\" border=\"1\"\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003eAlgorithm 1: Fe-NODE\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1:\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eGather data set of random trajectories\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2:\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eNormalize dynamics model\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e3:\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003efor iter\u0026thinsp;=\u0026thinsp;1 to max_iter do\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e4:\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003efor t\u0026thinsp;=\u0026thinsp;0 to n do\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e5:\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003ex\u0026thinsp;=\u0026thinsp;PolynomialFeatures(x,degree\u0026thinsp;=\u0026thinsp;2,3,4)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e6:\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003ex\u0026thinsp;=\u0026thinsp;ODESolver( f, x, u)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e7:\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eAppend x to X\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e8:\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eend for\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e9:\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eloss\u0026thinsp;=\u0026thinsp;MSEloss(X,X\u0026rsquo;)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e10:\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eTrain f by performing gradient descent\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e11:\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e \u003cp\u003eend for\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eIn summary, the proposed Fe-NODE algorithm consists of three parts: feature extension, state equation neural network, and integrator process. We use Fe-NODE to obtain an estimate of the state sequence. Then the black box control system is simulated and the true value of the state sequence is obtained. The loss value is obtained by comparing the true value and the estimated value, and the state equation neural network is iteratively updated. The training process of the Fe-NODE parameters is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e"},{"header":"4. Result","content":"\u003cp\u003eIn this section, we conducted a series of experiments on modeling continuous control systems in four environments to test the performance of Fe-NODE. Two of the environments are numerical examples with control parameters. The state equation for the first numerical example is as follows:\u003cdiv id=\"Equ7\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ7\" name=\"EquationSource\"\u003e\n$$\\begin{gathered} {{\\dot {x}}_1}={x_2} \\hfill \\\\ {{\\dot {x}}_2}={\\text{g}} \\cdot \\sin (u) \\hfill \\\\ \\end{gathered}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e7\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eIt contains two state parameters and one control parameter. The state equation for the second numerical example is as follows:\u003cdiv id=\"Equ8\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ8\" name=\"EquationSource\"\u003e\n$$\\begin{gathered} {{\\dot {x}}_1}={x_3} \\cdot \\sin (u) \\hfill \\\\ {{\\dot {x}}_2}= - {x_3} \\cdot \\cos (u) \\hfill \\\\ {{\\dot {x}}_3}={\\text{g}} \\cdot \\cos (u) \\hfill \\\\ \\end{gathered}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e8\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eIt contains three state parameters and one control parameter. The other two examples come from a black-box dynamic model with controls in OpenAI Gym [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e32\u003c/span\u003e]. For numerical examples, we use the integral method to obtain the sequence system state, and the black-box model uses the physical system simulator to obtain the data.\u003c/p\u003e \u003cp\u003eWe take a data-driven approach to obtaining the model, without combining physical or empirical models. Therefore, we compared the similar Dy-NODE method as the baseline algorithm, and different polynomial parameters were used to expand the input features and compare the influence of different polynomial parameters on the accuracy of the model.\u003c/p\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Experimental setup\u003c/h2\u003e \u003cp\u003eWe used a uniform random policy to collect samples from the environment for training and evaluation. As mentioned in section 3, we adopted the \u0026ldquo;Teacher Forcing\u0026rdquo; strategy during training, which uses the real value as the input of the model. In the testing process, we used the predicted value as the input of the model at the next moment, which is a more satisfying practical application for control problems which is difficult to obtain real-time feedback.\u003c/p\u003e \u003cp\u003eIn the model training process, each sample contains 40 to 200 sequences of 100 steps. For closed-loop dynamics prediction, 10 sequences of 100 steps were collected and 5 random numbers to construct multiple validation sets to comprehensively compare the prediction effect of the model.\u003c/p\u003e \u003cp\u003eWe use Torch to train the model, each method using the same number of neural network layers and neurons, training 100 epochs to obtain the model. We use the Polynomialfeatures function in Sklearn to implement the input feature extension. The influence of different polynomial parameters on the prediction accuracy of the model is compared.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Test result\u003c/h2\u003e \u003cp\u003eIn Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, we show the overall dependency of the root mean square error (RMSE) concerning the number of samples collected from the environment. The RMSE was computed over all state dimensions, across the entire length of the sequence. It is shown from the figure that the estimation error of the test set decreases with the increase of the training set, indicating that more training sets will effectively improve the estimation accuracy of the model. Several examples show that when the training set reaches a certain scale, the prediction accuracy of the model presents a relatively stable state. As can be seen from Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, the proposed Fe-NODE method outperforms the baseline in terms of prediction accuracy. The proposed method uses fewer training sets to obtain an accurate model and has higher modeling efficiency than the baseline method. We compare the effect of three different feature extensive coefficients on the model and find that the model has the highest prediction accuracy when the feature extensive coefficient is four.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eTo further demonstrate the advantages of the algorithm, we use the same number of training sets (60 samples) for each environment to construct the model and obtain the change in the cumulative prediction error as the sequence length increases. Figure\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e shows that the Fe-NODE method with a feature extension coefficient is four has high accuracy, the results are consistent with those in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e. Indicating that the method reduces the influence of error accumulation on sequence length.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e compares the accuracy of the models under different training sets of different sizes, and the results also prove that Fe-NODE is better than the baseline method, and in most cases, the prediction effect of the model is better when the feature coefficient is four, which is consistent with the results shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e\u0026ndash;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eRMSE for the four environments of the Fe-NODE\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNo.Samples\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMethods\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eExample 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eExample 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003ePendulum\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAcrobot\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"3\" rowspan=\"4\"\u003e \u003cp\u003e60\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDy-NODE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1.30E-03\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e7.06E-06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e4.30E-03\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.50E-04\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFe-NODE,2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2.69E-04\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3.37E-06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e4.34E-04\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.34E-04\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFe-NODE,3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1.30e-04\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e5.57E-06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e4.43E-04\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e9.97E-05\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFe-NODE,4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e1.11e-04\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e2.42E-06\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e3.08E-04\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.09E-04\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"3\" rowspan=\"4\"\u003e \u003cp\u003e120\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDy-NODE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1.54E-04\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2.93E-06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1.81E-04\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.39E-04\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFe-NODE,2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2.64E-05\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e8.31E-07\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e4.32E-05\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e5.52E-05\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFe-NODE,3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1.39E-05\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e4.10E-06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e4.48E-05\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e2.91E-05\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFe-NODE,4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e5.55E-06\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e4.07E-06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e2.47E-05\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e3.30E-05\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"3\" rowspan=\"4\"\u003e \u003cp\u003e200\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDy-NODE\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1.58e-05\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e4.42e-06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e5.97e-05\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.05e-04\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFe-NODE,2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e7.80e-06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1.20e-06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e2.59e-05\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e2.49e-05\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFe-NODE,3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e3.83e-06\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e4.77e-07\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e2.15e-05\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e1.35e-05\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eFe-NODE,4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6.11e-06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2.73e-06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e7.34e-06\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e9.98e-06\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe above results prove that the Fe-NODE method proposed in this paper is superior to the benchmark method in terms of model accuracy and training efficiency. We compare the effect of different feature parameters on the model through several examples, and the model has high accuracy when the feature extension coefficient is four.\u003c/p\u003e \u003c/div\u003e"},{"header":"5. Conclusion","content":"\u003cp\u003eWe propose a novel Fe-NODE method designed to effectively model black-box control systems. The primary innovation of this paper lies in extending the characteristics of the model inputs, enabling it to capture complex dynamic behaviors more efficiently and enhance the prediction accuracy of the model. During model training, we employ the strategies of intermediate state supervision and teacher forcing. Intermediate state supervision offers multilevel feedback, enhancing the model\u0026rsquo;s learning ability, while teacher coercion reduces error propagation during time series generation. These methods collectively increase the model\u0026rsquo;s efficiency and stability during training. We validate the effectiveness of the algorithm through four environments, including numerical examples and black-box control system examples. Results indicate that the proposed method achieves higher prediction accuracy than the baseline, allowing for more accurate models to be obtained with a smaller training set and enhanced modeling efficiency. Furthermore, we compare the effects of various feature extensive parameters on the model prediction accuracy, revealing that the model prediction accuracy is highest when the feature extension coefficient is four.\u003c/p\u003e \u003cp\u003eThe polynomial extension method increases the model input dimension with rising coefficients, consequently extending the model training time. Therefore, future research will focus on optimizing this method to reduce model training time while maintaining accuracy.\u003c/p\u003e"},{"header":"Declarations","content":" \u003ch2\u003eConflict of interest\u003c/strong\u003e \u003cp\u003eThe authors declare that they have no conflict of interest.\u003c/p\u003e \u003ch2\u003eFunding\u003c/h2\u003e \u003cp\u003eThis work was supported by the Equipment Pre-Research Common Technology Area Foundation of China (Grant No. 50923010101).\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eXia wrote the main manumanuscript text, Wu guided the reasearch of this paper, Xu and Xing prepared figures 1-2 . All authors reviewed the manuscript.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e \u003cp\u003eThe Python codes and the datasets generated during the current studies are available from the corresponding author upon reasonable request.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n \u003cli\u003eGhahramani, Z., Hinton, G.E.: Variational Learning for Switching State-Space Models. Neural. Comput. 12, 831-864 (2004). https://doi.org/10.1162/089976600300015619\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eKantas, N., Doucet, A., Singh, S.S.: On Particle Methods for Parameter Estimation in State Space Models. Stat. Sci. 30, 328-351 (2014). https://doi.org/10.1214/14-STS511\u003c/li\u003e\n \u003cli\u003eDadebo, S., Luus, R.: Optimal control of time‐delay systems by dynamic programming. Optim. Control Appl. Methods. 13, 29-41 (1992). https://doi.org/10.1002/oca.4660130103\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eZanon, M., Lars, G., Diehl, M.: Periodic Optimal Control, Dissipativity and MPC. IEEE Trans. on Autom. Control (2017). https://doi.org/10.1109/TAC.2016.2601881\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eKamalapurkar, R., Walters, P., Dixon, W.E.: Model-based reinforcement learning for approximate optimal regulation. Control Complex Syst. 64, 247-273 (2016).\u0026nbsp;\u003cbr\u003e\u0026nbsp;https://doi.org/10.1016/j.automatica.2015.10.039\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eJarzebowska, E., Augustynek, K., Urbas, A.: Dynamics modeling method dedicated to system models with open- and closed-loop structures subjected to kinematic and task based constraints. Nonlinear Dyn. 111, 12053\u0026ndash;12080 (2023). https://doi.org/10.1007/s11071-023-08471-1\u0026nbsp;\u003c/li\u003e\n \u003cli\u003ePark, J., Pao, G.M., Sugihara, G.: Empirical mode modeling A data-driven approach to recover and forecast nonlinear dynamics from noisy data. Nonlinear Dyn. 108:2147\u0026ndash;2160 (2022).\u003cbr\u003e\u0026nbsp;https://doi.org/10.1007/s11071-022-07311-y\u003c/li\u003e\n \u003cli\u003eChang, C.W., Ushio, M., Hsieh, C.H.: Empirical dynamic modeling for beginners.Ecol. Res. 32, 1-12 (2017). https://doi.org/10.1007/s11284-017-1469-9\u003c/li\u003e\n \u003cli\u003eKidger, P., Morrill, J., Foster, J.: Neural Controlled Differential Equations for Irregular Time Series. arXiv (2020). https://doi.org/10.48550/arXiv.2005.08926\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eRaissi, M., Perdikaris, P., Karniadakis G.E.: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686-707 (2019). https://doi.org/10.1016/j.jcp.2018.10.045\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eBrunton, S.L., Proctor, J.L., Kutz, J.N.: Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl. Acad. Sci. 113, 3932-3937 (2016).\u003cbr\u003e\u0026nbsp;https://doi.org/10.1073/pnas.1517384113\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eRangapuram, S.S., Seeger. M.W., Gasthaus, J.: Deep State Space Models for Time Series Forecasting. Neural Information Processing Systems (2018)\u003c/li\u003e\n \u003cli\u003eMasti, D., Bemporad, A.: Learning Nonlinear State-Space Models Using Deep Autoencoders. IEEE Conference on Decision and Control (2018). https://doi.org/10.1109/CDC.2018.8619475\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eSkomski, E., J\u0026aacute;n, D., Tuor A.: Automating Discovery of Physics-Informed Neural State Space Models via Learning and Evolution. Proceedings of the 3rd Conference on Learning for Dynamics and Control. 144, 980-991 (2021)\u003c/li\u003e\n \u003cli\u003eRahman, A., Drgoňa, J., Tuor, A.: Neural Ordinary Differential Equations for Nonlinear System Identification. American Control Conference (2022). https://doi.org/10.23919/ACC53348.2022.9867586\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eHerrera, C., Krach, F., Teichmann, J.: Neural jump ordinary differential equations: Consistent continuous-time prediction and filtering. International Conference on Learning Representation (2021)\u003c/li\u003e\n \u003cli\u003eChen, R.T.Q., Rubanova, Y., Bettencourt, J.: Neural ordinary differential equations. Advances in neural information processing systems. (2018)\u003c/li\u003e\n \u003cli\u003eMassaroli, S., Poli, M., Park, J.: Dissecting neural odes. Advances in Neural Information Processing Systems. 33, 3952-3963 (2020)\u003c/li\u003e\n \u003cli\u003eTzen, B., Raginsky, M.: Neural stochastic differential equations: Deep latent Gaussian models in the diffusion limit. arXiv (2019). https://doi.org/10.48550/arXiv.1905.09883\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eChen, Z., Zhang, J., Arjovsky, M.: Symplectic recurrent neural networks. arXiv (2019).\u0026nbsp;https://doi.org/arXiv:1909.13334, 2019\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eLong, Z., Lu, Y., Ma, X.: Pde-net: Learning pdes from data. International conference on machine learning. 3208-3216 (2018)\u003c/li\u003e\n \u003cli\u003eSorourifar, F., Peng, Y., Castillo, I.: Physics-Enhanced neural ordinary differential equations: Application to industrial chemical reaction systems. Ind. Eng. Chem. Res. 62, 15563-15577(2023)\u003c/li\u003e\n \u003cli\u003eLi, H.W., Hao, S., Ni, Y.Q.: Hybrid structural analysis integrating physical model and continuous time state‐space neural network model. Comput.-Aided Civ. Infrastruct. Eng. (2024)\u003c/li\u003e\n \u003cli\u003eGorodetskiy, A., Mironov, K., Panov A.: Model-Based Policy Optimization with Neural Differential Equations for Robotic Arm Control. International Conference on Interactive Collaborative Robotics. 258-266 (2023)\u003c/li\u003e\n \u003cli\u003eHashimoto, T., Matsui, N., Kamiura, N.: Exploring Model Structures to Reduce Data Requirements for Neural ODE Learning in Control Systems. J. Adv. Comput. Intell. Intell. Inform. 27, 537-542 (2023)\u003c/li\u003e\n \u003cli\u003eAlvarez, V.M.M., Roşca, R., Fălcuţescu, C.G.: Dynode: Neural ordinary differential equations for dynamics modeling in continuous control. arXiv (2020).\u003cbr\u003e\u0026nbsp;https://doi.org/10.48550/arXiv.2009.04278\u003c/li\u003e\n \u003cli\u003eLi, Q., Evje, S., Geng, J.: Learning parameterized odes from data. IEEE Access. 11, \u0026nbsp;54897-54909 (2023)\u003c/li\u003e\n \u003cli\u003eLin, T.Y., Doll\u0026aacute;r, P., Girshick, R.: Feature pyramid networks for object detection. Proceedings of the IEEE conference on computer vision and pattern recognition. 2117-2125 (2017)\u003c/li\u003e\n \u003cli\u003eChen, C., Hong, J., Lu, Y.: Stochastic differential equation with piecewise continuous arguments: Markov property, invariant measure and numerical approximation. Discreet. Contin. Dyn. Syst. Ser. B. 28, 765\u0026ndash;807 (2023).\u0026nbsp;https://doi.org/10.3934/dcdsb.2022098\u003c/li\u003e\n \u003cli\u003eWu, F., Yin, G., Mei, H.: Stochastic functional differential equations with infinite delay: Existence and uniqueness of solutions, solution maps, Markov properties, and ergodicity. \u0026nbsp;J. Differ. Equ. 262, 1226-1252 (2017)\u003c/li\u003e\n \u003cli\u003eLi, R., Wang, X., Huang, G.: A comprehensive review on deep supervision: Theories and applications. arXiv (2022).\u0026nbsp;https://arxiv.org/pdf/2207.02376\u003c/li\u003e\n \u003cli\u003eFeng, Y., Gu, S., Guo, D.: Guiding teacher forcing with seer forcing for neural machine translation. arXiv (2021)\u003c/li\u003e\n \u003cli\u003eLamb, A., Anirudh, G.: Professor forcing: A new algorithm for training recurrent networks. Advances in neural information processing systems, 29 (2016)\u003c/li\u003e\n \u003cli\u003eBrockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym. arXiv (2016). https://arxiv.org/abs/1606. 01540. cite arxiv:1606.01540\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nonlinear-dynamics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"nody","sideBox":"Learn more about [Nonlinear Dynamics](https://www.springer.com/journal/11071)","snPcode":"11071","submissionUrl":"https://submission.nature.com/new-submission/11071/3","title":"Nonlinear Dynamics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Neural ODE, Feature extension, Black-box control systems, System identification.","lastPublishedDoi":"10.21203/rs.3.rs-5030219/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5030219/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis paper introduces a novel Feature-Extended Neural Ordinary Differential Equation (Fe-NODE) method for modeling black-box control systems by incorporating control parameters into the ordinary differential equation. Since the input to the model should consider both the system state and the control parameters, this is a challenge to the modeling process. The primary contribution of the proposed method is employing feature expansion to add information to the model input and enhance the perception ability of the model while utilizing intermediate state supervision and teacher forcing to optimize the training process, and effectively identify potential models for black-box control systems. The experimental results demonstrate that the Fe-NODE method outperforms the baseline in prediction accuracy and modeling efficiency. Additionally, we investigate the impact of different feature expansion coefficients on the model\u0026rsquo;s prediction accuracy, identifying the optimal coefficient through comparative analysis. This approach provides a novel methodology for modeling black-box control systems and lays the foundation for model-based reinforcement learning to achieve optimal control.\u003c/p\u003e","manuscriptTitle":"Fe-NODE: Feature extension Neural Ordinary Differential Equations for Modeling Black-box Control Systems","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-10-08 11:16:30","doi":"10.21203/rs.3.rs-5030219/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorAssigned","content":"","date":"2024-09-05T10:30:27+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-09-05T04:29:37+00:00","index":"","fulltext":""},{"type":"submitted","content":"Nonlinear Dynamics","date":"2024-09-04T09:04:00+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nonlinear-dynamics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"nody","sideBox":"Learn more about [Nonlinear Dynamics](https://www.springer.com/journal/11071)","snPcode":"11071","submissionUrl":"https://submission.nature.com/new-submission/11071/3","title":"Nonlinear Dynamics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"e983059e-cde2-4ed8-912b-a63c4968b0bb","owner":[],"postedDate":"October 8th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2024-10-08T11:16:31+00:00","versionOfRecord":[],"versionCreatedAt":"2024-10-08 11:16:30","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5030219","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5030219","identity":"rs-5030219","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}