2.1 Problem statement

The centroid dynamic model of unpowered reentry vehicle subjected to unknown disturbances is shown as follows:

(1) \begin{align}\left\{ {\begin{array}{l}{\dot x = V{\rm{cos}}\theta {\rm{cos}}{\psi _v}}\\[3pt]{\dot y = V{\rm{sin}}\theta }\\[3pt]{\dot z = - V{\rm{cos}}\theta {\rm{sin}}{\psi _v}}\\[3pt]{\dot V = - \dfrac{X}{m} - g{\rm{sin}}\theta }\\[11pt]{\dot \theta = \dfrac{{Y{\rm{cos}}{\gamma _v}}}{{mV}} - \dfrac{g}{V}{\rm{cos}}\theta + {d_1}\left( t \right)}\\[11pt]{{{\dot \psi }_v} = - \dfrac{{Y{\rm{sin}}{\gamma _v}}}{{mV{\rm{cos}}\theta }} + {d_2}\left( t \right)}\end{array}} \right.\end{align}

where $x,y,z$ are the positions of the vehicle in the inertial frame. $V$ is the velocity. $\theta $ represents the angle between the velocity vector and the horizontal plane, i.e., the path angle. When the velocity vector is above the horizontal plane, the $\theta $ is positive. ${\psi _v}$ represents the angle between the projection of the velocity vector in the horizontal plane and the $x$ axis, i.e., the deflection angle, measured counterclockwise in the horizontal plane. $\alpha $ and ${\gamma _v}$ are the angle-of-attack and back angle. $m$ and $g$ are mass and gravitational acceleration. ${d_1}\left( t \right)$ and ${d_2}\left( t \right)$ are the unknown disturbances to the vehicle. $X$ and $Y$ are the drag and lift forces, and the expression of $Y$ is

(2) \begin{align}Y = \left( {C_y^0 + C_y^\alpha \alpha } \right)q{S_{ref}}\end{align}

where $C_y^0$ and $C_y^\alpha $ are the lift coefficients. ${S_{ref}}$ is the reference area. $q$ is the dynamic pressure, and its expression is

(3) \begin{align}q = \frac{1}{2}\rho {V^2}\end{align}

where $\rho $ is the atmospheric density.

Define $\xi = {\left[ {\theta, {\psi _v}} \right]^T}$ as the state variables. Then, when the reentry vehicle performs a spiral-diving manoeuver, the desired guidance commands can be expressed as ${\xi _d} = {\left[ {{\theta _d},{\psi _{vd}}} \right]^T}$ . Up to now, the problem of spiral-diving guidance for reentry vehicle subject to unknown disturbances can be essentially translated into the problem of tracking the desired guidance command. And this problem can be organized as follows:

(4) \begin{align}\left\{ {\begin{array}{c}{\dot \xi = {f_1}\left( \xi \right) + {g_1}\left( {\xi, u} \right) + d}\\[5pt]{{\chi _o} = \xi }\end{array}} \right.\end{align}

where $u = {\left[ {\alpha, {\gamma _v}} \right]^T}$ are the control variables. ${\chi _o}$ is the output vector. $d = {\left[ {{d_1},{d_2}} \right]^T}$ . ${f_1}$ and ${g_1}$ are smooth nonlinear functions that can be expressed as

(5) \begin{align}{f_1}\left( \xi \right) = \left[ {\begin{array}{c}{ - \dfrac{g}{V}{\rm{cos}}\theta }\\[10pt]0\end{array}} \right]\end{align}

(6) \begin{align}{g_1}\left( {\xi, u} \right) = \left[ {\begin{array}{c}{\dfrac{{\left( {C_y^0 + C_y^\alpha \alpha } \right)q{S_{ref}}{\rm{cos}}{\gamma _v}}}{{mV}}}\\[10pt]{ - \dfrac{{\left( {C_y^0 + C_y^\alpha \alpha } \right)q{S_{ref}}{\rm{sin}}{\gamma _v}}}{{mV{\rm{cos}}\theta }}}\end{array}} \right]\end{align}

The tracking error can be expressed as

(7) \begin{align}{e_1} = \xi - {\xi _d}\end{align}

Assumption 1. The system represented by Equation 4 is controllable, which satisfies

(8) \begin{align}\left| {\frac{{\partial {g_1}\left( {\xi, u} \right)}}{{\partial u}}} \right| \ne 0\end{align}

Lemma 1. [Reference He and Dong34] For a Lyapunov function $L\left( t \right)$ , if its initial value $L\left( 0 \right)$ is bounded and its time derivative satisfies

(9) \begin{align}\dot L\left( t \right) \le - \kappa L\left( t \right) + \delta \end{align}

where $\kappa \gt 0$ and $\delta \gt 0$ are constants, then $L\left( t \right)$ is bounded.

Lemma 2. [Reference Xing-Kai35] For vectors $A \in {\mathbb{R}^n}$ and $B \in {\mathbb{R}^n}$ , there always holds

(10) \begin{align}2{A^T}B \le \| A \|^2 + \| B \|^2\end{align}

(11) \begin{align}\|AB\| \le \| A \| \cdot \| B \|\end{align}

2.2 Spiral trajectory parameters solving

The logarithmic spiral trajectory can be defined as

(12) \begin{align}{r_s} = r\left( \vartheta \right) = {r_0}{e^{\vartheta {\rm{cot\Lambda }}}}\end{align}

where ${r_0}$ is the initial polar diameter. $\vartheta $ is the polar angle. ${\rm{\Lambda }}$ is the angle between the component of the vehicle velocity vector in yaw plane and the polar diameter. It can be found from Equation 12 that once the values of ${r_0}$ and ${\rm{\Lambda }}$ are acquired, the shape of the spiral trajectory can be determined uniquely.

Figure 3 in He et al. [Reference He, Yan and Tang20] shows the geometric representation of the spiral trajectory in the yaw plane. Where ${M_0}$ and ${M_s}$ are the initial position and the current desired position of the vehicle. $T$ is the target position. $p{x_p}{z_p}$ denotes the polar frame, the pole $p$ coincides with the rotation centre of the spiral trajectory, the polar axis $p{z_p}$ points to the ${M_0}$ , and the polar axis $p{x_p}$ is perpendicular to $p{z_p}$ . ${r_{s0}}$ , ${r_s}$ and ${r_{s1}}$ are the polar diameters at ${M_0}$ , ${M_s}$ and $T$ , and the corresponding polar angles and deflection angles are ${\vartheta _0}$ , $\vartheta $ , ${\vartheta _1}$ and ${\psi _{vs0}}$ , ${\psi _{vs}}$ , ${\psi _{vs1}}$ , respectively. $\eta $ is the rotation angle of the polar frame with respect to the inertial frame. If let $\left( {{x_0},{z_0}} \right)$ and $\left( {{x_1},{z_1}} \right)$ represent the coordinates at ${M_0}$ and $T$ , respectively, then the initial condition set ${H_0}$ and terminal constraint set ${H_1}$ of the vehicle can be defined as

\begin{align*}{H_0} = \left\{ {\left( {{x_0},{z_0}} \right),{\vartheta _0},{\psi _{vs0}}} \right\},{\rm{\;\;\;\;}}{H_1} = \left\{ {\left( {{x_1},{z_1}} \right),{\psi _{vs1}}} \right\}\end{align*}

It is worth noting that the definition of deflection angle direction in this paper is contrary to that in He et al. [Reference He, Yan and Tang20]. Therefore, in order to facilitate understanding, it is necessary to deduce new expressions related to the determination of spiral trajectory parameters. Referring to Fig. 3 in He et al. [Reference He, Yan and Tang20], the main geometrical relation can be expressed as follows:

(13) \begin{align}\frac{\pi }{2} + {\psi _{vs}} = \vartheta + \eta + {\rm{\Lambda }}\end{align}

Substituting the ${\psi _{vs0}}$ and ${\psi _{vs1}}$ into Equation 13, we get

(14) \begin{align}\eta = \frac{\pi }{2} + {\psi _{vs0}} - {\rm{\Lambda }}\end{align}

(15) \begin{align}{\vartheta _1} = {\psi _{vs1}} - {\psi _{vs0}}\end{align}

The coordinates of the $p$ can be solved by

(16) \begin{align}\left[ {\begin{array}{c}{{x_p}}\\[3pt]{{z_p}}\end{array}} \right] = \frac{1}{{{K_1} - {K_0}}}\left[ {\begin{array}{c}{{K_1}{x_0} - {K_0}{x_1} + {K_0}{K_1}\left( {{z_1} - {z_0}} \right)}\\[3pt]{{K_1}{z_1} - {K_0}{z_0} - \left( {{x_1} - {x_0}} \right)}\end{array}} \right]\end{align}

where ${K_0}$ and ${K_1}$ are slopes of the rays $p{M_0}$ and $pT$ , respectively, and their expressions are:

(17) \begin{align}{K_i} = {\rm{tan}}\left( {\frac{\pi }{2} + {\psi _{vsi}} - {\rm{\Lambda }}} \right),{\rm{\;\;\;\;}}i = 0,1\end{align}

Refer to He et al. [Reference He, Yan and Tang20] for the calculation of ${x_p}$ and ${z_p}$ when ${K_0}$ and ${K_1}$ do not exist. Next, the lengths of the polar diameters ${r_{s0}}$ and ${r_{s1}}$ can be calculated:

(18) \begin{align}\|{r_{s0}}\| = \left| {\frac{{{z_0} - {z_p}}}{{{\rm{sin}}\left( {{\psi _{vs0}} - {\rm{\Lambda }}} \right)}}} \right|\end{align}

(19) \begin{align}\|{r_{s1}}\| = \left| {\frac{{{z_1} - {z_p}}}{{{\rm{sin}}\left( {{\psi _{vs1}} - {\rm{\Lambda }}} \right)}}} \right|\end{align}

Dividing Equation 19 by Equation 18 and combining Equations 12 and 17 yields

(20) \begin{align}In\left| {\frac{{{\rm{cos}}\left( {{\psi _{vs0}} - {\rm{\Lambda }} - \mu } \right)}}{{{\rm{cos}}\left( {{\psi _{vs1}} - {\rm{\Lambda }} - \mu } \right)}}} \right| = {\vartheta _1}{\rm{cot\Lambda }}\end{align}

where $\mu = {\rm{arctan}}\left[ {\left( {{x_0} - {x_1}} \right)/\left( {{z_0} - {z_1}} \right)} \right]$ .

Once the values in the sets ${H_0}$ and ${H_1}$ are given, ${\rm{\Lambda }}$ can be acquired by solving Equation 20. By substituting ${\rm{\Lambda }}$ into Equations 16, 18, 14 and 15, the polar coordinates $\left( {{x_p},{z_p}} \right)$ , initial polar diameter ${r_0}$ , the rotation angle $\eta $ of the polar frame with respect to the inertial frame and the terminal polar angle ${\vartheta _1}$ can be calculated, respectively.

2.3 Neural networks in reinforcement learning

Neural networks are an important part of reinforcement learning and are powerful in coping with nonlinearities. With the help of neural networks, a nonlinear function $f$ can be expressed as

(21) \begin{align}f = {W^T}{\rm{\Phi }}\left( {\bar Z} \right) + \varepsilon \left( {\bar Z} \right)\end{align}

where $W \in {\mathbb{R}^l}$ is the weight vector, $l$ is the number of nodes in the hidden layer. $\bar Z = {\left[ {{{\bar z}_1},{{\bar z}_2}, \ldots, {{\bar z}_m}} \right]^T} \in {\mathbb{R}^m}$ are the inputs of neural networks with dimension $m$ . ${\rm{\Phi }}\left( {\bar Z} \right) = {\left[ {{\varphi _{b1}},{\varphi _{b2}}, \ldots {\varphi _{bl}}} \right]^T} \in {\mathbb{R}^l}$ are the basis functions. $\varepsilon \left( {\bar Z} \right)$ is the function reconstruction error. The optimal approximation can be obtained by properly selecting the number of network nodes.

The radial basis functions (RBF) neural networks are selected as the basic network frame in this paper, and its basis functions can be described as

(22) \begin{align}{\varphi _j}\left( Z \right) = {\rm{exp}}\left( { - \frac{{\|Z - {\zeta _j}\|^2}}{{\varsigma _j^2}}} \right)\end{align}

where ${\zeta _j} = {\left[ {{\zeta _{j1}},{\zeta _{j2}}, \ldots, {\zeta _{jm}}} \right]^T}$ is the centre vector of the $j$ -th node in the hidden layer. ${\varsigma _j}$ is the width value.

Lemma 3. [Reference Ouyang, Dong, Wei and Sun31] The basis function ${\rm{\Phi }}\left( {\bar Z} \right)$ of neural networks is bounded, which satisfies $\left\|{\rm{\Phi }}\left( {\bar Z} \right)\right\| \le {{\rm{\Phi }}_M}$ and $\left\|\dot{\Phi}\left( {\bar Z} \right)\right\| \le {{\rm{\Phi }}_{dM}}$ , where ${{\rm{\Phi }}_M}$ and ${{\rm{\Phi }}_{dM}}$ are positive constants.

Lemma 4. [Reference Ouyang, Dong, Wei and Sun31] If the ideal weight ${W^{\rm{*}}}$ is obtained, then there exists $\left| {\varepsilon \left( {\bar Z} \right)} \right| \le {\varepsilon _m}$ and $\left| {\dot \varepsilon \left( {\bar Z} \right)} \right| \le {\varepsilon _{dm}}$ , where ${\varepsilon _m}$ and ${\varepsilon _{dm}}$ are positive constants.

3.1 Desired proportional navigation guidance law

Figure 4 in He et al. [Reference He, Yan and Tang20] displays the motion of the vehicle and the virtual sliding target in the yaw plane. Where $M$ is the projection of the current position of the vehicle in the yaw plane, and ${M_s}$ is the closest point of $M$ to the spiral trajectory.

Under Assumption 2, the polar angle $\vartheta $ corresponding to the $M$ can be obtained by solving the following equation:

(23) \begin{align}\left( {x - {x_p}} \right){\rm{sin}}\left( {\vartheta + \eta + {\rm{\Lambda }}} \right) + \left( {z - {z_p}} \right){\rm{cos}}\left( {\vartheta + \eta + {\rm{\Lambda }}} \right) = a{e^{\vartheta {\rm{cot\Lambda }}}}{\rm{cos\Lambda }}\end{align}

The time derivative of Equation 23 can be arranged to obtain the time derivative of $\vartheta $ :

(24) \begin{align}\dot \vartheta = \frac{{V{\rm{cos}}\theta {\rm{sin}}\left( {\vartheta + \eta + {\rm{\Lambda }} - {\psi _v}} \right) - {V_t}{\rm{sin}}\left( {\vartheta + \eta + {\rm{\Lambda }} - {\psi _{vt}}} \right)}}{{\left[ {\|{r_s}\|{\rm{co}}{{\rm{s}}^2}{\rm{\Lambda }}/{\rm{sin\Lambda }} - \left( {x - {x_p}} \right){\rm{cos}}\left( {\vartheta + \eta + {\rm{\Lambda }}} \right) + \left( {z - {z_p}} \right){\rm{sin}}\left( {\vartheta + \eta + {\rm{\Lambda }}} \right)} \right]}}\end{align}

where ${V_t}$ and ${\psi _{vt}}$ are the size and direction angle of the target velocity, respectively. Note that ${\psi _{vt}}$ is meaningless when the target is stationary, i.e., ${V_t} = 0$ .

Figure 1. Reinforcement learning adaptive spiral-diving manoeuver guidance framework.

The trajectory of the virtual sliding target $T{\rm{'}}$ is designed based on the curve involute principle and is denoted as

(25) \begin{align}{r_{vt}} = \left[ {\begin{array}{c}{{x_p}}\\[3pt]{{z_p}}\end{array}} \right] + {r_s}\left[ {\begin{array}{c}{{\rm{sin}}\left( {\vartheta + \eta } \right)}\\[3pt]{{\rm{cos}}\left( {\vartheta + \eta } \right)}\end{array}} \right] + {l_{go}}\left[ {\begin{array}{c}{{\rm{sin}}\left( {\vartheta + \eta + {\rm{\Lambda }}} \right)}\\[3pt]{{\rm{cos}}\left( {\vartheta + \eta + {\rm{\Lambda }}} \right)}\end{array}} \right]\end{align}

where ${r_{vt}} = {\left[ {{x_{vt}},{z_{vt}}} \right]^T}$ represents the coordinate vector of the $T{\rm{'}}$ . ${l_{go}}$ is the remaining length of the spiral trajectory and its value can be obtained by integrating Equation 12. The time derivative of Equation 25 yields

(26) \begin{align}{V_{vt}} = \left[ {\begin{array}{c}{{{\dot x}_{vt}}}\\[3pt]{{{\dot z}_{vt}}}\end{array}} \right] = {V_t}\left[ {\begin{array}{c}{{\rm{cos}}{\psi _{vt}}}\\[3pt]{ - {\rm{sin}}{\psi _{vt}}}\end{array}} \right] + {l_{go}}\dot \vartheta \left[ {\begin{array}{c}{{\rm{cos}}\left( {\vartheta + \eta + {\rm{\Lambda }}} \right)}\\[3pt]{ - {\rm{sin}}\left( {\vartheta + \eta + {\rm{\Lambda }}} \right)}\end{array}} \right]\end{align}

The virtual line-of-sight deflection angle from the current position of $T{\rm{'}}$ pointing to $M$ is defined as

(27) \begin{align}\varphi = {\rm{arctan}}\frac{{x - {x_{vt}}}}{{z - {z_{vt}}}}\end{align}

Taking the time derivative of Equation 27, and combining with Equation 1 and Equation 26, the following equation can be obtained:

(28) \begin{align}\dot \varphi = \frac{{V{\rm{cos}}\theta }}{s}{\rm{sin}}\left( {\delta {\psi _v}} \right) + \dot \vartheta {\rm{co}}{{\rm{s}}^2}\left( {{\rm{\Delta }}\varphi } \right) - \frac{{{V_t}}}{s}{\rm{cos}}\left( {{\psi _{vt}} - \varphi } \right)\end{align}

where

(29) \begin{align}\delta {\psi _v} = \frac{\pi }{2} + \varphi - {\psi _v}\end{align}

(30) \begin{align}{\rm{\Delta }}\varphi = - \frac{\pi }{2} + {\psi _{vs}} - \varphi \end{align}

$s = \sqrt {{{\left( {x - {x_{vt}}} \right)}^2} + {{\left( {z - {z_{vt}}} \right)}^2}} $ is the remaining flight distance.

Similarly, the virtual line-of-sight path angle from the current position of $T{\rm{'}}$ pointing to $M$ is defined as

(31) \begin{align}\phi = {\rm{arctan}}\left( {\frac{y}{s}} \right)\end{align}

Combining Equations 1 and 26, the time derivative of Equation 31 can be derived:

(32) \begin{align}\dot \phi = \frac{V}{r}\left( {{\rm{cos}}\phi {\rm{sin}}\theta + {\rm{sin}}\phi {\rm{cos}}\theta {\rm{cos}}\left( {\delta {\psi _v}} \right)} \right) + \frac{{{l_{go}}}}{r}\dot \vartheta {\rm{sin}}\phi {\rm{sin}}\left( {{\rm{\Delta }}\varphi } \right) + \frac{{{V_t}}}{r}{\rm{sin}}\phi {\rm{sin}}\left( {\varphi - {\psi _{vt}}} \right)\end{align}

where $r = \sqrt {{s^2} + {y^2}} $ is the distance of the vehicle from the virtual target.

Furthermore, taking the time deriving of Equation 13 yields

(33) \begin{align}{\dot \psi _{vs}} = \dot \vartheta \end{align}

Based on Equations 28, 32 and 33, the desired proportional navigation guiding law of the vehicle with respect to the virtual sliding target as shown in Equations 34 and 35 can be designed:

(34) \begin{align}{\dot \psi _{vd}} = - {\lambda _1}\dot \varphi + \left( {1 + {\lambda _1}} \right){\dot \psi _{vs}}\end{align}

(35) \begin{align}{\dot \theta _d} = - {\lambda _2}\dot \phi \end{align}

where ${\lambda _1}$ and ${\lambda _2}$ are user-defined guidance parameters, and their value ranges will be determined later.

3.2 Reinforcement learning adaptive controller

For the first-order multivariate tightly coupled system 4 considering the effects of unknown disturbances, treat ${g_1}\left( {\xi, u} \right)$ as the virtual control variable and define

(36) \begin{align}{e_2} = {g_1}\left( {\xi, u} \right) - \upsilon \left( {\xi, u} \right)\end{align}

where $\upsilon $ is the virtual control law.

Taking the time derivative of Equation 7 and combining it with Equation 36 yields

(37) \begin{align}{\dot e_1} = {f_1} + {e_2} + \upsilon + d - {\dot \xi _d}\end{align}

so the virtual controller can be designed as

(38) \begin{align}\upsilon = - {k_1}{e_1} - {f_1} - \hat d + {\dot \xi _d}\end{align}

where ${k_1} \gt 0$ is a user-defined control gain. $\hat d$ is the estimation of $d$ .

Taking the time derivative of Equation 36 leads to

(39) \begin{align}{\dot e_2} = \frac{{\partial {g_1}}}{{\partial \xi }}\dot \xi + \frac{{\partial {g_1}}}{{\partial u}}\dot u - \dot \upsilon \end{align}

so the time derivative of the controller $u$ can be designed as

(40) \begin{align}\dot u = {\left( {\frac{{\partial {g_1}}}{{\partial u}}} \right)^{ - 1}}\left( { - {k_2}{e_2} - {e_1} - \frac{{\partial {g_1}}}{{\partial \xi }}\left( {{f_1} + {g_1} + \hat d} \right) + \dot \upsilon } \right)\end{align}

where ${k_2} \gt 0$ is a user-defined control gain. By integrating Equation 40, $u$ can be obtained.

From Equations 38 and 40, it is clear that how to obtain the estimate of $d$ is the premise of designing the controller $u$ . Ingeniously, the actor-critic networks in reinforcement learning provide a superior alternative to deal with the problem.

In the framework of the actor network, $d$ can theoretically be expressed by

(41) \begin{align}d = W_a^{{\rm{*}}T}{{\rm{\Phi }}_a}\left( \xi \right) + {\varepsilon _a}\end{align}

where ${{\rm{\Phi }}_a} \in {\mathbb{R}^{{l_a}}}$ is the basis function of dimension ${l_a}$ , which satisfies $\|{{\rm{\Phi }}_a}\| \le {{\rm{\Phi }}_{aM}}$ . ${\varepsilon _a} \in {\mathbb{R}^2}$ is the actor reconstruction error and satisfies $\|{\varepsilon _a}\| \le {\varepsilon _{am}}$ . $W_a^{\rm{*}} \in {\mathbb{R}^{{l_a} \times 2}}$ is the real actor network weight. The reality is that only the estimation of $d$ can be obtained:

(42) \begin{align}\hat d = \hat W_a^T{{\rm{\Phi }}_a}\left( \xi \right)\end{align}

where ${\hat W_a} \in {\mathbb{R}^{{l_a} \times 2}}$ is the estimated weight of the actor network.

In the framework of the critic network, the integral penalty function can be designed as

(43) \begin{align}J\left( t \right) = \mathop \smallint \nolimits_\tau ^\infty \,{\mathcal{L}}\left( t \right)dt\end{align}

where ${\mathcal{L}}\left( t \right) = e_1^TQ{e_1}$ , $Q \in {\mathbb{R}^{2 \times 2}}$ is a positive definite matrix. $J$ can theoretically be expressed by

(44) \begin{align}J = W_c^{{\rm{*}}T}{{\rm{\Phi }}_c}\left( {{e_1}} \right) + {\varepsilon _c}\end{align}

where ${{\rm{\Phi }}_c} \in {\mathbb{R}^{{l_c}}}$ is the basis function of dimension ${l_c}$ , which satisfies $\|{\dot{\Phi}_c}\| \le {{\rm{\Phi }}_{cdM}}$ . ${\varepsilon _c}$ is the critic reconstruction error and satisfies $\left| {{{\dot \varepsilon }_c}} \right| \le {\varepsilon _{cdm}}$ . $W_c^{\rm{*}} \in {\mathbb{R}^{{l_c}}}$ is the real critic network weight. The reality is that only the estimation of $J$ can be obtained:

(45) \begin{align}\hat J = \hat W_c^T{{\rm{\Phi }}_c}\left( {{e_1}} \right)\end{align}

where ${\hat W_c} \in {\mathbb{R}^{{l_c}}}$ is the estimated weight of the critic network.

Define the weight error of critic network as ${\hat W_c} = {\hat W_c} - W_c^{\rm{*}}$ . In addition, define the critic error as

(46) \begin{align}{e_c} = {\mathcal{L}} + \dot{\hat{J}} = {\mathcal{L}} + \hat W_c^T{\dot{\Phi}_c}\end{align}

and the critic error function can be designed as

(47) \begin{align}{E_c} = \frac{1}{2}e_c^2\end{align}

According to the gradient descent criterion, the adaptive update law of ${\hat W_c}$ can be deduced as follows:

(48) \begin{align}{\dot{\hat{W_c}}} = - {\lambda _c}\left( {{\mathcal{L}} + \hat W_c^T{{\dot{\Phi}}_c}} \right){\dot{\Phi}_c} - {\lambda _c}{\hbar _c}{\hat W_c}\end{align}

where ${\lambda _c} \gt 0$ and ${\hbar _c} \gt 0$ are the user-defined learning rates of the critic network.

Define the weight error of actor network as ${\hat W_a} = {\hat W_a} - W_a^{\rm{*}}$ . And define the approximation error ${H_a}$ as

(49) \begin{align}{H_a} = \hat W_a^T{{\rm{\Phi }}_a} - W_a^{{\rm{*}}T}{{\rm{\Phi }}_a} = \hat W_a^T{{\rm{\Phi }}_a}\end{align}

Then, the actor error can be defined as

(50) \begin{align}{e_a} = {H_a} + {{\rm{\Omega }}_a}\hat J\end{align}

where ${{\rm{\Omega }}_a} \in {\mathbb{R}^{2 \times 1}}$ is the user-defined gain matrix satisfying $\|{{\rm{\Omega }}_a}\| \le {{\rm{\Omega }}_{aM}}$ , and ${{\rm{\Omega }}_{aM}}$ is a positive constant.

Furthermore, the actor error function can be designed as

(51) \begin{align}{E_a} = \frac{1}{2}e_a^T{e_a}\end{align}

According to the gradient descent criterion, the adaptive update law of ${\hat W_a}$ can be deduced as follows:

(52) \begin{align} {\dot{\hat{W_a}}} & = - {\lambda _a}{{\rm{\Phi }}_a}e_a^T - {\lambda _a}{\hbar _a}{\hat W_a}\nonumber\\& = - {\lambda _a}{{\rm{\Phi }}_a}\left( {{\rm{\Phi }}_a^T{{\hat W}_a} + \hat J{\rm{\Omega }}_a^T} \right) - {\lambda _a}{\hbar _a}{\hat W_a}\end{align}

where $a \gt 0$ and ${\hbar _a} \gt 0$ are the user-defined learning rates of the actor network.

3.3 Stability and convergence analysis

Proof. Construct the Lyapunov candidate function as follows:

(53) \begin{align}L = {L_1} + {L_2} + {L_3}\end{align}

where

(54) \begin{align}{L_1} = \frac{1}{2}e_1^T{e_1} + \frac{1}{2}e_2^T{e_2}\end{align}

(55) \begin{align}{L_2} = \frac{1}{2}Tr\left( {\hat W_a^T\lambda _a^{ - 1}{{\hat W}_a}} \right)\end{align}

(56) \begin{align}{L_3} = \frac{1}{2}Tr\left( {\hat W_c^T\lambda _c^{ - 1}{{\hat W}_c}} \right)\end{align}

By taking time derivative of Equation 54 and combining Equations 37–42, it can be deduced that

(57) \begin{align} {\dot L_1} & = e_1^T{\dot e_1} + e_2^T{\dot e_2}\nonumber\\&= - {k_1}e_1^T{e_1} + e_1^T{e_2} - e_1^T\hat W_a^T{{\rm{\Phi }}_a} + e_1^T{\varepsilon _a} - {k_2}e_2^T{e_2} - e_2^T{e_1} - e_2^T\frac{{\partial g}}{{\partial \xi }}\hat W_a^T{{\rm{\Phi }}_a} + e_2^T\frac{{\partial g}}{{\partial \xi }}{\varepsilon _a}\nonumber\\& \le - {k_1}e_1^Te + \frac{1}{2}e_1^T{e_1} + \frac{1}{2}{\rm{\Phi }}_a^T{\hat W_a}\hat W_a^T{{\rm{\Phi }}_a} + \frac{1}{2}e_1^T{e_1} + \frac{1}{2}\varepsilon _a^T{\varepsilon _a} \nonumber\\& \quad -{k_2}e_2^T{e_2} + \frac{1}{2}e_2^T\frac{{\partial g}}{{\partial \xi }}{\left( {\frac{{\partial g}}{{\partial \xi }}} \right)^T}{e_2} + \frac{1}{2}{\rm{\Phi }}_a^T{\hat W_a}\hat W_a^T{{\rm{\Phi }}_a} + \frac{1}{2}e_2^T\frac{{\partial g}}{{\partial \xi }}{\left( {\frac{{\partial g}}{{\partial \xi }}} \right)^T}{e_2} + \frac{1}{2}\varepsilon _a^T{\varepsilon _a}\nonumber\\& \le - \left( {{k_1} - 1} \right)e_1^T{e_1} - \left( {{k_2} - Tr\left( {\frac{{\partial g}}{{\partial \xi }}{{\left( {\frac{{\partial g}}{{\partial \xi }}} \right)}^T}} \right)} \right)e_2^T{e_2} + {\rm{\Phi }}_{aM}^2Tr\left( {\hat W_a^T{{\hat W}_a}} \right) + \varepsilon _{am}^2\end{align}

By taking time derivative of Equation 55 and combining Equation 52, it can be deduced that

(58) \begin{align}{\dot L_2} & = Tr\left( {\hat W_a^T\lambda _a^{ - 1}{{\dot{\hat{W}}}_a}} \right)\nonumber\\& = Tr\left( {\tilde W_a^T\lambda _a^{ - 1}\left[ { - {\lambda _a}{{\rm{\Phi }}_a}\left( {{\rm{\Phi }}_a^T{{\hat W}_a} + \hat J{\rm{\Omega }}_a^T} \right) - {\lambda _a}{\hbar _a}{{\hat W}_a}} \right]} \right)\nonumber\\& \le - Tr\left( {\tilde W_a^T{{\rm{\Phi }}_a}{\rm{\Phi }}_a^T{{\tilde W}_a}} \right) + \frac{1}{2}Tr\left( {\tilde W_a^T{{\rm{\Phi }}_a}{\rm{\Phi }}_a^T{{\tilde W}_a}} \right) + \frac{1}{2}Tr\left( {W_a^T{{\rm{\Phi }}_a}{\rm{\Phi }}_a^T{W_a}} \right)\nonumber\\& \quad + \frac{1}{2}Tr\left( {\tilde W_a^T{{\rm{\Phi }}_a}{\rm{\Phi }}_a^T{{\tilde W}_a}} \right) + \frac{1}{2}Tr\left( {\tilde W_c^T{{\rm{\Phi }}_c}{\rm{\Omega }}_a^T{{\rm{\Omega }}_a}{\rm{\Phi }}_c^T{{\tilde W}_c}} \right)\nonumber\\& \quad + \frac{1}{2}Tr\left( {\tilde W_a^T{{\rm{\Phi }}_a}{\rm{\Phi }}_a^T{{\tilde W}_a}} \right) + \frac{1}{2}Tr\left( {W_c^T{{\rm{\Phi }}_c}{\rm{\Omega }}_a^T{{\rm{\Omega }}_a}{\rm{\Phi }}_c^T{W_c}} \right)\nonumber\\& \quad - {\hbar _a}Tr\left( {\tilde W_a^T{{\tilde W}_a}} \right) + \frac{1}{2}{\hbar _a}Tr\left( {\tilde W_a^T{{\tilde W}_a}} \right) + \frac{1}{2}{\hbar _a}Tr\left( {W_a^T{W_a}} \right)\nonumber\\& \le - \frac{1}{2}\left( {{\hbar _a} - {\rm{\Phi }}_{aM}^2} \right)Tr\left( {\tilde W_a^T{{\tilde W}_a}} \right) + \frac{1}{2}\left( {{\hbar _a} + {\rm{\Phi }}_{aM}^2} \right)Tr\left( {W_a^T{W_a}} \right)\nonumber\\& \quad + \frac{1}{2}{\rm{\Phi }}_{cM}^2{\rm{\Omega }}_{aM}^2Tr\left( {\tilde W_c^T{{\tilde W}_c}} \right) + \frac{1}{2}{\rm{\Phi }}_{cM}^2{\rm{\Omega }}_{aM}^2Tr\left( {W_c^T{W_c}} \right)\end{align}

Similarly, by taking time derivative of Equation 3.3 and combining Equation 48, it can be deduced that

(59) \begin{align} {\dot L_3} & = Tr\left( {\tilde W_c^T\lambda _c^{ - 1}{{\tilde W}_c}} \right)\nonumber\\[3pt]& = Tr\left( {\tilde W_c^T\lambda _c^{ - 1}\left[ { - {\lambda _c}\left( {{\mathcal{L}} + \hat W_c^T{{\dot{\Phi}}_c}} \right){{\dot{\Phi}}_c} - {\lambda _c}{\hbar _c}{{\hat W}_c}} \right]} \right)\nonumber\\[3pt]& \le - Tr\left( {\tilde W_c^T{{\dot{\Phi}}_c}\dot{\Phi}_c^T{{\tilde W}_c}} \right) + Tr\left( {\tilde W_c^T{{\dot{\Phi}}_c}\dot{\Phi}_c^T{{\tilde W}_c}} \right) + Tr\left( {W_c^T{{\dot{\Phi}}_c}\dot{\Phi}_c^T{W_c}} \right) + \frac{1}{2}Tr\left( {\tilde W_c^T{{\dot{\Phi}}_c}\dot{\Phi}_c^T{{\tilde W}_c}} \right)\nonumber\\[3pt]& \quad + \frac{1}{2}\varepsilon _{cdm}^2 - {\hbar _c}Tr\left( {\tilde W_c^T{{\tilde W}_c}} \right) + \frac{1}{2}{\hbar _c}Tr\left( {\tilde W_c^T{{\tilde W}_c}} \right) + \frac{1}{2}{\hbar _c}Tr\left( {W_c^T{W_c}} \right)\nonumber\\[3pt]& \le - \frac{1}{2}\left( {{\hbar _c} - {\rm{\Phi }}_{cdM}^2} \right)Tr\left( {\tilde W_c^T{{\tilde W}_c}} \right) + \frac{1}{2}\left( {{\hbar _c} + 2{\rm{\Phi }}_{cdM}^2} \right)Tr\left( {W_c^T{W_c}} \right) + \frac{1}{2}\varepsilon _{cdm}^2\end{align}

At last, by taking time derivative of Equation 53 and substituting Equations 57–59, we get

(60) \begin{align} \dot L & = {\dot L_1} + {\dot L_2} + {\dot L_3}\nonumber\\[3pt]& \le - \left( {{k_1} - 1} \right)e_1^T{e_1} - \left( {{k_2} - Tr\left( {\frac{{\partial g}}{{\partial \xi }}{{\left( {\frac{{\partial g}}{{\partial \xi }}} \right)}^T}} \right)} \right)e_2^T{e_2} - \frac{1}{2}\left( {{\hbar _a} - 3{\rm{\Phi }}_{aM}^2} \right)Tr\left( {\tilde W_a^T{{\tilde W}_a}} \right)\nonumber\\[3pt]& \quad - \frac{1}{2}\left( {{\hbar _c} - {\rm{\Phi }}_{cdM}^2 - {\rm{\Phi }}_{cM}^2{\rm{\Omega }}_{aM}^2} \right)Tr\left( {\tilde W_c^T{{\tilde W}_c}} \right) + \varepsilon _{am}^2 + \frac{1}{2}\varepsilon _{cdm}^2\nonumber\\[3pt]& \quad + \frac{1}{2}\left( {{\hbar _a} + {\rm{\Phi }}_{aM}^2} \right)Tr\left( {W_a^T{W_a}} \right) + \frac{1}{2}\left( {{\hbar _c} + 2{\rm{\Phi }}_{cdM}^2 + {\rm{\Phi }}_{cM}^2{\rm{\Omega }}_{aM}^2} \right)Tr\left( {W_c^T{W_c}} \right)\end{align}

Equation 60 satisfies $\dot L \le - \kappa L\left( t \right) + \delta $ under the condition that $\left( {{k_1} - 1} \right) \gt 0$ , $\left( {{k_2} - Tr\left( {\frac{{\partial g}}{{\partial \xi }}{{\left( {\frac{{\partial g}}{{\partial \xi }}} \right)}^T}} \right)} \right) \gt 0$ , $\left( {{\hbar _a} - 3{\rm{\Phi }}_{aM}^2} \right) \gt 0$ and $\left( {{\hbar _c} - {\rm{\Phi }}_{cdM}^2 - {\rm{\Phi }}_{cM}^2{\rm{\Omega }}_{aM}^2} \right) \gt 0$ , where

\begin{align*}\kappa & = {\rm{min}}\left\{ {\begin{array}{c}{2\left( {{k_1} - 1} \right),2\left( {{k_2} - Tr\left( {\frac{{\partial g}}{{\partial \xi }}{{\left( {\frac{{\partial g}}{{\partial \xi }}} \right)}^T}} \right)} \right),}\\[3pt]{\left( {{\hbar _a} - 3{\rm{\Phi }}_{aM}^2} \right),\left( {{\hbar _c} - {\rm{\Phi }}_{cdM}^2 - {\rm{\Phi }}_{cM}^2{\rm{\Omega }}_{aM}^2} \right)}\end{array}} \right\}\\[5pt]\delta & = \frac{1}{2}\left( {{\eta _a} + {\rm{\Phi }}_{aM}^2} \right)Tr\left( {W_a^T{W_a}} \right) + \frac{1}{2}{\rm{\Phi }}_{cM}^2{\rm{\Omega }}_a^2Tr\left( {W_c^T{W_c}} \right) + \varepsilon _{am}^2 + \frac{1}{2}\varepsilon _{cdm}^2\end{align*}

Therefore, ${e_1}$ , ${e_2}$ , ${\tilde W_a}$ and ${\tilde W_c}$ are uniformly ultimately bounded.

Proof. It follows from Theorem 1 that

(61) \begin{align}{\psi _v} = {\psi _{vd}} + {e_{12}}\end{align}

where $\left| {{e_{12}}} \right|$ is an arbitrarily small constant. And the time derivative of Equation 61 gives

(62) \begin{align}{\dot \psi _v} = {\dot \psi _{vd}}\end{align}

Taking the time derivative of Equation 29 and substituting Equations 28, 34 and 62, the following equation can be derived:

(63) \begin{align} \delta {\dot \psi _v} & = \dot \varphi - {\dot \psi _v} = \left( {1 + {\lambda _1}} \right)\left( {\dot \varphi - {{\dot \psi }_{vs}}} \right)\nonumber\\[5pt]& = \left( {1 + {\lambda _1}} \right)\left( {\frac{{V{\rm{cos}}\theta }}{s}{\rm{sin}}\left( {\delta {\psi _v}} \right) - \frac{{{V_t}}}{s}{\rm{cos}}\left( {{\psi _{vt}} - \varphi } \right) - \dot \vartheta {\rm{si}}{{\rm{n}}^2}\left( {{\rm{\Delta }}\varphi } \right)} \right)\end{align}

Neglecting the second-order small quantity $\dot \vartheta {\rm{si}}{{\rm{n}}^2}\left( {{\rm{\Delta }}\varphi } \right)$ and the action term $\frac{{{V_t}}}{s}{\rm{cos}}\left( {{\psi _{vt}} - \varphi } \right)$ of the low-speed moving target in Equation 63, it can be rewritten as

(64) \begin{align}\delta {\dot \psi _v} = \left( {1 + {\lambda _1}} \right)\frac{{V{\rm{cos}}\theta }}{s}{\rm{sin}}\left( {\delta {\psi _v}} \right)\end{align}

A similar treatment to time derivative of $s$ produces

(65) \begin{align}\dot s = - V{\rm{cos}}\theta {\rm{cos}}\left( {\delta {\psi _v}} \right)\end{align}

Dividing Equation 65 by Equation 64, the Equation 66 can be obtained:

(66) \begin{align}\frac{{ds}}{{d\left( {\delta {\psi _v}} \right)}} = - \frac{s}{{1 + {\lambda _1}}}\frac{{{\rm{cos}}\left( {\delta {\psi _v}} \right)}}{{{\rm{sin}}\left( {\delta {\psi _v}} \right)}}\end{align}

And the Equation 67 can be obtained by integrating the Equation 66:

(67) \begin{align}s = \ell {\left| {{\rm{sin}}\left( {\delta {\psi _v}} \right)} \right|^{ - \frac{1}{{1 + {\lambda _1}}}}}\end{align}

where $\ell \gt 0$ is the bounded integration constant.

If $\delta {\psi _v}\left( t \right)$ satisfies $0 \lt \delta {\psi _v}\left( 0 \right) \lt \frac{\pi }{2}$ at $t = 0$ , then by substituting Equation 67 into Equation 64, we can get

(68) \begin{align}\delta {\dot \psi _v} = \left( {1 + {\lambda _1}} \right)\frac{{V{\rm{cos}}\theta }}{\ell }{\left( {{\rm{sin}}\left( {\delta {\psi _v}} \right)} \right)^{\frac{{{\lambda _1} + 2}}{{{\lambda _1} + 1}}}}\end{align}

Note that $V{\rm{cos}}\theta \gt 0$ always holds no matter in which flight state. In the case $0 \lt \delta {\psi _v}\left( 0 \right) \lt \frac{\pi }{2}$ , when the guidance parameter ${\lambda _1} \lt - 1$ , there is $\delta {\dot \psi _v} \lt 0$ , which indicates that $\delta {\psi _v} \to 0$ as $t \to \infty $ . From Equation 67, it can be found that the remaining flight distance between the vehicle and the virtual sliding target $s \to 0$ as $\delta {\psi _v} \to 0$ . Furthermore, from Equation 29, it can be found that $\varphi - {\psi _v} \to - \frac{\pi }{2}$ as $\delta {\psi _v} \to 0$ . From Equation 68, when ${\lambda _1} \lt - 2$ , there is $\delta {\dot \psi _v} \to 0$ as $\delta {\psi _v} \to 0$ , so $\dot \varphi - {\dot \psi _v} \to 0$ , and combining Equations 34 and 62, it can be observed that ${\dot \psi _{vs}} \to {\dot \psi _v}$ . Therefore, the flight trajectory converges to the spiral trajectory. This means that ${\rm{\Delta }}\varphi \to 0$ , i.e. $\varphi - {\psi _{vs}} \to - \frac{\pi }{2}$ . $\delta {\psi _v} \to 0$ and ${\rm{\Delta }}\varphi \to 0$ indicate that velocity vector of the vehicle converges to the virtual line-of-sight. The same conclusion can be obtained when $ - \frac{\pi }{2} \lt \delta {\psi _v}\left( 0 \right) \lt 0$ .

Proof. It follows from Theorem 1 that

(69) \begin{align}\theta = {\theta _d} + {e_{11}}\end{align}

where $\left| {{e_{11}}} \right|$ is an arbitrarily small constant. And the time derivative of Equation 69 gives

(70) \begin{align}\dot \theta = {\dot \theta _d}\end{align}

Define $\sigma = \phi + \theta $ , by taking its time derivative and combining Equations 35 and 70, we get

(71) \begin{align}\dot \sigma = \dot \phi + \dot \theta = \left( {1 - {\lambda _2}} \right)\dot \phi \end{align}

Taking the time derivative of $r$ , and then neglecting the low-speed moving target action term and noting that $\delta {\psi _v} \to 0$ , ${\rm{\Delta }}\varphi \to 0$ as $t \to \infty $ , yields

(72) \begin{align}\dot r = - V{\rm{cos}}\sigma \end{align}

Similarly, there is

(73) \begin{align}\dot \phi = \frac{V}{r}{\rm{sin}}\sigma \end{align}

Dividing Equation 72 by Equation 71 yields

(74) \begin{align}\frac{{dr}}{{d\sigma }} = \frac{r}{{{\lambda _2} - 1}}\frac{{{\rm{cos}}\sigma }}{{{\rm{sin}}\sigma }}\end{align}

And the Equation 75 can be obtained by integrating the Equation 74:

(75) \begin{align}r = \ell {\rm{'}}{\left| {{\rm{sin}}\sigma } \right|^{\frac{1}{{{\lambda _2} - 1}}}}\end{align}

where $\ell {\rm{'}} \gt 0$ is the bounded integration constant. Combining Equations 71, 73 and 75, the Equation 76 can be organized:

(76) \begin{align}\dot \sigma = \left( {1 - {\lambda _2}} \right)\frac{V}{{\ell {\rm{'}}}}{\left( {{\rm{sin}}\sigma } \right)^{\frac{{{\lambda _2} - 2}}{{{\lambda _2} - 1}}}}\end{align}

when $0 \lt \sigma \left( 0 \right) \lt \pi $ , if the guidance parameter ${\lambda _2} \gt 1$ , then $\dot \sigma \lt 0$ . Therefore $\sigma \to 0$ as $t \to \infty $ . From Equation 75, it can be found that spatial distance from the vehicle to the virtual sliding target $r \to 0$ as $\sigma \to 0$ . If ${\lambda _2} \gt 2$ , there is $\dot \sigma \to 0$ . Because $\dot \sigma = \left( {1 - {\lambda _2}} \right)\dot \phi $ , so $\dot \phi \to 0$ . Thus, $\phi $ approaches a constant and $\theta $ approaches the negative of the same constant. That is, in the pitch plane, $\phi + \theta \to 0$ . The same conclusion can be obtained when $ - \pi \lt \sigma \left( 0 \right) \lt 0$ .