2.1. Tracking error model

The wheeled mobile robot is shown in Fig. 1, and it is assumed that its particle and center point are coincident. The model is described as

(1) \begin{equation} [\dot{x},\dot{y},\dot{\theta }, \dot{v},\dot{\omega }]^{\top }=[v\cos \theta, v\sin \theta,\omega, a_{v}, a_{\omega }]^{\top }, \end{equation}

where $q=(x,y)$ is the position of the robot in the global coordinate frame and $\theta$ is the heading angle related to the robot’s kinematic model. $v$ and $\omega$ denote the linear velocity and angular velocity of the robot, respectively. $a_{v}$ and $a_{\omega }$ are the accelerations of $v$ and $\omega$ , respectively, related to the dynamic model of the robot. The control input is $u=[a_{v},a_{\omega }]^{\top }$ , and the state of the robot is defined as $p=[x,y,\theta,v,\omega ]^{\top }$ .

Fig. 1. Tracking error model of the wheeled robot.

Suppose a virtual robot guides the robot to move forward, and its state is described as $p_r=[{x}_{r},{y}_{r},{\theta }_{r},{v}_{r},{\omega }_{r}]^{\top }$ , satisfying:

(2) \begin{equation} [\dot x_r,\dot y_r,\dot \theta _r, \dot v_r,\dot \omega _r]^{\top }=[v_r\cos \theta _r, v_r\sin \theta _r,\omega _r, a_{v,r}, a_{\omega,r}]^{\top }, \end{equation}

where $u_r=[a_{v,r}, a_{w,r}]^{\top }$ is the control input of the virtual leader robot.

The tracking error model of the robot can be written as

(3) \begin{equation} \begin{aligned} e=T(p_r-p), \end{aligned} \end{equation}

where $T$ is the coordinate transformation matrix from the global coordinate frame $XOY$ to the robot’s local coordinate frame $xoy$ , which is

(4) \begin{equation} T=\left [ \begin{array}{c@{\quad}c@{\quad}c} \cos \theta & \sin \theta & 0\\ \\[-8pt] -\sin \theta & \cos \theta & 0\\ \\[-8pt] 0& 0& I_3\\ \end{array} \right ]. \end{equation}

Then, we can get the tracking error model in derivative form by differentiating both sides of (3), which is

(5) \begin{equation} \dot{e}=T{(\dot{p_r}-\dot{p})}+\dot{T}{(p_r-p)}. \end{equation}

According to Eqs. (1)–(2) and (5), the discrete-time tracking error model according to the forward-Euler discretization can be written as

(6) \begin{equation} \left \{\begin{aligned} e_x\left( k+1 \right) &=e_x\left( k \right) +\Delta h\left( \omega \left( k \right) e_y\left( k \right) -v\left( k \right) + \right. \left. v_r\left( k \right) \cos e_{\theta}\left( k \right) \right)\\e_y\left( k+1 \right) &=e_y\left( k \right) +\Delta h\left( -\omega \left( k \right) e_x\left( k \right) + \right. \left. v_r\left( k \right) \sin e_{\theta}\left( k \right) \right)\\e_{\theta}\left( k+1 \right) &=e_{\theta}\left( k \right) +\Delta h\left( \omega _r\left( k \right) -\omega \left( k \right) \right)\\e_v\left( k+1 \right) &=e_v\left( k \right) +\Delta h\left( a_{v,r}\left( k \right) -a_v\left( k \right) \right)\\e_{\omega}\left( k+1 \right) &=e_{\omega}\left( k \right) +\Delta h\left( a_{\omega ,r}\left( k \right) -a_{\omega}\left( k \right) \right) , \end{aligned}\right. \end{equation}

where $e=[e_{x},e_{y},e_{\theta },e_{v},e_{\omega }]^{\top }$ , $v(k)=v_r(k)-e_{v}(k)$ , $\omega (k)=\omega _r(k)-e_{\omega }(k)$ , $k$ is the discrete-time index, and $\Delta h$ is the adopted sampling interval.

The nonlinear tracking error model can be abbreviated as

(7) \begin{equation} e(k+1)=f(e(k))+g(u(k)), \end{equation}

where $f$ is the error transition function, and $g$ is the input mapping function.

2.2. Problem formulation

We define the performance index of path-following control as

(8) \begin{equation} J(e(k))=\sum \nolimits _{k=0}^{\infty } \gamma ^{k} r(e(k),u(k)), \end{equation}

where

(9) \begin{equation} r(e(k),u(k))=\lVert e(k) \rVert _Q^2+\lVert u(k)-u_r(k)\rVert _{R}^2, \end{equation}

and $\gamma \in (0,1]$ is the discounting factor, $Q^{\top }=Q \in \mathbb{R}^{5\times 5}$ , $R^{\top }=R\in \mathbb{R}^{2\times 2}$ , $Q\succ 0$ , $R\succ 0$ .

The cost $J(e(k))$ is minimized subject to the following constraints $q(k)\in \mathcal{X}, \forall k\in [1,\infty ]$ , where

(10) \begin{equation} \begin{aligned} \mathcal{X}=\{q|\lVert q-q_{o}\rVert _2 \gt r+r_o\}, \end{aligned} \end{equation}

$q_{o},o\in \mathbb{N}_{1}^{n}$ , is the position of the obstacle $o$ . $r$ and $r_o$ are the radii of the smallest approximate circles surrounding the robot and obstacle $o$ , $o\in \mathbb{N}_{1}^{n}$ , respectively.

Below, we give the definition of the potential field function and the assumption on the detection rules for the robot.

Definition 1 (Potential field function):

For the robot’s position $q$ and the obstacle’s position $q_{o}\in \mathbb{R}^2$ , $o\in \mathbb{N}_{1}^{n}$ , the potential field function $\mathcal{B}_{o}$ is defined as

(11) \begin{equation} \mathcal{B}_{o}(q)= \begin{cases} \dfrac{1}{2} \kappa _{o}\left(\dfrac{1}{\lVert q-q_{o} \rVert _2}-\dfrac{1}{r_{o}+r}\right)^2, &\lVert q-q_{o} \rVert _2 \leq r_{o}+r \\ 0, & \lVert q-q_{o} \rVert _2 \gt r_{o}+r,\end{cases} \end{equation}

where $\kappa _{o}$ is the relaxation factor of the potential field. $\lVert q-q_{o} \rVert _2$ , $o\in \mathbb{N}_{1}^{n}$ , represents the distance between the robot and the obstacle $o$ . The depiction of the above parameters is shown in Fig. 2.

Fig. 2. Depiction of robot’s parameters in obstacle avoidance.

3.1. Design of P-DHP

To solve the path-following control problem with safety constraints, we propose a control policy that has two control parts, which is described as

(12) \begin{equation} {u}(k)=\bar{u}^{i_1}(e(k))+\sum \nolimits _{o=1}^{n}\eta _{o}(k)\nabla \mathcal{B}_{o}(k), \end{equation}

where $\bar{u}^{i_1}(e(k))$ is a control policy that corresponds to a fixed weight matrix without the obstacles’ constraints. $\eta _o \in \mathbb{R}^{2\times 2}$ , $o\in \mathbb{N}_{1}^{n}$ , are variables to be optimized for obstacles of different sizes and positions (see Section 3.2). $\nabla \mathcal{B}_{o}=\partial \mathcal{B}_{o}/\partial q$ .

To decouple the conflicting learning objectives of path-following control and obstacle avoidance, the learning process is divided into two parts, where $\bar u$ and $\eta _o$ in (12) are learned asynchronously by value iteration to guarantee convergence.

Then, the performance index of path-following control is redefined as

(13) \begin{equation} J_1( \bar e(k))=\sum \nolimits _{k=0}^{\infty } \gamma ^{k} r_1(\bar e(k),\bar u(k)), \end{equation}

where $r_1(\bar e(k),\bar u(k))=\lVert \bar e(k) \rVert _{Q_1}^2+\lVert \bar u(k)-u_r(k)\rVert _{R_1}^2$ , $Q_1^{\top }=Q_1 \in \mathbb{R}^{5\times 5}$ , $R_1^{\top }=R_1\in \mathbb{R}^{2\times 2}$ , $Q_1\succ 0$ , $R_1\succ 0$ . $\bar e(k+1)$ is the state of the robot that satisfies $\bar e(k+1)=f(\bar e(k))+g(\bar u(k))$ .

Define $J_1^*$ as the optimal value function of $J_1$ under the optimal control policy $\bar u^*$ . According to the Hamilton–Jacobi–Bellman (HJB) equation, one has

(14) \begin{equation} \begin{aligned} J_1^{*}(k)=\underset{\,\, {\bar u}(k)}{\rm min}\,\,{{r_1(\bar e(k),{\bar u}(k))}+\gamma J_1^{*}(k+1)},\\ \end{aligned} \end{equation}

and

(15) \begin{equation} \begin{aligned}{\bar u}^{*}(k)&=\underset{\,\,{\bar u}(k)}{\mathrm{argmin}}\,\,{r_1(\bar e(k),\bar u(k))+ \gamma J_1^{*}(k+1)}. \end{aligned} \end{equation}

Define $i$ as the number of iterations. Firstly, the control policy $\bar u(k)$ in the unconstrained scenario is learned within iterations $i=1,\ldots,i_{1}$ , and the update rule in each iteration of $J_1(k)$ and $\bar u(k)$ can be obtained as

(16) \begin{equation} \begin{aligned} J_1^{i+1}(k)={r_1(\bar e(k),{\bar u}^{i}(k))}+\gamma J_1^{i}(k+1),\\ \end{aligned} \end{equation}

(17) \begin{equation} \begin{aligned}{\bar u}^{i+1}(k)&=\underset{\,\,{\bar u}(k)}{\mathrm{argmin}}\,\,{r_1(\bar e(k),\bar u(k))+ \gamma J_1^{i+1}(k+1)}, \end{aligned} \end{equation}

where $i_{1}$ is the iterative value as $J_1^{i}(k)$ converges (see Algorithm 1 for details).

With the fixed control policy $\bar u^{i_1}(k)$ , the variable $\eta _o$ in the second item of (12) is updated within iterations $i=i_{1},\ldots,i_{2}$ to minimize the reconstructed cost function $ J_2(k)$ for collision avoidance constraints, which is

(18) \begin{equation} J_2(k)=\sum _{o=1}^{n}J_{2,o}(k), \end{equation}

(19) \begin{equation} J_{2,o}(k)=\sum _{k=0}^{\infty } \gamma ^{k}r_2(\mathcal{B}_{o}(k),e(k),u(k)), \end{equation}

where $r_2(\mathcal{B}_{o}(k),e(k),u(k))=\lVert e(k)-{\bar e}(k)\rVert _{Q_2}^2+\lVert u(k)-\bar{u}^{i_1}(k)\rVert _{R_2}^2+\mu \mathcal{B}_{o}(k)$ , ${Q}_2^{\top }={Q_2}\in \mathbb{R}^{5\times 5}$ , ${R}_2^{\top }={R_2}\in \mathbb{R}^{2\times 2}$ , ${Q_2}\succ 0$ , ${R_2}\succ 0$ , and $\mu \gt 0$ is a tuning parameter.

Let $\overrightarrow{\eta }$ be the sequence of ${\eta }_{1},\ldots,{\eta }_{n}$ . Then $J_{2}$ and $\overrightarrow{\eta }$ can be updated in each iteration $i=i_{1},\ldots,i_{2}$ by

(20) \begin{equation} \begin{aligned} J_{2}^{i+1}(k)=\sum _{o=1}^n{r_2(\mathcal{B}_{o}(k),e(k),{u}^{i}(k))}+\gamma J_{2}^{i}(k+1),\\ \end{aligned} \end{equation}

(21) \begin{equation} \begin{aligned} \overrightarrow{\eta }^{i+1}(k)&=\underset{\,\, \overrightarrow{\eta }(k)}{\mathrm{argmin}}\,\,\sum _{o=1}^n{r_2(\mathcal{B}_{o}(k),e(k),u(k))+ \gamma J_{2}^{i+1}(k+1).} \end{aligned} \end{equation}

The main steps of the proposed P-DHP can be seen in Algorithm 1.

In the following, we prove the convergence of our proposed P-DHP.

Algorithm 1. Pseudocode of P-DHP.

3.2. AC learning for P-DHP

In this section, the AC learning structures (see Fig. 3) are introduced to implement Algorithm 1. The AC learning structures consist of two decoupling parts. The actor-1–critic-1 network corresponds to the iteration of $J_1(k)$ and $\bar u(k)$ , and the actor-2–critic-2 network corresponds to the iteration of $J_2(k)$ and $\overrightarrow{\eta}$ , respectively. The AC network weights are updated in an incremental learning mode.

The critic-1 network is constructed as

(22) \begin{equation} \nabla \hat{J_1}(k) =W_{c}^{\top }(k)h_{c}(\bar e(k)), \end{equation}

where the costate $\nabla \hat{J_1}(k)=\partial \hat{J_1}(k)/\partial \bar e(k)$ , $W_{c}\in \mathbb{R}^{5\times 5}$ is the weighting matrix, and $h_{c}$ is the activation function.

Define $\nabla{J_1^{*}}=\partial{J_1^{*}}/\partial \bar e$ . To minimize the bias between $\nabla \hat{J_1}$ and $\nabla J_1^*$ , we designed a target to be followed by $\nabla \hat{J_1}$ , i.e.,

(23) \begin{equation} \begin{aligned} \nabla J_1^{d}(k)=2Q_1\bar e(k)+\gamma \nabla f(k)^{\top }\nabla \hat{J_1}(k+1), \end{aligned} \end{equation}

where $\nabla f=\partial{f}/\partial{\bar e}$ .

Let the quadratic approximation error of the critic-1 network be defined as $E_{c}(k)=||\nabla{\hat J_1}(k)-\nabla J_1^d(k)||_2^2$ . The goal of the critic-1 network is to minimize the distance between $\nabla{\hat{J_1}}(k)$ and $\nabla J_1^d(k)$ by updating weights. At each time instant $k$ , the update rule of $W_{c}$ is

(24) \begin{equation} \begin{aligned} W_{c}(k+1) &=W_{c}(k) -\alpha _{c}\frac{\partial E_{c}(k)}{\partial W_{c}(k)}, \end{aligned} \end{equation}

where $\alpha _{c}$ is the learning rate of $W_{c}$ .

Then we design the actor-1 network for the robot, which is

(25) \begin{equation} \hat{\bar u}(k) =W_{a}^{\top }(k)h_{a}(\bar e(k)), \end{equation}

where the $W_{a}\in \mathbb{R}^{5\times 2}$ is the weighting matrix, and $h_{a}$ is the activation function.

In view of (17), we define the $ \bar u^{d}(k)$ as the desired value of $\hat{\bar u}(k)$ , which is expressed as

(26) \begin{equation} \bar u^{d}(k) =-\frac{1}{2}\gamma{R_1}^{-1}\nabla g(k)^{\top }\nabla \hat{J_1}(k+1) +u_r(k), \end{equation}

where $\nabla g=\partial{g}/\partial{\bar u}$ , and $\nabla \hat{J_1}$ is the output of the critic-1 network.

Fig. 3. The structure of the actor–critic learning algorithm.

The quadratic approximation error of actor-1 network can be defined as $E_{a}(k)=||\hat{\bar u}(k)-\bar u^d(k)||_2^2$ , and the update rule of $W_{a}$ is

(27) \begin{equation} \begin{aligned} W_{a}(k+1) &=W_{a}(k) -\alpha _{a}\frac{\partial E_{a}(k)}{\partial W_{a}(k)},\\ \end{aligned} \end{equation}

where $\alpha _{a}$ is the learning rate of $W_{a}$ . Note that the learning process (27) is performed in $i_1$ iterations. We define the converged matrix $W_{a}$ as $\textbf{W}_{a}$ and the learned control policy $\hat{\bar u}$ as ${\hat{\bar{\textbf{u}}}}$ , respectively.

Suppose that the potential fields of obstacles do not overlap. In the following design, we decompose the actor-2–critic-2 network into $n$ subnetworks to learn the variable $\eta _o$ , $o\in \mathbb{N}_1^{n}$ , in (12) respectively, which corresponds to obstacles $o$ , $o\in \mathbb{N}_1^{n}$ , of different sizes and positions.

The sub-critic-2 network for each obstacle $o$ is given as

(28) \begin{equation} \nabla{\hat{J}_{2,o}}(k) =\mu _{c}{\textbf{W}_{\textbf{c}}}^{\top }{h}_{c}(e(k))+\rho _{o}(k)\nabla \mathcal{B}_{o}(k), \end{equation}

where $\rho _{o}\in \mathbb{R}^{2\times 2}$ , and $\nabla \hat J_{2,o}=\partial \hat J_{2,o}/\partial e$ . $\textbf{W}_c$ is the learned matrix of $W_{c}$ , and $\mu _{c}\gt 0$ is a tuning parameter.

The desired value of $\nabla \hat J_{2,o}$ i.e., $\nabla{J}_{2,o}^{d}$ is defined as

(29) \begin{equation} \begin{aligned} \nabla \hat{J}_{2,o}^{d}(k)=2Q_2(e(k)-\bar e(k))+\mu \left(\frac{\partial{q}(k)}{\partial{e}(k)}\right)^{\top }\nabla \mathcal{B}_{o}(k)+\gamma \nabla f(k)^{\top }\nabla \hat{J}_{2,o}(k+1). \end{aligned} \end{equation}

where $\nabla f=\partial{f}/\partial{e}$ .

Denote ${E}_{c,o}(k)=||\nabla \hat {J}_{2,o}(k)-\nabla {J}_{2,o}^d(k)||_2^2$ as the quadratic approximation error of the sub-critic-2 network. The update rule of $\rho _{o}$ can be obtained as

(30) \begin{equation} \begin{aligned} \rho _{o}(k+1)=\rho _{o}(k) -\beta _{c}\frac{\partial{E}_{c,o}(k)}{\partial \rho _{o}(k)},\\ \end{aligned} \end{equation}

where $\beta _{c}$ is the learning rate of $\rho _{o}$ , $o\in \mathbb{N}_{1}^{n}$ .

In view of (12), the sub-actor-2 network for each obstacle $o$ is designed with the following form:

(31) \begin{equation} \hat{u}_o(k) =\textbf{W}_{\textbf{a}}^{\top }{h}_{a}(e(k))+\eta _{o}(k)\nabla \mathcal{B}_{o}(k), \end{equation}

where $\hat{u}_o(k)$ is the control policy for the robot to avoid the obstacle $o$ , which is equal to ${\hat u}(k)$ in the case of only one obstacle $o$ to be avoided.

Define $\varepsilon _o =2R_2({\hat u_o}-\hat{\bar{\textbf{u}}})$ . Then the desired value of $\varepsilon _o(k)$ is redefined as

(32) \begin{equation} \varepsilon^d_o(k) =-\gamma\nabla g(k)^{\top} \nabla \hat{J}_{2,o}(k+1). \end{equation}

where $\nabla g=\partial{g}/\partial{u}$ .

In the same way, the quadratic approximation error of the sub-actor-2 network and the update rule of $\eta _o$ are given as

(33) \begin{equation} \begin{aligned}E_{a,o}(k)=||\varepsilon_o(k)-\varepsilon_o^d(k)||_2^2, \end{aligned} \end{equation}

(34) \begin{equation} \begin{aligned} \eta _{o}(k+1) =\eta _{o}(k) -\beta _{a}\frac{\partial{E}_{a,o}(k)}{\partial \eta _{o}(k)}, \\ \end{aligned} \end{equation}

where $\beta _{a}$ is the learning rate of $\eta _{o}$ , $o\in \mathbb{N}_{1}^{n}$ .

Given that all variables $\eta _o$ , $o\in \mathbb{N}_{1}^{n}$ , have been trained until converging, the overall control policy to be deployed to the robot is

(35) \begin{equation} \hat{u}(k) =\textbf{W}_{\textbf{a}}^{\top }{h}_{a}(e(k))+\sum \nolimits _{o=1}^{n}\eta _{o}(k)\nabla \mathcal{B}_{o}(k). \end{equation}

4.1. Simulation experiments

4.1.1. Parameter settings and training process

In the simulation, a relaxed potential function is defined and used as follows to guarantee the smoothness in the control process, i.e.,

(36) \begin{equation} \mathcal{B}_{o}^{r}(q)= \begin{cases}\mathcal{B}_{o}(q), &d_{o} \geq \delta _{o} \\ \\[-8pt] \sigma _{o}(q), &d_{o} \lt \delta _{o}, \end{cases} \end{equation}

where $d_o=\lVert q-q_{o} \Vert_2$ and $0\lt \delta _{o}\lt d_{o}$ is an adjustable relaxing factor. $\sigma _{o}(q)$ is a derivative and monotonic function, where $\sigma_{o}(q)=\frac{1}{2}\sigma_{1}(d_{o}-{\delta_{o}})^2+\sigma_{2}(d_{o}-{\delta_{o}})+\sigma_{3}$ , $\sigma _{1}=\nabla ^2_{d_{o}} \mathcal{B}_{o}(q)|_{d_{o}=\delta _{o}}$ , $\sigma _{2}=\nabla _{d_{o}} \mathcal{B}_{o}(q)|_{d_{o}=\delta _{o}}$ , $\sigma _{3}=\mathcal{B}_{o}(q)|_{d_{o}=\delta _{o}}$ .

Also, the perceived radius values of obstacles are properly expanded to construct $\nabla \mathcal{\bar B}_{o}(q)$ , which replaces $\nabla \mathcal{B}_{o}(q)$ in (12) to ensure control safety. The reconstructed $\mathcal{\bar B}_{o}(q)$ is defined as

(37) \begin{equation} \mathcal{\bar B}_{o}(q)= \begin{cases} \dfrac{1}{2} \bar \kappa _{o}\left(\dfrac{1}{d_{o}}-\dfrac{1}{r_{d}}\right)^2, &d_{o} \leq r_{d} \\ \\[-8pt] 0, & d_{o} \gt r_{d},\end{cases} \end{equation}

where $r_{d}$ indicates the maximum distance that the robot can detect the obstacles, and the definitions of relevant parameters can be seen in Fig. 2. The parameters satisfy $\kappa _{o}\lt \bar \kappa _{o}$ and $r_{o}+r\lt r_{d}$ . Similar to $\mathcal{B}_{o}^{r}(q)$ in (36), $ \mathcal{\bar B}_{o}(q)$ is deflated into $\mathcal{\bar B}_{o}^{r}(q)$ by $\bar \delta _{o}$ , $0\lt \bar \delta _{o}\lt r_{d}$ .

In the training process, the discounting factor was $\gamma =0.95$ . The penalty matrices were set as $Q_1=I_5$ , $R_1=0.01I_2$ , $Q_2=0.001I_5$ , $R_2=0.001I_2$ , $\mu =1$ , $\mu _c=0.001$ , $\alpha _{a}=0.2$ , $\alpha _{c}=0.4$ , $\beta _{a}=0.4$ , $\beta _{c}=0.4$ , $\delta _o=1.5$ , $\bar \delta _o=5$ , $\kappa _{o}=5.5$ , $\bar \kappa _{o}=600$ , $r_d=6$ , and $\Delta h=0.05\,\text{s}$ . In the learning process, the weights were initialized with uniformly distributed random values. The simulations were implemented in the Matlab 2019b environment on a laptop with an AMD Ryzen 7 4800H CPU.

Firstly, we tested our algorithm in a straight-line path scenario with one obstacle in the center of the road. The recorded performance costs were defined as $J^i_{r,1}=\sum \nolimits _{k=1}^{K_1}J_1^i(k)$ and $ J^i_{r,2}=\sum \nolimits _{k=1}^{K_2}J_2^i(k)$ with $K_1=1000$ and $K_2=1000$ in each training iteration. The variation results of $J^i_{r,1}$ and $J^i_{r,2}$ in Fig. 4 show that both the performance costs decrease with iteration increases and converge in 30 iterations. Moreover, the robot’s trajectories in the learning process and the actor weights of the robot in a single-obstacle scenario are displayed in Figs. 5 and 6. The results illustrate that the robot gradually moves away from the obstacle by continuous iterative learning until the robot’s trajectory is circumscribed by the edge of the obstacle’s potential field.

Fig. 4. Variations of $J^{i}_{r,1}$ and $J^{i}_{r,2}$ in the learning process.

Fig. 5. Iteration of robot’s trajectories in a single-obstacle scenario based on P-DHP algorithm. The radii of the robot and obstacle are $r=1\,\text{m}$ and $r_o=1\,\text{m}$ , respectively.

Fig. 6. Actor-2 weights of robot with the increase of iteration, where $ \eta_{o}(i,j)$ refers to the element in the i-th row and j-th column of matrix $\eta_{o},$ and $o=1$ .

Table I. Comparative results of P-DHP and CS-based RL in 100 tests.

4.1.2. Comparison results with state-of-the-art approaches

In this subsection, we compared our approach with the reward/cost shaping-based RL approach (CS-based RL) [Reference Perkins and Barto27], the MPC using a disjunctive chance constraint (MPC-dc) [Reference Castillo-Lopez, Ludivig, Sajadi-Alamdari, Sanchez-Lopez, Olivares-Mendez and Voos28], and the MPC using a linearized chance constraint (MPC-lc) [Reference Zhu and Alonso-Mora29]. Please refer to Fig. 9 for the adopted reference paths in the simulation tests.

In the CS-based RL approach, the cost function was constructed as $r=\lVert e(k) \rVert _{Q_1}^2+\lVert u(k)-u_r(k)\rVert _{R_1}^2+\mu\nabla\mathcal{B}_{o}(k)$ , and the parameters of $\nabla\mathcal{B}_{o}$ were fine-tuned. For a fair comparison, the same DHP-based AC learning structure(actor-1-critic-1) was used in CS-based RL. The comparative results in 100 tests with different learning rates are illustrated in Table I and Fig. 7, which show that our approach outperforms the CS-based RL approach in the success rate of the learning process. Although the success rate between ours and CS-based RL is similar as the learning rate is set small, as shown in Fig. 8,  the CS-based RL approach has a probability of failure to avoid obstacles due to the lack of corrective terms $\nabla \mathcal{B}_{o}$ in the control policy. As the learning rate increases, the learning failure rate of the CS-based RL approach also increases and has a longer obstacle avoidance distance with more energy loss, whereas our method possesses stable control performance for different learning rates. This illustrates, in part, that only shaping the cost function may not be sufficient to learn a convergent control policy efficiently via a standard AC learning algorithm. To solve this problem, our P-DHP approach are proposed to resolve the conflict between the path-following control and obstacle avoidance goals in the learning process.

Fig. 7. Simulation results of our approach and the CS-based RL [Reference Perkins and Barto27] with $v_r=1\,\text{m/s}$ , $r=1.3\,\text{m}$ , $r_o=0.7\,\text{m}$ or $r_o=1\,\text{m}$ . The CS-based RL method has a certain probability of learning failure, see Table I for details.

Fig. 8. Statistics on the closest distance from the robot to the obstacle in 100 repetitive tests. The comparisons were made between P-DHP and the CS-based RL [Reference Perkins and Barto27] in the single-obstacle scenario (shown in Fig.5). The safe bound is represented in red, and the outliers are drawn as blue circles.

The MPC-dc [Reference Castillo-Lopez, Ludivig, Sajadi-Alamdari, Sanchez-Lopez, Olivares-Mendez and Voos28] and MPC-lc [Reference Zhu and Alonso-Mora29] algorithms were implemented by the CasADi toolbox with an Ipopt solver [Reference Andersson, Gillis, Horn, Rawlings and Diehl30]. The stage costs in MPC-dc and MPC-lc were designed the same as (9) in our approach. The performance indicator $J_e$ was defined as $J_e=\text{1/}\bar K\sum_{k\in K}{\lVert e(k) \rVert_2}$ , where $K$ is the time interval in obstacle avoidance, and $\bar K$ is the length of $K$ . The comparative study was carried out under the scenario with three different sizes of obstacles, where the side lengths of the bounding boxes were set as $2.3/\sqrt{2}\,\text{m}$ , $2/\sqrt{2}\,\text{m}$ , and $2.6/\sqrt{2} \,\text{m}$ . The prediction horizon was set as $N=30$ . For the details of the simulation parameters, please refer to ref. [Reference Castillo-Lopez, Ludivig, Sajadi-Alamdari, Sanchez-Lopez, Olivares-Mendez and Voos28].

The comparative simulation results are illustrated in Fig. 9 and Tables II and III. The results show that our approach outperforms the fixed-parameters MPC-lc and MPC-dc with smaller cost values and shorter path length in obstacle avoidance, demonstrating that our approach has better control performance at different reference velocities. This is due to the learning optimization mechanism and policy design of our algorithm. In addition, our approach has advantages in computational efficiency (see Table III).

Table II. Numerical comparison of P-DHP, MPC-dc and MPC-lc.

Table III. Computational time comparison ( $v_r=1\,\text{m/s}$ ).

Fig. 9. Simulation results of our approach, MPC-lc [Reference Zhu and Alonso-Mora29] and MPC-dc [Reference Castillo-Lopez, Ludivig, Sajadi-Alamdari, Sanchez-Lopez, Olivares-Mendez and Voos28] with $v_r=1.6\,\text{m/s}$ : the filled blue circular area represents the obstacles and the light blue dotted line indicates the influence range of the obstacle potential field. The simulation scene contains three sizes of obstacles, corresponding to $r_o=0.7\,\text{m}$ , $r_o=1\,\text{m}$ and $r_o=1.3\,\text{m}$ respectively. The radius of the robot is $r = 1.3\,\text{m}$ .

4.2. Real-world experiments

We also tested our algorithm on the experimental platform of a real-world differential-drive wheeled robot. As shown in Fig.10,the robot was controlled by a laptop equipped with the Ubuntu operating system, and the state of the robot and the positions of obstacles were obtained by mounted satellite inertial. In the experiment, we deployed the offline training weights from the simulation to generate control policy $u=[a_{v}, a_{\omega }]^{\top }$ to drive the underlying system of the robot, and the sampling interval was $\Delta h=0.1\,\text{s}$ .

Fig. 10. Experimental platform of the differential-drive wheeled robot.

Fig. 11. Experiment results of robot’s path aerial view in a real-world scenario.

Fig. 12. Experiment results of robot’s state errors in real-world scenario.

The real-world experimental results are shown in Figs. 11 and 12, which illustrate that the robots can follow the desired path from an initial state, meanwhile avoiding all encountered obstacles on the path and recovering path following after collision avoidance. Moreover, the above-reported results show that our approach can resolve the weight divergence problem caused by the conflict between path following and obstacle avoidance.