2.1. Environment and problem definition

As shown in Fig. 1, the whole environment consists of a vascular robotic system, a 3D-printed vascular model, a guidewire, and a catheter. The vascular robotic system (hereinafter referred to as robot) has two degrees of freedom (translation and rotation) [Reference Zhao, Liu, Zhou, Xie, Hou, Zhou, Zhang, Gui and Wang35]. The robot needs to deliver the guidewire to a target in the vascular model. The target is illustrated in Fig. 1(c). The size of the vascular model is 15 $\times$ 15 $\times$ 1.5 cm, where vessels are about 3−5 mm thick. The environment is randomly reset to avoid overfitting to specific states. Episodes start in initial states where the distal tip of the guidewire is randomly 10−15 pixels from the outset and the guidewire is rotated randomly, and end when the distal tip reaches the target or borders.

Figure 1. The preclinical environment. (a) The whole environment. The camera and lights are omitted for simplicity. (b) Close-up of the guidewire and the vascular model. (c) The target in the vascular model and the correct path from the outset to the target.

The guidewire delivery problem is defined as a partially observable Markov decision process $\langle \mathcal{S}, \mathcal{O}, \mathcal{A}, P, R, \gamma \rangle$ , where $\mathcal{S}$ is the state space, $\mathcal{O}$ is the observation space, $\mathcal{A}$ is the action space, $P: \mathcal{S}\times \mathcal{A}\times \mathcal{S}\mapsto [0,1 ]$ is the transition probability, $R:\mathcal{S}\times \mathcal{A}\mapsto \mathbb{R}$ is the reward function, and $\gamma$ is the discount factor. At time-step $t$ , the robot receives an observation $o_t\in \mathcal{O}$ and chooses an action $a_t\in \mathcal{A}$ based on the stochastic policy $\pi (\cdot |o_t)$ . Observations are 140 $\times$ 140 size images taken from a fixed camera directly above the vascular model. For generality and simplicity, each observation is prepossessed into binary images of the vessel and the guidewire as in Fig. 2. Actions are speed commands in translation and rotation freedom. The action space consists of 10 discrete actions. There are two sub-actions in the translation freedom, which are forward and backward with the same speed, and five sub-actions in the rotation freedom, including static, two-speed counterclockwise, and clockwise rotation. The interval between actions is 0.5 s. Then the state of environment $s_t$ changes to $s_{t+1}$ according to the transition probability $P(s_{t+1}|s_t, a_t)$ , and the robot gets a reward $r_t=R(s_t, a_t)$ and a new observation $o_{t+1}$ . The goal is to find optimal policy that maximizes the maximum entropy objective $\sum _{t}\mathbb{E}_{o_t}\{\mathbb{E}_{a_t\sim \pi (\cdot |o_t)}\{r_t+\alpha \mathcal{H}[\pi (\cdot |o_t)]\}\}$ [Reference Zie, bart, Maas, Bagnell and Dey36], where $\alpha$ is the temperature parameter that balances the reward and the policy stochasticity. In the following, the subscript indicating the time-step will be truncated if the time-step does not need to be considered.

Figure 2. Prepossessed images as observations in RL. (a) The vessel. (b) The guidewire.

The reward $r_t$ guides the learning process and requires careful design. To ensure efficacy and safety, the reward consists of three sub-rewards :

1. (1) The sparse sub-reward that indicates completion of guidewire delivery. The agent will receive a large sparse bonus of size 400 if the target is reached; otherwise, the sparse sub-reward is 0. The guidewire is considered as reaching the target when the distal tip of the guidewire is within five pixels from the target.

2. (2) The dense sub-reward that encourages the guidewire to approach the target along the correct path. The correct path is the shortest path from the outset to the target as shown in Fig. 1(c), which is automatically generated by the Dijkstra’s algorithm. When the guidewire keeps on the correct path, the robot will get a reward that is equal to the decrease in distance to the target. The distance of each point to the target is calculated by the Dijkstra’s algorithm simultaneously with the correct path. The robot will get a penalty of size 20 if the guidewire deviates from the correct path and goes into a wrong vessel branch, and a bonus of size 20 if the guidewire leaves the wrong vessel branch and goes back to the correct path. The robot will receive another penalty of size 50 if the guidewire exits the vascular model.

3. (3) The safe sub-reward. The robot will get a large penalty of size 200 if the contact force exceeds a safe threshold.

As it is a challenge to measure the contact force between the guidewire and the vascular model, safety during RL is ensured by limiting the contact force between the robot and the guidewire. The contact force between the robot and the guidewire is represented by the motor current.

2.2. Discrete soft actor-critic with auto-encoder

To improve sample efficiency, we propose discrete soft actor-critic with auto-encoder (DSAC-AE) that uses an auxiliary reconstruction task as SAC-AE [Reference Yarats, Zhang, Kostrikov, Amos, Pineau and Fergus33]. The whole architecture of DSAC-AE is shown in Fig. 3. The shared encoder compresses the input images into a vectorized embedding. Following the encoder, the decoder reconstructs the input, while the two multilayer perceptrons (MLP) represent soft Q-function $Q$ and policy $\pi$ , respectively. The output of soft Q-function $Q$ and policy $\pi$ is $|\mathcal{A}|$ -dimension vectors representing soft Q-value and probability of each action, respectively. In the following, $Q(o)$ and $\pi (o)$ represent output vectors of soft Q-function $Q$ and policy $\pi$ respectively, while $Q(o,a)$ and $\pi (a|o)$ represent elements corresponding to action $a$ in $Q(o)$ and $\pi (o)$ , respectively.

Figure 3. DSAC-AE: SAC-Discrete is augmented with an auto-encoder. The encoder is updated with gradients from soft Bellman residual $J(Q)$ and the auxiliary reconstruction task. Focal loss is used as the loss function of auxiliary reconstruction task to mitigate the impact of foreground-background imbalance.

Soft Q-function $Q$ and policy $\pi$ is updated according to SAC [Reference Haarnoja, Zhou, Hartikainen, Tucker, Ha, Tan, Kumar, Zhu, Gupta, Abbeel and Levine37]. Soft Q-function $Q$ are trained by minimizing the soft Bellman residual

(1) \begin{equation} J(Q) = \mathbb{E}_{(o_t, a_t, r_t, o_{t+1})} \left\{ \left[Q(o_t, a_t) - r_t - \gamma V(o_{t+1})\right]^2 \right\}, \end{equation}

where $V$ is the soft state value function

(2) \begin{equation} V(o) = \mathbb{E}_{a \sim \pi (\cdot |o)} \left[\bar{Q}(o, a) - \alpha \log \pi (a|o) \right]. \end{equation}

$\bar{Q}$ denotes the target Q-function whose parameters are the exponentially moving average of soft Q-function parameters. Policy $\pi$ is updated by minimizing the following formula

(3) \begin{equation} J(\pi ) = \mathbb{E}_{o} \big \{\mathbb{E}_{a \sim \pi (\cdot |o)} \{\alpha \log [\pi (a|o)]-Q(o, a)\}\big \}. \end{equation}

The temperature parameter $\alpha$ is updated by minimizing the following loss function

(4) \begin{equation} J(\alpha ) = \mathbb{E}_{o} \big \{\mathbb{E}_{a \sim \pi (\cdot |o)} \{-\alpha \log [\pi (a|o)]-\alpha \bar{\mathcal{H}}\}\big \}, \end{equation}

where hyperparameter $\bar{\mathcal{H}}$ is the target entropy. Since the action space $\mathcal{A}$ is discrete, expectations on actions are directly computed without Monte Carlo method as in SAC-Discrete [Reference Christodoulou38] to decrease variance of loss functions. This means that Eqs. (2), (3), and (4) are changed to:

(5) \begin{equation} V(o) = \pi (o)^{T}[\bar{Q}(o) - \alpha \log \pi (o) ], \end{equation}

(6) \begin{equation} J(\pi ) = \mathbb{E}_{o} \big \{\pi (o)^{T} \{\alpha \log [\pi (o)]-Q(o)\}\big \}, \end{equation}

(7) \begin{equation} J(\alpha ) =\mathbb{E}_{o}\left\{ -\alpha \pi (o)^{T}\log [\pi (o)]-\alpha \bar{\mathcal{H}}\right\}. \end{equation}

The decoder is updated by minimizing reconstruction error. Since vessels and guidewires are narrow, the foreground is much smaller than the background in prepossessed images. To mitigate the impact of foreground-background imbalance, focal loss [Reference Lin, Goyal, Girshick, He and Dollár39] is used for the auxiliary reconstruction task:

(8) \begin{equation} \textrm{FL}(\hat{o}, o) = \sum \limits _{i,j} -o_{ij}\left(1-\hat{o}_{ij}\right)^{\tau }\log (\hat{o}_{ij}) - (1-o_{ij})\hat{o}_{ij}^{\tau }\log \left(1-\hat{o}_{ij}\right), \end{equation}

where $\hat{o}$ is the reconstruction obtained from the decoder, hyperparameter $\tau$ controls sensitivity to the foreground, and the subscript indicates the position of image pixels.

The shared encoder is updated by the gradient from the decoder and soft Q-function, while the gradient from policy is prevented to update the encoder to improve stability [Reference Yarats, Zhang, Kostrikov, Amos, Pineau and Fergus33].

2.3. Distributed actor and learner

Using digital simulation, sample collection can be performed quickly and in parallel with negligible time. In such cases, RL algorithm are often applied with the paradigm that alternately collects samples and updates parameters. But in the environment which uses a real robot as shown in Section 2.1, each step of sample collection has to wait for the action execution. Therefore, both sample collection and parameter update are time-consuming. To shorten the training time, sample collection and parameter update of DSAC-AE are decoupled and performed parallelly in two processes.

The distributed structure is shown in Fig. 4. The structure is composed of an actor, a dataset, and a learner. The actor interacts with the environment and transmits transitions $(o_t, a_t, r_t, o_{t+1})$ to the dataset. The actor copies parameters from the leaner at set intervals. It is worth noting that only the encoder and the policy are used in the actor. The dataset acts as a common replay buffer in off-policy RL algorithms, storing transitions obtained from the actor and providing randomly sampled transitions to the learner. The learner updates parameters of the whole model as described in Section 2.2. Sample collection and parameter update are executed asynchronously. The data transmission among the actor, the dataset, and the learner is implemented using the ray package [Reference Moritz, Nishihara, Wang, Tumanov, Liaw, Liang, Paul, Jordan and Stoica40].

Figure 4. Distributed actor, dataset, and learner: collecting transitions, storing and sampling transitions, and updating parameters of DSAC-AE are performed distributedly by actor, dataset, and learner.