2. System Model

As shown in Figure 1, an information transmitter communicates with its receiver while perceived by K nearby RF energy harvesters [Reference Xing and Dong8]. Both the transmitter and the receiver are equipped with M antennas, while each RF energy harvester is equipped with one receive antenna. The baseband received signal at the receiver can be represented as

(1) $\begin{array}{}y=Hx+z,\end{array}$

where $H\in {ℂ}^{M\times M}$ denotes the normalized baseband equivalent channel from the information transmitter to its receiver, $x\in {ℂ}^{M\times 1}$ represents the transmitted signal, and $z\in {ℂ}^{M\times 1}$ is the zero-mean circularly symmetric complex Gaussian noise with $z\sim \mathrm{CN}\left(0,{\rho }^{2}I\right)$ .

FIGURE 1: Wireless information transmitter and receiver surrounded by multiple RF energy harvesters.

The transmit covariance matrix is denoted by Q, i.e., $Q=\text{E}\left(x{x}^H\right)$ . The covariance matrix is Hermitian positive semidefinite, i.e., $Q\succcurlyeq 0$ . The transmit power is restricted by the transmitter’s power constraint P, i.e., $\text{Tr}\left(Q\right)\le P$ . For the information transmission, we assume that a Gaussian codebook with infinitely many code words is used for the symbols and the expectation of the transmit covariance matrix is taken over the entire codebook. Therefore, x is the zero-mean circularly symmetric complex Gaussian with $x\sim \mathrm{CN}\left(0,Q\right)$ . With transmitter precoding and receiver filtering, the capacity of the MIMO channel is the sum of the capacities of the parallel noninterfering single-input single-output (SISO) channels (eigenmodes of channel H) [Reference Biglieri, Calderbank, Constantinides, Goldsmith, Paulraj and Poor19]. We convert the MIMO channel to M eigenchannels for information and energy transfer [Reference Timotheou, Krikidis, Karachontzitis and Berberidis20, Reference Mishra and Alexandropoulos21]. A singular value decomposition (SVD) on H gives $H=U\Sigma V^H$ , where $\Sigma =\text{diag}\left({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_M\right)$ contains the M singular values of H. Since the MIMO channel is decomposed into M parallel SISO channels, the information rate can be given by

(2) $\begin{array}{}r=\sum _{m=1}^M\log \left(1+{\rho }^{-2}{\left({\sigma }_m\right)}^2{\hat{q}}_m\right),\end{array}$

where $\left({\hat{q}}_m\right)$ are the diagonal elements of $\hat{Q}$ with $\hat{Q}=V^{H}QV$ .

The RF energy harvester received power specifies the harvested energy normalized by the baseband symbol period and scaled by the energy conversion efficiency. The received power at the ith energy harvester is

(3) $\begin{array}{}{p}_i=g_i^{H}Q{g}_i,\end{array}$

where $g_i\in {ℂ}^{M\times 1}$ is the channel vector from the transmitter to the ith energy harvester. With MIMO channel decomposition, the received power at energy harvester i is denoted as

(4) $\begin{array}{}{p}_i=\sum _{m=1}^M{\left({\hat{g}}_{im}\right)}^2{\hat{q}}_m,\end{array}$

where $\left({\hat{g}}_{im}\right)$ are the elements of vector ${\hat{g}}_i$ with ${\hat{g}}_i=V^H{g}_i$ .

We define the simplified channel vector from the transmitter to the ith RF energy harvester as

(5) $\begin{array}{}{c}_i={\left({\left({\hat{g}}_{i1}\right)}^2,{\left({\hat{g}}_{i2}\right)}^2,\dots ,{\left({\hat{g}}_{iM}\right)}^2\right)}^T,\end{array}$

for each $i\in K=\left(\mathrm{1,2},\dots ,K\right)$ . The simplified channel vector contains no phase information. The K simplified channel vectors compose matrix $C\in {ℝ}^{M\times K}$ as

(6) $\begin{array}{}C=\left(c_1,c_2,\dots ,c_K\right).\end{array}$

In what follows, we assume that time is slotted, each time slot as a duration T, and that each energy harvester is equipped with an energy buffer of size $B_i\in \left(0,B_{\max }\right)$ , $i\in K$ . Without loss of generality, we assume that, at t = 0, all harvesters’ buffers are empty, which corresponds to system state $s_0=\left(\mathrm{0,0},\dots ,0\right)$ . At a generic time slot t, the transmitter transmits with one of the designed beam patterns. Each harvester i can harvest the specific amount of power p_i, and its energy buffer values increase to $B_i^{t+1}=B_i^t+p_{i}T$ . Therefore, each state of the system includes the accumulated harvested energy information of all K harvesters, i.e.,

(7) $\begin{array}{}{s}_t=\left(B_1^t,B_2^t,\dots ,B_K^t\right),\end{array}$

where $B_i^t$ denotes the ith energy harvester’s accumulated energy up to time slot t.

Once all harvesters are fully charged, we assume that the system arrives at a final goal state denoted as $s_G=\left(B_{\max },B_{\max },\dots ,B_{\max }\right)$ . We note that the energy buffer level B _max also accounts for situations in which $B_i>B_{\max }$ .

3. Problem Formulation for Time-Varying Channel Conditions

In this section, we suppose that the communication link is characterized by strong LOS transmission, which results in an invariant channel matrix H, while the energy harvesting channel vector g varies over time slots. We model the wireless charging problem as a Markov decision process (MDP) and show how to solve the optimization problem using reinforcement learning (RL). When the number of system states is very large, we apply a deep Q-network algorithm to acquire the optimal strategy at each particular system state.

3.1. Problem Formulation

In order to model our optimization problem as a RL problem, we define the beam pattern chosen in a particular time slot t as the action a_t. The set of possible actions A is determined by equally generating L different beam patterns with power allocation vector $\hat{q}=\left({\hat{q}}_1,\dots ,{\hat{q}}_m\right)$ that satisfies the power and information rate constraints, i.e., ${\sum }_{i=1}^M{\hat{q}}_m=P$ , ${\sum }_{i=1}^M\log \left(1+{\rho }^{-2}{\left({\sigma }_m\right)}^2{\hat{q}}_m\right)\ge R$ . Each beam pattern corresponds to a particular power level p_i, which depends not only on the action a^t but also on the channel condition experienced by the harvester during time slot t.

Given the above, the simultaneous wireless information and energy transfer problem for a time-varying channel can be formulated as minimizing the time-consumption n to fully charge all the energy harvesters while maintaining the information rate between the information transceivers:

(8) $\begin{array}{}{P}_1:\begin{array}{cc}\underset{\left(a^t\right)}{\text{minimize}}& n\\ \text{subject to}& a_m^t\ge 0\\ & \sum _{m=1}^M{a}_m^t\le P\\ & \sum _{m=1}^M\log \left(1+{\rho }^{-2}{\left({\sigma }_m\right)}^2{a}_m^t\right)\ge R\\ & \sum _{t=1}^n\sum _{m=1}^M{\left({\hat{g}}_{im}^t\right)}^2{a}_m^{t}T\ge B_{\max },\hspace{1em}\forall i\in K\end{array}.\end{array}$

In general, the action selected at each time slot will be different to adapt to the current channel conditions and current energy buffer state of the harvesters. Therefore, the evolution of our system can be described by a Markov chain, where the generic state s is identified by the current buffer levels of the harvester, i.e., $s=\left(B_1,B_2,\dots ,B_K\right)$ . The set of all states is denoted by S. Among all states, we are interested in the state in which all harvesters’ buffer is empty, namely, $s_0=\left(0,\dots ,0\right)$ , and the state s_G in which all the harvesters are fully charged, i.e., $s_G=\left(B_{\max },\dots ,B_{\max }\right)$ . If we suppose that we know all the channel coefficients at each time slot, problem P ₁ can be seen as a stochastic shortest path (SSP) problem from state s ₀ to state s_G. At each time slot, the system is in a generic state s, the transmitter selects a beam pattern or action $a\in A$ , and the system moves to a new state s′. The dynamics of the system is captured by transition probabilities $p_{s,s^{\prime }}\left(a\right)$ , $s,s^{\prime }\in S$ , and $a\in A$ , describing the probability that the harvesters’ energy buffers reach the levels in s′ after a transmission with beam pattern a. We note that the goal state s_G is absorbing, i.e., $P_{s_G,s_G}\left(a\right)=1,\forall a\in A$ .

Each transition also has an associated reward, $w\left(s,a,s^{\prime }\right)$ , that denotes the reward when the current state is $s\in S$ , action $a\in A$ is selected, and the system moves to state $s^{\prime }\in S$ . Since we aim at reaching s_G in the fewest transmission time slots, we consider that the action entails a positive reward related to the difference between the current energy buffer level and the full energy buffer level of all harvesters. When the system reaches state s_G, we set the reward as 0. In this way, the system not only tries to fully charge all harvesters in the shortest time but will also uniformly charge all the harvesters. In detail, we define the reward function as

(9) $\begin{array}{}w\left(s,a,s^{\prime }\right)=-\lambda \left({\text{KB}}_{\max }-\sum _{i=1}^K\min \left(s_i^{\prime },B_{\max }\right)\right),\end{array}$

where

(10) $\begin{array}{}\lambda s_i^{\prime }=\lambda s_i+\lambda \sum _{m=1}^M{\left({\hat{g}}_{im}\right)}^2{a}_{m}T,\end{array}$

and λ denotes the unit price of the harvested energy.

It is noted that different reward functions can also be selected. As an example, it is also possible to set a constant negative reward (e.g., a unitary cost) for each transmission that the system does not reach the goal state and a big positive reward only for the states and actions that bring the system to the goal state s_G. This can be expressed as follows:

(11) $\begin{array}{}w\left(s,a,s^{\prime }\right)=\left(\begin{array}{cc}+\infty ,& s^{\prime }=s_G,\\ -1,& \text{otherwise}.\end{array}\right)\end{array}$

We note that the reward formulation of equation (11) is actually equivalent to minimizing the number of time slots required to reach state s_G starting from state s ₀.

Using the above formulation, the optimization problem $P=\left(S,A,p,w,s_0,s_G\right)$ can then be seen as a stochastic shortest path search from state s ₀ to state s_G on the Markov chain with states S and probabilities $\left(p_{s,s^{\prime }}\left(a\right)\right)$ , actions $a\in A$ , and rewards $w\left(s,a,s^{\prime }\right)$ . Our objective is to find, for each possible state $s\in S$ , an optimal action $a^{\ast }\left(s\right)$ so that the system will reach the goal state following the path with maximum average reward. A generic policy can be written as $\pi =\left(a\left(s\right):s\in S\right)$ .

Different techniques can be applied to solve problem P ₁, as it represents a particular class of MDPs. In this paper, however, we assume that the channel conditions at each time slot are unknown, which corresponds to not knowing the transition probabilities $\left(p_{s,s^{\prime }}\left(a\right)\right)$ . Therefore, in the next section, we describe how to solve the above problem using reinforcement learning.

3.2. Optimal Power Allocation with Reinforcement Learning

Reinforcement learning is suitable for solving optimization problems in which the system dynamics follow a particular transition probability function, however, the probabilities $\left(p_{s,s^{\prime }}\left(a\right)\right)$ are unknown. In what follows, we first show how to apply the Q-learning algorithm [Reference Barto, Bradtke and Singh22] to solve the optimization problem and then show how we can combine the reinforcement learning approach with a neural network to approximate the system model in case of large states and action sets, using deep Q-network [Reference Mnih, Kavukcuoglu and Silver12].

3.2.1. Q-Learning Method

If the number of system states is small, we can depend on the traditional Q-learning method to find the optimal strategy at each system state, as defined in the previous section.

To this end, we define the cost function of action a on system state s as $\left(p_{s,s^{\prime }}\left(a\right)\right)$ , with $s\in S,a\in A$ . The algorithm initializes with $Q\left(s,a\right)=0$ and then updates the Q values using the following equation:

(12) $\begin{array}{}Q\left(s,a\right)=\left(1-\alpha \left(s,a\right)\right)Q\left(s,a\right)+\alpha \left(s,a\right)\left(w\left(s,a,s^{\prime }\right)+\gamma f\left(s^{\prime },a\right)\right),\end{array}$

where

(13) $\begin{array}{}f\left(s^{\prime },a\right)=\underset{a\in A}{\min }Q\left(s^{\prime },a\right),\end{array}$

and $\alpha \left(s^{\prime },a\right)$ denotes the learning rate. In each time slot, only one Q value is updated, and hence, all the other Q values remain the same.

At the beginning of the learning iterations, since the Q-table does not have enough information to choose the best action at each system state, the algorithm randomly explores new actions. Hence, we first define threshold ${\epsilon }_c\in \left(0.5,1\right)$ , and we then randomly generate a probability $p\in \left(\mathrm{0,1}\right)$ . In the case that $p\ge {\epsilon }_c$ , we choose the action a as

(14) $\begin{array}{}a=\underset{a\in A}{\max }Q\left(s,a\right).\end{array}$

On the contrary, if $p<{\epsilon }_c$ , we randomly select one action from the action set A.

When Q ^∗ converges, the optimal strategy at each state is determined as

(15) $\begin{array}{}{\pi }^{\ast }\left(s\right)=\text{arg}\underset{a\in A}{\max }{Q}^{\ast }\left(s,a\right),\end{array}$

which corresponds to finding the optimal beam pattern for each system state during the charging process.

3.2.2. Deep Q-Network

When considering a complex system with multiple harvesters, large energy buffers, and time-varying channel conditions, the number of system states dramatically increases. In order to learn the optimal transmit strategy at each system state, the Q-learning algorithm described before requires a Q-table with a large number of elements, making it very difficult for all the values in the Q-table to converge. Therefore, in what follows, we describe how to apply the deep Q-network (DQN) approach to find the optimal transmission policy.

The main idea of DQN is to train a neural network to find the Q function of a particular system state and action combination. When the system is in state s, and action a is selected, the Q function is denoted as $Q\left(s,a,\theta \right)$ . θ denotes the parameters of the Q-network. The purpose of training the neural network is to make

(16) $\begin{array}{}Q\left(s,a,\theta \right)\approx Q^{\ast }\left(s,a\right).\end{array}$

According to the DQN algorithm [Reference Van Hasselt, Guez and Silver17], two neural networks are used to solve the problem: the evaluation network and the target network, which are denoted as eval_net and target_net, respectively. Both the eval_net and the target_net are set up with several hidden layers. The input of the eval_net and the target_net are denoted as s and s′, which describe the current system state s and the next system state s′, respectively. The output of eval_net and target_net are denoted as $Q_e\left(s,a,\theta \right)$ and $Q_t\left(s,a,\theta \right)$ , respectively. The evaluation network is continuously trained to update the value of θ; however, the target network only copies the weight parameters from the evaluation network intermittently (i.e., ${\theta }^{\prime }=\theta$ ). In each neural network learning epoch, the loss function is defined as

(17) $\begin{array}{}\text{loss}\left(\theta \right)=E\left({\left(y-Q_e\left(s,a,\theta \right)\right)}^2\right).\end{array}$

where y represents the real Q value and is calculated as

(18) $\begin{array}{}y=w\left(s,a,s^{\prime }\right)+\epsilon \underset{a^{\prime }\in A}{\max },Q_t\left(s^{\prime },a^{\prime },{\theta }^{\prime }\right),\end{array}$

where ε is the learning rate. As the loss function updates, the values are backpropagated to the neural network to update the weight of the eval_net.

ALGORITHM 1: Deep Q-network algorithm training process.

In order to better train the neural network, we apply the experience reply method to remove the correlation between different training data. Each experience consists of the current system state s, the action a, the next system state s′, and the corresponding reward $w\left(s,a,s^{\prime }\right)$ . The experience is denoted by the set $ep=\left(s,a,w\left(s,a,s^{\prime }\right),s^{\prime }\right)$ . The algorithm records D experiences, and randomly select D_s (with $D_s<D$ ) experiences from D for training. After the training is finished, target_net clones all the weight parameters from the eval_net (i.e., ${\theta }^{\prime }=\theta$ ).

The algorithm used for the DQN training process is presented in Algorithm 1. In the algorithm, we define in each training iteration, we generate D usable experiences ep and select D_s of all for training the eval_net. In total, we suppose there are U training iterations. We consider that, for both the eval_net and the target_net, there are N_l layers in the neural network. In the learning process, we use C to denote all energy harvesters’ channel condition in a particular time slot.

3.2.3. Dueling Double Deep Q-Network

Since more harvesters and time-varying channel conditions incur more system states, even if we utilize the original DQN, it is hard to study the transmit rules for the transmitter. Therefore, we can apply dueling double DQN in order to deal with the overestimating problem during the training process and improve the learning efficiency of the neural network. Doubling DQN is a technique that strengthens the traditional DQN algorithm by preventing overestimating to happen [Reference Van Hasselt, Guez and Silver17]. In traditional DQN, as shown in equation (18), we utilize the target_net to predict the maximum Q value of the next state. However, the target_net is not updated at every training episode, which may lead to an increase in the training error and therefore complicate the training process. In doubling DQN, we utilize both the target_net and the eval_net to predict the Q value. The eval_net is used to determine the optimal action to be taken for the system state s′ as follows:

(19) $\begin{array}{}y=w\left(s,a,s^{\prime }\right)+\epsilon \underset{a^{\prime }\in A}{\max }{Q}_e\left(s^{\prime },\underset{a\in A}{\text{arg}}\max Q\left(s^{\prime },a,\theta \right),{\theta }^{\prime }\right).\end{array}$

It can be shown that, following this approach, the training error considerably decreases [Reference Van Hasselt, Guez and Silver17].

In traditional DQN, the neural network only has the Q value as the output. In order to speed up the convergence, we apply dueling DQN by setting up two output streams from the neural network. The first stream is represented by the output value $V\left(s,\theta ,\beta \right)$ results of the neural network, which represents the Q value of each system state. The second stream is called advantage output $A\left(s^{\prime },a,\theta ,\alpha \right)$ and describes the advantage of applying each particular action to the current system state [Reference Wang, Schaul, Hessel, Van Hasselt, Lanctot and De Freitas18]. α and β are parameters that relate the two streams and the neural network output, which is denoted as

(20) $\begin{array}{}Q\left(s,a,\theta ,\alpha ,\beta \right)=V\left(s,\theta ,\beta \right)+\left(A\left(s^{\prime },a,\theta ,\alpha \right)-\frac{1}{\left(A\right)}\sum _{a^{\prime }}A\left(s^{\prime },a,\theta ,\alpha \right)\right).\end{array}$

Dueling DQN can efficiently eliminate the extra training freedom, which speeds up the training [Reference Wang, Schaul, Hessel, Van Hasselt, Lanctot and De Freitas18].

4. Simulation Results

We simulate a MIMO wireless communication system with nearby RF energy harvesters. The wireless transmitter has at most M = 4 antennas. The 4 × 4 communication MIMO channel matrix H is measured by two Wireless Open-Access Research Platform (WARP) v3 boards. Both WARP boards are mounted with the FMC-RF-2X245 dual-radio module, which is operated in 5.805 GHz frequency band. The Xilinx Virtex-6 FPGA operates as the central processing system and the WARPLab is used for rapid physical layer prototyping which is compiled by MATLAB [Reference Dong and Liu23]. We deploy two transceivers as line-of-sight transmission. The maximum transmitted power is P = 12W. ${\rho }^2=-70$ dBm. The information rate requirement R is 53 bps/Hz. The average channel gain from the transmitter to the energy harvester is −30 dB. The energy conversion efficiency is 0.1. The duration of one time slot is defined as T = 100 ms.

DQN is trained to solve for the optimal transmit strategies for each system state. The simulation parameters used for DQN are presented in Table 1.

TABLE 1: DQN simulation parameters.

As described in Section 3.2, the exploration rate ${\epsilon }_c$ determines the probability that the network selects an action randomly or follows the values of the Q-table. Initially, we set ${\epsilon }_c=1$ because the experience pool has to accumulate reasonable amount of data to train the neural network. ${\epsilon }_c=1$ decreases with 0.001 at each training interval and finally stops at ${\epsilon }_{ch}=0.1$ , since the experience pool has collected enough training data.

Refer to [Reference Xing, Pan, Xu, Zhao, Tapparello and Qian24]. The dueling double DQN is used in our paper, which is shown in Figure 2. The software environment for simulation is TensorFlow 0.12.1 with Python 3.6 in Jupyter Notebook 5.6.0.

FIGURE 2: Dueling double deep Q-network structure.

For the energy harvesters’ channel, to show an example of the performance achievable by the proposed algorithm, we consider the Rician channel fading model [Reference Cavers25]. We suppose within each time slot t, the channel is invariant and varies in different time slots [Reference Wu and Yang26]. At the end of each time slot, the energy harvester feedbacks the current energy level back to the transmitter. For the Rician fading channel model, the total gain of the signal is denoted as $g=g^s+g^d$ , where g^s is the invariant LOS component and g^d denotes a zero-mean Gaussian diffuse component. The channel between the transmit antenna m and the energy harvester i can be denoted as $g_{im}=g_{im}^s+g_{im}^d$ . The magnitude of the faded envelope can be modeled using the Rice factor K^r such that $K_{im}^r={\rho }_{im}^2/2{\sigma }_{im}^2$ , where ${\rho }_{im}^2$ denotes the average power of the main LOS component between the transmit antenna m and energy harvester i and ${\sigma }_{im}^2$ denotes the variance of the scatter component. We can derive the magnitude of the main LOS component as $\left(g_{im}^s\right)=\sqrt{2K_{im}^r}{\sigma }_{im}$ since $1/2E\left({\left(\left(g_{im}^d\right)\right)}^2\right)={\sigma }_{im}^2$ . The mean and the variance of g_im are denoted as ${\mu }_{g_{im}}=g_{im}^s$ and ${\sigma }_{g_{im}}^2={\sigma }_{im}^2$ , respectively. In polar coordinates, $g_{im}=r_{im}{e}^{j{\theta }_{im}}$ .

First, we explore the optimal deep Q-network structure under fading channels. We suppose the number of antennas is M = 3 and the number of energy harvesters is K = 2. The channel between each antenna of the transmitter and each harvester is individually Rician distributed. The action set A contains 13 actions satisfying the information rate requirement: ${\left(\mathrm{2,2,8}\right)}^T$ , ${\left(\mathrm{2,4,6}\right)}^T$ , ${\left(\mathrm{2,6,4}\right)}^T$ , ${\left(\mathrm{2,8,2}\right)}^T$ , ${\left(\mathrm{4,2,6}\right)}^T$ , ${\left(\mathrm{4,4,4}\right)}^T$ , ${\left(\mathrm{4,6,2}\right)}^T$ , ${\left(\mathrm{4,8,0}\right)}^T$ , ${\left(\mathrm{6,2,4}\right)}^T$ , ${\left(\mathrm{6,4,2}\right)}^T$ , ${\left(\mathrm{6,6,0}\right)}^T$ , ${\left(\mathrm{8,2,2}\right)}^T$ , and ${\left(\mathrm{8,4,0}\right)}^T$ .

The LOS amplitude components of all channel links are defined as $r_{im}=0.5$ , with i = 1, 2 and m = 1, 2, 3. The LOS phase components of all channel links are defined as ${\theta }_{11}=\pi /4$ , ${\theta }_{12}=\pi /2$ , ${\theta }_{13}=-\pi /4$ , ${\theta }_{21}=-\pi /2$ , ${\theta }_{22}=0$ , and ${\theta }_{23}=3\pi /4$ . The standard deviation of the g_im amplitude and phase is denoted as ${\sigma }_{im}$ and $1/\sqrt{2K_{im}^r}$ , respectively. We suppose ${\sigma }_{im}=0.05,\forall i,m$ . Hence, $1/\sqrt{2K_{im}^r}={\left(r_{im}/{\sigma }_{im}\right)}^{-1}=0.1,\forall i,m$ .

Using the fading channel model above, in Figure 3, we show how the structure of the neural network together with the learning rate can affect the performance of the DQN, for a fixed number of training episodes (i.e., 40000). The performance of DQN is measured by the average number of time slots required to fully charge two harvesters. The average time-consumption is obtained over 1000 testing data. Figure 3 shows that if the deep Q-network has multiple hidden layers, a smaller learning rate is necessary to achieve better performance. When the learning rate is 0.1, the DQN with 4 hidden layers performs worse than a neural network with 2 or 3 hidden layers. On the other side, when the learning rate decreases, we can see that the neural network with 4 hidden layers and a learning rate of 0.00005 achieves the best overall performance. We do not see a monotonic decrease in the average number of time slots due to the stochastic nature of the channel that causes some fluctuations in the DQN optimization. After an initial improvement, decreasing the learning rate results in a slight increase in the average number of charging steps for all three neural network structures. This is due to the fixed number of training episodes. As a result, for all the simulations presented in this section, we consider a DQN algorithm using a 4 hidden layer deep neural network, with 100 nodes in each layer and a learning rate of 0.00005.

FIGURE 3: Deep Q-network performance on different learning rates and number of hidden layers for the neural network.

In Figure 4, we can observe that the size of the experience pool also affects the performance of DQN (40000 training episodes). To eliminate the correlation between the training data, we select part of the experience pool for training. In our simulation, this parameter, called mini-batch, is set to 10. Larger experience pool contains more training data; hence, selecting the mini-batch from it for training can eliminate the correlation between the training data. However, we need to balance the size of the experience pool and the target_net weight replacement interval. If the experience pool is large but the replacement iteration interval is small, even if we address the correlation problem between the training data, the neural network does not have enough training episodes to reduce the training error before the weight of the target_net is replaced. From Figure 4, we can observe that a large number of replacement iteration intervals may not be the best choice too. Therefore, we determine that, for our problem, DQN achieves the best performance when the size of the experience pool and the neural network replacement iteration interval are 60000 and 1000, respectively.

FIGURE 4: Deep Q-network performance for different values of neural network replacement iteration interval and experience pool.

Figure 5 shows the impact of the reward function (see Section 3.1) on the DQN performance. In this figure, we consider the following reward functions: Reward₁: $w\left(s,a,s^{\prime }\right)=0$ if $s^{\prime }=s_G$ and $w\left(s,a,s^{\prime }\right)=-\lambda \left({\text{KB}}_{\max }-{\sum }_{i=1}^K\min \left(s_i^{\prime },B_{\max }\right)\right)$ otherwise; Reward₂: $w\left(s,a,s^{\prime }\right)=10$ if $s^{\prime }=s_G$ and $w\left(s,a,s^{\prime }\right)=-1$ otherwise; Reward₃: $w\left(s,a,s^{\prime }\right)=1$ if $s^{\prime }=s_G$ and $w\left(s,a,s^{\prime }\right)=-1$ otherwise. Here, K = 2 and λ = 0.25. All three reward functions are designed to minimize the number of time slots required to fully charge all the harvesters. However, from Figure 5, we can observe that the best performance can be obtained using Reward₁. In this case, the energy level accumulated by each harvester increases uniformly, which results in the DQN to converge faster to the optimal policy. Both Reward₂ and Reward₃, instead, do not penalize states that unevenly charge the harvesters and therefore require more iterations to converge to the optimal solution (not shown in the figure) due to the large number of system states to explore. Therefore, in the following simulations, we use the reward function Reward₁ in both Figures 5 and 6, we average 40000 training steps every 100 steps in order to better show the convergence of the algorithm.

FIGURE 5: The deep Q-network performance for different reward functions.

FIGURE 6: The deep Q-network performance for different energy buffer size B _max.

Figure 6 shows that when each energy harvester in the system is equipped with a larger energy buffer, the number of system states increases, and therefore, DQN requires more training period to converge to the steady transmit strategy for each system state. We can observe that when B _max = 1.6 mJ, the system only needs less than 5000 training episodes to converge to the optimal strategy, and when B _max = 3.2 mJ, the system needs around 12000 training episodes to converge to the optimal policy. However, for B _max = 4.8 mJ, the system needs as many as 20000 training episodes to converge to the optimal strategy.

In the following simulations, we explore the impact of the channel model on optimization problem P ₁. For the Rician fading channel model, we consider $K_{im}^r\ge 10$ and to be the same for all i,m. In this way, we can approximate the Rician distribution as a Gaussian distribution. We fix $r_{im}=0.5,\forall i,m$ , but allow the standard deviation of both the amplitude and the phase of the channel to change to evaluate the performance on the system under different channel conditions. Since $r_{im}=0.5$ and $\sqrt{2K_{im}^r}=r_{im}/{\sigma }_{im}$ , $1/\sqrt{2K_{im}^r}=2{\sigma }_{im}$ . We define ${\sigma }_{im}\le 0.1$ to guarantee $K_{im}^r\ge 10$ .

In Figure 7, we express the standard deviation ${\sigma }_{\text{amp}}={\sigma }_{im},\forall i,m$ of the phase and amplitude of the channel, and we compare the performance attained by the optimal policy with the performance of different other algorithms. The multiarmed bandit (MAB) algorithm is also implemented to compare with the DQN. In MAB, each bandit arm represents a particular transmission pattern. The upper confidence bound (UCB) algorithm [Reference Slivkins27] is implemented to maximize the reward $w\left(s,a,s^{\prime }\right)$ and determine the optimal action. Once the action is selected from the action space A, it will be used for transmission continuously. The myopic algorithm is another machine learning algorithm that can be compared with DQN. Myopic solution has the same structure as the DQN; however, the reward discount is defined as γ = 0. As a result, the optimal strategy is determined only according to the current observation instead of considering the future consequence. Myopic solution has been widely used to solve the complex optimization in wireless communication problem and achieve good system performance [Reference Wang, Liu, Gomes and Krishnamachari28]. Besides two machine learning algorithms, another two heuristic algorithms are also used for system performance comparison. For even power allocation, the transmit power P is evenly allocated on parallel channel for transmission. The random action selection is also applied for performance comparison. The random action selection has the worst performance while DQN performs best. Compared to the optimal existing algorithm multiarmed bandit algorithm, the DQN can consume 20% fewer time slots to complete charging. In some channel conditions, the myopic solution can achieve a similar performance as the DQN. However, the myopic solution cannot perform stably. For example, as the standard deviation of the channel amplitude is ${\sigma }_{\text{amp}}=0.025$ , DQN can outperform myopic solution by 45%. The instability can be explained as the myopic solution makes the decision only on the current system state and the current reward, which does not consider the future consequence. Hence, the training effects cannot be guaranteed. Overall, the DQN has superiority in both the charging time consumption and performing stability corresponding to different channel conditions.

FIGURE 7: The comparison of average time consumption between DQN and other algorithms (myopic solution, multiarmed bandit, even power allocation, and random action selection) in the Rician fading channel model. The number of transmit antennas is M = 3. The number of energy harvesters is N = 2.

To better explain the performance of the optimal policy, in Figure 8, we plot the action selected by DQN at a particular system state when ${\sigma }_{\text{amp}}=0.05$ . When ${\sigma }_{\text{amp}}=0.05$ , the optimal action selected by multiarmed bandit is the third action $a_3={\left(\mathrm{2,6,4}\right)}^T$ , which can finish charging both harvesters in around 60 time slots. Meanwhile, the optimal policy determined by DQN can finish charging in around 43 time slots. To this end, Figure 8 shows that the charging process can actually be divided into two parts: before harvester 1 accumulates 1.2 mJ energy and harvester 2 accumulates 0.8 mJ energy, mostly action 4 $a_4={\left(\mathrm{2,8,2}\right)}^T$ is selected. After that, mostly action 1 $a_1={\left(\mathrm{2,2,8}\right)}^T$ is selected. As defined above, both the amplitude and the phase of the channel are Gaussian distributed with zero standard deviation, ${\hat{g}}_1^0={\left(0.05,0.59,0.11\right)}^T$ and ${\hat{g}}_2^0={\left(0.04,0.19,0.51\right)}^T$ . So when both the amplitude and the phase of the channel change, the simplified channel state information will be distributed around ${\hat{g}}_1^0$ and ${\hat{g}}_2^0$ . As a result, it can be shown that a policy that selects either action 1 or action 4 with different probabilities can have better performance than the policy that only selects action 3. Henceforth, the DQN can consume 40% fewer time slots to fully charge two energy harvesters.

FIGURE 8: The action selection process of two harvesters scenario when ${\sigma }_{\text{amp}}=0.05$ .

In Figure 9, the performance of the DQN is compared with the other four algorithms by varying the number of energy harvesters in the system. In general, as the number of energy harvesters increases, all four algorithms consume more time slots to complete the wireless charging process. Compared to the random action selection, DQN can consume at least 58% less time slots to complete the charging. The performance of the multiarmed bandit and the even power allocation is very similar, which can be explained as the optimal action determined by the multiarmed bandit algorithm is close to the even power allocation strategy. Compared with two fixed action selection strategies (multiarmed bandit and even power allocation), DQN can reduce the time consumption by up to 72% (when the number of energy harvesters is N = 3). The myopic solution is still not the optimal strategy. From the figure, we can observe that the myopic solution outperforms two fixed action selection algorithms. Even though in some conditions (N = 6), the performance difference between DQN and myopic solution is very small, the myopic solution consumes more than 15% of the time slot than DQN in average. Overall, the DQN is the optimal algorithm which consumes fewest time slots to fully charge all the energy harvesters regardless of the number of energy harvesters.

FIGURE 9: The comparison of average time consumption between DQN and other algorithms (myopic solution, multiarmed bandit, even power allocation, and random action selection) when the number of energy harvesters is N = 2, 3, 4, 5, 6. The number of transmit antennas is M = 3. The standard deviation of the channel amplitude is ${\sigma }_{\text{amp}}=0.05$ .

In Figure 10, the number of transmit antennas is increased from M = 3 to M = 4. The number of energy harvesters varies from N = 2 to N = 6. Though the number of antennas increases, the channel conditions between the transmitter and the energy harvesters become more complicated; DQN still outperforms all the other four algorithms. Compared with myopic solution, multiarmed bandit, even action selection, and random action selection, DQN can consume up to 27%, 54%, 55%, and 76% fewer time slots to fulfill the wireless charging, respectively. As the number of energy harvesters increases, the superiority of the DQN becomes more obvious compared to two fixed action selection algorithms, which can be explained as it is more inefficient to select one fixed action to deal with a more complicated varying channel environment. Even though in some conditions, the performance of the myopic solution and DQN is similar, the myopic solution is not stable in dealing with different energy harvesters conditions. The results from both Figures 9 and 10 demonstrate the superiority of the DQN in optimizing the time consumption for wireless power transfer.

FIGURE 10: The comparison of average time consumption between DQN and other algorithms (myopic solution, multiarmed bandit, even power allocation, and random action selection) when the number of energy harvesters is N = 2, 3, 4, 5, 6. The number of transmit antennas is M = 4. The standard deviation of the channel amplitude is ${\sigma }_{\text{amp}}=0.05$ .