2.1. Planning of landing position control

The problem of ball landing position control can be considered an inverse operation problem. As illustrated in Figure 8, the ball velocity after impact, angular velocity, racket orientation during impact, and racket velocity are all unknown in this problem. These unknowns can only be determined using the incoming ball velocity, angular velocity, desired landing point, and desired ball flight duration. The physical models, consisting of the aerodynamic and bouncing models as referenced by Nakashima et al. [Reference Nakashima, Ogawa, Liu and Hayakawa15], are essential to this approach. The ADM accounts for drag and lift forces, which are crucial for understanding the ball’s flight path, especially under varying spin conditions. The bouncing model encompasses the dynamics of the ball’s interactions with both the table and racket, providing a detailed understanding of the ball’s behavior upon impact. Because a table tennis ball movement model can be viewed as a quadratic nonlinear formula, using these physical models to perform inverse operations for controlling the landing point of different spinning balls is, in reality, both complex and difficult.

Figure 8. Inverse operations using physical models to control the landing point of a returned ball.

The machine learning methods proposed in previous studies are not applicable to the problem of landing position control for various spinning balls because they were typically designed for specific conditions like topspin or backspin and relied on assumptions about the ball’s rotational state. These assumptions limited their ability to accurately control the landing position under varied spin conditions. In this study, we modified these methods to handle the complexities of different spins. The problem of return ball landing position control is associated with four sequential subtasks, as shown in Figure 9. Our approach utilizes a CNN model that divides the task into these subtasks, allowing for more precise and adaptable control over the ball’s trajectory and landing position, regardless of the spin type. This modification enhances the generalizability and effectiveness of the control system, providing a more comprehensive solution than previous methods. Additionally, we integrated a physical model of the table tennis robot, considering factors such as aerodynamics and the interactions between the ball and the racket. This integration enables the system to account for real-world factors like air resistance and spin effects, which were not adequately addressed in previous methods. This combination of machine learning and physical modeling offers a more robust solution for predicting and controlling the landing position of various spinning balls.

Figure 9. Planning of ball landing position control subtasks.

Subtask 1: In this subtask, the flight duration of a ball from the moment of impact to the desired landing point is determined. In studies regarding the hitting control of table tennis robots, the racket velocity and racket surface orientation are the parameters for implementing control. However, the true velocity at the terminal end of the arm used in this study cannot be controlled; hence, the velocity and acceleration percentages of the robotic arm are set as constant to ensure that the racket’s hitting velocity approximates constant values and the ball landing point is controlled by the orientation of the racket. Assume that at constant racket velocity and incoming ball velocity, only the time a ball takes to reach a certain flight distance after impact is adjusted under the condition of racket orientation, then flight duration is primarily affected by the ball velocity after impact and flight distance. However, the ball velocity after impact is unknown, and other known conditions must be employed to determine the flight duration. At constant racket velocity, the ball velocity after impact is proportional to the velocity of the incoming ball, and in the absence of a high-flying ball, flight duration and flight distance can be assumed to be proportional. In [Reference Li, Chen, Huang, Chen, Yu and Duo7], the states of topspin and backspin balls after impact were determined, and the flight trajectory of a topspin ball was found to be higher, whereas the backspin ball began falling after being hit by the racket. This reveals that the rotational state of the incoming ball considerably influences the state of a ball after impact. Because the rotational velocity of a spherical body cannot be determined for this study’s robotic table tennis system and because the ball’s acceleration varies during its process of rotation, this study used acceleration to represent the rotational state of the ball. According to the aforementioned factors that influence flight duration, the Subtask 1 model was defined to comprise the following inputs: distance between the hitting point and desired landing point, incoming ball velocity, and incoming ball acceleration. The model’s output is flight duration.

Subtask 2: In this subtask, the ball’s flight trajectory from the desired landing point to the hitting point is reverse-predicted from among the trajectory points segmented by the same time interval. The obtained point is used to increase the accuracy of the ball velocity estimate after impact. Unlike trajectory prediction of an incoming ball, forward trajectory prediction is not applicable in this case because the ball velocity at the initial position (hitting point) is unknown and the end point of the trajectory (desired landing point) is self-defined (i.e., a known term). In addition, the trajectory of a moving ball after impact is related to flight distance, flight duration, and the rotational state of the ball, whereas the displacement from the trajectory at each moment is related to that in the previous moment. Hence, in Subtask 2, the displacement from hitting point to desired landing point, flight duration, and acceleration of incoming ball are the initial inputs for estimating the displacement in the previous moment. Displacement in the next moment is predicted using the displacement in the previous moment, flight duration, and incoming ball acceleration, subsequently obtaining, from among the trajectory points that are segmented by the same time interval, the trajectory point that is closest to the hitting point.

Subtask 3: In this subtask, the velocity of the outgoing ball after impact, $V_{out}$ , is estimated at the last point of flight trajectory, which was predicted in Subtask 2. Because this position is closer to the hitting point, the state of the ball at this position is similar to that at the hitting point. The velocity of the ball after impact is more accurately estimated than if using the total displacement and total flight duration.

Subtask 4: According to the physical model employed in ref. [Reference Nakashima, Ogawa, Liu and Hayakawa15] for illustrating the impact between a table tennis ball and racket, the velocity of an outgoing ball after impact is related to the ball velocity before impact, ball rotational velocity, racket orientation, and racket velocity. In the present study, the racket velocity is constant, and the racket surface orientation is used to control the landing position of the returned ball. Hence, ball velocity before and after impact and incoming ball acceleration are the input parameters for estimating racket orientation.

2.2. Cascade forward and recurrent neural networks

On the basis of the subtasks of ball landing position control planned in the previous section, a CNN based on a forward and recurrent neural network (RNN) was designed (Figure 10) to achieve control of ball landing position when using a robotic table tennis system. In Parts 1, 3, and 4 of the model, control is implemented using the forward NN, whereas in Part 2, the RNN is employed to exercise control. Compared with other regression algorithms, NNs do not require input of hypothesized relationships between input and output variables such as index, polynomial, and trigonometric functions. By adjusting the hidden layer and number of neurons, the network’s ability to express models can be improved to learn the relationships of complex input–output maps. Therefore, Parts 1, 3, and 4 of the model in this study employ forward NNs as the model architecture. The forward NN handles the non-linearities associated with the ball’s motion, such as drag and spin effects, by providing initial estimations of the ball’s trajectory. Regarding Part 2, because a ball’s displacement at every moment is related to the displacement at the next moment, a model based on the RNN, which contains state memory units, is used to predict trajectory points by referring to information on the displacement at the last moment and the history stored in the state memory unit. The RNNs refine these predictions by capturing temporal dependencies, essential for precise motion planning and control, especially in dynamic scenarios like table tennis. In the network architecture designed in this study, Part 2 is the reverse trajectory prediction model, which is only used to predict the $Z$ -direction trajectory, and Part 3 is the postimpact ball velocity estimation model, which is only employed to estimate the $Z$ -direction velocity of the ball because the movement trajectory of a topspin or backspin ball after impact is similar to that of an isokinetic movement.

Figure 10. Proposed cascade neural network.

In the flight duration estimation model (Part 1), the inputs are the displacement of the incoming ball from the hitting point to the desired landing point, denoted $\Delta x_0$ , $\Delta y_0$ , $\Delta z_0$ ; the acceleration of the incoming ball $V_{in}^{ball}=(V_{ix},V_{iy},V_{iz})$ ; and the $Z$ -direction acceleration of the incoming ball $a_{iz}^{ball}$ . The output is the flight duration $t_0$ from hitting point to desired landing point. In this study, topspin, backspin, and no-spin balls were considered. Topspin and backspin influence only the $Z$ -direction acceleration of the ball; therefore, only the acceleration in this direction was considered.

In the reverse trajectory prediction model (Part 2), the flight trajectory of the ball from hitting point to landing point was divided into 10 segments with equal time intervals. As illustrated in Figure 11, $P_0$ is the ball landing point and $P_{10}$ is the ball-hitting point. Based on Figure 11, the definition of the RNN architecture for reverse trajectory prediction is displayed in Figure 12, where the time step $T=0,1, \cdots, 9$ ; $S_T$ is the state memory unit of the RNN; $\Delta z_T$ is the $Z$ -direction displacement of the ball from hitting point $P_{10}$ to $P_T$ ; $t_T$ is the flight duration of the ball from hitting point $P_{10}$ to $P_T$ in Figure 11; and $a_{iz}^{ball}$ is the $Z$ -direction acceleration of the incoming ball once it has reached hitting point $P_{10}$ . In the RNN architecture, when the time step = $T$ , the network receives $S_{T-1}$ , $\Delta z_{T-1}$ , $t_{T-1}$ , $a_{iz}^{ball}$ as the inputs and outputs $\Delta z_T$ .

Figure 11. Outgoing ball flight trajectory divided into 10 segments with equal time intervals.

Figure 12. Architecture of the recurrent neural network for reverse trajectory prediction.

The ball velocity estimation model (Part 3) receives the $Z$ -direction displacement $\Delta z_9$ and flight duration $t_9$ of the ball from hitting point $P_{10}$ to $P_9$ in Figure 11 as its input and outputs the $Z$ -direction velocity of the ball at hitting point $P_{10}$ after it has been hit. Because the movement of the ball in the $X$ and $Y$ directions is similar to an isokinetic movement, the $X$ - and $Y$ -direction velocity of the ball at hitting point $P_{10}$ after it has been hit can be calculated as follows:

(1) \begin{equation} V_{ox}=\frac{\Delta x_0}{t_0}, V_{oy}=\frac{\Delta y_0}{t_0}, \end{equation}

Finally, the change in velocity and racket orientation mapping model (Part 4) receives the velocity of the ball at hitting point $P_{10}$ before ( $V_{in}^{ball}$ ) and after ( $V_{out}^{ball}$ ) it hits the racket and the Z-direction acceleration of incoming ball ( $a_{iz}^{ball}$ ) as the inputs and outputs the corresponding angles of racket orientation $\alpha$ and $\beta$ .

2.3. Landing position control based on LWR

Unlike the systems used in previous studies, the robotic table tennis system proposed in this study does not use racket velocity as the parameter for controlling the ball’s landing position. Hence, the landing position control methods developed by past studies cannot be compared with the method proposed in this study. For this reason, the LWR-based landing position control method in ref. [Reference Huang, Xu, Tan and Su5] was modified to act as a comparison method for the present study’s system. Figure 13 illustrates the architecture of the modified LWR-based landing position control method. Part 1 of the method is the flight duration estimation model, which is identical to Part 1 of the multilayer NN architecture proposed by this study. Part 2 is the postimpact ball velocity estimation model. In ref. [Reference Huang, Xu, Tan and Su5], only the displacement of the ball from the hitting point to the desired landing point was used to estimate the velocity of the ball at the hitting point. However, the same level of displacement may occur at different ball velocities. Hence, the ball velocity estimation model was modified to receive the Z-direction displacement $\Delta z_0$ and flight duration $t_9$ of the ball as inputs and to output the $Z$ -direction velocity $V_{oz}$ of the ball after the ball has been hit. Subsequently, the ball velocities in the $X$ and $Y$ directions can be calculated using (1). Part 3 is the change in velocity and racket orientation mapping model, which receives the change in ball velocity $\Delta V_{ball}$ at the hitting point and outputs the corresponding angles of racket orientation $\alpha$ and $\beta$ .

Figure 13. Modified locally weighted regression-based landing position control method.

3.1. Simulation

The ball landing position control results of a robotic table tennis system are affected by not only training errors in the landing position control model itself but also errors in the 3D calculation of imaging systems, hitting point prediction, estimation of incoming ball velocity and rotation, and hitting time point control. These factors cause greater landing position errors. Therefore, this study first used simulation to compare the performance of the proposed CNN and LWR models in landing position control when only errors in landing position control model exist. The physical model of the table tennis ball movement process in ref. [Reference Nakashima, Ogawa, Liu and Hayakawa15] was employed to simulate the entire process of a ball flying, bouncing on the table, and being hit by a racket.

3.2. Generating training and test data

Incoming balls with different movement directions, velocities, and rotational states were generated by adjusting the initial position, serving angle, velocity, and rotational velocity of the incoming ball. The ball landing position control only considered topspin and backspin balls; hence, the initial rotational state of the incoming ball was configured to include topspin and backspin balls. To ensure that the landing points of the returned ball in the training data were evenly distributed among the nine grid squares on the server’s side (Figure 14), 900 sets of data (consisting of the trajectories of the incoming ball and return ball) were obtained for model training. Among the 900 sets of data, 100 sets were data on the return ball’s landing points in each grid square; 720 sets were training data, and 180 sets were test data. Next, another 900 sets of data were obtained, 100 of which were data on the landing points in each grid square. These datasets were used to evaluate the landing position control method and verify that landing position control was achieved. Half the training data and test data involved topspin balls, and the other half, backspin balls.

Table II. Structural settings for cascade neural network in simulation experiment.

Figure 14. Planning of the landing area of a return ball.

3.3. Model evaluation

The previous section described the method by which the training data and test data in the landing position control model were obtained. This section presents an evaluation of the CNN proposed in this study and the LWR-based landing position control mode. The purpose was to evaluate each part of the model that causes errors in the landing position to accumulate. Table II shows the structural settings for each part of the proposed CNN shown in Figure 10 in the simulation experiment.

To assess the accumulation of errors in each part of the CNN, 4 bits were used to express whether each part was activated. Take 0001 as an example; 0001 means that Parts 1, 2, and 3 are unactivated, whereas Part 4 is activated. Similarly, the LWR-based landing position control model was planned using the same approach. Because the LWR-based model has only three parts, 3 bits are used to indicate whether each part is activated.

After the input data required by each model were incorporated into the model, the racket orientation parameters $\alpha$ and $\beta$ could be obtained, which were then input to the physical model to obtain the return ball’s landing point. Subsequently, the landing distance error was calculated on the basis of the desired and actual landing point. Figure 15 shows the errors in landing distance for the training data based on the four types of models planned by the CNN. Models 0001 and 0011 had extremely similar average errors and SDs, suggesting that adding a model to Part 3 did not significantly influence the errors. Next, the errors of Models 0011 and 0111 were compared, and if Part 2 was added, the average error increased substantially from 33.1 to 40.7 mm whereas the SD increased from 51.5 to 60.2 mm, both of which are a considerable increase compared with the two aforementioned comparisons. This difference was obtained because the error in flight duration t0 estimated by the Part 1 model simultaneously influenced the input parameter $t_9$ of the Part 3 model and the input parameter $V_{out}^{ball}$ of the Part 4 model. Unlike the trend prior to the addition of Part 1, for every part added to the model, the errors accumulated gradually. In Model 0111, the errors were higher because Part 2 had been added, causing the output errors of Part 3 to be greater, which indirectly increased the output errors of Part 4. Once Part 1 had been added, the errors accumulated gradually, and the errors input to the Part 3 and Part 4 models increased, thereby further raising the errors in the model’s final output, which was the orientation of the racket surface. According to Figure 16, the LWR-based landing position control produced greater errors in landing distance than the CNN for the training data based on three models. Figure 16 shows that the average errors of the three models were all greater than 260 mm (SD = 190 mm).

Figure 15. Landing distance errors of four cascade neural network-based models for training data.

Figure 16. Landing distance errors of three locally weighted regression-based models for training data.

Figure 17 illustrates the errors in the landing distance for test data when using the four CNN models. The error trend was identical to that for the training data. Models 0001 and 0011 had similar errors. Model 0111 generated considerably greater errors, and the errors of Model 1111 were dramatically higher. Figure 18 shows the errors in the landing distance for test data when using the three LWR-based models. The error trend was again identical to that for the training data. The average errors and SDs of the three models were similar; however, the average error and SD for the test data were 20 and 40 mm greater than those for the training data, respectively. This result reveals that the LWR-based landing position control model performed poorly for the test data.

Figure 17. Landing distance errors of four cascade neural network-based models for test data.

Figure 18. Landing distance errors of three locally weighted regression-based models for test data.

3.4. Landing position control

To compare the landing position control performance of the CNN proposed in this study and the LWR-based landing position control model, the desired landing points are displayed in Figure 19 were selected. The server’s side was divided into three equal parts in the $X$ and $Y$ directions, forming nine grid squares of equal size. The center point of each square was designated as the desired landing point. In this experiment, 100 incoming ball trajectories (50 topspin and 50 backspin balls) were employed for each desired landing point to examine the distribution of return ball landing points generated using the two model types. Figure 20 presents the distributions of ball landing points for desired landing points 1–9, respectively. In the diagrams, red circles represent the desired landing point and blue circles represent the actual landing point of a returned ball.

Figure 19. Planning of desired landing points.

Figure 20. Distribution of landing points for desired landing point 9.

Examination of Figure 20 reveals that when the CNN was used for control, the landing points were relatively more centralized and closer to the desired landing points than when the LWR-based model was used. In this experiment, landing position control was more effective with the proposed CNN than with the LWR-based model.

3.5. Experiment

The objective of this experiment was to compare the landing position control performance of the proposed CNN and LWR-based control models in a robotic table tennis system. Figure 21 illustrates the experimental environment and placement of relevant equipment in the robotic table tennis system. The visible range of the visual system in the robotic table tennis system could not include the whole of the server’s side; therefore, the blue area (shown in Figure 21) was defined as the controllable range of the return ball landing point according to the visible range of the visual system and distribution of landing points of a ball returned by the robot when using a randomly oriented racket. This blue area was divided into six grid squares of equal size. The center of each grid was the desired landing point in this experiment.

Figure 21. Experimental environment and controllable area of return ball landing point and planning of desired landing points.

Given the 3D calculation errors inherent in a visual system, for this experiment, a 40 $\times$ 40 cm² block (where the ball should land and bounce on the table) was designed as shown in Figure 22(a). The block consisted of four layers. The topmost layer was a 2-cm-wide white frame that facilitated positioning of the block when testing different desired landing points; the second layer was carbon paper so that when a ball hit the block, it would leave a print on the piece of white paper at the bottom [Figure 22(b)]; the third layer was a frame of the same size as the topmost layer to avoid leaving prints on the white paper at the bottom when the ball landed outside the block; and the bottommost layer was the aforementioned piece of white paper.

Figure 22. The block (where the ball should land and bounce on the table) used in this experiment. (a) Placement of the block during the experiment. (b) Prints left on white paper when the block was hit by the ball.

In this experiment, the errors in the returned ball landing point were affected by numerous factors, including the errors in the CNN, control of hitting time and hitting point, incoming ball velocity, and acceleration prediction. Therefore, blocks of three sizes were planned, as illustrated in Figure 23. The success rates for these three blocks were averaged to represent the success rate of a desired landing point. In this experiment, the trajectories of topspin, backspin, and no-spin balls returned by a randomly oriented racket were collected for model training.

Figure 23. Blocks of differing sizes.

Once the robotic table tennis system completed estimation of the hitting point and incoming ball velocity and acceleration, the landing position control model was employed to calculate the racket orientation, and finally, the robotic arm was ordered to move and return the ball. To ensure that the robotic arm had sufficient time to complete a hitting movement once the calculation had been completed, the calculation time of the model had to be minimized as much as possible. Figure 24 shows the average amount of time spent by the CNN and LWR-based models to make 100 calculations. Because both models took less than 1 ms, the time required by robotic arm to execute the remaining movement was not adversely affected. The average execution time of the CNN and LWR-based models was 0.012 and 0.458 ms, respectively. Compared with the LWR-based model, the CNN model took 97.38% less time. Figure 25 shows the number of parameters associated with the CNN and LWR-based models. The LWR-based model had 3766 parameters, whereas the CNN model had 424 parameters, 88.74% fewer. When predicting every new input, the LWR-based model must recalculate the model coefficients by using the training data; therefore, a large volume of training data must be stored in the matrix, taking up a considerable amount of memory.

Figure 24. Comparison of execution time of the cascade neural network and locally weighted regression-based models.

Table III. Average total success rate at various desired landing points in blocks of varying sizes.

Figure 25. Comparison of the number of parameters associated with the cascade neural network and locally weighted regression-based models.

Table III summarizes the average success rates at the six desired landing points when the CNN and LWR-based models were used. The results indicate that the success rate at desired landing points 1 and 2 was considerably higher for the CNN model than for the LWR-based model. The success rates at desired landing points 3, 4, and 5 differed between methods by 3%, and this difference can be considered to be the result of measurement errors. The two methods had similar control effects at these three desired landing points. At desired landing point 6, the CNN model was somewhat superior to the LWR-based model. Overall, the CNN model achieved favorable landing position control. Figure 26 shows the sequential stages of the table tennis ball interaction during experimental testing. The first stage captures the ball coming from the launcher machine, the second stage shows the ball being hit by the robotic arm’s racket, and the final stage illustrates the ball landing on the block used for measuring landing positions.

Figure 26. Sequential stages of table tennis ball interaction during experimental testing.

Next, the success rates for topspin, backspin, and no-spin balls were calculated. Tables IV and V show the success rates of controlling the landing positions for the three spin types using the CNN and LWR models, respectively. Based on the results, CNN model maintains a relatively stable control effect on landing points, with success rates not less than 50% for all six points.

Table IV. Average success rates of cascade neural networks model for different spin types.

Table V. Average success rates of locally weighted regression model for different spin types.

Additionally, comparing the success rates across 18 sets of data (covering three spin types and six landing points), the success rates of topspin balls at desired landing points 3, 4, and 5 and no-spin balls at desired landing points 5 and 6 were lower than those obtained using the LWR-based model. Therefore, the CNN overall more effectively controlled the landing of differently spinning balls.