2.1. Stroke division

Stroke division is one of the difficulties in the study of Chinese characters because it is tough to learn the connections and intersections in Chinese characters. When the endpoints of two strokes are at the same point, as shown in Fig. 1, the point is defined as a connection. Improper recognising this point will tend to misidentify multiple strokes as the same stroke, as shown in Fig. 2. The intersection is the point formed when two strokes intersect, and improper handling of this point will result in incorrect recognition of the strokes, as shown in Fig. 3.

Figure 1. The connection and intersection of Chinese character “zhong”.

Figure 2. (a) The correct stroke division for the Chinese character “zhong”. (b) The incorrect stroke division after the improper handling of the connections.

Figure 3. (a) The correct stroke division for the Chinese character “shi”. (b) and (c) The incorrect stroke division after the improper handling of the intersection.

The Chinese character strokes are shown in Fig. 4. Analysing the strokes of Chinese characters, we can see that each stroke can be divided into one or more approximately straight curves. These curves can be summarised as dot, horizontal, vertical, left-falling and right-falling as shown in Fig. 5. These curves are named stroke components. Thus, each Chinese character stroke can be formed by combining one or more Chinese character stroke components. For example, as shown in Fig. 6, a Chinese character stroke “dot” can be seen as a combination of a Chinese character stroke component “dot”. A Chinese character stroke “vertical-turning” can be decomposed as a combination of a Chinese character stroke component “vertical” and a Chinese character stroke component “horizontal”. For another example, a Chinese character “horizontal-break-hook” can be seen as a combination of a “horizontal”, a “vertical” and a “dot”.

Figure 4. Chinese character strokes.

Figure 5. Chinese character stroke components.

Figure 6. Chinese character stroke can be formed by combining one or more Chinese character stroke components.

As the Chinese character stroke components all extend in the same direction with no turning points, the Chinese character stroke components can be extracted when the direction of extension is known. At the same time, the determination of the direction of extension also deals with the issue of connections and intersections. A turning place connects two stroke components, and thus the turning identification is important in the basic component extraction. The intersection occurs when the strokes have been extended to different directions. When the different stroke components’ writing directions are determined, the stroke component extensions can cross the intersection in the specified direction to prevent misidentification at the intersection. Therefore, the stroke extraction algorithm has two parts: stroke components extraction and stroke components connection.

2.1.1. Stroke components extraction

Firstly, the handwritten Chinese character images were binarised and refined. The algorithm for stroke components extraction is built on the basis of a refined image of a white Chinese character on a black background. The image is then traversed to find the white pixel, which is recorded as the starting point $P_{sta}$ and treated as $P_{0}$ . As shown in Fig. 7, for a set of eight neighbourhoods with $P_{0}$ as the centroid, we set $P_{0}$ as the origin and the length of the pixel as the unit length. The pixel points in this eight-neighbourhood are therefore named by the coordinates in which they are located. Starting from the pixel point above $P_{0}$ , the pixels in a clockwise circle are each named $P_{n}=(x_n,y_n)(n=1,2,3,\ldots,8)$ . The process of extracting the strokes component is shown in Fig. 8. Define a direction vector $\textbf{D} =(\alpha,\beta )$ with an initial value of $(0,0)$ . $\textbf{D}$ describes the direction from the start pixel to the end pixel of the searched stroke component. The values of $\textbf{D}$ are normalised to obtain the coordinate points $P_{D}$ in the eight-neighborhood coordinate system.

(1) \begin{equation} {P_{D} =(x_{D},y_{D})} \end{equation}

(2) \begin{equation} {x_{D} = \frac{\alpha }{\sqrt{\alpha ^{2}+\beta ^{2} } } } \end{equation}

(3) \begin{equation} {y_{D} = \frac{\beta }{\sqrt{\alpha ^{2}+\beta ^{2} } } } \end{equation}

Figure 7. A set of eight neighbourhoods with $P_{0}$ as the centroid.

Figure 8. The process of extracting the strokes component.

The distance between $P_{D}$ and the coordinates of each pixel in the eight neighbourhoods is

(4) \begin{equation} {d_{n} =|P_{n}-P_{D}|} \end{equation}

Arrange $d_1, d_2\ldots, d_n$ according to their values (from small to large values). To ensure that the stroke component extends in the direction indicated by $\textbf{D}$ , perform a sequential search for the $P_{n}$ corresponding to the first 4 $d_{n}$ . If all the $P_{n}$ picked out are black, the search stops. If $P_{n}$ is a white pixel, extend the base stroke to $P_{n}$ and update direction vector

(5) \begin{equation} {\textbf{D}_{\textbf{new}} =(\alpha _{new},\beta _{new})} \end{equation}

(6) \begin{equation} {\alpha _{new} = \alpha +x_n} \end{equation}

(7) \begin{equation} {\beta _{new} = \beta +y_n} \end{equation}

Subsequently, $P_{n}$ is used as the next $P_{0}$ and $\textbf{D}_{\textbf{new}}$ as the next $\textbf{D}$ in order to continue extending the stroke component until the search stops.

However, the starting point $P_{sta}$ is not always the endpoint of the stroke component. So after the extension along direction $\textbf{D}$ is complete, the direction vector $\textbf{D}$ needs to be reversed

(8) \begin{equation} {\textbf{D}_{\textbf{R}} = -\textbf{D}} \end{equation}

It is then extended to the opposite direction from the starting point $P_{sta}$ , and finally, the two curves are joined to form a complete stroke component.

In order to prevent the same stroke component from being searched repeatedly, each pixel point is blacked out after it has been recorded as a point in the stroke component. However, this creates the interruption problem. As shown in Fig. 9, an intersection point is shared by multiple stroke components and when the intersection is blacked out, another stroke component passing through this intersection will be interrupted at the intersection. Therefore, we use two identical refined images $I_{1}$ and $I_{2}$ of white characters on a black background. Image $I_{1}$ is blacked out at the recorded pixels and is used to find the starting point $P_{sta}$ . This prevents repeated searching of the same basic stroke. Image $I_{2}$ is not blacked out at the recorded pixels and is used to extend the stroke component so that the stroke components are not interrupted at the intersection point.

2.1.2. Stroke components connection

When the end point of stroke component $SC_{1}$ and the start point of stroke component $SC_{2}$ are connected, $SC_{1}$ and $SC_{2}$ will make a stroke. Therefore, the starting and ending points of the stroke components need to be identified, i.e. the points between the connected stroke components. According to the conventional writing rules, the strokes are written from left to right or from top to bottom. The specific sorting algorithm is shown in Algorithm 1.

Algorithm 1: Sorting of points

Figure 9. (a) The darkening at the intersection divides the vertical into two parts. (b) Using two identical images $I_{1}$ and $I_{2}$ can split the stroke components correctly.

However, there is a special Chinese stroke called “hook” as shown in Fig. 10, which is written from bottom to top and from right to left. Similar to the stroke component “dot”, it is much shorter than the other basic strokes. It is also connected to the other stroke components. Thus, a stroke component is considered to be a ”hook” if it satisfies the following conditions:

Figure 10. The “hook” in Chinese character “horizontal-break-hook”.

1. 1. the end point is connected to the end points of the other stroke components;

2. 2. the start point is not connected to the end points of any of the other stroke components;

3. 3. the length is less than one-half the length of the average stroke component.

When combining strokes, the points of the “hook” need to be reverse ordered and then connected to other stroke components.

2.2. Force-position hybrid control

For a robotic writing system, the position control mode provides a limited performance, e.g. nonsmooth and nonsimilar to human’s writing characters. To write smoothly on the writing board, the contact force between the pen tip and the writing board needs to be controlled. Therefore, we use the force-position hybrid control. A modified DMP is used to control the position and posture of the robot’s end-effector. At the same time, the contact force between the pen tip and the writing board is controlled using admittance control. As a result, the robot can perform writing tasks.

2.2.1. Modified DMPs

Based on a spring damping model, the DMPs use a non-linear adjusting term to achieve the desired point attractor behaviour. It has the properties of a second-order dynamical system such as convergence at the target point, robustness to disturbances and generalisation in time and space. At the same time, the non-linear terms can be made to produce a continuous smooth trajectory. DMPs are classified as discrete DMPs and periodic DMPs. We use the discrete DMPs to model writing skills. The discrete DMPs model can be expressed as:

(9) \begin{equation} {\tau \dot{v}=\alpha \left (y_{g}-y\right )-\beta v+f(x)} \end{equation}

(10) \begin{equation} {\tau \dot{y}=v} \end{equation}

where $\tau$ is the temporal scaling constant, and changing it allows the trajectory to generalise in the time dimension. $\alpha$ and $\beta$ represent the damping factor and the spring constant, respectively, and $\alpha$ can generally be set to $\alpha =\beta ^{2}/4$ . $y_{g}$ is the target position. $y$ is the position of the demonstration. $v$ is the velocity of the demonstration. The forcing term $f(x)$ can be expressed as:

(11) \begin{equation} {f\left (x,y_{g}\right )=\frac{\sum _{i=1}^{N}{\psi _i\cdot \omega _i}}{\sum _{i=1}^{N}\psi _i}\cdot x\cdot (y_{g}-y_0)} \end{equation}

(12) \begin{equation} {\psi _i=\exp ({-}h_i(x-c_i)^2)} \end{equation}

(13) \begin{equation} {\tau \dot{x}=-\alpha _x\cdot x} \end{equation}

where $\omega _i$ is the weight of the Gaussian kernel function, $y_0$ is the starting point of the trajectory, $c_i$ and $h_i$ are the centre and width of the Gaussian kernel function, respectively, $N$ denotes the number of Gaussian kernel functions, and $\alpha _x$ is a positive gain coefficient.

However, as shown in Fig. 11, when the start and target points are rotated, the original DMP cannot rotate the generalised trajectory in its original shape and the trajectory is distorted. In order to apply the DMPs to this case where both the start and target points are rotated, we have improved the original DMPs inspired by [Reference Koutras and Doulgeri31]. The generalised space is obtained by rotating the original space. Once the rotation matrix between the two spaces is known, the corresponding rotation can be applied to the non-linear term. Thus, equation (9) becomes

(14) \begin{equation} {\tau \ddot{y}=\alpha _y(\beta _y\left (y_{g,1}-y\right )-\dot{y})+s\cdot \textbf{R} \cdot f } \end{equation}

(15) \begin{equation} {y_{g,1}=\textbf{R}\cdot y_{g,0}+\textbf{t}} \end{equation}

(16) \begin{equation} {y_{0,1}=\textbf{R}\cdot y_{0,0}+\textbf{t}} \end{equation}

(17) \begin{equation} {s=\frac{\left \| y_{g,1}-y_{0,1} \right \| }{\left \| y_{g,0}-y_{0,0} \right \| } } \end{equation}

where $y_{g,1}$ and $y_{g,0}$ are the target points in the generalised space and the original space, respectively. $y_{0,1}$ and $y_{0,0}$ are the starting points in the generalised space and the original space, respectively. $s$ is the scaling constant. $\textbf{R}$ is the rotation matrix and $\textbf{t}$ is the translation matrix. Throughout this article, $\textbf{R}$ and $\textbf{t}$ are calculated from visual information.

Figure 11. Comparison of original DMPs and modified DMPs. For linear and curvilinear demonstrations, generalisations are carried out using the original DMPs and modified DMPs, respectively. The trajectories generalised from the original DMPs have distortions. The trajectories generalised by the improved DMPs conform to the expected trajectories.

2.2.2. Admittance control

The admittance model is defined as follows:

(18) \begin{equation} {M_{A}\ddot{x}_{A}+D_{A}\dot{x}_A+K_{A}(x_A-x_0)=f_h} \end{equation}

where $M_{A}$ is the quality matrix, $D_{A}$ is the damping matrix and $K_{A}$ is the spring factor matrix. $f_h$ is the traction of the demonstrator. $x_A$ is the pose of the robot end-effector, $\dot{x}_{A}$ is the speed and $\ddot{x}_{A}$ is the acceleration. $x_0$ is the original point of the pose. The robot end-effector needs to move on the writing board during the writing process, so $x_A=x_0$ . The admittance model can be simplified as:

(19) \begin{equation} {M_{A}\ddot{x}_{A}+D_{A}\dot{x}_A=f_h}. \end{equation}

Therefore, the acceleration can be calculated using the traction force $f_h$ and then the velocity and position can be calculated by integration to obtain the trajectory. In this article, $f_h$ is the contact force between the pen tip and the writing board, with an expected value of $2N$ .

2.3. Designed system

Based on the above approach, we have designed a multi-sensor-based robotic writing system. As shown in Fig. 12, at the demonstration stage, the strokes are extracted from the demonstrations. During the trajectory generalisation phase, the camera detects changes in the pose of the writing board and calculates the rotation matrix $\textbf{R}$ and the translation matrix $\textbf{t}$ . The DMPs model calculates the generalised trajectory points based on the $\textbf{R}$ and $\textbf{t}$ and feeds the trajectory points into the admittance control model. The admittance control algorithm calculates the execution position based on the given trajectory points and real-time force information and outputs the execution position to the robot. At the same time, the execution position will be different from the desired position and in the next step this differential will be used to correct the trajectory points in advance to improve accuracy.

Figure 12. Robotic writing system.

3.1. Stroke extraction

In order to verify the accuracy of Chinese character stroke extraction, we randomly selected 100 handwriting Chinese characters with different structures and complexities for experiments. As shown in Fig. 13, the experiments show that the algorithm works well. Statistically, 93% of the strokes of handwritten Chinese characters can be extracted accurately, with an average speed of 0.4 s per image. The characters boxed in Fig. 13 are characters with inaccurate stroke recognition. After studying them, we have divided them into three categories. Category 1 is for those boxed in red, whose strokes are broken. This means that a single stroke has been split into multiple strokes or is incompletely recognised. Those boxed in green are in category 2: misconnected strokes. This means that a stroke has been incorrectly connected to another stroke. Boxed in blue is category 3, there are redundant strokes. Excess strokes take the form of burrs consisting of just one or two pixels. Analysis of these three categories shows that almost all recognition errors happen in the refinement process. When inaccurate refinement leads to a change in the direction of the strokes, deformations are observed at the intersection and junction points of the Chinese character’s skeleton, errors in category 1 are likely to occur. During the writing process, variations in pen pressure or ink leakage can result in changes in the thickness of strokes, leading to the occurrence of jaggies in the refinement process, errors in category 3 are likely to occur. Errors in category 2 are caused by inaccurate refinement and the poor writing where multiple interconnected or intersecting stroke components share a consistent direction.

Figure 13. Stroke extraction results. Different colours indicate different strokes.

3.2. Modified DMPs

There are two methods to generalise a trajectory based on disturbances in real time. Method 1 firstly generalises the new trajectory using the original DMPs and then adjusts the new trajectory using rotation and translation matrices, $\textbf{R}$ and $\textbf{t}$ . Method 2 uses the modified DMPs to generalise the trajectory directly based on $\textbf{R}$ and $\textbf{t}$ . We set up a writing track and use visual information to obtain the pose of the writing board to calculate $\textbf{R}$ and $\textbf{t}$ . These two methods are then used to make the trajectory follow the desired pose on the writing board. The effects of the two methods are shown in Fig. 14. It can be seen that the trajectory output of method 1 is more oscillating, not smooth enough and has large errors. Method 2 outputs a smoother trajectory with less oscillation, which is more effective. In comparison to Method 1, the trajectories produced by Method 2 exhibit a difference of up to 7.1977 mm in the z-direction, which poses a potential safety hazard in practical applications, as it could result in the inadvertent puncturing of the writing surface. We analysed the reasons for the large differences in the results between the two methods. When using method 1, outputs of the original DMP have errors in practice, such as distortion of the trajectory and robot movement errors. Also, $\textbf{R}$ and $\textbf{t}$ are calculated from visual information and have errors. Because $\textbf{R}$ and $\textbf{t}$ do not fall within the framework of DMPs, the multiplication with the trajectory points is not constrained by DMPs, leaving the trajectory with a large offset. Whereas method 2 incorporates $\textbf{R}$ and $\textbf{t}$ into the framework of DMPs, the generalisation of DMPs will change with $\textbf{R}$ and $\textbf{t}$ . At the same time, the trajectory does not produce large offsets due to the constraints of the DMPs framework.

Figure 14. The effect of the two methods.

3.3. Admittance control

We have measured the contact force between the pen tip and the writing board when a person writes. We find that the best writing performance is achieved when the force sensor has $2N$ contact force on the z-axis. If the contact force is not $2N$ when the robot arm moves to the specified position, the admittance control algorithm can adjust the end position of the robot arm in time to guarantee the contact force always be $2N$ . As shown in Fig. 15, when the initial contact force is 0, i.e. when the end-effector is not touching the writing board, the admittance control moves the end-effector closer to the writing board until the contact force converges to $2N$ . When the initial contact force is $6N$ , the admittance control policy steers the end-effector moving away from the writing board until the contact force is stabilised at $2N$ . The admittance control also adjusts the contact force in time when there is any interference during the adjustment process.

Figure 15. The effect of the admittance control.

3.4. Application

In this experiment, we verify the effectiveness of the designed interference-resistant robotic Chinese character writing system. A human randomly writes a Chinese character. Then the robot imitates the human handwriting on the writing board. While the robot writes, the human randomly moves, turns and flips the board, and the robot can still follow the posture of the board to adjust its trajectory. As shown in Fig. 16, the experimental platform is composed of an Elite Collaborative Robot EC66, an Intel RealSense Depth Camera D435 and a LCD Writing Board. The Elite Collaborative Robot EC66 is a lightweight and flexible collaborative robot. Its weight is 17.5 kg and the largest payload is 6 kg. It runs efficiently and can reach a maximum speed of 2.8 m/s, with a soft and consistent trajectory. As a collaborative robot, it enables safe human–robot interaction with a high degree of reliability and safety. It can therefore complete different tasks while ensuring the safety of humans. With a range of 10 m and a wide field of view, the Intel RealSense Depth Camera D435 has a wide range of applications in robotics development. By affixing four calibration papers to the four corners of the writing board, it is able to estimate the posture of the writing board very well in this experiment, i.e. to recognise human interference. During the robot’s writing process, the human randomly and continuously moves, rotates and flips the writing board. The robot can write complete Chinese characters on the writing board.

Figure 16. Experimental platform.

The results of the experiment are shown in Fig. 17. The robot is able to adjust its end trajectory in time to write complete and accurate Chinese characters on the writing pad under human interferences.

Figure 17. The results of the experiment.