2.1. Mobile robot odometry: Kinematic model

The mobile robot platform used in this research is Kobuki differential wheeled robot platform [Reference Siegwart, Nourbakhsh and Scaramuzza31]. As a non-holonomic mobile robot, the Kobuki has its two-dimensional model, which gives the position and orientation of the local frame, with respect to the global one, as shown in Fig. 1 [Reference Siegwart, Nourbakhsh and Scaramuzza31].

Figure 1. The local reference frame and global reference frame.

The global generalized velocity vector $\dot{G}$ shown in Eq. (1) includes three variables, $X_{\text{odom}}$ , $Y_{\text{odom}}$ (the state in 2D frame), and $\theta _{\text{odom}}$ (heading),

(1) \begin{equation}\dot{G}=\left(\begin{array}{c} \dot{X}_{\text{odom}}\\[4pt] \dot{Y}_{\text{odom}}\\[4pt] \dot{\theta }_{\text{odom}} \end{array}\right)\end{equation}

where the local frame velocity matrix (Eq. 2),

(2) \begin{equation}\dot{P}=\left(\begin{array}{c} \dfrac{r(\dot{{\varnothing }}_{R}+\dot{{\varnothing }}_{L})}{2}\\[7pt] 0\\[4pt]\dfrac{r(\dot{{\varnothing }}_{R}-\dot{{\varnothing }}_{L})}{2l} \end{array}\right)\end{equation}

where $r$ is the radius of the Kobuki robot wheel which equals 0.0352 m, $l$ is the distance between the two wheels which equals 0.21 m, and ${\varnothing }$ represents the robot wheel angle for right and left wheels.

In order to relate the robot’s local frame to the global frame, a rotation matrix needs to be introduced ( $R$ ) as shown in Eq. (3).

(3) \begin{equation}R=\left(\begin{array}{c@{\quad}c@{\quad}c} \cos\!\left(\theta _{\text{odom}}\right) & -\sin\! \left(\theta _{\text{odom}}\right) & 0\\[4pt] \sin\! \left(\theta _{\text{odom}}\right) & \cos\!\left(\theta _{\text{odom}}\right) & 0\\[4pt] 0 & 0 & 1 \end{array}\right)\end{equation}

The rotation matrix calculates the global frame state matrix in relation to the local frame state matrix. The relationship between the local frame and the global frame is shown in Eq. (4).

(4) \begin{equation}\dot{G}=R\dot{P}\end{equation}

To track the position and orientation of the robot in the global reference frame, one must measure the local change of position and orientation of the robot and use the $R$ matrix to obtain the needed information. This process is usually referred to as dead reckoning, which consists of estimating the value of the position and orientation using the earlier value and adding the current changes.

Diverse types of sensors provide different types of information about the position and orientation of the robot. IMUs and embarked depth cameras sensors, for example, provide local pose information individually, which needs to be converted to the global frame that represents the robot global state.

In the rest of this section, the different sensors used in this work are presented along with the methodology needed to extract the pose information related to the above model.

2.2. Localization using IMU sensor

Accelerometers and gyroscopes sensors are basic components in the IMU unit, which work together to measure the three accelerations and the angular velocities, along the three axes in space.

Typically, IMUs have one accelerometer, gyro, and magnetometer per axis, for each of the three axes: pitch, roll, and yaw. These sensors detect the motion and output a signal, which could be an acceleration from the accelerometer sensors or an angular velocity from the gyroscope sensors.

To enhance the performance of the accelerometer, the jerk has been taken into consideration to remove the abrupt changes of velocity, as in ref. [Reference Ibrahim and Moselhi32]. The use of jerk requires four consecutive accelerations readings in order to filter out the noises or unwanted readings as in Eqs. (5)–(7).

(5) \begin{equation}J_{1}=\frac{\text{acc}_{2}-\text{acc}_{1}}{dt}\end{equation}

(6) \begin{equation}J_{2}=\frac{\text{acc}_{4}-\text{acc}_{3}}{dt}\end{equation}

(7) \begin{equation}A_{t}=\frac{J_{1}-J_{2}}{2}dt\end{equation}

where $\text{ac}\text{c}_{i}$ and $J_{i}$ are the measured acceleration and the jerk values of the ith iteration, respectively. Then the local acceleration $A_{t}$ is estimated according to Eq. (7). Therefore, the acceleration is taken every four iterations. These equations will be applied to the three directions so that it gives the filtered acceleration values in the three local axes. The use of jerk calculations will automatically detect the gravity component and remove it at any initial state. Moreover, it will continue to cancel gravity at the next states as shown in ref. [Reference Ibrahim and Moselhi32]. A rotation matrix is used to convert the IMU measured accelerations to the global axes [Reference Premerlani and Bizard33, Reference Khan, Clark, Woodward and Lindeman34]. Eq. (8) shows the rotation matrix that needs to be applied, followed by Eq. (9) that shows the global acceleration $\boldsymbol{A}_{t}$ . As the IMU measurements would always be relative to its initial position (local position), the differences in the angles can be used to estimate the position in the global frame. Consequently, the global acceleration values will be integrated using Eqs. (10) and (11), resulting in the 3D position in the global frame.

(8) \begin{equation}\boldsymbol{R}_{\textbf{IMU}}=\left(\begin{array}{c@{\quad}c@{\quad}c} {\cos }\boldsymbol{\theta }{\cos }\boldsymbol{\psi } & {\sin }\boldsymbol{\varnothing }{\sin }\boldsymbol{\theta }{\cos }\boldsymbol{\psi }-{\cos }\boldsymbol{\varnothing }{\sin }\boldsymbol{\psi } & {\cos }\boldsymbol{\varnothing }{\sin }\boldsymbol{\theta }{\cos }\boldsymbol{\psi }+{\sin }\boldsymbol{\varnothing }{\sin }\boldsymbol{\psi }\\[4pt] {\cos }\boldsymbol{\theta }{\sin }\boldsymbol{\psi } & {\sin }\boldsymbol{\varnothing }{\sin }\boldsymbol{\theta }{\sin }\boldsymbol{\psi }+{\cos }\boldsymbol{\varnothing }{\cos }\boldsymbol{\psi } & {\cos }\boldsymbol{\varnothing }{\sin }\boldsymbol{\theta }{\sin }\boldsymbol{\psi }-{\sin }\boldsymbol{\varnothing }{\cos }\boldsymbol{\psi }\\[4pt] -{\sin }\boldsymbol{\theta } & {\sin }\boldsymbol{\varnothing }{\cos }\boldsymbol{\theta } & {\cos }\boldsymbol{\varnothing }{\cos }\boldsymbol{\theta } \end{array}\right)\end{equation}

(9) \begin{equation}\boldsymbol{A}_{t}=\left[\begin{array}{c} \boldsymbol{A}_{{\boldsymbol{x}_{t}}}\\[4pt] \boldsymbol{A}_{{\boldsymbol{y}_{t}}}\\[4pt] \boldsymbol{A}_{{\boldsymbol{z}_{t}}} \end{array}\right]=\boldsymbol{R}_{\textbf{IMU}}A_{t}\end{equation}

(10) \begin{equation}\boldsymbol{v}_{k}=\boldsymbol{A}_{t}dt+\boldsymbol{v}_{k-1}\end{equation}

(11) \begin{equation}\boldsymbol{P}_{\text{k}}=\boldsymbol{v}_{k}dt+\boldsymbol{P}_{k-1}\end{equation}

where $\theta, \phi, \psi$ represent the rotation angles in the three dimensions (yaw, pitch, and roll), the vector $\boldsymbol{v}_{k}$ represents the current velocity, and the vector $\boldsymbol{P}_{k}$ represents the current position which includes $\boldsymbol{X}_{\textbf{IMU}}$ , $\boldsymbol{Y}_{\textbf{IMU}}$ , and $\boldsymbol{Z}_{\textbf{IMU}}$ . Some IMUs are equipped with a pressure sensor to measure the altitude above the sea level using Eq. (12) [Reference Ibrahim and Moselhi32].

(12) \begin{equation}\text{Altitude}_{\text{barometer}}=\text{C}_{1}\left(1-\left(\frac{P}{P_{0}}\right)^{\frac{1}{\text{C}_{2}}}\right)\end{equation}

where $C_{1}$ and $C_{2}$ are non-dimensional constants and equal to 44,330 and 5.255, respectively, at 15°C, while $P$ is the measured pressure and $P_{0}$ is the pressure at the sea level (1013.25 hPa). An additional improvement that can be used is the magnetometer sensor in the IMU.

2.3. Depth-based localization

The chosen depth camera (Intel d435) provides two techniques of depth-based localization, which makes it less vulnerable to noises and vibrations. The first technique relies on the idea of “stereo camera,” which tries to imitate the human eyes, by utilizing the focal lengths of two adjacent lenses. As shown in Fig. 2, using triangulation, the distance from the camera reference to the object is calculated. Therefore, each point seen by the lenses can be considered as a feature to be used in the localization process. Eqs. (13)–(16) explain the stereo localization process [Reference Kang, Webb, Zitnick and Kanade35].

(13) \begin{equation}\frac{f}{{z}}=\frac{x_{1}-x_{2}}{b}\end{equation}

(14) \begin{equation}{z}=f\frac{b}{x_{1}-x_{2}}\end{equation}

(15) \begin{equation}x=x_{1}\frac{{z}}{f}\end{equation}

where $f$ is the focal length, $z$ is the depth distance from the camera to the object, $x_{1}$ and $x_{2}$ are the horizontal distances from both of the cameras to the object, d is the disparity, $b$ is the fixed distance between the two cameras (baseline), and $x$ is the horizontal distance from the camera frame to the object. Similarly, the vertical distance $y$ in Eq. (16) is formulated. However, for this application, the robot performs a planar motion, hence $y$ is constant.

(16) \begin{equation}y=y_{1}\frac{{z}}{f}\end{equation}

The position can be estimated by iterative closest point algorithm [Reference Ruan36], which relates two point-clouds by a rotation and translation matrix that can be calculated using least mean squares minimization. Moreover, the rotational angles (pitch, roll, and yaw) can be computed from the current plane orientation (i.e., $\Phi =\tan ^{-1} (z/x)$ is used for the roll angle) [Reference Sappa, Dornaika, Ponsa, Gerónimo and López37]. Hence, the pose is estimated.

Figure 2. Path resulted from the selected points according to the 2D cost map.

Figure 3. Path resulted from the selected points according to the 2D cost map.

On the other hand, an assistive algorithm is used along with the stereo camera to ensure more reliability, known as “structured light technology” [Reference Gupta, Agrawal, Veeraraghavan and Narasimhan38]. This technique uses a cloud of infrared lights projected on the map and received by an infrared receiver (IR) as shown in Fig. 3. Using triangulation, the distance to a certain point of projection can be calculated as shown in Eqs. (17)–(20).

(17) \begin{equation}\frac{d}{\sin\! \left(\alpha \right)}=\frac{b}{\sin\! \left(\gamma \right)}\end{equation}

Since $\sin \gamma =\sin\! (\pi -\gamma )$ , where $\alpha +\beta =\pi -\gamma$

(18) \begin{equation}d=b\frac{\sin\! \left(\alpha \right)}{\sin\! (\alpha +\beta )}\end{equation}

accordingly,

(19) \begin{equation}d_{x}=d \cos (\beta )\end{equation}

(20) \begin{equation}d_{z}=d \sin\! (\beta )\end{equation}

The selected method of depth localization in Intel d435 performs localization by taking features from the environment by the fusion of its stereo cameras and IR. These features are produced by Harris corner detection [39] with adaptive non-maximum suppression [Reference Liu, Huang and Wang40], and then the scale-invariant feature transform algorithm was used to create a 128-dimensional descriptor for every feature [Reference Lowe41, 42]. Therefore, these features will be continuously compared to previously saved features (using sum of squared distances ratio SSD Ratio) to detect closed-loop (viewing loop) and enhance the localization process. This method is known as RTABMAP [Reference Labbé and Michaud43]. RTABMAP uses ORB-SLAM to keep track of the location. ORB-SLAM, in turn, is based on “stereo camera” and “structured light technology” principles [Reference Ruan36]. The camera frames will help construct the map; in turn, the depth images will be used to localize the camera with respect to its environment. Accordingly, the 2D results from the camera are longitude and latitude values.

Figure 3 shows the depth camera from a top view, resulting in a two-dimensional frame. However, it is worth noting that this is a dense representation which incorporates a third dimensional component.

Since the use of the camera and the RTABMAP module yield the pose of the camera, one needs to apply another transformation to obtain the pose of the Kobuki, which is the global frame. This transformation can be represented by a homogeneous matrix $\boldsymbol{H}$ given by:

(21) \begin{equation}\boldsymbol{H}=\left[\begin{array}{c@{\quad}c} \boldsymbol{R} & \boldsymbol{T}\\[4pt] \textbf{0}_{(\textbf{1}\times \textbf{3})} & \textbf{1} \end{array}\right]\end{equation}

where $\boldsymbol{R}$ is the rotation matrix and $\boldsymbol{T}$ is the translation vector [44]. With reference to the experimental setup shown in Fig. 13, the rotational matrix $\boldsymbol{R}=\left[\begin{array}{c@{\quad}c@{\quad}c} 0 & 0 & 1\\[4pt] 1 & 0 & 0\\[4pt] 0 & 1 & 0 \end{array}\right]$ and the translational matrix $\boldsymbol{T}=\left[\begin{array}{c} 0\\[4pt] 0\\[4pt] h=-0.2763 \end{array}\right]$ , since the camera is fixed at the center of the Kobuki with a fixed measured height of 27.63 cm. These quantities are fixed throughout the missions to relate the local camera reference frame to the global reference of the Kobuki.

Points projected and received by the sensor vary depending on the environment and the graphical processing unit (GPU) used to operate the Intel d435 camera. The more distinctive is the environment, the more features are in the frame. Moreover, the fastest frequency at which the camera can operate is 1 Hz, this frequency is set as a default setting. However, this frequency can be reduced if a low-end GPU is used. Lower frequencies cause fewer features in the frame, which reduces the quality of the localization accordingly.

2.4. Depth/IMU intelligent sensor fusion framework

The above sensors are mounted on a Kobuki mobile robot developing a low-cost depth/IMU intelligent sensor fusion approach. In this approach, the slow depth camera localization and the fast-inaccurate IMU localization are fused based on a modified CCN neural network approach as shown in Fig. 4 [Reference Chiang, Noureldin and El-Sheimy30, Reference Malleswaran, Vaidehi and Jebarsi45–Reference Malleswaran, Vaidehi, Saravanaselvan and Mohankumar47]. Initially, the IMU sensor provides the robot absolute position based on Eqs. (5)–(9), and this solution is generated based on the “Direct Cosine Matrix,” correction of the gravity, and “Double Integration” blocks [Reference Bikonis and Demkowicz48], providing the globalized IMU 2D frame measurements. The performed localization process by the IMU sensors is completed by the “Position Estimation” block; the outputs $\boldsymbol{X}_{\textbf{IMU}} \text{and} \boldsymbol{Y}_{\textbf{IMU}}$ are fed to the CNN neural network as inputs. IMU sensors output is normalized in order to provide flexibility and scalability for other paths. On the other hand, the depth camera extracts the features in the frame, to generate the global frame for the camera and estimate $\boldsymbol{P}_{\text{depth}}$ using RTABMAP python library. However, the depth frame 2D pose is considered as neural network target outputs. The collected heading ( $\theta _{\text{IMU}}$ and $\theta _{\text{Depth}}$ ) and altitude ( $Z_{\text{IMU}}$ and $Z_{\text{Depth}}$ ) from both sensors are averaged and are fed to the final state estimation block in Fig. 4. The averaging of the heading and the altitude is to minimize the small drift in the gyroscope in short missions [Reference Li, Gao, Meng, Wang and Guan49], and the absence of it in the barometer [Reference Mummadi, Leo, Verma, Kasireddy, Scholl, Kempfle and Laerhoven50], respectively. Moreover, drift in the gyroscope can be corrected with the help of the accelerometer, thus yielding more reliable results.

Figure 4. The localization framework using IMU and depth sensor.

2.5. Localization using cascade correlation neural network fusion

Although the camera depth sensor is a reliable source and its accuracy is higher than the IMU accuracy [Reference Lukianto, Hönniger and Sternberg7], it is not as fast as the IMU sensors, and it does not work in all conditions. The camera depth sensor can lose availability when it is too close to the wall, or when a severe swerve occurs by the robot. Therefore, fusing both sensors would enhance the localization process and would decrease the dependency on a certain sensor for localization.

Filters such as KF have several limitations such as the need of knowing the covariance values and the statistical properties in advance. Several limitations of KF were discussed in refs. [Reference Grewal and Andrews51–Reference Mohammed57]. Thus, neural networks are superior, since they do not require the model of the system, making them easily implementable on any hardware.

Determining the best topology when applying neural network creates a challenge when designing the ANN, which also includes the hidden neurons and hidden layers in the network. Several ways were discussed to determine the best size (hidden neurons) and depth (hidden layers) for the neural networks in refs. [Reference Borkowf58] and [Reference Bishop59], which indicate that the size and depth are mainly dependent on the applications itself.

The chosen fusion method is a constructive neural network since it can adjust its layers based on the needs. It starts constructing the layers one by one, while continuously checking if the number of layers is sufficient.

Several methods for constructive neural networks were proposed in refs. [Reference Chiang, Noureldin and El-Sheimy30, Reference Malleswaran, Vaidehi and Jebarsi45–Reference Malleswaran, Vaidehi, Saravanaselvan and Mohankumar47, 60]. The CCN [Reference Nachev, Hogan and Stoyanov28] was selected due to the aforementioned advantages and uses.

The topology of the CCN starts as shown in Fig. 5, followed by an explanation in Table I. The weights linking the input neurons to the output ones are started to be trained. This topology will be extended adaptively based on the application needs. As shown in Fig. 4, the input neurons receive the IMU-based pose data in the global frame ( $X_{\text{IMU}}$ and $Y_{\text{IMU}}$ ). The output neurons provide the estimated corrected IMU measurements, while the measured pose values from the depth camera solution ( $X_{\text{Depth}}$ and $Y_{\text{Depth}}$ ) are used as the target values for training.

Table I. The explanation of the CCN topology shown in this section.

Figure 5. Initial topology for CCN (no hidden layers).

Figure 6. Second phase of CCN when adding a hidden layer.

The model keeps on building layers based on the initial topology until no further improvement can be made. The input data are trained through the Quickprop algorithm which does not require any backpropagation to adjust the weights [Reference Fahlman61]. Quickprop is relatively fast and simple compared to other algorithms [Reference Bishop59].

The next phase is to recruit a new pool of candidate neurons as shown in Fig. 6 by declaring initial weights. The candidate neurons are firstly trained based on their connections to input neurons, while the connections to the output neurons have not been made yet. The training is done to ultimately maximize the correlation among all connections going from the candidate neurons to the output neurons. Thus, a pseudo connection is made to deliver information about the residual error that was initially randomized, resulting in a maximization of the correlation in the connections from the candidate neurons and the residual error at the output neurons [Reference Chiang, Noureldin and El-Sheimy30]. Eq. (22) represents the maximization formula; its parameters are explained in Table II.

(22) \begin{equation}S=\sum _{o}\left| \sum _{p}(V_{p}-\overline{V})(E_{p,o}-\overline{E_{o}})\right|\end{equation}

Maximizing $S$ can be done through differentiation which uses the Quickprop algorithm to adjust concerning weights. Afterwards, the winner neurons layer based on the incoming weights will be frozen, resulting in the insertion of the winner layer into the active network once the training is completed. This engages the second step of recruitment, in which the weights of the connections from the hidden neutrons to the output neurons become adjustable and start to be trained, through a sigmoidal activation function. In addition, the input neurons connections to the output neurons are retrained as shown in Fig. 7.

Table II. Parameters of Eq. (22).

Figure 7. Winner neurons (WN) trained to match output neurons.

When another layer of hidden neurons is added, the entire process starts again as shown in Figs. 8 and 9. This process repeats itself and the layers of hidden neurons are added until no further improvement can be done, or the number of training passes reaches its maximum. Finally, the network can be represented with (n) hidden neurons and layers modified in a feedforwarded manner as shown in Fig. 10.

Figure 8. Third phase of CCN when adding the second hidden layer.

Figure 9. The second set of (WN) are trained to match the output neurons.

Figure 10. Full architecture with N hidden layers (neurons).

To reduce the complex interactions between the hidden neurons, the backpropagation is replaced by a feedforward algorithm. Moreover, the feedforward algorithm can always be incremented; hence, the trained layers will stay and will not need to be retrained again as shown in Fig. 9. CCN topology is determined automatically depending on the application complexity, so the need of performing additional trials to re-decide the hidden neurons and hidden layers does not exist.

Finally, the trained network is used in the fusion process. Based on the average error histogram among 60%, 70%, 80%, 90%, and 100% training data, 80% was observed to be the best percentage among the other percentages. The final structure of the CCN has 12 layers where each hidden layer has two neurons while being cascaded until the last layer.

3.1. Mapping and path-planning algorithms

With the prior knowledge of an indoor environment, a grid-cost map can be introduced based on the starting node and the goal node. A node-based algorithm such as A^* can be used to give the path in real time, to generate the path and the associated waypoints. Part of the path-planning algorithm is obstacle avoidance, which involves using a pre-existing map. Therefore, obstacles should have an infinite cost to force the algorithm to avoid the nodes of the obstacle in the cost map. A 2D grid-cost map is shown in Fig. 11.

The black box in Fig. 11 represents the change in the values of the cost map to infinity where the obstacle exists. Then, the different obstacles of the map can be added similarly. A^* is used due to its lower computational cost and faster path planning. The grid is based on the location of the starting node and the ending one in addition to the distance between them, which will define the step size for the grid-cost map.

The shown grid-cost map in Fig. 11 creates the waypoints while considering the need of changing the heading. In other words, since the used robot is non-holonomic, the rotation of the robot will be considered as an extra waypoint. In order to generate a more efficient navigational path and reduce the number of waypoints, the waypoints at which the orientation is changing are selected.

Table III. The generated waypoints of the previous path.

Figure 11. 2D Grid cost showing the cost (intensity of the color) with the presence of an obstacle.

An illustrative example is shown in Fig. 12, with a reduced number of points on the map for faster convergence. The resolution of the provided grid depends mainly on object sizes, shapes in the environment, and size of the environment. Table III shows the generated waypoints, which will be fed to the robot navigation program to track. The epoch number is just a representation of the time at each waypoint passage. The axes in Figs. 11 and 12 are unitless as they represent the steps in the map.

Figure 12. Path resulted from the selected points according to the 2D cost map.

Figure 13. Experimental setup.

3.2. Used framework and platform

The framework of all the used sensors and actuators is the robotic operating system (ROS) [62]. The main advantage of ROS is that it provides a domain for all the sensors and actuators to communicate with each other in different patterns, distinct types, and various locations. The mobile robot used is Kobuki. The Kobuki robot is a ROS-enabled robot, which makes it integrable with a variety of sensors and other actuators.

The used IMU is the XSens Mti-G-710 GNSS. It is a 10 Degrees of Freedom IMU that consists of a three-axis accelerometer, a three-axis gyroscope, a three-axis magnetometer, and a barometer. The magnetometer is not used due to disturbances that may arise in the environment. The GPS mounted on the XSens- IMU is automatically enabled when there is an available signal in an outdoor environment. The use of the GPS/IMU, in this work, is to compare the indoor environment result to the outdoor one using a commercial off-the-shelf available solution [Reference Jaradat and Abdel-Hafez22]. This is done through extended KF fusion between the IMU and GPS in the XSens Module as in ref. [Reference Bikonis and Demkowicz48]. The XSens solution (IMU/GPS fusion) is used as a reference in the outdoor environment. On the other hand, Kobuki mobile robot platform is utilizing the odometry solution based on its internal IMU sensor and wheels’ encoders to provide the localization measurements for indoor environment. To verify the accuracy of the Kobuki odometry solution, a test has been conducted to follow a certain path. The conducted test is based on the validation of the non-holonomic robot model presented in ref. [Reference Renawi, Jaradat and Abdel-Hafez63]. During the test, actual waypoints in a U-shaped path were compared to the desired waypoints put on the floor. The mean error in robot position for X and Y axis is 0.011 and 0.033 m, respectively.

Figure 14. The recorded path by depth camera, Kobuki odometry, GPS, and IMU.

Many options are available in the market for depth sensors [Reference Alkhawaja, Jaradat and Romdhane5]; however, the chosen depth sensor was the Intel d435 depth camera. The mentioned camera was chosen based on two main reasons. First, the weight of the Intel d435 is 71 g which is low compared to the other existing depth cameras. Second, unlike several depth cameras and IR sensors, Intel d435 has a standalone USB-C port that is responsible for power and data transfer. USB-C port is capable of providing high power without compromising the speed of the data transfer. Other sensors might need a battery which increases the weight of the whole platform. Intel d435 is a ROS-enabled camera; as indicated in Fig. 4, the depth camera solution provides the estimated position of the robot based on fusion of the collected depth measurements by the camera, and more information about the Kobuki robot can be found at ref. [64]. Figure 13 shows all the sensors (IMU, GPS, and depth camera) and the laptop mounted on the Kobuki robot.

3.3. Control

The Kobuki mobile robot is a non-holonomic platform, where the wheel odometry (from the encoder) is linked to its heading. Wheels’ encoders are used to sense its movement in the X and Y axes. However, the robot wheels motor can only actuate in its local X axis. Therefore, the controller is controlling the robot linear velocity along the X axis, only. Mainly a proportional integral (PI) controller is used to control the speed of the robot as in Eq. (23).

(23) \begin{equation}v=K_{p}e_{v}+K_{i}\int _{0}^{T}e_{v} dt\end{equation}

where $e_{v}$ is the error in the linear speed. The gains of the PI controller were obtained using the process reaction curve through Cohen–Coon method [Reference Joseph65] ( $K_{p}=9.465, K_{i}=0.642$ ). The accepted error range was 0.01 m before moving onto the next waypoint.

Once the waypoints are fed to the user-defined navigation program, the robot divides the waypoints into two main categories (linear and twist). The linear category is for the robot linear movement considering that the orientation has been achieved already. The twist category is for the robot rotation to adjust and achieve the needed orientation.