In this work, we propose a novel approach for tomato pollination that utilizes visual servo control. The objective is to meet the growing demand for automated robotic pollinators to overcome the decline in bee populations. Our approach focuses on addressing this challenge by leveraging visual servo control to guide the pollination process. The proposed method leverages deep learning to estimate the orientations and depth of detected flower, incorporating CAD-based synthetic images to ensure dataset diversity. By utilizing a 3D camera, the system accurately estimates flower depth information for visual servoing. The robustness of the approach is validated through experiments conducted in a laboratory environment with a 3D printed tomato flower plant. The results demonstrate a high detection rate, with a mean average precision of 91.2 %. Furthermore, the average depth error for accurately localizing the pollination target is impressively minimal, measuring only 1.1 cm. This research presents a promising solution for tomato pollination, showcasing the effectiveness of visual-guided servo control and its potential to address the challenges posed by diminishing bee populations in greenhouses.

This article presents a detailed examination of circular target localization techniques for measuring robot pose and performing online pose correction. The investigated target localization methods include centroiding, ellipse fitting with point data and gradient information, and ellipse fitting methods with augmented and corrected input data. The performance of each method is evaluated in terms of accuracy and precision of measurements through experimental comparison with a laser tracker. This study provides technical and practical insights for selecting an appropriate target localization method in robotic applications. It also introduces a vision-based solution for robot relative error correction, comprising the calibration procedure and a closed-loop control with a proportional–integral-derivative controller for pose correction. Results show enhanced accuracy in robot positioning relative to workpiece, highlighting the effectiveness of the proposed solution in robotic drilling applications.

This article designs a robotic Chinese character writing system that can resist random human interference. Firstly, an innovative stroke extraction method of Chinese characters was devised. A basic Chinese character stroke extraction method based on cumulative direction vectors is used to extract the components that make up the strokes of Chinese characters. The components are then stitched together into strokes based on the sequential base stroke joining method. To enable the robot to imitate handwriting Chinese character skills, we utilised stroke information as the demonstration and modelled the skills using dynamic movement primitives (DMPs). To suppress random human interference, this article combines improved DMPs and conductance control to adjust robot trajectories based on real-time visual measurements. The experimental results show that the proposed method can accurately extract the strokes of most Chinese characters. The designed trajectory adjustment method offers better smoothness and robustness than direct rotating and translating curves. The robot is able to adjust its posture and trajectory in real time to eliminate the negative impacts of human interference.

In this study, we present a new approach to improving vocal fold access to perform phonomicrosurgery. It is done by shooting the laser through a mirror to reach the vocal fold hidden parts. A geometrical study of laser shooting path was conducted for vocal fold anatomical constraints, followed by devising a laser-shooting system conceptual design. Control laws were developed and tested by simulation and validated experimentally on a test bench in a monocular and stereoscopic configuration. Simulation and experimental results are provided to demonstrate the effectiveness of the developed approach.

This article proposes the design and development of a novel custom-built, autonomous scaled multiwheeled vehicle that features an eight-wheel drive and eight-wheel steer system. In addition to the mechanical and electrical design, high-level path planning and low-level vehicle control algorithms are developed and implemented including a two-stage autonomous parking algorithm is developed. A modified position-based visual servoing algorithm is proposed and developed to achieve precise pose correction. The results show significant gains in accuracy and efficiency comparing with an open-source path planner. It is the aim of this work to expand the research of autonomous platforms taking the form of commercial and off-road vehicles using actuated steering and other mechanisms attributed to passenger vehicles. The outcome of this work is a unique autonomous research platform that features independently driven wheels, steering, autonomous navigation, and parking.

Interaction between a robot and its environment requires perception about the environment, which helps the robot in making a clear decision about the object type and its location. After that, the end effector will be brought to the object’s location for grasping. There are many research studies on the reaching and grasping of objects using different techniques and mechanisms for increasing accuracy and robustness during grasping and reaching tasks. Thus, this paper presents an extensive review of research directions and topics of different approaches such as sensing, learning and gripping, which have been implemented within the current five years.

In this paper, we present and implement a novel approach for position-based visual servoing. The challenge of controlling the mobile robot while simultaneously estimating the camera to mobile robot transformation is solved. This is achieved using gradient descent (GD)-based estimation and the sliding-mode approach. The GD approach allows online parameter estimation for controlling the robot to achieve a desired position and orientation. The adaptive nature of the parameters demonstrates the robustness of the system. In contrast to existing work, the proposed technique achieves both estimation and control tasks in a single experiment. Simulation and experimental results are provided to validate the performance of the proposed scheme.

During visual servoing space activities, the attitude of free-floating space robot may be disturbed due to dynamics coupling between the satellite base and the manipulator. And the disturbance may cause communication interruption between space robot and control center on earth. However, it often happens that the redundancy of manipulator is not enough to fully eliminate this disturbance. In this paper, a method named off-line optimizing visual servoing algorithm is innovatively proposed to minimize the base disturbance during the visual servoing process where the degrees-of-freedom of the manipulator is not enough for a zero-reaction control. Based on the characteristic of visual servoing process and the robot system modeling, the optimal control method is applied to achieve the optimization, and a pose planning method is presented to achieve a second-order continuity of quaternion getting rid of the interruption caused by ambiguity. Then simulations are carried out to verify the method, and the results show that the robot is controlled with optimized results during visual servoing process and the joint trajectories are smooth.

This paper deals with a generic modeling and vision-based control approach for a broad class of parallel mechanisms. First, a generic architecture representing several families is proposed. Second, inspired by the geometry of lines, a generic differential inverse kinematic model according to the proposed generic structure is introduced. Finally, based on the image projection of cylindrical legs, a kinematic vision-based control using legs observation is presented. The approach is illustrated and validated on the Gough–Stewart and Par4 parallel robots.

This paper presents a novel system for human–robot interaction in object-grasping applications. Consisting of an RGB-D camera, a projector and a robot manipulator, the proposed system provides intuitive information to the human by analyzing the scene, detecting graspable objects and directly projecting numbers or symbols in front of objects. Objects are detected using a visual attention model that incorporates color, shape and depth information. The positions and orientations of the projected numbers are based on the shapes, positions and orientations of the corresponding objects. Users select a grasping target by indicating the corresponding number. Projected arrows are then created on the fly to guide a robotic arm to grasp the selected object using visual servoing and deliver the object to the human user. Experimental results are presented to demonstrate how the system is used in robot grasping tasks.

Although image-based visual servoing (IBVS) provides good performance in many dual-arm manipulation applications, it reveals some fatal limitations when dealing with a large position and orientation uncertainty. The object features may leave the camera's field of view, and the dual-arm robot may not converge to their goal configurations. In this paper, a novel vision-based control strategy is presented to resolve these limitations. A visual path planning method for dual-arm end-effector features is proposed to regulate the large initial poses to the pre-alignment poses. Then, the visual constraints between the position and orientation of two objects are established, and the sequenced subtasks are performed to attain the pose alignment of two objects by using a multi-tasks IBVS method. The proposed strategy has been implemented on a MOTOMAN robot to perform the alignment tasks of plug–socket and cup–lid, and results indicate that the plug and socket with the large initial pose errors 145.4 mm, 43.8○ (the average errors of three axes) are successfully aligned with the allowed pose alignment errors 3.1 mm, 1.1○, and the cup and lid with the large initial pose errors 131.7 mm, 20.4○ are aligned with the allowed pose alignment errors −2.7 mm, −0.8○.

It is well known that the use of multi-rate control techniques have improved the performance of many systems in general, and robotic systems, in particular. The main contribution of this paper is the generalization of the Reference Filtering control strategy from a dual-rate point of view, improving its inherent properties by overcoming the problem of sensor latency. In the paper, we discuss and analyze the improvements introduced by the novel dual-rate reference filtering control strategy in terms of convergence time, reachability and robustness. More specifically, we discuss the capability to solve positioning tasks, when hardware limitations are present with large sampling rates. In addition, a comparison is made between the single-rate and the proposed dual-rate control strategies to prove the advantages of the latter approach. A complete set-up has been prepared for validation, including a six degree of freedom (DOF) industrial manipulator, a smart camera and embedded hardware used as a high level controller.

This paper proposes a new visual control approach based on sinusoidal inputs to be used on a nonholonomic robot. We present several contributions: In our method, developed considering a unicycle kinematic model, sinusoids are used in such a way that the generated vehicle trajectories are feasible, smooth and versatile. Our technique improves previous sinusoidal-based control works in terms of efficiency and flexibility. As further contributions, we present analytical expressions for the evolution of the robot's state, and propose a new state-feedback control law based on these expressions. All the information used in the control scheme is obtained from omnidirectional vision by means of the one-dimensional trifocal tensor. Stability analysis of the proposed approach is presented, and its performance is illustrated through experiments.

In this paper we address the problem of stabilization and local positioning of a four-rotor rotorcraft using computer vision. Our approaches to estimate the orientation and position of the rotorcraft combine the measurements from an Inertial Measurement Unit (IMU) and a vision system composed of a single camera. In the first stage, the vision system is used to estimate the position and yaw angle of the rotorcraft, while in the second stage the vision system is used to estimate the translational velocity of the flying robot. In both cases the IMU gives the pitch and roll angles at a higher rate. The technique used to estimate the position of the rotorcraft in the first stage combines the homogeneous transformation approach for the camera calibration process with the plane-based pose method for estimating the position. In the second stage, a navigation system using the optical flow is also developed to estimate the translational velocity of the aircraft. We present real-time experiments of stabilization and location of a four-rotor rotorcraft.

We present two visual servoing controllers (pose-based and image-based) enabling mobile robots with a fixed pinhole camera to reach and follow a continuous path drawn on the ground. The first contribution is the theoretical and experimental comparison between pose-based and image-based techniques for a nonholonomic robot task. Moreover, our controllers are appropriate not only for path following, but also for path reaching, a problem that has been rarely tackled in the past. Finally, in contrast with most works, which require the path geometric model, only two path features are necessary in our image-based scheme and three in the pose-based scheme. For both controllers, a convergence analysis is carried out, and the performance is validated by simulations, and outdoor experiments on a car-like robot.

The uncalibrated robotic hand–eye coordination problem is firstly modeled by a dynamic system, where the unknown hand–eye relationship is regarded as the system's unmodeled dynamics. A state observer is then designed to estimate impacts of this modeling error together with the system's external disturbances. With the estimation results as the compensation, the system control is thus accomplished based on a nonlinear combination of the system state errors. Convergence analysis of the whole system under the proposed control scheme is emphasized. Simulations and experiment results are presented to verify the performance of the proposed approach.

For efficient visual servoing of an “eye-in-hand” robot, the concepts of Augmented Image Space and Transformed Feature Space are presented in the paper. A formal definition of image features as functionals is given along with a technique to use defined image features for visual servoing. Compared with other known methods, the proposed concepts reduce the computational burden for visual feedback, and enhance the flexibility in describing the vision-based task. Simulations and real experiments demonstrate that the proposed concepts are useful and versatile tools for the industrial robot vision tasks, and thus the visual servoing problem can be dealt with more systematically.

When computer vision technique is used in robotics, robotic hand–eye calibration is a very important research task. Many algorithms have been proposed for hand–eye calibration. Based on these algorithms, we introduce a new hand–eye calibration algorithm in this paper, which employs the screw motion theory to establish a hand–eye matrix equation by using quaternion and gets a simultaneous result for rotation and translation by solving linear equations. The algorithm proposed in this paper has high accuracy and stable computational efficiency and can be understood easily. Both simulations and real experiments show the superiority of our algorithm over the comparative algorithms.

Robotic manipulators that have interacted with uncalibrated environments typically have limited positioning and tracking capabilities, if control tasks cannot be appropriately encoded using available features in the environments. Specifically, to perform 3-D trajectory following operations employing binocular vision, it seems necessary to have a priori knowledge on pointwise correspondence information between two image planes. However, such an assumption cannot be made for any smooth 3-D trajectories. This paper describes how one might enhance autonomous robotic manipulation for 3-D trajectory following tasks using eye-to-hand binocular visual servoing. Based on a novel encoded error, an image-based feedback control law is proposed without assuming pointwise binocular correspondence information. The proposed control approach can guarantee task precision by employing only an approximately calibrated binocular vision system. The goal of the autonomous task is to drive a tool mounted on the end-effector of the robotic manipulator to follow a visually determined smooth 3-D target trajectory in desired speed with precision. The proposed control architecture is suitable for applications that require precise 3-D positioning and tracking in unknown environments. Our approach is successfully validated in a real task environment by performing experiments with an industrial robotic manipulator.