This paper proposes a virtual reality-based dual-mode teleoperation architecture to assist human operators in remotely operating robotic manipulation systems in a safe and flexible way. The architecture, implemented via a finite state machine, enables the operator to switch between two operational modes: the Approach mode, where the operator indirectly controls the robotic system by specifying its target configuration via the immersive virtual reality (VR) interface, and the Telemanip mode, where the operator directly controls the robot end-effector motion via input devices. The two independent control modes have been tested along the task of reaching a glass on a table by a sample population of 18 participants. Two working groups have been considered to distinguish users with previous experience with VR technologies from the novices. The results of the user study presented in this work show the potential of the proposed architecture in terms of usability, both physical and mental workload, and user satisfaction. Finally, a statistical analysis showed no significant differences along these three metrics between the two considered groups demonstrating ease of use of the proposed architecture by both people with and with no previous experience in VR.

As the proportion of the elderly population in the USA expands, so will the demand for rehabilitation and social care, which play an important role in maintaining function and mediating motor and cognitive decline in older adults. The use of social robotics and telemedicine are each potential solutions but each have limitations. To address challenges with classical telemedicine for rehabilitation, we propose to use a social robot-augmented telepresence (SRAT), Flo, which was deployed for long-term use in a community-based rehabilitation facility catering to older adults. Our goals were to explore how clinicians and patients would use and respond to the robot during rehab interactions. In this pilot study, three clinicians were recruited and asked to rate usability after receiving training for operating the robot and two of them conducted multiple rehab interactions with their patients using the robot (eleven patients with cognitive impairment and/or motor impairment and 23 rehab sessions delivered via SRAT in total). We report on the experience of both therapists and patients after the interactions.

With increasing attention to physical human-machine interaction (pHMI), new control methods involving contact force regulation in collaborative and coexistence scenarios have spread in recent years. Thanks to its internal robustness, high dynamic performance, and capabilities to avoid constraint violations, a Model Predictive Control (MPC) action can pose a viable solution to manage the uncertainties involved in those applications. This paper uses an MPC-driven control method that aims to apply a well-defined and tunable force impulse on a human subject. After describing a general control design suitable to achieve this goal, a practical implementation of such a logic, based on an MPC controller, is shown. In particular, the physical interaction considered is the one occurring between the body of a patient and an external perturbation device in a dynamic posturography trial. The device prototype is presented in both its hardware architecture and software design. The MPC-based main control parameters are thus tuned inside hardware-in-the-loop and human-in-the-loop environments to get optimal behaviors. Finally, the device performance is analyzed to assess the MPC algorithm’s accuracy, repeatability, flexibility, and robustness concerning the several uncertainties due to the specific pHMI environment considered.

We create a new tactile recording system with which we develop a three-axis fingernail-color sensor that can measure a three-dimensional force applied to fingertips by observing the change of the fingernail’s color. Since the color change is complicated, the relationships between images and three-dimensional forces were assessed using convolution neural network (CNN) models. The success of this method depends on the input data size because the CNN model learning requires big data. Thus, to efficiently obtain big data, we developed a novel measuring device, which was composed of an electronic scale and a load cell, to obtain fingernail images with 0$^\circ$ to 360$^\circ$ directional tangential force. We performed a series of evaluation experiments to obtain movies of the color changes caused by the three-axis forces and created a data set for the CNN models by transforming the movies to still images. Although we produced a generalized CNN model that can evaluate the images of any person’s fingernails, its root means square error (RMSE) exceeded both the whole and individual models, and the individual models showed the smallest RMSE. Therefore, we adopted the individual models, which precisely evaluated the tangential-force direction of the test data in an $F_x$-$F_y$ plane within around $\pm$2.5$^\circ$ error at the peak points of the applied force. Although the fingernail-color sensor possessed almost the same level of accuracy as previous sensors for normal-force tests, the present fingernail-color sensor acts as the best tangential sensor because the RMSE obtained from tangential-force tests was around 1/3 that of previous studies.

Detecting and recognizing objects is one of the most important uses of vision systems in nature and is consequently highly evolved. This paper aims to accurately detect an object using its shape and color information from a complex background. In particular, we evaluated our algorithm to detect 19 different integrated circuits (IC) from 10 different printed circuit boards (PCB) of different colors. We have compared three different shape descriptors for four different color space models. We have evaluated shape detection algorithms in different lighting conditions (indoor, outdoor, and controlled light source) to find suitable illumination for image acquisition. We undertook statistical hypothesis testing to find the effect of color space models and shape descriptors on the accuracy, false positive and false negative rates. While measuring accuracy, we have noted that L*a*b* color space is significantly worse, and the best result is obtained in YCbCr color space using bounding box shape descriptors for 2500 Lux using LED.

This paper presents a novel approach to implement bilateral control loops between local haptic devices and remote industrial manipulators using a layer of simulation and virtual reality. The remote scene of manipulation has been visualized in an open-source software environment, where forward and inverse kinematics of the manipulators can be computed. Therefore, the explicit knowledge of mathematical models of the robots is not required for the implementation of the proposed bilateral control schemes. A haptic coupling has been designed between the human operator and the task in the remote environment. Virtually introduced force feedback has contributed to the performance of the proposed bilateral loop by facilitating the adaptation of unexperienced human operators. Teleoperation of one remote manipulator has been experimentally demonstrated with the proposed controllers. Structural modularity of the bilateral haptic control schemes makes them directly extendable for the teleoperation of multiple collaborative robots. Stability and transparency of the proposed bilateral haptic controllers have been theoretically and experimentally investigated.

This paper treats the systematic injury analysis of lower arm robot–human impacts. For this purpose, a passive mechanical lower arm (PMLA) was developed that mimics the human impact response and is suitable for systematic impact testing and prediction of mild contusions and lacerations. A mathematical model of the passive human lower arm is adopted to the control of the PMLA. Its biofidelity is verified by a number of comparative impact experiments with the PMLA and a human volunteer. The respective dynamic impact responses show very good consistency and support the fact that the developed device may serve as a human substitute in safety analysis for the described conditions. The collision tests were performed with two different robots: the DLR Lightweight Robot III (LWR-III) and the EPSON PS3L industrial robot. The data acquired in the PMLA impact experiments were used to encapsulate the results in a robot independent safety curve, taking into account robot's reflected inertia, velocity and impact geometry. Safety curves define the velocity boundaries on robot motions based on the instantaneous manipulator dynamics and possible human injury due to unforeseen impacts.

Fall-related accidents are among the most serious concerns in elderly people, amputees and subjects with neurological disorders. The aim of this paper was to investigate the behaviour of healthy subjects wearing a novel light-weight pelvis exoskeleton controlled in zero-torque mode while carrying out unperturbed locomotion and managing unexpected perturbations. Results showed that the proposed exoskeleton was unobtrusive and had a minimum loading effect on the human biomechanics during unperturbed locomotion. Conversely, it affected the movement of the trailing leg while subjects managed unexpected slipping-like perturbations. These findings support further investigations on the potential use of powered exoskeletons to assist locomotion and, possibly prevent incipient falls.

It is important to develop a robotic orthosis or exoskeleton that can provide back-drivable and good assistive performances with lightweight structures for overground gait rehabilitation of stroke patients. In this paper, we describe a robotic knee device with a five-bar linkage to allow low-impedance voluntary knee motion within a specified rotation range during the swing phase, and to assist knee extension during the stance phase. The device can provide free motion through the five-bar linkage with 2-degree-of-freedom (DOF) actuation via the patient's shank using a linear actuator, and can assist knee extension at any controlled knee angle while bearing weight via a geared five-bar linkage with 1 DOF actuation of the linear actuator. The kinematic transition between the two modes can be implemented by contact with a circular structure and a linear link, and the resultant range of motion can be determined by the linear actuator. The kinematic weight of the device was optimized using the simple genetic algorithm to reduce the mass. The optimization cost function was based on the sum of the total link lengths and the actuator power. The optimization results reduced the total link length and motor power by 47% and 43%, respectively, compared to the initial design. We expect that the device will facilitate rehabilitation of stroke patients by allowing safe and free overground walking while providing support for stumbling.

This paper presents a virtual headstick system as an alternative to the conventional passive headstick for persons with limited upper extremity function. The system is composed of a pair of kinematically dissimilar master-slave robots with the master robot being operated by the user's head. At the remote site, the end-effector of the slave robot moves as if it were at the tip of an imaginary headstick attached to the user's head. A unique feature of this system is that through force-reflection, the virtual headstick provides the user with proprioceptive information as in a conventional headstick, but with an augmentation of workspace volume and additional mechanical power. This paper describes the test-bed development, system identification, bilateral control implementation, and system performance evaluation.