A predictive display method and man-virtual robot interaction based on augmented reality are applied to control a telerobot. We first discuss the process of the augmented reality environment development. Then, we present the advantages of predictive display. Simulation of virtual robot's tasks in the augmented environment improves the safety of the telerobot when it executes the planned tasks. In addition, the immediate feedback from the virtual robot avoids the exacerbation of maneuverability caused by time-delay. For a more natural operation process, we apply multi man-virtual robot interactive methods. Lastly, the experiment of pick & place is conducted to validate the system.

The Multimodal User Supervised Interface and Intelligent Control (MUSIIC) project focuses on a multimodal human-machine interface which addresses user need to manipulate familiar objects in an unstructured environment. The control of a robot by individuals with significant physical limitations presents a challenging problem of telemanipulation. This is addressed by a unique user-interface integrating the user's command (speech) and gestures (pointing) with autonomous planning techniques (knowledge-bases and 3-D vision). The resultant test-bed offers the opportunity to study telemanipulation by individuals with physical disabilities, and can be generalized to an effective technique for other, including remote and time-delayed, telemanipulation. This paper focuses on the knowledge-driven planning mechanism that is central to the MUSIIC system.