Designing natural interaction for camera viewpoint control in teleoperation

Abstract

Teleoperation can be used to remove human beings or provide replaceable surrogates in hazardous or difficult working environments. Multi-tasking situations are common in teleoperation. For many teleoperation activities, operators usually have multiple devices or equipment to control via conventional hand operated interfaces, such as joysticks, switches, wheels, mice and keyboards. A typical example of such a situation is controlling a robot for either navigation or manipulation as the primary task and controlling a remote camera as the major visual feedback to obtain situational awareness from the remote environment, all by manual interfaces. Apart from switching hands frequently between different control interfaces, the working attention of the operator will also be distracted from the primary task under such circumstances, which results in increased workload and reducing performance.

This thesis is motivated by a real-world industrial setting in mining teleoperation: Remote rock breaking, in which operators have to face such a multi-tasking situation. Instead of using conventional control interfaces, we explore several possible solutions by integrating natural human interactive information to design and develop new remote camera viewpoint control models. Natural interaction based interfaces allow people to interact with technologies, applications and systems as they are used to when they interact with other people in everyday life through movements, gestures, expressions, discovering the world by looking around and manipulating physical objects. In this research, we particularly investigate the use of natural head movements, eye gaze, weight-based body gestures and foot movements with prototype systems as alternative inputs for the camera viewpoint control in teleoperation.

We present a number of user studies which compare these camera viewpoint control models with conventional control interfaces. Several novel lab-based experimental models have been designed using the concept of a Functional Physical Model, which is used to simulate the real-world multi-tasking setting to deal with the difficulty of direct access to real industry settings. Both objective and subjective measures were used in all user studies conducted.

From the results, we demonstrate the remote viewpoint control models we developed are either comparable to or an improvement over conventional control interfaces used in the experiments. Particularly for the empirical user study comparing our Natural Interaction Model (combining eye gaze and head motion) with optimised manual control and autonomous tracking models, the results indicate the advantages of using our natural interaction model, while the manual control model performed the worst.

Existing issues, discovered problems and challenges of using natural interaction for remote camera viewpoint control are also discussed. Finally, suggestions are made for further research in the area of importing more natural interaction based interfaces for teleoperation. Show more