Virtual Reality and Robots: History
Please note this is an old version of this entry, which may differ significantly from the current revision.
Subjects: Others
Contributor:

There is a significant amount of synergy between virtual reality (VR) and the field of robotics. However, it has only been in approximately the past five years that commercial immersive VR devices have been available to developers. This new availability has led to a rapid increase in research using VR devices in the field of robotics, especially in the development of VR interfaces for operating robots. 

  • virtual reality
  • robotics
  • interfaces
  • human robot interaction

1. Introduction

Even though the concept of virtual reality (VR) has been around since the 1960s [1], its usage in the field of robotics has only been widely adopted in robot-assisted surgery[2][3][4] and robot-assisted rehabilitation [5][6][7]. However, there is a great deal of synergy between VR and robotics[8]. Robots can be used in VR to help provide more immersive VR experiences by providing haptic feedback to users[9][10] or by providing items for user interaction[11][12]. On the flip side, VR is an enabling technology in robotics to provide immersive robot teleoperation[13], to aid in robot programming[14], to conduct human-robot interaction and collaboration studies[15][16][17], and even to train individuals on how to collaborate with robots[18].

The lack of VR integration in the field of robotics has been largely caused by the absence of availability and affordability of commercial VR devices. In the past decade though, there has been considerable advancement to make VR devices with immersive visualization and 6 degree-of-freedom (DOF) tracking commercially available and relatively affordable. With these advancements, there has been an increase in utilizing VR in the field of robotics. One area that has seen particular growth is in utilizing VR devices for human-robot interaction and collaboration.

VR provides an opportunity to create more natural and intuitive interfaces by immersing users in a 3D environment where they can view and interact with robots in 3D shared or remote environments. This can allow for better situational awareness and easier interaction. Robot interfaces can be of two types, either teleoperation, where an operator controls a robot’s end-effector, or shared-control, where an operator provides high-level commands to the robot. Both interface types allow operators to control a robot from remote locations. This allows for robots to be used in dangerous, distant and daring jobs and removes risk from the operator but keeps their knowledge and expertise in the loop. By using VR these interfaces can be used in a wider application due to the immersive nature of VR.

2. Current State of VR Robot Interfaces

VR interfaces for operating robots have come a long way in the past four years. VR devices becoming commercially available and affordable for many researchers has helped tremendously in furthering the state-of-the-art. There is also a growing VR community to help support development, which lowers the barrier of entry for new developers. This is particularly relevant in the area of system architecture and infrastructure for VR robot interfaces, as there are now several open-source solutions in connecting ROS and Unity making it much easier to get started with VR robot interface development. Additionally, there have been several works that show the promise of VR interfaces over traditional ones, such as 2D computer interfaces. Overall, it has been shown that VR interfaces reduce task completion time, increase operator performance, and are generally preferred over traditional interfaces. These works help support the need for continue development in VR interfaces for robot operation.

Although, there has been a significant amount of foundational work in VR robot interfaces, there are still several areas that require further development. For example, there has been a significant amount of work in creating VR teleoperation interfaces, even though, teleoperation may not always be a viable option and a shared-control interface may be more appropriate solution, especially for complex systems. However, there has been limited research in shared-control VR interfaces despite the fact that these interfaces bring the advantage of allowing the robot to act semi-autonomously, which in turn allows the user to only provide input when needed and focus on the critical elements of a task. In addition, most VR interfaces so far have been designed for robot manipulators, aerial robots, or mobile robots. At the moment, there is very limited work in using bipedal humanoid robots. Humanoids are general purpose platforms though and can easily be used in diverse environments designed for humans. However, they are complex dynamic systems and can immensely benefit from the immersive and 3D interaction environment that VR devices provide.

Furthermore, as VR interface development continues, it is important to continually evaluate the usability and likability of these new advancements among users. It is also important to evaluate these interfaces in real-world applications with actual potential users of the systems.

This entry is adapted from the peer-reviewed paper 10.3390/app10249051

References

  1. • Mazuryk, T.; Gervautz, M. Virtual Reality-History, Applications, Technology and Future; Vienna University of Technology: Vienna, Austria, 1996.
  2. • Bric, J.D.; Lumbard, D.C.; Frelich, M.J.; Gould, J.C. Current state of virtual reality simulation in robotic surgery training: A review. Surg. Endosc. 2016, 30, 2169–2178.
  3. • Moglia, A.; Ferrari, V.; Morelli, L.; Ferrari, M.; Mosca, F.; Cuschieri, A. A systematic review of virtual reality simulators for robot-assisted surgery. Eur. Urol. 2016, 69, 1065–1080.
  4. • Van der Meijden, O.A.; Schijven, M.P. The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: A current review. Surg. Endosc. 2009, 23, 1180–1190.
  5. • Adamovich, S.V.; Fluet, G.G.; Tunik, E.; Merians, A.S. Sensorimotor training in virtual reality: A review. Neuro Rehabil. 2009, 25, 29–44.
  6. • Baur, K.; Schättin, A.; de Bruin, E.D.; Riener, R.; Duarte, J.E.; Wolf, P. Trends in robot-assisted and virtual reality-assisted neuromuscular therapy: A systematic review of health-related multiplayer games. J. Neuroeng. Rehabil. 2018, 15, 107.
  7. • Howard, M.C. A meta-analysis and systematic literature review of virtual reality rehabilitation programs. Comput. Hum. Behav. 2017, 70, 317–327.
  8. • Burdea, G.C. Invited review: The synergy between virtual reality and robotics. IEEE Trans. Robot. Autom. 1999, 15, 400–410.
  9. • Al-Sada, M.; Jiang, K.; Ranade, S.; Kalkattawi, M.; Nakajima, T. HapticSnakes: Multi-haptic feedback wearable robots for immersive virtual reality. Virtual Real. 2020, 24, 191–209.
  10. • Vonach, E.; Gatterer, C.; Kaufmann, H. VRRobot: Robot actuated props in an infinite virtual environment. In Proceedings of the 2017 IEEE Virtual Reality (VR), Los Angeles, CA, USA, 18–22 March 2017; pp. 74–83.
  11. • Zhao, Y.; Kim, L.H.; Wang, Y.; Le Goc, M.; Follmer, S. Robotic Assembly of Haptic Proxy Objects for Tangible Interaction and Virtual Reality. In Proceedings of the Interactive Surfaces and Spaces on ZZZ—ISS ’17, Brighton, UK, 17–20 October 2017; pp. 82–91.
  12. • Suzuki, R.; Hedayati, H.; Zheng, C.; Bohn, J.L.; Szafir, D.; Do, E.Y.L.; Gross, M.D.; Leithinger, D. RoomShift: Room-Scale Dynamic Haptics for VR with Furniture-Moving Swarm Robots. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, Honolulu, HI, USA, 25–30 April 2020; pp. 1–11.
  13. • Elobaid, M.; Hu, Y.; Romualdi, G.; Dafarra, S.; Babic, J.; Pucci, D. Telexistence and Teleoperation for Walking Humanoid Robots. In Intelligent Systems and Applications; Bi, Y., Bhatia, R., Kapoor, S., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 1106–1121.
  14. • Bolano, G.; Roennau, A.; Dillmann, R.; Groz, A. Virtual Reality for Offline Programming of Robotic Applications with Online Teaching Methods. In Proceedings of the 2020 17th International Conference on Ubiquitous Robots (UR), Kyoto, Japan, 22–26 June 2020; pp. 625–630.
  15. • Liu, O.; Rakita, D.; Mutlu, B.; Gleicher, M. Understanding human-robot interaction in virtual reality. In Proceedings of the 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Lisbon, Portugal, 28 August–1 September 2017; pp. 751–757.
  16. • Villani, V.; Capelli, B.; Sabattini, L. Use of Virtual Reality for the Evaluation of Human-Robot Interaction Systems in Complex Scenarios. In Proceedings of the 2018 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Nanjing, China, 27–31 August 2018; pp. 422–427.
  17. • Wijnen, L.; Lemaignan, S.; Bremner, P. Towards using Virtual Reality for Replicating HRI Studies. In Proceedings of the Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’20, Cambridge, UK, 23–26 March 2020; pp. 514–516.
  18. • Matsas, E.; Vosniakos, G.C. Design of a virtual reality training system for human–robot collaboration in manufacturing tasks. Int. J. Interact. Des. Manuf. (IJIDeM) 2017, 11, 139–153.
More
This entry is offline, you can click here to edit this entry!