You're using an outdated browser. Please upgrade to a modern browser for the best experience.
Submitted Successfully!
Thank you for your contribution! You can also upload a video entry or images related to this topic. For video creation, please contact our Academic Video Service.
Version Summary Created by Modification Content Size Created at Operation
1
Dimitris Mourtzis
 -- 2141 2022-09-08 18:10:26 |
2 layout
Camila Xu
 Meta information modification 2141 2022-09-09 02:01:07 | |
3 layout
Camila Xu
 + 5 word(s) 2146 2022-09-16 02:48:35 |

Video Upload Options

We provide professional Academic Video Service to translate complex research into visually appealing presentations. Would you like to try it?
No, upload directly Yes

Confirm

Are you sure to Delete?
Yes No
Cite
If you have any further questions, please contact Encyclopedia Editorial Office.
Mourtzis, D.;  Angelopoulos, J.;  Panopoulos, N. Human–Robot Collaboration based on Mixed Reality. Encyclopedia. Available online: https://encyclopedia.pub/entry/27026 (accessed on 17 July 2025).
Mourtzis D,  Angelopoulos J,  Panopoulos N. Human–Robot Collaboration based on Mixed Reality. Encyclopedia. Available at: https://encyclopedia.pub/entry/27026. Accessed July 17, 2025.
Mourtzis, Dimitris, John Angelopoulos, Nikos Panopoulos. "Human–Robot Collaboration based on Mixed Reality" Encyclopedia, https://encyclopedia.pub/entry/27026 (accessed July 17, 2025).
Mourtzis, D.,  Angelopoulos, J., & Panopoulos, N. (2022, September 08). Human–Robot Collaboration based on Mixed Reality. In Encyclopedia. https://encyclopedia.pub/entry/27026
Mourtzis, Dimitris, et al. "Human–Robot Collaboration based on Mixed Reality." Encyclopedia. Web. 08 September, 2022.
Human–Robot Collaboration based on Mixed Reality
Edit
This entry is adapted from the peer-reviewed paper 10.3390/app12062972

Human–Robot Interaction (HRI) poses new challenges to the manufacturing landscape, such as safety, autonomy, and social acceptance, as the demand for collaborative robots or cobots to interact, collaborate, and assist human operators grows. Concretely, since there is also a great development of the Information and Communication Technologies (ICT), both in terms of hardware and software, several other digital technologies, such as Extended Reality (XR), have become more popular in the industrial world. Extended Reality (XR) is an umbrella term, including Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR).

Mixed Reality (MR) robotic manipulation human–robot interface (HRI)

1. Introduction

Modern manufacturing systems working under the Fourth Industrial Revolution (or Industry 4.0) paradigm, are constantly evolving. Industry 4.0 has introduced a wide variety of technologies and techniques for improving both productivity and working conditions in modern manufacturing plants. However, it is of great importance to keep in mind that human resources and, more specifically, shop-floor technicians must retain the engineers’ center of attention [1]. With recent technological advances, the integration of robotic manipulation has been leveraged during the last decades, and the path to the factories of the future leads to the design and development of collaborative environments [2]. Therefore, what is needed is the provision of suitable tools that will enable continuous and flawless communication between human operators and machines [3][4]. Additionally, Human–Robot Interaction (HRI) poses new challenges to the manufacturing landscape, such as safety, autonomy, and social acceptance, as the demand for collaborative robots or cobots [5] to interact, collaborate, and assist human operators grows. Smart manufacturing technologies [6] are gradually displacing jobs that are repetitive, monotonous, and low-skilled. Artificial Intelligence (AI)-based systems have great potential for automating jobs that previously required human intelligence for adaptive decision making [7]. In collaborative manufacturing cells, safety is also a major issue. Robotics and automation are creating new and more skill-demanding job opportunities. This shift has led to the reshaping of manufacturing to make it smarter and safer, not only in terms of production processes, but also in terms of human labor, with new skills and competencies required [8]. Human–Robot Collaboration (HRC) [9] also poses significant challenges, especially in terms of safety [10]. The ability to predict human actions [10][11] and the capability to plan and continuously replan safe robot trajectories based on predicted/observed human actions [11] have been identified as two major challenges in the literature.
As mentioned in the previous paragraph, the current era is characterized by immense technological advances. Concretely, since there is also a great development of the Information and Communication Technologies (ICT), both in terms of hardware and software, several other digital technologies, such as Extended Reality (XR), have become more popular in the industrial world [12]. Extended Reality (XR) is an umbrella term, including Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR) [13]. Under the scope of the current research work, special attention will be given to MR. MR is similar to AR, since it partially immerses the users, i.e., it involves the registration of digital content on their field of view (FoV). However, the main difference with AR is that MR enables the interaction of the user with the digital contents, i.e., the holograms. The capabilities of the above-mentioned technologies are leveraged by the fact that Artificial Intelligence (AI) technologies have become mainstream [14]. As a result of the above-mentioned transition and progress, HRC is one of the outcomes that allows humans and robots to collaborate to achieve common goals. As a result, new HRI methods encourage collaboration, especially in more complex scenarios.
Safety is a critical consideration in the design and implementation of any new technology that aims to work in close collaboration with operators during the age of industrialization and automation [15], particularly as the human-centric industrial revolution or Industry 5.0 approaches [16]. In the research work of Gualtieri et al. [17], safety risks in HRCs are identified mainly in the field of collaborative assembly stations, in which non-intentional contact between humans and robots is one of the main safety risks. Similarly, in [18] the authors have investigated the available literature in an attempt to highlight challenges in HRC implementation. Among the key findings of this research work, safety hazards also contain ergonomics issues. Moreover, collision avoidance and mitigation are among the key topics providing fertile ground for further research. Interestingly, safety assurance in collaborative environments, according to Bi et al. [19], requires (i) integration of recent Industry 4.0 technologies in order to adequately acquire data and (ii) the development of suitable algorithmic approaches for processing these data and constantly adapting system parameters in order to ensure that humans and robots can safely co-exist. According to the OECD (Organization for Economic Co-operation and Development), 14% of jobs in OECD countries are at risk of automation [20], owing to a lack of meaning, increased repetitiveness, or a high risk of injury [21].
Based on the abovementioned challenges, the identified challenges as well as the limitations of the investigated key relevant publications in the field of HRC are summarized in Table 1, below.
Table 1. Identified limitations and challenges in the field of HRC.
A/A Ref. Challenges Limitation
1 [5]
  • Limitation of a robot’s interface design
  • Bottleneck caused by a combination of multimodal control commands for intuitive control of the robot
  • Typical challenge of Human–Robot Collaboration (HRC) assembly: HRC assembly and adaptive responses to commands and correct triggering of the relevant controls
  • Proposed framework is a proof of concept
2 [9]
  • Confusions exist surrounding the relationships between robots and humans: coexistence, interaction, cooperation, and collaboration
  • Lack of standards
  • Lack of safety solutions
  • Low acceptance of the human–robot combination
3 [11]
  • Ability to understand and imitate human behavior will be a key skill to allow collaborative robots to operate and become useful in the human environment
  • Limited investigation of behavior of motion; primitive learning in the presence of multiple demonstrators
  • No investigation of the higher order model for improving the segmentation results and for predicting human motion
  • Application and validation of the proposed approach with other types of human motion, including task and goal-based motion (e.g., interaction with the environment)
4 [15]
  • Safety is not always explicitly mentioned as an application in research works
  • Safety in Human–Robot Interaction remains an open problem
  • Novel, robust, and generalizable safety methods are required in order to enable safe incorporation of robots into homes, offices, factories, or any other setting
  • Perception view: Active vision mechanisms should be incorporated into robots
  • Cognition view: Incorporation of Machine Learning techniques into the action robotic skills
  • Incorporation of probabilistic learning into task planning and decision making
5 [17]
  • Safety: unwanted and unexpected contacts between human and robotic systems may cause injuries and therefore limit the potential for collaboration
  • There is a lack of simple and practical tools for helping system designers to overcome such limiting conditions
  • The proposed validation of the Collaborative Assembly System process is based on a virtual model
  • The validation was based on only one test case which involved three manufacturing engineers
  • This work did not analyze the hierarchical relationships between the various guidelines, as well as possible inconsistencies in their implementation
6 [18]
  • Occupational health and safety criteria are of crucial importance in the implementation of collaborative robotics
  • Collaborative robotics could be helpful for small and medium-sized enterprises (SMEs) As such, future reviews could include the term ‘SMEs’ as a search keyword
  • Contact avoidance research should be improved
  • Contact detection and mitigation should be improved
  • Physical Ergonomics
  • Cognitive Ergonomics
7 [19]
  • Acquiring, processing, and fusing diversified data for risk classification
  • Update the control to avoid any interference in a real-time mode
  • Developing technologies to improve HMI performance
  • Reducing the overall cost of safety assurance features
  • Develop standards in expressing the safety features of a functional module
  • Define the technical implementations to enforce corresponding guidelines and regulations
  • Classify and specify the methods of recognition for hazard scenarios
 System Programming and Control
•
Intuitive programming
•
Task-driven programming
•
Skill-based programming
◦
Risk management
◦
Evaluation of biomechanical loads
◦
Real-time estimation of stopping distances
Sensing Systems
  • New instrumentations and algorithms for effective sensing, processing, and fusing of diverse data
  • Machine learning for high-level complexity and uncertainty
8 [22]
  • Physical human–robot contacts are not allowed during the actual polishing task
  • An innovative coexistence modality and human–robot communication with gestural commands were demonstrated for the collaborative phases of setup operations of the cell/tools and of quality assessment of the workpiece
  • The investigated case study-cell is still a research project and not all safety functions have achieved the performance requirements of industrial robot safety standards

2. Advances of Human–Robot Collaboration based on Mixed Reality

Robotics, automation, and AI have gained a rapidly growing position in the workplace, faster than many organizations had ever expected the introduction of automation to be [23]. Although companies are gradually using these technologies in order to automate internal processes, true pioneers are fundamentally rethinking the work environment to optimize the value of both humans and machines by creating new opportunities to coordinate work more efficiently and to redefine the skills and professions of human staff [24]. Due to the fact that even more organizations are rushing to adopt these technologies, the market for AI tools and robotics is blooming. Leading companies, such as Microsoft, IBM, Facebook, and other technology giants, are investing heavily in this field. CEOs are becoming increasingly aware that these systems are most successful when they complement, instead of replacing, human operators [25]. Research suggests that while automation is capable of improving scale, speed, and quality, it does not do away with jobs. It might actually do just the opposite [2].
Human–Robot Collaboration (HRC) aims at creating work environments in the manufacturing context where human operators can work side by side in close proximity with robots. In such configurations, the main goal is to achieve efficient and high-quality manufacturing processes. In the literature, several recent works have demonstrated such implementations of HRC systems in real industrial manufacturing tasks, taking into consideration both human safety and communication. The authors of [26] proposed an AR-based wearable interface integrated into an off-the-shelf safety system. This wearable AR assists the assembly line operator by providing visual guidance on how to execute the current task in the form of textual details or parts representation in a 3D model. This research work has been applied in an automotive assembly task. Next, the author of [27] used a standardized and control and communication architecture in conjunction with fused sensor data in order to ensure safety robot control. Apart from the safety aspect, one of the key challenges of industrial HRC is the interaction and coordination between human and robot resources, as presented in [28]. More similar to this research work, a context-aware MR approach was used in car door assembly and tested against two standard methods, i.e., printed and screen display instructions [29]. In addition, the authors of [30] focused on enabling human operators to communicate with mobile dual arm robots, namely, Mobile Robot Platforms (MRPs), via an AR-based software suite. The novelties of the systems proposed lie in the end-to-end (E2E) integration of the human side interface AR-based framework, with mobile robot controllers exploiting the Digital Twin capabilities of the production entities [31].
Moving on, a recent study presenting the problems of Human–Robot Interaction (HRI) [21] suggests that AR interfaces can enhance the process of interaction by manipulating robots. Moreover, MR has been used in order to embed the user in a virtual environment deeper than AR. Furthermore, a similar study in [32] proposed an intuitive robot programming based on MR. A methodology to plan the geometric path, including orientation, has been developed. Shared autonomy systems enhance the ability of people to carry out everyday life tasks using robotic manipulators. The authors of [33] describe a robotic cell that manipulates, assembles, and packages geometrically complex products using cognitive control and actuation systems. Individual mechatronic components, such as a 6 DoF (Degrees of Freedom) gripper and a flexible assembly mechanism, were designed by decomposing the actual assembly and handling tasks into functional components. Additionally, a problem for users that cannot change their point of view has been addressed in [33] with the introduction of the InvisibleRobot, which is a diminished reality-based approach that overlays the background information onto the robot in the FoV of the user, through an Optical See-Through Head-Mounted Display. The authors of [34] developed an AR system allowing for safer online programming of industrial robots. Lastly, [35] presented the results of a project to develop an AR-based HRC system to improve safety when working with robots, the solutions consisting of volumes of safe working zones and audio and visual instructions to indicate danger. Furthermore, the levels of collaboration between an operator and a robot are classified in [36] as (1) Coexistence, (2) Cooperation, and (3) Collaboration, these being the three pillars of coexistence. As a result, as defined in ISO/TS 15,066 [37], different levels of collaboration necessitate different safety actions and measures.
Therefore, following the literature investigation, there is only a limited number of similar studies proposing a method for near real-time wireless robot manipulation with MR capabilities. Additional user-experience-enhancing features, such as safety zones, robot reachability zones, and so on, are also supported.

References

  1. Barrett, T.R.; Llion, M.E.; Mike, F.; Fred, D.; Simon, C.M.; Daniel, I.; Elizabeth, S. Virtual Engineering of a Fusion Reactor: Application to Divertor Design, Manufacture, and Testing. IEEE Trans. Plasma Sci. 2018, 47, 889–896.
  2. Mourtzis, D.; Siatras, V.; Angelopoulos, J.; Panopoulos, N. An Augmented Reality Collaborative Product Design Cloud-Based Platform in the Context of Learning Factory. Procedia Manuf. 2020, 45, 546–551.
  3. Mourtzis, D.; Angelopoulos, J.; Dimitrakopoulos, G. Design and development of a flexible manufacturing cell in the concept of learning factory paradigm for the education of generation 4.0 engineers. Procedia Manuf. 2020, 45, 361–366.
  4. Hermann, M.; Tobias, P.; Boris, O. Design Principles for Industrie 4.0 Scenarios. In Proceedings of the 2016 49th Hawaii International Conference on System Sciences (HICSS), Koloa, HI, USA, 5–8 January 2016; pp. 3928–3937.
  5. Liu, S.; Wang, L.; Wang, X.V. Symbiotic human-robot collaboration: Multimodal control using function blocks. Procedia CIRP 2020, 93, 1188–1193.
  6. Mourtzis, D. Simulation in the design and operation of manufacturing systems: State of the art and new trends. Int. J. Prod. Res. 2020, 58, 1927–1949.
  7. Michalos, G.; Makris, S.; Papakostas, N.; Mourtzis, D.; Chryssolouris, G. Automotive assembly technologies review: Challenges and outlook for a flexible and adaptive approach. CIRP J. Manuf. Sci. Technol. 2010, 2, 81–91.
  8. Whiting, K. These Are the Top 10 Job Skills of Tomorrow—And How Long It Takes to Learn Them; World Economic Forum: Geneva, Switzerland, 2020; Volume 21.
  9. Wang, L.; Gao, R.; Váncza, J.; Krüger, J.; Wang, X.; Makris, S.; Chryssolouris, G. Symbiotic human-robot collaborative assembly. CIRP Ann. 2019, 68, 701–726.
  10. Takata, S.; Hirano, T. Human and robot allocation method for hybrid assembly systems. CIRP Ann. 2011, 60, 9–12.
  11. Krüger, J.; Lien, T.; Verl, A. Cooperation of human and machines in assembly lines. CIRP Ann. 2009, 58, 628–646.
  12. Kulic, D.; Nakamura, Y. Incremental Learning of Human Behaviors Using Hierarchical Hidden Markov Models. In Proceedings of the 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, Taipei, Taiwan, 18–22 October 2010; pp. 4649–4655.
  13. ElMaraghy, H.; Monostori, L.; Schuh, G.; ElMaraghy, W. Evolution and future of manufacturing systems. CIRP Ann. 2021, 70, 635–658.
  14. Demir, K.A.; Cicibaş., H. The Next Industrial Revolution: Industry 5.0 and Discussions on Industry 4.0. In Industry 4.0 from the Management Information Systems Perspectives; Peter Lang Publishing House: New York, NY, USA, 2019.
  15. De Nul, L.; Maija, B.; Petridis, A. Industry 5.0: Towards a Sustainable, Human-Centric and Resilient European Industry; Publications Office of the European Union: Luxembourg, 2021.
  16. Zacharaki, A.; Kostavelis, I.; Gasteratos, A.; Dokas, I. Safety bounds in human robot interaction: A survey. Saf. Sci. 2020, 127, 104667.
  17. Wang, L. A futuristic perspective on human-centric assembly. J. Manuf. Syst. 2021, 62, 199–201.
  18. Gualtieri, L.; Rauch, E.; Vidoni, R. Development and validation of guidelines for safety in human-robot collaborative assembly systems. Comput. Ind. Eng. 2021, 163, 107801.
  19. Gualtieri, L.; Rauch, E.; Vidoni, R. Emerging research fields in safety and ergonomics in industrial collaborative robotics: A systematic literature review. Robot. Comput. Integr. Manuf. 2021, 67, 101998.
  20. Bi, Z.; Luo, M.; Miao, Z.; Zhang, B.; Zhang, W.; Wang, L. Safety assurance mechanisms of collaborative robotic systems in manufacturing. Robot. Comput. Manuf. 2021, 67, 102022.
  21. Smids, J.; Nyholm, S.; Berkers, H. Robots in the Workplace: A Threat to—Or opportunity for—Meaningful Work? Philos. Technol. 2020, 33, 503–522.
  22. Tsarouchi, P.; Makris, S.; Chryssolouris, G. Human-robot interaction review and challenges on task planning and programming. Int. J. Comput. Integr. Manuf. 2016, 29, 916–931.
  23. Georgieff, A.; Milanez, A. What Happened to Jobs at High Risk of Automation? 2021. Available online: https://www.oecd.org/future-of-work/reports-and-data/what-happened-to-jobs-at-high-risk-of-automation-2021.pdf (accessed on 13 February 2022).
  24. Mourtzis, D.; Vlachou, A.; Zogopoulos, V. Cloud-based augmented reality remote maintenance through shop-floor monitoring: A product-service system approach. J. Manuf. Sci. Eng. 2017, 39, 061011.
  25. Rishab, A. 2021 Digital Transformation Assessment. The Manufacturer and IBM Report 2021. Available online: https://www.ibm.com/downloads/cas/MPQGMEN9 (accessed on 15 February 2022).
  26. Berglund, F.Å.; Gong, L.; Li, D. Testing and validating Extended Reality (xR) technologies in manufacturing. In Proceedings of the 8th Swedish Production Symposium (SPS 2018), Stokholm, Sweden, 16–18 May 2018; pp. 31–38.
  27. Bonci, A.; Cen Cheng, P.D.; Indri, M.; Nabissi, G.; Sibona, F. Human-Robot Perception in Industrial Environments: A Survey. Sensors 2021, 21, 1571.
  28. Nahavandi, S. Industry 5.0—A human-centric solution. Sustainability 2019, 11, 4371.
  29. Chryssolouris, G. Manufacturing: Systems Theory and Practice, 2nd ed.; Springer: New York, NY, USA, 2006.
  30. Agarwal, D.; Bersin, J.; Lahiri, G.; Schwartz, J.; Volini, E. AI, Robotics, and Automation: Put Humans in the Loop. Global Human Capital Trends. Deloitte Insights 2018. Available online: https://www2.deloitte.com/us/en/insights/focus/human-capital-trends/2018/ai-robotics-intelligent-machines.html (accessed on 15 February 2022).
  31. Bessen, E.J. How computer automation affects occupations: Technology, jobs, and skills. Boston Univ. Sch. Law Law Econ. Res. Pap. 2016, 15–49.
  32. Karagiannis, P.; Michalos, G.; Andronas, D.; Matthaiakis, A.-S.; Giannoulis, C.; Makris, S. Cognitive Mechatronic Devices for Re-configurable Production of Complex Parts. Appl. Sci. 2021, 11, 5034.
  33. Magrini, E.; Ferraguti, F.; Ronga, A.J.; Pini, F.; De Luca, A.; Leali, F. Human-robot coexistence and interaction in open industrial cells. Rob. Comput.-Integr. Manuf. (RCIM) 2020, 61, 120–143.
  34. Ganesan, R.K.; Rathore, Y.K.; Ross, H.M.; Amor, H.B. Better teaming through visual cues: How projecting imagery in a workspace can improve human-robot collaboration. IEEE Rob. Autom. Mag. 2018, 25, 59–71.
  35. Alpaslan, D.K.; Caymaz, E.; Elci, M. Issues in Integrating Robots into Organizations. In Proceedings of the 12th International Scientific Conference on Defense Resources Management in the 21st Century, Brașov, Romania, 9–10 November 2017.
  36. Aaltonen, I.; Salmi, T.; Marstio, I. Refining levels of collaboration to support the design and evaluation of human-robot interaction in the manufacturing industry. Procedia CIRP 2018, 72, 93–98.
  37. ISO/TS 15066:2016; Robots and Robotic Devices—Collaborative Robots. International Organization for Standardization: Geneva, Switzerland, 2016.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.
Upload a video for this entry
Information
Subjects: Engineering, Manufacturing
Contributors MDPI registered users' name will be linked to their SciProfiles pages. To register with us, please refer to https://encyclopedia.pub/register :
Dimitris Mourtzis
,
John Angelopoulos
,
Nikos Panopoulos
View Times: 709
Revisions: 3 times (View History)
Update Date: 16 Sep 2022
Table of Contents
    1000/1000
    Hot Most Recent
    Academic Video Service
    Feedback