Action Recognition for Human–Robot Teaming

Version	Summary	Created by	Modification	Content Size	Created at	Operation
1		Shakra Mehak	--	1919	2024-01-30 08:58:08	\|
2	format correct	Mona Zou	-11 word(s)	1908	2024-02-01 08:13:03	\|

This entry is adapted from the peer-reviewed paper 10.3390/machines12010045

Human–robot teaming (HrT) is being adopted in an increasing range of industries and work environments. Effective HrT relies on the success of complex and dynamic human–robot interaction. Although it may be optimal for robots to possess all the social and emotional skills to function as productive team members, certain cognitive capabilities can enable them to develop attitude-based competencies for optimizing teams. Despite the extensive research into the human–human team structure, the domain of HrT research remains relatively limited. In this sense, incorporating established human–human teaming (HhT) elements may prove practical.

human–robot interaction human–robot collaboration teaming AI human–robot teaming ML human factors

1. Human–Robot Teaming: A System Modeling Perspective

The concept of HrT depends on merging the capabilities of humans and robots. From a system modeling perspective, a human–robot team can be regarded as a system comprising two subsystems: a human subsystem and a robot subsystem. Each subsystem has its own characteristics and capabilities, and the interaction between these two subsystems determines the overall performance of the system. In order to model the human–robot team as a system, it is necessary to understand the capabilities and limitations of each subsystem, as well as how they can interact and cooperate to achieve a common goal.

Humans possess intrinsic flexibility, cognition, and problem-solving skills ^[1], whereas robots offer high accuracy, speed, and repeatability ^[2]. As the ability of robots to act intelligently and the potential for them to be installed without cages increases, manufacturers have developed guidelines and design criteria such as autonomy and mechanical design ^[3]^[4]^[5]^[6]. The purpose of these guidelines and design criteria is to ensure the safety and reliability of the system. However, HrT brings unique challenges to these guidelines. Due to cognitive/computational and physiological differences between robots and humans ^[7], robots need to be programmed with the ability to predict and comprehend the tasks/intentions of human teammates.Humans acquire the ability to predict the behaviour of other humans over time ^[8], but robots must be explicitly trained on how to do this. In addition, when confronted with the volatility of real-world applications, robots cannot rely on a human teammate to stick to the defined task ^[9], nor can they consistently predict how their human partner would act when something goes wrong. The potential solution to this problem is to equip the robot with explicit models of their human teammates. Ideally, these models should learn the generalized features of human behavior without requiring individuals to act in a certain manner. To this end, machine learning (ML) offers emerging techniques for constructing cognitive models and behavioral blocks ^[10], providing a vital perspective for human–robot collaborative activities. By modeling the human–robot team from a system perspective, it is possible to gain a better understanding of how the team functions as a whole and identify potential improvements or enhancements to the system.

The human–robot interaction (HRI) field has advanced, and the need for applications requiring humans and robots to form a team and collaborate to solve complicated issues has increased ^[11]^[12]. Therefore, addressing the teaming configurations and elements and showing the path to incorporate them using advanced algorithms is necessary. However, determining the optimal teaming configuration for HrT can be complicated, particularly due to the unpredictable nature of HRI, as well as the diverse range of tasks and environments that human–robot teams may encounter.

2. Trust in Human–Robot Teaming

Trust plays a vital part in collaborative efforts, particularly in the context of HrT. In teams where tasks are related and mutually dependent, the efficacy relies primarily on the trust between the human and robot team members. This mutual trust is a cornerstone of teamwork and is important for the success of HrT ^[13].

A human’s trust in a robot involves a belief in the robot’s proficiency ^[14]; as demonstrated by the robot’s capacity to understand and conform to human preferences and the ability to make a valuable contribution to a common objective. In exploring the factors that influence trust in HRI, Hancock et al. ^[15] highlighted that various robot characteristics and performance-based factors are particularly crucial. Their research suggests that trust in HRI is most significantly influenced by aspects related to the robot’s ability and performance. Therefore, manipulating and improving these performance aspects can have a substantial impact on the level of trust established between humans and robots.

In order for robots to establish trust and work effectively alongside human teammates, the researchers argue that gaining an understanding of human behavior would indeed enhance the robot’s adaptability and behavior, which are key performance factors. This would involve analyzing contextual patterns and predicting human task performance based on this analysis. By doing so, robots can be better equipped to anticipate the needs and actions of their human counterparts, leading to trust building and, ultimately, effective teaming. The study by Hancock et al. ^[15] stated that higher trust is associated with higher reliability. Therefore, the implication is that if a capacity for performance monitoring in a robot is to be developed, it also needs to be highly reliable; otherwise, it could potentially have a detrimental effect on the overall HRI trust dynamic. Hancock et al. included team collaboration characteristics and tasking factors as relevant factors of the HRI trust dynamic. However, further specification of the team- and task-related effects could not be provided because of an insufficient number of empirically codable studies. To gain more insights, additional research experiments are required in this field.

Overall, trust in HrT is a dynamic element that should facilitate smooth interactions, effective task execution, and adaptive coordination between human and robot team members. The dataset the researchers used for this preliminary analysis did not contain data regarding trust and performance from the perspective of the human operator, as it was collected solely for providing action recognition capabilities.

3. Translating Human–Human Teaming Characteristics into a Human–Robot Teaming Setting

Teamwork is a collection of each team member’s interlinked thoughts, behaviors, and emotions required for the team to function ^[16]. Teammates offer emotional support; individuals usually feel more assured when communicating with others who share a similar experience. Moreover, humans enjoy the sense of belonging that comes with being a team member, especially in a well-functioning team ^[17].

The main elements of teamwork include coordinating teammates’ tasks, anticipating needs based on knowledge of assigned roles, adaptability, team orientation, backup behaviors, closed communication, and mutual performance monitoring. The aforementioned teaming elements are extracted from the Big Five teamwork model, a theoretical foundation for implementing team learning theory in HhT structures ^[18].

Despite the fact that many researchers have discussed teaming features in the context of HhT, relatively few studies ^[9]^[19]^[20]^[21]^[22]^[23]^[24]^[25] have underlined their importance in HrT. Incorporating HhT features into a HrT framework may increase the team’s efficacy and efficiency. By applying these elements, the team members may be better able to understand each other’s strengths and weaknesses, communicate more effectively, and work together toward shared goals and objectives. Moreover, the concept may improve overall productivity and reduce the risk of errors or miscommunication. These teaming components are seen as universally applicable to collaborative processes ^[18]. It is imperative to adapt these elements to optimize the efficacy of HrT. To this end, the researchers have reinterpreted these HhT elements in the context of HrT and identified current methods that hold promise for prospective implementation while also highlighting the distinct challenges associated with these approaches:

Team leadership roles can be dynamically allocated to robots or humans depending on the task. For instance, a human can lead a task requiring creativity and decision making, and a robot with advanced data processing capabilities can lead a task requiring large data analysis. In this regard, multi-agent systems ^[26] can be employed to assign roles. However, the challenge lies in human safety, and humans may resist being led by a robot.

Team orientation is important to align the objectives of both human and robot team members. The field of social robotics can be instrumental in this regard. Utilizing Natural Language Processing (NLP) models and social signal processing can equip robots with social intelligence ^[27]. However, the implementation of NLP models requires substantial data and model training.

Backup behaviors involve team members supporting each other when required. In HrT, robots can be programmed to support human team members in tasks that are hazardous or difficult for humans. Scheduling algorithms ^[28] can be used to optimize backup behaviors in HrT. Additionally, multi-agent systems ^[26] can be used to develop a backup behavior framework. Developing algorithms that can assess the needs of human team members and switch tasks accordingly requires detailed design and testing to ensure functional team collaboration.

Adaptability is a key aspect that entails the ability of a team to adjust to environmental or task-related transformations. To expedite adaptation among team members, shared mental models are employed to modify task requirements. Advanced ML methods, such as reinforcement learning ^[29] and online optimization algorithms ^[30], can enable robots to adapt to environmental changes. However, the challenge lies in ensuring that the adaption is timely, contextually appropriate, and aligns with the preferences of the human team members.

Mutual performance monitoring involves understanding the performance of team members to ensure that the team is working toward the goals. This requires a comprehensive understanding of each teammate’s actions or tasks. Sensors and real-time data analytics can be employed to monitor teammates’ actions. Robots can be equipped to interpret teammates’ actions by integrating cognitive abilities using ML algorithms ^[31]. The development of reliable algorithms demands a significant amount of training data and involves feature selection, model selection, and hyperparameter tuning to optimize the performance of these algorithms.

4. The Need for Mutual Performance Monitoring in Human–Robot Teaming

Fundamental to the development of HrT is the question of how well robots can engage in implicit collaboration, which is the process of synchronizing with team members’ actions based on understanding what each teammate is most likely to do or is doing. It has always been challenging for team researchers to specify cognitive skills or attributes that enable teams to engage, adapt, and synchronize task-relevant information ^[32]. However, researchers have discovered that MPM is a practical teaming component that contributes to successful teaming ^[17]. MPM is the capacity to monitor the tasks of other teammates while executing one’s own tasks ^[18]. It is an important aspect of teaming that can be characterized in the context of HrT as the reciprocal understanding and monitoring of team actions, progress, and outcomes. Here, the researchers consider the innate human ability to understand and interpret teammates’ actions and apply this concept to the team relations between humans and robots. However, it is important to note that although robots may lack the natural capabilities humans possess, they have the potential to acquire them through artificial intelligence (AI) and programming. Through MPM, team members can gain valuable insights into their intentions and task-related challenges, enabling effective communication and shared decision making for successful task accomplishment. Although humans possess the innate ability to interpret their teammates’ actions or intentions ^[11], not all social skills may be necessary for robots. However, certain task-related cognitive abilities can empower robots to function as effective team members in collaborative settings. Further, teaming abilities involve robots’ responsiveness to adaptable team dynamics and individual preferences. Robots should be equipped with the ability to adjust their behavior, communication style, and task allocation to their human teammates’ preferences and work styles ^[9]. In this regard, the modeling of MPM can be a step forward in achieving adaptability, allowing for enhanced teamwork and collaboration.

References

Devlin, S.P.; Flynn, J.R.; Riggs, S.L. Connecting the big five taxonomies: Understanding how individual traits contribute to team adaptability under workload transitions. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Baltimore, MD, USA, 3–8 October 2021; SAGE Publications Sage CA: Los Angeles, CA, USA, 2018; Volume 62, pp. 119–123.
Wolf, F.D.; Stock-Homburg, R. Human-robot teams: A review. In Proceedings of the International Conference on Social Robotics, Golden, CO, USA, 14–18 November 2020; pp. 246–258.
Martinetti, A.; Chemweno, P.K.; Nizamis, K.; Fosch-Villaronga, E. Redefining safety in light of human-robot interaction: A critical review of current standards and regulations. Front. Chem. Eng. 2021, 3, 32.
Tuncer, S.; Licoppe, C.; Luff, P.; Heath, C. Recipient design in human–robot interaction: The emergent assessment of a robot’s competence. AI Soc. 2023, 1–16.
Mutlu, B.; Forlizzi, J. Robots in organizations: The role of workflow, social, and environmental factors in human-robot interaction. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction, Amsterdam, The Netherlands, 12–15 March 2008; pp. 287–294.
Harper, C.; Virk, G. Towards the development of international safety standards for human robot interaction. Int. J. Soc. Robot. 2010, 2, 229–234.
Hoffman, G.; Breazeal, C. Collaboration in human-robot teams. In Proceedings of the AIAA 1st Intelligent Systems Technical Conference, Chicago, IL, USA, 20–22 September 2004; p. 6434.
De Visser, E.; Parasuraman, R. Adaptive aiding of human-robot teaming: Effects of imperfect automation on performance, trust, and workload. J. Cogn. Eng. Decis. Mak. 2011, 5, 209–231.
Gombolay, M.C.; Huang, C.; Shah, J. Coordination of human-robot teaming with human task preferences. In Proceedings of the 2015 AAAI Fall Symposium Series, Arlington, VA, SUA, 12–14 November 2015.
Tabrez, A.; Luebbers, M.B.; Hayes, B. A survey of mental modeling techniques in human–robot teaming. Curr. Robot. Rep. 2020, 1, 259–267.
Zhang, Q.; Lee, M.L.; Carter, S. You complete me: Human-ai teams and complementary expertise. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 30 April–5 May 2022; pp. 1–28.
Webber, S.S.; Detjen, J.; MacLean, T.L.; Thomas, D. Team challenges: Is artificial intelligence the solution? Bus. Horizons 2019, 62, 741–750.
Lewis, M.; Sycara, K.; Walker, P. The role of trust in human-robot interaction. Found. Trust. Auton. 2018, 117, 135–159.
Guo, Y.; Yang, X.J. Modeling and predicting trust dynamics in human–robot teaming: A Bayesian inference approach. Int. J. Soc. Robot. 2021, 13, 1899–1909.
Hancock, P.A.; Billings, D.R.; Schaefer, K.E.; Chen, J.Y.; De Visser, E.J.; Parasuraman, R. A meta-analysis of factors affecting trust in human-robot interaction. Hum. Factors 2011, 53, 517–527.
Onnasch, L.; Roesler, E. A taxonomy to structure and analyze human–robot interaction. Int. J. Soc. Robot. 2021, 13, 833–849.
Albon, R.; Jewels, T. Mutual performance monitoring: Elaborating the development of a team learning theory. Group Decis. Negot. 2014, 23, 149–164.
Salas, E.; Sims, D.E.; Burke, C.S. Is there a “big five” in teamwork? Small Group Res. 2005, 36, 555–599.
Ma, L.M.; IJtsma, M.; Feigh, K.M.; Pritchett, A.R. Metrics for human-robot team design: A teamwork perspective on evaluation of human-robot teams. ACM Trans. Hum.-Robot Interact. (THRI) 2022, 11, 1–36.
You, S.; Robert, L. Teaming up with robots: An IMOI (inputs-mediators-outputs-inputs) framework of human-robot teamwork. Int. J. Robot. Eng. 2018, 2.
Guznov, S.; Lyons, J.; Pfahler, M.; Heironimus, A.; Woolley, M.; Friedman, J.; Neimeier, A. Robot transparency and team orientation effects on human–robot teaming. Int. J. Hum. Interact. 2020, 36, 650–660.
Yasar, M.S.; Iqbal, T. Robots That Can Anticipate and Learn in Human-Robot Teams. In Proceedings of the 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Sapporo, Hokkaido, Japan, 7–10 March 2022; pp. 1185–1187.
De Visser, E.J.; Peeters, M.M.; Jung, M.F.; Kohn, S.; Shaw, T.H.; Pak, R.; Neerincx, M.A. Towards a theory of longitudinal trust calibration in human–robot teams. Int. J. Soc. Robot. 2020, 12, 459–478.
Shah, J.; Breazeal, C. An empirical analysis of team coordination behaviors and action planning with application to human–robot teaming. Hum. Factors 2010, 52, 234–245.
Gervasi, R.; Mastrogiacomo, L.; Maisano, D.A.; Antonelli, D.; Franceschini, F. A structured methodology to support human–robot collaboration configuration choice. Prod. Eng. 2022, 2022, 1–17.
Dahiya, A.; Aroyo, A.M.; Dautenhahn, K.; Smith, S.L. A survey of multi-agent Human–Robot Interaction systems. Robot. Auton. Syst. 2023, 161, 104335.
Lemaignan, S.; Cooper, S.; Ros, R.; Ferrini, L.; Andriella, A.; Irisarri, A. Open-source Natural Language Processing on the PAL Robotics ARI Social Robot. In Proceedings of the Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, Stockholm, Sweden, 13–16 March 2023; pp. 907–908.
Hari, S.K.K.; Nayak, A.; Rathinam, S. An approximation algorithm for a task allocation, sequencing and scheduling problem involving a human-robot team. IEEE Robot. Autom. Lett. 2020, 5, 2146–2153.
Singh, S.; Heard, J. Human-aware reinforcement learning for adaptive human robot teaming. In Proceedings of the 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Sapporo, Hokkaido, Japan, 7–10 March 2022; pp. 1049–1052.
Tian, C.; Xu, Z.; Wang, L.; Liu, Y. Arc fault detection using artificial intelligence: Challenges and benefits. Math. Biosci. Eng. 2023, 20, 12404–12432.
Naser, M.; Alavi, A. Insights into performance fitness and error metrics for machine learning. arXiv 2020, arXiv:2006.00887.
Chakraborti, T.; Kambhampati, S.; Scheutz, M.; Zhang, Y. Ai challenges in human-robot cognitive teaming. arXiv 2017, arXiv:1707.04775.

© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.

Upload a video for this entry

Information

Subjects: Robotics

Contributors MDPI registered users' name will be linked to their SciProfiles pages. To register with us, please refer to https://encyclopedia.pub/register :

Shakra Mehak

John D. Kelleher

Michael Guilfoyle

Maria Chiara Leva

View Times: 90

Update Date: 01 Feb 2024

Table of Contents

Video Upload Options

Confirm