AI-Informed Decision Making: Comparison
View Latest Version
Please note this is a comparison between Version 2 by Catherine Yang and Version 1 by Jianlong Zhou.

AI-assisted decision-making that impacts individuals raises critical questions about transparency and fairness in artificial intelligence (AI). Much research has highlighted the reciprocal relationships between the transparency/explanation and fairness in AI-assisted decision-making. Thus, considering their impact on user trust or perceived fairness simultaneously benefits responsible use of socio-technical AI systems, but currently receives little attention.

  • AI explanation
  • AI fairness
  • trust
  • perception of fairness
  • AI ethics

1. Introduction

Artificial Intelligence (AI) informed decision-making is claimed to lead to faster and better decision outcomes. It has been increasingly used in our society from decision-making of daily lives such as recommending movies and books to making more critical decisions such as medical diagnosis, credit risk prediction, and shortlisting talents in recruitment. In 2020, the EU proposed the European approach to excellence and trust with their White Paper on AI [1]. They stated that AI will change lives by improving not only healthcare but also increasing the efficiency of farming and contributing to climate change mitigation. Thus, their approach is to improve lives, while respecting rights. Among such AI-informed decision-making tasks, trust and perception of fairness have been found to be critical factors driving human behaviour in human–machine interactions [2,3][2][3]. The black-box nature of AI models makes it hard for users to understand why a decision is made or how the data are processed for the decision-making [4,5,6][4][5][6]. Thus, trustworthy AI has experienced a significant surge in interest from the research community in various application domains, especially in high stake domains which usually require testing and verification for reasonability by domain experts not only for safety but also for legal reasons [7,8,9,10,11][7][8][9][10][11].

1.1. AI Explanation

Explanation and trust are common partners in everyday life, and extensive research has investigated the relations between AI explanations and trust from different perspectives ranging from philosophical to qualitative and quantitative dimensions [12]. For instance, Zhou et al. [13] showed that the explanation of influences of training data points on predictions significantly increased the user trust in predictions. Alam and Mueller [14] investigated the roles of explanations in AI-informed decision-making in medical diagnosis scenarios. The results show that visual and example-based explanations integrated with rationales had a significantly better impact on patient satisfaction and trust than no explanations, or with text-based rationales alone. The previous studies that empirically tested the importance of explanations to users, in various fields, consistently showed that explanations significantly increase user trust. Furthermore, with the advancement of AI explanation research, different explanation approaches such as local and global explanations, as well as feature importance-based and example-based explanations are proposed [6]. As a result, besides the explanation presentation styles such as visualisation and text [14[14][15],15], it is also critical to understand how different explanation approaches affect user trust in AI-informed decision-making. In addition, Edwards [16] stated that the main challenge for AI-informed decision-making is to know whether an explanation that seems valid is accurate. This information is also needed to ensure transparency and accountability of the decision.

1.2. AI Fairness

The data used to train machine learning models are often historical records or samples of events. They are usually not a precise description of events and conceal discrimination with sparse details which are very difficult to identify. AI models are also imperfect abstractions of reality because of their statistical nature. All these lead to imminent imprecision and discrimination (bias) associated with AI. As a result, the investigation of fairness in AI has been becoming an indispensable component for responsible socio-technical AI systems in various decision-making tasks [17,18][17][18]. In addition, extensive research focuses on fairness definitions and unfairness quantification. Furthermore, human’s perceived fairness (perception of fairness) plays an important role in AI-informed decision-making since AI is often used by humans and/or for human-related decision-making [19].
Duan et al. [20] argue that AI-informed decision-making can help users make better decisions. Furthermore, the authors propose that AI-informed decisions will be mostly accepted by humans when used as a support tool. Thus, it is crucial to consider the human perception of AI in general, and to what extent users would be willing to use such systems [21]. Considerable research on perceived fairness has evidenced its links to trust such as in management and organizations [22,23][22][23].

2. Fairness and Explanation in AI-Informed Decision Making

2.1. Perception of Fairness

Current machine learning outlines fairness in the context of different protected attributes (race, sex, culture, etc.) receiving equal treatments by algorithms [25,26,27][24][25][26]. Definitions of fairness are formalised ranging from statistical bias, group fairness, and individual fairness, to process fairness, and others. Various metrics are proposed to quantify the unfairness (bias) of algorithms [28,29,30][27][28][29].
The research on the perception of fairness can be categorised into the following dimensions [19]: First, algorithmic factors study how the technical design of an AI system affects people’s fairness perceptions. For example, Lee et al. [31,32][30][31] investigated people’s perception of fairness regarding the allocation of resources based on equality, equity, or efficiency. They found that people had many variations in the preferences for the three fairness metrics (equality, equity, efficiency) impacted by the decision. Dodge et al. [24][32] found that people’s perception of fairness is evaluated primarily based on features that are used and not used in the model, algorithm errors, and errors or flaws in input data. Secondly, human factors investigate how human-related information affects the perception of fairness. For example, Helberger et al. [33] found that education and age affected both perceptions of algorithmic fairness and people’s reasons for the perception of AI fairness. Thirdly, comparative effects investigate how individuals react in fairness to humans compared to algorithmic decision-makers. For example, Helberger et al. [33] found that people believe that AI makes fairer decisions than human decision-makers. Some studies found the opposite results in the criminal justice system [34]. Fourthly, the consequence of the perception of fairness aims to investigate the impact of the perception of fairness on AI-informed decision-making. For example, Shin and Park [35] investigated the effects of perception of fairness on satisfaction and found that people’s perception of fairness has a positive impact on satisfaction with algorithms. Moreover, Shin et al. [36] argued that the algorithmic experience is inherently related to the perception of fairness, transparency and the underlying trust. Zhou et al. [3] investigated the relationship between induced algorithmic fairness and its perception in humans. It was found that introduced fairness is positively related to the perception of fairness, i.e., the high level of introduced fairness resulted in a high level of perception of fairness.
People’s perception of fairness has close relations with AI explanations. Shin [37] looked at explanations for an algorithmic decision as a critical factor of perceived fairness, and it was found that explanations for an algorithmic decision significantly increased people’s perception of fairness in an AI-based news recommender system. Dodge et al. [24][32] found that case-based and sensitivity-based explanations effectively exposed fairness discrepancies between different cases, while demographic explanations (offering information about the classification for individuals in the same demographic categories) and input influence (presenting all input features and their impact in the classification) enhanced fairness perception by increasing people’s confidence in understanding the model. Binns et al. [38] examined people’s perception of fairness in AI-informed decision-making under four explanation types (input influence, sensitivity, case-based, and demographic). It was found that people did consider fairness in AI-informed decision-making. However, depending on when and how explanations were presented, explanations had different effects on people’s perception of fairness: (1) when multiple explanation types were presented, case-based explanations (presenting a case from the model’s training data which is most similar to the decision being explained) had a negative influence on the perception of fairness. (2) When only one explanation type was presented to people, the explanation did not show effects on people’s perception of fairness.
Besides explanation types, mathematical fairness inherently introduced by AI models and/or data (also refers to introduced fairness in this paper) can affect people’s perceived fairness [3]. However, little work is found in understanding whether different explanation types and introduced fairness together affect people’s perception of fairness.

2.2. AI Fairness and Trust

User trust in algorithmic decision-making has been investigated from different perspectives. Zhou et al. [39,40][39][40] argued that communicating user trust benefits the evaluation of the effectiveness of machine learning approaches. Kizilcec [41] found that appropriate transparency of algorithms by explanation benefited the user trust. Other empirical studies found the effects of confidence score, model accuracy and users’ experience of system performance on user trust [8,42,43][8][42][43].
Understanding relations between fairness and trust is nontrivial in the social interaction context such as marketing and services. Roy et al. [23] showed that perceptions of fair treatment of customers play a positive role in engendering trust in the banking context. Earle and Siegrist [44] found that the issue’s importance affected the relations between fairness and trust. They showed that procedural fairness did not affect trust when the issue importance was high, while procedural fairness had moderate effects on trust when issue importance was low. Nikbin et al. [45] showed that perceived service fairness had a significant effect on trust, and confirmed the mediating role of satisfaction and trust in the relationship between perceived service fairness and behavioural intention.
Kasinidou et al. [46] investigated the perception of fairness in algorithmic decision-making and found that people’s perception of a system’s decision as ‘not fair’ affects the participants’ trust in the system. Shin’s investigations [27,37][26][37] showed that perception of fairness had a positive effect on trust in an algorithmic decision-making system such as recommendations. Zhou et al. [3] obtained similar conclusions that introduced fairness is positively related to user trust in AI-informed decision-making.
These previous works motivate us to further investigate how multiple factors such as AI fairness and AI explanation together affect user trust in AI-informed decision-making.

2.3. AI Explanation and Trust

Explainability is indispensable to foster user trust in AI systems, particularly in sensible application domains. Holzinger et al. [47] introduced the concept of causability and demonstrated the importance of causability in AI explanations [48,49][48][49]. Shin [37] used causability as an antecedent of explainability to examine their relations to trust, where causability gives the justification for what and how AI results should be explained to determine the relative importance of the properties of explainability. Shin argued that the inclusion of causability and explanations would help to increase trust and help users to assess the quality of explanations, e.g., with the Systems Causability Scale [50].
The influence of training data points on predictions is one of the typical AI explanation approaches [51]. Zhou et al. [13] investigated the effects of influence on user trust and found that the presentation of influences of training data points significantly increased the user trust in predictions, but only for training data points with higher influence values under the high model performance condition. Papenmerer et al. [52] investigated the effects of model accuracy and explanation fidelity, and found that model accuracy is more important for user trust than explainability. When adding nonsensical explanations, explanations can potentially harm trust. Larasati et al. [53] investigated the effects of different styles of textual explanations on user trust in an AI medical support scenario. Four textual styles of explanations including contrastive, general, truthful, and thorough were investigated. It was found that contrastive and thorough explanations produced higher user trust scores compared to the general explanation style, and truthful explanations showed no difference compared to the rest of the explanations. Wang et al. [54] compared different explanation types such as feature importance, feature contribution, nearest neighbour and counterfactual explanation from three perspectives of improving people’s understanding of the AI model, helping people recognize the model uncertainty, and supporting people’s calibrated trust in the model. They highlighted the importance of selecting different AI explanation types in designing the most suitable AI methods for a specific decision-making task.

References

  1. White Paper on Artificial Intelligence—A European Approach to Excellence and Trust. Available online: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52020DC0065 (accessed on 31 May 2022).
  2. Zhou, J.; Arshad, S.Z.; Luo, S.; Chen, F. Effects of Uncertainty and Cognitive Load on User Trust in Predictive Decision Making. In Human-Computer Interaction—INTERACT 2017; Bernhaupt, R., Dalvi, G., Joshi, A.K., Balkrishan, D., O’Neill, J., Winckler, M., Eds.; Springer: Cham, Switzerland, 2017; pp. 23–39.
  3. Zhou, J.; Verma, S.; Mittal, M.; Chen, F. Understanding Relations between Perception of Fairness and Trust in Algorithmic Decision Making. In Proceedings of the International Conference on Behavioral and Social Computing (BESC 2021), Doha, Qatar, 29–31 October 2021; pp. 1–5.
  4. Castelvecchi, D. Can we open the black box of AI? Nat. News 2016, 538, 20.
  5. Zhou, J.; Khawaja, M.A.; Li, Z.; Sun, J.; Wang, Y.; Chen, F. Making Machine Learning Useable by Revealing Internal States Update—A Transparent Approach. Int. J. Comput. Sci. Eng. 2016, 13, 378–389.
  6. Zhou, J.; Gandomi, A.H.; Chen, F.; Holzinger, A. Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics. Electronics 2021, 10, 593.
  7. Zhou, J.; Chen, F. 2D Transparency Space—Bring Domain Users and Machine Learning Experts Together. In Human and Machine Learning: Visible, Explainable, Trustworthy and Transparent; Human–Computer Interaction Series; Springer International Publishing: Cham, Switzerland, 2018; pp. 3–19.
  8. Zhou, J.; Chen, F. (Eds.) Human and Machine Learning: Visible, Explainable, Trustworthy and Transparent; Springer: Cham, Switzerland, 2018.
  9. Holzinger, K.; Mak, K.; Kieseberg, P.; Holzinger, A. Can we Trust Machine Learning Results? Artificial Intelligence in Safety-Critical Decision Support. ERCIM News 2018, 112, 42–43.
  10. Stoeger, K.; Schneeberger, D.; Kieseberg, P.; Holzinger, A. Legal aspects of data cleansing in medical AI. Comput. Law Secur. Rev. 2021, 42, 105587.
  11. Stoeger, K.; Schneeberger, D.; Holzinger, A. Medical Artificial Intelligence: The European Legal Perspective. Commun. ACM 2021, 64, 34–36.
  12. Pieters, W. Explanation and trust: What to tell the user in security and AI? Ethics Inf. Technol. 2011, 13, 53–64.
  13. Zhou, J.; Hu, H.; Li, Z.; Yu, K.; Chen, F. Physiological Indicators for User Trust in Machine Learning with Influence Enhanced Fact-Checking. In Machine Learning and Knowledge Extraction; Springer: Cham, Switzerland, 2019; pp. 94–113.
  14. Alam, L.; Mueller, S. Examining the effect of explanation on satisfaction and trust in AI diagnostic systems. BMC Med Inform. Decis. Mak. 2021, 21, 178.
  15. Zhou, J.; Chen, F. Making machine learning useable. Int. J. Intell. Syst. Technol. Appl. 2015, 14, 91–109.
  16. Dwivedi, Y.K.; Hughes, L.; Ismagilova, E.; Aarts, G.; Coombs, C.; Crick, T.; Duan, Y.; Dwivedi, R.; Edwards, J.; Eirug, A.; et al. Artificial Intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. Int. J. Inf. Manag. 2021, 57, 101994.
  17. Berk, R.; Heidari, H.; Jabbari, S.; Kearns, M.; Roth, A. Fairness in criminal justice risk assessments: The state of the art. Sociol. Methods Res. 2018, 50, 0049124118782533.
  18. Feldman, M.; Friedler, S.A.; Moeller, J.; Scheidegger, C.; Venkatasubramanian, S. Certifying and removing disparate impact. In Proceedings of the KDD2015, Sydney, NSW, Australia, 10–13 August 2015; pp. 259–268.
  19. Starke, C.; Baleis, J.; Keller, B.; Marcinkowski, F. Fairness Perceptions of Algorithmic Decision-Making: A Systematic Review of the Empirical Literature. arXiv 2021, arXiv:2103.12016.
  20. Duan, Y.; Edwards, J.S.; Dwivedi, Y.K. Artificial intelligence for decision-making in the era of Big Data—Evolution, challenges and research agenda. Int. J. Inf. Manag. 2019, 48, 63–71.
  21. Kuzior, A.; Kwilinski, A. Cognitive Technologies and Artificial Intelligence in Social Perception. Manag. Syst. Prod. Eng. 2022, 30, 109–115.
  22. Komodromos, M. Employees’ Perceptions of Trust, Fairness, and the Management of Change in Three Private Universities in Cyprus. J. Hum. Resour. Manag. Labor Stud. 2014, 2, 35–54.
  23. Roy, S.K.; Devlin, J.F.; Sekhon, H. The impact of fairness on trustworthiness and trust in banking. J. Mark. Manag. 2015, 31, 996–1017.
  24. Kilbertus, N.; Carulla, M.R.; Parascandolo, G.; Hardt, M.; Janzing, D.; Schölkopf, B. Avoiding discrimination through causal reasoning. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 656–666.
  25. Bellamy, R.K.E.; Dey, K.; Hind, M.; Hoffman, S.C.; Houde, S.; Kannan, K.; Lohia, P.; Martino, J.; Mehta, S.; Mojsilovic, A.; et al. AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias. arXiv 2018, arXiv:1810.01943.
  26. Shin, D. User Perceptions of Algorithmic Decisions in the Personalized AI System:Perceptual Evaluation of Fairness, Accountability, Transparency, and Explainability. J. Broadcast. Electron. Media 2020, 64, 541–565.
  27. Corbett-Davies, S.; Goel, S. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv 2018, arXiv:1808.00023.
  28. Nabi, R.; Shpitser, I. Fair inference on outcomes. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2–7 February 2018; Volume 2018, p. 1931.
  29. Glymour, B.; Herington, J. Measuring the biases that matter: The ethical and casual foundations for measures of fairness in algorithms. In Proceedings of the Conference on Fairness, Accountability, and Transparency, Atlanta, GA, USA, 29–31 January 2019; pp. 269–278.
  30. Lee, M.K.; Baykal, S. Algorithmic Mediation in Group Decisions: Fairness Perceptions of Algorithmically Mediated vs. Discussion-Based Social Division. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, Portland, OR, USA, 25 February–1 March 2017; pp. 1035–1048.
  31. Lee, M.K.; Jain, A.; Cha, H.J.; Ojha, S.; Kusbit, D. Procedural Justice in Algorithmic Fairness: Leveraging Transparency and Outcome Control for Fair Algorithmic Mediation. Proc. ACM Hum. Comput. Interact. 2019, 3, 1–26.
  32. Dodge, J.; Liao, Q.V.; Zhang, Y.; Bellamy, R.K.E.; Dugan, C. Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment. In Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI’19), Marina del Ray, CA, USA, 17–20 March 2019; pp. 275–285.
  33. Helberger, N.; Araujo, T.; de Vreese, C.H. Who is the fairest of them all? Public attitudes and expectations regarding automated decision-making. Comput. Law Secur. Rev. 2020, 39, 105456.
  34. Harrison, G.; Hanson, J.; Jacinto, C.; Ramirez, J.; Ur, B. An Empirical Study on the Perceived Fairness of Realistic, Imperfect Machine Learning Models. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* ’20, Barcelona, Spain, 27–30 January 2020; pp. 392–402.
  35. Shin, D.; Park, Y.J. Role of fairness, accountability, and transparency in algorithmic affordance. Comput. Hum. Behav. 2019, 98, 277–284.
  36. Shin, D.; Zhong, B.; Biocca, F.A. Beyond user experience: What constitutes algorithmic experiences? Int. J. Inf. Manag. 2020, 52, 102061.
  37. Shin, D. The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable AI. Int. J. Hum. Comput. Stud. 2021, 146, 102551.
  38. Binns, R.; Van Kleek, M.; Veale, M.; Lyngs, U.; Zhao, J.; Shadbolt, N. ‘It’s Reducing a Human Being to a Percentage’: Perceptions of Justice in Algorithmic Decisions. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, Montreal, QC, Canada, 21–26 April 2018; pp. 1–14.
  39. Zhou, J.; Bridon, C.; Chen, F.; Khawaji, A.; Wang, Y. Be Informed and Be Involved: Effects of Uncertainty and Correlation on User’s Confidence in Decision Making. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, Association for Computing Machinery, CHI EA ’15, Seoul, Korea, 18–23 April 2015; pp. 923–928.
  40. Zhou, J.; Sun, J.; Chen, F.; Wang, Y.; Taib, R.; Khawaji, A.; Li, Z. Measurable Decision Making with GSR and Pupillary Analysis for Intelligent User Interface. ACM Trans. Comput.-Hum. Interact. 2015, 21, 1–23.
  41. Kizilcec, R.F. How Much Information? Effects of Transparency on Trust in an Algorithmic Interface. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, CHI ’16, San Jose, CA, USA, 7–12 May 2016; pp. 2390–2395.
  42. Zhang, Y.; Liao, Q.V.; Bellamy, R.K.E. Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* ’20, Barcelona, Spain, 27–30 January 2020; pp. 295–305.
  43. Yin, M.; Vaughan, J.W.; Wallach, H. Does Stated Accuracy Affect Trust in Machine Learning Algorithms? In Proceedings of the ICML2018 Workshop on Human Interpretability in Machine Learning (WHI 2018), Stockholm, Sweden, 14 July 2018; pp. 1–2.
  44. Earle, T.C.; Siegrist, M. On the Relation Between Trust and Fairness in Environmental Risk Management. Risk Anal. 2008, 28, 1395–1414.
  45. Nikbin, D.; Ismail, I.; Marimuthu, M.; Abu-Jarad, I. The effects of perceived service fairness on satisfaction, trust, and behavioural intentions. Singap. Manag. Rev. 2011, 33, 58–73.
  46. Kasinidou, M.; Kleanthous, S.; Barlas, P.; Otterbacher, J. I Agree with the Decision, but They Didn’t Deserve This: Future Developers’ Perception of Fairness in Algorithmic Decisions. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, Virtual Event, 3–10 March 2021; pp. 690–700.
  47. Holzinger, A.; Langs, G.; Denk, H.; Zatloukal, K.; Müller, H. Causability and Explainability of Artificial Intelligence in Medicine. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 2019, 9, 1–13.
  48. Holzinger, A.; Malle, B.; Saranti, A.; Pfeifer, B. Towards Multi-Modal Causability with Graph Neural Networks enabling Information Fusion for explainable AI. Inf. Fusion 2021, 71, 28–37.
  49. Hudec, M.; Minarikova, E.; Mesiar, R.; Saranti, A.; Holzinger, A. Classification by ordinal sums of conjunctive and disjunctive functions for explainable AI and interpretable machine learning solutions. Knowl. Based Syst. 2021, 220, 106916.
  50. Holzinger, A.; Carrington, A.; Mueller, H. Measuring the Quality of Explanations: The System Causability Scale (SCS). Comparing Human and Machine Explanations. KI -Kuenstliche Intell. 2020, 34, 193–198.
  51. Koh, P.W.; Liang, P. Understanding Black-box Predictions via Influence Functions. Proc. ICML 2017, 70, 1885–1894.
  52. Papenmeier, A.; Englebienne, G.; Seifert, C. How model accuracy and explanation fidelity influence user trust. arXiv 2019, arXiv:1907.12652.
  53. Larasati, R.; Liddo, A.D.; Motta, E. The Effect of Explanation Styles on User’s Trust. In Proceedings of the Workshop on Explainable Smart Systems for Algorithmic Transparency in Emerging Technologies co-located with IUI 2020, Cagliari, Italy, 17 March 2020; pp. 1–6.
  54. Wang, X.; Yin, M. Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making. In Proceedings of the 26th International Conference on Intelligent User Interfaces, College Station, TX, USA, 14–17 April 2021; pp. 318–328.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/)
Video Production Service
Feedback