Decentralized Federated Learning and Knowledge Graph Embedding: Comparison
View Latest Version
Please note this is a comparison between Version 2 by Jessie Wu and Version 1 by Xiangjie Kong.

Anomaly detection plays a crucial role in data security and risk management across various domains, such as financial insurance security, medical image recognition, and Internet of Things (IoT) device management. Researchers rely on machine learning to address potential threats in order to enhance data security.

  • knowledge graph embedding
  • anomaly detection
  • relation

1. Introduction

The development of Internet technology makes digitized data and information easy to be transmitted and analyzed, and the subtle connections between data and information are easier to mine [1]. But at the same time, hidden crises and potential risks, such as abnormal data and fraudulent behavior, are also mixed in. Whether it is fraud detection in the financial field, device quality monitoring in the IoT industry, disease diagnosis in the healthcare field, or intrusion detection in network security, all rely on anomaly detection to ensure system reliability and data integrity. Regardless of the industry sector, all involve serious economic losses and trust crises. Therefore, the research and development of effective detection mechanisms for the management and analysis of digitized information have become crucial.
However, anomaly samples are still rare and traditional auto insurance fraud detection relies directly on expert manual review. This results in extremely inefficient fraud detection [2]. To reduce human error and missed inspections, insurance companies start to leverage the automated intelligence of machine learning [3]. In addition to using unsupervised learning, semi-supervised learning, and other methods to improve the performance of anomaly detection models, researchers also use these methods: (1) Data generation: generating abnormal data through transformation, expansion, or learning from existing data for sample expansion, such as how Zhang et al. utilized MetaGAN, based on a Generative Adversarial Network, to generate images to strengthen the performance of sample-level image classification [4]. (2) Transfer Learning [5]: learning anomaly detection models from the original domain dataset and transferring them to the target domain. (3) Active Learning [6]: improving model performance by intelligently selecting which samples should be labeled, thereby reducing dependence on labeled data. (4) Cooperative train [7]: Collaborating between different data holders to jointly build anomaly detection models can also help solve the problem of data scarcity. The centralized training of data can indeed improve the efficiency of detection, but this completely disregards privacy concerns [8]. Especially in the financial insurance industry, when it comes to a substantial amount of customer information, data sharing needs to be carried out with the precondition of ensuring privacy protection.

2. Decentralized Federated Learning

Artificial Intelligence is steadily developing, and its technical underlying support remains data. The quantity, quality, and dimension of data have become some of the most important factors constraining the progress of science and technology. As data owners need to consider data security protection, competition relations, and legal regulations when facing data exchange and sharing, it leads to the problem of “data silos” between enterprises and industries [21][9]. How to share data safely and effectively has become a popular research topic.
In 2017, Google first proposed and constructed a Federated Learning (FL) framework to realize the idea of model updating locally [22][10]. They aimed to improve the prediction accuracy of what Android users associate with their next input when typing on their mobile terminals. Subsequently, a large number of scholars conducted more in-depth research on data security and personalized models. In 2019, Google released the first FL framework in the world, TensorFlow Federated Framework. And in the same year, Professor Yang Qiang with his team open-sourced the first FL framework in China, named Federated AI Technology Enabler [23][11], as a secure computing framework to support the Federated AI system.
FL can not only break through the “data silos” and “small data” limitations during the training process, but also ensures a certain degree of data privacy and security while benefitting all participants [24][12]. Because of this, it has been a high concern of researchers in various fields. This framework has a wide range of applications, including medical image processing [25][13], auto plate recognition [26][14], air handwriting recognition [27][15], and so on.
However, due to the high dependence of FL on the central server, it is unable to cope with the problem of a single point of failure of the central server, and thus DFL emerges [28][16]. A more decentralized federation aggregation is achieved through communication and interaction between participants. Currently, this framework has been applied in several fields. Lu et al. [29][17] extracted medical patient features more securely by constructing a DFL model that conforms to realistic cooperation. In addition, Kalapaaking et al. [30][18] combined blockchain with FL to improve the security of the system by using the traceable and untamperable characteristics of blockchain. They used blockchain with a Trusted Execution Environment to replace the central server to improve the fault tolerance and attack resistance of the system.
Compared with traditional FL, DFL has no central communication bottleneck, but it generates a huge client–client communication overhead. To deal with this problem, Liu et al. [31][19] pioneered the application of the Lloyd–Max algorithm to DFL. They utilized the exchange of model information between neighboring nodes to adaptively adjust the quantization level, and succeeded in improving the communication efficiency by reducing the amount of data of the federated transmission model parameters. Sun et al. [32][20] investigated Decentralized FedAVG with Momentum (DFedAvgM) based on the FedAVG paradigm, which reduces the communication overhead by mixing the matrices, Momentum, multiple local iterations of client training, and quantization of sending models.

3. Knowledge Graph Embedding

Knowledge Graph as a kind of mesh database manages loose multi-source heterogeneous data through a standardized structural organization (head entity, relation, tail entity) [33][21]. With the advantages of graph structure to reflect and manage information, and to help accurate positioning and searching, KG provides strong underlying support for specific downstream applications such as Internet semantic searches, personalized recommendations, an intelligent Question Answering System and big data decision-making.
However, such a ternary structure is difficult to deal with directly due to the low portability of its underlying symbolic properties. Therefore, researchers apply knowledge graph embedding to ensure computational simplicity by embedding entities and relations into a continuous low-dimensional vector space, while preserving the structural information of the KG [34][22]. The entities and relations are downscaled and stored in the low-dimensional space in the form of vector matrices or tensors.
The training of KGE involves the semantic understanding level. It is necessary to consider how to extract relations and entities from non-aligned heterogeneous data, and how to understand the real meaning of different relations and entities for the alignment task [35][23].
Existing knowledge graph embedding methods are mainly categorized into three types: (i) Translational Distance Models; (ii) Semantic Matching Models; and (iii) Neural Network Models. Based on the above three types of models, many models have been derived.
The Translational Distance Models are based on TransE [36][24] and extend to derive models such as TransR [37][25], RotatE [38][26], and HAKE [39][27]. This type of method defines the scoring function by modeling the relation as the distance from the head entity to the tail entity like the Euclidean distance of TransE and the rotation transformation of RotatE. Semantic Matching Models measure the rationality of triples at the semantic level to construct score functions, mainly including RESCAL (bilinear model) models [40][28]. Due to the lack of clear Euclidean inner product correspondence in hyperbolic spaces, in order to extend the calculation to hyperbolic spaces, Ivana Balaževic et al. first used a combination of a bilinear model and Poincaré ball [41][29]. The MuRP model proposed by them can outperform Euclidean models on the link prediction task at lower dimensionality. As for the Neural Network Models, they score by embedding the head entity, relation, and tail entity into the neural network. Jiarui Zhang et al. [42][30] found that data-driven link prediction tasks rely on various labels and only utilize the structural information of the graph. Inspired by knowledge distillation, the DA-GCN proposed by them makes use of logical rules to reduce the dependence of graph neural networks on data and iterative rules to construct graph convolutional networks. The KGE trained by DA-GCN can perform excellently in link prediction tasks.

References

  1. Wang, H.; Fu, T.; Du, Y.; Gao, W.; Huang, K.; Liu, Z.; Chandak, P.; Liu, S.; Van Katwyk, P.; Deac, A.; et al. Scientific discovery in the age of artificial intelligence. Nature 2023, 620, 47–60.
  2. Zhang, L.; Wu, T.; Chen, X.; Lu, B.; Na, C.; Qi, G. Auto Insurance Knowledge Graph Construction and Its Application to Fraud Detection. In Proceedings of the 10th International Joint Conference on Knowledge Graphs, Virtual, 6–8 December 2021; Association for Computing Machinery: New York, NY, USA, 2022; pp. 64–70.
  3. Dhieb, N.; Ghazzai, H.; Besbes, H.; Massoud, Y. Extreme Gradient Boosting Machine Learning Algorithm for Safe Auto Insurance Operations. In Proceedings of the 2019 IEEE International Conference on vehicular Electronics and Safety (ICVES), Cairo, Egypt, 4–6 September 2019; pp. 1–5.
  4. Zhang, R.; Che, T.; Ghahramani, Z.; Bengio, Y.; Song, Y. MetaGAN: An Adversarial Approach to Few-Shot Learning. Adv. Neural Inf. Process. Syst. 2018, 31, 2365–2374.
  5. Vincent, V.; Wannes, M.; Jesse, D. Transfer learning for anomaly detection through localized and unsupervised instance selection. In Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA, 7–12 February 2020; Volume 34, pp. 6054–6061.
  6. Chen, Z.; Duan, J.; Kang, L.; Qiu, G. Supervised Anomaly Detection via Conditional Generative Adversarial Network and Ensemble Active Learning. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 7781–7798.
  7. Li, W.; Du, Q. Collaborative Representation for Hyperspectral Anomaly Detection. IEEE Trans. Geosci. Remote Sens. 2014, 53, 1463–1474.
  8. Peng, H.; Li, H.; Song, Y.; Zheng, V.; Li, J. Differentially Private Federated Knowledge Graphs Embedding. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, Virtual, 1–5 November 2021; pp. 1416–1425.
  9. Kong, X.; Gao, H.; Shen, G.; Duan, G.; Das, S.K. FedVCP: A Federated-Learning-Based Cooperative Positioning Scheme for Social Internet of Vehicles. IEEE Trans. Comput. Soc. Syst. 2022, 9, 197–206.
  10. Chen, M.; Zhang, W.; Yuan, Z.; Jia, Y.; Chen, H. Fede: Embedding knowledge graphs in federated setting. In Proceedings of the 10th International Joint Conference on Knowledge Graphs, Virtual, 6–8 December 2021; pp. 80–88.
  11. Federated AI Technology Enabler. Available online: https://github.com/FederatedAI/FATE (accessed on 21 August 2023).
  12. Li, L.; Fan, Y.; Tse, M.; Lin, K.Y. A review of applications in federated learning. Comput. Ind. Eng. 2020, 149, 106854.
  13. Niu, Y.; Deng, W. Federated learning for face recognition with gradient correction. In Proceedings of the AAAI Conference on Artificial Intelligence, Palo Alto, CA, USA, 22 February–1 March 2022; Volume 36, pp. 1999–2007.
  14. Kong, X.; Wang, K.; Hou, M.; Hao, X.; Shen, G.; Chen, X.; Xia, F. A Federated Learning-Based License Plate Recognition Scheme for 5G-Enabled Internet of Vehicles. IEEE Trans. Ind. Inform. 2021, 17, 8523–8530.
  15. Kong, X.; Zhang, W.; Qu, Y.; Yao, X.; Shen, G. FedAWR: An Interactive Federated Active Learning Framework for Air Writing Recognition. IEEE Trans. Mob. Comput. 2023, in press.
  16. Hegedus, I.; Danner, G.; Jelasity, M. Decentralized learning works: An empirical comparison of gossip learning and federated learning. J. Parallel Distrib. Comput. 2021, 148, 109–124.
  17. Lu, S.; Zhang, Y.; Wang, Y.; Mack, C. Learn electronic health records by fully decentralized federated learning. arXiv 2019, arXiv:1912.01792.
  18. Kalapaaking, A.P.; Khalil, I.; Rahman, M.S.; Atiquzzaman, M.; Yi, X.; Almashor, M. Blockchain-based federated learning with secure aggregation in trusted execution environment for internet-of-things. IEEE Trans. Ind. Inform. 2022, 19, 1703–1714.
  19. Liu, W.; Chen, L.; Chen, Y.; Wang, W. Communication-Efficient Design for Quantized Decentralized Federated Learning. arXiv 2023, arXiv:2303.08423.
  20. Sun, T.; Li, D.; Wang, B. Decentralized federated averaging. IEEE Trans. Pattern Anal. Mach. Intell. 2022, 45, 4289–4301.
  21. Ji, S.; Pan, S.; Cambria, E.; Marttinen, P.; Philip, S.Y. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Trans. Neural Netw. Learn. Syst. 2021, 33, 494–514.
  22. Huang, X.; Zhang, J.; Li, D.; Li, P. Knowledge graph embedding based question answering. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, Melbourne, Australia, 11–15 February 2019; pp. 105–113.
  23. Wang, Q.; Mao, Z.; Wang, B.; Guo, L. Knowledge graph embedding: A survey of approaches and applications. IEEE Trans. Knowl. Data Eng. 2017, 29, 2724–2743.
  24. Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; Yakhnenko, O. Translating embeddings for modeling multi-relational data. Adv. Neural Inf. Process. Syst. 2013, 26, 1–9.
  25. Lin, Y.; Liu, Z.; Sun, M.; Liu, Y.; Zhu, X. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, Austin, TX, USA, 25–30 January 2015; Volume 29.
  26. Sun, Z.; Deng, Z.H.; Nie, J.Y.; Tang, J. RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. In Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA, 6–9 May 2019.
  27. Zhang, Z.; Cai, J.; Zhang, Y.; Wang, J. Learning hierarchy-aware knowledge graph embeddings for link prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 6–9 May 2020; Volume 34, pp. 3065–3072.
  28. Nickel, M.; Tresp, V.; Kriegel, H.P. A three-way model for collective learning on multi-relational data. In Proceedings of the ICML, Bellevue, WA, USA, 28 June–2 July 2011; Volume 11, pp. 3104482–3104584.
  29. Balazevic, I.; Allen, C.; Hospedales, T. Multi-relational poincaré graph embeddings. Adv. Neural Inf. Process. Syst. 2019, 32, 4463–4473.
  30. Zhang, J.; Huang, J.; Gao, J.; Han, R.; Zhou, C. Knowledge graph embedding by logical-default attention graph convolution neural network for link prediction. Inf. Sci. 2022, 593, 201–215.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/)
Video Production Service
Feedback