Advanced Persistent Threat Predictive Analytics

Version	Summary	Created by	Modification	Content Size	Created at	Operation
1		Haitian Liu	--	2150	2023-12-27 12:28:29	\|
2	update references and layout	Rita Xu	Meta information modification	2150	2023-12-28 03:19:45	\|

This entry is adapted from the peer-reviewed paper 10.3390/electronics12081849

Advanced persistent threat (APT) audit logs information and uses a combination of causal graphs and deep learning techniques to perform predictive analysis of APT.

APT causal graph evolving graph

1. Introduction

Cyberattacks have become more common, which can often cause significant economic damage and can even hinder the operation of core public services. In addition, advanced, persistent cyber threats have recently re-emerged due to the advent of the Internet of Things and the increased number of interconnected devices ^[1]. For example, in May 2017, the “WannaCry” ransomware attack was detected after targeting 200,000 servers in over 150 countries ^[2]. In the same year, another form of the same attack caused disruptions to most government websites and several companies in Ukraine, and eventually, the attack spread globally ^[3].

The forms of network attacks are complex and diverse, and more types of attacks have changed from simple one-step attacks to new composite attacks ^[4]. In response, the techniques used by attackers to attack computer systems and networks have reached an unprecedented level of sophistication, using a combination of multiple steps to achieve their goals in a premeditated manner, represented by the presence of advanced persistent threats (APTs) ^[5]^[6].

It requires the execution of a series of attack stages; however, the individual stages may be benign or malicious, and very occasionally, each attack stage can behave as a benign stage without raising any suspicion. In addition, attacks may last for weeks or years, and traditional intrusion detection systems (IDSs) may not be able to detect these attacks due to the time variation between attack stages ^[3]. A new intrusion detection model is necessary to address these threats, identify the ongoing attacks early, and anticipate the attacker’s further strategies as much as possible. This approach will provide network analysts with a foundation for preventing attacks. Yet, detection and prediction of various types of dynamic attacks is always a challenging task ^[7].

In this regard, a variety of different mechanisms can be used to achieve the detection and prediction of multi-stage attacks. These mechanisms include discrete models such as attack graphs, Bayesian networks, Markov models, and game theory or continuous models such as time series and grey models. Among the various graph models, causal graphs appear to be an ideal threat analysis approach, linking causal events in a system, with powerful semantic representation and attack history correlation capabilities.

Audit log data are a good source of information for online monitoring and anomaly/attack detection, considering that they record system status and significant events at various critical moments to help debug performance problems and failures, and for root cause analysis. In addition, as system logs record noteworthy events that occur during active running processes, such log data are universally available in almost all computer systems. This makes it a natural advantage to construct causal graphs from audit log data, which is moreover a very common practice.

However, the prediction of multi-stage attacks based on causal graphs remains an open problem, and previous research on cyber threat events based on causal graphs has mostly stopped at the detection of malicious events that have already occurred and the tracing of attack scenarios, while rarely considering the speculation of specific malicious attacks that will occur next.

When faced with predicting malicious events in multi-stage attacks based on audit log data and causal graphs, there are a number of challenges that need to be considered, including but not limited to the following challenging issues.

(1): The log data themselves are unstructured and may come from different operating platforms, their format and semantics may vary from platform to platform, and it is already challenging to use unstructured logs to diagnose problems
(2): How can one reduce log complexity, minimize data storage size, and balance the space efficiency of causal graph storage with the time efficiency for attack investigation while maintaining the original semantics in audit log data?
(3): Despite experiencing the same type of attack, there is no definitive pattern indicating that a malicious event will always precede or follow another. It is possible to observe unrelated noisy events first or multiple malicious events occurring simultaneously from different adversary groups with distinct attacks.
(4): How can one design an efficient and robust malicious event prediction algorithm that ensures prediction accuracy while minimizing the response time for investigative forensics?
(5): There is a lack of a standard public dataset available to provide real log data from different multi-stage attacks and similarly, a lack of a solid standard to quantify and measure the malicious event prediction performance of such an architecture

2. Advanced Persistent Threat Predictive Analytics

Unlike common network attack detection, the analysis and investigation of an attack often begins after the attack is completed. Predictive analysis of network attacks is more focused on timeliness and is faster to be effective so that users can intervene in ongoing attacks or system performance issues.

Typically, predictive methods in cybersecurity use discrete models to represent attacks or network security situations. Clear examples are graphical models of attack processes or game-theoretic representations of interactions between attackers and defenders.

Figure 1 provides a simple profile of the cybersecurity use case, based on the discrete models that compose the approach under consideration. When it comes to predictive analytics of multi-stage attacks, the focus is primarily on attack projection. This involves recording the attacker’s behavior and constructing an attack description for future reference. If a series of events conform to the attack pattern, it can be assumed that the attack will continue along the same lines. In addition, researchers may be more interested in predicting novel attacks rather than analyzing previously observed attacks. Alternatively, researchers may prioritize forecasting the overall security situation rather than examining individual attacks.

Figure 1. Cybersecurity use case profiling.

Among the various graph models, the attack graph is a graphical representation of an attack scenario proposed by Phillips and Swiler ^[8] in 1998 and quickly became a popular formal attack representation. Hughes et al. ^[9] provided an effective method for analyzing and predicting network threats based on network models in 2003, which is considered as one of the earliest practical approaches to attack graphs and an effective static analysis method. Based on this, Polatidis et al. ^[10]^[11] constructed attack graphs using information about the underlying infrastructure and proposed a method for predicting network attacks using attack graphs and recommendation systems.

Another practical approach for attack prediction is using Bayesian networks, which are closely related to prediction methods based on attack graphs, as Bayesian networks are often constructed based on attack graphs. For example, Bayesian attack graphs are attack graphs in the form of Bayesian networks ^[12].

Hidden Markov models (HMMs) have been widely used in intrusion detection and attack prediction methods due to their ability to eliminate the dependence on complete information in graphical models, particularly when unobservable states and transitions exist. An early example is the alert correlation and prediction system proposed by Farhadi et al. ^[13] in 2011, which uses the Attack Scenario Extraction Algorithm (ASEA) to correlate and extract important alerts and then applies HMM for predictive analysis of intruder behavior. Another example is a new method based on HMM proposed by P. Holgado et al. ^[14] in 2020, which considers hidden states as similar stages of specific types of attacks and can easily adapt to multi-stage attacks and anticipate the attacker’s subsequent stages. Similarly, T. Shawly et al. ^[15] proposed a novel framework in 2021 based on HMM modeling to address the challenges of modeling and detecting complex network attacks (such as multiple interleaved attacks), which have not been addressed by previous methods.

Unlike the various graphical methods mentioned above, knowledge graphs are more geared toward dealing with larger and more dynamically changing real-time network attacks. For example, Jia Yan et al. ^[7] proposed a method for network security knowledge graph and deduction rules based on the five-tuple model in 2018. Qi et al. ^[4] further stored prior knowledge in the network security knowledge graph and attack rule library as computer-understandable data and then mined attack chains from massive data with temporal and spatial constraints, thus proposing an attack analysis framework for a network attack and defense testing platform.

In contrast to other graph models, causal/dependency graphs are often not directly applied to the problem of proactive attack prediction analysis but are widely used as a promising tool for the problem of APT attack detection. It has a strong abstract representation and relatively high efficiency to abstract the interactions between components in opaque systems through a high-fidelity and visible approach, enough to link events in the system with cause and effect, regardless of the time between events. Thus, a comprehensive understanding of the entire attack is possible, which provides a natural observation platform for the predictive analysis of cyberattacks.

Causal graphs are more commonly used in the detection of APT and the backtracking of attack scenarios and are also known as dependency graphs or provenance graphs. In Backtracker ^[16]^[17], researchers first explored the problem of piecing together the causal chains leading to an attack, i.e., the concept of attack tracing, based on the dependency graph for OS-level attack tracing, where backtracking is able to traverse the entire historical context of system execution by given a detection point. Subsequent studies ^[18]^[19] have improved the accuracy of the dependency chains constructed by Backtracker. However, these efforts run in a purely forensic setting, i.e., backtracking all relevant events of the entire attack scenario, which requires a complete traceability graph and excessive manual intervention that is neither timely nor efficient. It cannot cope with the analysis of attack activities executed in real time, much less include proactive attack prediction.

For the current causal graph-based threat analysis system, first, a comprehensive system can be divided into three modules: the data collection module, the data management module, and the threat detection module. Each module contains several components that address different research questions. In the end-to-end model, each module can be considered independently of the other. In the proposed causal graph-based malicious event prediction model, the first two modules are not significantly different from other traceability/causal graph-based threat detection systems.

Second, an ideal traceability graph-based threat analysis system needs to consider three attributes simultaneously: fast response, high efficiency, and high accuracy ^[20]. However, even after pruning, the size of the causal graph is very large. Therefore, threat analysis based on causal graphs may introduce high space and computational overhead. In previous work on causal graph-based threat detection, many attempts have been made by researchers to find a balance between these three properties. Based on the main detection designs, these approaches can be classified into three categories.

Finally, these approaches can be broadly divided into three categories based on the main design options for attack detection. The tag propagation-based approaches (Hossain et al., 2017 ^[21]; Milajerdi et al., 2019 ^[22]) try to store system execution history incrementally in tags and utilize the tag propagation process to trace the causality. These algorithms have a roughly linear time complexity. Moreover, they can take streaming graphs as input and respond fast. The abnormal detection approaches (Hassan et al., 2019 ^[23]; Liu et al., 2018 ^[24]; Xie et al., 2018, 2021 ^[25]^[26]) try to identify anomalous interactions between nodes. Therefore, these approaches will simulate normal behaviors by collecting historical data or data from parallel systems. The graph-matching-based approaches (Han et al., 2020 ^[27]; Liu et al., 2019 ^[28]) try to identify suspicious behavior by matching sub-structures in graphs. However, graph matching is computationally complex. Researchers have tried to extract graph features through graph embedding or graph sketching algorithms or using approximation methods.

As a powerful artificial intelligence technology, deep learning has been widely applied in various fields such as computer vision, natural language processing, and bioinformatics. It can learn complex patterns and relationships and extract valuable information from large amounts of data. This makes it possible to combine with various traditional models and algorithm techniques, greatly improving the automation and efficiency of models. In recent years, it has also been used in threat prediction tasks based on various function environments. For example, Deeplog proposed by Du et al. ^[29] uses a deep neural network model with long short-term memory (LSTM) to model system logs as natural language sequences. However, although it makes predictions, it is essentially an anomaly detection model rather than a prediction model. If the error between the predicted and observed value vectors is within the high confidence interval of the Gaussian distribution, the parameter value vector of the incoming log entry is considered normal; otherwise, it is considered abnormal. Deepag ^[30] further proposes a new method for threat detection and attack path prediction using bi-directional deep learning based on Deeplog. Unlike the previous two methods, Tiresias ^[31] does not consider system logs but models security events themselves and demonstrates the feasibility of predicting security events through a recurrent neural network with recurrent memory cells, filling the gap in predicting the specific steps that attackers will take when carrying out attack activities. In addition, there are other examples of attack prediction, such as the prediction of system calls ^[32] and the combination of attack prediction and network security situation forecasting, using deep learning to predict different types of threats ^[33]^[34].

References

Ghafir, I.; Kyriakopoulos, K.G.; Lambotharan, S.; Aparicio-Navarro, F.J.; Assadhan, B.; Binsalleeh, H.; Diab, D.M. Hidden Markov models and alert correlations for the prediction of advanced persistent threats. IEEE Access 2019, 7, 99508–99520.
CNET. ‘Wannacry’ Ransomware: Everything You Need to Know. Available online: https://www.windowscentral.com/wannacry-ransomware-attack-windows (accessed on 22 October 2017).
Washington Post. Massive Cyberattack Hits Europe with Widespread Ransom Demands. Available online: https://www.thegazette.com/nation-world/massive-cyberattack-hits-europe-with-widespread-ransom-demands (accessed on 22 October 2017).
Qi, Y.; Jiang, R.; Jia, Y.; Li, A. Attack Analysis Framework for Cyber-Attack and Defense Test Platform. Electronics 2020, 9, 1413.
Steinberger, J.; Sperotto, A.; Golling, M.; Baier, H. How to exchange security events? Overview and evaluation of formats and protocols. In Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management (IM), Ottawa, ON, Canada, 11–15 May 2015; pp. 261–269.
Kaspersky. What Is WannaCry Ransomware. Available online: https://usa.kaspersky.com/resource-center/threats/ransomware-wannacry (accessed on 10 January 2022).
Jia, Y.; Qi, Y.; Shang, H.; Jiang, R.; Li, A. A Practical Approach to Constructing a Knowledge Graph for Cybersecurity. Engineering 2018, 4, 117–133.
Phillips, C.; Swiler, L.P. A Graph-Based System for Network-Vulnerability Analysis. In Proceedings of the Workshop New Security Paradigms, Charlottesville, VA, USA, 22–26 September 1998; pp. 71–79.
Hughes, T.; Sheyner, O. Attack Scenario Graphs for Computer Network Threat Analysis and Prediction. Complexity 2003, 9, 15–18.
Polatidis, N.; Pimenidis, E.; Pavlidis, M.; Kameas, A. Recommender Systems Meeting Security: From Product Recommendation to Cyber-Attack Prediction. In Proceedings of the Engineering Applications of Neural Networks: 18th International Conference, Athens, Greece, 25–27 August 2017; pp. 508–519.
Polatidis, N.; Pimenidis, E.; Pavlidis, M.; Papastergiou, S.; Mouratidis, H. From product recommendation to cyber-attack prediction: Generating attack graphs and predicting future attacks. Evol. Syst. 2020, 11, 479–490.
Ramaki, A.A.; Khosravi-Farmad, M.; Bafghi, A.G. Real time alert correlation and prediction using Bayesian networks. In Proceedings of the 12th International Iranian Society of Cryptology Conference on Information Security and Cryptology (ISCISC), Rasht, Iran, 8–10 September 2015; pp. 98–103.
Farhadi, H.; AmirHaeri, M.; Khansari, M. Alert correlation and prediction using data mining and HMM. ISeCure 2011, 3, 77–101.
Holgado, P.; Villagrá, V.A.; Vazquez, L. Real-time multistep attack prediction based on hidden markov models. IEEE Trans. Dependable Secur. Comput. 2017, 17, 134–147.
Shawly, T.; Elghariani, A.; Kobes, J.; Ghafoor, A. Architectures for Detecting Interleaved Multi-Stage Network Attacks Using Hidden Markov Models. IEEE Trans. Dependable Secur. Comput. 2021, 18, 2316–2330.
King, S.T.; Chen, P.M. Backtracking intrusions. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles (SOSP), Bolton, NY, USA, 19–22 October 2003.
King, S.T.; Mao, Z.M.; Lucchetti, D.G.; Chen, P.M. Enriching intrusion alerts through multi-host causality. In Proceedings of the Network and Distributed System Security Symposium, San Diego, CA, USA, 8–11 February 2005.
Lee, K.H.; Zhang, X.; Xu, D. High accuracy attack provenance via binary-based execution partition. In Proceedings of the Network and Distributed System Security Symposium, San Diego, CA, USA, 24–27 February 2013.
Ma, S.; Zhang, X.; Xu, D. ProTracer: Towards practical provenance tracing by alternating between logging and tainting. In Proceedings of the Network and Distributed System Security Symposium, San Diego, CA, USA, 21–24 February 2016.
Li, Z.; Chen, Q.A.; Yang, R.; Chen, Y.; Ruan, W. Threat Detection and Investigation with System-level Provenance Graphs: A Survey. Comput. Secur. 2021, 106, 102282.
Hossain, M.N.; Milajerdi, S.M.; Wang, J.; Eshete, B.; Gjomemo, R.; Sekar, R.; Stoller, S.D.; Venkatakrishnan, V.N. SLEUTH: Real-time Attack Scenario Reconstruction from COTS Audit Data. In Proceedings of the USENIX Security Symposium, Vancouver, BC, Canada, 16–18 August 2017.
Milajerdi, S.M.; Eshete, B.; Gjomemo, R.; Venkatakrishnan, V.N. Poirot: Aligning Attack Behavior with Kernel Audit Records for Cyber Threat Hunting. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, London, UK, 11–15 November 2019; pp. 1795–1812.
Hassan, W.U.; Guo, S.; Li, D.; Chen, Z.; Jee, K.; Li, Z.; Bates, A. NoDoze: Combatting Threat Alert Fatigue with Automated Provenance Triage. In Proceedings of the Network and Distributed System Security Symposium, San Diego, CA, USA, 24–27 February 2019.
Liu, Y.; Zhang, M.; Li, D.; Jee, K.; Li, Z.; Wu, Z.; Rhee, J.; Mittal, P. Towards a Timely Causality Analysis for Enterprise Security. In Proceedings of the Network and Distributed System Security Symposium, San Diego, CA, USA, 18–21 February 2018.
Xie, Y.; Feng, D.; Hu, Y.; Li, Y.; Sample, S.; Long, D.L. Pagoda: A Hybrid Approach to Enable Efficient Real-time Provenance Based Intrusion Detection in Big Data Environments. IEEE Trans. Dependable Secur. Comput. 2018, 17, 1283–1296.
Xie, Y.; Wu, Y.; Feng, D.; Long, D. P-Gaussian: Provenance-Based Gaussian Distribution for Detecting Intrusion Behavior Variants Using High Efficient and Real Time Memory Databases. IEEE Trans. Dependable Secur. Comput. 2021, 18, 2658–2674.
Han, X.; Pasquier, T.; Bates, A.; Mickens, J.; Seltzer, M. Unicorn: Runtime Provenance-Based Detector for Advanced Persistent Threats. arXiv, 2020; arXiv:2001.01525.
Liu, F.; Wen, Y.; Zhang, D.; Jiang, X.; Xing, X.; Meng, D. Log2vec: A Heterogeneous Graph Embedding Based Approach for Detecting Cyber Threats within Enterprise. In Proceedings of the 2019 ACM SIGSAC Conference, London, UK, 11–15 November 2019.
Du, M.; Li, F.; Zheng, G.; Srikumar, V. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Dallas, TX, USA, 30 October–3 November 2017; pp. 1285–1298.
Li, T.; Jiang, Y.; Lin, C.; Obaidat, M.S.; Shen, Y.; Ma, J. Deepag: Attack graph construction and threats prediction with bi-directional deep learning. IEEE Trans. Dependable Secur. Comput. 2022, 20, 740–757.
Shen, Y.; Mariconti, E.; Vervier, P.A.; Stringhini, G. Tiresias: Predicting security events through deep learning. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, Toronto, ON, USA, 15–19 October 2018; pp. 592–605.
Lv, S.; Wang, J.; Yang, Y.; Liu, J. Intrusion prediction with system-call sequence-to-sequence model. IEEE Access 2018, 6, 71413–71421.
Yin, K.; Yang, Y.; Yao, C.; Yang, J. Long-Term Prediction of Network Security Situation Through the Use of the Transformer-Based Model. IEEE Access 2022, 10, 56145–56157.
Hu, C.; Liu, G.; Li, M. A Network Security Situation Prediction Method Based on Attention-CNN-BiGRU. In Proceedings of the 2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD), Hangzhou, China, 4–6 May 2022; pp. 257–262.

© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.

Upload a video for this entry

Information

Subjects: Computer Science, Information Systems

Contributors MDPI registered users' name will be linked to their SciProfiles pages. To register with us, please refer to https://encyclopedia.pub/register :

Haitian Liu

Rong Jiang

View Times: 115

Update Date: 28 Dec 2023

Table of Contents

Video Upload Options

Confirm

1. Introduction

2. Advanced Persistent Threat Predictive Analytics

References