Detection of Rootkits Using Memory Analysis

Detection of Rootkits Using Memory Analysis: History

Please note this is an old version of this entry, which may differ significantly from the current revision.

Contributor:

Rootkits are malicious programs designed to conceal their activities on compromised systems, making them challenging to detect using conventional methods. As the threat landscape continually evolves, rootkits pose a serious threat by stealthily concealing malicious activities, making their early detection crucial to prevent data breaches and system compromise. A promising strategy for monitoring system activities involves analyzing volatile memory. The process can be made more reliable by the use of Machine learning and deep learning algorithms too.

memory analysis
rootkits
deep learning
machine learning
execution time

1. Introduction

A rootkit is a malicious program with a highly deceptive nature. It operates by concealing its presence in the system and enabling unauthorized root access to attackers, allowing then to gain complete control over the compromised system. Rootkits are designed to evade detection and this makes it challenging for anti-malware tools to detect their infiltration. Once installed, a system may exhibit unusual behaviour, indicating remote access by an attacker. Rootkits are particularly hazardous and can lead to significant data loss and damage.

The term “rootkit” is derived from the combination of two words, “root” and “kit”. In UNIX and Linux environments, “root” refers to the system administrator who possesses the highest level of access. “Kit” denotes a collection of tools and techniques. Consequently, a rootkit is defined as a set of tools or techniques enabling unauthorized individuals to gain and sustain administrator-level access during an attack while evading detection by authorized users and administrators ^[1].

Rootkits have been a significant concern for security engineers since the late 1980s. Initially, they were primarily used to hide log files and application binaries. The early generation of these malicious software targeted user-level programs, making them relatively easy to detect using simple checksum methods. However, the threat landscape has evolved, and modern rootkits pose more and more danger, as operating systems are now widely employed in various devices such as smartphones, IoT nodes, computers, and embedded devices ^[2]^[3]. Consequently, researchers are actively working on developing efficient methods to detect these sophisticated rootkits that can evade traditional detection techniques.

The frequency of rootkit attacks is also on the rise. According to reports from Avast, a leading cybersecurity company, the number of users affected by rootkits increased from 10,000 in 2020 to 100,000 in 2021. Therefore, the main objective herein is to devise a reliable and efficient detection technique specifically tailored for rootkits that are challenging to detect.

Rootkits are designed to conceal themselves from system administrators and users, allowing them to operate covertly and pose significant security threats. They pose a threat not only to the host systems, but also to the virtual machines ^[4]. Therefore, there is a crucial requirement for a standardized mechanism that can efficiently detect rootkits.

Memory analysis with tools like Volatility can greatly aid in the detection of rootkits by examining the contents of a system's volatile memory. In this process, a memory dump is acquired from the target system, and Volatility is utilized to parse and interpret the information within. Initial steps involve identifying the profile of the memory dump to ensure compatibility with Volatility's analysis tools. Subsequently, the framework is employed to list running processes, scrutinize loaded kernel modules, and unveil potentially hidden processes or threads. Rootkits often utilize these tactics to remain stealthy on a compromised system. Moreover, the analysis extends to inspecting network connections, registry entries, and files in memory, as rootkits commonly manipulate these elements for persistence and covert operations. By leveraging the capabilities of Volatility, analysts gain a comprehensive view of a system's memory, enabling the identification of irregularities indicative of rootkit activity.

In practical terms, commands such as `pslist`, `modscan`, `dlllist`, `netscan`, `svscan`, `filescan`, and `dumpfiles` are executed with appropriate parameters to extract and analyze relevant information from the memory dump. The output from these commands aids in uncovering any anomalies or indicators of malicious activity. Rootkit detection through memory analysis is an essential aspect of digital forensics and incident response, allowing security professionals to understand and mitigate the impact of these sophisticated forms of malware on compromised systems.

2. Detection of Rootkits Using Memory Analysis

In ^[5], Djenna et al. contribute to the detection of rootkits by integrating dynamic DL-based methods and heuristic approaches within a malware detection framework. By analyzing the behavior patterns and employing advanced detection techniques, their model achieved effective identification and classification of rootkits, as well as enhanced the overall capabilities of the malware detection system.

In ^[6] Sihwail et al. conducted a study on the effectiveness of extracting memory-based images to detect malware. They created a binary memory-based dataset available on GitHub ^[7] and employed classification algorithms such as SVM, RF, k-NN, Naïve Bayes (NB), and DT. Their model achieved an accuracy of 98.5% using the Volatility v2.6 framework.

Bozkir et al. ^[8] utilized computer vision and ML techniques to detect and classify malware by analyzing memory dumps as RGB images. Their approach improved the detection of unknown malware by up to 20.78% across multiple ML algorithms. They employed RF, SVM, and XGBoost algorithms and demonstrated the practicality of computer vision-based schemes for protection against malicious applications.

Another study by ^[9] emphasized the significance of memory analysis in capturing malware footprints and extracting hidden code from obfuscated malware. The authors developed a Python-based plugin called VolMemLyzer for Volatility v2.6 that is capable of extracting 36 features and converting the results into CSV format. The plugin showed high accuracy in malware classification, and a dataset of approximately 1,900 instances was created using the tool.

Addressing the challenges of detecting obfuscated and hidden malware, Carrier et al. ^[10] transformed the VolMemLyzer framework to extract 26 new memory features, enhancing its efficiency ^[10]. The plugin was employed to detect ransomware, Trojan Horses, and spyware, and achieved an accuracy of 99% and an F1-Score of 99.02%. An extended dataset was created, contributing positively to research in this field.

In ^[11], the Trusted Kernel Rootkit Detection (TKRD) system combined memory forensic analysis with bio-inspired ML techniques to detect kernel rootkits. It achieved very high accuracy. In ^[12], a hardware-assisted Virtualization-based Kernel-level Rootkit Detection (VKRD) system was introduced. It employed ML techniques and dynamic analysis to intercept and isolate the operations of kernel modules, albeit with performance overhead. Nagy et al. ^[13] addressed the challenge of detecting rootkits in embedded IoT devices by utilizing dynamic analysis within Trusted Execution Environments (TEE) available in popular IoT platforms.

These studies highlight the significance of memory analysis and various detection techniques, including ML, computer vision, and dynamic analysis, in combating the threat of malware and rootkits. Collectively, these studies reveal that a multi-dimensional approach, combining memory analysis and a range of detection methodologies, holds great promise for addressing the evolving landscape of cyber threats. Table 1 presents a summary of the research discussed in this section.

Table 1. Comparison of Related Work on Rootkit Detection using Memory Analysis.

Study	Learning Algorithm	Performance Metrics	Dataset	Strengths	Limitations
Djenna et al. ^[5]	DNN, CNN, RF, DT	Accuracy, precision, recall, F1-score	CICAndMal2017 ^[14]	Utilizes dynamic deep learning and heuristic.	Lack of in-depth analysis on potential false positives. Limited exploration of feature engineering.
Sihwail et al. ^[6]	SVM, Naïve Bayes, k-NN, RF, DT	Accuracy, precision, recall, F1-score, False positive rates	Sihwail ^[7]	Utilizes memory features extracted from memory images. Incorporates feature engineering and binary vectors for training and testing.	Potential over-fitting due to high accuracy rate on training data.
Bokzir et al. ^[8]	CNN	Accuracy, precision, recall, F1-score, ROC-AUC	Dampware10 ^[15]	Comprehensive dataset with malware and benign samples. Inclusion of GIST descriptors and HOG features.	Limited explanation of dataset creation process. Limited discussion on feature extraction methods.
Lashkari et al. ^[9]	Adaboost, RF, k-NN, DT	False positives, False negatives, Accuracy, F1- Score, Precision	VolMemLyzer ^[16]	Handles feature selection automatically. Resistant to over-fitting.	Could be computationally expensive.
Carrier et al. ^[10]	RF, DT, k-NN, Naïve Bayes, SVM, Logistic Regression	Accuracy, F1- Score, Precision, Recall	CIC-MalMem-2022 ^[17]	High accuracy and fast classification.	Prone to over-fitting.
Wang et al. ^[11]	RF, DT, Bayesian	TPR, FPR, AUC, F-measure, Accuracy	^[11]	Integration of memory forensics analysis.	Limited exploration of feature selection.

3. Conclusion

In conclusion, the escalating threat landscape posed by rootkits necessitates advanced detection mechanisms, particularly in the face of their evolving sophistication and the increasing frequency of attacks. Memory analysis, exemplified by tools like Volatility, emerges as a crucial aspect of rootkit detection, offering insight into a system's volatile memory to identify irregularities indicative of malicious activity. The comprehensive review of contemporary research underscores the effectiveness of diverse methodologies, including dynamic deep learning, SVM, RF, computer vision, and dynamic analysis, in combating rootkit threats. The summarized comparison of related work provides a valuable reference for understanding the strengths and limitations of different approaches. Collectively, these studies illuminate a multi-dimensional approach, combining memory analysis and various detection methodologies, as a promising strategy to address the complex and evolving challenges posed by rootkits in the dynamic landscape of cybersecurity.

This entry is adapted from the peer-reviewed paper 10.3390/app131910730

References

Manap, S. Rootkit: Attacker Undercover Tools. Available online: http://forum.ouah.org/salirootkit.pdf (accessed on 6 April 2020).
Bickford, J.; O’Hare, R.; Baliga, A.; Ganapathy, V.; Iftode, L. Rootkits on smartphones. In Proceedings of the Eleventh Workshop on Mobile Computing Systems Applications-HotMobile ’10, Annapolis, MD, USA, 22–23 February 2010.
Bunten, A. UNIX and Linux based Rootkits Techniques and Countermeasures. 2004. Available online: https://www.semanticscholar.org/paper/UNIXand-Linux-based-Rootkits-Techniques-and-Bunten/ (accessed on 31 March 2023).
Huseynov, H.; Saadawi, T.; Kourai, K. Hardening the Security of Multi-Access Edge Computing through Bio-Inspired VM Introspection. Big Data Cogn. Comput. 2021, 5, 52.
Djenna, A.; Bouridane, A.; Rubab, S.; Marou, I.M. Artificial Intelligence-Based Malware Detection, Analysis, and Mitigation. Symmetry 2023, 15, 677.
Sihwail, R.; Omar, K.; Ariffin, K.A.Z. An Effective Memory Analysis for Malware Detection and Classification. Comput. Mater. Contin. 2021, 67, 2301–2320.
Sihwail. Sihwail/Malware-Memory-Dataset. GitHub. 5 February 2021. Available online: https://github.com/sihwail/malware-memory-dataset (accessed on 14 April 2023).
Bozkir, A.S.; Tahillioglu, E.; Aydos, M.; Kara, I. Catch them alive: A malware detection approach through memory forensics, manifold learning and computer vision. Comput. Secur. 2021, 103, 102166.
Lashkari, A.H.; Li, B.; Carrier, T.L.; Kaur, G. VolMemLyzer: Volatile Memory Analyzer for Malware Classification using Feature Engineering. In Proceedings of the 2021 Reconciling Data Analytics, Automation, Privacy, and Security: A Big Data Challenge (RDAAPS), Hamilton, ON, Canada, 18–19 May 2021; Available online: https://ieeexplore.ieee.org/document/9452028 (accessed on 23 April 2023).
Carrier, T.; Victor, P.; Tekeoglu, A.; Lashkari, A. Detecting Obfuscated Malware using Memory Feature Engineering. In Proceedings of the 8th International Conference on Information Systems Security and Privacy, Online, 9–11 February 2022.
Wang, X.; Zhang, J.; Zhang, A.; Ren, J. TKRD: Trusted kernel rootkit detection for cybersecurity of VMs based on Machine Learning and memory forensic analysis. Math. Biosci. Eng. 2019, 16, 2650–2667.
Tian, D.; Ma, R.; Jia, X.; Hu, C. A Kernel Rootkit Detection Approach Based on Virtualization and Machine Learning. IEEE Access 2019, 7, 91657–91666.
Nagy, R.; Németh, K.; Papp, D.; Buttyán, L. Rootkit Detection on Embedded IoT Devices. Acta Cybern. 2021, 25, 369–400.
Lashkari, A.H.; Kadir, A.F.A.; Taheri, L.; Ghorbani, A.A. Toward Developing a Systematic Approach to Generate Benchmark Android Malware Datasets and Classification. In Proceedings of the 2018 International Carnahan Conference on Security Technology (ICCST), Montreal, QC, Canada, 22–25 October 2018; pp. 1–7.
Dumpware 10 Dataset Homepage. Available online: https://web.cs.hacettepe.edu.tr/~selman/dumpware10/ (accessed on 17 August 2023).
Volatility Memory Analyzer. 2019. Available online: https://github.com/ahlashkari/VolMemLyzer (accessed on 23 April 2023).
CIC-MalMem-2022. 2022. Available online: https://www.unb.ca/cic/datasets/malmem-2022.html (accessed on 23 April 2023).

© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.