A rootkit is a malicious program with a highly deceptive nature. It operates by concealing its presence in the system and enabling unauthorized root access to attackers, allowing then to gain complete control over the compromised system. Rootkits are designed to evade detection and this makes it challenging for anti-malware tools to detect their infiltration. Once installed, a system may exhibit unusual behaviour, indicating remote access by an attacker. Rootkits are particularly hazardous and can lead to significant data loss and damage.

The term “rootkit” is derived from the combination of two words, “root” and “kit”. In UNIX and Linux environments, “root” refers to the system administrator who possesses the highest level of access. “Kit” denotes a collection of tools and techniques. Consequently, a rootkit is defined as a set of tools or techniques enabling unauthorized individuals to gain and sustain administrator-level access during an attack while evading detection by authorized users and administrators ^[1].

Rootkits have been a significant concern for security engineers since the late 1980s. Initially, they were primarily used to hide log files and application binaries. The early generation of these malicious software targeted user-level programs, making them relatively easy to detect using simple checksum methods. However, the threat landscape has evolved, and modern rootkits pose more and more danger, as operating systems are now widely employed in various devices such as smartphones, IoT nodes, computers, and embedded devices ^[2][3]. Consequently, researchers are actively working on developing efficient methods to detect these sophisticated rootkits that can evade traditional detection techniques.

The frequency of rootkit attacks is also on the rise. According to reports from Avast, a leading cybersecurity company, the number of users affected by rootkits increased from 10,000 in 2020 to 100,000 in 2021. Therefore, the main objective herein is to devise a reliable and efficient detection technique specifically tailored for rootkits that are challenging to detect.

Rootkits are designed to conceal themselves from system administrators and users, allowing them to operate covertly and pose significant security threats. They pose a threat not only to the host systems, but also to the virtual machines ^[4]. Therefore, there is a crucial requirement for a standardized mechanism that can efficiently detect rootkits.

Memory analysis with tools like Volatility can greatly aid in the detection of rootkits by examining the contents of a system's volatile memory. In this process, a memory dump is acquired from the target system, and Volatility is utilized to parse and interpret the information within. Initial steps involve identifying the profile of the memory dump to ensure compatibility with Volatility's analysis tools. Subsequently, the framework is employed to list running processes, scrutinize loaded kernel modules, and unveil potentially hidden processes or threads. Rootkits often utilize these tactics to remain stealthy on a compromised system. Moreover, the analysis extends to inspecting network connections, registry entries, and files in memory, as rootkits commonly manipulate these elements for persistence and covert operations. By leveraging the capabilities of Volatility, analysts gain a comprehensive view of a system's memory, enabling the identification of irregularities indicative of rootkit activity.

In practical terms, commands such as `pslist`, `modscan`, `dlllist`, `netscan`, `svscan`, `filescan`, and `dumpfiles` are executed with appropriate parameters to extract and analyze relevant information from the memory dump. The output from these commands aids in uncovering any anomalies or indicators of malicious activity. Rootkit detection through memory analysis is an essential aspect of digital forensics and incident response, allowing security professionals to understand and mitigate the impact of these sophisticated forms of malware on compromised systems.

In ^[5], Djenna et al. contribute to the detection of rootkits by integrating dynamic DL-based methods and heuristic approaches within a malware detection framework. By analyzing the behavior patterns and employing advanced detection techniques, their model achieved effective identification and classification of rootkits, as well as enhanced the overall capabilities of the malware detection system.

In ^[6] Sihwail et al. conducted a study on the effectiveness of extracting memory-based images to detect malware. They created a binary memory-based dataset available on GitHub ^[7] and employed classification algorithms such as SVM, RF, k-NN, Naïve Bayes (NB), and DT. Their model achieved an accuracy of 98.5% using the Volatility v2.6 framework.

Bozkir et al. ^[8] utilized computer vision and ML techniques to detect and classify malware by analyzing memory dumps as RGB images. Their approach improved the detection of unknown malware by up to 20.78% across multiple ML algorithms. They employed RF, SVM, and XGBoost algorithms and demonstrated the practicality of computer vision-based schemes for protection against malicious applications.

Another study by ^[9] emphasized the significance of memory analysis in capturing malware footprints and extracting hidden code from obfuscated malware. The authors developed a Python-based plugin called VolMemLyzer for Volatility v2.6 that is capable of extracting 36 features and converting the results into CSV format. The plugin showed high accuracy in malware classification, and a dataset of approximately 1,900 instances was created using the tool.

Addressing the challenges of detecting obfuscated and hidden malware, Carrier et al. ^[10] transformed the VolMemLyzer framework to extract 26 new memory features, enhancing its efficiency ^[10]. The plugin was employed to detect ransomware, Trojan Horses, and spyware, and achieved an accuracy of 99% and an F1-Score of 99.02%. An extended dataset was created, contributing positively to research in this field.

In ^[11], the Trusted Kernel Rootkit Detection (TKRD) system combined memory forensic analysis with bio-inspired ML techniques to detect kernel rootkits. It achieved very high accuracy. In ^[12], a hardware-assisted Virtualization-based Kernel-level Rootkit Detection (VKRD) system was introduced. It employed ML techniques and dynamic analysis to intercept and isolate the operations of kernel modules, albeit with performance overhead. Nagy et al. ^[13] addressed the challenge of detecting rootkits in embedded IoT devices by utilizing dynamic analysis within Trusted Execution Environments (TEE) available in popular IoT platforms.

These studies highlight the significance of memory analysis and various detection techniques, including ML, computer vision, and dynamic analysis, in combating the threat of malware and rootkits. Collectively, these studies reveal that a multi-dimensional approach, combining memory analysis and a range of detection methodologies, holds great promise for addressing the evolving landscape of cyber threats. Table 1 presents a summary of the research discussed in this section.