Multi-level Sandboxing Techniques for Execution-based Stealthy Malware Detection
详细信息
- 作者：Liu ; Lei ; Ph.D.
- 学历：Ph.D.
- 年：2011
- 关键词：Malware detection ; Web browsers ; Computer security
- 导师：Chen, Songqing
- 毕业院校：George Mason University
- 专业：Computer Engineering, Computer science
- ISBN：9781124623443
- CBH：3455188
- Country：USA
- 语种：English
- FileSize：843074
- Pages：126
文摘
These days all kinds of malware are pervasive on the Internet. Compared to their ancestors that were commonly used for vandalism or demonstration of skills, modern malware, such as Bots, are driven by the underground economics. Often consisting of hundreds to thousands of bots, botnets are one of the most serious threats on the Internet, responsible for various attacks, such as spamming and distributed denial of service (DDoS). As web browsers are the main interface for the majority of Internet users to surf the Internet today, many of such stealthy malware seek to invade via web browsers in the form of browser helper objects (BHO) and browser toolbars.
To defend against Internet malware, existing schemes mainly rely on either signature-based or anomaly-based detection approaches. Signature-based detection is effective for known malware if the malware signature has been generated. However, the effectiveness of signature-based schemes is challenged by polymorphism, metamorphism, obfuscation, encryption, and other techniques. Moreover, signature-based schemes do not work for zero-day (or unknown) malware. On the other hand, anomaly-based detection schemes seek to detect behavior patterns that do not conform to the established normal patterns. Anomaly-based detection schemes do not require malware signatures. However, modern computer software and systems are often complicated, building and analyzing a comprehensive behavior model is time consuming and even impractical.
To overcome these challenges, we propose a novel execution-based approach for stealthy malware detection. In order to facilitate such run-time detection, we aim to design and implement multi-level sandboxing techniques to create controlled running environments to execute testing programs so that their behaviors can be closely observed and analyzed.
First, we leverage virtual machines for OS-level sandboxing to detect bots on individual hosts. By cloning the host image to a virtual machine and screening user input on the virtual machine, the detection noise is significantly reduced. We find that a typical bot exhibits three invariant features along its onset: (1) the startup of a bot is automatic without requiring any user actions; (2) a bot must establish a command and control channel with its botmaster; and (3) a bot will perform local or remote attacks sooner or later. These invariants indicate three indispensable phases (startup, preparation, and attack) for a bot attack. Thus, we propose [Special characters omitted.]
to detect these three phases with the assistance of OS-level sandboxing techniques. To validate [Special characters omitted.]
, we implement a prototype of [Special characters omitted.]
based on [Special characters omitted.]
. The results show that [Special characters omitted.]
can successfully detect all the bots in the experiments. However, [Special characters omitted.]
may slightly degrade the user performance. Furthermore, advanced malware could evade [Special characters omitted.]
by performing virtual machine fingerprinting.
Second, to overcome the limitations of OS-level sandboxes, we build [Special characters omitted.]
based on process-level sandboxes for malware detection. The key of [Special characters omitted.]
is to defeat malware anti-detection mechanisms at startup and runtime so that malware behaviors during execution can be accurately captured and distinguished. For analysis, [Special characters omitted.]
always starts a copy, referred to as a shadow process, of any suspicious process in the process-level sandbox by defeating all startup anti-detection mechanisms employed in the suspicious process. To defeat internal runtime anti-detection attempts, [Special characters omitted.]
further makes this shadow process mutually invisible to the original suspicious process. To defeat external anti-detection attempts, [Special characters omitted.]
makes as if the shadow process runs on a different machine to the outside. Since ultimately malware will conduct local information harvesting or dispersion, [Special characters omitted.]
constantly monitors the shadow processs behaviors and adopts a hybrid scheme for its behavior analysis. In our experiments, [Special characters omitted.]
can accurately detect all malware samples that employ various anti-detection techniques.
Lastly, to detect and contain malicious browser plugins, we develop [Special characters omitted.]
with intra-process sandboxing techniques. With an intra-process sandbox, only plugins are closely monitored for misbehavior detection without confining the entire process. This further reduces the detection overhead while maintaining transparency to end-users. Based on intra-process sandboxing techniques, we build [Special characters omitted.]
to enhance the security of a browser by enforcing security policies on plugins' accessing requests to the browser's internal objects and external system-level resources, such as file systems and network interfaces. [Special characters omitted.]
deals with both native and .NET-based plugins and its unique design renders it possible to work with commodity web browsers without requiring any modifications to the legacy browser architecture or plugin code. We implement [Special characters omitted.]
in Windows XP and IE8.
To defend against Internet malware, existing schemes mainly rely on either signature-based or anomaly-based detection approaches. Signature-based detection is effective for known malware if the malware signature has been generated. However, the effectiveness of signature-based schemes is challenged by polymorphism, metamorphism, obfuscation, encryption, and other techniques. Moreover, signature-based schemes do not work for zero-day (or unknown) malware. On the other hand, anomaly-based detection schemes seek to detect behavior patterns that do not conform to the established normal patterns. Anomaly-based detection schemes do not require malware signatures. However, modern computer software and systems are often complicated, building and analyzing a comprehensive behavior model is time consuming and even impractical.
To overcome these challenges, we propose a novel execution-based approach for stealthy malware detection. In order to facilitate such run-time detection, we aim to design and implement multi-level sandboxing techniques to create controlled running environments to execute testing programs so that their behaviors can be closely observed and analyzed.
First, we leverage virtual machines for OS-level sandboxing to detect bots on individual hosts. By cloning the host image to a virtual machine and screening user input on the virtual machine, the detection noise is significantly reduced. We find that a typical bot exhibits three invariant features along its onset: (1) the startup of a bot is automatic without requiring any user actions; (2) a bot must establish a command and control channel with its botmaster; and (3) a bot will perform local or remote attacks sooner or later. These invariants indicate three indispensable phases (startup, preparation, and attack) for a bot attack. Thus, we propose [Special characters omitted.]
to detect these three phases with the assistance of OS-level sandboxing techniques. To validate [Special characters omitted.]
, we implement a prototype of [Special characters omitted.]
based on [Special characters omitted.]
. The results show that [Special characters omitted.]
can successfully detect all the bots in the experiments. However, [Special characters omitted.]
may slightly degrade the user performance. Furthermore, advanced malware could evade [Special characters omitted.]
by performing virtual machine fingerprinting.
Second, to overcome the limitations of OS-level sandboxes, we build [Special characters omitted.]
based on process-level sandboxes for malware detection. The key of [Special characters omitted.]
is to defeat malware anti-detection mechanisms at startup and runtime so that malware behaviors during execution can be accurately captured and distinguished. For analysis, [Special characters omitted.]
always starts a copy, referred to as a shadow process, of any suspicious process in the process-level sandbox by defeating all startup anti-detection mechanisms employed in the suspicious process. To defeat internal runtime anti-detection attempts, [Special characters omitted.]
further makes this shadow process mutually invisible to the original suspicious process. To defeat external anti-detection attempts, [Special characters omitted.]
makes as if the shadow process runs on a different machine to the outside. Since ultimately malware will conduct local information harvesting or dispersion, [Special characters omitted.]
constantly monitors the shadow processs behaviors and adopts a hybrid scheme for its behavior analysis. In our experiments, [Special characters omitted.]
can accurately detect all malware samples that employ various anti-detection techniques.
Lastly, to detect and contain malicious browser plugins, we develop [Special characters omitted.]
with intra-process sandboxing techniques. With an intra-process sandbox, only plugins are closely monitored for misbehavior detection without confining the entire process. This further reduces the detection overhead while maintaining transparency to end-users. Based on intra-process sandboxing techniques, we build [Special characters omitted.]
to enhance the security of a browser by enforcing security policies on plugins' accessing requests to the browser's internal objects and external system-level resources, such as file systems and network interfaces. [Special characters omitted.]
deals with both native and .NET-based plugins and its unique design renders it possible to work with commodity web browsers without requiring any modifications to the legacy browser architecture or plugin code. We implement [Special characters omitted.]
in Windows XP and IE8.