一种分布式的僵尸网络实时检测算法
|
摘要
僵尸网络通过控制的主机实现多类恶意行为,使得当前的检测方法失效,其中窃取敏感数据已经成为主流。鉴于僵尸网络实现的恶意行为,检测和减轻方法的研究已经势在必行。提出了一种新颖的分布式实时僵尸网络检测方法,该方法通过将Netflow组织成主机Netflow图谱和主机关系链,并提取隐含的C&C通信特征来检测僵尸网络。同时,基于Spark Streaming分布式实时流处理引擎,使用该算法实现了BotScanner分布式检测系统。为了验证该系统的有效性,采用5个主流的僵尸网络家族进行训练,并分别使用模拟网络流量和真实网络流量进行测试。实验结果表明,在无需深度包解析的情况下,BotScanner分布式检测系统能够实时检测指定的僵尸网络,并获得了较高的检测率和较低的误报率。而且,在真实的网络环境中,BotScanner分布式检测系统能够进行实时检测,加速比接近线性,验证了Spark Streaming引擎在分布式流处理方面的优势,以及用于僵尸网络检测方面的可行性。
Compared with other types of malware,botnets have recently been adopted by hackers for their resiliency against take-down efforts.Besides being harder to take down,modern botnets tend to be stealthier in the way they perform malicious activities by using the infected computer,making current detection approaches ineffective.Given the malicious activities botnets can realize,detection and mitigation of botnet threats are imperative.In this paper,we presented a novel approach for botnet detection,called distributed real-time botnet detection algorithm.It uses Spark engine,where Netflow related data are correlated as the host Netflow graph structure and the host access chain structure,and a feature extraction method based on the Spark Streaming is leveraged for exacting implicit characteristics.Meanwhile,this paper established distributed BotScanner detection system based on the Spark Streaming,which is a distributed real-time steam processing engine.We trained BotScanner system on the five representative bot families and evaluated BotScanner on simulated network traffic and real-world network traffic.The experimental results show that the BotScanner is able to detect bots in network traffic without the need of deep packet inspection,and achieves high detection rates with very few false positives.When the traffic data from the Internet service provider are very large,the BotScanner is able to detect botnets in real-time by adding the compute nodes,and BotScanner has approximate linear speedup.It proves the feasibility of Applying Spark Streaming engine to distributed botnet detection.
Compared with other types of malware,botnets have recently been adopted by hackers for their resiliency against take-down efforts.Besides being harder to take down,modern botnets tend to be stealthier in the way they perform malicious activities by using the infected computer,making current detection approaches ineffective.Given the malicious activities botnets can realize,detection and mitigation of botnet threats are imperative.In this paper,we presented a novel approach for botnet detection,called distributed real-time botnet detection algorithm.It uses Spark engine,where Netflow related data are correlated as the host Netflow graph structure and the host access chain structure,and a feature extraction method based on the Spark Streaming is leveraged for exacting implicit characteristics.Meanwhile,this paper established distributed BotScanner detection system based on the Spark Streaming,which is a distributed real-time steam processing engine.We trained BotScanner system on the five representative bot families and evaluated BotScanner on simulated network traffic and real-world network traffic.The experimental results show that the BotScanner is able to detect bots in network traffic without the need of deep packet inspection,and achieves high detection rates with very few false positives.When the traffic data from the Internet service provider are very large,the BotScanner is able to detect botnets in real-time by adding the compute nodes,and BotScanner has approximate linear speedup.It proves the feasibility of Applying Spark Streaming engine to distributed botnet detection.
引文
[1]Zhuge J W,Han X H,Zhou Y L,et al.Research and development of botnets[J].Journal of Software,2008,19(3):702-715(in Chinese)诸葛建伟,韩心慧,周勇林,等.僵尸网络研究[J].软件学报,2008,19(3):702-715
[2]Gu G,Porras P,Yegneswaran V,et al.BotHunter:Detecting Malware Infection Through IDS-Driven Dialog Correlation[C]∥16th Usenix Security Symposium.2007
[3]Goebel J,Holz T.Rishi:Identify bot contaminated hosts by irc nickname evaluation[C]∥Proceedings of the first conference on First Workshop on Hot Topics in Understanding Botnets.2007:8
[4]Binkley J R,Singh S.An algorithm for anomaly-based botnet detection[C]∥Proceedings of USENIX Steps to Reducing Unwanted Traffic on the Internet Workshop(SRUTI).2006:43-48
[5]Wurzinger P,Bilge L,Holz T,et al.Automatically generating models for botnet detection[M]∥Computer Security-ESORICS2009.Springer Berlin Heidelberg,2009:232-249
[6]Perdisci R,Lee W,Feamster N.Behavioral Clustering of HTTPBased Malware and Signature Generation Using Malicious Network Traces[C]∥NSDI.2010:391-404
[7]Giroire F,Chandrashekar J,Taft N,et al.Exploiting temporal persistence to detect covert botnet channels[M]∥Recent Advances in Intrusion Detection.Springer Berlin Heidelberg,2009:326-345
[8]Gu G,Zhang J,Lee W.BotSniffer:Detecting Botnet Command and Control Channels in Network Traffic[C]∥15th Annual Network and Distributed System Security Symposium(NDSS).2008
[9]Gu G,Perdisci R,Zhang J,et al.BotMiner:Clustering Analysis of Network Traffic for Protocol-and Structure-Independent Botnet Detection[C]∥USENIX Security Symposium.2008:139-154
[10]Yen T F,Reiter M K.Traffic aggregation for malware detection[M]∥Detection of Intrusions and Malware,and Vulnerability Assessment.Springer Berlin Heidelberg,2008:207-227
[11]Strayer W T,Walsh R,Livadas C,et al.Detecting botnets with tight command and control[C]∥Proceedings of the 31st IEEE Conference on Local Computer Networks.2006:195-202
[12]Nagaraja S,Mittal P,Hong C Y,et al.BotGrep:Finding P2P Bots with Structured Graph Analysis[C]∥USENIX Security Symposium.2010:95-110
[13]Franois J,Wang S,Engel T.BotTrack:tracking botnets using NetFlow and PageRank[M]∥NETWORKING 2011.Springer Berlin Heidelberg,2011:1-14
[14]Tegeler F,Fu X,Vigna G,et al.Botfinder:Finding bots in network traffic without deep packet inspection[C]∥Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies.ACM,2012:349-360
[15]Coskun B,Dietrich S,Memon N.Friends of an enemy:identifying local members of peer-to-peer botnets using mutual contacts[C]∥Proceedings of the 26th Annual Computer Security Applications Conference.ACM,2010:131-140
[16]Kheir N,Wolley C.BotSuer:Suing stealthy P2Pbots in network traffic through netflow analysis[M]∥Cryptology and Network Security.Springer International Publishing,2013:162-178
[17]Fan Y,Xu N.A P2PBotnet Detection Method Used On-line Monitoring and Off-line Detection[J].International Journal of Security&Its Applications,2014,8(3):87-96
[18]Amini P,Azmi R,Araghizadeh M A.Botnet Detection using NetFlow and Clustering[J].Advances in Computer Science:an International Journal,2014,3(2):139-149
[19]Garg S,Sarje A K,Peddoju S K.Improved Detection of P2PBotnets through Network Behavior Analysis[M]∥Recent Trends in Computer Networks and Distributed Systems Security.Springer Berlin Heidelberg,2014:334-345
[20]Vania J,Meniya A,Jethva H B.A Review on Botnet and Detection Technique[J].International Journal of Computer Trends and Technology,2013,4(1):23-29
[21]Zhao Y,Xie Y,Yu F,et al.BotGraph:Large Scale Spamming Botnet Detection[C]∥NSDI.2009,9:321-334
[22]Jiang Hong-ling,Shao Xiu-li,Li Yao-fang.Online Botnet Detection Algorithm Using MapReduce[J].Journal of Electronics&Information Technology,2013,35(7):1732-1738(in Chinese)蒋鸿玲,邵秀丽,李耀芳.基于MapReduce的僵尸网络在线检测算法[J].电子与信息学报,2013,35(7):1732-1738
[23]Batcher K E.Design of a massively parallel processor[J].IEEE Transactions on Computers,1980,100(9):836-840
[24]Gropp W,Lusk E,Doss N,et al.A high-performance,portable implementation of the MPI message passing interface standard[J].Parallel Computing,1996,22(6):789-828
[25]Geist A,Beguelin A,Dongarra J,et al.PVM:Parallel virtual machine-a users’guide and tutorial for networked parallel computing[M].MIT press,1994
[26]Zaharia M,Chowdhury M,Franklin M J,et al.Spark:cluster computing with working sets[C]∥Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing.2010:10
[27]Dean J,Ghemawat S.MapReduce:simplified data processing on large clusters[J].Communications of the ACM,2008,51(1):107-113
[28]Bahmani B,Moseley B,Vattani A,et al.Scalable k-means++[J].Proceedings of the VLDB Endowment,2012,5(7):622-633
[2]Gu G,Porras P,Yegneswaran V,et al.BotHunter:Detecting Malware Infection Through IDS-Driven Dialog Correlation[C]∥16th Usenix Security Symposium.2007
[3]Goebel J,Holz T.Rishi:Identify bot contaminated hosts by irc nickname evaluation[C]∥Proceedings of the first conference on First Workshop on Hot Topics in Understanding Botnets.2007:8
[4]Binkley J R,Singh S.An algorithm for anomaly-based botnet detection[C]∥Proceedings of USENIX Steps to Reducing Unwanted Traffic on the Internet Workshop(SRUTI).2006:43-48
[5]Wurzinger P,Bilge L,Holz T,et al.Automatically generating models for botnet detection[M]∥Computer Security-ESORICS2009.Springer Berlin Heidelberg,2009:232-249
[6]Perdisci R,Lee W,Feamster N.Behavioral Clustering of HTTPBased Malware and Signature Generation Using Malicious Network Traces[C]∥NSDI.2010:391-404
[7]Giroire F,Chandrashekar J,Taft N,et al.Exploiting temporal persistence to detect covert botnet channels[M]∥Recent Advances in Intrusion Detection.Springer Berlin Heidelberg,2009:326-345
[8]Gu G,Zhang J,Lee W.BotSniffer:Detecting Botnet Command and Control Channels in Network Traffic[C]∥15th Annual Network and Distributed System Security Symposium(NDSS).2008
[9]Gu G,Perdisci R,Zhang J,et al.BotMiner:Clustering Analysis of Network Traffic for Protocol-and Structure-Independent Botnet Detection[C]∥USENIX Security Symposium.2008:139-154
[10]Yen T F,Reiter M K.Traffic aggregation for malware detection[M]∥Detection of Intrusions and Malware,and Vulnerability Assessment.Springer Berlin Heidelberg,2008:207-227
[11]Strayer W T,Walsh R,Livadas C,et al.Detecting botnets with tight command and control[C]∥Proceedings of the 31st IEEE Conference on Local Computer Networks.2006:195-202
[12]Nagaraja S,Mittal P,Hong C Y,et al.BotGrep:Finding P2P Bots with Structured Graph Analysis[C]∥USENIX Security Symposium.2010:95-110
[13]Franois J,Wang S,Engel T.BotTrack:tracking botnets using NetFlow and PageRank[M]∥NETWORKING 2011.Springer Berlin Heidelberg,2011:1-14
[14]Tegeler F,Fu X,Vigna G,et al.Botfinder:Finding bots in network traffic without deep packet inspection[C]∥Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies.ACM,2012:349-360
[15]Coskun B,Dietrich S,Memon N.Friends of an enemy:identifying local members of peer-to-peer botnets using mutual contacts[C]∥Proceedings of the 26th Annual Computer Security Applications Conference.ACM,2010:131-140
[16]Kheir N,Wolley C.BotSuer:Suing stealthy P2Pbots in network traffic through netflow analysis[M]∥Cryptology and Network Security.Springer International Publishing,2013:162-178
[17]Fan Y,Xu N.A P2PBotnet Detection Method Used On-line Monitoring and Off-line Detection[J].International Journal of Security&Its Applications,2014,8(3):87-96
[18]Amini P,Azmi R,Araghizadeh M A.Botnet Detection using NetFlow and Clustering[J].Advances in Computer Science:an International Journal,2014,3(2):139-149
[19]Garg S,Sarje A K,Peddoju S K.Improved Detection of P2PBotnets through Network Behavior Analysis[M]∥Recent Trends in Computer Networks and Distributed Systems Security.Springer Berlin Heidelberg,2014:334-345
[20]Vania J,Meniya A,Jethva H B.A Review on Botnet and Detection Technique[J].International Journal of Computer Trends and Technology,2013,4(1):23-29
[21]Zhao Y,Xie Y,Yu F,et al.BotGraph:Large Scale Spamming Botnet Detection[C]∥NSDI.2009,9:321-334
[22]Jiang Hong-ling,Shao Xiu-li,Li Yao-fang.Online Botnet Detection Algorithm Using MapReduce[J].Journal of Electronics&Information Technology,2013,35(7):1732-1738(in Chinese)蒋鸿玲,邵秀丽,李耀芳.基于MapReduce的僵尸网络在线检测算法[J].电子与信息学报,2013,35(7):1732-1738
[23]Batcher K E.Design of a massively parallel processor[J].IEEE Transactions on Computers,1980,100(9):836-840
[24]Gropp W,Lusk E,Doss N,et al.A high-performance,portable implementation of the MPI message passing interface standard[J].Parallel Computing,1996,22(6):789-828
[25]Geist A,Beguelin A,Dongarra J,et al.PVM:Parallel virtual machine-a users’guide and tutorial for networked parallel computing[M].MIT press,1994
[26]Zaharia M,Chowdhury M,Franklin M J,et al.Spark:cluster computing with working sets[C]∥Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing.2010:10
[27]Dean J,Ghemawat S.MapReduce:simplified data processing on large clusters[J].Communications of the ACM,2008,51(1):107-113
[28]Bahmani B,Moseley B,Vattani A,et al.Scalable k-means++[J].Proceedings of the VLDB Endowment,2012,5(7):622-633