贝叶斯算法在垃圾邮件过滤中的应用研究
|
摘要
电子邮件是网络中最方便的应用程序之一,如今它已经在网络中成为人们沟通交流的便捷方式。但是,垃圾邮件的出现对互联网造成了很大的安全威胁和资源滥用。起初,垃圾邮件主要是一些来路不明的商业宣传电子邮件,而现在,更多与色情、政治、诈骗相关的垃圾邮件在网络中不断蔓延,而且呈不断持续增长的态势。另外,计算机病毒和木马以垃圾邮件为途径进行大量的传播,危及网络安全。
垃圾邮件的影响对于广大的互联网使用者是很大的,一方面是垃圾邮件需要人们利用时间来处理并且垃圾还占据大量系统和网络资源；另外,很多的安全问题也从垃圾邮件中产生。大量网络资源被垃圾邮件占用,这是毋庸置疑的。安全性差的邮件服务器因为被垃圾邮件作为转发站而被域名机构封堵IP、列入黑名单的情况时有发生,网络资源的大量消耗使得网络利用率和速度降低,对人们正常业务的开展造成影响。随着反垃圾邮件技术在国际上的不断发展,一些正常的邮件服务器被当做垃圾邮件器大范围屏蔽,造成严重的用户使用问题。病毒、木马结合垃圾邮件的黑客攻击和也越来越密切,随着垃圾邮件类型的不断变化,支持垃圾邮件的恶意程序和控制软件等黑客攻击手段也呈急速上升趋势。很多捆绑了木马病毒的邮件、伪造的钓鱼邮件越来越具有欺骗性,让很多企业和个人受到危害,普通互联网用户很难正确判断邮件的合法性和非法性,因此垃圾邮件直接对终端个人用户造成了损失。
本文通过阐述数据挖掘的不同方法,对不同的分类方法进行综合比较,考虑影响垃圾邮件过滤的各种因素,以朴素贝叶斯算法为基础形成一个邮件文本分类器。通过实验采用朴素贝叶斯分类交叉验证方法,以收集的一些邮件文本为词组,通过训练对测试集中的邮件文本进行分类判断,最终形成了一个较为高效的数据过滤实验模型。同时,结合数据挖掘工具weka证明,在不断对贝叶斯分类进行动态调整之后可使分类结果达到最优。
E-mail is one of the most popular network applications, network communication has become an important way. But spam (spam) is being abused and cause great harm on the Internet. Initially, spam was mainly a number of unsolicited commercial advertising e-mail, and now more about sex, politics, increasing spam, junk mail and even reached the total amount of 40%, and still growing the trend. On the other hand, spam has become a computer virus, new, fast transmission.
Spam can bring a great impact for the majority of users on the Internet, this impact is not just that people take time to deal with spam, and other system resources, it also brings a lot of security issues. Spam takes up a lot of network resources, it is obvious. Some mail servers because of poor security, spam relay station as to be warned, blocked IP and other incidents have occurred, making the network resources consumed by a large number of normal business operations become slow. With the development of international anti-spam, blacklist sharing between organizations, making the innocent server is a wider screen, which undoubtedly will give normal users to use cause serious problems. Spam and hacker attacks, viruses, etc. more and more closely combined with the evolution of spam with malicious code or monitoring software to support the spam has been significantly increased. More deceptive e-mail virus, so many companies suffer, for the ordinary user, it is difficult to make the right judgments, but the loss is very straightforward.
This paper describes the different data mining methods, a comprehensive comparison of different classification methods, consider the impact of various factors, spam filtering, a Bayesian classifier to form the message text data filtering model. Bayesian classification by experiments using cross-validation method to collect some of the message text for the phrase, by training on the test set to determine the classification of text messages, and ultimately resulted in a more efficient experimental data. Meanwhile, the combination of data mining tools that weka, constantly on the Bayesian classifier can be dynamically adjusted to achieve optimal classification results.
垃圾邮件的影响对于广大的互联网使用者是很大的,一方面是垃圾邮件需要人们利用时间来处理并且垃圾还占据大量系统和网络资源；另外,很多的安全问题也从垃圾邮件中产生。大量网络资源被垃圾邮件占用,这是毋庸置疑的。安全性差的邮件服务器因为被垃圾邮件作为转发站而被域名机构封堵IP、列入黑名单的情况时有发生,网络资源的大量消耗使得网络利用率和速度降低,对人们正常业务的开展造成影响。随着反垃圾邮件技术在国际上的不断发展,一些正常的邮件服务器被当做垃圾邮件器大范围屏蔽,造成严重的用户使用问题。病毒、木马结合垃圾邮件的黑客攻击和也越来越密切,随着垃圾邮件类型的不断变化,支持垃圾邮件的恶意程序和控制软件等黑客攻击手段也呈急速上升趋势。很多捆绑了木马病毒的邮件、伪造的钓鱼邮件越来越具有欺骗性,让很多企业和个人受到危害,普通互联网用户很难正确判断邮件的合法性和非法性,因此垃圾邮件直接对终端个人用户造成了损失。
本文通过阐述数据挖掘的不同方法,对不同的分类方法进行综合比较,考虑影响垃圾邮件过滤的各种因素,以朴素贝叶斯算法为基础形成一个邮件文本分类器。通过实验采用朴素贝叶斯分类交叉验证方法,以收集的一些邮件文本为词组,通过训练对测试集中的邮件文本进行分类判断,最终形成了一个较为高效的数据过滤实验模型。同时,结合数据挖掘工具weka证明,在不断对贝叶斯分类进行动态调整之后可使分类结果达到最优。
E-mail is one of the most popular network applications, network communication has become an important way. But spam (spam) is being abused and cause great harm on the Internet. Initially, spam was mainly a number of unsolicited commercial advertising e-mail, and now more about sex, politics, increasing spam, junk mail and even reached the total amount of 40%, and still growing the trend. On the other hand, spam has become a computer virus, new, fast transmission.
Spam can bring a great impact for the majority of users on the Internet, this impact is not just that people take time to deal with spam, and other system resources, it also brings a lot of security issues. Spam takes up a lot of network resources, it is obvious. Some mail servers because of poor security, spam relay station as to be warned, blocked IP and other incidents have occurred, making the network resources consumed by a large number of normal business operations become slow. With the development of international anti-spam, blacklist sharing between organizations, making the innocent server is a wider screen, which undoubtedly will give normal users to use cause serious problems. Spam and hacker attacks, viruses, etc. more and more closely combined with the evolution of spam with malicious code or monitoring software to support the spam has been significantly increased. More deceptive e-mail virus, so many companies suffer, for the ordinary user, it is difficult to make the right judgments, but the loss is very straightforward.
This paper describes the different data mining methods, a comprehensive comparison of different classification methods, consider the impact of various factors, spam filtering, a Bayesian classifier to form the message text data filtering model. Bayesian classification by experiments using cross-validation method to collect some of the message text for the phrase, by training on the test set to determine the classification of text messages, and ultimately resulted in a more efficient experimental data. Meanwhile, the combination of data mining tools that weka, constantly on the Bayesian classifier can be dynamically adjusted to achieve optimal classification results.
引文
[1]英国SOPHOS安全公司.[EB].2010年信息安全威胁报告[EB]http://www.sophos. com/en-us/security-news-trends/security-trends/security-report-2010.a spx.2010-2-1.
[2]美国McAfee实验室.2011年第一季度信息安全威胁报告[EB].http://www.mcafee. con/us/resources/reports/rp-quarterly-threat-q1-2011.pdf.2011-3-20.
[3]Nick Kelly,图像垃圾邮件：新的电子邮件灾难[EB].www.emanu.cn/upload/McAf ee/ImageSpam.pdf.2007-9-15.
[4]梁雪松.基于SMTP会话层的垃圾邮件行为识别技术[J].信息技术.2010.4.
[5]曾小宁.一种新的垃圾邮件过滤技术的研究与实现[J].计算机应用与软件.2009.6.
[6]孙吉谭.基于内容的垃圾邮件意图分析方法研究[C].吉林：吉林大学,2011.
[7]黄明生、杨巍.一种设置多层反垃圾网关框架的方法200510114441[P].2005.
[8]王娜.weka实验指南[EB].http://wenku.baidu.com/view/3fd5f5d6195f312b3169 a5cc.html?from=rec&pos=O&weight=10&lastweight=5&count=5.2011.1
[9]廖芹,郝志峰.数据挖掘与教学建模[M].北京：国防工业出版社,2010.
[10]喻钢,周定康.联机分析处理(OLAP)技术的研究[J].计算机应用.2001.
[11]赵芳,马玉磊.浅析数据挖掘技术的发展及应用[J].新乡学院学报.2010.
[12]刘伟,孟小峰.Deep Web数据集成问题研究[J]WAMDM Technical Report,2006.
[13]范广玲,李春生.数据挖掘模型选择的通用建模研究[J].科学技术与工程.2011-4.
[14]徐传胜.托马斯·贝叶斯的逆概率思想研究[J].西北大学学报,2009-2.
[15]程春明.数据挖掘-决策树[EB].http://wenku.baidu.com/view/Oc85e76527d324 Oc8447eff5.html.2010-4.
[16]文贵华,向君.基于理论的大规模S V M分类算法[J].2008-8.
[17]李建中.反传神经网络的二分法使用[J].2005-1.
[18]白洁,吴渝.基于多层自组织映射和主成分分析的入侵检测方法[J].2007-1.
[19]崔学忠.基于Hopfield网解决T S P问题[J].2008-7.
[20]凌瑞良.玻尔兹曼统计分布律推导的新方法[J].计算机科学.1992-3.
[21]周志华.适应谐振理论综述[J].计算机科学.1999-3.
[22]闭小梅,闭瑞华.KNN算法综述[J].科技创新导报.2009-8.
[23]石洪波.贝叶斯分类方法研究[M].北京：中国科技出版社.2005-3.
[24]王双成.贝叶斯网络学习、推理与应用.北京：立信会计出版社.2010-2.
[25]王峻.一种基于属性相关性度量的朴素贝叶斯分类模型[J].2007-2.
[26]尹蔷.谈谈P和NP问题[J].大连教育学院学报.2005-4.
[27]姜丹.信息论与编码[M].北京：中国科学技术大学出版社.2009-12.
[28]搜狗实验室.一个较大规模的标准中文文本分类测试平台文本分类语料库[EB].http://www.sogou.com/labs/dl/c.html.
[29]张艳,李毅.垃圾邮件与反垃圾邮件新技术追踪[J].全国计算机安全学术交流会论文集.2010.
[2]美国McAfee实验室.2011年第一季度信息安全威胁报告[EB].http://www.mcafee. con/us/resources/reports/rp-quarterly-threat-q1-2011.pdf.2011-3-20.
[3]Nick Kelly,图像垃圾邮件：新的电子邮件灾难[EB].www.emanu.cn/upload/McAf ee/ImageSpam.pdf.2007-9-15.
[4]梁雪松.基于SMTP会话层的垃圾邮件行为识别技术[J].信息技术.2010.4.
[5]曾小宁.一种新的垃圾邮件过滤技术的研究与实现[J].计算机应用与软件.2009.6.
[6]孙吉谭.基于内容的垃圾邮件意图分析方法研究[C].吉林：吉林大学,2011.
[7]黄明生、杨巍.一种设置多层反垃圾网关框架的方法200510114441[P].2005.
[8]王娜.weka实验指南[EB].http://wenku.baidu.com/view/3fd5f5d6195f312b3169 a5cc.html?from=rec&pos=O&weight=10&lastweight=5&count=5.2011.1
[9]廖芹,郝志峰.数据挖掘与教学建模[M].北京：国防工业出版社,2010.
[10]喻钢,周定康.联机分析处理(OLAP)技术的研究[J].计算机应用.2001.
[11]赵芳,马玉磊.浅析数据挖掘技术的发展及应用[J].新乡学院学报.2010.
[12]刘伟,孟小峰.Deep Web数据集成问题研究[J]WAMDM Technical Report,2006.
[13]范广玲,李春生.数据挖掘模型选择的通用建模研究[J].科学技术与工程.2011-4.
[14]徐传胜.托马斯·贝叶斯的逆概率思想研究[J].西北大学学报,2009-2.
[15]程春明.数据挖掘-决策树[EB].http://wenku.baidu.com/view/Oc85e76527d324 Oc8447eff5.html.2010-4.
[16]文贵华,向君.基于理论的大规模S V M分类算法[J].2008-8.
[17]李建中.反传神经网络的二分法使用[J].2005-1.
[18]白洁,吴渝.基于多层自组织映射和主成分分析的入侵检测方法[J].2007-1.
[19]崔学忠.基于Hopfield网解决T S P问题[J].2008-7.
[20]凌瑞良.玻尔兹曼统计分布律推导的新方法[J].计算机科学.1992-3.
[21]周志华.适应谐振理论综述[J].计算机科学.1999-3.
[22]闭小梅,闭瑞华.KNN算法综述[J].科技创新导报.2009-8.
[23]石洪波.贝叶斯分类方法研究[M].北京：中国科技出版社.2005-3.
[24]王双成.贝叶斯网络学习、推理与应用.北京：立信会计出版社.2010-2.
[25]王峻.一种基于属性相关性度量的朴素贝叶斯分类模型[J].2007-2.
[26]尹蔷.谈谈P和NP问题[J].大连教育学院学报.2005-4.
[27]姜丹.信息论与编码[M].北京：中国科学技术大学出版社.2009-12.
[28]搜狗实验室.一个较大规模的标准中文文本分类测试平台文本分类语料库[EB].http://www.sogou.com/labs/dl/c.html.
[29]张艳,李毅.垃圾邮件与反垃圾邮件新技术追踪[J].全国计算机安全学术交流会论文集.2010.