基于网页浏览行为的反爬虫研究
|
摘要
在大数据的背景下,数据的潜在价值被不断地挖掘出来。能够有效识别或阻挡爬取行为的反爬虫方法对于商业服务网站来说尤为重要。基于网页浏览行为,提出一种新的反爬虫方法。该方法通过对真实用户和网络爬虫浏览网页的行为进行特征提取,然后构造并使用决策树对一个用户是否属于爬虫进行预测。该方法对网络爬虫的敏感性高,并具有较低的假阴率。
In the era of the big data, the potential value of data is constantly being explored. Anti-spider methods that effectively identify or block web spider crawling behavior are especially important for websites who offer commercial services. Proposes a new anti-spider method based on the behavior of browsing the web page. The method extracts feature from web browsing behavior of real users and web spider, and then con?structs and uses decision tree to predict whether a user is web spider. This method has high sensitivity to web spider and a low false-nega?tive rate.
In the era of the big data, the potential value of data is constantly being explored. Anti-spider methods that effectively identify or block web spider crawling behavior are especially important for websites who offer commercial services. Proposes a new anti-spider method based on the behavior of browsing the web page. The method extracts feature from web browsing behavior of real users and web spider, and then con?structs and uses decision tree to predict whether a user is web spider. This method has high sensitivity to web spider and a low false-nega?tive rate.
引文
[1]2018 Bad Bot Report[EB/OL]. https://resources.distilnetworks.com/whitepapers/2018-bad-bot-report,2018.
[2]刘庆杰,孙旭光,王小英.通过Filter抵御网页爬虫[J].网络安全技术与应用,2010(1):70-71.
[3]陈利婷.大数据时代的反爬虫技术[J].电脑与信息技术,2016(6):60-61.
[4]顾流,万仲保,石红芹.基于Web页信息隐藏的研究与实现[J].微计算机信息,2006,22(24):186-187.
[5]张宁.群体兴趣网的统计特性研究[J].上海理工大学学报,2008,30(3):243-248.
[6]Witten I H,Frank E. Data Mining:Practical Machine Learning Tools and Techniques(Third Edition)[M].机械工业出版社,2005.
[7]史忠植.知识发现.第2版[M].清华大学出版社,2011.
[2]刘庆杰,孙旭光,王小英.通过Filter抵御网页爬虫[J].网络安全技术与应用,2010(1):70-71.
[3]陈利婷.大数据时代的反爬虫技术[J].电脑与信息技术,2016(6):60-61.
[4]顾流,万仲保,石红芹.基于Web页信息隐藏的研究与实现[J].微计算机信息,2006,22(24):186-187.
[5]张宁.群体兴趣网的统计特性研究[J].上海理工大学学报,2008,30(3):243-248.
[6]Witten I H,Frank E. Data Mining:Practical Machine Learning Tools and Techniques(Third Edition)[M].机械工业出版社,2005.
[7]史忠植.知识发现.第2版[M].清华大学出版社,2011.