基于客户-服务器双端去重的Web预取新方法
|
摘要
网络数据量的不断增长和网络内容的不断多样化,使得提高网络访问效率、有效降低网络访问延迟成为当前网络存储领域面临的严峻挑战。Web缓存技术和预取技术是目前解决该难题的有效方法。针对缓存技术受限于命中率、预取技术受限于带宽等诸多问题,在Web预取系统引入数据去重技术,提出一种客户-服务器端双端数据去重的Web预取系统改进策略。首先改进传统预取系统框架,分别引入代理服务器端数据去重模块SDM和客户端数据去重模块CDM以及相关模块;然后采用LBFS算法实现双端数据去重;最后,将该方法应用于预取系统,评价其预取性能。实验结果表明,相对于标准传输,该系统一方面可以减少大约34%的字节传输量,另一方面降低大约8%的用户预期的访问延迟,从而优化预取系统。
The increasing amount of network data and the diversification of network contents make it a serious challenge to improve network access efficiency and reduce network access delay. Web caching technology and prefetching technology are currently effective ways to solve this problem. For the problem that the cache technology is limited by the hit rate and the prefetching technology is limited by the bandwidth,we propose a Web prefetching method based on the client-server double-end data deduplication technology. Firstly,the traditional Web prefetching system is improved. The proxy server data deduplication module(SDM) and the client data deduplication module(CDM) are introduced to implement the client-server dual-end data deduplication. Then,LBFS algorithm is used to achieve double-end data deduplication. Finally,this method is applied to the prefetching system to evaluate its prefetching performance. The experiment shows that compared to standard transmission,on the one hand this system can reduce the amount of bytes transferred by about 34%,and on the other hand,in order to optimize the prefetching system,it can reduce the expected access delay by about 8% of users.
The increasing amount of network data and the diversification of network contents make it a serious challenge to improve network access efficiency and reduce network access delay. Web caching technology and prefetching technology are currently effective ways to solve this problem. For the problem that the cache technology is limited by the hit rate and the prefetching technology is limited by the bandwidth,we propose a Web prefetching method based on the client-server double-end data deduplication technology. Firstly,the traditional Web prefetching system is improved. The proxy server data deduplication module(SDM) and the client data deduplication module(CDM) are introduced to implement the client-server dual-end data deduplication. Then,LBFS algorithm is used to achieve double-end data deduplication. Finally,this method is applied to the prefetching system to evaluate its prefetching performance. The experiment shows that compared to standard transmission,on the one hand this system can reduce the amount of bytes transferred by about 34%,and on the other hand,in order to optimize the prefetching system,it can reduce the expected access delay by about 8% of users.
引文
[1] 程龙泉.基于预测模型和缓存替换策略的网络资源访问研究[J].科技通报,2017,33(10):134-136.
[2] 姚瑶,王战红,石磊.一种基于页面聚类的Web概念化建模新方法[J].微电子学与计算机,2015,32(1):156-160.
[3] ARONOVICHA L,ASHERB R,HARNIK D,et al.Similarity based deduplication with small data chunks[J].Discrete Applied Mathematics,2016,212:10-22.
[4] ZHANG Panfeng,HUANG Ping,HE Xubin,et al.Resemblance and mergence based indexing for high performance data deduplication[J].Journal of Systems and Software,2017,128:11-24.
[5] HIRSCHA M,ISH-SHALOMA A,KLEINB S T.Optimal partitioning of data chunks in deduplication systems[J].Discrete Applied Mathematics,2016,212:104-114.
[6] WIDODO R N S,LIM H,ATIQUZZAMAN M.SDM:smart deduplication for mobile cloud storage[J].Future Generation Computer Systems,2017,70:64-73.
[7] 王闪,谭良.Web大数据环境下的相似重复数据清理[J].计算机工程与设计,2017,38(3):646-651.
[8] 熊忠阳,牙漫,张玉芳.基于网页正文结构和特征串的相似网页去重算法[J].计算机应用,2013,33(2):554-557.
[9] 李超,王树鹏,云晓春,等.一种基于流水线的重复数据删除系统读性能优化方法[J].计算机研究与发展,2013,50(1):90-100.
[10] 孙爱玲,冉禄纯.一种基于重复数据删除的网络文件备份系统设计与实现[J].计算机应用与软件,2014,31(10):86-90.
[11] WIDODO R N S,LIM H,ATIQUZZAMAN Z.A new content-defined chunking algorithm for data deduplication in cloud storage[J].Future Generation Computer Systems,2017,71:145-156.
[12] 王红梅,李芬田,王泽儒.基于滑动窗口数据流频繁项集挖掘模型综述[J].长春工业大学学报,2017,38(5):484-490.
[13] 王歧,卢毓海,刘洋,等.支持模式串动态更新的多模式匹配Karp-Rabin算法[J].计算机工程与应用,2017,53(4):39-44.
[2] 姚瑶,王战红,石磊.一种基于页面聚类的Web概念化建模新方法[J].微电子学与计算机,2015,32(1):156-160.
[3] ARONOVICHA L,ASHERB R,HARNIK D,et al.Similarity based deduplication with small data chunks[J].Discrete Applied Mathematics,2016,212:10-22.
[4] ZHANG Panfeng,HUANG Ping,HE Xubin,et al.Resemblance and mergence based indexing for high performance data deduplication[J].Journal of Systems and Software,2017,128:11-24.
[5] HIRSCHA M,ISH-SHALOMA A,KLEINB S T.Optimal partitioning of data chunks in deduplication systems[J].Discrete Applied Mathematics,2016,212:104-114.
[6] WIDODO R N S,LIM H,ATIQUZZAMAN M.SDM:smart deduplication for mobile cloud storage[J].Future Generation Computer Systems,2017,70:64-73.
[7] 王闪,谭良.Web大数据环境下的相似重复数据清理[J].计算机工程与设计,2017,38(3):646-651.
[8] 熊忠阳,牙漫,张玉芳.基于网页正文结构和特征串的相似网页去重算法[J].计算机应用,2013,33(2):554-557.
[9] 李超,王树鹏,云晓春,等.一种基于流水线的重复数据删除系统读性能优化方法[J].计算机研究与发展,2013,50(1):90-100.
[10] 孙爱玲,冉禄纯.一种基于重复数据删除的网络文件备份系统设计与实现[J].计算机应用与软件,2014,31(10):86-90.
[11] WIDODO R N S,LIM H,ATIQUZZAMAN Z.A new content-defined chunking algorithm for data deduplication in cloud storage[J].Future Generation Computer Systems,2017,71:145-156.
[12] 王红梅,李芬田,王泽儒.基于滑动窗口数据流频繁项集挖掘模型综述[J].长春工业大学学报,2017,38(5):484-490.
[13] 王歧,卢毓海,刘洋,等.支持模式串动态更新的多模式匹配Karp-Rabin算法[J].计算机工程与应用,2017,53(4):39-44.