基于内存的分布式缓存系统设计
摘要
随着互联网WEB2.0时代的到来,信息传播的广度和深度都达到了一个新的高度。传统的网站都是用数据库来进行数据的存储、检索和更新,但随着访问量的增加,将导致数据库服务器系统资源紧缺,查询操作响应变慢,网站用户体验变差。因此,开发缓存系统具有广泛的应用前景。
本文根据应用中一部分检索数据库的操作具有逻辑相同,请求频繁的特点,结合内存读写速率快的特点,提出并设计了一款分布式内存缓存系统。该系统采用物理内存作为数据存储介质,使用客户端／服务器的软件结构,通过在客户端应用分布式哈希算法实现了分布式存储的功能,并开发了PHP及Python的编程接口,给应用开发提供了一套可供简单调用的高性能存储服务。
本文主要介绍了分布式内存缓存系统的体系结构以及各个层次的具体实现。本设计主要分为以下四个方面的内容：在数据库检索方案性能有限,迫切需要提高网站浏览体验的前提下,调研并提出了分布式内存缓存系统的完整设计方案；介绍了作为系统存储单元的内存缓存系统的设计及主要功能模块；通过采用哈希算法的客户端来将请求分散到不同的内存缓存节点上,达到分布式的效果；最后根据前台开发的特点,实现了PHP及Python的调用接口。
通过将缓存系统读性能和数据库检索的压力测试结果进行对比,表明该系统具有较好的读性能和实用性。
With the advent of WEB2.0era in the Internet revolution, the breadth and depth of information dissemination have reached a new height. Traditional sites use database system for data storage. But with the increase of the visits, the user experience becomes poor. Therefore, the development of the cache system has a wide range of prospect in application.
According to some database search operations have the features of the same logic and frequent request, combined with the memory's characteristics of fast access rate, the thesis proposes and designs a distributed memory cache system. The system uses physical memory as storage, adopts the client and server software architecture, implements the distributed storage functionality by using hash algorithm in the client and develops a high-performance storage services by the PHP and Python programming interface for application developers.
The thesis describes the architecture of the distributed memory cache system, and each level of the system. The design is mainly divided into the following four aspects: because of the limited performance of the database system and an urgent need to improve the website's user experience, research and propose a complete design of the distributed memory cache system; introduce the design of the memory based cache system as the system's storage unit; the request by the client by using a hashing algorithm, make the request to the different memory based cache nodes; Finally, according to the characteristics of the application development, implement the application programming interface of PHP and Python.
By comparing the performance of the cache system with database by running pressure test, indicate that the system has better read performance and practicality.
本文根据应用中一部分检索数据库的操作具有逻辑相同,请求频繁的特点,结合内存读写速率快的特点,提出并设计了一款分布式内存缓存系统。该系统采用物理内存作为数据存储介质,使用客户端／服务器的软件结构,通过在客户端应用分布式哈希算法实现了分布式存储的功能,并开发了PHP及Python的编程接口,给应用开发提供了一套可供简单调用的高性能存储服务。
本文主要介绍了分布式内存缓存系统的体系结构以及各个层次的具体实现。本设计主要分为以下四个方面的内容：在数据库检索方案性能有限,迫切需要提高网站浏览体验的前提下,调研并提出了分布式内存缓存系统的完整设计方案；介绍了作为系统存储单元的内存缓存系统的设计及主要功能模块；通过采用哈希算法的客户端来将请求分散到不同的内存缓存节点上,达到分布式的效果；最后根据前台开发的特点,实现了PHP及Python的调用接口。
通过将缓存系统读性能和数据库检索的压力测试结果进行对比,表明该系统具有较好的读性能和实用性。
With the advent of WEB2.0era in the Internet revolution, the breadth and depth of information dissemination have reached a new height. Traditional sites use database system for data storage. But with the increase of the visits, the user experience becomes poor. Therefore, the development of the cache system has a wide range of prospect in application.
According to some database search operations have the features of the same logic and frequent request, combined with the memory's characteristics of fast access rate, the thesis proposes and designs a distributed memory cache system. The system uses physical memory as storage, adopts the client and server software architecture, implements the distributed storage functionality by using hash algorithm in the client and develops a high-performance storage services by the PHP and Python programming interface for application developers.
The thesis describes the architecture of the distributed memory cache system, and each level of the system. The design is mainly divided into the following four aspects: because of the limited performance of the database system and an urgent need to improve the website's user experience, research and propose a complete design of the distributed memory cache system; introduce the design of the memory based cache system as the system's storage unit; the request by the client by using a hashing algorithm, make the request to the different memory based cache nodes; Finally, according to the characteristics of the application development, implement the application programming interface of PHP and Python.
By comparing the performance of the cache system with database by running pressure test, indicate that the system has better read performance and practicality.
引文
[1]苗雪兰,刘瑞新,王怀峰.数据库系统原理及应用教程[M].机械工业出版社,2004(1)：7-12.
[2]潘名莲,马争,丁庆生.微计算机原理[M].电子工业出版社,2003(9)：118-134.
[3]王晓艳,吕宁,李会娟.基于SQUID+Memcached Web缓存技术的应用与研究[J].2009国际信息技术与应用论坛论文集(上),2009(5).
[4]Abraham Silberschatz, Henry F. Korth, S. Sudarshan.数据库系统概念[M].机械工业出版社,2008(6)：168-186.
[5]David Karger, Alex Sherman, Andy Berkheimer, Bill Bogstad, Rizwan Dhanidina Ken Iwamoto, Brian Kim, Luke Matkins, Yoav Yerushalmi. Web Caching with Consistent Hashing[J]. MIT Laboratory for Computer Science,2009(5).
[6]Marc J. Rochkind.高级UNIX编程[M].机械工业出版社,2006(5)：216-218.
[7]Andrew S. Tanenbaum, Maartenvan Steen.分布式系统原理与范型[M].清华大学出版社,2007(1)：347-352.
[8]丘恩(Chun, W. J.). Python核心编程(第二版)[M].人民邮电出版社,2008(8)：245-258.
[9]米歇尔.精通Unix Shell脚本编程[M].清华大学出版社,2007(12)：312-324.
[10]Zhaoqing Jia, Ruonan Rao, Minglu Li, Jinyuan You. Random Walk Spread and Search In Unstructured P2P[J].The 5th International Conference Parallel and Distributed Computing,2004(11).
[11]斯蒂文斯.UNIX网络编程卷1：套接字联网API(第3版)[M].人民邮电出版社,2004(11)：311-324.
[12]周淦淼,谭石强.基于Memcached的MySQL查询优化[J].现代计算机(专业版),2009(5).
[13]张飞,姜进磊,李庆虎.利用MySQL构建分布式应用[J].计算机工程与应用,2001(11).
[14]赵庆民,李伟平.多处理机系统分析[J].微计算机应用,2005(3).
[15]Stephanos Androutsellis-Theotokis, Diomldis. A survey of peer-to-peer content distribution technologies[J]. ACM Computing Surveys,2004(11).
[16]郭欣.构建高性能Web站点[M].电子工业出版社,2010(2)：467-478.
[17]D. Darger, E. Lehman, T. Leighton, M. Levine, D. Lewin and R. Panigrahy. Consistent Hashing and Random Trees:Distributed Caching Protocols for Relieving Hot Spots On the World Wide Web[J]. ACM Symposium on Theory of Computing:2007(2).
[18]胡志华,王珏明.动态Web Cache系统研究与探讨[J].计算机应用与软件,2006(9).
[19]许艳美,肖宗水,梁异.Web缓存服务器技术研究与应用[J].计算机工程与设计,2005(1).
[20]郝晓艳,孙厚庆,孙奕奇.应用服务器缓存机制构建高效信息平台[J].山东省计算机学会2005年信息技术与信息化研讨会论文集(一),2005(3).
[21]PHP manual.
[22]杜江,PHP 5与MySQL5 Web开发技术详解[M].电子工业出版社,2008(5)：321-337.
[23]Python manual.
[2]潘名莲,马争,丁庆生.微计算机原理[M].电子工业出版社,2003(9)：118-134.
[3]王晓艳,吕宁,李会娟.基于SQUID+Memcached Web缓存技术的应用与研究[J].2009国际信息技术与应用论坛论文集(上),2009(5).
[4]Abraham Silberschatz, Henry F. Korth, S. Sudarshan.数据库系统概念[M].机械工业出版社,2008(6)：168-186.
[5]David Karger, Alex Sherman, Andy Berkheimer, Bill Bogstad, Rizwan Dhanidina Ken Iwamoto, Brian Kim, Luke Matkins, Yoav Yerushalmi. Web Caching with Consistent Hashing[J]. MIT Laboratory for Computer Science,2009(5).
[6]Marc J. Rochkind.高级UNIX编程[M].机械工业出版社,2006(5)：216-218.
[7]Andrew S. Tanenbaum, Maartenvan Steen.分布式系统原理与范型[M].清华大学出版社,2007(1)：347-352.
[8]丘恩(Chun, W. J.). Python核心编程(第二版)[M].人民邮电出版社,2008(8)：245-258.
[9]米歇尔.精通Unix Shell脚本编程[M].清华大学出版社,2007(12)：312-324.
[10]Zhaoqing Jia, Ruonan Rao, Minglu Li, Jinyuan You. Random Walk Spread and Search In Unstructured P2P[J].The 5th International Conference Parallel and Distributed Computing,2004(11).
[11]斯蒂文斯.UNIX网络编程卷1：套接字联网API(第3版)[M].人民邮电出版社,2004(11)：311-324.
[12]周淦淼,谭石强.基于Memcached的MySQL查询优化[J].现代计算机(专业版),2009(5).
[13]张飞,姜进磊,李庆虎.利用MySQL构建分布式应用[J].计算机工程与应用,2001(11).
[14]赵庆民,李伟平.多处理机系统分析[J].微计算机应用,2005(3).
[15]Stephanos Androutsellis-Theotokis, Diomldis. A survey of peer-to-peer content distribution technologies[J]. ACM Computing Surveys,2004(11).
[16]郭欣.构建高性能Web站点[M].电子工业出版社,2010(2)：467-478.
[17]D. Darger, E. Lehman, T. Leighton, M. Levine, D. Lewin and R. Panigrahy. Consistent Hashing and Random Trees:Distributed Caching Protocols for Relieving Hot Spots On the World Wide Web[J]. ACM Symposium on Theory of Computing:2007(2).
[18]胡志华,王珏明.动态Web Cache系统研究与探讨[J].计算机应用与软件,2006(9).
[19]许艳美,肖宗水,梁异.Web缓存服务器技术研究与应用[J].计算机工程与设计,2005(1).
[20]郝晓艳,孙厚庆,孙奕奇.应用服务器缓存机制构建高效信息平台[J].山东省计算机学会2005年信息技术与信息化研讨会论文集(一),2005(3).
[21]PHP manual.
[22]杜江,PHP 5与MySQL5 Web开发技术详解[M].电子工业出版社,2008(5)：321-337.
[23]Python manual.