分布式缓存系统MCACHE的优化与测试
摘要
高性能分布式数据缓存系统的研究,是电信服务提供商、电子商务平台,包括其它业务高速增长的企业都必须面临的问题和挑战,其高并发性和数据服务的低延迟和高效性使得对系统的架构设计、性能指标等各方面都提出了很高的要求。随着互联网技术的发展以及内存等硬件设施越来越廉价,高性能分布式数据缓存系统得到越来越多的关注。
本文首先对传统的网络I/O技术做了一定的分析,根据各自的特点以及适用的场合做了对比,介绍了mcache系统所使用的最优和最稳定的epoll模型。接着本文介绍了mcache系统的数据缓存层技术--memcached协议及内存分配和数据删除方式,在后端数据库方面,本文分析了mysql数据库优化的目标,以及影响后端数据库性能的主要因素。针对mcache系统的要求,进行数据库存储引擎的选择,同时调整server端的参数以尽量提高系统的性能。最后结合所研究的内容,在充分了解高性能分布式数据缓存系统mcache的框架后,在不同的场景下,对mcache系统进行了详细的功能测试以及在并发性和数据处理这两方面的性能测试。
本文针对高性能分布式缓存系统mcache的应用场景,进行了性能上的优化和测试,测试结果进一步证明了,‘经过优化的mcache系统比传统的磁盘数据库在并发量和数据处理方面存在较大的优势。
Telecommunications Service Provider, E-Commerce Platform, and the other enterprises which get rapid growth of business and face the same problems and challenges, attache great importance on the The research of the high-performance distributed data cache system. Its high concurrency, low latency and efficient data services make the system architecture design, performance indicators get high demands. With the development of Internet technology、cheap memory and other hardware facilities, high-performance distributed data cache system will be got more and more attention.
In this paper, I firstly analysis the traditional network I/O technology, according to its characteristics and possible applications,I introduce the best and most stable epoll model used by MCACHE system firstly. Then This article describes the MCACHE system data cache layer, memory allocation and data deletion method of the memcached, In the back-end database, the paper analyzes the optimal target of mysql database,the factors that affect the back-end database performance, and the choice of the database storage engine for MCACHE system's requirements, Finally, I research the high-performance distributed data caching framework MCACHE system,The system was conducted the functional testing in different scenarios.
This paper presents a high-performance distributed cache service system---MCACHE, and through the optimization and testing, The results further show that,there is a huge advantage about the optimized MCACHE systems than traditional disk database in concurrency and data processing.
本文首先对传统的网络I/O技术做了一定的分析,根据各自的特点以及适用的场合做了对比,介绍了mcache系统所使用的最优和最稳定的epoll模型。接着本文介绍了mcache系统的数据缓存层技术--memcached协议及内存分配和数据删除方式,在后端数据库方面,本文分析了mysql数据库优化的目标,以及影响后端数据库性能的主要因素。针对mcache系统的要求,进行数据库存储引擎的选择,同时调整server端的参数以尽量提高系统的性能。最后结合所研究的内容,在充分了解高性能分布式数据缓存系统mcache的框架后,在不同的场景下,对mcache系统进行了详细的功能测试以及在并发性和数据处理这两方面的性能测试。
本文针对高性能分布式缓存系统mcache的应用场景,进行了性能上的优化和测试,测试结果进一步证明了,‘经过优化的mcache系统比传统的磁盘数据库在并发量和数据处理方面存在较大的优势。
Telecommunications Service Provider, E-Commerce Platform, and the other enterprises which get rapid growth of business and face the same problems and challenges, attache great importance on the The research of the high-performance distributed data cache system. Its high concurrency, low latency and efficient data services make the system architecture design, performance indicators get high demands. With the development of Internet technology、cheap memory and other hardware facilities, high-performance distributed data cache system will be got more and more attention.
In this paper, I firstly analysis the traditional network I/O technology, according to its characteristics and possible applications,I introduce the best and most stable epoll model used by MCACHE system firstly. Then This article describes the MCACHE system data cache layer, memory allocation and data deletion method of the memcached, In the back-end database, the paper analyzes the optimal target of mysql database,the factors that affect the back-end database performance, and the choice of the database storage engine for MCACHE system's requirements, Finally, I research the high-performance distributed data caching framework MCACHE system,The system was conducted the functional testing in different scenarios.
This paper presents a high-performance distributed cache service system---MCACHE, and through the optimization and testing, The results further show that,there is a huge advantage about the optimized MCACHE systems than traditional disk database in concurrency and data processing.
引文
[1]Oracle TimesTen内存数据库技术[EB/OL].http://www.oracle.com/technology/global/cn/events/index.html,2008.03
[2]Hybrid DBMS-ALTIBASE数据库技术白皮书[EB/OL]. http://www.altibase.com.cn/download/Altibase.pdf,2010.04.
[3]C10K problem[EB/OL].http://www.kegel.com/c10k.html,2006.
[4]sys_epoll-making poll fast[EB/OL].http://lwn.net/Articles/14168,2005.
[5]Louay Gammo,Tim Brecht,Amol Shukla and David Pariag.Comparing and Evaluating epoll,select and poll Event Mechanisms.In the Proceedings of Ottawa Linux Symposium,2004.07.
[6]张震波,杨鹤标,马振华.基于LRU算法的web系统缓存机制[J].计算机工程.32卷19期,pages 68-70.2006.
[7]D. Karger, E. Lehman, T. Leighton, M. Levine,D. Lewin, and R. Panigrahy. Consistent hashing and random trees:Distributed caching protocols for relieving hot spots on the World Wide Web[J]. In Proceedings of the Twenty-Ninth Annual ACM Symposium on Theory of Computing, pages 654-663,1997.05
[8]Mitra Nasri,Mohsen Sharifi. Load Balancing Using Consistent Hashing:A Real Challenge for Large Scale Distributed Web Crawlers[J]. Proceedings of the 2009 International Conference on Advanced Information Networking and Applications Workshops, pages715-720,2009.05.
[9]简超阳.Mysql性能调优与架构设计.电子工业出版社.2009.06.
[10]林宇,郭凌云.Linux网络编程.人民邮电出版社.2000.11.
[11]李昊,刘志敬.线程池技术的研究[J].现代电子技术.2004.03.
[12]UNIX System Programming:Communication, Concurrency and Threads 2/e[M].Prentice Hall.2004:50-124.
[2]Hybrid DBMS-ALTIBASE数据库技术白皮书[EB/OL]. http://www.altibase.com.cn/download/Altibase.pdf,2010.04.
[3]C10K problem[EB/OL].http://www.kegel.com/c10k.html,2006.
[4]sys_epoll-making poll fast[EB/OL].http://lwn.net/Articles/14168,2005.
[5]Louay Gammo,Tim Brecht,Amol Shukla and David Pariag.Comparing and Evaluating epoll,select and poll Event Mechanisms.In the Proceedings of Ottawa Linux Symposium,2004.07.
[6]张震波,杨鹤标,马振华.基于LRU算法的web系统缓存机制[J].计算机工程.32卷19期,pages 68-70.2006.
[7]D. Karger, E. Lehman, T. Leighton, M. Levine,D. Lewin, and R. Panigrahy. Consistent hashing and random trees:Distributed caching protocols for relieving hot spots on the World Wide Web[J]. In Proceedings of the Twenty-Ninth Annual ACM Symposium on Theory of Computing, pages 654-663,1997.05
[8]Mitra Nasri,Mohsen Sharifi. Load Balancing Using Consistent Hashing:A Real Challenge for Large Scale Distributed Web Crawlers[J]. Proceedings of the 2009 International Conference on Advanced Information Networking and Applications Workshops, pages715-720,2009.05.
[9]简超阳.Mysql性能调优与架构设计.电子工业出版社.2009.06.
[10]林宇,郭凌云.Linux网络编程.人民邮电出版社.2000.11.
[11]李昊,刘志敬.线程池技术的研究[J].现代电子技术.2004.03.
[12]UNIX System Programming:Communication, Concurrency and Threads 2/e[M].Prentice Hall.2004:50-124.