高并发搜索系统下内存池的设计和实现
|
摘要
为了在输入关键词后能从网络搜索中快速、准确地返回信息,并有效降低搜索系统在高并发状态下频繁分配和回收内存对程序性能的影响.文中根据搜索引擎中不同的场景,设计出了可回收定长内存池,可回收变长内存池和只分配不释放内存池.实例计算结果表明:与系统默认的内存分配器对比,可回收定长内存池的效率提升了70.20%;可回收变长内存池的效率提升了13.84%;只分配不释放内存池的效率提升了90.80%.
In order to quickly and accurately return the information to the user after the keywords are entered,and to effectively reduce the effect on the performance of the program when the search system allocates and deallocates memory frequently under the high concurrency,the Recoverable Fixed Length Memory Pool,Recoverable Variable Length Memory pool and Allocate Not Free Memory Pool were designed.A according to the different scenes features of the search engine.The result shows that,compared with the default system memory allocator,the efficiency of the Recoverable Fixed Length Memory Pool efficiency is increased by 70.20%,the efficiency of the Recoverable Variable Length Memory Pool is increased by 13.84% and the efficiency of the Allocate Not Free Memory Pool is increased by 90.80%.
In order to quickly and accurately return the information to the user after the keywords are entered,and to effectively reduce the effect on the performance of the program when the search system allocates and deallocates memory frequently under the high concurrency,the Recoverable Fixed Length Memory Pool,Recoverable Variable Length Memory pool and Allocate Not Free Memory Pool were designed.A according to the different scenes features of the search engine.The result shows that,compared with the default system memory allocator,the efficiency of the Recoverable Fixed Length Memory Pool efficiency is increased by 70.20%,the efficiency of the Recoverable Variable Length Memory Pool is increased by 13.84% and the efficiency of the Allocate Not Free Memory Pool is increased by 90.80%.
引文
[1]戴春燕,徐智文.对C++中Malloc/Free和New/Delete的探讨[J].包钢科技,2009(35):59.DAI Chunyan,XU Zhiwen.Discussion of Malloc/Free and New/Delete in C++[J].Science&Technology of Baotou Steel(Group)Corporation,2009(35):59.(in Chinese)
[2]李倩,潘敏学,李宣东.内存泄漏检测工具与评估方法[J].计算机科学与探索,2010(1):29.LI Qian,PAN Minxue,LI Xuandong.Benchmark of Tools for Memory Leak[J].Journal of Frontiers of Computer Science and Technology,2010(1):29.(in Chinese)
[3]王小银,陈莉君.Linux内核中内存池的实现及应用[J].西安邮电学院学报,2011(4):40.WANG Xiaoyin,CHEN Lijun.Implementation and Application of the Memory Pool in Linux Kernel[J].Journal of Xi’an University of Posts and Telecommunications,2011(4):40.(in Chinese)
[4]曲卫华,王群.搜索引擎原理介绍与分析[J].电脑知识与技术,2006(6):113.QU Weihua,WANG Qun.Introduction to and Analysis of Search Engine Principle[J].Computer Knowledge and Technology,2006(6):113.(in Chinese)
[5]段兵营.搜索引擎中网络爬虫的研究与实现[D].西安:西安电子科技大学,2014.DUAN Bingying.Study and Design of Web Crawler in A Search Engine[D].Xi’an:Xidian University,2014.(in Chinese)
[6]郭丙轩,张京莉,张志超.基于内存池的空间数据调度算法[J].计算机工程,2008,34(6):63.GUO Bingxuan,ZHANG Jingli,ZHANG Zhichao.Algorithm for Spatial Data Scheduling Based on Memory Pool[J].Computer Engineering,2008,34(6):63.(in Chinese)
[7]刘涛,聂晓峰,荆继武,王跃武.基于小型对象分配技术的GTNetS蠕虫仿真内存管理[J].中国科学院研究生学报,2012,29(1):131.LIU Tao,NIE Xiaofeng,JING Jiwu,WANG Yuewu.Memory Management in Worm Simulation Based on Small Object Memory Allocation Technique on The GTNetS[J].Journal of Graduate University of Chinese Academy of Sciences,2012,29(1):131.(in Chinese)
[8]郭旭峰,于芳,刘忠立.基于哈希表的高效存储器内建自修复方法[J].电子学报,2013(7):1371.GUO Xufeng,YU Fang,LIU Zhongli.An Efficient Memory Built-in Self-Repair Method Based on Hash Table[J].Acta Electronica Sinica,2013(7):1371.(in Chinese)
[9]赖祥芳.选择合适的STL容器[J].数字技术与应用,2015(9):177.LAI Xiangfang.Selecting The Appropriate STL Containers[J].Digital Technology and Application,2015(9):177.(in Chinese)
[10]ALEXANDRESCU A.Modern C++Design:Generic Programming and Design Patterns Applied[M].Boston:Addison-Wesley Professional,2001.
[11]ROBERT W P,Luka,Wai Lamb.Efficient In-Memory Extensible Inverted File[J].Information Systems,2007(32):733.
[2]李倩,潘敏学,李宣东.内存泄漏检测工具与评估方法[J].计算机科学与探索,2010(1):29.LI Qian,PAN Minxue,LI Xuandong.Benchmark of Tools for Memory Leak[J].Journal of Frontiers of Computer Science and Technology,2010(1):29.(in Chinese)
[3]王小银,陈莉君.Linux内核中内存池的实现及应用[J].西安邮电学院学报,2011(4):40.WANG Xiaoyin,CHEN Lijun.Implementation and Application of the Memory Pool in Linux Kernel[J].Journal of Xi’an University of Posts and Telecommunications,2011(4):40.(in Chinese)
[4]曲卫华,王群.搜索引擎原理介绍与分析[J].电脑知识与技术,2006(6):113.QU Weihua,WANG Qun.Introduction to and Analysis of Search Engine Principle[J].Computer Knowledge and Technology,2006(6):113.(in Chinese)
[5]段兵营.搜索引擎中网络爬虫的研究与实现[D].西安:西安电子科技大学,2014.DUAN Bingying.Study and Design of Web Crawler in A Search Engine[D].Xi’an:Xidian University,2014.(in Chinese)
[6]郭丙轩,张京莉,张志超.基于内存池的空间数据调度算法[J].计算机工程,2008,34(6):63.GUO Bingxuan,ZHANG Jingli,ZHANG Zhichao.Algorithm for Spatial Data Scheduling Based on Memory Pool[J].Computer Engineering,2008,34(6):63.(in Chinese)
[7]刘涛,聂晓峰,荆继武,王跃武.基于小型对象分配技术的GTNetS蠕虫仿真内存管理[J].中国科学院研究生学报,2012,29(1):131.LIU Tao,NIE Xiaofeng,JING Jiwu,WANG Yuewu.Memory Management in Worm Simulation Based on Small Object Memory Allocation Technique on The GTNetS[J].Journal of Graduate University of Chinese Academy of Sciences,2012,29(1):131.(in Chinese)
[8]郭旭峰,于芳,刘忠立.基于哈希表的高效存储器内建自修复方法[J].电子学报,2013(7):1371.GUO Xufeng,YU Fang,LIU Zhongli.An Efficient Memory Built-in Self-Repair Method Based on Hash Table[J].Acta Electronica Sinica,2013(7):1371.(in Chinese)
[9]赖祥芳.选择合适的STL容器[J].数字技术与应用,2015(9):177.LAI Xiangfang.Selecting The Appropriate STL Containers[J].Digital Technology and Application,2015(9):177.(in Chinese)
[10]ALEXANDRESCU A.Modern C++Design:Generic Programming and Design Patterns Applied[M].Boston:Addison-Wesley Professional,2001.
[11]ROBERT W P,Luka,Wai Lamb.Efficient In-Memory Extensible Inverted File[J].Information Systems,2007(32):733.