基于分布式Redis集群的WEB共享管理研究
|
摘要
目前WEB共享系统多数采用基于物理磁盘读写的数据存储系统,虽然磁盘数据存储系统和分布式大数据平台的结合可以提供较大的存储空间和较好的物理拓展性,但是在面对高并发的海量数据的实时存取时,物理磁盘读写效率以及系统内存的瓶颈会严重制约WEB共享系统的数据处理的效率。针对这种情况,论文通过对基于分布式的Redis集群的稳定的节点容错能力、优异的数据读写能力和高效的索引检索速度的理论研究和测试分析,证明了分布式的Redis集群能够有效提高WEB共享系统的海量数据实时处理的能力,是WEB共享系统升级数据存储系统性能的高效的解决方案。
At present,most of the WEB sharing system data storage system is based on the physical disk read and write,although this combination of data storage system and distributed data platform can provide larger storage space and better physical development,but facing massive data and high concurrent read and write,the physical disk read and write efficiency and system memory bottleneck will seriously restrict the efficiency of data processing in WEB sharing system. In view of this situation,based on the theoretical research and test analysis of node fault tolerance,Redis cluster distributed stable data based on excellent reading and writing skills and efficient indexing retrieval speed,this paper proves that Redis can effectively improve the Web Cluster distributed sharing system real-time mass data processing ability,and it's an efficient solution of WEB sharing system performance upgrade the data storage system.
At present,most of the WEB sharing system data storage system is based on the physical disk read and write,although this combination of data storage system and distributed data platform can provide larger storage space and better physical development,but facing massive data and high concurrent read and write,the physical disk read and write efficiency and system memory bottleneck will seriously restrict the efficiency of data processing in WEB sharing system. In view of this situation,based on the theoretical research and test analysis of node fault tolerance,Redis cluster distributed stable data based on excellent reading and writing skills and efficient indexing retrieval speed,this paper proves that Redis can effectively improve the Web Cluster distributed sharing system real-time mass data processing ability,and it's an efficient solution of WEB sharing system performance upgrade the data storage system.
引文
[1]李强,孙震宇,孙功星.一种面向HDFS的数据随机访问方法[J].计算机工程与应用,2017(10):1-7.LI Qiang,SUN Zhenyu,SUN Gongxing.A HDFS oriented data random access method of[J].Computer engineering and applications,2017(10):1-7.
[2]温云龙.云应用管理平台的设计与实现[J].移动通信,2017(8):69-73.WEN Yunlong.Design and implementation of cloud application management platform[J].Mobile communication,2017(8):69-73.
[3]姚经纬,杨福军.Redis分布式缓存技术在Hadoop平台上的应用[J].计算机技术与发展,2017(6):1-5.YAO Jingwei,YANG Fujun.Redis Distributed caching technology on Hadoop platform[J].Computer Technology and Development,2017(6):1-5.
[4]柳纲,何晶.一种高性能WEB集群的方案设计[J].电力信息与通信技术,2017(3):95-100.LIU Gang,HE Jing.Scheme design of a high performance WEB cluster[J].Power information and communication technology,2017(3):95-100.
[5]孙杜靖,李玲娟.面向Redis的数据序列化算法研究[J].计算机技术与发展,2017(5):77-81,86.SUN Dujing,LI Lingjuan.Research on data serialization algorithms for[J].Redis computer technology and development,2017(5):77-81,86.
[6]伍绍佳,杜林,廖丽.基于云平台的数字化校园信息门户系统实践研究[J].软件,2017(1):29-33.WU Shaojia,DU Lin,LIAO Li.Research on digital campus information portal system based on cloud platform[J].Software,2017(1):29-33.
[7]何龙,陈晋川,杜小勇.一种面向HDFS的多层索引技术[J].软件学报,2017(3):502-513.HE Long,CHEN Jinchuan,DU Xiaoyong.A multi index technology for HDFS.Journal of[J].Software,2017(3):502-513.
[8]崔丹,史金鑫.基于Redis实现HBase二级索引的方法[J].软件,2016(11):64-67.CUI Dan,SHI Jinxin.Method of implementing HBase two index based on Redis[J].Software,2016(11):64-67.
[9]丁建立,郑峰弓,李永华,等.基于NoSQL的海量航空物流小文件分布式多级存储方法[J].计算机应用研究,2017(5):1433-1436,1441.DING Jianli,ZHENG Fenggong,LI Yonghua,et al.A distributed multilevel storage method based on NoSQL for large air logistics small files[J].Computer application re-(上接第2082页)search,2017(5):1433-1436,1441.
[10]张发扬,李玲娟,陈煜.VFDT算法基于Storm平台的实现方案[J].计算机技术与发展,2016(9):192-196.ZHANG Chang,LI Lingjuan,CHEN Yu.VFDT algorithm,implementation scheme based on Storm platform,[J].Computer technology and development,2016(9):192-196.
[11]王康,李东静,陈海光.分布式存储系统中改进的一致性哈希算法[J].计算机技术与发展,2016(7):24-29.WANG Kang,LI Dongjing,CHEN Haiguang.Improved consistent hashing algorithm in distributed storage systems[J].Computer technology and development,2016(7):24-29.
[12]郎泓钰,任永功.基于Redis内存数据库的快速查找算法[J].计算机应用与软件,2016(5):40-43,52.LANG Hongyu,REN Yonggong.Fast lookup algorithm based on Redis in memory detabase[J].Computer applications and software,Redis memory database,2016(5):40-43,52.
[13]邹振宇,郑烇,王嵩,等.基于HDFS的云存储系统小文件优化方案[J].计算机工程,2016(3):34-40,46.ZOU Zhenyu,ZHENG Quan,WANG Song,et al.A small file optimization scheme for cloud storage systems based on[J].HDFS Computer Engineering,2016(3):34-40,46.
[14]董书暕,汪璟玢,陈远.HMSST+:基于分布式内存数据库的HMSST算法优化[J].计算机科学,2016(3):220-224,230.DONG Shujian,WANG Jingbin,CHEN Yuan.HMSST+:HMSST algorithm[J].Computer science,distributed memory database,2016(3):220-224,230.
[15]黎勇,崔喆,余少剑.内存数据库在超宽带实时定位系统中应用[J].计算机应用,2014(S2):48-51.LI Yong,CUI Zhe,YU Shaojian.Memory database in ultra wideband real-time positioning system in the application of[J].Computer applications,2014(S2):48-51.
[2]温云龙.云应用管理平台的设计与实现[J].移动通信,2017(8):69-73.WEN Yunlong.Design and implementation of cloud application management platform[J].Mobile communication,2017(8):69-73.
[3]姚经纬,杨福军.Redis分布式缓存技术在Hadoop平台上的应用[J].计算机技术与发展,2017(6):1-5.YAO Jingwei,YANG Fujun.Redis Distributed caching technology on Hadoop platform[J].Computer Technology and Development,2017(6):1-5.
[4]柳纲,何晶.一种高性能WEB集群的方案设计[J].电力信息与通信技术,2017(3):95-100.LIU Gang,HE Jing.Scheme design of a high performance WEB cluster[J].Power information and communication technology,2017(3):95-100.
[5]孙杜靖,李玲娟.面向Redis的数据序列化算法研究[J].计算机技术与发展,2017(5):77-81,86.SUN Dujing,LI Lingjuan.Research on data serialization algorithms for[J].Redis computer technology and development,2017(5):77-81,86.
[6]伍绍佳,杜林,廖丽.基于云平台的数字化校园信息门户系统实践研究[J].软件,2017(1):29-33.WU Shaojia,DU Lin,LIAO Li.Research on digital campus information portal system based on cloud platform[J].Software,2017(1):29-33.
[7]何龙,陈晋川,杜小勇.一种面向HDFS的多层索引技术[J].软件学报,2017(3):502-513.HE Long,CHEN Jinchuan,DU Xiaoyong.A multi index technology for HDFS.Journal of[J].Software,2017(3):502-513.
[8]崔丹,史金鑫.基于Redis实现HBase二级索引的方法[J].软件,2016(11):64-67.CUI Dan,SHI Jinxin.Method of implementing HBase two index based on Redis[J].Software,2016(11):64-67.
[9]丁建立,郑峰弓,李永华,等.基于NoSQL的海量航空物流小文件分布式多级存储方法[J].计算机应用研究,2017(5):1433-1436,1441.DING Jianli,ZHENG Fenggong,LI Yonghua,et al.A distributed multilevel storage method based on NoSQL for large air logistics small files[J].Computer application re-(上接第2082页)search,2017(5):1433-1436,1441.
[10]张发扬,李玲娟,陈煜.VFDT算法基于Storm平台的实现方案[J].计算机技术与发展,2016(9):192-196.ZHANG Chang,LI Lingjuan,CHEN Yu.VFDT algorithm,implementation scheme based on Storm platform,[J].Computer technology and development,2016(9):192-196.
[11]王康,李东静,陈海光.分布式存储系统中改进的一致性哈希算法[J].计算机技术与发展,2016(7):24-29.WANG Kang,LI Dongjing,CHEN Haiguang.Improved consistent hashing algorithm in distributed storage systems[J].Computer technology and development,2016(7):24-29.
[12]郎泓钰,任永功.基于Redis内存数据库的快速查找算法[J].计算机应用与软件,2016(5):40-43,52.LANG Hongyu,REN Yonggong.Fast lookup algorithm based on Redis in memory detabase[J].Computer applications and software,Redis memory database,2016(5):40-43,52.
[13]邹振宇,郑烇,王嵩,等.基于HDFS的云存储系统小文件优化方案[J].计算机工程,2016(3):34-40,46.ZOU Zhenyu,ZHENG Quan,WANG Song,et al.A small file optimization scheme for cloud storage systems based on[J].HDFS Computer Engineering,2016(3):34-40,46.
[14]董书暕,汪璟玢,陈远.HMSST+:基于分布式内存数据库的HMSST算法优化[J].计算机科学,2016(3):220-224,230.DONG Shujian,WANG Jingbin,CHEN Yuan.HMSST+:HMSST algorithm[J].Computer science,distributed memory database,2016(3):220-224,230.
[15]黎勇,崔喆,余少剑.内存数据库在超宽带实时定位系统中应用[J].计算机应用,2014(S2):48-51.LI Yong,CUI Zhe,YU Shaojian.Memory database in ultra wideband real-time positioning system in the application of[J].Computer applications,2014(S2):48-51.