Hadoop分布式存储架构的性能分析
|
摘要
近几十年,随着互联网的发展,数据正在呈爆炸式增长。为了保存和处理这些数据,传统的IT架构已经满足不了,云计算应运而生。Hadoop是云计算下的一种分布式处理的软件架构,能够对大数据进行存储和处理,而它最核心的两大组件就是HDFS和Map Reduce,分别实现了分布式文件存储和分布式并行任务处理。文中,将基于PEPA对HDFS进行建模分析,并对HDFS的性能进行评估。
In recent decades,data has been in explosive growth with the development of the Internet. As the traditional IT architecture cannot meet the requirements of storing and processing the data,cloud computing has emerged. Hadoop is a distributed processing software architecture using cloud computing,which can store and process big data. The two main core components of Hadoop is the Hadoop Distributed File System(HDFS)and Map Reduce,which implement distributed file storage and distributed parallel task processing respectively. In this paper,modeling analysis for the HDFS is conducted based on the Performance Evaluation Process Algebra(PEPA),so as to evaluate the performance of the HDFS.
In recent decades,data has been in explosive growth with the development of the Internet. As the traditional IT architecture cannot meet the requirements of storing and processing the data,cloud computing has emerged. Hadoop is a distributed processing software architecture using cloud computing,which can store and process big data. The two main core components of Hadoop is the Hadoop Distributed File System(HDFS)and Map Reduce,which implement distributed file storage and distributed parallel task processing respectively. In this paper,modeling analysis for the HDFS is conducted based on the Performance Evaluation Process Algebra(PEPA),so as to evaluate the performance of the HDFS.
引文
[1]董新华,李瑞轩,周湾湾,等.Hadoop系统性能优化与功能增强综述[J].计算机研究与发展,2013,50(z2):1-15.DONG Xinhua,LI Ruixuan,ZHOU Wanwan,et al.Performance optimization and feature enhancements of Hadoop system[J].Journal of computer research and development,2013,50(S2):1-15.
[2]李成华,张新访,金海,等.Map Reduce:新型的分布式并行计算编程模型[J].计算机工程与科学,2011,33(3):129-135.LI Chenghua,ZHANG Xinfang,JIN Hai,et al.Map Reduce:a new programming model for distributed parallel computing[J].Computer engineering&science,2011,33(3):129-135.
[3]郝树魁.Hadoop HDFS和Map Reduce架构浅析[J].邮电设计技术,2012(7):37-42.HAO Shukui.Brief analysis of the architecture of Hadoop HDFS and Map Reduce[J].Designing techniques of posts and telecommunications,2012(7):37-42.
[4]DING J.A comparison of fluid approximation and stochastic simulation for evaluating content adaptation systems[J].Wireless personal communications,2015,84(1):231-250.
[5]DING J,HILLSTON J,LAURENSON D I.Evaluating the response time of large scale content adaptation systems using performance evaluation process algebra[C]//Proceedings of IEEE International Conference on Communications.Cape town:IEEE,2010:1-5.
[6]WANG H,LAURENSON D I,HILLSTON J.Evaluation of RSVP and mobility-aware RSVP using performance evaluation process algebra[C]//Proceedings of IEEE International Conference on Communications.Beijing:IEEE,2008:192-197.
[7]潘耘,王励成,余镇危,等.基于随机进程代数的Overlay网络上的Qo S性能评价[J].北京邮电大学学报,2009,32(z1):129-133.PAN Yun,WANG Licheng,YU Zhenwei,et al.Based on stochastic process algebra overlay network Qo S performance evaluation[J].Journal of Beijing University of Posts and Telecommunications,2009,32(S1):129-133.
[8]黄金才,陈洒然,程光权.舰艇编队防空过程建模及分析[J].国防科技大学学报,2014,36(3):184-190.HUANG Jincai,CHEN Saran,CHENG Guangquan.Modeling and analysis of air defense process for warship formation[J].Journal of National University of Defense Technology,2014,36(3):184-190.
[9]李宽.基于HDFS的分布式Namenode节点模型的研究[D].广州:华南理工大学,2011.LI Kuan.Research of the model of distributed Namenodes in HDFS[D].Guangzhou:South China University of Technology,2011.
[10]HILLSTON J.Fluid flow approximation of PEPA models[C]//Proceedings of Second International Conference on the Quantitative Evaluation of Systems.Torino:IEEE,2005:1-10.
[2]李成华,张新访,金海,等.Map Reduce:新型的分布式并行计算编程模型[J].计算机工程与科学,2011,33(3):129-135.LI Chenghua,ZHANG Xinfang,JIN Hai,et al.Map Reduce:a new programming model for distributed parallel computing[J].Computer engineering&science,2011,33(3):129-135.
[3]郝树魁.Hadoop HDFS和Map Reduce架构浅析[J].邮电设计技术,2012(7):37-42.HAO Shukui.Brief analysis of the architecture of Hadoop HDFS and Map Reduce[J].Designing techniques of posts and telecommunications,2012(7):37-42.
[4]DING J.A comparison of fluid approximation and stochastic simulation for evaluating content adaptation systems[J].Wireless personal communications,2015,84(1):231-250.
[5]DING J,HILLSTON J,LAURENSON D I.Evaluating the response time of large scale content adaptation systems using performance evaluation process algebra[C]//Proceedings of IEEE International Conference on Communications.Cape town:IEEE,2010:1-5.
[6]WANG H,LAURENSON D I,HILLSTON J.Evaluation of RSVP and mobility-aware RSVP using performance evaluation process algebra[C]//Proceedings of IEEE International Conference on Communications.Beijing:IEEE,2008:192-197.
[7]潘耘,王励成,余镇危,等.基于随机进程代数的Overlay网络上的Qo S性能评价[J].北京邮电大学学报,2009,32(z1):129-133.PAN Yun,WANG Licheng,YU Zhenwei,et al.Based on stochastic process algebra overlay network Qo S performance evaluation[J].Journal of Beijing University of Posts and Telecommunications,2009,32(S1):129-133.
[8]黄金才,陈洒然,程光权.舰艇编队防空过程建模及分析[J].国防科技大学学报,2014,36(3):184-190.HUANG Jincai,CHEN Saran,CHENG Guangquan.Modeling and analysis of air defense process for warship formation[J].Journal of National University of Defense Technology,2014,36(3):184-190.
[9]李宽.基于HDFS的分布式Namenode节点模型的研究[D].广州:华南理工大学,2011.LI Kuan.Research of the model of distributed Namenodes in HDFS[D].Guangzhou:South China University of Technology,2011.
[10]HILLSTON J.Fluid flow approximation of PEPA models[C]//Proceedings of Second International Conference on the Quantitative Evaluation of Systems.Torino:IEEE,2005:1-10.