A new data-grouping-aware dynamic data placement method that take into account jobs execute frequency for Hadoop
详细信息 查看全文
- 作者：Jia-xuan WuAuthor Vitae ; Chang-sheng ZhangAuthor Vitae ; Bin Zhang ; zhangbin@ise.neu.edu.cn ; paper820@sohu.comAuthor Vitae ; Peng WangAuthor Vitae
- 关键词：Hadoop ; Data placement ; Date-group-aware ; Job execute frequency ; Access correlate relation
- 刊名：Microprocessors and Microsystems
- 出版年：2016
- 出版时间：November 2016
- 年：2016
- 卷：47
- 期：part_PA
- 页码：161-169
- 全文大小：1756 K
- 卷排序：47
文摘
Recent years have seen an increasing number of scientists employing data parallel computing frameworks, such as Hadoop, in order to run data-intensive applications. Research on data-grouping-aware data placement for Hadoop has become increasingly popular. However, we observe that many data-grouping-aware data placement schemes are static, without taking MapReduce job execution frequency into consideration. Such data placements scheme will lead to severe performance degradation that is way below the potential efficiency of optimal data distribution when executing MapReduce jobs that are executed frequency. In this paper, we propose a new data-grouping-aware dynamic (DGAD) data placement method based on the job execution frequency. Firstly, we build a job access correlation relation model among the data blocks according to the relationships provided by the records about historical data block access. Then we use a clustering algorithm to divide data blocks into clusters according to the job access correlation relation model among the data blocks and propose a data placement algorithm based on data block clusters in order to put correlated data blocks within a cluster on the different nodes. Finally, a series of experiments are carried out in order to verify the method proposed in this paper. Experimental results show that the proposed method can effectively deal with the mass data and can obviously improve the execution efficiency of MapReduce.