MapReduce short jobs optimization based on resource reuse

详细信息    查看全文

• 作者：Yuliang Shi^a ; ^{shiyuliang@sdu.edu.cn}Author Vitae ; Kaihui Zhang^a ; ^{kaihuizhang@126.com}Author Vitae ; Lizhen Cui^a ; ^{clz@sdu.edu.cn}Author Vitae ; Lei Liu^a ; ^{l.liu@sdu.edu.cn}Author Vitae ; Yongqing Zheng^a ; ^{zhengyongqing@dareway.com.cn}Author Vitae ; Shidong Zhang^a ; ^{zsd@sdu.edu.cn}Author Vitae ; Han Yu^b ; ^{han.yu@ntu.edu.sg}Author Vitae
• 关键词：Hadoop ; Short job ; Performance optimization ; Resource utilization ; Task scheduling
• 刊名：Microprocessors and Microsystems
• 出版年：2016
• 出版时间：November 2016
• 年：2016
• 卷：47
• 期：part_PA
• 页码：178-187
• 全文大小：975 K
• 卷排序：47

文摘

Hadoop is an open-source implementation of MapReduce serving for processing large datasets in a massively parallel manner. It was designed aiming at executing large-scale jobs in an enormous number of computing nodes offering computing and storage. However, Hadoop is frequently employed to process short jobs. In practice, short jobs suffer from poor response time and run inefficiently. To fill this gap, this paper analyses the process of job execution and depicts the existing issues why short jobs run inefficiently in Hadoop. According to the characteristic of task execution in multi-wave under cluster overload, we develop a mechanism in light of resource reuse to optimize short jobs execution. This mechanism can reduce the frequency of resource allocation and recovery. Experimental results suggest that the developed mechanism based on resource reuse is able to improve effectiveness of the resource utilization. In addition, the runtime of short jobs can be significantly reduced.