Novel heuristic speculative execution strategies in heterogeneous distributed environments

详细信息    查看全文

• 作者：Xin Huang^a ; ^{huangxin@hnu.edu.cn" class="auth_mail" title="E-mail the corresponding author}Author Vitae ; Longxin Zhang^a ; ^b ; ^{longxinzhang@hnu.edu.cn" class="auth_mail" title="E-mail the corresponding author}Author Vitae ; Renfa Li^a ; ^{lirenfa@hnu.edu.cn" class="auth_mail" title="E-mail the corresponding author}Author Vitae ; Lanjun Wan^a ; ^{wancanjun2008@163.com" class="auth_mail" title="E-mail the corresponding author}Author Vitae ; Keqin Li^a ; ^b ; ^c ; ^{lik@newpaltz.edu" class="auth_mail" title="E-mail the corresponding author}Author Vitae
• 关键词：Cloud computing ; Dynamic loading ; Hadoop ; MapReduce ; Speculative execution
• 刊名：Computers & Electrical Engineering
• 出版年：2016
• 出版时间：February 2016
• 年：2016
• 卷：50
• 期：Complete
• 页码：166-179
• 全文大小：985 K

文摘

MapReduce is a promising distributed computing platform for large-scale data processing applications. Hadoop MapReduce has been considered as one of the most extensively used open-source implementations of MapReduce frameworks for its flexible customization and convenient usage. Despite these advantages, a relatively slow running task called straggler task impedes job progress. In this study, two novel speculative strategies, namely, Estimate Remaining time Using Linear relationship model (ERUL) and extensional Maximum Cost Performance (exMCP), are developed to improve the estimation of the remaining time of a task. ERUL is a dynamic system load-aware strategy; using this strategy, we can overcome some drawbacks of the Longest Approximate Time to End (LATE) that misleads speculative execution in some cases. In exMCP, different slot values are considered. Extensive experiments show that ERUL and exMCP are applied to accurately estimate the remaining execution times of running tasks and reduce the running time of a job.