FARMS: Efficient mapreduce speculation for failure recovery in short jobs

详细信息    查看全文

• 作者：Huansong Fu ; ^a ; ^{fu@cs.fsu.edu} ; Haiquan Chen^b ; ^{hachen@valdosta.edu} ; Yue Zhu^a ; ^{yzhu@cs.fsu.edu} ; Weikuan Yu^a ; ^{yuw@cs.fsu.edu}
• 关键词：Mapreduce ; YARN ; Speculation ; Failure recovery
• 刊名：Parallel Computing
• 出版年：2017
• 出版时间：January 2017
• 年：2017
• 卷：61
• 期：Complete
• 页码：68-82
• 全文大小：2157 K
• 卷排序：61

文摘

Existing speculation mechanism has fundamental flaws in mitigating intra-node and completed task stragglers, which are often caused by node failure. Those issues result in more than an order of magnitude performance breakdown of small jobs and serious performance degradation of large jobs upon failures. A hybrid solution includes a speculation mechanism to cope with the issues and a scheduling policy to enhance failure awareness and recovery. The implementation of the solution shows striking performance improvement for MapReduce failure recovery.