REDU: reducing redundancy and duplication for multi-failure recovery in erasure-coded storages

详细信息    查看全文

• 作者：Jing Zhang ; Shanshan Li ; Xiangke Liao
• 关键词：Erasure ; coded storage ; Multi ; failure recovery ; Redundancy ; De ; duplication ; Aggregation Decoding
• 刊名：The Journal of Supercomputing
• 出版年：2016
• 出版时间：September 2016
• 年：2016
• 卷：72
• 期：9
• 页码：3281-3296
• 全文大小：1,082 KB
• 刊物类别：Computer Science
• 刊物主题：Programming Languages, Compilers and Interpreters
  Processor Architectures
  Computer Science, general
  
• 出版者：Springer Netherlands
• ISSN：1573-0484
• 卷排序：72

文摘

Data reliability is a significant issue in large-scale storage systems. Erasure codes provide high data reliability via data recovery, which however generates a large amount of data transmission in the network. The bandwidth cost of the data transmission in recovery significantly impacts the performance of the located cluster. Existing work considers the single-failure as the most common failure pattern and mainly focuses on reducing the data transmission cost of single-failure recovery, which unfortunately fails to efficiently support the multi-failure recovery. In this work, first, we provide the Mean Time To Multi-Failure metric based on Markov model to demonstrate the frequency and pattern of multi-failure in erasure-coded storages. Then, we propose REDU to reduce the duplication and redundancy in multi-failure recovery of erasure-coded storages. In REDU, we propose merging-based de-duplication to reduce duplicated data transmission, and aggregating-based de-redundancy to reduce redundant information transmission, and we also propose cooperative routing to efficiently use the two schemes above based on the practical cluster topology. The analysis and experimental results demonstrate the importance of multi-failure recovery problem and the efficiency of REDU.