纠删码系统中基于协作方式的更新方法
|
摘要
为满足纠删码更新方法的效率需求,提出一种基于协作方式的更新方法Coop-U,以批量形式高效完成纠删码数据的多点更新。通过一种三层的更新框架保证数据更新的一致性,通过一种负载感知的分组方法动态调整分组大小,通过一种协作式的数据处理算法有效组织多层之间数据的传输与计算。基于数据驱动的模拟器对比实验结果表明,与典型的纠删码数据更新方法 Data-R和Full-W相比,Coop-U的平均更新时间分别减少了42%和30%。
To meet the efficiency requirement of the update schemes of erasure codes,a cooperative update scheme for multiple updates with erasure codes was proposed,called Coop-U.The data block was updated eagerly and the parity blocks were cooperatively updated in bulk.A three-layer update structure was proposed to ensure the generality.For efficiency,a workload-aware grouping algorithm was proposed to group the nodes to be updated and the group size was dynamically adjusted according to the update workload.A cooperative data processing algorithm was proposed to organize the data transmission and computation among layers.To evaluate the performance of Coop-U,it was implemented on HDFS-RAID and testbed experiments were conducted on different update schemes.Extensive experiments verify that Coop-U reduces the update time by 42% and 30% on average compared with the typical update schemes Data-R and Full-W respectively.
To meet the efficiency requirement of the update schemes of erasure codes,a cooperative update scheme for multiple updates with erasure codes was proposed,called Coop-U.The data block was updated eagerly and the parity blocks were cooperatively updated in bulk.A three-layer update structure was proposed to ensure the generality.For efficiency,a workload-aware grouping algorithm was proposed to group the nodes to be updated and the group size was dynamically adjusted according to the update workload.A cooperative data processing algorithm was proposed to organize the data transmission and computation among layers.To evaluate the performance of Coop-U,it was implemented on HDFS-RAID and testbed experiments were conducted on different update schemes.Extensive experiments verify that Coop-U reduces the update time by 42% and 30% on average compared with the typical update schemes Data-R and Full-W respectively.
引文
[1]LUO Xianghong,SHU Jiwu.Review of erasure codes in storage systems[J]. Computer Research and Development,2012,49(1):1-11(in Chinese).[罗象宏,舒继武.存储系统中的纠删码研究综述[J].计算机研究与发展,2012,49(1):1-11.]
[2]Li Bin,Aditya Ramamoorthy,Srikant R.Mean-field-analysis of coding versus replication in cloud storage systems[C]//35th Annual IEEE International Conference on Computer Communications,2016:2662-2670.
[3]Zhang Heng,Dong Mingkai,Chen Haibo.Efficient and available in-memory KV-store with hybrid erasure coding and replication[C]//14th USENIX Conference on File and Storage Technologies,2016:167-180.
[4]SONG Baoyan, WANG Junlu, WANG Yan.Resaerch on HDFS optimized storage strategy based on Vandermonde codes[J].Chinese Journal of Computers,2015,38(9):1825-1836(in Chinese).[宋宝燕,王俊陆,王妍.基于范德蒙码的HDFS优化存储策略研究[J].计算机学报,2015,38(9):1825-1836.]
[5]Arya Mazumdar,Venkat Chandar,Gregory W Wornell.Update efficiency and local repairability limits for capacity approaching codes[J].IEEE Journal on Selected Areas in Communications,2014,32(5):976-988.
[6]Chan JCW,Ding Q,Lee PPC,et al.Parity logging with reserved space:Towards efficient updates and recovery in erasurecoded clustered storage[C]//12th USENIX Conference on File and Storage Technologies,2014:163-176.
[7]Chan JCW,Ding Q,Lee PPC.Parity logging with reserved space:Towards efficient updates and recovery in erasure-coded clustered storage[C]//12th USENIX Conference on File and Storage Technologies,2014:163-176.
[8]FU Yingxun,LUO Shengmei,SHU Jiwu.Review of secure cloud storage and key technologies[J].Computer Research and Development,2015,50(1):136-145(in Chinese).[傅颖勋,罗圣美,舒继武.安全云存储系统与关键技术综述[J].计算机研究与发展,2015,50(1):136-145.]
[9]Fabio Margaglia,Gala Yadgar,Eitan Yaakobi,et al.The devil is in the details:Implementing flash page reuse with WOM codes[C]//14th USENIX Conference on File and Storage Technologies,2016:95-109.
[10]Zhang Fenghao, Huang Jianzhong,Xie Changsheng.Two efficient partial-updating schemes for erasure-coded storage clusters[C]//7th International Conference on Networking,Architecture,and Storage,2012:21-30.
[11]CHEN Fu,ZHANG Shuping.Research on efficient erasure codes reconstruction in distributed block storage systems[J].Software Guide,2016,15(7):33-35(in Chinese).[陈付,张淑萍.分布式块存储系统中高效纠删码重构技术研究[J].软件导刊,2016,15(7):33-35.]
[2]Li Bin,Aditya Ramamoorthy,Srikant R.Mean-field-analysis of coding versus replication in cloud storage systems[C]//35th Annual IEEE International Conference on Computer Communications,2016:2662-2670.
[3]Zhang Heng,Dong Mingkai,Chen Haibo.Efficient and available in-memory KV-store with hybrid erasure coding and replication[C]//14th USENIX Conference on File and Storage Technologies,2016:167-180.
[4]SONG Baoyan, WANG Junlu, WANG Yan.Resaerch on HDFS optimized storage strategy based on Vandermonde codes[J].Chinese Journal of Computers,2015,38(9):1825-1836(in Chinese).[宋宝燕,王俊陆,王妍.基于范德蒙码的HDFS优化存储策略研究[J].计算机学报,2015,38(9):1825-1836.]
[5]Arya Mazumdar,Venkat Chandar,Gregory W Wornell.Update efficiency and local repairability limits for capacity approaching codes[J].IEEE Journal on Selected Areas in Communications,2014,32(5):976-988.
[6]Chan JCW,Ding Q,Lee PPC,et al.Parity logging with reserved space:Towards efficient updates and recovery in erasurecoded clustered storage[C]//12th USENIX Conference on File and Storage Technologies,2014:163-176.
[7]Chan JCW,Ding Q,Lee PPC.Parity logging with reserved space:Towards efficient updates and recovery in erasure-coded clustered storage[C]//12th USENIX Conference on File and Storage Technologies,2014:163-176.
[8]FU Yingxun,LUO Shengmei,SHU Jiwu.Review of secure cloud storage and key technologies[J].Computer Research and Development,2015,50(1):136-145(in Chinese).[傅颖勋,罗圣美,舒继武.安全云存储系统与关键技术综述[J].计算机研究与发展,2015,50(1):136-145.]
[9]Fabio Margaglia,Gala Yadgar,Eitan Yaakobi,et al.The devil is in the details:Implementing flash page reuse with WOM codes[C]//14th USENIX Conference on File and Storage Technologies,2016:95-109.
[10]Zhang Fenghao, Huang Jianzhong,Xie Changsheng.Two efficient partial-updating schemes for erasure-coded storage clusters[C]//7th International Conference on Networking,Architecture,and Storage,2012:21-30.
[11]CHEN Fu,ZHANG Shuping.Research on efficient erasure codes reconstruction in distributed block storage systems[J].Software Guide,2016,15(7):33-35(in Chinese).[陈付,张淑萍.分布式块存储系统中高效纠删码重构技术研究[J].软件导刊,2016,15(7):33-35.]