虚拟数据代理云模型构建及数据布局
|
摘要
综合考虑数据集间的依赖关系以及数据中心的存储容量,引入一种名为"虚拟数据代理"的新实体,通过建立虚拟数据代理云模型,将数据布局问题转换为2个映射过程,即从数据集到虚拟数据代理的映射以及从虚拟数据代理到数据中心的映射,进而提出一种基于虚拟数据代理的云模型数据布局策略(CDPVDA)。仿真实验结果表明:CDPVDA与典型的数据放置策略相比,可以将数据中心之间的数据传输开销降低5%～20%。
Considering the dependence between data sets and the storage capacity of data centers, a new type of entity called virtual data agent was introduced. The data placement problem was converted into two mapping processes namely mapping from the data set to the virtual data agent and mapping from the virtual data agent to the data center. Cloud model based data placement algorithm with virtual data agent(CDPVDA) was proposed. The results of simulation experiment show that compared with two typical data placement strategies, CDPVDA can reduce data transmission overhead between data centers by 5% to 20%.
Considering the dependence between data sets and the storage capacity of data centers, a new type of entity called virtual data agent was introduced. The data placement problem was converted into two mapping processes namely mapping from the data set to the virtual data agent and mapping from the virtual data agent to the data center. Cloud model based data placement algorithm with virtual data agent(CDPVDA) was proposed. The results of simulation experiment show that compared with two typical data placement strategies, CDPVDA can reduce data transmission overhead between data centers by 5% to 20%.
引文
[1]宫学庆,金澈清,王晓玲,等.数据密集型科学与工程:需求和挑战[J].计算机学报,2012,35(8):1563-1578.GONG Xueqing,JIN Cheqing,WANG Xiaoling,et al.Data-intensive science and engineering:requirements and challenges[J].Chinese Journal of Computers,2012,35(8):1563-1578.
[2]BRIAN V E,HENRY H,SASHA A,et al.DI-MMAP-a scalable memory-map runtime for out-of-core data-intensive applications[J].Cluster Computing,2015,18:15-28.
[3]SENYO P K,EFFAH J,ADDAE E.Preliminary insight into cloud computing adoption in a developing country[J].Journal of Enterprise Information Management,2016,29(4):400-422.
[4]SHANG Pengju,WANG Jun.A novel power management for CMP systems in data-intensive environment[C]//IEEEInternational Symposium on Parallel and Distributed Processing.Anchorage,Alaska,USA,2011:92-103.
[5]WANG Shangguang,SUN Qibo,ZHANG Guangwei,et al.Uncertain Qos-aware skyline service selection based on cloud model[J].Journal of Software,2012,23(6):1397-1412.
[6]YUAN Dong,YANG Yun,LIU Xiao.A data placement strategy in scientific cloud workflows[J].Future Generation Computer Systems,2010,26(8):1200-1214.
[7]刘少伟,孔令梅,任开军,等.云环境下优化科学工作流执行性能的两阶段数据放置与任务调度策略[J].计算机学报,2011,34(11):2121-2130.LIU Shaowei,KONG Lingmei,REN Kaijun,et al.A two-step data placement and task scheduling strategy for optimizing scientific workflow performance on cloud computing platform[J].Chinese Journal of Computers,2011,34(11):2121-2130.
[8]DENG Kefeng,REN Kaijun,SONG Junqiang,et al.A clustering based co-scheduling strategy for efficient scientific workflow execution in cloud computing[J].Concurrency and Computation:Practice&Experience,2013,25(18):2523-2539.
[9]张甜甜,崔立真.基于释放和重构的科学工作流数据布局策略[J].计算机研究与发展,2013,50(S2):71-76.ZHANG Tiantian,CUI Lizhen.A data placement strategy based on relaxation and reconstruction for scientific workflow applications[J].Journal of Computer Research and Development,2013,50(S2):71-76.
[10]PANDEY S,WU Linlin,GURU S M,et al.A particle swarm optimization-based heuristic for scheduling workflow applications in cloud computing environments[C]//IEEEInternational Conference on Advanced Information NETWORKING and Applications.Aina,Perth,Australia,2010:400-407.
[11]XU Changlin,WANG Guoyin,ZHANG Qinghua.A new multi-step backward cloud transformation algorithm based on normal cloud model[J].Fundamental Informaticae,2014,133(1):55-85.
[12]CHEN Jinpeng,LIU Yu,LI Deyi.Enhancing recommender diversity using gaussian cloud transformation[J].International Journal of Uncertainty,Fuzziness and Knowledge-Based Systems,2015,23(4):521-544.
[13]MA Hua,HU Zhigang,LI Keqin.Toward trustworthy cloud service selection:a time-aware approach using interval neutrosophic set[J].Journal of Parallel&Distributed Computing,2016,96(C):75-94.
[14]ZHANG Xinxin,HU Zhigang,ZHENG Meiguang,et al.A novel cloud model based data placement strategy for data-intensive application in clouds[EB/OL].[2018-08-21].https://doi.org/10.1016/j.compeleceng.2018.07.007.
[15]WANG Guoyin,XU Changlin,LI Deyi.Generic normal cloud model[J].Information Science,2014,280:1-15.
[16]AUFFHAMMER M,HSIANG S M,SCHLENKER W,et al.Using Weather Data and Climate Model Output in Economic Analyses of Climate Change[J].Review of Environmental Economics&Policy,2013,7(2):181-198.
[17]BAUN C.Mobile clusters of single board computers:an option for providing resources to student projects and researchers[J].Springer Plus,2016,5(1):360.
[18]DALEY C S,GHOSHAL D,LOCKWOOD G K,et al.Performance characterization of scientific workflows for the optimal use of burst buffers[J].Future Generation Computer Systems,2017,53:69-73.
[19]HU Zhigang,LI Jia,ZHENG Meiguang,et al.Hypergraph-based data reduced scheduling policy for data-intensive workflow in clouds[C]//ICPCSEE 2017.Third International Conference of Pioneering Computer Scientists,Engineers and Educators.Changsha,China,2017:335-349.
[2]BRIAN V E,HENRY H,SASHA A,et al.DI-MMAP-a scalable memory-map runtime for out-of-core data-intensive applications[J].Cluster Computing,2015,18:15-28.
[3]SENYO P K,EFFAH J,ADDAE E.Preliminary insight into cloud computing adoption in a developing country[J].Journal of Enterprise Information Management,2016,29(4):400-422.
[4]SHANG Pengju,WANG Jun.A novel power management for CMP systems in data-intensive environment[C]//IEEEInternational Symposium on Parallel and Distributed Processing.Anchorage,Alaska,USA,2011:92-103.
[5]WANG Shangguang,SUN Qibo,ZHANG Guangwei,et al.Uncertain Qos-aware skyline service selection based on cloud model[J].Journal of Software,2012,23(6):1397-1412.
[6]YUAN Dong,YANG Yun,LIU Xiao.A data placement strategy in scientific cloud workflows[J].Future Generation Computer Systems,2010,26(8):1200-1214.
[7]刘少伟,孔令梅,任开军,等.云环境下优化科学工作流执行性能的两阶段数据放置与任务调度策略[J].计算机学报,2011,34(11):2121-2130.LIU Shaowei,KONG Lingmei,REN Kaijun,et al.A two-step data placement and task scheduling strategy for optimizing scientific workflow performance on cloud computing platform[J].Chinese Journal of Computers,2011,34(11):2121-2130.
[8]DENG Kefeng,REN Kaijun,SONG Junqiang,et al.A clustering based co-scheduling strategy for efficient scientific workflow execution in cloud computing[J].Concurrency and Computation:Practice&Experience,2013,25(18):2523-2539.
[9]张甜甜,崔立真.基于释放和重构的科学工作流数据布局策略[J].计算机研究与发展,2013,50(S2):71-76.ZHANG Tiantian,CUI Lizhen.A data placement strategy based on relaxation and reconstruction for scientific workflow applications[J].Journal of Computer Research and Development,2013,50(S2):71-76.
[10]PANDEY S,WU Linlin,GURU S M,et al.A particle swarm optimization-based heuristic for scheduling workflow applications in cloud computing environments[C]//IEEEInternational Conference on Advanced Information NETWORKING and Applications.Aina,Perth,Australia,2010:400-407.
[11]XU Changlin,WANG Guoyin,ZHANG Qinghua.A new multi-step backward cloud transformation algorithm based on normal cloud model[J].Fundamental Informaticae,2014,133(1):55-85.
[12]CHEN Jinpeng,LIU Yu,LI Deyi.Enhancing recommender diversity using gaussian cloud transformation[J].International Journal of Uncertainty,Fuzziness and Knowledge-Based Systems,2015,23(4):521-544.
[13]MA Hua,HU Zhigang,LI Keqin.Toward trustworthy cloud service selection:a time-aware approach using interval neutrosophic set[J].Journal of Parallel&Distributed Computing,2016,96(C):75-94.
[14]ZHANG Xinxin,HU Zhigang,ZHENG Meiguang,et al.A novel cloud model based data placement strategy for data-intensive application in clouds[EB/OL].[2018-08-21].https://doi.org/10.1016/j.compeleceng.2018.07.007.
[15]WANG Guoyin,XU Changlin,LI Deyi.Generic normal cloud model[J].Information Science,2014,280:1-15.
[16]AUFFHAMMER M,HSIANG S M,SCHLENKER W,et al.Using Weather Data and Climate Model Output in Economic Analyses of Climate Change[J].Review of Environmental Economics&Policy,2013,7(2):181-198.
[17]BAUN C.Mobile clusters of single board computers:an option for providing resources to student projects and researchers[J].Springer Plus,2016,5(1):360.
[18]DALEY C S,GHOSHAL D,LOCKWOOD G K,et al.Performance characterization of scientific workflows for the optimal use of burst buffers[J].Future Generation Computer Systems,2017,53:69-73.
[19]HU Zhigang,LI Jia,ZHENG Meiguang,et al.Hypergraph-based data reduced scheduling policy for data-intensive workflow in clouds[C]//ICPCSEE 2017.Third International Conference of Pioneering Computer Scientists,Engineers and Educators.Changsha,China,2017:335-349.