结合大数据流特征和改进SOM聚类的资源动态分配算法
|
摘要
在大数据流中,由于数据特征的未知性,如何分配数据资源是一个难题。为了解决这个问题,提出一种大数据环境下基于数据特征预测和改进自组织映射SOM(Self-organizing maps)的资源管理算法。根据数据的体积和速度变化,通过自回归模型对下一时间间隔到达的数据的特征进行估计,估计值用数据特征(CoD)向量表示;利用粒子群优化PSO算法来优化SOM算法的权重分布,形成改进型SOM算法,对CoD向量进行聚类,动态创建和分配云资源集群。这些集群以拓扑排序的方式创建,集群之间的联系越多,它们的排序越接近,利用这种拓扑排序来减少等待时间。实验结果表明,该算法能准确预测数据特征,有效提高了云资源的利用率。
In big data streams, it is a difficult problem to allocate data resources due to the unknown characteristics of data. To solve this problem, we proposed a resource management algorithm based on data characteristic prediction and improved self-organizing mapping(SOM) in big data environment. We estimated the data characteristics arriving at the next time interval by the autoregressive model according to the volume and velocity of the data, and the estimated value was represented by the characteristics of data(CoD) vector. Then we used the particle swarm optimization(PSO) to optimize the weight distribution of SOM algorithm, and an improved SOM algorithm was formed to cluster the CoD vectors, so as to dynamically create and allocate cloud resource clusters. These clusters were created in a topological sort. The more connections between clusters, the closer their sort was, so using this topological sort could reduce waiting time. The experimental results show that the proposed algorithm can not only accurately predict the data characteristics, but also effectively improve the utilization of cloud resources.
In big data streams, it is a difficult problem to allocate data resources due to the unknown characteristics of data. To solve this problem, we proposed a resource management algorithm based on data characteristic prediction and improved self-organizing mapping(SOM) in big data environment. We estimated the data characteristics arriving at the next time interval by the autoregressive model according to the volume and velocity of the data, and the estimated value was represented by the characteristics of data(CoD) vector. Then we used the particle swarm optimization(PSO) to optimize the weight distribution of SOM algorithm, and an improved SOM algorithm was formed to cluster the CoD vectors, so as to dynamically create and allocate cloud resource clusters. These clusters were created in a topological sort. The more connections between clusters, the closer their sort was, so using this topological sort could reduce waiting time. The experimental results show that the proposed algorithm can not only accurately predict the data characteristics, but also effectively improve the utilization of cloud resources.
引文
[1] Liu C,Yang C,Zhang X,et al.External integrity verification for outsourced big data in cloud and IoT:A big picture[J].Future Generation Computer Systems,2015,49(C):58-67.
[2] 王元卓,靳小龙,程学旗.网络大数据:现状与展望[J].计算机学报,2013,36(6):1125-1138.
[3] 郑宇超,夏学文,艾冬梅.基于队列理论的云资源分配收益最大化算法[J].计算机应用与软件,2017,34(11):252-257.
[4] Vasile M A,Pop F,Tutueanu R I,et al.Resource-aware hybrid scheduling algorithm in heterogeneous distributed computing[J].Future Generation Computer Systems,2015,51(C):61-71.
[5] Sandhu R,Sood S K.Scheduling of big data applications on distributed cloud based on QoS parameters[J].Cluster Computing,2015,18(2):817-828.
[6] Sun D,Zhang G,Yang S,et al.Re-Stream:Real-time and energy-efficient resource scheduling in big data stream computing environments[J].Information Sciences,2015,319:92-112.
[7] Rahman M M,Graham P.Responsive and efficient provisioning for multimedia applications[J].Computers & Electrical Engineering,2016,53(C):458-468.
[8] Zhang Q,Chen Z,Yang L T.A nodes scheduling model based on Markov chain prediction for big streaming data analysis[J].International Journal of Communication Systems,2015,28(9):1610-1619.
[9] Castiglione A,Pizzolante R,Santis A D,et al.Cloud-based adaptive compression and secure management services for 3D healthcare data[J].Future Generation Computer Systems,2015,43/44:120-134.
[10] Baughman A K,Bogdany R J,Mcavoy C,et al.Predictive Cloud Computing with Big Data:Professional Golf and Tennis Forecasting [Application Notes][J].Computational Intelligence Magazine IEEE,2015,10(3):62-76.
[11] 王旭,王国中,范涛.深度图像的分块自适应压缩感知[J].计算机应用研究,2016,33(3):903-906.
[12] Zhou A C,He B,Liu C.Monetary Cost Optimizations for Hosting Workflow-as-a-Service in IaaS Clouds[J].IEEE Transactions on Cloud Computing,2016,4(1):34-48.
[13] 张蕾,章毅.大数据分析的无限深度神经网络方法[J].计算机研究与发展,2016,53(1):68-79.
[14] 李玮,张大方,黄昆,等.面向大数据处理的高精度多维计数布鲁姆过滤器[J].电子学报,2015,43(4):652-657.
[15] Li K,Zhu Y,Yang J,et al.Video super-resolution using an adaptive superpixel-guided auto-regressive model[J].Pattern Recognition,2016,51(C):59-71.
[16] Agarwal A,Maheswaran R,Kurths J,et al.Wavelet Spectrum and Self-Organizing Maps-Based Approach for Hydrologic Regionalization -a Case Study in the Western United States[J].Water Resources Management,2016,30(12):4399-4413.
[2] 王元卓,靳小龙,程学旗.网络大数据:现状与展望[J].计算机学报,2013,36(6):1125-1138.
[3] 郑宇超,夏学文,艾冬梅.基于队列理论的云资源分配收益最大化算法[J].计算机应用与软件,2017,34(11):252-257.
[4] Vasile M A,Pop F,Tutueanu R I,et al.Resource-aware hybrid scheduling algorithm in heterogeneous distributed computing[J].Future Generation Computer Systems,2015,51(C):61-71.
[5] Sandhu R,Sood S K.Scheduling of big data applications on distributed cloud based on QoS parameters[J].Cluster Computing,2015,18(2):817-828.
[6] Sun D,Zhang G,Yang S,et al.Re-Stream:Real-time and energy-efficient resource scheduling in big data stream computing environments[J].Information Sciences,2015,319:92-112.
[7] Rahman M M,Graham P.Responsive and efficient provisioning for multimedia applications[J].Computers & Electrical Engineering,2016,53(C):458-468.
[8] Zhang Q,Chen Z,Yang L T.A nodes scheduling model based on Markov chain prediction for big streaming data analysis[J].International Journal of Communication Systems,2015,28(9):1610-1619.
[9] Castiglione A,Pizzolante R,Santis A D,et al.Cloud-based adaptive compression and secure management services for 3D healthcare data[J].Future Generation Computer Systems,2015,43/44:120-134.
[10] Baughman A K,Bogdany R J,Mcavoy C,et al.Predictive Cloud Computing with Big Data:Professional Golf and Tennis Forecasting [Application Notes][J].Computational Intelligence Magazine IEEE,2015,10(3):62-76.
[11] 王旭,王国中,范涛.深度图像的分块自适应压缩感知[J].计算机应用研究,2016,33(3):903-906.
[12] Zhou A C,He B,Liu C.Monetary Cost Optimizations for Hosting Workflow-as-a-Service in IaaS Clouds[J].IEEE Transactions on Cloud Computing,2016,4(1):34-48.
[13] 张蕾,章毅.大数据分析的无限深度神经网络方法[J].计算机研究与发展,2016,53(1):68-79.
[14] 李玮,张大方,黄昆,等.面向大数据处理的高精度多维计数布鲁姆过滤器[J].电子学报,2015,43(4):652-657.
[15] Li K,Zhu Y,Yang J,et al.Video super-resolution using an adaptive superpixel-guided auto-regressive model[J].Pattern Recognition,2016,51(C):59-71.
[16] Agarwal A,Maheswaran R,Kurths J,et al.Wavelet Spectrum and Self-Organizing Maps-Based Approach for Hydrologic Regionalization -a Case Study in the Western United States[J].Water Resources Management,2016,30(12):4399-4413.