基于Hadoop的区域土壤侵蚀强度计算GIS设计与实验
|
摘要
在开源分布式计算框架Hadoop的基础上,选择大凌河流域为研究区,构建降雨侵蚀力、土壤可侵蚀性、坡长等土壤侵蚀因子栅格数据分布式存储策略,利用MapReduce编程模式设计实现通用水土流失方程RUSLE的算法程序进行土壤侵蚀强度计算,并扩展Geoserver的栅格文件读取功能,增大对HDFS栅格文件的读取支持,实现土壤侵蚀强度计算结果的快速发布。结果表明,该系统可实现大数据量、高分辨率土壤侵蚀因子栅格数据的区域土壤侵蚀强度快速计算,可较好满足区域土壤侵蚀预测预报的时效性要求;程序运行过程中计算任务本地化效果明显,显著降低了计算过程中的网络传输压力,网络使用率约22%,且系统性能可通过增加计算节点进一步提升。本研究可为大数据量、高分辨率栅格数据下区域土壤侵蚀强度计算及发布GIS设计开发提供指导,为大凌河流域土壤侵蚀监测提供科学依据。
At present,the research trends and hotspots of regional soil erosion are related to efficiently accessing a large amount of soil erosion factors raster data,quickly calculating and publishing the date of soil erosion intensity. To solve these problem,a GIS was constructed,which consisted of the distributed storage of soil erosion factor raster data,the parallel computing program to calculate soil erosion intensity and the quickly publishing program of soil erosion intensity,based on the latest open source project called Hadoop. Then an experiment of calculating the Da Ling River Basin 's soil erosion intensity was taken through the system. The result showed that initially the system can much effectively calculate Da Ling River Basin's soil erosion intensity from a large amount of soil erosion factors raster data. Furthermore,the system can fulfill the requirement of regional soil erosion intensity prediction and forecast.Additionally,in the process of running the program,the localization effect of the calculation task was very obvious,the network transmission pressure in the calculation process was significantly reduced,the network usage rate was maintained at about 22%,and the system performance can be further improved by adding computing nodes. Finally,the studies can provide guidance for GIS design and development of regional soil erosion intensity calculating and publishing with a large amount of high-resolution raster data,and provide scientific basis for soil erosion monitoring in Da Ling River Basin.
At present,the research trends and hotspots of regional soil erosion are related to efficiently accessing a large amount of soil erosion factors raster data,quickly calculating and publishing the date of soil erosion intensity. To solve these problem,a GIS was constructed,which consisted of the distributed storage of soil erosion factor raster data,the parallel computing program to calculate soil erosion intensity and the quickly publishing program of soil erosion intensity,based on the latest open source project called Hadoop. Then an experiment of calculating the Da Ling River Basin 's soil erosion intensity was taken through the system. The result showed that initially the system can much effectively calculate Da Ling River Basin's soil erosion intensity from a large amount of soil erosion factors raster data. Furthermore,the system can fulfill the requirement of regional soil erosion intensity prediction and forecast.Additionally,in the process of running the program,the localization effect of the calculation task was very obvious,the network transmission pressure in the calculation process was significantly reduced,the network usage rate was maintained at about 22%,and the system performance can be further improved by adding computing nodes. Finally,the studies can provide guidance for GIS design and development of regional soil erosion intensity calculating and publishing with a large amount of high-resolution raster data,and provide scientific basis for soil erosion monitoring in Da Ling River Basin.
引文
1 LIU Y,FU B J,LU Y H,et al.Hydrological responses and soil erosion potential of abandoned cropland in the Loess Plateau,China[J].Geomorphology,2012,138(1):404-414.
2李锐,上官周平,刘宝元,等.近60年我国土壤侵蚀科学研究进展[J].中国水土保持科学,2009,7(5):1-6.LI Rui,SHANGGUAN Zhouping,LIU Baoyuan,et al.Advances of soil erosion research during the past 60 years in China[J].Science of Soil and Water Conservation,2009,7(5):1-6.(in Chinese)
3张岩,刘宪春,李智广,等.利用侵蚀模型普查黄土高原土壤侵蚀状况[J].农业工程学报,2012,28(10):156-170.ZHANG Yan,LIU Xianchun,LI Zhiguang,et al.Surveying soil erosion condition in Loess Plateau using soil erosion model[J].Transactions of the CSAE,2012,28(10):156-170.(in Chinese)
4杨琴科,李锐,刘咏梅,等.区域土壤侵蚀普查方法的初步讨论[J].中国水土保持科学,2008,6(3):1-7.YANG Qinke,LI Rui,LIU Yongmei,et al.Discussions on methods of national regional soil erosion survey[J].Science of Soil and Water Conservation,2008,6(3):1-7.(in Chinese)
5陈正发,史东梅,何伟,等.云南省降雨侵蚀力时空分布与演变趋势研究[J/OL].农业机械学报,2017,48(10):209-219.http:∥www.j-csam.org/jcsam/ch/reader/view_abstract.aspx?flag=1&file_no=20171026&journal_id=jcsam.DOI:10.6041/j.issn.1000-1298.2017.10.026.CHEN Zhengfa,SHI Dongmei,HE Wei,et al.Spatial-temporal distribution and trend of rainfall erosivity in Yunnan Province[J/OL].Transactions of the Chinese Society for Agricultural Machinery,2017,48(10):209-219.(in Chinese)
6李桂芳,郑粉莉,卢嘉,等.降雨和地形因子对黑土坡面土壤侵蚀过程的影响[J/OL].农业机械学报,2015,46(4):147-154.http:∥www.j-csam.org/jcsam/ch/reader/view_abstract.aspx?flag=1&file_no=20150422&journal_id=jcsam.DOI:10.6041/j.issn.1000-1298.2015.04.022.LI Guifang,ZHENG Fenli,LU Jia,et al.Effects of rainfall and topography on soil erosion processes of black soil hillslope[J/OL].Transactions of the Chinese Society for Agricultural Machinery,2015,46(4):147-154.(in Chinese)
7李智广,姜学兵,刘二佳,等.我国水土保持监测技术和方法的现状与发展方向[J].中国水土保持科学,2015,13(4):145-146.LI Zhiguang,JIANG Xuebing,LIU Erjia,et al.Present status and future development of technologies and methods for soil and water conservation monitoring in China[J].Science of Soil and Water Conservation,2015,13(4):145-146.(in Chinese)
8王文娟,张树文,李颖,等.高分辨率遥感影像在水土流失定量评价中的应用探讨---以Quickbird影像为例[J].土壤与作物,2008,24(4):441-446.WANG Wenjuan,ZHANG Shuwen,LI Ying,et al.Application of high-resolution images on soil loss quantitative estimation-a case of Quikbird[J].Soil and Crop,2008,24(4):441-446.(in Chinese)
9史明昌,刘宪春,罗志东,等.土壤侵蚀普查数据管理及数据库构建[J].中国水土保持,2013(10):51-53.SHI Mingchang,LIU Xianchun,LUO Zhidong,et al.Soil erosion survey data management and database construction[J].Soil and Water Conservation in China,2013(10):51-53.(in Chinese)
10李德仁,童庆禧,李荣兴,等.高分辨率对地观测的若干前沿科学问题[J].中国科学:地球科学,2012,42(6):805-809.LI Deren,TONG Qingxi,LI Rongxing,et al.Some frontier scientific issues of high resolution earth observation[J].Science China:Earth Science,2012,42(6):805-809.(in Chinese)
11何国金,王力哲,马艳,等.对地观测大数据处理:挑战与思考[J].科学通报,2015(5):470-478.HE Guojin,WANG Lizhe,MA Yan,et al.Processing of earth observation big data:challenges and countermeasures[J].Chinese Science Bulletin,2015(5):470-478.(in Chinese)
12李清泉,李德仁.大数据GIS[J].武汉大学学报(信息科学版),2014,39(6):642-643.LI Qingquan,LI Deren.Big data GIS[J].Geomatics and Information Science of Wuhan University,2014,39(6):642-643.(in Chinese)
13冯敏,尹芳,诸云强,等.基于MapReduce的分布式地形数据计算研究[J].华中科技大学学报(自然科学版),2011,39(增刊1):24-27.FENG Min,YIN Fang,ZHU Yunqiang,et al.Terrain data distributed computing using MapReduce[J].Journal of Huazhong University of Science and Technology(Natural Science Edition),2011,39(Supp.1):24-27.(in Chinese)
14刘磊,尹芳,冯敏,等.基于开源Hadoop的栅格数据分布式处理[J].华中科技大学学报(自然科学版),2013,41(7):103-108.LIU Lei,YIN Fang,FENG Min,et al.Distributed computation of raster data using open source Hadoop[J].Journal of Huazhong University of Science and Technology(Natural Science Edition),2013,41(7):103-108.(in Chinese)
15吕雪锋,程承旗,龚健雅,等.海量遥感数据存储管理技术综述[J].中国科学:技术科学,2011,41(12):1561-1573.LXuefeng,CHENG Chengqi,GONG Jianya,et al.Massive remote sensing data storage management technology[J].Science China:Technology Science,2011,41(12):1561-1573.(in Chinese)
16张永瀚,孙瑞志,李林,等.生态环境遥感评价模型库系统平台构建研究[J/OL].农业机械学报,2015,46(增刊):267-273.http:∥www.j-csam.org/jcsam/ch/reader/view_abstract.aspx?flag=1&file_no=2015S043&journal_id=jcsam.DOI:10.6041/j.issn.1000-1298.2015.S0.043.ZHANG Yonghan,SUN Ruizhi,LI Lin,et al.Research of remote sensing evaluation model library platform of ecological environment[J/OL].Transactions of the Chinese Society for Agricultural Machinery,2015,46(Supp.):267-273.(in Chinese)
17买凯乐,张文辉.黄土丘陵沟壑区遥感影像信息面向对象分类方法提取[J].农业机械学报,2011,42(4):153-158.MAI Kaile,ZHANG Wenhui.Object-oriented classification approach for remote sensing imagery information extraction in Loess Hilly-gully region[J].Transactions of the Chinese Society for Agricultural Machinery,2011,42(4):153-158.(in Chinese)
18 HONNUTAGI P S.The Hadoop distributed file system[J].International Journal of Computer Science and Information Technologies,2014,5(5):6238-6243.
19 WANG Ke,QIAO Kan,SADOOGHI I,et al.Load-balanced and locality-aware scheduling for data-intensive workloads at extreme scales[J].Concurrency&Computation Practice&Experience,2016,28(1):70-94.
20 MALAKAR R,VYDYANATHAN N.A CUDA-enabled Hadoop cluster for fast distributed image processing[C].2013 National Conference on Parallel Computing Technologies,2013.
21 JHUMMARWALA A,POTDAR M B,CHAUHAN P,et al.Parallel and distributed GIS for processing Geo-data:an overview[J].International Journal of Computer Applications,2014,106(16):9-16.
22 LI Jiyuan,MENG Lingkui,WANG F Z,et al.A Map-Reduce-enabled SOLAP cube for large-scale remotely sensed data aggregation[J].Computers&Geosciences,2014,70(9):110-119.
23中华人民共和国水利部.中国土壤侵蚀分类分级标准:SL 190-2007[S].北京:中国水利水电出版社,2007.
2李锐,上官周平,刘宝元,等.近60年我国土壤侵蚀科学研究进展[J].中国水土保持科学,2009,7(5):1-6.LI Rui,SHANGGUAN Zhouping,LIU Baoyuan,et al.Advances of soil erosion research during the past 60 years in China[J].Science of Soil and Water Conservation,2009,7(5):1-6.(in Chinese)
3张岩,刘宪春,李智广,等.利用侵蚀模型普查黄土高原土壤侵蚀状况[J].农业工程学报,2012,28(10):156-170.ZHANG Yan,LIU Xianchun,LI Zhiguang,et al.Surveying soil erosion condition in Loess Plateau using soil erosion model[J].Transactions of the CSAE,2012,28(10):156-170.(in Chinese)
4杨琴科,李锐,刘咏梅,等.区域土壤侵蚀普查方法的初步讨论[J].中国水土保持科学,2008,6(3):1-7.YANG Qinke,LI Rui,LIU Yongmei,et al.Discussions on methods of national regional soil erosion survey[J].Science of Soil and Water Conservation,2008,6(3):1-7.(in Chinese)
5陈正发,史东梅,何伟,等.云南省降雨侵蚀力时空分布与演变趋势研究[J/OL].农业机械学报,2017,48(10):209-219.http:∥www.j-csam.org/jcsam/ch/reader/view_abstract.aspx?flag=1&file_no=20171026&journal_id=jcsam.DOI:10.6041/j.issn.1000-1298.2017.10.026.CHEN Zhengfa,SHI Dongmei,HE Wei,et al.Spatial-temporal distribution and trend of rainfall erosivity in Yunnan Province[J/OL].Transactions of the Chinese Society for Agricultural Machinery,2017,48(10):209-219.(in Chinese)
6李桂芳,郑粉莉,卢嘉,等.降雨和地形因子对黑土坡面土壤侵蚀过程的影响[J/OL].农业机械学报,2015,46(4):147-154.http:∥www.j-csam.org/jcsam/ch/reader/view_abstract.aspx?flag=1&file_no=20150422&journal_id=jcsam.DOI:10.6041/j.issn.1000-1298.2015.04.022.LI Guifang,ZHENG Fenli,LU Jia,et al.Effects of rainfall and topography on soil erosion processes of black soil hillslope[J/OL].Transactions of the Chinese Society for Agricultural Machinery,2015,46(4):147-154.(in Chinese)
7李智广,姜学兵,刘二佳,等.我国水土保持监测技术和方法的现状与发展方向[J].中国水土保持科学,2015,13(4):145-146.LI Zhiguang,JIANG Xuebing,LIU Erjia,et al.Present status and future development of technologies and methods for soil and water conservation monitoring in China[J].Science of Soil and Water Conservation,2015,13(4):145-146.(in Chinese)
8王文娟,张树文,李颖,等.高分辨率遥感影像在水土流失定量评价中的应用探讨---以Quickbird影像为例[J].土壤与作物,2008,24(4):441-446.WANG Wenjuan,ZHANG Shuwen,LI Ying,et al.Application of high-resolution images on soil loss quantitative estimation-a case of Quikbird[J].Soil and Crop,2008,24(4):441-446.(in Chinese)
9史明昌,刘宪春,罗志东,等.土壤侵蚀普查数据管理及数据库构建[J].中国水土保持,2013(10):51-53.SHI Mingchang,LIU Xianchun,LUO Zhidong,et al.Soil erosion survey data management and database construction[J].Soil and Water Conservation in China,2013(10):51-53.(in Chinese)
10李德仁,童庆禧,李荣兴,等.高分辨率对地观测的若干前沿科学问题[J].中国科学:地球科学,2012,42(6):805-809.LI Deren,TONG Qingxi,LI Rongxing,et al.Some frontier scientific issues of high resolution earth observation[J].Science China:Earth Science,2012,42(6):805-809.(in Chinese)
11何国金,王力哲,马艳,等.对地观测大数据处理:挑战与思考[J].科学通报,2015(5):470-478.HE Guojin,WANG Lizhe,MA Yan,et al.Processing of earth observation big data:challenges and countermeasures[J].Chinese Science Bulletin,2015(5):470-478.(in Chinese)
12李清泉,李德仁.大数据GIS[J].武汉大学学报(信息科学版),2014,39(6):642-643.LI Qingquan,LI Deren.Big data GIS[J].Geomatics and Information Science of Wuhan University,2014,39(6):642-643.(in Chinese)
13冯敏,尹芳,诸云强,等.基于MapReduce的分布式地形数据计算研究[J].华中科技大学学报(自然科学版),2011,39(增刊1):24-27.FENG Min,YIN Fang,ZHU Yunqiang,et al.Terrain data distributed computing using MapReduce[J].Journal of Huazhong University of Science and Technology(Natural Science Edition),2011,39(Supp.1):24-27.(in Chinese)
14刘磊,尹芳,冯敏,等.基于开源Hadoop的栅格数据分布式处理[J].华中科技大学学报(自然科学版),2013,41(7):103-108.LIU Lei,YIN Fang,FENG Min,et al.Distributed computation of raster data using open source Hadoop[J].Journal of Huazhong University of Science and Technology(Natural Science Edition),2013,41(7):103-108.(in Chinese)
15吕雪锋,程承旗,龚健雅,等.海量遥感数据存储管理技术综述[J].中国科学:技术科学,2011,41(12):1561-1573.LXuefeng,CHENG Chengqi,GONG Jianya,et al.Massive remote sensing data storage management technology[J].Science China:Technology Science,2011,41(12):1561-1573.(in Chinese)
16张永瀚,孙瑞志,李林,等.生态环境遥感评价模型库系统平台构建研究[J/OL].农业机械学报,2015,46(增刊):267-273.http:∥www.j-csam.org/jcsam/ch/reader/view_abstract.aspx?flag=1&file_no=2015S043&journal_id=jcsam.DOI:10.6041/j.issn.1000-1298.2015.S0.043.ZHANG Yonghan,SUN Ruizhi,LI Lin,et al.Research of remote sensing evaluation model library platform of ecological environment[J/OL].Transactions of the Chinese Society for Agricultural Machinery,2015,46(Supp.):267-273.(in Chinese)
17买凯乐,张文辉.黄土丘陵沟壑区遥感影像信息面向对象分类方法提取[J].农业机械学报,2011,42(4):153-158.MAI Kaile,ZHANG Wenhui.Object-oriented classification approach for remote sensing imagery information extraction in Loess Hilly-gully region[J].Transactions of the Chinese Society for Agricultural Machinery,2011,42(4):153-158.(in Chinese)
18 HONNUTAGI P S.The Hadoop distributed file system[J].International Journal of Computer Science and Information Technologies,2014,5(5):6238-6243.
19 WANG Ke,QIAO Kan,SADOOGHI I,et al.Load-balanced and locality-aware scheduling for data-intensive workloads at extreme scales[J].Concurrency&Computation Practice&Experience,2016,28(1):70-94.
20 MALAKAR R,VYDYANATHAN N.A CUDA-enabled Hadoop cluster for fast distributed image processing[C].2013 National Conference on Parallel Computing Technologies,2013.
21 JHUMMARWALA A,POTDAR M B,CHAUHAN P,et al.Parallel and distributed GIS for processing Geo-data:an overview[J].International Journal of Computer Applications,2014,106(16):9-16.
22 LI Jiyuan,MENG Lingkui,WANG F Z,et al.A Map-Reduce-enabled SOLAP cube for large-scale remotely sensed data aggregation[J].Computers&Geosciences,2014,70(9):110-119.
23中华人民共和国水利部.中国土壤侵蚀分类分级标准:SL 190-2007[S].北京:中国水利水电出版社,2007.