基于新一代大数据处理引擎Flink的“智慧滁河”系统
|
摘要
概述了水利领域大数据的特点,展示了基于Flink构建的"智慧滁河"系统,并以在滁河监测中获取的传感器数据为实验数据,以常用的查询操作测试了系统性能。结果表明,采用Flink的"智慧滁河"系统的处理能力远超传统的多层架构系统,可为水利信息化迈向"智慧"提供可行的解决方案。
The characteristics of big data in water conservancy were summarized,and a "Smart Chuhe River"systems based on Flink was demonstrated.Taking the sensor data obtained from Chuhe River monitoring as experimental data,the system performance is tested by common query operations.The results show that"Smart Chuhe"system based on Flink has far more processing power than the traditional multi-tier framework system,providing feasible solutions for water conservancy informatization towards"smart".
The characteristics of big data in water conservancy were summarized,and a "Smart Chuhe River"systems based on Flink was demonstrated.Taking the sensor data obtained from Chuhe River monitoring as experimental data,the system performance is tested by common query operations.The results show that"Smart Chuhe"system based on Flink has far more processing power than the traditional multi-tier framework system,providing feasible solutions for water conservancy informatization towards"smart".
引文
[1]IBM商业价值研究院.智慧地球赢在中国[EB/OL].[2014-10-30].http://www-31.ibm.com/innovation/cn/think/downloads/smart_China.pdf
[2]HEY T,TANSLEY S,TOLLE K.The fourth paradigm:data-intensive scientific discovery[M].Redmond Washington:Microsoft Research,2009.
[3]孟小峰,慈祥.大数据管理:概念、技术与挑战[J].计算机研究与发展,2013,50(1):146-169.(MENGXiaofeng,CI Xiang.Big data management:concepts,techniques and challenges[J].Journal of Computer Research and Development,2013,50(1):146-169.(in Chinese))
[4]李国杰,程学旗.大数据研究:未来科技及经济社会发展的重大战略领域:大数据的研究现状与科学思考[J].中国科学院院刊,2012(6):647-657.(LI Guojie,CHENG Xueqi.Research status and scientific thinking of big data[J].Bulletin of Chinese Academy of Sciences,2012(6):647-657.(in Chinese))
[5]蒋云钟,冶运涛,王浩.基于物联网理念的流域智能调度技术体系刍议[J].水利信息化,2010(4):1-10.(JIANG Yunzhong,YE Yuntao,WANG Hao.Discussion on intelligent regulation technology architecture for river basin based on interact of things[J].Water Resources Informatization,2010(4):1-10.(in Chinese))
[6]蒋云钟,冶运涛,王浩.智慧流域及其应用前景[J].系统工程理论与实践,2011,31(6):1174-1181.(JIANGYunzhong,YE Yuntao,WANG Hao.Smart basin and its prospects for application[J].Systems Engineering:Theory&Practice,2011,31(6):1174-1181.(in Chinese))
[7]王忠静,王光谦,王建华,等.基于水联网及智慧水利提高水资源效能[J].水利水电技术,2013,44(1):1-6.(WANG Zhongjing,WANG Guangqian,WANG Jianhua,et al.Developing the internet of water to prompt water utilization efficiency[J].Water Resources and Hydropower Engineering,2013,44(1):1-6.(in Chinese))
[8]许峰,朱跃龙,叶枫.水利云平台的研究与应用[C]//电子政务理事会.中国电子政务年鉴(2012).北京:社会科学文献出版社,2012.
[9]芮晓玲,吴一凡.基于物联网技术的智慧水利系统[J].计算机系统应用,2012,21(6):161-163.(RUI Xiaoling,WU Yifan.Intelligent system of water conservancy based on internet of things[J].Computer Systems&Applications,2012,21(6):161-163.(in Chinese))
[10]冯钧,许潇,唐志贤等.水利大数据及其资源化关键技术研究[J].水利信息化,2013(8):6-9.(FENG Jun,XUXiao,TANG Zhixian,et al.Research on key technology of water big data and resource utilization[J].Water Resources Informatization,2013(8):6-9.(in Chinese))
[11]于嘉骥,张慧妍,王小艺,等.基于改进的投影寻踪-云模型的农业灌溉水质综合评价[J].水资源保护,2017,33(6):142-146.(YU Jiaji,ZHANG Huiyan,WANGXiaoyi,et al.Comprehensive evaluation of agricultural irrigation water quality based on modified projection pursuit-cloud model[J].Water Resources Protection,2017,33(6):142-146.(in Chinese))
[12]HELSEL D R,HIRSCH R M.Statistical methods in water resources[EB/OL].[2018-05-20].http://water.usgs.gov/pubs/twri/twri4a3/.
[13]黄黎明,张可,龚寻,等.大数据视角下水利工程质量风险管理[J].水利经济,2017,35(6):66-70.(HUANGLiming,ZHANG Ke,GONG Xun,et al.Management of quality risk of water conservancy projects from perspective of big data[J].Journal of Economics of Water Resources,2017,35(6):66-70.(in Chinese))
[14]FRIEDMAN E,TZOUMAS K.Introduction to apache flink:stream processing for real time and beyond[M].Sebastopol:O'Reilly Media,2016.
[15]DESHPANDE T.Learning apache flink[M].Birmingham:Packt Publishing,2017.
[16]CARBONE P,GBOR E.HERMANN G,et al.Largescale data stream processing systems[C]//Handbook of Big Data Technologies.Berlin:Springer,2017.
[17]LOPEZ M A,LOCATOR A G P,CARLOS O M B.Aperformance comparison of open-source stream processing platforms[C]//Proceedings of 2016 IEEEGlobal Communications Conference.New York:IEEEComputer Society,2016.
[18]CHINTAPALLI S,DAGIT D,EVANS B,et al.Benchmarking streaming computation engines:storm,flink and spark streaming[C]//Proceedings of IEEE 28th International Parallel and Distributed Processing Symposium Workshops.New York:IEEE Computer Society,2016.
[19]SPANGENBERG N,ROTH M,FRANCZYK B.Evaluating new approaches of big data analytics frameworks[C]//Proceedings of 18th International Conference on Business Information Systems.Berlin:Springer Verlag,2015.
[20]WANG Lizhe,RAJIV R,CHEN Jinjun,et a1.Cloud computing:methodology,systems and applications[M].Boca Raton:CRC Press,2012.
[21]TIWARI S.Professional No SQL[M].Indianapolis:John Wiley&Son,2011.
[22]陆嘉恒.大数据挑战与No SQL数据库技术[M].北京:电子工业出版社,2013.
[2]HEY T,TANSLEY S,TOLLE K.The fourth paradigm:data-intensive scientific discovery[M].Redmond Washington:Microsoft Research,2009.
[3]孟小峰,慈祥.大数据管理:概念、技术与挑战[J].计算机研究与发展,2013,50(1):146-169.(MENGXiaofeng,CI Xiang.Big data management:concepts,techniques and challenges[J].Journal of Computer Research and Development,2013,50(1):146-169.(in Chinese))
[4]李国杰,程学旗.大数据研究:未来科技及经济社会发展的重大战略领域:大数据的研究现状与科学思考[J].中国科学院院刊,2012(6):647-657.(LI Guojie,CHENG Xueqi.Research status and scientific thinking of big data[J].Bulletin of Chinese Academy of Sciences,2012(6):647-657.(in Chinese))
[5]蒋云钟,冶运涛,王浩.基于物联网理念的流域智能调度技术体系刍议[J].水利信息化,2010(4):1-10.(JIANG Yunzhong,YE Yuntao,WANG Hao.Discussion on intelligent regulation technology architecture for river basin based on interact of things[J].Water Resources Informatization,2010(4):1-10.(in Chinese))
[6]蒋云钟,冶运涛,王浩.智慧流域及其应用前景[J].系统工程理论与实践,2011,31(6):1174-1181.(JIANGYunzhong,YE Yuntao,WANG Hao.Smart basin and its prospects for application[J].Systems Engineering:Theory&Practice,2011,31(6):1174-1181.(in Chinese))
[7]王忠静,王光谦,王建华,等.基于水联网及智慧水利提高水资源效能[J].水利水电技术,2013,44(1):1-6.(WANG Zhongjing,WANG Guangqian,WANG Jianhua,et al.Developing the internet of water to prompt water utilization efficiency[J].Water Resources and Hydropower Engineering,2013,44(1):1-6.(in Chinese))
[8]许峰,朱跃龙,叶枫.水利云平台的研究与应用[C]//电子政务理事会.中国电子政务年鉴(2012).北京:社会科学文献出版社,2012.
[9]芮晓玲,吴一凡.基于物联网技术的智慧水利系统[J].计算机系统应用,2012,21(6):161-163.(RUI Xiaoling,WU Yifan.Intelligent system of water conservancy based on internet of things[J].Computer Systems&Applications,2012,21(6):161-163.(in Chinese))
[10]冯钧,许潇,唐志贤等.水利大数据及其资源化关键技术研究[J].水利信息化,2013(8):6-9.(FENG Jun,XUXiao,TANG Zhixian,et al.Research on key technology of water big data and resource utilization[J].Water Resources Informatization,2013(8):6-9.(in Chinese))
[11]于嘉骥,张慧妍,王小艺,等.基于改进的投影寻踪-云模型的农业灌溉水质综合评价[J].水资源保护,2017,33(6):142-146.(YU Jiaji,ZHANG Huiyan,WANGXiaoyi,et al.Comprehensive evaluation of agricultural irrigation water quality based on modified projection pursuit-cloud model[J].Water Resources Protection,2017,33(6):142-146.(in Chinese))
[12]HELSEL D R,HIRSCH R M.Statistical methods in water resources[EB/OL].[2018-05-20].http://water.usgs.gov/pubs/twri/twri4a3/.
[13]黄黎明,张可,龚寻,等.大数据视角下水利工程质量风险管理[J].水利经济,2017,35(6):66-70.(HUANGLiming,ZHANG Ke,GONG Xun,et al.Management of quality risk of water conservancy projects from perspective of big data[J].Journal of Economics of Water Resources,2017,35(6):66-70.(in Chinese))
[14]FRIEDMAN E,TZOUMAS K.Introduction to apache flink:stream processing for real time and beyond[M].Sebastopol:O'Reilly Media,2016.
[15]DESHPANDE T.Learning apache flink[M].Birmingham:Packt Publishing,2017.
[16]CARBONE P,GBOR E.HERMANN G,et al.Largescale data stream processing systems[C]//Handbook of Big Data Technologies.Berlin:Springer,2017.
[17]LOPEZ M A,LOCATOR A G P,CARLOS O M B.Aperformance comparison of open-source stream processing platforms[C]//Proceedings of 2016 IEEEGlobal Communications Conference.New York:IEEEComputer Society,2016.
[18]CHINTAPALLI S,DAGIT D,EVANS B,et al.Benchmarking streaming computation engines:storm,flink and spark streaming[C]//Proceedings of IEEE 28th International Parallel and Distributed Processing Symposium Workshops.New York:IEEE Computer Society,2016.
[19]SPANGENBERG N,ROTH M,FRANCZYK B.Evaluating new approaches of big data analytics frameworks[C]//Proceedings of 18th International Conference on Business Information Systems.Berlin:Springer Verlag,2015.
[20]WANG Lizhe,RAJIV R,CHEN Jinjun,et a1.Cloud computing:methodology,systems and applications[M].Boca Raton:CRC Press,2012.
[21]TIWARI S.Professional No SQL[M].Indianapolis:John Wiley&Son,2011.
[22]陆嘉恒.大数据挑战与No SQL数据库技术[M].北京:电子工业出版社,2013.