环境评估大数据管理平台初探及技术综述
|
摘要
随着软硬件成本的下降和相关技术的日趋成熟,传统的数据分析和管理方式已不能适应业务发展需求.由于环境保护监测和评估数据符合大数据的特征,因此利用大数据技术构建新的环境评估管理平台已成为可能.本文在通过对环境评估大数据管理平台三层架构以及物理实现方案的阐述,结合弹性扩展、流处理、数据湖、大规模并行处理、机器学习等技术的研究,尝试解决环境评估数据的高速、深入、智能处理问题,从而为业务管理提供更高效率的数据支撑.
As the cost decrease of hardware and software with proven technologies, the traditional way to analyze and manage data has obvious shortages to meet the needs of business development. Because the monitoring and evaluation data of environmental protection matches the characteristics of big data, it is possible to build a new data management platform using big data technology. In this study, we design the three-layer architecture and physical implementation proposal of big data management platform for environmental evaluation, with the research of elastic expansion, stream processing, data lake, massive parallel processing, machine learning, and other technologies. We try to resolve the data processing issues of high speed, deep insight, and intelligence so that to provide high efficient support to business management.
As the cost decrease of hardware and software with proven technologies, the traditional way to analyze and manage data has obvious shortages to meet the needs of business development. Because the monitoring and evaluation data of environmental protection matches the characteristics of big data, it is possible to build a new data management platform using big data technology. In this study, we design the three-layer architecture and physical implementation proposal of big data management platform for environmental evaluation, with the research of elastic expansion, stream processing, data lake, massive parallel processing, machine learning, and other technologies. We try to resolve the data processing issues of high speed, deep insight, and intelligence so that to provide high efficient support to business management.
引文
1环境保护部办公厅.关于印发《生态环境大数据建设总体方案》的通知.http://www.cac.gov.cn/2016-03/18/c_1118376330.htm.[2016-03-08]
2 Yang CW,Huang QY,Li ZL,et al.Big data and cloud computing:Innovation opportunities and challenges.International Journal of Digital Earth,2017,10(1):13-53.[doi:10.1080/17538947.2016.1239771]
3陈付梅,韩德志,毕坤,等.大数据环境下的分布式数据流处理关键技术探析.计算机应用,2017,37(3):620-627.
4 Wikimedia Foundation,Inc.Data lake.https://en.wikipedia.org/wiki/Data_lake.[2018-07-16]
5林荣智.并行数据库技术分析与展望.信息通信,2016,(12):200-201.[doi:10.3969/j.issn.1673-1131.2016.12.095]
6林志,茆云霞.深度学习技术在环保督查工作中的应用研究.信息通信,2017,(11):80-82.[doi:10.3969/j.issn.1673-1131.2017.11.035]
7林伟声.深度学习技术在信息系统数据分析中的应用.电脑与电信,2017,(6):51-53.
8 Sivarajah U,Kamal MM,Irani Z,et al.Critical analysis of big data challenges and analytical methods.Journal of Business Research,2017,70:263-286.[doi:10.1016/j.jbusres.2016.08.001]
2 Yang CW,Huang QY,Li ZL,et al.Big data and cloud computing:Innovation opportunities and challenges.International Journal of Digital Earth,2017,10(1):13-53.[doi:10.1080/17538947.2016.1239771]
3陈付梅,韩德志,毕坤,等.大数据环境下的分布式数据流处理关键技术探析.计算机应用,2017,37(3):620-627.
4 Wikimedia Foundation,Inc.Data lake.https://en.wikipedia.org/wiki/Data_lake.[2018-07-16]
5林荣智.并行数据库技术分析与展望.信息通信,2016,(12):200-201.[doi:10.3969/j.issn.1673-1131.2016.12.095]
6林志,茆云霞.深度学习技术在环保督查工作中的应用研究.信息通信,2017,(11):80-82.[doi:10.3969/j.issn.1673-1131.2017.11.035]
7林伟声.深度学习技术在信息系统数据分析中的应用.电脑与电信,2017,(6):51-53.
8 Sivarajah U,Kamal MM,Irani Z,et al.Critical analysis of big data challenges and analytical methods.Journal of Business Research,2017,70:263-286.[doi:10.1016/j.jbusres.2016.08.001]