基于NoSQL数据库的农田物联网云存储系统设计与实现
|
摘要
为了解决农田物联网大量图像、视频和传感器等结构化和非结构化数据实时处理与写入问题,该文基于分布式存储与NoSQL(NotOnlySQL)技术,结合农田物联网数据特征,利用HDFS(HadoopDistributedFileSystem)和HBase(Hadoop Database)存储非结构化和结构化数据,基于Redis缓存服务,设计了三层物联网数据云存储框架,实现了海量农田物联网数据存储中的业务处理、事务处理、图片打包与索引、负载均衡等关键技术。面对复杂业务下的事务数据一致性,该文采用基于HLock的乐观锁机制,实现了HBase对强事务性的支持,经过与传统MySQL集群事务对比测试,当数据量级在500万时,数据读取效率提升达35.75%。为了提高农田物联网中大量的小图片和小文件处理效率,基于图片打包合并策略,利用SequenceFile技术实现物联图片的快速索引读写技术,与原生HDFS存储效率相比,读写效率提升30%以上。该研究可以为海量农业物联网数据的存储和管理提供技术参考和理论支撑。
In order to solve the problems of storing large amounts of structured and unstructured data, such as images, video and sensors and so on, and real-time processing and writing, a data cloud storage system for farmland Internet of Things(IOT) with mass storage, high performance and easy expansion is constructed. Based on Hadoop platform, in this paper, we constructed a massive farmland IOT data cloud storage system by combining the characteristics of farmland IOT data, using distributed storage and NoSQL(Not Only SQL) technology. From the security, reliability, efficient reading and writing, data conversion, transaction processing, small file processing, cache strategy, load balancing and other issues of the system, HDFS was used to store unstructured data such as pictures and videos in the farmland IOT system, HBase was used to store structured data such as meteorology and moisture in the farmland IOT system, Redis was used in cache servers. Three layers of data cloud storage architecture for IOT were designed. The system classifies and processes video, image, text and structured data. For large video block storage, small file image packaging and merging storage, text classification and conversion strategy, unstructured data were written to HDFS, structured data were written to HBase, and Redis was used as the system cache to realize the data of the IOT writing and reading business. In distributed cluster environment, the reliability of cross-line transaction and long transaction processing was restricted. It was difficult to process cross-line transaction and long transaction accurately and orderly, and it was difficult to ensure data consistency in complex services such as massive data analysis. In this paper, a distributed transaction mechanism based on optimistic lock was designed. The transaction processing module cooperates with the HLock(optimistic lock) structure to control the state of the transaction. The NTP server guarantees the uniqueness of the transaction timestamp. The transaction ACID features, including reading and writing data, were solved. HBase's strong transactional support has been tested to improve query efficiency by 35.75% compared with traditional MySQL clusters when the data level was 5 million. Thus, NoSQL-based structured data storage scheme was feasible in dealing with high concurrent massive data scenarios. In order to solve the problem of a large number of small pictures and small files in the farmland IOT, the sampled pictures were packaged and measured. The "SequenceFile" technology was used to merge multiple pictures into a "Block" to realize the strategy of merging and storing small files. Fast index reading, compared with the original HDFS storage reading and writing efficiency, image file storage reading and writing efficiency improved by more than 30%. Therefore, based on the "SequenceFile" file merging technology, image file name design and index optimization strategy, it was suitable for large-scale image storage scene in the farmland IOT. The system had been applied to the monitoring system of farmland IOT in China Henan Province. It was distributed in more than 60 monitoring stations in Changge, Huaxian, Luohe and Fangcheng counties and cities, providing real-time data for storage, management and visualization, and considering the incorporation of more sensors and monitoring stations, the system was in good working order. In summary, based on Hadoop platform and NoSQL technology, we designed a massive farmland IOT data storage model, designed and implements the key technologies such as data reading and writing, transaction, picture packaging, index, load balancing module, and develops a massive farmland IOT data storage, management system. Based on NoSQL massive farmland IOT data storage scheme suitable for the storage and management needs of the IOT massive, real-time data, for farmland IOT storage transaction consistency, small file processing and other issues, for massive agricultural IOT data storage solutions. It can combine distributed computing and machine learning technology to compute the data of IOT in real time and provide real-time operation and decision-making services for agricultural production.
In order to solve the problems of storing large amounts of structured and unstructured data, such as images, video and sensors and so on, and real-time processing and writing, a data cloud storage system for farmland Internet of Things(IOT) with mass storage, high performance and easy expansion is constructed. Based on Hadoop platform, in this paper, we constructed a massive farmland IOT data cloud storage system by combining the characteristics of farmland IOT data, using distributed storage and NoSQL(Not Only SQL) technology. From the security, reliability, efficient reading and writing, data conversion, transaction processing, small file processing, cache strategy, load balancing and other issues of the system, HDFS was used to store unstructured data such as pictures and videos in the farmland IOT system, HBase was used to store structured data such as meteorology and moisture in the farmland IOT system, Redis was used in cache servers. Three layers of data cloud storage architecture for IOT were designed. The system classifies and processes video, image, text and structured data. For large video block storage, small file image packaging and merging storage, text classification and conversion strategy, unstructured data were written to HDFS, structured data were written to HBase, and Redis was used as the system cache to realize the data of the IOT writing and reading business. In distributed cluster environment, the reliability of cross-line transaction and long transaction processing was restricted. It was difficult to process cross-line transaction and long transaction accurately and orderly, and it was difficult to ensure data consistency in complex services such as massive data analysis. In this paper, a distributed transaction mechanism based on optimistic lock was designed. The transaction processing module cooperates with the HLock(optimistic lock) structure to control the state of the transaction. The NTP server guarantees the uniqueness of the transaction timestamp. The transaction ACID features, including reading and writing data, were solved. HBase's strong transactional support has been tested to improve query efficiency by 35.75% compared with traditional MySQL clusters when the data level was 5 million. Thus, NoSQL-based structured data storage scheme was feasible in dealing with high concurrent massive data scenarios. In order to solve the problem of a large number of small pictures and small files in the farmland IOT, the sampled pictures were packaged and measured. The "SequenceFile" technology was used to merge multiple pictures into a "Block" to realize the strategy of merging and storing small files. Fast index reading, compared with the original HDFS storage reading and writing efficiency, image file storage reading and writing efficiency improved by more than 30%. Therefore, based on the "SequenceFile" file merging technology, image file name design and index optimization strategy, it was suitable for large-scale image storage scene in the farmland IOT. The system had been applied to the monitoring system of farmland IOT in China Henan Province. It was distributed in more than 60 monitoring stations in Changge, Huaxian, Luohe and Fangcheng counties and cities, providing real-time data for storage, management and visualization, and considering the incorporation of more sensors and monitoring stations, the system was in good working order. In summary, based on Hadoop platform and NoSQL technology, we designed a massive farmland IOT data storage model, designed and implements the key technologies such as data reading and writing, transaction, picture packaging, index, load balancing module, and develops a massive farmland IOT data storage, management system. Based on NoSQL massive farmland IOT data storage scheme suitable for the storage and management needs of the IOT massive, real-time data, for farmland IOT storage transaction consistency, small file processing and other issues, for massive agricultural IOT data storage solutions. It can combine distributed computing and machine learning technology to compute the data of IOT in real time and provide real-time operation and decision-making services for agricultural production.
引文
[1]李瑾,郭美荣,高亮亮.农业物联网技术应用及创新发展策略[J].农业工程学报,2015,31(增刊2):200-209.Li Jin,Guo Meirong,Gao Liangliang.Application and innovation strategy of agricultural Internet of Things[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2015,31(Supp.2):200-209.(in Chinese with English abstract)
[2]张向飞.基于农业物联网的数据智能传输与大田监测应用[D].上海:东华大学,2016.Zhang Xiangfei.Intelligent Data Transmission And Field Monitoring Application Based on Agricultural Internet of Things[D].Shanghai:Donghua University,2016.(in Chinese with English abstract)
[3]杜克明,褚金翔,孙忠富,等.WebGIS在农业环境物联网监测系统中的设计与实现[J].农业工程学报,2016,32(4):171-178.Du Keming,Chu Jinxiang,Sun Zhongfu,et al.Design and implementation of monitoring system for agricultural environment based on Web GIS with Internet of Things[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2016,32(4):171-178.(in Chinese with English abstract)
[4]陈晓栋,原向阳,郭平毅,等.农业物联网研究进展与前景展望[J].中国农业科技导报,2015,17(2):8-16.Chen Xiaodong,Yuan Xiangyang,Guo Pingyi,et al.Progress and prospect in agricultural internet of things[J].Journal of Agricultural Science and Technology,2015,17(2):8-16.(in Chinese with English abstract)
[5]王萍,赵宏亮,李佩林,等.农业物联网技术在大豆生产中的应用[J].大豆科学,2018,37(5):809-813.Wang Ping,Zhao Hongliang,Li Peilin,et al.Application of agricultural internet of things in soybean production[J].Soybean Science,2018,37(5):809-813.(in Chinese with English abstract)
[6]瞿荣锦,韦琮,赵丽娟.基于物联网技术的农田环境监测系统的研究与构建[J].农业开发与装备,2018(9):104-105.
[7]琚书存,程文杰,徐建鹏,等.农业气象物联网数据采集系统[J].计算机与现代化,2018(9):105-109,117.Ju Shucun,Cheng Wenjie,Xu Jianpeng,et al.Data acquisition system for agricultural meteorological IOT[J].Computer and Modernization,2018(9):105-109,117.(in Chinese with English abstract)
[8]宋健玮.农田灌溉监控系统的设计与实现[D].济南:山东大学,2017.Song Jianwei.Design and Implementation of Farmland Irrigation Monitoring and Control System[D].Jinan:Shandong University,2017.(in Chinese with English abstract)
[9]刘艺蕾.农业物联网监控管理系统设计与实现[D].北京:北京邮电大学,2016.Liu Yilei.The Design and Implementation of Agricultural IOT Monitor and Management system[D].Beijing:Beijing University of Posts and Telecommunications,2016.(in Chinese with English abstract)
[10]薛文龙.基于物联网的农田环境信息采集控制与预警系统[J].江苏农业科学,2017,45(9):195-198.
[11]蔡绍堂,麻硕琪,乐英高,等.一种农田环境远程监测系统设计与实现方法[J].四川理工学院学报:自然科学版,2018,31(2):69-74.Cai Shaotang,Ma Shuoqi,Le Yinggao,et al.Design and realization of remote monitoring system for farmland environment[J].Journal of Sichuan University of Science&Engineering:Natural Science Edition,2018,31(2):69-74.(in Chinese with English abstract)
[12]姜岩,段杰,王茂励,等.基于物联网技术的水肥一体化智能管理系统[J].现代农业科技,2018(16):279-281.Jiang Yan,Duan Jie,Wang Maoli,et al.Intelligent system of water and fertilizer based on internet of things technology[J].Modern Agricultural Science and Technology,2018(16):279-281.(in Chinese with English abstract)
[13]徐识溥,刘勇,李双喜,等.基于农业物联网的农田土壤环境监测系统的研究与设计[J].中国农学通报,2018,34(23):145-150.Xu Shipu,Liu Yong,Li Shuangxi,et al.Research and design of farmland soil environmental monitoring system based on agricultural IOT[J].Chinese Agricultural Science Bulletin,2018,34(23):145-150.(in Chinese with English abstract)
[14]李雅丽,魏峰远,陈荣国,等.基于物联网和WebGIS果园监测系统的设计与实现[J].测绘与空间地理信息,2018,41(8):75-77,81.Li Yali,Wei Fengyuan,Chen Rongguo,et al.Design and application of orchard environment based on internet of things and Web GIS[J].Geomatics&Spatial Information Technology,2018,41(8):75-77,81.(in Chinese with English abstract)
[15]郝行军.物联网大数据存储与管理技术研究[D].合肥:中国科学技术大学,2017.Hao Xingjun.Reseach on Technology of Storage and Management of IOT Data[D].Hefei:University of Science and Technology of China,2017.(in Chinese with English abstract)
[16]陈威,郭书普.中国农业信息化技术发展现状及存在的问题[J].农业工程学报,2013,29(22):196-205.Chen Wei,Guo Shupu.Current situation and existing problems of agricultural informatization in China[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2013,29(22):196-205.(in Chinese with English abstract)
[17]赵立波,李凌霞,王旭.物联网海量异构传感器采样数据存储与查询处理[J].软件导刊,2017,16(12):182-183,187.Zhao Libo,Li Lingxia,Wang Xu.Storage and query processing of massive heterogeneous sensorsample data in internet of things[J].Soft Guide,2017,16(12):182-183,187.(in Chinese with English abstract)
[18]郑建忠,郑建荣.一种基于云计算技术的物联网平台设计[J].电力信息与通信技术,2018,16(6):57-61.Zheng Jianzhong,Zheng Jianrong.Design of internet of things platform based on cloud computing technology[J].Electric Power ICT,2018,16(6):57-61.(in Chinese with English abstract)
[19]刘佩增.面向物联网多维度数据的多级存储系统的设计与实现[D].北京:北京邮电大学,2018.Liu Peizeng.The Design and Implementation of Multilevel Storage System Oriented to Multidimensional Data[D].Beijing:Beijing University of Posts and Telecommunications,2018.(in Chinese with English abstract)
[20]杨鹏,林俊晖.一种基于MongoDB和Hadoop的海量非结构化物联网数据处理方案[J].微电子学与计算机,2018,35(4):68-72,78.Yang Peng,Lin Junhui.A scheme for massive unstructured iot data processingbased on mongodb and hadoop[J].Microelectronics&Computer,2018,35(4):68-72,78.(in Chinese with English abstract)
[21]李继蕊,李小勇,高雅丽,等.物联网环境下数据转发模型研究[J].软件学报,2018,29(1):196-224.Li Jirui,Li Xiaoyong,Gao Yali,et al.Review on data forwarding model in internet of things[J].Journal of Software,2018,29(1):196-224.(in Chinese with English abstract)
[22]王顺.面向农业物联网的异构数据存储方法研究[D].郑州:河南农业大学,2016.Wang Shun.Research on the Heterogeneous Data Storage Method for the Agricultural Internet of Things[D].Zhengzhou:Henan Agricultural University,2016.(in Chinese with English abstract)
[23]龚畅.大数据下的No SQL数据库技术分析[J].信息记录材料,2018,19(6):118-119.
[24]李林.基于hadoop的海量图片存储模型的分析和设计[D].杭州:杭州电子科技大学,2011.Li Lin.Design and Analysis of the Mass Image Storage Model Based on Hadoop[D].Hangzhou:Hangzhou Dianzi University,2011.(in Chinese with English abstract)
[25]卢冬海,何先波.浅析No SQL数据库[J].中国西部科技,2011,10(2):14,15-16.Lu Donghai,He Xianbo.The analysis of NoSQL database[J].Science and Technology of West China,2011,10(2):14,15-16.(in Chinese with English abstract)
[26]宋晓东.Hadoop分布式文件系统小文件数据存储性能的优化方法研究[D].北京:北京交通大学,2017.Song Xiaodong.The Optimization Method Research for Small File Data Storage Performance on Hadoop Distributed File System[D].Beijing:Beijing Jiaotong University,2017.(in Chinese with English abstract)
[27]宋俊辉,冯岩.负载均衡的分布式系统任务调度优化算法[J].吉林大学学报:理学版,2017,55(2):383-387.Song Junhui,Feng Yan.Task Scheduling optimization algorithm in distributed system with load balancing[J].Journal of Jilin University:Science Edition,2017,55(2):383-387.(in Chinese with English abstract)
[28]谭台哲,向云鹏.Hadoop平台下海量图像处理实现[J].计算机工程与设计,2017,38(4):976-980.Tan Taizhe,Xiang Yunpeng.Large-scale image processing implementation under hadoop platform[J].Computer Engineering and Design,2017,38(4):976-980.(in Chinese with English abstract)
[29]Daniel Peng,Frank Dabek.Large-scale Incremental Processing Using Distributed Transactions and Notifications[C]//USENIX Symposium on Operating Systems Design and Implementation,October 4-6,2010,Vancouver,BC:USENIX Association,c2010:251?264.
[30]苗星.HBase长事务实现方法研究[D].北京:北京交通大学,2015.Miao Xing.The Research on Long Transaction Support for HBase[D].Beijing:Beijing Jiaotong University,2015.(in Chinese with English abstract)
[31]吴龙,唐军.一种ntp服务器时间同步的方法:CN102916799A[P].2013-02-06.
[32]李三淼,李龙澍.Hadoop中处理小文件的四种方法的性能分析[J].计算机工程与应用,2016,52(9):44-49.Li Sanmiao,Li Longshu.Performance analysis of four methods for handling small files in Hadoop[J].Computer Engineering and Applications,2016,52(9):44-49.(in Chinese with English abstract)
[2]张向飞.基于农业物联网的数据智能传输与大田监测应用[D].上海:东华大学,2016.Zhang Xiangfei.Intelligent Data Transmission And Field Monitoring Application Based on Agricultural Internet of Things[D].Shanghai:Donghua University,2016.(in Chinese with English abstract)
[3]杜克明,褚金翔,孙忠富,等.WebGIS在农业环境物联网监测系统中的设计与实现[J].农业工程学报,2016,32(4):171-178.Du Keming,Chu Jinxiang,Sun Zhongfu,et al.Design and implementation of monitoring system for agricultural environment based on Web GIS with Internet of Things[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2016,32(4):171-178.(in Chinese with English abstract)
[4]陈晓栋,原向阳,郭平毅,等.农业物联网研究进展与前景展望[J].中国农业科技导报,2015,17(2):8-16.Chen Xiaodong,Yuan Xiangyang,Guo Pingyi,et al.Progress and prospect in agricultural internet of things[J].Journal of Agricultural Science and Technology,2015,17(2):8-16.(in Chinese with English abstract)
[5]王萍,赵宏亮,李佩林,等.农业物联网技术在大豆生产中的应用[J].大豆科学,2018,37(5):809-813.Wang Ping,Zhao Hongliang,Li Peilin,et al.Application of agricultural internet of things in soybean production[J].Soybean Science,2018,37(5):809-813.(in Chinese with English abstract)
[6]瞿荣锦,韦琮,赵丽娟.基于物联网技术的农田环境监测系统的研究与构建[J].农业开发与装备,2018(9):104-105.
[7]琚书存,程文杰,徐建鹏,等.农业气象物联网数据采集系统[J].计算机与现代化,2018(9):105-109,117.Ju Shucun,Cheng Wenjie,Xu Jianpeng,et al.Data acquisition system for agricultural meteorological IOT[J].Computer and Modernization,2018(9):105-109,117.(in Chinese with English abstract)
[8]宋健玮.农田灌溉监控系统的设计与实现[D].济南:山东大学,2017.Song Jianwei.Design and Implementation of Farmland Irrigation Monitoring and Control System[D].Jinan:Shandong University,2017.(in Chinese with English abstract)
[9]刘艺蕾.农业物联网监控管理系统设计与实现[D].北京:北京邮电大学,2016.Liu Yilei.The Design and Implementation of Agricultural IOT Monitor and Management system[D].Beijing:Beijing University of Posts and Telecommunications,2016.(in Chinese with English abstract)
[10]薛文龙.基于物联网的农田环境信息采集控制与预警系统[J].江苏农业科学,2017,45(9):195-198.
[11]蔡绍堂,麻硕琪,乐英高,等.一种农田环境远程监测系统设计与实现方法[J].四川理工学院学报:自然科学版,2018,31(2):69-74.Cai Shaotang,Ma Shuoqi,Le Yinggao,et al.Design and realization of remote monitoring system for farmland environment[J].Journal of Sichuan University of Science&Engineering:Natural Science Edition,2018,31(2):69-74.(in Chinese with English abstract)
[12]姜岩,段杰,王茂励,等.基于物联网技术的水肥一体化智能管理系统[J].现代农业科技,2018(16):279-281.Jiang Yan,Duan Jie,Wang Maoli,et al.Intelligent system of water and fertilizer based on internet of things technology[J].Modern Agricultural Science and Technology,2018(16):279-281.(in Chinese with English abstract)
[13]徐识溥,刘勇,李双喜,等.基于农业物联网的农田土壤环境监测系统的研究与设计[J].中国农学通报,2018,34(23):145-150.Xu Shipu,Liu Yong,Li Shuangxi,et al.Research and design of farmland soil environmental monitoring system based on agricultural IOT[J].Chinese Agricultural Science Bulletin,2018,34(23):145-150.(in Chinese with English abstract)
[14]李雅丽,魏峰远,陈荣国,等.基于物联网和WebGIS果园监测系统的设计与实现[J].测绘与空间地理信息,2018,41(8):75-77,81.Li Yali,Wei Fengyuan,Chen Rongguo,et al.Design and application of orchard environment based on internet of things and Web GIS[J].Geomatics&Spatial Information Technology,2018,41(8):75-77,81.(in Chinese with English abstract)
[15]郝行军.物联网大数据存储与管理技术研究[D].合肥:中国科学技术大学,2017.Hao Xingjun.Reseach on Technology of Storage and Management of IOT Data[D].Hefei:University of Science and Technology of China,2017.(in Chinese with English abstract)
[16]陈威,郭书普.中国农业信息化技术发展现状及存在的问题[J].农业工程学报,2013,29(22):196-205.Chen Wei,Guo Shupu.Current situation and existing problems of agricultural informatization in China[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2013,29(22):196-205.(in Chinese with English abstract)
[17]赵立波,李凌霞,王旭.物联网海量异构传感器采样数据存储与查询处理[J].软件导刊,2017,16(12):182-183,187.Zhao Libo,Li Lingxia,Wang Xu.Storage and query processing of massive heterogeneous sensorsample data in internet of things[J].Soft Guide,2017,16(12):182-183,187.(in Chinese with English abstract)
[18]郑建忠,郑建荣.一种基于云计算技术的物联网平台设计[J].电力信息与通信技术,2018,16(6):57-61.Zheng Jianzhong,Zheng Jianrong.Design of internet of things platform based on cloud computing technology[J].Electric Power ICT,2018,16(6):57-61.(in Chinese with English abstract)
[19]刘佩增.面向物联网多维度数据的多级存储系统的设计与实现[D].北京:北京邮电大学,2018.Liu Peizeng.The Design and Implementation of Multilevel Storage System Oriented to Multidimensional Data[D].Beijing:Beijing University of Posts and Telecommunications,2018.(in Chinese with English abstract)
[20]杨鹏,林俊晖.一种基于MongoDB和Hadoop的海量非结构化物联网数据处理方案[J].微电子学与计算机,2018,35(4):68-72,78.Yang Peng,Lin Junhui.A scheme for massive unstructured iot data processingbased on mongodb and hadoop[J].Microelectronics&Computer,2018,35(4):68-72,78.(in Chinese with English abstract)
[21]李继蕊,李小勇,高雅丽,等.物联网环境下数据转发模型研究[J].软件学报,2018,29(1):196-224.Li Jirui,Li Xiaoyong,Gao Yali,et al.Review on data forwarding model in internet of things[J].Journal of Software,2018,29(1):196-224.(in Chinese with English abstract)
[22]王顺.面向农业物联网的异构数据存储方法研究[D].郑州:河南农业大学,2016.Wang Shun.Research on the Heterogeneous Data Storage Method for the Agricultural Internet of Things[D].Zhengzhou:Henan Agricultural University,2016.(in Chinese with English abstract)
[23]龚畅.大数据下的No SQL数据库技术分析[J].信息记录材料,2018,19(6):118-119.
[24]李林.基于hadoop的海量图片存储模型的分析和设计[D].杭州:杭州电子科技大学,2011.Li Lin.Design and Analysis of the Mass Image Storage Model Based on Hadoop[D].Hangzhou:Hangzhou Dianzi University,2011.(in Chinese with English abstract)
[25]卢冬海,何先波.浅析No SQL数据库[J].中国西部科技,2011,10(2):14,15-16.Lu Donghai,He Xianbo.The analysis of NoSQL database[J].Science and Technology of West China,2011,10(2):14,15-16.(in Chinese with English abstract)
[26]宋晓东.Hadoop分布式文件系统小文件数据存储性能的优化方法研究[D].北京:北京交通大学,2017.Song Xiaodong.The Optimization Method Research for Small File Data Storage Performance on Hadoop Distributed File System[D].Beijing:Beijing Jiaotong University,2017.(in Chinese with English abstract)
[27]宋俊辉,冯岩.负载均衡的分布式系统任务调度优化算法[J].吉林大学学报:理学版,2017,55(2):383-387.Song Junhui,Feng Yan.Task Scheduling optimization algorithm in distributed system with load balancing[J].Journal of Jilin University:Science Edition,2017,55(2):383-387.(in Chinese with English abstract)
[28]谭台哲,向云鹏.Hadoop平台下海量图像处理实现[J].计算机工程与设计,2017,38(4):976-980.Tan Taizhe,Xiang Yunpeng.Large-scale image processing implementation under hadoop platform[J].Computer Engineering and Design,2017,38(4):976-980.(in Chinese with English abstract)
[29]Daniel Peng,Frank Dabek.Large-scale Incremental Processing Using Distributed Transactions and Notifications[C]//USENIX Symposium on Operating Systems Design and Implementation,October 4-6,2010,Vancouver,BC:USENIX Association,c2010:251?264.
[30]苗星.HBase长事务实现方法研究[D].北京:北京交通大学,2015.Miao Xing.The Research on Long Transaction Support for HBase[D].Beijing:Beijing Jiaotong University,2015.(in Chinese with English abstract)
[31]吴龙,唐军.一种ntp服务器时间同步的方法:CN102916799A[P].2013-02-06.
[32]李三淼,李龙澍.Hadoop中处理小文件的四种方法的性能分析[J].计算机工程与应用,2016,52(9):44-49.Li Sanmiao,Li Longshu.Performance analysis of four methods for handling small files in Hadoop[J].Computer Engineering and Applications,2016,52(9):44-49.(in Chinese with English abstract)