基于Spark和小波分析的水上交通异常数据实时检测方法研究
|
摘要
针对不断增加的水路运输产生的水上交通数据量增长,导致的水上交通监测难度更大、处理时间更长,提出一种基于Spark的交通异常数据实时检测方法,通过对船舶自动识别系统(AIS)数据进行处理,对不同类型的交通数据进行分析并写入分布式文件系统HDFS中。然后通过小波分析的方法对AIS数据进行多层分解,去除高频噪声并对数据进行重构,找出AIS数据中的异常信息。通过对异常信息进行分析,结合Spark的数据处理结果,最终实时检测出交通异常数据。实验结果表明能够在短时间内对异常数据进行检测和分析,处理速度快,异常数据检测结果符合该水域的交通情况,检测方法能够为海事部门提供实时、稳定的监管服务。
With the increasing amount of waterway transportation,the amount of data generated by water traffic is also increasing,which makes the monitoring of water traffic more difficult and more time-consuming. In this paper,a real-time detection method of traffic abnormal data based on Spark is proposed. By analyzing the AIS data,different types of traffic data are analyzed and written into the distributed file system HDFS. Then,the AIS data is decomposed by wavelet analysis,the high frequency noise is removed and the data is reconstructed to find the abnormal information in AIS data. Through the analysis of abnormal information,combined with Spark data processing results,the final real-time traffic abnormal data is detected. The experimental results show that the abnormal traffic data can be detected and analyzed in a short time,the processing speed is fast and the abnormal data test result is consistent with the traffic situation of the waters. The detection method can provide the maritime department with real-time and stable regulatory services.
With the increasing amount of waterway transportation,the amount of data generated by water traffic is also increasing,which makes the monitoring of water traffic more difficult and more time-consuming. In this paper,a real-time detection method of traffic abnormal data based on Spark is proposed. By analyzing the AIS data,different types of traffic data are analyzed and written into the distributed file system HDFS. Then,the AIS data is decomposed by wavelet analysis,the high frequency noise is removed and the data is reconstructed to find the abnormal information in AIS data. Through the analysis of abnormal information,combined with Spark data processing results,the final real-time traffic abnormal data is detected. The experimental results show that the abnormal traffic data can be detected and analyzed in a short time,the processing speed is fast and the abnormal data test result is consistent with the traffic situation of the waters. The detection method can provide the maritime department with real-time and stable regulatory services.
引文
[1] TUROCHY R E. Enhancing Short-Term Traffic Forecasting with Traffic Condition Information[J]. Journal of TransportationEngineering,2006,132(6):469-474.
[2]陈德旺,郑长青,章长彪.快速路交通流异常数据判断算法研究及实证[J].中国安全科学学报,2006,16(7):122-127.
[3]李成兵,姚琛.交通流异常数据检测研究及实证[J].计算机工程与应用,2013,49(20):244-246.
[4] CHEN Shu-yan,WANG Wei,ZUYLEN H V. Construct support vector machine ensemble to detect traffic incident[J]. Ex-pert Systems with Applications,2009,36(8):10976-10986.
[5] ZAHARIA M,CHOWDHURY M,FRANKLIN M J,et al. Spark:cluster computing with working sets[C]//Usenix Confer-ence on Hot Topics in Cloud Computing,2010.
[6] XIN R S,GONZALEZ J E,FRANKLIN M J,et al. Graph X:a resilient distributed graph system on Spark[C]//InternationalWorkshop on Graph Data Management Experiences and Systems,2013.
[7]马秀红,曹继平,董晟飞.小波分析及其应用[J].计算机技术与发展,2003,13(8):93-94.
[8] MATERASSI M,MITCHELL C N. Wavelet analysis of GPS amplitude scintillation:A case study[J]. Radio Science,2016,42(1):1-10.
[9]刘时华,张亚.基于小波分析对信号噪声的处理及应用[J].机械工程与自动化,2015(1):84-85.
[10] LIU Xu-hui,HAN Ji-zhong,ZHONG Yun-qin,et al. Implementing Web GIS on Hadoop:A case study of improving smallfile I/O performance on HDFS[C]//IEEE International Conference on CLUSTER Computing and Workshops,2009:1-8.
[11] XIA M,SAXENA M,BLAUM M,et al. A tale of two erasure codes in HDFS[C]//Usenix Conference on File and StorageTechnologies,2015:213-226.
[2]陈德旺,郑长青,章长彪.快速路交通流异常数据判断算法研究及实证[J].中国安全科学学报,2006,16(7):122-127.
[3]李成兵,姚琛.交通流异常数据检测研究及实证[J].计算机工程与应用,2013,49(20):244-246.
[4] CHEN Shu-yan,WANG Wei,ZUYLEN H V. Construct support vector machine ensemble to detect traffic incident[J]. Ex-pert Systems with Applications,2009,36(8):10976-10986.
[5] ZAHARIA M,CHOWDHURY M,FRANKLIN M J,et al. Spark:cluster computing with working sets[C]//Usenix Confer-ence on Hot Topics in Cloud Computing,2010.
[6] XIN R S,GONZALEZ J E,FRANKLIN M J,et al. Graph X:a resilient distributed graph system on Spark[C]//InternationalWorkshop on Graph Data Management Experiences and Systems,2013.
[7]马秀红,曹继平,董晟飞.小波分析及其应用[J].计算机技术与发展,2003,13(8):93-94.
[8] MATERASSI M,MITCHELL C N. Wavelet analysis of GPS amplitude scintillation:A case study[J]. Radio Science,2016,42(1):1-10.
[9]刘时华,张亚.基于小波分析对信号噪声的处理及应用[J].机械工程与自动化,2015(1):84-85.
[10] LIU Xu-hui,HAN Ji-zhong,ZHONG Yun-qin,et al. Implementing Web GIS on Hadoop:A case study of improving smallfile I/O performance on HDFS[C]//IEEE International Conference on CLUSTER Computing and Workshops,2009:1-8.
[11] XIA M,SAXENA M,BLAUM M,et al. A tale of two erasure codes in HDFS[C]//Usenix Conference on File and StorageTechnologies,2015:213-226.