基于增强Tiny YOLOV3算法的车辆实时检测与跟踪
|
摘要
针对深度学习方法在视觉车辆检测过程中对小目标车辆漏检率高和难以实现嵌入式实时检测的问题,该文基于Tiny YOLOV3算法提出了增强Tiny YOLOV3模型,并通过匈牙利匹配和卡尔曼滤波算法实现目标车辆的跟踪。在车载Jetson TX2嵌入式平台上,分别在白天和夜间驾驶环境下进行了对比试验。试验结果表明:与Tiny YOLOV3模型相比,增强Tiny YOLOV3模型的车辆检测平均准确率提高4.6%,平均误检率减少0.5%,平均漏检率降低7.4%,算法平均耗时增加43.8 ms/帧;加入跟踪算法后,本文算法模型的车辆检测平均准确率提高10.6%,平均误检率减少1.2%,平均漏检率降低23.6%,平均运算速度提高5倍左右,可达30帧/s。结果表明,所提出的算法能够实时准确检测出目标车辆,为卷积神经网络模型的嵌入式工程应用提供了参考。
For intelligent vehicles and advanced driving assistant systems, real-time and accurate vehicle objects detection and tracking through on-board visual sensors are conducive to discovering potential dangers, and can take timely warning to drivers or measures to control vehicle braking and steering systems to avoid traffic accidents by active safety system. In recent years, vehicle detection based on deep learning has become a research hotspot. Although the deep learning method has made a significant breakthrough in vehicle detection precision, it will lead to high missed detection rate of small vehicle targets and rely on expensive computing resources in visual vehicle detection tasks, which is difficult to achieve in embedded real-time applications. Further analysis shows that the main reason for the above problems is that deep convolution neural network cannot reasonably prune network layer parameters, especially cannot reasonably utilize the shallow semantic information. On the contrary, a series of operations at the lower sampling layers will lead to the loss of vehicle information, especially for the small vehicle objects. Therefore, how to effectively extract and utilize the semantic information of small vehicle objects is a problem to be solved in this paper. On this basis, the problem of pruning network layer parameters reasonably was discussed.For the detection algorithm, on the one hand, based on visual analysis of receptive field of Tiny YOLOV3 network shallow layers, the use of shallow semantic information was enhanced by constructing a shallow feature pyramid structure, on the other hand, the shallow down sampling layer was replaced by convolution layer to reduce the semantic information loss of shallow network layers and increase the shallow layer features of vehicle objects to be extracted. Combine the above 2 aspects, the enhanced Tiny YOLOV3 network was proposed. For the tracking algorithm, because of the high frame rate of the vehicle-mounted camera, assuming that the vehicle objects in the adjacent frame moving uniformly without considering the image information, Kalman filter algorithm was used to track the vehicle target, and its observation position was estimated optimally. The proposed enhanced Tiny YOLO V3 network was trained by using 50 000 images collected by the on-board vehicle camera during the day and night. The training strategy included pre-training model, multi-scale training, batch normalization and data augment methods, which same as Tiny YOLOV3 network. On the vehicle Jetson TX2 embedded platform, 8 groups of comparative experiments were carried out with Tiny YOLOV3 model, including day and night traffic scenes. The experimental results showed that compared with Tiny YOLO V3 model,the mean precision rate of the enhanced Tiny YOLOV3 model proposed in this paper was improved by 4.6%, the mean detection error rate was reduced by 0.5%, the mean missing detection rate was reduced by 7.4%, and the mean time consumption was increased by 43.8 ms/frame without tracking algorithm. After adding the vehicle tracking algorithm, the mean precision rate was improved by 10.6%, the mean detection error rate was reduced by 1.2%, the mean missing detection rate was reduced by 23.6%, and the mean operation speed was 5 times faster than that of the Tiny YOLOV3 model, reaching 30 frame/s. The study provides an important guidance for embedded vehicle detection and tracking algorithm application in intelligent vehicles and advanced driving assistant systems.
For intelligent vehicles and advanced driving assistant systems, real-time and accurate vehicle objects detection and tracking through on-board visual sensors are conducive to discovering potential dangers, and can take timely warning to drivers or measures to control vehicle braking and steering systems to avoid traffic accidents by active safety system. In recent years, vehicle detection based on deep learning has become a research hotspot. Although the deep learning method has made a significant breakthrough in vehicle detection precision, it will lead to high missed detection rate of small vehicle targets and rely on expensive computing resources in visual vehicle detection tasks, which is difficult to achieve in embedded real-time applications. Further analysis shows that the main reason for the above problems is that deep convolution neural network cannot reasonably prune network layer parameters, especially cannot reasonably utilize the shallow semantic information. On the contrary, a series of operations at the lower sampling layers will lead to the loss of vehicle information, especially for the small vehicle objects. Therefore, how to effectively extract and utilize the semantic information of small vehicle objects is a problem to be solved in this paper. On this basis, the problem of pruning network layer parameters reasonably was discussed.For the detection algorithm, on the one hand, based on visual analysis of receptive field of Tiny YOLOV3 network shallow layers, the use of shallow semantic information was enhanced by constructing a shallow feature pyramid structure, on the other hand, the shallow down sampling layer was replaced by convolution layer to reduce the semantic information loss of shallow network layers and increase the shallow layer features of vehicle objects to be extracted. Combine the above 2 aspects, the enhanced Tiny YOLOV3 network was proposed. For the tracking algorithm, because of the high frame rate of the vehicle-mounted camera, assuming that the vehicle objects in the adjacent frame moving uniformly without considering the image information, Kalman filter algorithm was used to track the vehicle target, and its observation position was estimated optimally. The proposed enhanced Tiny YOLO V3 network was trained by using 50 000 images collected by the on-board vehicle camera during the day and night. The training strategy included pre-training model, multi-scale training, batch normalization and data augment methods, which same as Tiny YOLOV3 network. On the vehicle Jetson TX2 embedded platform, 8 groups of comparative experiments were carried out with Tiny YOLOV3 model, including day and night traffic scenes. The experimental results showed that compared with Tiny YOLO V3 model,the mean precision rate of the enhanced Tiny YOLOV3 model proposed in this paper was improved by 4.6%, the mean detection error rate was reduced by 0.5%, the mean missing detection rate was reduced by 7.4%, and the mean time consumption was increased by 43.8 ms/frame without tracking algorithm. After adding the vehicle tracking algorithm, the mean precision rate was improved by 10.6%, the mean detection error rate was reduced by 1.2%, the mean missing detection rate was reduced by 23.6%, and the mean operation speed was 5 times faster than that of the Tiny YOLOV3 model, reaching 30 frame/s. The study provides an important guidance for embedded vehicle detection and tracking algorithm application in intelligent vehicles and advanced driving assistant systems.
引文
[1]Mukhtar Amir,Xia Likun,Tang Tongboon.Vehicle detection techniques for collision avoidance systems:A review[J].IEEETransactions on Intelligent Transportation Systems,2015,16(5):2318-2338.
[2]Song Wenjie,Yang Yi,Fu Mengyin,et al.Real-time obstacles detection and status classification for collision warning in a vehicle active safety system[J].IEEE Transactions on Intelligent Transportation Systems,2018,19(3):758-773.
[3]刘庆华,邱修林,谢礼猛,等.基于行驶车速的车辆防撞时间预警算法[J].农业工程学报,2017,33(12):99-106.Liu Qinghua,Qiu Xiulin,Xie Limeng,et al.Anti-collision warning time algorithm based on driving speed of vehicle[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2017,33(12):99-106.(in Chinese with English abstract)
[4]刘军,后士浩,张凯,等.基于单目视觉车辆姿态角估计和逆透视变换的车距测量[J].农业工程学报,2018,34(13):70-76.Liu Jun,Hou Shihao,Zhang Kai,et al.Vehicle distance measurement with implementation of vehicle attitude angle estimation and inverse perspective mapping based on monocular vision[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2018,34(13):70-76.(in Chinese with English abstract)
[5]He Kaiming,Zhang Xiangyu,Ren Shaoqing,et al.Spatial pyramid pooling in deep convolutional networks for visual recognition[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2014,37(9):1904-1916.
[6]He Kaiming,Gkioxari Georgia,Dollar Piotr,et al.Mask R-CNN[C]//IEEE International Conference on Computer Vision.IEEE,2017:2961-2969.
[7]Manana Mduduzi,Tu Chunling,Owolawi Piusadewale.Asurvey on vehicle detection based on convolution neural networks[C]//IEEE International Conference on Computer&Communications.IEEE,2017:1751-1755.
[8]Henriques Joaof,Caseiro Rui,Martins Pedro,et al.High-speed tracking with kernelized correlation filters[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,37(3):583-596.
[9]Tang Ming,Feng Jiayi.Multi-kernel correlation filter for visual tracking[C]//IEEE International Conference on Computer Vision.IEEE,2016:3038-3046.
[10]Girshick Ross,Donahue Jeff,Darrell Trevor,et al.Region-based convolutional networks for accurate object detection and segmentation[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2015,38(1):142-158.
[11]Girshick Ross.Fast R-CNN[C]//IEEE International Conference on Computer Vision.IEEE,2015:1440-1448.
[12]Ren Shaoqing,He Kaiming,Girshick Ross,et al.Faster R-CNN:towards real-time object detection with region proposal networks[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2017,39(6):1137-1149.
[13]Yang Biao,Zhang Yuyu,Cao Jinmeng,et al.On road vehicle detection using an improved Faster RCNN framework with small-size region up-scaling strategy[C]//Pacific-Rim Symposium on Image and Video Technology.Springer,Cham,2017:241-253.
[14]宋焕生,张向清,郑宝峰,等.基于深度学习方法的复杂场景下车辆目标检测[J].计算机应用研究,2018,35(4):1270-1273.Song Huansheng,Zhang Xiangqing,Zheng Baofeng,et al.Vehicle detection based on deep learning in complex scene[J].Application Research of Computers,2018,35(4):1270-1273.(in Chinese with English abstract)
[15]李琳辉,伦智梅,连静,等.基于卷积神经网络的道路车辆检测方法[J].吉林大学学报:工学版,2017,47(2):384-391.Li Linhui,Lun Zhimei,Lian Jing,et al.Convolution neural network-based vehicle detection method[J].Journal of Jilin University(Engineering and Technology Edition),2017,47(2):384-391.(in Chinese with English abstract)
[16]Lee Wonjae,Dong Sungpae,Dong Wonkim,et al.A vehicle detection using selective multi-stage features in convolutional neural networks[C]//International Conference on Control.IEEE,2017:1-3.
[17]刘军,高雪婷,王利明,等.基于OpenCV的前方车辆检测和前撞预警算法研究[J].汽车技术,2017(6):11-16.Liu Jun,Gao Xueting,Wang Liming,et al.Research on preceding vehicle detection and collision warning method based on OpenCV[J].Automobile Technology,2017(6):11-16.(in Chinese with English abstract)
[18]Danelljan Martin,H?ger Gustav,Khan Fahad,et al.Accurate scale estimation for robust visual tracking[C]//2014 British Machine Vision Conference.BMVA Press,2014:1-11.
[19]Danelljan Martin,H?ger Gustav,Khan Fahad,et al.Discriminative scale space tracking[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2017,39(8):1561-1575.
[20]纪筱鹏,魏志强.基于轮廓特征及扩展Kalman滤波的车辆跟踪方法研究[J].中国图象图形学报,2011,16(2):267-272.Ji Xiaopeng,Wei Zhiqiang.Tracking method based on contour feature of vehicles and extended Kalman filter[J].Journal of Image and Graphics,2011,16(2):267-272.(in Chinese with English abstract)
[21]Lecun Yann,Bengio Yoshua,Hinton Geoffrey.Deep learning[J].Nature,2015,521(7553):436-444.
[22]Liu Wei,Anguelov Dragomir,Erhan Dumitru,et al.SSD:single shot multiBox detector[C]//European Conference on Computer Vision.Springer,Cham,2016:21-37.
[23]Smeulders A,Chu D,Cucchiara R,et al.Visual tracking:an experimental survey[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2014,36(7):1442-1468.
[24]Joseph Redmon,Divvala Santosh,Girshick Ross,et al.You only look once:Unified,real-Time object detection[C]//IEEEConference on Computer Vision and Pattern Recognition.IEEE,2015:779-788.
[25]Joseph Redmon,Farhadi Ali.YOLO9000:Better,faster,stronger[C]//IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2017:6517-6525.
[26]Joseph Redmon,Farhadi Ali.YOLOv3:An Incremental Improvement[EB/OL].[2018-04-08].https://arxiv.org/pdf/1804.02767.pdf.
[27]Lin Tsungyi,Dollár Piotr,Girshick Ross,et al.Feature pyramid networks for object detection[C]//IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2017:936-944.
[28]Long Jonathan,Shelhamer Evan,Darrell Trevor.Fully convolutional networks for semantic segmentation[J].IEEETransactions on Pattern Analysis&Machine Intelligence,2014,39(4):640-651.
[29]Yosinski Jason,Clune Jeff,Nguyen Anh,et al.Understanding neural networks through deep visualization[EB/OL].[2015-06-22].https://arxiv.org/abs/1506.06579.
[30]Bochinski Erik,Eiselein Volker,Sikora Thomas.High-Speed tracking-by-detection without using image information[C]//IEEE International Conference on Advanced Video&Signal Based Surveillance.IEEE,2017:1-6.
[2]Song Wenjie,Yang Yi,Fu Mengyin,et al.Real-time obstacles detection and status classification for collision warning in a vehicle active safety system[J].IEEE Transactions on Intelligent Transportation Systems,2018,19(3):758-773.
[3]刘庆华,邱修林,谢礼猛,等.基于行驶车速的车辆防撞时间预警算法[J].农业工程学报,2017,33(12):99-106.Liu Qinghua,Qiu Xiulin,Xie Limeng,et al.Anti-collision warning time algorithm based on driving speed of vehicle[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2017,33(12):99-106.(in Chinese with English abstract)
[4]刘军,后士浩,张凯,等.基于单目视觉车辆姿态角估计和逆透视变换的车距测量[J].农业工程学报,2018,34(13):70-76.Liu Jun,Hou Shihao,Zhang Kai,et al.Vehicle distance measurement with implementation of vehicle attitude angle estimation and inverse perspective mapping based on monocular vision[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2018,34(13):70-76.(in Chinese with English abstract)
[5]He Kaiming,Zhang Xiangyu,Ren Shaoqing,et al.Spatial pyramid pooling in deep convolutional networks for visual recognition[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2014,37(9):1904-1916.
[6]He Kaiming,Gkioxari Georgia,Dollar Piotr,et al.Mask R-CNN[C]//IEEE International Conference on Computer Vision.IEEE,2017:2961-2969.
[7]Manana Mduduzi,Tu Chunling,Owolawi Piusadewale.Asurvey on vehicle detection based on convolution neural networks[C]//IEEE International Conference on Computer&Communications.IEEE,2017:1751-1755.
[8]Henriques Joaof,Caseiro Rui,Martins Pedro,et al.High-speed tracking with kernelized correlation filters[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,37(3):583-596.
[9]Tang Ming,Feng Jiayi.Multi-kernel correlation filter for visual tracking[C]//IEEE International Conference on Computer Vision.IEEE,2016:3038-3046.
[10]Girshick Ross,Donahue Jeff,Darrell Trevor,et al.Region-based convolutional networks for accurate object detection and segmentation[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2015,38(1):142-158.
[11]Girshick Ross.Fast R-CNN[C]//IEEE International Conference on Computer Vision.IEEE,2015:1440-1448.
[12]Ren Shaoqing,He Kaiming,Girshick Ross,et al.Faster R-CNN:towards real-time object detection with region proposal networks[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2017,39(6):1137-1149.
[13]Yang Biao,Zhang Yuyu,Cao Jinmeng,et al.On road vehicle detection using an improved Faster RCNN framework with small-size region up-scaling strategy[C]//Pacific-Rim Symposium on Image and Video Technology.Springer,Cham,2017:241-253.
[14]宋焕生,张向清,郑宝峰,等.基于深度学习方法的复杂场景下车辆目标检测[J].计算机应用研究,2018,35(4):1270-1273.Song Huansheng,Zhang Xiangqing,Zheng Baofeng,et al.Vehicle detection based on deep learning in complex scene[J].Application Research of Computers,2018,35(4):1270-1273.(in Chinese with English abstract)
[15]李琳辉,伦智梅,连静,等.基于卷积神经网络的道路车辆检测方法[J].吉林大学学报:工学版,2017,47(2):384-391.Li Linhui,Lun Zhimei,Lian Jing,et al.Convolution neural network-based vehicle detection method[J].Journal of Jilin University(Engineering and Technology Edition),2017,47(2):384-391.(in Chinese with English abstract)
[16]Lee Wonjae,Dong Sungpae,Dong Wonkim,et al.A vehicle detection using selective multi-stage features in convolutional neural networks[C]//International Conference on Control.IEEE,2017:1-3.
[17]刘军,高雪婷,王利明,等.基于OpenCV的前方车辆检测和前撞预警算法研究[J].汽车技术,2017(6):11-16.Liu Jun,Gao Xueting,Wang Liming,et al.Research on preceding vehicle detection and collision warning method based on OpenCV[J].Automobile Technology,2017(6):11-16.(in Chinese with English abstract)
[18]Danelljan Martin,H?ger Gustav,Khan Fahad,et al.Accurate scale estimation for robust visual tracking[C]//2014 British Machine Vision Conference.BMVA Press,2014:1-11.
[19]Danelljan Martin,H?ger Gustav,Khan Fahad,et al.Discriminative scale space tracking[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2017,39(8):1561-1575.
[20]纪筱鹏,魏志强.基于轮廓特征及扩展Kalman滤波的车辆跟踪方法研究[J].中国图象图形学报,2011,16(2):267-272.Ji Xiaopeng,Wei Zhiqiang.Tracking method based on contour feature of vehicles and extended Kalman filter[J].Journal of Image and Graphics,2011,16(2):267-272.(in Chinese with English abstract)
[21]Lecun Yann,Bengio Yoshua,Hinton Geoffrey.Deep learning[J].Nature,2015,521(7553):436-444.
[22]Liu Wei,Anguelov Dragomir,Erhan Dumitru,et al.SSD:single shot multiBox detector[C]//European Conference on Computer Vision.Springer,Cham,2016:21-37.
[23]Smeulders A,Chu D,Cucchiara R,et al.Visual tracking:an experimental survey[J].IEEE Transactions on Pattern Analysis&Machine Intelligence,2014,36(7):1442-1468.
[24]Joseph Redmon,Divvala Santosh,Girshick Ross,et al.You only look once:Unified,real-Time object detection[C]//IEEEConference on Computer Vision and Pattern Recognition.IEEE,2015:779-788.
[25]Joseph Redmon,Farhadi Ali.YOLO9000:Better,faster,stronger[C]//IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2017:6517-6525.
[26]Joseph Redmon,Farhadi Ali.YOLOv3:An Incremental Improvement[EB/OL].[2018-04-08].https://arxiv.org/pdf/1804.02767.pdf.
[27]Lin Tsungyi,Dollár Piotr,Girshick Ross,et al.Feature pyramid networks for object detection[C]//IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2017:936-944.
[28]Long Jonathan,Shelhamer Evan,Darrell Trevor.Fully convolutional networks for semantic segmentation[J].IEEETransactions on Pattern Analysis&Machine Intelligence,2014,39(4):640-651.
[29]Yosinski Jason,Clune Jeff,Nguyen Anh,et al.Understanding neural networks through deep visualization[EB/OL].[2015-06-22].https://arxiv.org/abs/1506.06579.
[30]Bochinski Erik,Eiselein Volker,Sikora Thomas.High-Speed tracking-by-detection without using image information[C]//IEEE International Conference on Advanced Video&Signal Based Surveillance.IEEE,2017:1-6.