虚拟现实全景图像显著性检测研究进展综述
|
摘要
随着虚拟现实处理技术的发展,虚拟现实全景图像的显著性检测成为近年来学术界和工业界关注的研究热点.本文分析虚拟现实全景图像的特性,综述虚拟现实全景图像显著性检测算法的研究进展.将已有的虚拟现实全景图像显著性检测算法进行分类、分析以及对比,本文总结了当前虚拟现实全景图像显著性检测面临的挑战,并对其发展趋势进行展望.
With the development of virtual reality(VR),the saliency detection for panoramic VR image has been a hot research topic in both academic and industry worlds.After analyzing the particular characteristics of panoramic VR image,this paper summarizes the research progress of the saliency detection algorithms of panoramic VR image.Existing VR image saliency detection algorithms are firstly classified,analyzed and compared.Then,the current research challenges and future directions of VR saliency detection are discussed.
With the development of virtual reality(VR),the saliency detection for panoramic VR image has been a hot research topic in both academic and industry worlds.After analyzing the particular characteristics of panoramic VR image,this paper summarizes the research progress of the saliency detection algorithms of panoramic VR image.Existing VR image saliency detection algorithms are firstly classified,analyzed and compared.Then,the current research challenges and future directions of VR saliency detection are discussed.
引文
[1] 解凯,郭恒业,张田文.图像Mosaics技术综述[J].电子学报,2004,32(4):630-634.XIE Kai,GUO Heng-ye,ZHANG Tian-wen.A survey of image mosaics technology[J].Acta Electronica Sinica,2004,32(4):630-634.(in Chinese)
[2] LUZ G,ASCENSO J,BRITES C,et al.Saliency-driven omni-directional imaging adaptive coding:modeling and assessment[A].Proceedings of the International Workshop on Multimedia Signal[C].USA:IEEE,2017.1-6.
[3] LIU K D,LIU Y W,LIU J X,et al.Joint source encoding and networking optimization for panoramic video streaming over LTE-A downlink[A].Proceedings of the GLOBECOM[C].USA:IEEE,2017.1-7.
[4] SERRANO A,SITZMANN V,RUIZ-BORAU J,et al.Movie editing and cognitive event segmentation in virtual reality video[J].ACM Transactions on Graphics,2017,36(4):1-7.
[5] BORJI A,CHENG M M,JIANG H,et al.Salient object detection:a survey[J].ArXiv Preprint,2014,2(4):1411-5878.
[6] JUDD T,DURAMD F,TORRALBA A.A benchmark of computational models of saliency to predict human fixations[J/OL].Technical Report,MIT,2012,http://hdl.handle.net/1721.1/68590.
[7] GUTIERREZ J,DAVID E,RAI Y,et al.Toolbox and dataset for the development of saliency and scanpath models for omni-directional/360° still images[J].Signal Processing Image Communication,2018,69:1-7.
[8] HE Y,VISHWANATH B,XIU X,et al.AHG8:interdigital’s projection format conversion tool [S].
[9] KOCH C,ULLMAN S.Shifts in Selective Visual Attention:Towards the Underlying Neural Circuitry[M].Springer,Dordrecht:Matters of Intelligence,1987.115-141.
[10] ITTI L,KOCH C,NIEBUR E.A model of saliency-based visual attention for rapid scene analysis[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1998,V20(11):1254-1259.
[11] HAREL J,KOCH C,PERONA P.Graph-based visual saliency[A].Proceedings of Advances in Neural Information Processing Systems[C].USA:IEEE,2007.545-552.
[12] ACHANTA R,HEMAMI S,ESTRADA F,et al.Frequency-tuned salient region detection[A].Proceedings of Computer Vision and Pattern Recognition[C].USA:IEEE,2009.1597-1604.
[13] HOU X,ZHANG L.Saliency detection:a spectral residual approach[A].Proceedings of Computer Vision and Pattern Recognition[C].USA:IEEE,2007.1-8.
[14] ZHANG J,SCLAROFF S.Saliency detection:a boolean map approach[A].Proceedings of International Conference on Computer Vision[C].USA:IEEE,2013.153-160.
[15] KRIZHEVSHY A,SUTSKEVER I,HINTON G E.Imagenet classification with deep convolutional neural networks[A].Proceedings of International Conference on Neural Information[C].Qatar:Neural Information Processing,2012.1097-1105.
[16] MATHIEU M,LeCun Y,FERGUS R,et al.OverFeat:integrated recognition,localization and detection using convolutional networks[J].ArXiv Preprint,2013,2(4):1-7.
[17] LING J,ZHANG K,ZHANG Y,et al.A saliency prediction model on 360 degree images using color dictionary based sparse representation[J].Signal Processing Image Communication,2018,69:60-68.
[18] LEBRETON P,RAAKE A.GBVS360,BMS360,ProSal:extending existing saliency prediction models from 2D to omni-directional images[J].Signal Processing Image Communication,2018,69:69-78.
[19] STARTSEV M,DORR M.360-aware saliency estimation with conventional image saliency predictors[J].Signal Processing Image Communication,2018,69:43-52.
[20] BATTISTI F,BALDONI S,BRIZZI M,et al.A feature-based approach for saliency estimation of omni-directional images[J].Signal Processing Image Communication,2018,69:53-59.
[21] DING Y,LIU Y,LIU J,et al.Panoramic image saliency detection by fusing visual frequency feature and viewing behavior pattern[A].Proceedings of the Pacific-Rim Conference on Multimedia[C].China:MTAP,2018.418-429.
[22] ZHU Y,ZHAI G,MIN X.The prediction of head and eye movement for 360 degree images[J].Signal Processing Image Communication,2018,69:15-25.
[23] BOGDANOVA I,BUR A,HUGLI H.Visual attention on the sphere[J].Computer Vision & Image Understanding,2010,114(1):100-110.
[24] MAUGEY T,MEUR O L,LIU Z.Saliency-based navigation in omni-directional image[A].Proceedings of Multimedia Signal Processing[C].USA:IEEE,2017.1-6.
[25] ABREU A,OZCINAR C,SMOLIC A.Look around you:saliency maps for omni-directional images in VR applications[A].Proceedings of Quality of Multimedia Experience[C].USA:IEEE,2017.1-6.
[26] UPENIK E,EBRAHIMI T.A simple method to obtain visual attention data in head mounted virtual reality[A].Proceedings of Multimedia & Expo Workshops[C].USA:IEEE,2017.73-78.
[27] SITZMANN V,SERRANO A,PAVEL A,et al.Saliency in VR:how do people explore virtual environments?[A].Proceedings of Transactions on Visualization and Computer Graphics[C].USA:IEEE,2018.1633-1642.
[28] MONROY R,LUTZ S,CHALASANI T,et al.SalNet360:saliency maps for omni-directional images with CNN[J].Signal Processing Image Communication,2018,69:26-34.
[29] ASSENS M,MCGUINNESS K,GIRO-I-NIETO X,et al.SaltiNet:scan-path prediction on 360 degree images using saliency volumes[J].International Conference on Computer Vision Workshops,2017,3:2331-2338.
[30] XU M,SONG Y,WANG J,et al.Modeling attention in panoramic video:a deep reinforcement learning approach[J/OL].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018.https://arxiv.org/abs/1710.10755v1.
[31] ZHANG Z,XU Y,YU J,et al.Saliency detection in 360° videos[A].Proceedings of the European Conference on Computer Vision[C].Germany:Springer,2018.488-503.
[32] CORBILLON X,DE Simone F,SIMON G.360-Degree video head movement dataset[A].Proceedings of the 8th ACM on Multimedia Systems Conference[C].New York:ACM,2017.199-204.
[33] WU C,TAN Z,WANG Z,et al.A dataset for exploring user behaviors in VR spherical video streaming[A].Proceedings of the 8th ACM on Multimedia Systems Conference[C].New York:ACM,2017.193-198.
[34] LO W,FAN C,LEE J,et al.360° video viewing dataset in head-mounted virtual reality[A].Proceedings of the 8th ACM on Multimedia Systems Conference[C].New York:ACM,2017.211-216.
[2] LUZ G,ASCENSO J,BRITES C,et al.Saliency-driven omni-directional imaging adaptive coding:modeling and assessment[A].Proceedings of the International Workshop on Multimedia Signal[C].USA:IEEE,2017.1-6.
[3] LIU K D,LIU Y W,LIU J X,et al.Joint source encoding and networking optimization for panoramic video streaming over LTE-A downlink[A].Proceedings of the GLOBECOM[C].USA:IEEE,2017.1-7.
[4] SERRANO A,SITZMANN V,RUIZ-BORAU J,et al.Movie editing and cognitive event segmentation in virtual reality video[J].ACM Transactions on Graphics,2017,36(4):1-7.
[5] BORJI A,CHENG M M,JIANG H,et al.Salient object detection:a survey[J].ArXiv Preprint,2014,2(4):1411-5878.
[6] JUDD T,DURAMD F,TORRALBA A.A benchmark of computational models of saliency to predict human fixations[J/OL].Technical Report,MIT,2012,http://hdl.handle.net/1721.1/68590.
[7] GUTIERREZ J,DAVID E,RAI Y,et al.Toolbox and dataset for the development of saliency and scanpath models for omni-directional/360° still images[J].Signal Processing Image Communication,2018,69:1-7.
[8] HE Y,VISHWANATH B,XIU X,et al.AHG8:interdigital’s projection format conversion tool [S].
[9] KOCH C,ULLMAN S.Shifts in Selective Visual Attention:Towards the Underlying Neural Circuitry[M].Springer,Dordrecht:Matters of Intelligence,1987.115-141.
[10] ITTI L,KOCH C,NIEBUR E.A model of saliency-based visual attention for rapid scene analysis[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1998,V20(11):1254-1259.
[11] HAREL J,KOCH C,PERONA P.Graph-based visual saliency[A].Proceedings of Advances in Neural Information Processing Systems[C].USA:IEEE,2007.545-552.
[12] ACHANTA R,HEMAMI S,ESTRADA F,et al.Frequency-tuned salient region detection[A].Proceedings of Computer Vision and Pattern Recognition[C].USA:IEEE,2009.1597-1604.
[13] HOU X,ZHANG L.Saliency detection:a spectral residual approach[A].Proceedings of Computer Vision and Pattern Recognition[C].USA:IEEE,2007.1-8.
[14] ZHANG J,SCLAROFF S.Saliency detection:a boolean map approach[A].Proceedings of International Conference on Computer Vision[C].USA:IEEE,2013.153-160.
[15] KRIZHEVSHY A,SUTSKEVER I,HINTON G E.Imagenet classification with deep convolutional neural networks[A].Proceedings of International Conference on Neural Information[C].Qatar:Neural Information Processing,2012.1097-1105.
[16] MATHIEU M,LeCun Y,FERGUS R,et al.OverFeat:integrated recognition,localization and detection using convolutional networks[J].ArXiv Preprint,2013,2(4):1-7.
[17] LING J,ZHANG K,ZHANG Y,et al.A saliency prediction model on 360 degree images using color dictionary based sparse representation[J].Signal Processing Image Communication,2018,69:60-68.
[18] LEBRETON P,RAAKE A.GBVS360,BMS360,ProSal:extending existing saliency prediction models from 2D to omni-directional images[J].Signal Processing Image Communication,2018,69:69-78.
[19] STARTSEV M,DORR M.360-aware saliency estimation with conventional image saliency predictors[J].Signal Processing Image Communication,2018,69:43-52.
[20] BATTISTI F,BALDONI S,BRIZZI M,et al.A feature-based approach for saliency estimation of omni-directional images[J].Signal Processing Image Communication,2018,69:53-59.
[21] DING Y,LIU Y,LIU J,et al.Panoramic image saliency detection by fusing visual frequency feature and viewing behavior pattern[A].Proceedings of the Pacific-Rim Conference on Multimedia[C].China:MTAP,2018.418-429.
[22] ZHU Y,ZHAI G,MIN X.The prediction of head and eye movement for 360 degree images[J].Signal Processing Image Communication,2018,69:15-25.
[23] BOGDANOVA I,BUR A,HUGLI H.Visual attention on the sphere[J].Computer Vision & Image Understanding,2010,114(1):100-110.
[24] MAUGEY T,MEUR O L,LIU Z.Saliency-based navigation in omni-directional image[A].Proceedings of Multimedia Signal Processing[C].USA:IEEE,2017.1-6.
[25] ABREU A,OZCINAR C,SMOLIC A.Look around you:saliency maps for omni-directional images in VR applications[A].Proceedings of Quality of Multimedia Experience[C].USA:IEEE,2017.1-6.
[26] UPENIK E,EBRAHIMI T.A simple method to obtain visual attention data in head mounted virtual reality[A].Proceedings of Multimedia & Expo Workshops[C].USA:IEEE,2017.73-78.
[27] SITZMANN V,SERRANO A,PAVEL A,et al.Saliency in VR:how do people explore virtual environments?[A].Proceedings of Transactions on Visualization and Computer Graphics[C].USA:IEEE,2018.1633-1642.
[28] MONROY R,LUTZ S,CHALASANI T,et al.SalNet360:saliency maps for omni-directional images with CNN[J].Signal Processing Image Communication,2018,69:26-34.
[29] ASSENS M,MCGUINNESS K,GIRO-I-NIETO X,et al.SaltiNet:scan-path prediction on 360 degree images using saliency volumes[J].International Conference on Computer Vision Workshops,2017,3:2331-2338.
[30] XU M,SONG Y,WANG J,et al.Modeling attention in panoramic video:a deep reinforcement learning approach[J/OL].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018.https://arxiv.org/abs/1710.10755v1.
[31] ZHANG Z,XU Y,YU J,et al.Saliency detection in 360° videos[A].Proceedings of the European Conference on Computer Vision[C].Germany:Springer,2018.488-503.
[32] CORBILLON X,DE Simone F,SIMON G.360-Degree video head movement dataset[A].Proceedings of the 8th ACM on Multimedia Systems Conference[C].New York:ACM,2017.199-204.
[33] WU C,TAN Z,WANG Z,et al.A dataset for exploring user behaviors in VR spherical video streaming[A].Proceedings of the 8th ACM on Multimedia Systems Conference[C].New York:ACM,2017.193-198.
[34] LO W,FAN C,LEE J,et al.360° video viewing dataset in head-mounted virtual reality[A].Proceedings of the 8th ACM on Multimedia Systems Conference[C].New York:ACM,2017.211-216.