改进聚类的深度神经网络压缩实现方法
|
摘要
深度神经网络通常是过参数化的,并且深度学习模型存在严重冗余,这导致了计算和存储的巨大浪费.针对这个问题,本文提出了一种基于改进聚类的方法来对深度神经网络进行压缩.首先通过剪枝策略对正常训练后的网络进行修剪,然后通过K-Means++聚类得到每层权重的聚类中心从而实现权值共享,最后进行各层权重的量化.本文在LeNet,AlexNet和VGG-16上分别进行了实验,提出的方法最终将深度神经网络整体压缩了30到40倍,并且没有精度损失.实验结果表明通过基于改进聚类的压缩方法,深度神经网络在不损失精度的条件下实现了有效压缩,这使得深度网络在移动端的部署成为了可能.
Deep neural networks are typically over-parametrized and there is significant redundancy for deep learning models,which results in a waste of both computation and memory usage.In order to solve the problem,a new method based on improved clustering to compress the deep neural network is proposed.First of all,the network is pruned after the normal training.Then through the K-Means++ clustering the clustering center of each layer is gotten to achieve weight sharing.After the first two steps network weight quantization are also performed.The experiments on LeNet,AlexNet and VGG-16 are carried out,in which the deep neural network are compressed by 30 to 40 times without any loss of precision.The experimental results show that the deep neural network achieves effective compression without loss of accuracy through the method based on improved clustering,which makes the deployment of deep network on the mobile end possible.
Deep neural networks are typically over-parametrized and there is significant redundancy for deep learning models,which results in a waste of both computation and memory usage.In order to solve the problem,a new method based on improved clustering to compress the deep neural network is proposed.First of all,the network is pruned after the normal training.Then through the K-Means++ clustering the clustering center of each layer is gotten to achieve weight sharing.After the first two steps network weight quantization are also performed.The experiments on LeNet,AlexNet and VGG-16 are carried out,in which the deep neural network are compressed by 30 to 40 times without any loss of precision.The experimental results show that the deep neural network achieves effective compression without loss of accuracy through the method based on improved clustering,which makes the deployment of deep network on the mobile end possible.
引文
[1]ZHAO Dongbin,SHAO Kun,ZHU Yuanheng,et al.Review of deepreinforcement learning and discussions on the development of computer go.Control Theory&Applications,2016,33(6):701-717.(赵冬斌,邵坤,朱圆恒,等.深度强化学习综述:兼论计算机围棋的发展.控制理论与应用,2016,33(6):701-717.)
[2]LIU Zhiqing,WU Xiuzhu.Interpretation of the artificial intelligence technology behind Alphago.Control Theory&Applications,2016,33(12):1685-1687.(刘知青,吴修竹.解读AlphaGo背后的人工智能技术.控制理论与应用,2016,33(12):1685-1687.)
[3]DENIL MXSHAKIBI B,DINH L,et al.Predicting parameters in deep learning.International Conference on Neural Information Processing Systems.North Miami Beach,FL:Curran Associates Inc,2013:2148-2156.
[4]PRATT L Y.Comparing biases for minimal network construction with back-propagation.Inte rnational Co nference on Ne ural Information Processing Systems.Massachusetts:MIT Press,1988:177-185.
[5]CUN Y L,DENKER J S,SOLLA S A.Optimal brain damage.International Conference on Neural Information Processing Systems.Massachusetts:MIT Press,1989:598-605.
[6]HASSIBI B,STORK D G.Second order derivatives for network pruning:Optimal brain surgeon.Advances in Neural Information Processing Systems,1993,5:164-171.
[7]HAN S,POOL J,TRAN J,et al.Learning both weights and connections for efficient neural networks.International Conference on Neural onfornrtion Processing Systems.Massachusetts:MIT Press,2015:1135-1143.
[8]NGUYEM H V,ZHOU K,VEMULAPALLI R.Cross-domain synthesis of medical images using efficient location-sensitive deep network.Medical rmage Computing and Computer-Assisted Intervention—MICCAI 2015.Switzerland:Springer International Publishing,2015.
[9]HWANG K,SUNG W.Fixed-point feedforward deep neural network design using weights+1,0,and-1.Signal Processing Systems.San Diego,CA,USA:IEEE,2014:1-6.
[10]ANWAR S,HWANG K,SUNG W.Fixed point optimization of deep convolutional neural networks for object recognition.IEEE International Conference on Acoustics,Speech and Signal Processing.Brisbane,Q.D,Australia:IEEE,2015:1131-1135.
[11]CHEN W,WILSON J T,TYREE S,et al.Compressing neural networks with the hashing trick.Computer Science,2015:2285-2294.
[12]GONG Y,LIU L,YANG M,et al.Compressing deep convolutional networks using vector quantization.arXiv preprint arXiv:1412.6115,2014.
[13]DENTON E,ZAREMBA W,BRUNA J,et al.Exploiting linear structure within convolutional networks for efficient evaluation.International Conference on Neural Information Processing Systems.Massachusetts:MIT Press,2014:1269-1277.
[14]LIN M,CHEN Q,YAN S.Network in network.arXiv prep rint arXiv:1312.4400,2013.
[15]CHEN Y H,EMER J,SZE V.Eyeriss:A spatial architecture for energy-efficient dataflow for convolution al neural networks.The 43 rd Annual International Symposium on Computer Architecture(ISCA).Seoul,South Korea:IEEE,2016:367-379.
[16]SHAFIEE A,NAG A,MURALIMANOHAR N,et al.ISAAC:A convolutional neural network accelerator with in-situ analog arithmetic in crossbars.The 43rd Annual International Symposium on Computer Architecture.Seoul,South Korea:IEEE,2016:14-26.
[17]HINTON G,VINYALS O,DEAN J.Distilling the knowledge in a neural network.Computer Science,20 1 5,14(7):38-39.
[18]ZAGORUYKO S,KOMODAKIS N.Paying more attention to attention:Improving the performance of convolutional neural networks via attention transfer.arXiv preprint arXiv:1612.03928,20 16.
[19]HOWARD A G,ZHU M,CHEN B,et al.Mobilenets:Efficient convolutional neural networks for mobile vision applications.arXiv preprint arXiv:1 704.04861,20 1 7.
[20]XIE SiGIRSHICK R,DOLLAR P,et al.Aggregated residual transformations for deep neural networks.IEEE Conference on Computer Vision and Pattern Recognition(CVPR).Hawaii:IEEE,2017:5987-5995.
[21]HU H,PENG R,TAI Y W,et al.Network trimming:A data-driven neuron pruning approach towards efficient deep architectures.arXiv preprint arXiv:1607.03250,2016.
[22]YANG T J,CHEN Y H,SZE V.Designing energy-efficient convolutional neural networks using energy-aware pruning.arXiv preprint arXiv:1611,05128,2016.
[23]BELL N,GARLAND M.Efficient sparse matrix-vector multiplication on CUDA.Nvidia Technical Report NVR-2008-004.Santaclara:Nvidia Corporation,2008.
[24]HAN S,MAO H,DALLY W J.Deep compression:Compressing deep neural networks with pruning,trained quantization and huffman coding.arXiv preprint arXiv:1510.00149,20 1 5.
[25]ARTHUR D,VASSILVITSKIS S.K-Means++:The advantages of careful seeding.Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms.New Orleans,Louisiana,USA:SIAM,2007:10927-1035.
[2]LIU Zhiqing,WU Xiuzhu.Interpretation of the artificial intelligence technology behind Alphago.Control Theory&Applications,2016,33(12):1685-1687.(刘知青,吴修竹.解读AlphaGo背后的人工智能技术.控制理论与应用,2016,33(12):1685-1687.)
[3]DENIL MXSHAKIBI B,DINH L,et al.Predicting parameters in deep learning.International Conference on Neural Information Processing Systems.North Miami Beach,FL:Curran Associates Inc,2013:2148-2156.
[4]PRATT L Y.Comparing biases for minimal network construction with back-propagation.Inte rnational Co nference on Ne ural Information Processing Systems.Massachusetts:MIT Press,1988:177-185.
[5]CUN Y L,DENKER J S,SOLLA S A.Optimal brain damage.International Conference on Neural Information Processing Systems.Massachusetts:MIT Press,1989:598-605.
[6]HASSIBI B,STORK D G.Second order derivatives for network pruning:Optimal brain surgeon.Advances in Neural Information Processing Systems,1993,5:164-171.
[7]HAN S,POOL J,TRAN J,et al.Learning both weights and connections for efficient neural networks.International Conference on Neural onfornrtion Processing Systems.Massachusetts:MIT Press,2015:1135-1143.
[8]NGUYEM H V,ZHOU K,VEMULAPALLI R.Cross-domain synthesis of medical images using efficient location-sensitive deep network.Medical rmage Computing and Computer-Assisted Intervention—MICCAI 2015.Switzerland:Springer International Publishing,2015.
[9]HWANG K,SUNG W.Fixed-point feedforward deep neural network design using weights+1,0,and-1.Signal Processing Systems.San Diego,CA,USA:IEEE,2014:1-6.
[10]ANWAR S,HWANG K,SUNG W.Fixed point optimization of deep convolutional neural networks for object recognition.IEEE International Conference on Acoustics,Speech and Signal Processing.Brisbane,Q.D,Australia:IEEE,2015:1131-1135.
[11]CHEN W,WILSON J T,TYREE S,et al.Compressing neural networks with the hashing trick.Computer Science,2015:2285-2294.
[12]GONG Y,LIU L,YANG M,et al.Compressing deep convolutional networks using vector quantization.arXiv preprint arXiv:1412.6115,2014.
[13]DENTON E,ZAREMBA W,BRUNA J,et al.Exploiting linear structure within convolutional networks for efficient evaluation.International Conference on Neural Information Processing Systems.Massachusetts:MIT Press,2014:1269-1277.
[14]LIN M,CHEN Q,YAN S.Network in network.arXiv prep rint arXiv:1312.4400,2013.
[15]CHEN Y H,EMER J,SZE V.Eyeriss:A spatial architecture for energy-efficient dataflow for convolution al neural networks.The 43 rd Annual International Symposium on Computer Architecture(ISCA).Seoul,South Korea:IEEE,2016:367-379.
[16]SHAFIEE A,NAG A,MURALIMANOHAR N,et al.ISAAC:A convolutional neural network accelerator with in-situ analog arithmetic in crossbars.The 43rd Annual International Symposium on Computer Architecture.Seoul,South Korea:IEEE,2016:14-26.
[17]HINTON G,VINYALS O,DEAN J.Distilling the knowledge in a neural network.Computer Science,20 1 5,14(7):38-39.
[18]ZAGORUYKO S,KOMODAKIS N.Paying more attention to attention:Improving the performance of convolutional neural networks via attention transfer.arXiv preprint arXiv:1612.03928,20 16.
[19]HOWARD A G,ZHU M,CHEN B,et al.Mobilenets:Efficient convolutional neural networks for mobile vision applications.arXiv preprint arXiv:1 704.04861,20 1 7.
[20]XIE SiGIRSHICK R,DOLLAR P,et al.Aggregated residual transformations for deep neural networks.IEEE Conference on Computer Vision and Pattern Recognition(CVPR).Hawaii:IEEE,2017:5987-5995.
[21]HU H,PENG R,TAI Y W,et al.Network trimming:A data-driven neuron pruning approach towards efficient deep architectures.arXiv preprint arXiv:1607.03250,2016.
[22]YANG T J,CHEN Y H,SZE V.Designing energy-efficient convolutional neural networks using energy-aware pruning.arXiv preprint arXiv:1611,05128,2016.
[23]BELL N,GARLAND M.Efficient sparse matrix-vector multiplication on CUDA.Nvidia Technical Report NVR-2008-004.Santaclara:Nvidia Corporation,2008.
[24]HAN S,MAO H,DALLY W J.Deep compression:Compressing deep neural networks with pruning,trained quantization and huffman coding.arXiv preprint arXiv:1510.00149,20 1 5.
[25]ARTHUR D,VASSILVITSKIS S.K-Means++:The advantages of careful seeding.Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms.New Orleans,Louisiana,USA:SIAM,2007:10927-1035.