A highly scalable modular bottleneck neural network for image dimensionality reduction and image transformation

详细信息    查看全文

• 作者：Manuel Carcenac ; Soydan Redif
• 关键词：Multilayer perceptron ; Modular neural network ; Levenberg ; Marquardt method ; Parallelization ; Dimensionality reduction ; Nonlinear principal component analysis ; Bottleneck neural network ; Stretched network
• 刊名：Applied Intelligence
• 出版年：2016
• 出版时间：April 2016
• 年：2016
• 卷：44
• 期：3
• 页码：557-610
• 全文大小：8,093 KB
• 参考文献：1.Carcenac M (2008) A modular neural network applied to image transformation and mental images. Neural Comput Appl 17(5):549–568CrossRef
  2.Osherson DN, Weinstein S, Stoli M (1990) Modular learning. In: Schwartz EL (ed) Computational neuroscience. MIT Press, Cambridge, MA, pp 369–377
  3.Jacobs RA, Jordan MI, Nowlan SJ, Hinton GE (1991) Adaptive mixtures of local experts. Neural Comput 3:79–87CrossRef
  4.Jordan MI (1994) A statistical approach to decision tree modeling. Proceedings of the ACM Conference on Computer Learning Theory
  5.Wu CL, Chau KW, Fan C (2010) Prediction of rainfall time series using modular artificial neural networks coupled with data-preprocessing techniques. J Hydrol 389(1-2):146–167CrossRef
  6.Carcenac M (2009) A modular neural network for super-resolution of human faces. Appl Intell 30(2):168–186CrossRef
  7.Kramer MA (1991) Nonlinear principal component analysis using autoassociative neural networks. AIChE J 37(2):233–243CrossRef
  8.CUDA Toolkit Documentation http://​docs.​nvidia.​com/​cuda , Nvidia Corporation
  9.Linsker R (1986) From basic network principles to neural architecture: emergence of spatial-opponent cells. P Natl Acad Sci USA 83:7508–7512CrossRef
  10.Linsker R (1988) Self-organization in a perceptual network. Computer 21(3):105–117CrossRef
  11.LeCun Y (1993) Efficient learning and second-order methods. A tutorial at NIPS 93, Denver
  12.Marquardt D (1963) An algorithm for least-squares estimation of nonlinear parameters. SIAM J Soc Ind Appl Math 11(2):431–441MathSciNet CrossRef MATH
  13.Hinton GE (1989) Connectionist learning procedures. Artif Intell 40:185–234CrossRef
  14.Hsieh WW, Tang B (1998) Applying neural network models to prediction and data analysis in meteorology and oceanography. B Am Meteorol Soc 79(9):1855–1870CrossRef
  15.CUDA Toolkit Documentation http://​docs.​nvidia.​com/​cuda/​cublas , Nvidia Corporation
  16.Carcenac M (2014) From tile algorithm to stripe algorithm: a CUBLAS-based parallel implementation on GPUs of Gauss method for the resolution of extremely large dense linear systems stored on an array of solid state devices. J Supercomput 68(1):365–413CrossRef
  17.Buttari A, Langou J, Kurzak J, Dongarra J (2009) A class of parallel tiled linear algebra algorithms for multicore architectures. Parallel Comput 35:38–53MathSciNet CrossRef
  18.Agullo E, Augonnet C, Dongarra J, Faverge M, Langou J, Ltaief H, Tomov S (2011) LU factorization for accelerator-based systems AICCSA’ 11 Conference 217–224
  19.Hsieh WW (2001) Nonlinear principal component analysis by neural networks. Tellus 53A:599–615CrossRef
  20.Monahan AH (2001) Nonlinear principal component analysis: tropical Indo-Pacific sea surface temperature and sea level pressure. J Climate 14:219–233CrossRef
  21.Hsieh WW (2007) Nonlinear principal component analysis of noisy data. Neural Netw 20:434–443CrossRef MATH
  22.3D models http://​www.​turbosquid.​com , TurboSquid
  23.Monahan AD (2000) Nonlinear principal component analysis by neural networks: theory and application to the Lorentz system. J Climate 13:821–835CrossRef
  24.Lorenz EN (1963) Deterministic nonperiodic flow. J Atmos Sci 20:130–141CrossRef
  25.Kirby MJ, Miranda R (1996) Circular nodes in neural networks. Neural Comput 8:390–402CrossRef
  26.Bengio Y, LeCun Y (2007) Scaling learning algorithms towards AI. In: Bottou L, Chapelle O, DeCoste D, Weston J (eds) Large Scale Kernel Machines. MIT, Press, Cambridge, MA, pp 321–360
  27.Chen K (2015) Deep and Modular Neural Networks Kacprzyk J, Pedrycz W (eds)
  28.Bengio Y (2009) Learning Deep Architectures for AI. Found and Trends in Mach Learn 2(1):1–127MathSciNet CrossRef MATH
  29.Hinton GE, Osindero S, Teh Y (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(9):1527–1554MathSciNet CrossRef MATH
  30.Erhan D, Bengio Y, Courville A, Manzagol PA, Vincent P, Bengio S (2010) Why does unsupervised pre-training help deep learning J Mach Learn Res 11:625–660MathSciNet MATH
  31.Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS-10), 249–256
  32.Schmidhuber J (2014) Deep Learning in Neural Networks: An Overview. Technical Report IDSIA-03-14. arXiv:1404.​7828v4
  33.Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504– 507MathSciNet CrossRef MATH
  34.Salakhutdinov RR, Hinton GE Training a deep autoencoder or a classifier on MNIST digits. http://​www.​cs.​toronto.​edu/​~hinton/​MatlabForScience​Paper.​html
  35.Kailath T (1980) Linear systems. Prentice-Hall, Englewood Cliffs, NJMATH
  36.McWhirter JG, Baxter PD, Cooper T, Redif S, Foster J (2007) An EVD algorithm for para-Hermitian polynomial matrices. IEEE T Signal Proces 55(5):2158–2169MathSciNet CrossRef
  37.Redif S, McWhirter JG, Weiss S (2011) Design of FIR paraunitary filter banks for subband coding using a polynomial eigenvalue decomposition. IEEE T Signal Proces 59(11):5253–5264MathSciNet CrossRef
  38.Redif S, McWhirter JG, Baxter P, Cooper T (2006) Robust broadband adaptive beamforming via polynomial eigenvalues Proceedings of the IEEE Oceans Conference, 1–6
  39.Weiss S, Redif S, Cooper T, Liu C, Baxter PD, McWhirter JG (2006) Paraunitary oversampled filter bank design for channel coding. EURASIP J Appl Si Pr 2006:1–10CrossRef MATH
  40.Ta CH, Weiss S (2007) A design of precoding and equalisation for broadband MIMO systems. Proceedings of the 15th International Conference on Digital Signal Processing, Cardiff, UK, 571–574
  41.Alrmah MA, Weiss S, Redif S, Lambotharan S, McWhirter JG (2013) Angle of arrival estimation for broadband signals: A comparison. Proceedings of the Intelligent Signal Processing Conference, IET, London, UK
  42.Redif S, Weiss S, McWhirter JG (2015) Sequential matrix diagonalisation algorithms for polynomial EVD of parahermitian matrices. IEEE T Signal Proces 63(1):81–89MathSciNet CrossRef
  43.Corr J, Thomson K, Weiss S, McWhirter JG, Redif S, Proudler IK (2014) Multiple shift maximum element sequential matrix diagonalisation for parahermitian matrices. Proceedings of the IEEE Workshop on Statistical Signal Processing, Gold Coast, Australia, 312–315
  44.Krauth W, Mézard M, Nadal JP (1988) Basins of attraction in a perceptron-like neural network. Complex Syst 2:387–408MathSciNet MATH
  45.Uppala SM, Kallberg PW, Simmons AJ, Andrae U, Da Costa Bechtold V, Fiorino M, Gibson JK, Haseler J, Hernandez A, Kelly GA, Li X, Onogi K, Saarinen S, Sokka N, Allan RP, Andersson E, Arpe K, Balmaseda MA, Beljaars ACM, Van De Berg L, Bidlot J, Bormann N, Caires S, Chevallier F, Dethof A, Dragosavac M, Fisher M, Fuentes M, Hagemann S, Holm E, Hoskins BJ, Isaksen L, Janssen PAEM, Jenne R, Mcnally AP, Mahfouf JF, Morcrette JJ, Rayner NA, Saunders RW, Simon P, Sterl A, Trenberth KE, Untch A, Vasiljevic D, Viterbo P, Woollen J (2005) The ERA-40 re-analysis. Q J Roy Meteor Soc 131(612):2961–3012CrossRef
  
• 作者单位：Manuel Carcenac (1)
  Soydan Redif (1)
  
  1. Faculty of Engineering, European University of Lefke, Gemikonaǧı, via Mersin 10, Turkey
  
• 刊物类别：Computer Science
• 刊物主题：Artificial Intelligence and Robotics
  Mechanical Engineering
  Manufacturing, Machines and Tools
  
• 出版者：Springer Netherlands
• ISSN：1573-7497

文摘

This paper presents a highly scalable modular bottleneck neural network and its application to image dimensionality reduction and image transformation. The network is a three-dimensional lattice of modules that implements a complex mapping with full connectivity between two high-dimensional datasets. These datasets correspond to input and output pixel-based images of three airplanes with various spatial orientations. The modules are multilayer perceptrons trained with Levenberg-Marquardt method on GPUs. They are locally connected together in an original manner that allows the gradual elaboration of the global mapping. The lattice of modules is squeezed in its middle into a bottleneck, thereby reducing the dimensionality of images. Afterward, the bottleneck itself is stretched to enforce a specific transformation directly on the reduced data. Analysis of the neural values at the bottleneck shows that we can extract from them robust and discriminative descriptors of the airplanes. The approach compares favorably to other dimensionality reduction techniques.