Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems

详细信息    查看全文

• 作者：Shan Sung Liew^a ; ^{ssliew2@live.utm.my" class="auth_mail" title="E-mail the corresponding author}Author Vitae ; Mohamed Khalil-Hani^a ; ^{khalil@fke.utm.my" class="auth_mail" title="E-mail the corresponding author}Author Vitae ; Rabia Bakhteri^b ; ^{rbakhteri@sightlineinnovation.com" class="auth_mail" title="E-mail the corresponding author}Author Vitae
• 关键词：Activation function ; Output boundary ; Generalization performance ; Training stability ; Deep neural network ; Convolutional neural network
• 刊名：Neurocomputing
• 出版年：2016
• 出版时间：5 December 2016
• 年：2016
• 卷：216
• 期：Complete
• 页码：718-734
• 全文大小：1421 K

文摘

This paper focuses on the enhancement of the generalization ability and training stability of deep neural networks (DNNs). New activation functions that we call bounded rectified linear unit (ReLU), bounded leaky ReLU, and bounded bi-firing are proposed. These activation functions are defined based on the desired properties of the universal approximation theorem (UAT). An additional work on providing a new set of coefficient values for the scaled hyperbolic tangent function is also presented. These works result in improved classification performances and training stability in DNNs. Experimental works using the multilayer perceptron (MLP) and convolutional neural network (CNN) models have shown that the proposed activation functions outperforms their respective original forms in regards to the classification accuracies and numerical stability. Tests on MNIST, mnist-rot-bg-img handwritten digit, and AR Purdue face databases show that significant improvements of 17.31%, 9.19%, and 74.99% can be achieved in terms of the testing misclassification error rates (MCRs), applying both mean squared error (MSE) and cross-entropy (CE) loss functions This is done without sacrificing the computational efficiency. With the MNIST dataset, bounding the output of an activation function results in a 78.58% reduction in numerical instability, and with the mnist-rot-bg-img and AR Purdue databases the problem is completely eliminated. Thus, this work has demonstrated the significance of bounding an activation function in helping to alleviate the training instability problem when training a DNN model (particularly CNN).