Online gradient method with smoothing ℓub>0 ub> regularization for feedforward neural networks
详细信息 查看全文
- 作者：Huisheng Zhang ; zhhuisheng@dlmu.edu.cnAuthor Vitae ; Yanli TangAuthor Vitae
- 关键词：Online learning ; Gradient training algorithm ; Smoothing ℓ
- 刊名：Neurocomputing
- 出版年：2017
- 出版时间：8 February 2017
- 年：2017
- 卷：224
- 期：Complete
- 页码：1-8
- 全文大小：537 K
- 卷排序：224
文摘
ℓub>p regularization has been a popular pruning method for neural networks. The parameter p was usually set as 0<p≤20<p≤2 in the literature, and practical training algorithms with ℓub>0 regularization are lacking due to the NP-hard nature of the ℓub>0 regularization problem; however, the ℓub>0 regularization tends to produce the sparsest solution, corresponding to the most parsimonious network structure which is desirable in view of the generalization ability. To this end, this paper considers an online gradient training algorithm with smoothing ℓub>0 regularization (OGTSL0) for feedforward neural networks, where the ℓub>0 regularizer is approximated by a series of smoothing functions. The underlying principle for the sparsity of OGTSL0 is provided, and the convergence of the algorithm is also theoretically analyzed. Simulation examples support the theoretical analysis and illustrate the superiority of the proposed algorithm.