Convergence analyses on sparse feedforward neural networks via group lasso regularization

详细信息    查看全文

• 作者：Jian Wang ; ^a ; ^d ; ^{wangjiannl@upc.edu.cn} ; Qingling Cai^b ; Qingquan Chang^c ; Jacek M. Zurada^d ; ^e ; ^{jacek.zurada@louisville.edu}
• 关键词：Clarke gradient ; Convergence ; Feedforward neural networks ; Group Lasso ; Non-differentiability
• 刊名：Information Sciences
• 出版年：2017
• 出版时间：March 2017
• 年：2017
• 卷：381
• 期：Complete
• 页码：250-269
• 全文大小：1267 K
• 卷排序：381

文摘

In this paper, a new variant of feedforward neural networks has been proposed for a class of nonsmooth optimization problems. The penalty term of the presented neural networks stems from the Group Lasso method which selects hidden variables in a grouped manner. To deal with the non-differentiability of the original penalty term (l1−l2 norm) and avoid oscillations, smoothing techniques have been used to approximate the objective function. It is assumed that the training samples are supplied to the networks in a specific incremental way during training, that is, in each cycle samples are supplied in a fixed order. Then, under suitable assumptions on learning rate, penalization coefficients and smoothing parameters, the weak and strong convergence of the training process for the smoothing neural networks have been proved. The convergence analysis shows that the gradient of the smoothing error function approaches zero and the weight sequence converges to a fixed point, respectively. We demonstrate how the smoothing approximation parameter can be updated in the training procedure so as to guarantee the convergence of the procedure to a Clarke stationary point of the original optimization problem. In addition, we have proved that the original nonsmoothing algorithm with l1−l2l1−l2 norm penalty converges consistently to the same optimum solution with the corresponding smoothed algorithm. Numerical simulations demonstrate the convergence and effectiveness of the proposed training algorithm.