Adaptive Natural Gradient Method for Learning of Stochastic Neural Networks in Mini-Batch Mode
Source abstract
Abstract
Gradient descent method is an essential algorithm for learning of neural networks. Among diverse variations of gradient descent method that have been developed for accelerating learning speed, the natural gradient learning is based on the theory of information geometry on stochastic neuromanifold, and is known to have ideal convergence properties. Despite its theoretical advantages, the pure natural gradient has some limitations that prevent its practical usage. In order to get the explicit value of the natural gradient, it is required to know true probability distribution of input variables, and to calculate inverse of a matrix with the square size of the number of parameters. Though an adaptive estimation of the natural gradient has been proposed as a solution, it was originally developed for online learning mode, which is computationally inefficient for the learning of large data set. In this paper, we propose a novel adaptive natural gradient estimation for mini-batch learning mode, which is commonly adopted for big data analysis. For two representative stochastic neural network models, we present explicit rules of parameter updates and learning algorithm. Through experiments on three benchmark problems, we confirm that the proposed method has superior convergence properties to the conventional methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Blind Source Separation Techniques · Neural Networks and Applications · Machine Learning and ELM
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, Kyungpook National University, Daegu 41566, Korea
Department of Computer Science, Korea National Open University, Seoul 03087, Korea
References (30)
Show source-provided references
1. Cayamcela, M.E.M., and Lim, W. (2018, January 17–19). Artificial Intelligence in 5G Technology: A Survey. Proceedings of the 2018 International Conference on Information and Communication Technology Convergence (ICTC), Jeju, Korea.
2. Di Giorgio, A., Giuseppi, A., Liberati, F., Ornatelli, A., Rabezzano, A., and Celsi, L.R. (2017, January 3–6). On the optimization of energy storage system placement for protecting power transmission grids against dynamic load altering attacks. Proceedings of the 25th Mediterranean Conference on Control and Automation (MED), Valletta, Malta.10.1109/MED.2017.7984247 ↗
3. Suraci, V., Celsi, L.R., Giuseppi, A., and Di Giorgio, A. (2017, January 3–6). A distributed wardrop control algorithm for load balancing in smart grids. Proceedings of the 25th Mediterranean Conference on Control and Automation (MED), Valletta, Malta.10.1109/MED.2017.7984210 ↗
4. Rumelhart, D., Hinton, G.E., and Williams, R.J. (1986). Learning Internal Representations by Error Backpropagation. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, 1, Foundations, MIT Press.
5. Bengio, Y. (2009). Learning Deep Architectures for AI., Now Publishers.10.1561/9781601982957 ↗
6. Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R.R. (2012). Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors. arXiv.
7. Kingma, D.P., and Ba, J.L. (2015, January 7–9). ADAM: A method for stochastic optimization. Proceedings of the International Conference on Learning Representation, San Diego, CA, USA.
8. Schraudolph, N., Yu, J., and Gunter, S. (2007). A Stochastic Quasi-Newton Method for Online Convex Optimization. Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, PMLR.
9. Sohl-Dickstein, J., Poole, B., and Ganguli, S. (2014, January 21–26). Fast large-scale optimization by unifying stochastic gradient and quasi-newton methods. Proceedings of the 31st International Conference on Machine Learning (ICML-14), Beijing, China.
10. Novel preconditioners based on quasi-Newton updates for nonlinear conjugate gradient methods10.1007/s11590-016-1060-2 ↗
11. Exploiting damped techniques for nonlinear conjugate gradient methods10.1007/s00186-017-0593-1 ↗
12. Preconditioned Nonlinear Conjugate Gradient methods based on a modified secant equation
13. Glorot, X., and Bengio, Y. (2010). Understanding the Difficulty of Training Deep Feedforward Neural Networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR.
14. Natural Gradient Works Efficiently in Learning10.1162/089976698300017746 ↗
15. Singularities Affect Dynamics of Learning in Neuromanifolds10.1162/neco.2006.18.5.1007 ↗
16. Slow Dynamics Due to Singularities of Hierarchical Learning Machines10.1143/PTPS.157.275 ↗
17. On-Line Learning Theory of Soft Committee Machines with Correlated Hidden Units—Steepest Gradient Descent and Natural Gradient Descent10.1143/JPSJ.72.805 ↗
18. Pascanu, R., and Bengio, Y. (2013). Revisiting Natural Gradient for Deep Networks. arXiv.
19. Slow Dynamics and Singularity in Neural Networks—Standard Gradient vs. Natural Gradient10.1007/978-3-540-28633-2_31 ↗
20. Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons10.1162/089976600300015420 ↗
21. Adaptive natural gradient learning algorithms for various stochastic models10.1016/S0893-6080(00)00051-4 ↗
22. Park, H., and Lee, K. (2019, January 11–13). Adaptive Natural Gradient Method for Learning Neural Networks with Large Data Set in Mini-Batch Model. Proceedings of the 1st International Conference on Artificial Intelligence in Communications, Okinawa, Japan.10.1109/ICAIIC.2019.8669082 ↗
23. Saad, D. (1998). Online Algorithms and Stochastic Approximations. Online Learning and Neural Networks, Cambridge University Press.
24. Goodfellow, I., Bengio, Y., and Courvill, A. (2016). Deep Learning, MIT Press.
25. Multilayer Perceptron and Natural Gradient Learning10.1007/BF03037294 ↗
26. Natural Gradient Descent for On-Line Learning10.1103/PhysRevLett.81.5461 ↗
27. Wei, W.G.H., Liu, T., Song, A., and Zhang, K. (2017, January 20–22). An adaptive natural gradient method with adaptive step size in multilayer perceptrons. Proceedings of the 2017 Chinese Automation Congress (CAC), Jinan, China.
28. Petersen, K.B., and Pedersen, M.S. (2012). The Matrix Cookbook, Technical University of Denmark.
29. Guo, C., Ma, Y., Yang, B., Jensen, C.S., and Kaul, M. (2012). EcoMark: Evaluation Models of Vehicular Environmental Impact. Proceedings of the 20th International Conference on Advances in Geographic Information Systems, ACM.10.1145/2424321.2424356 ↗
30. Riemannian metrics for neural networks, I: Feedforward networks10.1093/imaiai/iav006 ↗