PrimeNet: Adaptive Multi-Layer Deep Neural Structure for Enhanced Feature Selection in Early Convolution Stage
Source abstract
Abstract
The colossal depths of the deep neural network sometimes suffer from ineffective backpropagation of the gradients through all its depths, whereas the strong performance of shallower multilayer neural structures proves their ability to increase the gradient signals in the early stages of training, which easily gets backpropagated for global loss corrections. Shallow neural structures are always a good starting point for encouraging the sturdy feature characteristics of the input. In this research, a shallow, deep neural structure called PrimeNet is proposed. PrimeNet is aimed to dynamically identify and encourage the quality visual indicators from the input to be used by the subsequent deep network layers and increase the gradient signals in the lower stages of the training pipeline. In addition to this, the layer-wise training is performed with the help of locally generated errors, which means the gradient is not backpropagated to previous layers, and the hidden layer weights are updated during the forward pass, making this structure a backpropagation free variant. PrimeNet has obtained state-of-the-art results on various image datasets, attaining the dual objective of: (1) a compact dynamic deep neural structure, which (2) eliminates the problem of backward-locking. The PrimeNet unit is proposed as an alternative to traditional convolution and dense blocks for faster and memory-efficient training, outperforming previously reported results aimed at adaptive methods for parallel and multilayer deep neural systems.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Image Enhancement Techniques · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer and Information Sciences Department, Universiti Teknologi PETRONAS, Seri Iskander 31750, Perak, Malaysia
Center for Research in Data Science (CeRDaS), Universiti Teknologi PETRONAS, Seri Iskander 31750, Perak, Malaysia
Funding
Yayasan UTP
References (34)
Show source-provided references
1. Lin, M., Chen, Q., and Yan, S. (2013). Network in network. arXiv.
2. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
3. Rosenblatt, F. (1961). Principles of Neurodynamics. Perceptrons and the Theory of Brain Mechanisms, Cornell Aeronautical Lab. Inc.. Technical Report.10.21236/AD0256582 ↗
4. Nøkland, A., and Eidnes, L.H. (2019, January 9–15). Training neural networks with local error signals. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
5. Bolukbasi, T., Wang, J., Dekel, O., and Saligrama, V. (2017). Adaptive neural networks for fast test-time prediction. arXiv.
6. Huang, G., Chen, D., Li, T., Wu, F., van der Maaten, L., and Weinberger, K.Q. (2017). Multi-scale dense networks for resource efficient image classification. arXiv.
7. Lin, J., Rao, Y., Lu, J., and Zhou, J. (2017, January 4–9). Runtime neural pruning. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
8. Wang, X., Yu, F., Dou, Z.Y., Darrell, T., and Gonzalez, J.E. (2018, January 8–14). Skipnet: Learning dynamic routing in convolutional networks. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01261-8_25 ↗
9. Veit, A., and Belongie, S. (2018, January 8–14). Convolutional networks with adaptive inference graphs. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01246-5_1 ↗
10. Figurnov, M., Collins, M.D., Zhu, Y., Zhang, L., Huang, J., Vetrov, D., and Salakhutdinov, R. (2017, January 21–26). Spatially adaptive computation time for residual networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.194 ↗
11. Kong, S., and Fowlkes, C. (2018). Pixel-wise attentional gating for parsimonious pixel labeling. arXiv.10.1109/WACV.2019.00114 ↗
12. Li, Z., Yang, Y., Liu, X., Zhou, F., Wen, S., and Xu, W. (2017, January 22–29). Dynamic computational time for visual attention. Proceedings of the IEEE International Conference on Computer Vision Workshops, Venice, Italy.10.1109/ICCVW.2017.145 ↗
13. Ying, C., and Fragkiadaki, K. (2017). Depth-adaptive computational policies for efficient visual tracking. International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, Springer.10.1007/978-3-319-78199-0_8 ↗
14. Wu, Z., Nagarajan, T., Kumar, A., Rennie, S., Davis, L.S., Grauman, K., and Feris, R. (2018, January 18–22). Blockdrop: Dynamic inference paths in residual networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00919 ↗
15. McIntosh, L., Maheswaranathan, N., Sussillo, D., and Shlens, J. (2018, January 18–22). Recurrent segmentation for variable computational budgets. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00216 ↗
16. Incorporating Side Information by Adaptive Convolution10.1007/s11263-020-01345-8 ↗
17. Li, H., Zhang, H., Qi, X., Yang, R., and Huang, G. (2019, January 27–28). Improved techniques for training adaptive deep networks. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.10.1109/ICCV.2019.00198 ↗
18. Huang, G., Liu, S., Van der Maaten, L., and Weinberger, K.Q. (2018, January 18–22). Condensenet: An efficient densenet using learned group convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00291 ↗
19. Mnih, V., Heess, N., Graves, A., and Kavukcuoglu, K. (2014). Recurrent models of visual attention. arXiv.
20. Pluggable Micronetwork for Layer Configuration Relay in a Dynamic Deep Neural Surface10.1109/ACCESS.2021.3110709 ↗
21. He, Y., Kang, G., Dong, X., Fu, Y., and Yang, Y. (2018). Soft filter pruning for accelerating deep convolutional neural networks. arXiv.10.24963/ijcai.2018/309 ↗
22. Singh, P., Verma, V.K., Rai, P., and Namboodiri, V.P. (2019). Play and prune: Adaptive filter pruning for deep model compression. arXiv.10.24963/ijcai.2019/480 ↗
23. Lin, M., Ji, R., Zhang, Y., Zhang, B., Wu, Y., and Tian, Y. (2020). Channel pruning via automatic structure search. arXiv.10.24963/ijcai.2020/94 ↗
24. Wang, L., Dong, X., Wang, Y., Ying, X., Lin, Z., An, W., and Guo, Y. (2021). Exploring Sparsity in Image Super-Resolution for Efficient Inference. arXiv.10.1109/CVPR46437.2021.00488 ↗
25. Kim, J., Chang, S., Yun, S., and Kwak, N. (2021). Prototype-based Personalized Pruning. arXiv.10.1109/ICASSP39728.2021.9414526 ↗
26. Luo, C., Zhan, J., Hao, T., Wang, L., and Gao, W. (2021). Shift-and-Balance Attention. arXiv.
27. Lan, X., Zhu, X., and Gong, S. (2018). Knowledge distillation by on-the-fly native ensemble. arXiv.
28. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv.
29. Kingma, D.P., and Ba, J. (2017). Adam: A Method for Stochastic Optimization. arXiv.
30. Springenberg, J.T., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2015). Striving for Simplicity: The All Convolutional Net. arXiv.
31. Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. (2014). How transferable are features in deep neural networks?. arXiv.
32. Determination of over-learning and over-fitting problem in back propagation neural network10.5121/ijsc.2011.2204 ↗
33. Visualizing Data using t-SNE
34. He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.10.1109/CVPR.2016.90 ↗