journal article·2019-10

Multitask bilateral learning for real‐time image enhancement

Kyeongbo Kong, Junggi Lee, Woo‐Jin Song, Minsung Kang, Kyung Joon Kwon, Seong Gyun Kim

Publisher page

Abstract

Abstract Nowadays, deep neural networks (DNNs) for image processing are becoming more complex; thus, reducing computational cost is increasingly important. This study highlights the construction of a DNN for real‐time image processing, training various image processing operators efficiently through multitask learning. For real‐time image processing, the proposed algorithm takes a joint upsampling approach through bilateral guided upsampling. For multitask learning, the overall network is based on an encoder‐decoder architecture, which consists of encoding, processing, and decoding components, in which the encoding and decoding components are shared by all the image processing operators. In the processing component, a semantic guidance map, which contains processing information for each image processing operator, is estimated using simple linear shifts of the shared deep features. Through these components, the proposed algorithm requires an increase of only 5% in the number of parameters to add another image processing operator and achieves faster and higher performance than that of deep‐learning‐based joint upsampling methods in local image processing as well as global image processing.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Advanced Neural Network Applications · Image Enhancement Techniques · Image Processing Techniques and Applications

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Electrical Engineering Pohang University of Science and Technology Pohang South Korea

LG Display Seoul South Korea

Funding

LG Display

References (35)

Show source-provided references
1. 10.1145/2601097.260117410.1145/2601097.2601174 ↗
2. 10.1145/2897824.292595210.1145/2897824.2925952 ↗
3. 10.1145/2185520.218552810.1145/2185520.2185528 ↗
4. 10.1145/1276377.127649710.1145/1276377.1276497 ↗
5. HeK SunJ.Fast guided filter. Arxiv. [Preprint]2015. Available from:https://arxiv.org/abs/1505.00996[Accessed 5 May 2015].
6. 10.1145/2980179.298242310.1145/2980179.2982423 ↗
7. 10.1145/3072959.307359210.1145/3072959.3073592 ↗
8. XuL RenJ YanQ LiaoR JiaJ.Deep edge‐aware filters. In: Proceedings of the International Conference on Machine Learning.2015; pp.1669–1678.
9. LiuS PanJ YangMH.Learning recursive filters for low‐level vision via a hybrid neural network. In: Proceedings of the European Conference on Computer Vision. Springer;2016; p.560–576.10.1007/978-3-319-46493-0_34 ↗
10. 10.1145/279029610.1145/2790296 ↗
11. ChenQ XuJ KoltunV.Fast image processing with fully‐convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision;2017. p.2497–2506.10.1109/ICCV.2017.273 ↗
12. LiY FangC YangJ WangZ LuX YangMH.Universal style transfer via feature transforms. In: Proceedings of the Advances in Neural Information Processing Systems;2017. p.386–396.
13. WuH ZhengS ZhangJ HuangK.Fast end‐to‐end trainable guided filter. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition;2018. p.1838–1847.10.1109/CVPR.2018.00197 ↗
14. 10.1109/TPAMI.2017.269918410.1109/TPAMI.2017.2699184 ↗
15. ZhaoH ShiJ QiX WangX JiaJ.Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition;2017. p.2881–2890.10.1109/CVPR.2017.660 ↗
16. 10.1145/1141911.114191810.1145/1141911.1141918 ↗
17. PorikliF.Constant time O (1) bilateral filtering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE;2008. p.1–8.10.1109/CVPR.2008.4587843 ↗
18. YangQ TanKH AhujaN.Real‐time O (1) bilateral filtering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition IEEE;2009. p.557–564.10.1109/CVPR.2009.5206542 ↗
19. SaeediA HoffmanM DiVerdiS GhandehariounA JohnsonM AdamsR.Multimodal prediction and personalization of photo edits with deep generative models. In: Proceedings of the International Conference on Artificial Intelligence and Statistics;2018. p.1309–1317.
20. SimonyanK ZissermanA.Very deep convolutional networks for large‐scale image recognition. In: Proceedings of the International Conference on Learning Representations;2015.
21. LeH BorjiA.What are the receptive effective receptive and projective fields of neurons in convolutional neural networks?. [Preprint] 2018. Available from:https://arxiv.org/abs/1705.07049[Accessed 7 Apr 2018].
22. ZhouB KhoslaA LapedrizaA OlivaA TorralbaA.Object detectors emerge in deep scene cnns. In: Proceedings of the International Conference on Learning Representations;2015.
23. 10.1145/1276377.127650610.1145/1276377.1276506 ↗
24. 10.1007/s11263-005-4939-z10.1007/s11263-005-4939-z ↗
25. BengioY MesnilG DauphinY RifaiS.Better mixing via deep representations. In: Proceedings of the International Conference on Machine Learning;2013. p.552–560.
26. UpchurchP GardnerJ PleissG et al.Deep feature interpolation for image content changes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition;2017. p.7064–7073.10.1109/CVPR.2017.645 ↗
27. ChenYC LinH ShuM LiR TaoX ShenX et al.Facelet‐bank for fast portrait manipulation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition;2018. p.3541–3549.10.1109/CVPR.2018.00373 ↗
28. GlorotX BengioY.Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth International Conference on Artificial Intelligence and Statistics;2010. p.249–256.
29. IoffeS SzegedyC.Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the International Conference on Machine Learning;2015. p.448–456.
30. SimonyanK ZissermanA.Adam: a method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations;2015.
31. AbadiM BarhamP ChenJ et al.Tensorflow: a system for large‐scale machine learning. In: 12th Symposium on Operating Systems Design and Implementation;2016. p.265–283.
32. BychkovskyV ParisS ChanE DurandF.Learning photographic global tonal adjustment with a database of input/output image pairs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition;2011. p.97–104.10.1109/CVPR.2011.5995413 ↗
33. RuderS.An overview of multi‐task learning in deep neural networks. arXiv preprint arXiv:170605098.2017.
34. LiuZ LuoP WangX TangX.Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision;2015. p.3730–3738.10.1109/ICCV.2015.425 ↗
35. JainV Learned‐MillerE.Fddb: a benchmark for face detection in unconstrained settings. UMass Amherst Technical Report;2010.