Depth Map Super-Resolution Reconstruction Based on Multi-Channel Progressive Attention Fusion Network
Source abstract
Abstract
Depth maps captured by traditional consumer-grade depth cameras are often noisy and low-resolution. Especially when upsampling low-resolution depth maps with large upsampling factors, the resulting depth maps tend to suffer from vague edges. To address these issues, we propose a multi-channel progressive attention fusion network that utilizes a pyramid structure to progressively recover high-resolution depth maps. The inputs of the network are the low-resolution depth image and its corresponding color image. The color image is used as prior information in this network to fill in the missing high-frequency information of the depth image. Then, an attention-based multi-branch feature fusion module is employed to mitigate the texture replication issue caused by incorrect guidance from the color image and inconsistencies between the color image and the depth map. This module restores the HR depth map by effectively integrating the information from both inputs. Extensive experimental results demonstrate that our proposed method outperforms existing methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Advanced Vision and Imaging · Image Processing Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Electronic Engineering, Zhejiang Gongshang University, Hangzhou 310018, China
Control System Laboratory, Graduate School of Engineering, Kogakuin University, Tokyo 163-8677, Japan
References (42)
Show source-provided references
1. Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., and Davison, A. (2011, January 16–19). Kinectfusion: Real-time 3d reconstruction and interaction using a moving depth camera. Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, Santa Barbara, CA, USA.10.1145/2047196.2047270 ↗
2. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., and Lu, H. (2019, January 15–20). Dual attention network for scene segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00326 ↗
3. Sinha, G., Shahi, R., and Shankar, M. (2010, January 19–21). Human computer interaction. Proceedings of the IEEE/CVF Conference on 3rd International Conference on Emerging Trends in Engineering and Technology, Goa, India.10.1109/ICETET.2010.85 ↗
4. Cubic convolution interpolation for digital image processing10.1109/TASSP.1981.1163711 ↗
5. Joint bilateral upsampling10.1145/1276377.1276497 ↗
6. Depth map super-resolution based on edge-guided joint trilateral upsampling10.1007/s00371-021-02057-x ↗
7. Mac Aodha, O., Campbell, N.D., Nair, A., and Brostow, G.J. (2012, January 7–13). Patch based synthesis for single depth image super-resolution. Proceedings of the Computer Vision—ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy. Proceedings, Part III 12.10.1007/978-3-642-33712-3_6 ↗
8. Li, Y., Min, D., Do, M.N., and Lu, J. (2016, January 11–14). Fast guided global interpolation for depth and motion. Proceedings of the Computer Vision—ECCV 2016: 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part III 14.10.1007/978-3-319-46487-9_44 ↗
9. Image super-resolution using deep convolutional networks10.1109/TPAMI.2015.2439281 ↗
10. Li, J., Fang, F., Mei, K., and Zhang, G. (2018, January 8–14). Multi-scale residual network for image super-resolution. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01237-3_32 ↗
11. Multi-scale feature fusion residual network for single image super-resolution10.1016/j.neucom.2019.10.076 ↗
12. Mei, Y., Fan, Y., Zhou, Y., Huang, L., Huang, T.S., and Shi, H. (2020, January 14–19). Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00573 ↗
13. Deeply supervised depth map super-resolution as novel view synthesis10.1109/TCSVT.2018.2866399 ↗
14. Pyramid-structured depth map super-resolution based on deep dense-residual network10.1109/LSP.2019.2944646 ↗
15. Xian, C., Qian, K., Zhang, Z., and Wang, C.C. (2020). Multi-scale progressive fusion learning for depth map super-resolution. arXiv.
16. Edge-guided single depth image super resolution10.1109/TIP.2015.2501749 ↗
17. Zhao, L., Bai, H., Liang, J., Wang, A., and Zhao, Y. (2017, January 10–14). Single depth image super-resolution with multiple residual dictionary learning and refinement. Proceedings of the 2017 IEEE International Conference on Multimedia and Expo (ICME), Hong Kong, China.10.1109/ICME.2017.8019331 ↗
18. Zhou, D., Wang, R., Lu, J., and Zhang, Q. (2018). Depth image super resolution based on edge-guided method. Appl. Sci., 8.10.3390/app8020298 ↗
19. Song, X., Dai, Y., Zhou, D., Liu, L., Li, W., Li, H., and Yang, R. (2020, January 14–19). Channel attention based iterative residual learning for depth map super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00567 ↗
20. Ye, X., Sun, B., Wang, Z., Yang, J., Xu, R., Li, H., and Li, B. (2020, January 12–16). Depth super-resolution via deep controllable slicing network. Proceedings of the 28th ACM International Conference on Multimedia, Virtual Event.10.1145/3394171.3413874 ↗
21. An application of markov random fields to range sensing
22. Depth upsampling based on deep edge-aware learning10.1016/j.patcog.2020.107274 ↗
23. Hui, T.W., Loy, C.C., and Tang, X. (2016, January 11–14). Depth map super-resolution by deep multi-scale guidance. Proceedings of the Computer Vision—ECCV 2016: 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part III 14.10.1007/978-3-319-46487-9_22 ↗
24. Multi-scale frequency reconstruction for guided depth map super-resolution via deep residual network10.1109/TCSVT.2018.2890271 ↗
25. High-resolution depth maps imaging via attention-based hierarchical multi-modal fusion10.1109/TIP.2021.3131041 ↗
26. Color image-guided very low-resolution depth image reconstruction10.1007/s11760-022-02425-7 ↗
27. Guo, J., Xiong, R., Ou, Y., Wang, L., and Liu, C. (2021, January 20–21). Depth Image Super-resolution via Two-Branch Network. Proceedings of the Cognitive Systems and Information Processing: 6th International Conference, ICCSIP 2021, Suzhou, China. Revised Selected Papers 6.
28. Sun, B., Ye, X., Li, B., Li, H., Wang, Z., and Xu, R. (2021, January 20–25). Learning scene structure guidance via cross-task knowledge transfer for single depth super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00770 ↗
29. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
30. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
31. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
32. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms10.1023/A:1014573219977 ↗
33. Butler, D.J., Wulff, J., Stanley, G.B., and Black, M.J. (2012, January 7–13). A naturalistic open source movie for optical flow evaluation. Proceedings of the Computer Vision—ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy. Proceedings, Part VI 12.10.1007/978-3-642-33783-3_44 ↗
34. Kiechle, M., Hawe, S., and Kleinsteuber, M. (2013, January 1–8). A joint intensity and depth co-sparse analysis model for depth map super-resolution. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.195 ↗
35. Lai, W.S., Huang, J.B., Ahuja, N., and Yang, M.H. (2017, January 21–26). Deep laplacian pyramid networks for fast and accurate super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.618 ↗
36. Color-guided depth recovery from RGB-D data using an adaptive autoregressive model10.1109/TIP.2014.2329776 ↗
37. Kim, J., Lee, J.K., and Lee, K.M. (July, January 26). Accurate image super-resolution using very deep convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
38. Residual dense network for intensity-guided depth map enhancement10.1016/j.ins.2019.05.003 ↗
39. Bansal, A., Jonna, S., and Sahay, R.R. (2019). Pag-net: Progressive attention guided depth super-resolution network. arXiv.
40. PMBANet: Progressive multi-branch aggregation network for scene depth super-resolution10.1109/TIP.2020.3002664 ↗
41. PDR-Net: Progressive depth reconstruction network for color guided depth map super-resolution10.1016/j.neucom.2022.01.050 ↗
42. Ferstl, D., Reinbacher, C., Ranftl, R., Rüther, M., and Bischof, H. (2013, January 1–8). Image guided depth upsampling using anisotropic total generalized variation. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.127 ↗