journal article·2023-01-29Open access

Monocular 3D Object Detection Based on Pseudo Multimodal Information Extraction and Keypoint Estimation

Dan Zhao, Chaofeng Ji, Guizhong Liu

Applied Sciences· Volume 13· Issue 3

Abstract

Three-dimensional object detection is an essential and fundamental task in the field of computer vision which can be widely used in various scenarios such as autonomous driving and visual navigation. In view of the current insufficient utilization of image information in current monocular camera-based 3D object detection algorithms, we propose a monocular 3D object detection algorithm based on pseudo-multimodal information extraction and keypoint estimation. We utilize the original image to generate pseudo-lidar and a bird’s-eye view, and then feed the fused data of the original image and pseudo-lidar to the keypoint-based network for an initial 3D box estimation, finally using the bird’s-eye view to refine the initial 3D box. The experimental performance of our method exceeds state-of-the-art algorithms under the evaluation criteria of 3D object detection and localization on the KITTI dataset, achieving the best experimental performance so far.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Advanced Neural Network Applications · Advanced Vision and Imaging · Robotics and Sensor-Based Localization

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

School of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China

References (47)

Show source-provided references
1. Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., and Beijbom, O. (November, January 27). PointPillars: Fast encoders for object detection from point clouds. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
2. Zhang, Y., Hu, Q., Xu, G., Ma, Y., Wan, J., and Guo, Y. (2022, January 18–24). Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds. Proceedings of the IEEE/CVF International Conference on Computer Vision, New Orleans, LA, USA.10.1109/CVPR52688.2022.01838 ↗
3. Shi, S., Wang, X., and Li, H. (November, January 27). PointRCNN: 3d object proposal generation and detection from point cloud. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
4. Chen, Y., Liu, S., Shen, X., and Jia, J. (2020, January 13–19). DSGN: Deep Stereo Geometry Network for 3D Object Detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01255 ↗
5. Li, P., Chen, X., and Shen, S. (November, January 27). Stereo R-CNN based 3d object detection for autonomous driving. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
6. Pon, A.D., Ku, J., Li, C., and Waslander, S.L. (August, January 31). Object-Centric Stereo Matching for 3D Object Detection. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), online.10.1109/ICRA40945.2020.9196660 ↗
7. Ku, J., Pon, A.D., and Waslander, S.L. (November, January 27). Monocular 3d object detection leveraging accurate proposals and shape reconstruction. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
8. Li, P., Zhao, H., Liu, P., and Cao, F. (2020, January 23–28). RTM3D: Real-time monocular 3d detection from object keypoints for autonomous driving. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58580-8_38 ↗
9. Li, B., Ouyang, W., Sheng, L., Zeng, X., and Wang, X. (November, January 27). GS3D: An efficient 3d object detection framework for autonomous driving. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
10. Weng, X., and Kitani, K. (November, January 27). Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
11. Ku, J., Mozifian, M., Lee, J., Harakeh, A., and Waslander, S.L. (2018, January 18–23). Joint 3D Proposal Generation and Object Detection from View Aggregation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Salt Lake City, UT, USA.10.1109/IROS.2018.8594049 ↗
12. Ma, X., Liu, S., Xia, Z., Zhang, H., Zeng, X., and Ouyang, W. (2020, January 23–28). Rethinking Pseudo-LiDAR Representation. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58601-0_19 ↗
13. Chen, X., Ma, H., Wan, J., Li, B., and Xia, T. (2017, January 22–29). Multi-View 3D Object Detection Network for Autonomous Driving. Proceedings of the IEEE/CVF International Conference on Computer Vision, Venice, Italy.10.1109/CVPR.2017.691 ↗
14. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 17). Are we ready for autonomous driving? the kitti vision benchmark suite. Proceedings of the IEEE/CVF International Conference on Computer Vision, Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
15. Liu, L., Lu, J., Xu, C., Tian, Q., and Zhou, J. (November, January 27). Deep fitting degree scoring network for monocular 3d object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
16. Mousavian, A., Anguelov, D., Flynn, J., and Kosecka, J. (2017, January 22–29). 3D bounding box estimation using deep learning and geometry. Proceedings of the IEEE/CVF International Conference on Computer Vision, Venice, Italy.10.1109/CVPR.2017.597 ↗
17. Ma, X., Wang, Z., Li, H., Ouyang, W., and Pengbo, Z. (November, January 27). Accurate monocular 3d object detection via color-embedded 3d reconstruction for autonomous driving. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
18. Ding, M., Yi, H., Wang, Z., Shi, J., Lu, Z., and Luo, P. (2020, January 13–19). Learning Depth-Guided Convolutions for Monocular 3D Object Detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seattle, WA, USA.10.1109/CVPR42600.2020.01169 ↗
19. Wang, Y., Chao, W., Garg, D., Hariharan, B., Campbell, M., and Weinberger, K. (November, January 27). Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
20. Zhou, Y., and Tuzel, O. (2018, January 18–23). VoxelNet: End-to-end learning for point cloud based 3d object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Salt Lake City, UT, USA.10.1109/CVPR.2018.00472 ↗
21. Yan, Y., Mao, Y., and Li, B. (2018). Second: Sparsely embedded convolutional detection. Sensors, 18.10.3390/s18103337 ↗
22. Yang, B., Luo, W., and Urtasun, R. (2018, January 18–23). Pixor: Realtime 3d object detection from point clouds. Proceedings of the IEEE/CVF International Conference on Computer Vision, Salt Lake City, UT, USA.10.1109/CVPR.2018.00798 ↗
23. Chen, Y., Liu, S., Shen, X., and Jia, J. (November, January 27). Fast-Point-RCNN. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
24. Wang, Y., Yang, B., Hu, R., Liang, M., and Urtasun, R. (October, January 27). PLUMENet: Efficient 3D Object Detection from Stereo Images. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Prague, Czech Republic.
25. You, Y., Wang, Y., Chao, W.-L., Garg, D., Pleiss, G., Hariharan, B., Campbell, M., and Weinberger, K.Q. (2019). Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving. arXiv.
26. Peng, W., Pan, H., Liu, H., and Sun, Y. (2020, January 13–19). IDA-3D: Instance-Depth-Aware 3D Object Detection from Stereo Vision for Autonomous Driving. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seattle, WA, USA.10.1109/CVPR42600.2020.01303 ↗
27. Guo, X., Shi, S., Wang, X., and Li, H. (2021, January 11–17). LIGA-Stereo: Learning LiDAR Geometry Aware Representations for Stereo-based 3D Detector. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00314 ↗
28. Sun, J., Chen, L., Xie, Y., Zhang, S., Jiang, Q., Zhou, X., and Bao, H. (2020, January 13–19). Disp R-CNN: Stereo 3D Object Detection via Shape Prior Guided Instance Disparity Estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01056 ↗
29. Liu, Y., Wang, L., and Liu, M. (June, January 30). y YOLOStereo3D: A Step Back to 2D for Efficient Stereo 3D Detection. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.10.1109/ICRA48506.2021.9561423 ↗
30. Brazil, G., and Liu, X. (November, January 27). M3D-RPN: Monocular 3d region proposal network for object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
31. Yang, L., Zhang, X., Wang, L., Zhu, M., and Li, J. (2021). Lite-FPN for Keypoint-based Monocular 3D Object Detection. arXiv.
32. Manhardt, F., Kehl, W., and Gaidon, A. (November, January 27). ROI-10D: Monocular lifting of 2d detection to 6d pose and metric shape. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
33. Liu, X., Xue, N., and Wu, T. (March, January 22). Learning Auxiliary Monocular Contexts Helps Monocular 3D Object Detection. Proceedings of the AAAI Conference on Artificial Intelligence, Virtually.
34. Liu, Z., Wu, Z., and Toth, R. (2020, January 13–19). SMOKE: Single-stage monocular 3d object detection via keypoint estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seattle, WA, USA.10.1109/CVPRW50498.2020.00506 ↗
35. Zhou, X., Wang, D., and Krahenbuhl, P. (2019). Objects as points. arXiv.
36. Xu, B., and Chen, Z. (2018, January 18–23). Multi-level fusion based 3D object detection from monocular images. Proceedings of the IEEE/CVF International Conference on Computer Vision, Salt Lake City, UT, USA.10.1109/CVPR.2018.00249 ↗
37. Reading, C., Harakeh, A., and Chae, J. (2021, January 11–17). Categorical Depth Distribution Network for Monocular 3D Object Detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/CVPR46437.2021.00845 ↗
38. Gupta, I., Rangesh, A., and Trivedi, M. (2018, January 8–14). 3D Bounding Boxes for Road Vehicles: A One-Stage, Localization Prioritized Approach using Single Monocular Images. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-11021-5_39 ↗
39. Cai, Z., Fan, Q., Feris, R.S., and Vasconcelos, N. (2016, January 11–14). A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46493-0_22 ↗
40. Fu, H., Gong, M., Wang, C., Batmanghelich, K., and Tao, D. (2018, January 18–23). Deep Ordinal Regression Network for Monocular Depth Estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Salt Lake City, UT, USA.10.1109/CVPR.2018.00214 ↗
41. Qi, C.R., Liu, W., Wu, C., Su, H., and Guibas, L.J. (2018, January 18–23). Frustum pointnets for 3d object detection from RGB-D data. Proceedings of the IEEE/CVF International Conference on Computer Vision, Salt Lake City, UT, USA.10.1109/CVPR.2018.00102 ↗
42. Cai, Y., Li, B., Jiao, Z., Li, H., Zeng, X., and Wang, X. (2020, January 7–12). Monocular 3d object detection with decoupled structured polygon estimation and height-guided depth estimation. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6618 ↗
43. Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. arXiv.
44. Chen, Y., Tai, L., Sun, K., and Li, M. (2020, January 13–19). MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seattle, WA, USA.10.1109/CVPR42600.2020.01211 ↗
45. Shi, X., Chen, Z., and Kim, T.-K. (2020, January 23–28). Distance-Normalized Unified Representation for Monocular 3D Object Detection. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58526-6_6 ↗
46. Godard, C., Aodha, O.M., and Brostow, G.J. (2017, January 22–29). Unsupervised monocular depth estimation with left-right consistency. Proceedings of the IEEE/CVF International Conference on Computer Vision, Venice, Italy.10.1109/CVPR.2017.699 ↗
47. Chang, J.-R., and Chen, Y.-S. (2018, January 18–23). Pyramid stereo matching network. Proceedings of the IEEE/CVF International Conference on Computer Vision, Salt Lake City, UT, USA.10.1109/CVPR.2018.00567 ↗