journal article·2022-08-02Open access

Complementary Segmentation of Primary Video Objects with Reversible Flows

Junjie Wu, Jia Li, Long Xu

Applied Sciences· Volume 12· Issue 15

Abstract

Segmenting primary objects in a video is an important yet challenging problem in intelligent video surveillance, as it exhibits various levels of foreground/background ambiguities. To reduce such ambiguities, we propose a novel formulation via exploiting foreground and background context as well as their complementary constraint. Under this formulation, a unified objective function is further defined to encode each cue. For implementation, we design a complementary segmentation network (CSNet) with two separate branches, which can simultaneously encode the foreground and background information along with joint spatial constraints. The CSNet is trained on massive images with manually annotated salient objects in an end-to-end manner. By applying CSNet on each video frame, the spatial foreground and background maps can be initialized. To enforce temporal consistency effectively and efficiently, we divide each frame into superpixels and construct a neighborhood reversible flow that reflects the most reliable temporal correspondences between superpixels in far-away frames. With such a flow, the initialized foregroundness and backgroundness can be propagated along the temporal dimension so that primary video objects gradually pop out and distractors are well suppressed. Extensive experimental results on three video datasets show that the proposed approach achieves impressive performance in comparisons with 22 state-of-the-art models.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Visual Attention and Saliency Detection · Video Surveillance and Tracking Methods · Advanced Image and Video Retrieval Techniques

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University, Beijing 100190, China

State Key Laboratory of Space Weather, National Space Science Center, Chinese Academy of Sciences, Beijing 100190, China

Funding

National Natural Science Foundation of China

References (78)

Show source-provided references
1. Prest, A., Leistner, C., Civera, J., Schmid, C., and Ferrari, V. (2012, January 16–21). Learning object class detectors from weakly annotated video. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248065 ↗
2. Jiang, H., Wang, J., Yuan, Z., Wu, Y., Zheng, N., and Li, S. (2013, January 23–28). Salient object detection: A discriminative regional feature integration approach. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Portland, OR, USA.10.1109/CVPR.2013.271 ↗
3. Co-saliency detection via a self-paced multiple-instance learning framework10.1109/TPAMI.2016.2567393 ↗
4. Lee, G., Tai, Y.W., and Kim, J. (2016, January 27–30). Deep saliency with encoded low level distance map and high level features. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.78 ↗
5. Li, G., and Yu, Y. (2015, January 7–12). Visual saliency based on multiscale deep features. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
6. Re-thinking co-salient object detection
7. Zhao, R., Ouyang, W., Li, H., and Wang, X. (2015, January 7–12). Saliency detection by multi-context deep learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298731 ↗
8. Zhuge, M., Fan, D.P., Liu, N., Zhang, D., Xu, D., and Shao, L. (2022). Salient object detection via integrity learning. IEEE Trans. Pattern Anal. Mach. Intell.10.1109/TPAMI.2022.3179526 ↗
9. Wang, G., Chen, C., Fan, D., Hao, A., and Qin, H. (2021). Weakly Supervised Visual-Auditory Saliency Detection with Multigranularity Perception. arXiv.
10. Fan, D.P., Zhang, J., Xu, G., Cheng, M.M., and Shao, L. (2022). Salient objects in clutter. IEEE Trans. Pattern Anal. Mach. Intell.10.1109/TPAMI.2022.3166451 ↗
11. Zhao, J.X., Liu, J.J., Fan, D.P., Cao, Y., Yang, J., and Cheng, M.M. (2019, January 27–28). EGNet: Edge guidance network for salient object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.10.1109/ICCV.2019.00887 ↗
12. Fan, D.P., Cheng, M.M., Liu, J.J., Gao, S.H., Hou, Q., and Borji, A. (2018, January 8–14). Salient objects in clutter: Bringing salient object detection to the foreground. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01267-0_12 ↗
13. Wang, L., Lu, H., Ruan, X., and Yang, M.H. (2015, January 7–12). Deep networks for saliency detection via local estimation and global search. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298938 ↗
14. Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., and Sorkine-Hornung, A. (2016, January 27–30). A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.85 ↗
15. A benchmark dataset and saliency-guided stacked autoencoders for video-based salient object detection10.1109/TIP.2017.2762594 ↗
16. Maerki, N., Perazzi, F., Wang, O., and Sorkine-Hornung, A. (2016, January 27–30). Bilateral space video segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.87 ↗
17. Ramakanth, S.A., and Babu, R.V. (2014, January 23–28). SeamSeg: Video object segmentation using patch seams. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.55 ↗
18. Seguin, G., Bojanowski, P., Lajugie, R., and Laptev, I. (2016, January 27–30). Instance-level video segmentation from object tracks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.400 ↗
19. Tsai, Y.H., Zhong, G., and Yang, M.H. (2016, January 8–14). Semantic co-segmentation in videos. Proceedings of the 14th European Conference on Computer Vision, Munich, Germany.10.1007/978-3-319-46493-0_46 ↗
20. Zhang, Y., Chen, X., Li, J., Wang, C., and Xia, C. (2015, January 7–12). Semantic object segmentation via detection in weakly labeled video. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298987 ↗
21. Zhang, D., Javed, O., and Shah, M. (2013, January 23–28). Video Object Segmentation through Spatially Accurate and Temporally Dense Extraction of Primary Object Regions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.87 ↗
22. Jang, W.D., Lee, C., and Kim, C.S. (2016, January 27–30). Primary object segmentation in videos via alternate convex optimization of foreground and background distributions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.82 ↗
23. Papazoglou, A., and Ferrari, V. (2013, January 23–28). Fast object segmentation in unconstrained video. Proceedings of the IEEE International Conference on Computer Vision, Portland, OR, USA.10.1109/ICCV.2013.223 ↗
24. Xiao, F., and Jae Lee, Y. (2016, January 27–30). Track and segment: An iterative unsupervised approach for video object proposals. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.107 ↗
25. Ji, G.P., Fu, K., Wu, Z., Fan, D.P., Shen, J., and Shao, L. (2021, January 10–17). Full-duplex strategy for video object segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00488 ↗
26. Fan, D.P., Wang, W., Cheng, M.M., and Shen, J. (2019, January 15–20). Shifting More Attention to Video Salient Object Detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00875 ↗
27. Chiu, W.C., and Fritz, M. (2013, January 23–28). Multi-class video co-segmentation with a generative multi-video model. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.48 ↗
28. Object-based multiple foreground video co-segmentation via multi-state selection graph10.1109/TIP.2015.2442915 ↗
29. Zhang, D., Javed, O., and Shah, M. (2014, January 6–12). Video object co-segmentation by regulated maximum weight cliques. Proceedings of the 13th European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10584-0_36 ↗
30. Yu, C.P., Le, H., Zelinsky, G., and Samaras, D. (2015, January 7–13). Efficient video segmentation using parametric graph partitioning. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.361 ↗
31. Saliency-aware video object segmentation10.1109/TPAMI.2017.2662005 ↗
32. Global contrast based salient region detection10.1109/TPAMI.2014.2345401 ↗
33. Jiang, P., Ling, H., Yu, J., and Peng, J. (2013, January 23–28). Salient region detection by ufo: Uniqueness, focusness and objectness. Proceedings of the IEEE International Conference on Computer Vision, Portland, OR, USA.10.1109/ICCV.2013.248 ↗
34. Zhu, W., Liang, S., Wei, Y., and Sun, J. (2014, January 23–28). Saliency Optimization from Robust Background Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.360 ↗
35. Zhang, J., and Sclaroff, S. (2013, January 23–28). Saliency detection: A boolean map approach. Proceedings of the IEEE International Conference on Computer Vision, Portland, OR, USA.10.1109/ICCV.2013.26 ↗
36. Dynamic background estimation and complementary learning for pixel-wise foreground/background segmentation10.1016/j.patcog.2016.01.031 ↗
37. Koh, Y.J., and Kim, C.S. (2017, January 21–26). Primary Object Segmentation in Videos Based on Region Augmentation and Reduction. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.784 ↗
38. Tsai, Y.H., Yang, M.H., and Black, M.J. (2016, January 27–30). Video segmentation via object flow. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.423 ↗
39. Li, J., Zheng, A., Chen, X., and Zhou, B. (2017, January 22–29). Primary video object segmentation via complementary cnns and neighborhood reversible flow. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.158 ↗
40. Liu, H., Jiang, S., Huang, Q., Xu, C., and Gao, W. (2007, January 24–29). Region-based visual attention analysis with its application in image browsing on small displays. Proceedings of the 15th ACM International Conference on Multimedia, Augsburg, Germany.10.1145/1291233.1291298 ↗
41. Perazzi, F., Krähenbühl, P., Pritch, Y., and Hornung, A. (2012, January 16–21). Saliency filters: Contrast based filtering for salient region detection. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6247743 ↗
42. Context-aware saliency detection10.1109/TPAMI.2011.272 ↗
43. Wei, Y., Wen, F., Zhu, W., and Sun, J. (2012, January 7–13). Geodesic saliency using background priors. Proceedings of the 12th European Conference on Computer Vision, Florence, Italy.10.1007/978-3-642-33712-3_3 ↗
44. Schauerte, B., and Stiefelhagen, R. (2012, January 7–13). Quaternion-based spectral saliency detection for eye fixation prediction. Proceedings of the 12th European Conference on Computer Vision, Florence, Italy.10.1007/978-3-642-33709-3_9 ↗
45. Learning to detect a salient object10.1109/TPAMI.2010.70 ↗
46. Yang, C., Zhang, L., Lu, H., Ruan, X., and Yang, M.H. (2013, January 23–28). Saliency detection via graph-based manifold ranking. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.407 ↗
47. Qin, Y., Lu, H., Xu, Y., and Wang, H. (2015, January 7–12). Saliency detection via cellular automata. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
48. Zhang, J., Sclaroff, S., Lin, Z., Shen, X., Price, B., and Mech, R. (2015, January 7–13). Minimum barrier salient object detection at 80 fps. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.165 ↗
49. Deepsaliency: Multi-task deep neural network model for salient object detection10.1109/TIP.2016.2579306 ↗
50. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
51. Liu, N., and Han, J. (2016, January 27–30). Dhsnet: Deep hierarchical saliency network for salient object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.80 ↗
52. Kuen, J., Wang, Z., and Wang, G. (2016, January 27–30). Recurrent attentional networks for saliency detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.399 ↗
53. Li, G., and Yu, Y. (2016, January 27–30). Deep contrast learning for salient object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.58 ↗
54. Yu, H., Li, J., Tian, Y., and Huang, T. (2010, January 25–29). Automatic interesting object extraction from images using complementary saliency maps. Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Italy.10.1145/1873951.1874105 ↗
55. Learning complementary saliency priors for foreground object segmentation in complex scenes10.1007/s11263-014-0737-1 ↗
56. Superpixel-based spatiotemporal saliency detection10.1109/TCSVT.2014.2308642 ↗
57. Wang, W., Shen, J., and Porikli, F. (2015, January 7–12). Saliency-aware geodesic video object segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
58. Segmentation of moving objects by long term video analysis10.1109/TPAMI.2013.242 ↗
59. Li, F., Kim, T., Humayun, A., Tsai, D., and Rehg, J.M. (2013, January 23–28). Video segmentation by tracking many figure-ground segments. Proceedings of the IEEE International Conference on Computer Vision, Portland, OR, USA.10.1109/ICCV.2013.273 ↗
60. Consistent video saliency using local gradient flow optimization and global refinement10.1109/TIP.2015.2460013 ↗
61. Nilsson, D., and Sminchisescu, C. (2016). Semantic video segmentation by gated recurrent flow propagation. arXiv.
62. Semantic video cnns through representation warping
63. Zhu, X., Xiong, Y., Dai, J., Yuan, L., and Wei, Y. (2017, January 21–26). Deep feature flow for video recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.441 ↗
64. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
65. (2016, January 01). MSRA10K. Available online: http://mmcheng.net/gsal/.
66. Yan, Q., Xu, L., Shi, J., and Jia, J. (2013, January 23–28). Hierarchical Saliency Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.153 ↗
67. SLIC Superpixels Compared to State-of-the-art Superpixel Methods10.1109/TPAMI.2012.120 ↗
68. Accurate image search using the contextual dissimilarity measure10.1109/TPAMI.2008.285 ↗
69. Jain, S.D., and Grauman, K. (2014, January 6–12). Supervoxel-consistent foreground propagation in video. Proceedings of the 13th European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10593-2_43 ↗
70. Salient object detection via structured matrix decomposition10.1109/TPAMI.2016.2562626 ↗
71. Tong, N., Lu, H., Ruan, X., and Yang, M.H. (2015, January 7–12). Salient object detection via bootstrap learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298798 ↗
72. Tu, W.C., He, S., Yang, Q., and Chien, S.Y. (2016, January 27–30). Real-time salient object detection with a minimum spanning tree. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.256 ↗
73. Wang, L., Wang, L., Lu, H., Zhang, P., and Ruan, X. (2016, January 8–14). Saliency detection with recurrent fully convolutional networks. Proceedings of the 14th European Conference on Computer Vision, Munich, Germany.10.1007/978-3-319-46493-0_50 ↗
74. Faktor, A., and Irani, M. (2014, January 1–5). Video Segmentation by Non-Local Consensus Voting. Proceedings of the British Machine Vision Conference, Nottingham, UK.10.5244/C.28.21 ↗
75. Liu, J.J., Hou, Q., Liu, Z.A., and Cheng, M.M. (2022). Poolnet+: Exploring the potential of pooling for salient object detection. IEEE Trans. Pattern Anal. Mach. Intell.10.1109/TPAMI.2021.3140168 ↗
76. Dynamic Feature Integration for Simultaneous Detection of Salient Object, Edge and Skeleton10.1109/TIP.2020.3017352 ↗
77. SAMNet: Stereoscopically Attentive Multi-scale Network for Lightweight Salient Object Detection10.1109/TIP.2021.3065239 ↗
78. Ma, M., Xia, C., and Li, J. (2021, January 2–9). Pyramidal feature shrinking for salient object detection. Proceedings of the AAAI Conference on Artificial Intelligence, Virtually.10.1609/aaai.v35i3.16331 ↗