MPVT: An Efficient Multi-Modal Prompt Vision Tracker for Visual Target Tracking
Source abstract
Abstract
Visual target tracking is a fundamental task in computer vision. Combining multi-modal information with tracking leverages complementary information, which improves the precision and robustness of trackers. Traditional multi-modal tracking methods typically employ a full fine-tuning scheme, i.e., fine-tuning pre-trained single-modal models to multi-modal tasks. However, this approach suffers from low transfer learning efficiency, catastrophic forgetting, and high cross-task deployment costs. To address these issues, we propose an efficient model named multi-modal prompt vision tracker (MPVT) based on an efficient prompt-tuning paradigm. Three key components are involved in the model: a decoupled input enhancement module, a dynamic adaptive prompt fusion module, and a fully connected head network module. The decoupled input enhancement module enhances input representations via positional and type embedding. The dynamic adaptive prompt fusion module achieves efficient prompt tuning and multi-modal interaction using scaled convolution and low-rank cross-modal attention mechanisms. The fully connected head network module addresses the shortcomings of traditional convolutional head networks such as inductive biases. Experimental results from RGB-T, RGB-D, and RGB-E scenarios show that MPVT outperforms state-of-the-art methods. Moreover, MPVT can save 43.8% GPU memory usage and reduce training time by 62.9% compared with a full-parameter fine-tuning model.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Video Surveillance and Tracking Methods · Visual Attention and Saliency Detection · Infrared Target Detection Methodologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
Chengdu Union Big Data Tech. Inc., Chengdu 610041, China
Zhuhai Yiyuan Technology Co., Ltd., Zhuhai 519040, China
Suining Institute of Digital Economy, Suining 629018, China
Funding
Major Science and Technology Projects in Sichuan Province · Major Program of National Natural Science Foundation of China
References (44)
Show source-provided references
1. Cui, Y., Jiang, C., Wang, L., and Wu, G. (2022, January 18–24). MixFormer: End-to-End Tracking with Iterative Mixed Attention. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01324 ↗
2. Zhang, Z., and Peng, H. (2019, January 15–20). Deeper and Wider Siamese Networks for Real-Time Visual Tracking. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00472 ↗
3. Chen, X., Peng, H., Wang, D., Lu, H., and Hu, H. (2023, January 17–24). SeqTrack: Sequence to sequence learning for visual object tracking. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01400 ↗
4. Hui, T., Xun, Z., Peng, F., Huang, J., Wei, X., Wei, X., Dai, J., Han, J., and Liu, S. (2023, January 17–24). Bridging search region interaction with template for RGB-T tracking. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01310 ↗
5. Attribute-based progressive fusion network for RGBT tracking
6. Learning reliable modal weight with transformer for robust RGBT tracking10.1016/j.knosys.2022.108945 ↗
7. Yan, S., Yang, J., Käpylä, J., Zheng, F., Leonardis, A., and Kämäräinen, J.K. (2021, January 10–17). DepthTrack: Unveiling the power of RGBD tracking. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.01055 ↗
8. Qian, Y., Yan, S., Lukežič, A., Kristan, M., Kämäräinen, J.K., and Matas, J. (2021, January 10–15). DAL: A deep depth-aware long-term tracker. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.10.1109/ICPR48806.2021.9412984 ↗
9. VisEvent: Reliable object tracking via collaboration of frame and event flows10.1109/TCYB.2023.3318601 ↗
10. Zhang, J., Wang, Y., Liu, W., Li, M., Bai, J., Yin, B., and Yang, X. (2023, January 17–24). Frame-event alignment and fusion network for high frame rate tracking. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.00943 ↗
11. Zhu, Z., Hou, J., and Wu, D.O. (2023, January 1–6). Cross-modal orthogonal high-rank augmentation for RGB-event transformer-trackers. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.02015 ↗
12. Challenge-aware RGBT tracking10.1007/978-3-030-58542-6_14 ↗
13. Learning adaptive attribute-driven representation for real-time RGB-T tracking10.1007/s11263-021-01495-3 ↗
14. Li, C.L., Lu, A., Zheng, A.H., Tu, Z., and Tang, J. (2019, January 27–28). Multi-adapter RGBT tracking. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Republic of Korea.10.1109/ICCVW.2019.00279 ↗
15. RGBT tracking via multi-adapter network with hierarchical divergence loss10.1109/TIP.2021.3087341 ↗
16. Duality-gated mutual condition network for RGBT tracking10.1109/TNNLS.2022.3157594 ↗
17. Quality-aware feature aggregation network for robust RGBT tracking10.1109/TIV.2020.2980735 ↗
18. Wang, C., Xu, C., Cui, Z., Zhou, L., Zhang, T., Zhang, X., and Yang, J. (2020, January 13–19). Cross-modal pattern-propagation for RGB-T tracking. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00709 ↗
19. Zhu, Y., Guo, R., Wang, Y., and Lu, W. (2023, January 17–24). ViPT: Visual prompt multi-modal tracking. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.00918 ↗
20. Hou, R., Xu, B., Ren, T., and Wu, G. (2023, January 10–14). MTNet: Learning modality-aware representation with transformer for RGBT tracking. Proceedings of the 2023 IEEE International Conference on Multimedia and Expo (ICME), Brisbane, Australia.10.1109/ICME55011.2023.00203 ↗
21. Xue, Y., Zhang, J., Lin, Z., Li, C., Huo, B., and Zhang, Y. (2023). SiamCAF: Complementary attention fusion-based siamese network for RGBT tracking. Remote Sens., 15.10.3390/rs15133252 ↗
22. Multi-modal adapter for RGB-T tracking10.1016/j.inffus.2025.102940 ↗
23. DS-KCF: A real-time tracker for RGB-D data10.1007/s11554-016-0654-3 ↗
24. Robust fusion of color and depth data for RGB-D target tracking using adaptive range-invariant depth models and spatio-temporal consistency constraints10.1109/TCYB.2017.2740952 ↗
25. Hierarchical multi-modal fusion FCN with attention model for RGB-D tracking10.1016/j.inffus.2018.09.014 ↗
26. Kart, U., Lukežič, A., Kristan, M., Kämäräinen, J.K., and Matas, J. (2019, January 15–20). Object tracking by reconstruction with view-specific discriminative correlation filters. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00143 ↗
27. RGB-D visual object tracking with transformer-based multi-modal feature fusion10.1016/j.knosys.2025.113531 ↗
28. Zhang, J., Yang, X., Fu, Y., Wei, X., Yin, B., and Dong, B. (2021, January 10–17). Object tracking by jointly exploiting frame and event domain. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.01280 ↗
29. Yang, J., Li, Z., Zheng, F., Leonardis, A., and Song, J. (2022, January 10–14). Prompting for multi-modal tracking. Proceedings of the MM ’22: 30th ACM International Conference on Multimedia, Lisboa, Portugal.10.1145/3503161.3547851 ↗
30. Hong, L., Yan, S., Zhang, R., Li, W., Zhou, X., Guo, P., Jiang, K., Chen, Y., Li, J., and Chen, Z. (2024, January 16–22). OneTracker: Unifying visual object tracking with foundation models and efficient tuning. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01805 ↗
31. Explicit visual prompts for visual object tracking10.1609/aaai.v38i5.28286 ↗
32. Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., and Hassner, T. (2022, January 23–27). Joint feature learning and relation modeling for tracking: A one-stream framework. Proceedings of the Computer Vision—ECCV 2022, Tel Aviv, Israel.10.1007/978-3-031-19778-9 ↗
33. LasHeR: A large-scale high-diversity benchmark for RGBT tracking10.1109/TIP.2021.3130533 ↗
34. Loshchilov, I., and Hutter, F. (2017). SGDR: Stochastic gradient descent with warm restarts. arXiv.
35. Glorot, X., and Bengio, Y. (2010, January 13–15). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, Chia Laguna Resort, Italy.
36. Nam, H., and Han, B. (2016, January 27–30). Learning multi-domain convolutional neural networks for visual tracking. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.465 ↗
37. Danelljan, M., Van Gool, L., and Timofte, R. (2020, January 13–19). Probabilistic regression for visual tracking. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00721 ↗
38. Dai, K., Zhang, Y., Wang, D., Li, J., Lu, H., and Yang, X. (2020, January 13–19). High-performance long-term tracking with meta-updater. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00633 ↗
39. Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., and Lu, H. (2021, January 20–25). Transformer tracking. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00803 ↗
40. Temporal adaptive RGBT tracking with modality prompt10.1609/aaai.v38i6.28352 ↗
41. Temporal adaptive bidirectional bridging for RGB-D tracking10.1016/j.patcog.2024.111053 ↗
42. Wu, Z., Zheng, J., Ren, X., Vasluianu, F.A., Ma, C., Paudel, D.P., Van Gool, L., and Timofte, R. (2024, January 16–22). Single-model and any-modality for video object tracking. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01812 ↗
43. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv.
44. CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models