Convolutional Sparse Modular Fusion Algorithm for Non-Rigid Registration of Visible–Infrared Images
Source abstract
Abstract
Existing image fusion algorithms involve extensive models and high computational demands when processing source images that require non-rigid registration, which may not align with the practical needs of engineering applications. To tackle this challenge, this study proposes a comprehensive framework for convolutional sparse fusion in the context of non-rigid registration of visible–infrared images. Our approach begins with an attention-based convolutional sparse encoder to extract cross-modal feature encodings from source images. To enhance feature extraction, we introduce a feature-guided loss and an information entropy loss to guide the extraction of homogeneous and isolated features, resulting in a feature decomposition network. Next, we create a registration module that estimates the registration parameters based on homogeneous feature pairs. Finally, we develop an image fusion module by applying homogeneous and isolated feature filtering to the feature groups, resulting in high-quality fused images with maximized information retention. Experimental results on multiple datasets indicate that, compared with similar studies, the proposed algorithm achieves an average improvement of 8.3% in image registration and 30.6% in fusion performance in mutual information. In addition, in downstream target recognition tasks, the fusion images generated by the proposed algorithm show a maximum improvement of 20.1% in average relative accuracy compared with the original images. Importantly, our algorithm maintains a relatively lightweight computational and parameter load.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Fusion Techniques · Infrared Target Detection Methodologies · Remote-Sensing Image Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Mechanical and Energy Engineering, Zhejiang University of Science and Technology, Hangzhou 310023, China
References (33)
Show source-provided references
1. Deep Learning-Based Multi-Focus Image Fusion: A Survey and a Comparative Study
2. Wang, J., Xi, X., Li, D., Li, F., and Zhang, G. (2023). GRPAFusion: A Gradient Residual and Pyramid Attention-Based Multiscale Network for Multimodal Image Fusion. Entropy, 25.10.3390/e25010169 ↗
3. PIAFusion: A Progressive Infrared and Visible Image Fusion Network Based on Illumination Aware10.1016/j.inffus.2022.03.007 ↗
4. Rethinking the Necessity of Image Fusion in High-Level Vision Tasks: A Practical Infrared and Visible Image Fusion Network Based on Progressive Semantic Injection and Scene Fidelity10.1016/j.inffus.2023.101870 ↗
5. KDFuse: A High-Level Vision Task-Driven Infrared and Visible Image Fusion Method Based on Cross-Domain Knowledge Distillation10.1016/j.inffus.2025.102944 ↗
6. Image Fusion in the Loop of High-Level Vision Tasks: A Semantic-Aware Real-Time Infrared and Visible Image Fusion Network10.1016/j.inffus.2021.12.004 ↗
7. Edge-Assisted Object Segmentation Using Multimodal Feature Aggregation and Learning10.1145/3612922 ↗
8. Zhao, Z., Bai, H., Zhang, J., Zhang, Y., Xu, S., Lin, Z., Timofte, R., and Van Gool, L. (2023, January 17–24). CDDFuse: Correlation-driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00572 ↗
9. DCAFusion: A Novel General Image Fusion Framework Based on Reference Image Reconstruction and Dual-Cross Attention Mechanism10.1016/j.ins.2024.121772 ↗
10. TUFusion: A Transformer-Based Universal Fusion Algorithm for Multimodal Images10.1109/TCSVT.2023.3296745 ↗
11. A Degradation-Aware Guided Fusion Network for Infrared and Visible Image10.1016/j.inffus.2025.102931 ↗
12. Shahsavarani, S., Lopez, F., Ibarra-Castanedo, C., and Maldague, X.P.V. (2024). Robust Multi-Modal Image Registration for Image Fusion Enhancement in Infrastructure Inspection. Sensors, 24.10.3390/s24123994 ↗
13. Infrared and Visible Image Fusion via Sparse Representation and Adaptive Dual-Channel PCNN Model Based on Co-Occurrence Analysis Shearlet Transform10.1109/TIM.2024.3522423 ↗
14. Interpretable Multi-Modal Image Registration Network Based on Disentangled Convolutional Sparse Coding10.1109/TIP.2023.3240024 ↗
15. Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., and Hassner, T. (2022, January 23–27). ReCoNet: Recurrent Correction Network for Fast and Efficient Multi-Modality Image Fusion. Proceedings of the Computer Vision—ECCV 2022, Tel Aviv, Israel.10.1007/978-3-031-20083-0 ↗
16. F3Net: Adaptive Frequency Feature Filtering Network for Multimodal Remote Sensing Image Registration10.1109/TGRS.2024.3459416 ↗
17. SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness10.1109/JAS.2022.106082 ↗
18. Semantic-Injected Bidirectional Multiscale Flow Estimation Network for Infrared and Visible Image Registration10.1109/JSTARS.2025.3527175 ↗
19. Sreter, H., and Giryes, R. (2018, January 15–20). Learned Convolutional Sparse Coding. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462313 ↗
20. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., and Yang, M. (2022, January 21–24). Restormer: Efficient Transformer for High-Resolution Image Restoration. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LU, USA.10.1109/CVPR52688.2022.00564 ↗
21. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18–23). MobileNetV2: Inverted Residuals and Linear Bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
22. Effective Pan-Sharpening with Transformer and Invertible Neural Network
23. Why Does Mutual-Information Work for Image Registration? A Deterministic Explanation10.1109/TPAMI.2014.2361512 ↗
24. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
25. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
26. Zhang, X., Ye, P., and Xiao, G. (2020, January 13–19). VIFB: A Visible and Infrared Image Fusion Benchmark. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Seattle, WA, USA.10.1109/CVPRW50498.2020.00060 ↗
27. U2Fusion: A Unified Unsupervised Image Fusion Network10.1109/TPAMI.2020.3012548 ↗
28. The TNO Multiband Image Data Collection10.1016/j.dib.2017.09.038 ↗
29. (2024, July 12). FREE—FLIR Thermal Dataset for Algorithm Training|Teledyne FLIR. Available online: https://www.flir.com/oem/adas/adas-dataset-form/.
30. Takumi, K., Watanabe, K., Ha, Q., Tejero-De-Pablos, A., Ushiku, Y., and Harada, T. (2017, January 23–27). Multispectral Object Detection for Autonomous Vehicles. Proceedings of the Thematic Workshops of ACM Multimedia 2017, New York, NY, USA. Thematic Workshops ’17.10.1145/3126686.3126727 ↗
31. Jia, X., Zhu, C., Li, M., Tang, W., and Zhou, W. (2021, January 11–17). LLVIP: A Visible-Infrared Paired Dataset for Low-Light Vision. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, BC, Canada.10.1109/ICCVW54120.2021.00389 ↗
32. Liu, J., Fan, X., Huang, Z., Wu, G., Liu, R., Zhong, W., and Luo, Z. (2022, January 21–24). Target-Aware Dual Adversarial Learning and a Multi-Scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00571 ↗
33. Jocher, G., Chaurasia, A., and Qiu, J. (2023, December 25). Ultralytics YOLOv8. Available online: https://github.com/ultralytics/ultralytics.