Volumes and Issues
Home » Optics and Spectroscopy » Year 2026, issue 5 » Article p. 489
<<<>>>
Methods for hyperspectral and multispectral images fusion based on per pixel spectral transformations and cyclic consistency
Russian version
Khodyrev R. R.1, Bazhenov S. A.1, Kabanova T. V. 1, Vrazhnov D. A. 1, Kistenev Y.V. 1
1Tomsk State University, Tomsk, Russia
Email: hodarev.ruslan@mail.ru, bazhenov.stepan1@gmail.com, tvk@bk.ru, vda@iao.ru, yuk@iao.ru
PDF
The paper presents two approaches for hyperspectral and multispectral images fusion. The first approach is based on the WMatrixGenerator module, which generates per pixel spectral transformation matrices with quadratic correction. The second approach uses the HSIMSICycleGAN architecture with cyclical consistency and physically motivated losses. The experimental evaluation showed the competitiveness of the proposed methods for the PSNR, SSIM, SAM and ERGAS metrics on the Real HSI/MSI/PAN dataset. The methods work on paired observations of hyperspectral and multispectral images of the same scene, but do not require high-resolution reference data. Keywords: image fusion, hyperspectral images, multispectral images, deep learning, non-negative least squares, CycleGAN, remote sensing, neural networks.
  1. Hyperspectral Data Exploitation: Theory and Applications, ed. by C.I. Chang (John Wiley \& Sons, 2007). DOI: 10.1002/9780470124628.ch3
  2. T. Adao, J. Hruv ska, L. Padua, J. Bessa, E. Peres, R. Morais, J.J. Sousa. Remote Sensing, 9 (11), 1110 (2017). DOI: 10.3390/rs9111110
  3. G. Avola, A. Matese, E. Riggi. Remote Sensing, 15 (7), 1917 (2023). DOI: 10.3390/rs15071917
  4. S. Du, Y. Zou, Z. Wang, X. Li, Y. Li, C. Shang, Q. Shen. arXiv:2412.04802, (2024). DOI: 10.48550/arXiv.2412.04802
  5. J. Yang, L. Xiao, Y.-Q. Zhao, J.C.-W. Chan. IEEE Transactions on Neural Networks and Learning Systems, 35 (9), 13017-13031 (2024). DOI: 10.1109/TNNLS.2023.3266038
  6. J. Liu, Z. Wu, L. Xiao, X.-J. Wu. IEEE Transactions on Geoscience and Remote Sensing, 60, 1-12 (2021). DOI: 10.1109/TGRS.2022.3143156
  7. K. Zheng, L. Gao, W. Liao, D. Hong, B. Zhang, X. Cui. IEEE Transactions on Neural Networks and Learning Systems, 59 (3), 2487-2502 (2021). DOI: 10.1109/TGRS.2020.3006534
  8. J. Yao, D. Hong, J. Chanussot, D. Meng, X. Zhu, Z. Xu. European Conference on Computer Vision (Cham: Springer International Publishing, 2020), p. 208-224. DOI: 10.1007/978-3-030-58526-6_13
  9. Y. Qu, H. Qi, C. Kwan. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2511-2520 (2018). DOI: 10.48550/arXiv.1904.12175
  10. S. Li, R. Dian, L. Fang, J.M. Bioucas-Dias. IEEE Transactions on Image Processing, 2018. DOI: 10.1109/TIP.2018.2836307
  11. M. Simoes, J. Bioucas-Dias, L.B. Almeida, J. Chanussot. IEEE Transactions on Geoscience and Remote Sensing, 2015. DOI: 10.1109/TGRS.2014.2375320
  12. W. Dong, F. Fu, G. Shi, X. Cao, J. Wu, G. Li, X. Li. IEEE Transactions on Image Processing, 2016. DOI: 10.1109/TIP.2016.2542360
  13. J. Yang, L. Xiao, Y. Zhao, J.C.-W. Chan. IEEE Transactions on Neural Networks and Learning Systems. 35 (9), 13017-13031 (2023). DOI: 10.1109/TNNLS.2023.3266038
  14. B. Li, X. Wang, H. Xu. arXiv:2409.06705, 2024. DOI: 10.48550/arXiv.2409.06705
  15. H. Wang, Y. Xu, Z. Wu, Z. Wei. IEEE Transactions on Neural Networks and Learning Systems, 2024. DOI: 10.1109/TNNLS.2024.3457781
  16. K. Yang, M. Bai, T. Lu. arXiv preprint arXiv:2407.05279, 2024. DOI: 10.48550/arXiv.2407.05279
  17. L. Wald. Data Fusion: Definitions and Architectures: Fusion of Images of Different Spatial Resolutions (Presses des MINES, 2002)
  18. R.A. Schowengerdt. Remote Sensing: Models and Methods for Image Processing, 3rd ed. (Academic Press, London, 2006)
  19. A. Hore, D. Ziou. 20th International Conference on Pattern Recognition. IEEE, 2366-2369 (2010). DOI: 10.1109/ICPR.2010.579
  20. Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli. IEEE Transactions on Image Processing, 13 (4), 600-612 (2004). DOI: 10.1109/TIP.2003.819861
  21. R.H. Yuhas, A.F.H. Goetz, J.W. Boardman. JPL, Summaries of the Third Annual JPL Airborne Geoscience Workshop, vol. 1: AVIRIS Workshop (1992)
  22. K. Kolikov, G. Krastev, Y. Epitropov, D. Hristozov. Chemometrics and Intelligent Laboratory Systems, 102 (1), 15-19 (2010). DOI: 10.1016/j.chemolab.2010.03.001
  23. G.H. Golub, C.F. Van Loan. Matrix Computations, 4th ed. (Baltimore, MD, 2013). DOI: 10.56021/9781421407944
  24. G. Strang. Introduction to Linear Algebra (Wellesley-Cambridge Press, 2022). DOI: 10.1137/1.9781733146678
  25. K. He, X. Zhang, S. Ren, J. Sun. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 770-778 (2016). DOI: 10.1109/CVPR.2016.90
  26. J.-Y. Zhu, T. Park, P. Isola, A. Efros. Proceedings of the IEEE International Conference on Computer Vision, 2223-2232 (2017). DOI: 10.1109/ICCV.2017.244
  27. Z. Salahuddin, H.C. Woodruff, A. Chatterjee, P. Lambin. Computers in Biology and Medicine, 140, 105111 (2022). DOI: 10.1016/j.compbiomed.2021.105111
  28. M. Yako. Optical Review, 1-13 (2025). DOI: 10.1007/s10043-025-01001-x
  29. M.M. Antony, C.S.S. Sandeep, H.-T. Lim, M.V. Matham. J. Biomedical Photonics \& Engineering, 9 (3), 030304 (2023). DOI: 10.18287/JBPE23.09.030304
  30. I.A. Bratchenko, V.P. Sherendak, O.O. Myakinin, D.N. Artemyev, A.A. Moryatov, E. Borisova, L. Avramov, L.A. Zherdeva, A.E. Orlov, S.V. Kozlov, V.P. Zakharov J. Biomedical Photonics \& Engineering, 4 (1), 5-12 (2018). DOI: 10.18287/JBPE17.04.010301
  31. B.V. Grechkin, V.O. Vinokurov, Y.A. Khristoforova, I.A. Matveeva. J. Biomedical Photonics \& Engineering, 9 (4), 50-55 (2023). DOI: 10.18287/JBPE23.09.040304
  32. I.A. Bratchenko, O.O. Myakinin, V.P. Sherendak, P.N. Volkhin, Y.A. Khristoforova, L.A. Bratchenko, D.N. Artemyev, A.A. Moryatov, O.V. Polschikova, A.S. Machikhin, V.E. Pozhar, S.V. Kozlov, V.P. Zakharov. J. Biomedical Photonics \& Engineering, 4 (4), 040301 (2018). DOI: 10.18287/JBPE18.04.040301
  33. Z. Lai, Y. Fu, J. Zhang. IEEE Transactions on Neural Networks and Learning Systems, 2024. DOI: 10.1109/TNNLS.2023.3340561
Publisher:

Ioffe Institute

Institute Officers:

Director: Sergei V. Ivanov

Contact us:

26 Polytekhnicheskaya, Saint Petersburg 194021, Russian Federation
Fax: +7 (812) 297 1017
Phone: +7 (812) 297 2245
E-mail: post@mail.ioffe.ru