journal article·2025-04-08Open access

Spatial-spectral graph convolutional network for automatic pigment mapping of historical artifacts

Bei Cao, David W. Messinger

npj Heritage Science· Volume 13· Issue 1

Abstract

Hyperspectral image classification is a challenging task due to the lack of ground-truth labels and high dimensionality of spectral bands. For hyperspectral image (HSI) data of cultural heritage artifacts, which are generally man-made objects, we typically have much higher spatial resolution posing extra challenges. Aiming to differentiate subtle spectral differences between varied pigments with little manual effort, we propose a Spatial-Spectral Graph Convolutional Network (SSGCN) as a powerful learning framework for realizing automatic pigment mapping. First, we introduce a novel spatial pattern descriptor “Edge Response Map” to extract structural information of a training ROI, upon which a Spatial-Spectral graph is built by incorporating spatial node connections into the spectral-based graph. Then, we infuse the Spatial-Spectral graph into the graph convolutional network and build a SSGCN learning pipeline. We train the network on the training ROI via semi-supervised classification. By aggregating node features in terms of spectral affinity and spatial context, SSGCN is capable of learning the intrinsic community structures in a non-linear way, and thus can be directly applied to another region for pigment mapping without ground-truth labels required. Furthermore, the graph-based nature of the approach empowers the SSGCN with scalability to learn new pigment classes by flexibly extending the current model. The experiments on multiple ROIs in the Selden Map of China demonstrate the strength and advantages of our method. Compared to an adaptive density-weighted KNN graph, the Spatial-Spectral graph is able to better identify spatial patterns. With only 15% labeled training pixels, the SSGCN achieves mean per-class accuracy above 90%, obtaining automatic pigment mapping on test ROIs with sub-visual spectral differences well discriminated. In terms of classification accuracy and time complexity, the SSGCN greatly outperforms a 3D CNN architecture, HybridSN, and demonstrates much better inductive learning ability on unseen pixels. The proposed graph-based learning pipeline brings a new prospect for processing large-scale HSI classification/clustering in an intelligent manner, and would aid historians in pigment analysis and codicological studies of artifacts with little manual cost.
Source: openalex_inverted_index · Quality: baseline

Research context

OpenAlex topics: Cultural Heritage Materials Analysis · Remote-Sensing Image Classification · Image Retrieval and Classification Techniques

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

References (51)

Show source-provided references
1. Manolakis, D. et al. Hyperspectral image processing for automatic target detection applications. Linc. Lab. J. 14, 79–116 (2003).
2. Song, A. & Kim, Y. Deep learning-based hyperspectral image classification with application to environmental geographic information systems. Korean J. Remote Sens. 33, 1061–1073 (2017).
3. Ishida, T. et al. A novel approach for vegetation classification using uav-based hyperspectral imaging. Comput. Electron. Agric. 144, 80–85 (2018).10.1016/j.compag.2017.11.027 ↗
4. Vali, A., Comai, S. & Matteucci, M. Deep learning for land use and land cover classification based on hyperspectral and multispectral earth observation data: a review. Remote Sens. 12, 2495 (2020).10.3390/rs12152495 ↗
5. Jiang, J. et al. Superpca: A superpixelwise pca approach for unsupervised feature extraction of hyperspectral imagery. IEEE Trans. Geosci. Remote Sens. 56, 4581–4593 (2018).10.1109/TGRS.2018.2828029 ↗
6. Lunga, D., Prasad, S., Crawford, M. M. & Ersoy, O. Manifold-learning-based feature extraction for classification of hyperspectral data: a review of advances in manifold learning. IEEE Signal Process. Mag. 31, 55–66 (2013).10.1109/MSP.2013.2279894 ↗
7. Huang, H., Shi, G., He, H., Duan, Y. & Luo, F. Dimensionality reduction of hyperspectral imagery based on spatial–spectral manifold learning. IEEE Trans. Cybern. 50, 2604–2616 (2019).10.1109/TCYB.2019.2905793 ↗
8. Wang, M., Yu, J., Niu, L. & Sun, W. Unsupervised feature extraction for hyperspectral images using combined low rank representation and locally linear embedding. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1428–1431 (IEEE, 2017).10.1109/ICASSP.2017.7952392 ↗
9. Hou, B., Zhang, X., Ye, Q. & Zheng, Y. A novel method for hyperspectral image classification based on laplacian eigenmap pixels distribution-flow. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 6, 1602–1618 (2013).10.1109/JSTARS.2013.2259470 ↗
10. Cahill, N. D., Czaja, W. & Messinger, D. W. Schroedinger eigenmaps with nondiagonal potentials for spatial-spectral clustering of hyperspectral imagery. In Algorithms and technologies for multispectral, hyperspectral, and ultraspectral imagery XX, vol. 9088, 27–39 (SPIE, 2014).10.1117/12.2050651 ↗
11. Sun, W., Yang, G., Du, B., Zhang, L. & Zhang, L. A sparse and low-rank near-isometric linear embedding method for feature extraction in hyperspectral imagery classification. IEEE Trans. Geosci. Remote Sens. 55, 4032–4046 (2017).10.1109/TGRS.2017.2686842 ↗
12. Shao, Z., Zhang, L., Zhou, X. & Ding, L. A novel hierarchical semisupervised svm for classification of hyperspectral images. IEEE Geosci. Remote Sens. Lett. 11, 1609–1613 (2014).10.1109/LGRS.2014.2302034 ↗
13. Cao, F. et al. Extreme sparse multinomial logistic regression: A fast and robust framework for hyperspectral image classification. Remote Sens. 9, 1255 (2017).10.3390/rs9121255 ↗
14. Peng, J., Li, L. & Tang, Y. Y. Maximum likelihood estimation-based joint sparse representation for the classification of hyperspectral remote sensing images. IEEE Trans. neural Netw. Learn. Syst. 30, 1790–1802 (2018).10.1109/TNNLS.2018.2874432 ↗
15. Batchelor, R. The selden map rediscovered: a chinese map of east asian shipping routes, c. 1619. Imago Mundi 65, 37–63 (2013).10.1080/03085694.2013.731203 ↗
16. Nie, H. A. The selden map of china: a new understanding of the ming dynasty (Bodleian Library, University of Oxford, 2019).
17. Bai, D., Messinger, D. W. & Howell, D. Hyperspectral analysis of cultural heritage artifacts: pigment material diversity in the gough map of britain. Opt. Eng. 56, 081805 (2017).10.1117/1.OE.56.8.081805 ↗
18. Kogou, S. et al. The origins of the selden map of china: scientific analysis of the painting materials and techniques using a holistic approach. Herit. Sci. 4, 28 (2016).10.1186/s40494-016-0098-x ↗
19. Bai, D., Messinger, D. W. & Howell, D. Pigment diversity estimation for hyperspectral images of the selden map of china. In Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXIV, vol. 10644, 1064415 (International Society for Optics and Photonics, 2018).10.1117/12.2304041 ↗
20. Messinger, D. W., Ziemann, A. K., Schlamm, A. & Basener, W. Metrics of spectral image complexity with application to large area search. Opt. Eng. 51, 036201 (2012).10.1117/1.OE.51.3.036201 ↗
21. Bai, D., Messinger, D. W. & Howell, D. Deep learning spatial-spectral processing of hyperspectral images for pigment mapping of cultural heritage artifacts. In International Conference on Pattern Recognition, 200–214 (Springer, 2021).10.1007/978-3-030-68787-8_14 ↗
22. Delano-Smith, C. et al. New light on the medieval gough map of britain. Imago Mundi 69, 1–36 (2017).10.1080/03085694.2017.1242838 ↗
23. Fontanella, F., Colace, F., Molinara, M., Di Freca, A. S. & Stanco, F. Pattern recognition and artificial intelligence techniques for cultural heritage (2020).10.1016/j.patrec.2020.06.018 ↗
24. Llamas, J., M. Lerones, P., Medina, R., Zalama, E. & Gómez-García-Bermejo, J. Classification of architectural heritage images using deep learning techniques. Appl. Sci. 7, 992 (2017).10.3390/app7100992 ↗
25. Janković, R. Machine learning models for cultural heritage image classification: Comparison based on attribute selection. Information 11, 12 (2019).10.3390/info11010012 ↗
26. Navarro, P. et al. Learning feature representation of iberian ceramics with automatic classification models. J. Cult. Herit. 48, 65–73 (2021).10.1016/j.culher.2021.01.003 ↗
27. Grilli, E. & Remondino, F. Classification of 3d digital heritage. Remote Sens. 11, 847 (2019).10.3390/rs11070847 ↗
28. Mehta, S., Kukreja, V. & Bordoloi, D. Heritage coin identification using convolutional neural networks: A multi-classification approach for numismatic research. In 2023 Second International Conference on Augmented Intelligence and Sustainable Systems (ICAISS), 1–6 (IEEE, 2023).10.1109/ICAISS58487.2023.10250481 ↗
29. Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M. & Monfardini, G. The graph neural network model. IEEE Trans. neural Netw. 20, 61–80 (2008).10.1109/TNN.2008.2005605 ↗
30. Wu, Z. et al. A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst. 32, 4–24 (2020).10.1109/TNNLS.2020.2978386 ↗
31. Asif, N. A. et al. Graph neural network: A comprehensive review on non-euclidean space. IEEE Access 9, 60588–60606 (2021).10.1109/ACCESS.2021.3071274 ↗
32. Defferrard, M., Bresson, X. & Vandergheynst, P. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems 29 (2016).
33. Kipf, T. N. & Welling, M. Semi-supervised classification with graph convolutional networks. 2017. ArXiv abs/1609.02907 (2017).
34. Gao, H., Wang, Z. & Ji, S. Large-scale learnable graph convolutional networks. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, 1416–1424 (2018).10.1145/3219819.3219947 ↗
35. Khan, N., Chaudhuri, U., Banerjee, B. & Chaudhuri, S. Graph convolutional network for multi-label vhr remote sensing scene recognition. Neurocomputing 357, 36–46 (2019).10.1016/j.neucom.2019.05.024 ↗
36. Fout, A., Byrd, J., Shariat, B. & Ben-Hur, A.Protein interface prediction using graph convolutional networks. Advances in neural information processing systems 30 (2017).
37. Sun, M. et al. Graph convolutional networks for computational drug development and discovery. Brief. Bioinforma. 21, 919–935 (2020).10.1093/bib/bbz042 ↗
38. He, L., Li, J., Liu, C. & Li, S. Recent advances on spectral–spatial hyperspectral image classification: An overview and new guidelines. IEEE Trans. Geosci. Remote Sens. 56, 1579–1597 (2017).10.1109/TGRS.2017.2765364 ↗
39. Xia, J., Dalla Mura, M., Chanussot, J., Du, P. & He, X. Random subspace ensembles for hyperspectral image classification with extended morphological attribute profiles. IEEE Trans. Geosci. Remote Sens. 53, 4768–4786 (2015).10.1109/TGRS.2015.2409195 ↗
40. Kang, X., Li, S. & Benediktsson, J. A. Spectral–spatial hyperspectral image classification with edge-preserving filtering. IEEE Trans. Geosci. remote Sens. 52, 2666–2677 (2013).10.1109/TGRS.2013.2264508 ↗
41. Gu, Y., Liu, T., Jia, X., Benediktsson, J. A. & Chanussot, J. Nonlinear multiple kernel learning with multiple-structure-element extended morphological profiles for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 54, 3235–3247 (2016).10.1109/TGRS.2015.2514161 ↗
42. Qin, A. et al. Spectral–spatial graph convolutional networks for semisupervised hyperspectral image classification. IEEE Geosci. Remote Sens. Lett. 16, 241–245 (2018).10.1109/LGRS.2018.2869563 ↗
43. Wan, S., Gong, C., Pan, S., Yang, J. & Yang, J. Multi-level graph convolutional network with automatic graph learning for hyperspectral image classification. arXiv preprint arXiv:2009.09196 (2020).
44. Cao, B. G. & Messinger, D. W. Automatic clustering of inks in cultural heritage artifacts via optimal selection of graph modularity. In Algorithms, Technologies, and Applications for Multispectral and Hyperspectral Imaging XXVII, vol. 11727, 117270Q (International Society for Optics and Photonics, 2021).10.1117/12.2585726 ↗
45. Mercovich, R. A., Albano, J. & Messinger, D. Techniques for the graph representation of spectral imagery. In 2011 3rd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), 1–4 (IEEE, 2011).10.1109/WHISPERS.2011.6080912 ↗
46. Bruna, J., Zaremba, W., Szlam, A. & LeCun, Y. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203 (2013).
47. Henaff, M., Bruna, J. & LeCun, Y. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163 (2015).
48. Stevens, J. R., Resmini, R. G. & Messinger, D. W. Spectral-density-based graph construction techniques for hyperspectral image analysis. IEEE Trans. Geosci. Remote Sens. 55, 5966–5983 (2017).10.1109/TGRS.2017.2718547 ↗
49. Cao, B. G. & Messinger, D. W. Graph convolutional network for automatic pigment clustering of cultural heritage artifacts. In Algorithms, Technologies, and Applications for Multispectral and Hyperspectral Imaging XXVIII, vol. 12094, 306–322 (SPIE, 2022).10.1117/12.2618456 ↗
50. Mulholland, R., Howell, D., Beeby, A., Nicholson, C. & Domoney, K. Identifying eighteenth century pigments at the bodleian library using in situ raman spectroscopy, xrf and hyperspectral imaging. Herit. Sci. 5, 1–19 (2017).10.1186/s40494-017-0157-y ↗
51. Roy, S. K., Krishna, G., Dubey, S. R. & Chaudhuri, B. B. Hybridsn: Exploring 3-d–2-d cnn feature hierarchy for hyperspectral image classification. IEEE Geosci. Remote Sens. Lett. 17, 277–281 (2019).10.1109/LGRS.2019.2918719 ↗