journal article·2025-02-22Open access

Image-driven batik product knowledge graph construction

Xingjie Wu, Qingni Yuan, Pengju Qu, Man Su

npj Heritage Science· Volume 13· Issue 1

Abstract

A text-based knowledge graph is limited to providing a basic textual introduction and retrieval of intangible cultural heritage, which hinders the in-depth exploration of the intricate artistic connotations of batik products. This study focuses on batik products and employs an image-driven approach to construct a knowledge graph. Initially, we conduct image data collection and processing for batik products, establishing an ontology specific to the domain of batik using an image-driven approach that defines classes and relationships within the domain. Subsequently, we construct a knowledge graph for batik products by identifying entities and relationships. We employ a method to obtain representative pattern entities through image clustering, where patterns of the same type are clustered again to reveal evolutionary relationships based on clustering results. Subsequently, the DySnake convolution module is integrated into the backbone network of the object detection model ResNet. Moreover, a channel gate control mechanism based on low-rank tensor fusion is proposed to establish an automatic extraction object detection model for batik patterns and facilitate knowledge graph automatic updates. Finally, we visualize the knowledge graph using Gephi and Neo4j. Through comparative experiments and ablation experiments on our self-built batik dataset with multiple object detection models, we validate the effectiveness of our methods.
Source: openalex_inverted_index · Quality: baseline

Research context

OpenAlex topics: Edcuational Technology Systems · Blockchain Technology in Education and Learning · Image Retrieval and Classification Techniques

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

References (59)

Show source-provided references
1. Mardani, D. A. & Pranowo, S. A. J. Deep learning for recognition of Javanese batik patterns. AIP Conf. Proc. 2217 (2020). https://doi.org/10.1063/5.0000686.10.1063/5.0000686 ↗
2. Agastya, I. M. A. & Setyanto, A. Classification of Indonesian Batik using deep learning techniques and data augmentation. In 3rd International Conference on Information Technology, Information System and Electrical Engineering (ICITISEE), Yogyakarta, Indonesia, Vol. 13, 27–31 (IEEE, 2018).10.1109/ICITISEE.2018.8720990 ↗
3. Chen, Z., Ren, X. & Zhang, Z. Cultural heritage as rural economic development: Batik production amongst China’s Miao population. J. Rural Stud. 81, 182–93, https://doi.org/10.1016/j.jrurstud.2020.10.024 (2021).10.1016/j.jrurstud.2020.10.024 ↗
4. Luo, W. Analysis of artistic modeling of opera stage clothing based on big data clustering algorithm. Secur. Commun. Netw. 2021, 1–9 (2021).
5. Ji, S., Pan, S., Cambria, E., Marttinen, P. & Yu, P. S. A survey on knowledge graphs: representation, acquisition, and applications. IEEE Trans. Neural Netw. Learn. Syst. 33, 494–514 (2022).10.1109/TNNLS.2021.3070843 ↗
6. Dou, J., Qin, J., Jin, Z. & Li, Z. Knowledge graph based on domain ontology and natural language processing technology for Chinese intangible cultural heritage. J. Vis. Lang. Comput. 48, 19–28 (2018).10.1016/j.jvlc.2018.06.005 ↗
7. Lu, L., Liang, X., Yuan, G., Jing, L., Wei, C. & Cheng, C. A study on the construction of knowledge graph of Yunjin video resources under productive conservation. Herit. Sci. 11, 83 (2023).10.1186/s40494-023-00932-5 ↗
8. Fan, T., Wang, H. & Hodel, T. Multimodal knowledge graph construction of Chinese traditional operas and sentiment and genre recognition. J. Cult. Herit. 62, 32–44 (2023).10.1016/j.culher.2023.05.003 ↗
9. Fan, T., Wang, H. & Hodel, T. CICHMKG: a large-scale and comprehensive Chinese intangible cultural heritage multimodal knowledge graph. Herit. Sci. 11, 115 (2023).10.1186/s40494-023-00927-2 ↗
10. Wang, M., Wang, H., Qi, G. & Zheng, Q. Richpedia: a large-scale, comprehensive multi-modal knowledge graph. Big Data Res. 22, 100159 (2020).10.1016/j.bdr.2020.100159 ↗
11. Deng, Y., Tang, F., Dong, W., Wu, F., Deussen, O. & Xu, C. Selective clustering for representative paintings selection. Multimed. Tools Appl. 78, 19305–23 (2019).10.1007/s11042-019-7271-7 ↗
12. Han, S., Ren, F., Du, Q. & Gui, D. Extracting representative images of tourist attractions from flickr by combining an improved cluster method and multiple deep learning models. Int. J. Geo-Inf. 9, 81 (2020).10.3390/ijgi9020081 ↗
13. Zhao, Y. et al. Detrs beat YOLOs on real-time object detection. Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, 16965–16974 (IEEE COMP SOC., 2024).10.1109/CVPR52733.2024.01605 ↗
14. Qi, Y., He, Y., Qi, X., Zhang, Y. & Yang, G. Dynamic snake convolution based on topological geometric constraints for tubular structure segmentation. In Proc. IEEE/CVF International Conference on Computer Vision, 6047–6056 (IEEE, Los Alamitos, Paris, France, 2023).10.1109/ICCV51070.2023.00558 ↗
15. Wei, T., Roche, C., Papadopoulou, M. & Jia, Y. An ontology of Chinese ceramic vases. In 12th International Conference on Knowledge Engineering and Ontology Development (eds Aveiro, D., Dietz, J. & Filipe, J.), 53–63 (Scitepress, Setubal, Budapest Hungary, 2020).10.5220/0010110600530063 ↗
16. Kim, H., Matuszka, T., Kim, J.-I., Kim, J. & Woo, W. Ontology-based mobile augmented reality in cultural heritage sites: information modeling and user study. Multimed. Tools Appl. 76, 26001–29 (2017).10.1007/s11042-017-4868-6 ↗
17. Ferrada, S., Bustos, B. & Hogan, A. IMGpedia: a linked dataset with content-based analysis of wikimedia images. In 16th International Semantic Web Conference (ISWC) (eds DAmato, C. et al.), 84–93 (Springer, Berlin, Vienna, Austria, 2017).10.1007/978-3-319-68204-4_8 ↗
18. Li, N., Shen, Q., Song, R., Chi, Y. & Xu, H. MEduKG: a deep-Learning-based approach for multi-modal educational knowledge graph construction. Information 13, 91 (2022).10.3390/info13020091 ↗
19. Min, W., Liu, C., Xu, L. & Jiang, S. Applications of knowledge graphs for food science and industry. Patterns 3, 100484 (2022).10.1016/j.patter.2022.100484 ↗
20. Wilcke, W. X., Bloem, P., de Boer, V., van t Veer, R. H. & van Harmelen, F. A. H. End-to-end entity classification on multimodal knowledge graphs. arXiv preprint https://doi.org/10.48550/arXiv.2003.12383 (2020).10.48550/arXiv.2003.12383 ↗
21. Schleider, T. et al. Searching silk fabrics by images leveraging on knowledge graph and domain expert rules. In Proc. 3rd Workshop on Structuring and Understanding of Multimedia Heritage Contents 41–49 (Association for Computing Machinery, New York, NY, United States 2021).10.1145/3475720.3484445 ↗
22. Jiang, S., Qian, X., Xue, Y., Li, F. & Hou, X. Generating representative images for landmark by discovering high frequency shooting locations from community-contributed photos. In 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), San Jose, CA (IEEE, New York, 2013).
23. Yan, Y., Hao, H., Xu, B., Zhao, J. & Shen, F. Image clustering via deep embedded dimensionality reduction and probability-based triplet loss. IEEE Trans. Image Process. 29, 5652–61 (2020).10.1109/TIP.2020.2984360 ↗
24. Castellano, G. & Vessio, G. A deep learning approach to clustering visual arts. Int. J. Comput. Vis. 130, 2590–605 (2022).10.1007/s11263-022-01664-y ↗
25. Zhang, F., Li, L., Hua, Q., Dong, C.-R. & Lim, B.-H. Improved deep clustering model based on semantic consistency for image clustering. Knowl.-Based Syst. 253, 109507 (2022).10.1016/j.knosys.2022.109507 ↗
26. Girshick, R., Donahue, J., Darrell, T. & Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA 580–587 (IEEE, New York, 2014).10.1109/CVPR.2014.81 ↗
27. Girshick, R. Fast R-CNN. In 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile 1440–1448 (IEEE, New York, 2015).10.1109/ICCV.2015.169 ↗
28. Ren, S., He, K., Girshick, R. & Sun, J. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39, 1137–49 (2017).10.1109/TPAMI.2016.2577031 ↗
29. Redmon, J., Divvala, S., Girshick, R. & Farhadi, A. You only look once: unified, real-time object detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA 779–788 (IEEE, New York, 2016).10.1109/CVPR.2016.91 ↗
30. Liu, W. et al. SSD: single shot multibox detector. In 14th European Conference on Computer Vision (ECCV) (eds Leibe, B., Matas, J., Sebe, N. & Welling, M.), 21–37 (Springer, Berlin, Amsterdam, Netherlands 2016).10.1007/978-3-319-46448-0_2 ↗
31. Dosovitskiy, A. et al. An Image is worth 16×16 words: transformers for image recognition at scale. arXiv preprint https://doi.org/10.48550/arXiv.2010.11929 (2020).10.48550/arXiv.2010.11929 ↗
32. Carion, N. et al. End-to-end object detection with transformers. arXiv preprint https://doi.org/10.48550/arXiv.2005.12872.10.48550/arXiv.2005.12872 ↗
33. Meyer, L. et al. Algorithmic ways of seeing: using object detection to facilitate art exploration. arXiv preprint https://doi.org/10.48550/arXiv.2403.19174 (2024).10.48550/arXiv.2403.19174 ↗
34. Ahmad, T. & Schich, M. Toward cross‐domain object detection in artwork images using improved YoloV5 and XGBoosting. IET Image Process. 17, 2437–2349 (2023).10.1049/ipr2.12806 ↗
35. Kadish, D., Risi, S. & Løvlie, A. Improving object detection in art Images using only style transfer. In International Joint Conference on Neural Networks (IJCNN), Electr Network (IEEE, New York, 2021).10.1109/IJCNN52387.2021.9534264 ↗
36. Gonthier, N., Gousseau, Y., Ladjal, S. & Bonfait, O. Weakly supervised object detection in artworks. arXiv preprint https://doi.org/10.48550/arXiv.1810.02569 (2019).10.48550/arXiv.1810.02569 ↗
37. Ibrahim, B. I. E., Eyharabide, V., Le Page, V. & Billiet, F. Few-shot object detection: application to medieval musicological studies. J. Imaging 8, 18 (2022).10.3390/jimaging8020018 ↗
38. Sindel, A., Maier, A. & Christlein, V. Artfacepoints: high-resolution facial landmark detection in paintings and prints. arXiv preprint https://doi.org/10.48550/arXiv.2210.09204 (2022).10.48550/arXiv.2210.09204 ↗
39. Bengamra, S., Mzoughi, O., Bigand, A. & Zagrouba, E. A comprehensive survey on object detection in visual art: taxonomy and challenge. Multimed. Tools Appl. 83, 14637–70 (2023).10.1007/s11042-023-15968-9 ↗
40. Wang, M. Study on the Art of Miao Batik Patterns https://kns.cnki.net/kcms2/article/abstract?v=lQz6UQjnwp-Q5tn40S34r4uv9l7hMEWxevf6vxXVANdNM4NWWBtcDE702QIh_tq2kMxj3oAEY5-34PSeV_VSLhoJxjfgs7J8ineowtCnak_Ubs6e7OqulPL2MB1mdWtO3mQiMvvQi9vBavwPVPe3wA9I5ElfSIO_&uniplatform=NZKPT&language=CHS (2016).
41. Wang, T. Study on the art of Miao Batik https://kns.cnki.net/kcms2/article/abstract?v=lQz6UQjnwp-k5IcNyBJtNuLssN5mjuU8xnIdXYyhoyPvgsR6rVhyBqb_gve_dCUxJWViYsglGW0oeqtiMkAfxa2KpXbTdtc9Iy2QU_lC30sruJlAEADK-4DITfSN2O1PyzSCri_4Qq0LdeMQfMOZcWX7z3LWqSZ1&uniplatform=NZKPT&language=CHS (2014).
42. Yang, X. Research on the Application of Miao Batik Art Elements in My Creative Practice https://doi.org/10.27667/d.cnki.gzymu.2023.000085 (2023).10.27667/d.cnki.gzymu.2023.000085 ↗
43. Xu, B. Modern Design and Application of Visual Language of Batik of Miao Nationality in Danzhai https://doi.org/10.27391/d.cnki.gxagu.2021.000490 (2021).10.27391/d.cnki.gxagu.2021.000490 ↗
44. Frey, B. J. & Dueck, D. Clustering by passing messages between data points. Science 315, 972–6 (2007).10.1126/science.1136800 ↗
45. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, 770–778 (IEEE, New York, 2016).10.1109/CVPR.2016.90 ↗
46. Liu, Z. et al. Efficient low-rank multimodal fusion with modality-specific factors. In Proc. 56th Annual Meeting of the Association for Computational Linguistics Vol. 1: Long Papers (eds Gurevych, I. & Miyao, Y.), 2247–2256 (Association of Computational Linguistics-ACL, Stroudsburg, Melbourne, Australia, 2018).10.18653/v1/P18-1209 ↗
47. Zadeh, A., Chen, M., Poria, S., Cambria, E. & Morency, L.-P. Tensor fusion network for multimodal sentiment analysis. arXiv preprint https://doi.org/10.48550/arXiv.1707.07250 (2017).10.48550/arXiv.1707.07250 ↗
48. Sun, P. et al. Sparse R-CNN: end-to-end object detection with learnable proposals. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Electr Network 14449–14458 (IEEE COMP SOC., Los Alamitos, 2021).10.1109/CVPR46437.2021.01422 ↗
49. Feng, C., Zhong, Y., Gao, Y., Scott, M. & Huang, W. TOOD: task-aligned one-stage object detection. In 18th IEEE/CVF International Conference on Computer Vision (ICCV), Electr Network 3490–3499 (IEEE, New York, 2021).10.1109/ICCV48922.2021.00349 ↗
50. Zhang, H. et al. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. arXiv preprint https://doi.org/10.48550/arXiv.2203.03605 (2022).10.48550/arXiv.2203.03605 ↗
51. Meng, D. et al. Conditional DETR for fast training convergence. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Electr Network 3631–3640 (IEEE, New York, 2021).10.1109/ICCV48922.2021.00363 ↗
52. Kang, M., Ting, C.-M., Ting, F. & Phan, Re-W. ASF-YOLO: a novel YOLO model with attentional scale sequence fusion for cell instance segmentation. Image Vis. Comput. 147, 105057 (2024).10.1016/j.imavis.2024.105057 ↗
53. Wang, C. et al. Gold-YOLO: efficient object detector via gather-and-distribute mechanism. arXiv preprint https://doi.org/10.48550/arXiv.2309.11331 (2023)10.48550/arXiv.2309.11331 ↗
54. Chen, Y. et al. Accurate leukocyte detection based on deformable-DETR and multi-level feature fusion for aiding diagnosis of blood diseases. Comput. Biol. Med. 170, 107917 (2024).10.1016/j.compbiomed.2024.107917 ↗
55. Lin, T. Y., Maire, M., Belongie, S. & Hays, J. Zitnick CLJSIP. Microsoft COCO: common objects in context. In 13th European Conference on Computer Vision (ECCV) (eds Fleet, D., Pajdla, T., Schiele, B. & Tuytelaars, T.), 740–755 (Springer, Berlin, Zurich, Switzerland, 2014).10.1007/978-3-319-10602-1_48 ↗
56. Bolya, D., Foley, S., Hays, J. & Hoffman, J. TIDE: a general toolbox for identifying object detection errors. arXiv preprint https://doi.org/10.48550/arXiv.2008.08115 (2020).10.48550/arXiv.2008.08115 ↗
57. Chen, K. et al. MMDetection: open MMLab detection toolbox and benchmark. arXiv preprint https://doi.org/10.48550/arXiv.1906.07155 (2019).10.48550/arXiv.1906.07155 ↗
58. Liu, J., Zheng, J., Liu, Y. & Li, H. GLID: Pre-training a Generalist Encoder-Decoder Vision Model. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22851–22860 (IEEE, COMPUTER SOC, Seattle, WA, CA, USA, 2024).10.1109/CVPR52733.2024.02156 ↗
59. Cheng, T., Song, L., Ge, Y., Liu, W., Wang, X. & Shan, Y. YOLO-world: real-time open-vocabulary object detection. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16901–16911 (IEEE, COMPUTER SOC, Seattle, WA, CA, USA, 2024).10.1109/CVPR52733.2024.01599 ↗