The problem of automatic classification of pictures using an intelligent decision-making system based on the knowledge graph and fine-grained image analysis
DOI:
https://doi.org/10.20535/SRIT.2308-8893.2022.4.05Keywords:
automatic multi-task classification, knowledge graph, attention mechanism, fine-grained image analysis, museum expertise, paintings, convolutional neural networksAbstract
In order to prevent the illegal export of paintings abroad, a museum examination using various methods for studying a work of art is carried out. At the same time, an analysis is also made of historical, art history, financial and other information and documents confirming the painting’s authenticity — provenance. Automation of such examination is hampered by the need to take into account numerical values of visual features, quality indicators, and verbal descriptions from provenance. In this paper, we consider the problem of automatic multi-task classification of paintings for museum expertise. A system architecture is proposed that checks provenance, implements a fine-grained image analysis (FGIA) of visual image features, and automatically classifies a painting by authorship, genre, and time of creation. Provenance is contained in a knowledge graph; for its vectorization, it is proposed to use a graph2vec type encoder with an attention mechanism. Fine-grained image analysis is proposed to be performed using searching discriminative regions (SDR) and learning discriminative regions (LDR) allocated by convolutional neural networks. To train the classifier, a generalized loss function is proposed. A data set is also proposed, including provenance and images of paintings by European and Ukrainian artists.
References
“Constitution of Ukraine dated June 28, 1996 No. 254k/96-BP,” Bulletin of the Verkhovna Rada of Ukraine, 1996.
“Law of Ukraine “Fundamentals of the Legislation of Ukraine on Culture” dated February 14, 1992 N 2117-XII,” Bulletin of the Verkhovna Rada of Ukraine, 1992.
“Law of Ukraine “On export, import and return of cultural values” of September 21, 1999 N 1068-XIV,” Bulletin of the Verkhovna Rada of Ukraine, 1999.
Resolution of the CMU of June 20, 2000 No. 983 “On the approval of the Regulation on the State Service of Control over the Movement of Cultural Values across the State Border of Ukraine”.
Letter of the State Customs Service dated 30.12.99 N 09/1619 “Regarding customs control over the movement of cultural values”.
Order of the Ministry of Culture and Tourism of Ukraine dated 11.15.2002 N 647 “On Approval of the List of State Institutions, Cultural Institutions, and Other Organizations Granted the Right to Conduct State Expertise of Cultural Values”.
A. Martynenko, V. Moroz, and I. Nulina, “An intelligent decision support system for cultural property identification,” Computer-IntegratedTechnologies: Education, Science, Production, 39, pp. 78–82, 2020. Available: http://cit-journal.com.ua/index.php/cit/article/view/126
A.A. Martynenko, A.D. Tevyashev, N.Ye. Kulishova, and B.I. Moroz, "System analysis of the problem of establishing the authenticity and authority of painting works,” System Research and Information Technologies, no. 1, pp. 1–16, 2022.
A.A. Martynenko, A.D. Tevyashev, N.Ye. Kulishova, B.I. Moroz, and A.S. Sergienko, “Automatic classification of paintings by year of creation,” Radio Electronics, Computer Science, Control, no. 2(61), pp. 80–89, 2022.
D. Pancaroglu, Artist, style and year classification using face recognition and clustering with convolutional neural networks; Eds.: D.C. Wyld et al., COMIT, SIPO, AISCA, MLIOB, BDHI, 2020, CS & IT, CSCP 2020, pp. 41–54. doi: 10.5121/csit.2020.101604.
N. Banar, M. Sabatelli, P.Geurts, W. Daelemans, and M. Kestemont, “Transfer Learning with Style Transfer between the Photorealistic and Artistic Domain,” Electronic Imaging, Computer Vision and Image Analysis of Art, pp. 41-1–41-9, 2021. Available: doi.org/10.2352/ISSN.2470-1173.2021.14.CVAA-041
E. Cetinic, T. Lipic, and S. Grgic, “Fine-tuning Convolutional Neural Networks for fine art classification,” Expert Systems with Applications, vol. 114, pp. 107–118, 2018. Available: doi.org/10.1016/j.eswa.2018.07.026
T. Chen and J. Yang, “A Novel Multi-Feature Fusion Method in Merging Information of Heterogenous-View Data for Oil Painting Image Feature Extraction and Recognition,” Front. Neurorobot., 12, 2021. Available: doi.org/10.3389/fnbot.2021.709043
W. Zhao, D. Zhou, X. Qiu, W. Jiang, “Compare the performance of the models in art classification,” PLoS ONE, 16(3), 2021. Available: doi.org/10.1371/journal.pone.0248414
Data Analytics for Cultural Heritage: Current Trends and Concepts; Eds.: A. Belhi, A. Bouras, A. Kh. Al-Ali, A.H. Sadka. Springer, 2021, 279 p.
M. Fiorucci, M. Khoroshiltseva, M. Pontil, A. Traviglia, A. Del Bue, and S. James, “Machine learning for cultural heritage: A survey”, Pattern Recognition Letters, vol. 133, pp.102–108, 2020.
M. Kelek, N. Calik, and T. Yildirim, “Painter Classification Over the Novel Art Painting Data Set via The Latest Deep Neural Networks,” Procedia Computer Science, 154, pp. 369–376, 2019. doi: 10.1016/j.procs.2019.06.053.
M. Sabatelli, M. Kestemont, W. Daelemans, P. Geurts, “Deep transfer learning for art classification problems,” Proceedings of the European Conference on Computer Vision (ECCV), 2018. Available: https://doi.org/10.1007/978-3-030-11012-3_48
G. Castellano, G. Sansaro, and G. Vessio, “Integrating Contextual Knowledge to Visual Features for Fine Art Classification,” arXiv:2105.15028v2 [cs.CV], 28 Sep 2021.
N. Garcia, B. Renoust, and Y. Nakashima, “ContextNet: Representation and exploration for painting classification and retrieval in context,” International Journal of Multimedia Information Retrieval, 9(1), pp.17–30, 2020.
X.-S. Wei, J.-H. Luo, J. Wu, and Z.-H. Zhou, “Selective convolutional descriptor aggregation for fine-grained image retrieval,” IEEE Trans. Image Process., vol. 26, no. 6, pp. 2868–2881, 2017.
X. Zeng, Y. Zhang, X. Wang, K. Chen, D. Li, and W. Yang, “Fine-grained image retrieval via piecewise cross entropy loss,” Image and Vision Computing, vol. 93, p.103820, 2020.
Open Access Images - National Gallery of Art. Available: https://www.nga.gov/open-access-images.html
O.L. Kalashnikova, “Legal consequences of imperfect description and accounting of museum collections. International law and the law of the European Union,” Actual problems of domestic jurisprudence, no. 2, vol. 2, 2018. Available: http://apnl.dnu.in.ua/2_2018/tom_2/32.pdf
A. Grover and J.Leskovec, “node2vec: Scalable Feature Learning for Networks,” ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2016.
Deep Residual Networks (ResNet, ResNet50) – 2022 Guide. Available: https://viso.ai/deep-learning/resnet-residual-neural-network/
RESNET50. Available: https://pytorch.org/vision/main/models/generated/torchvision.models.resnet50.html#resnet50
K. Sun and J. Zhu, “Searching and Learning Discriminative Regions for Fine-Grained Image Retrieval and Classification,” IEICE TRANS. INF. & SYST., vol. e105–d, no.1, pp. 141–147, 2022.
Best artworks of all time. Available: https://www.kaggle.com/ikarus777/best-artworks-of-all-time/tasks
National Art Museum of Ukraine. Available: https://namumuseum.business.site/
