Using machine learning to predict artistic styles: an analysis of trends and the research agenda

Machine learning models designed to forecast artistic styles in paintings leverage advanced algorithms capable of acquiring highly abstract and hierarchical representations from extensive datasets encompassing both historical and contemporary artistic images. These models prominently utilize deep neural network architectures, including convolutional neural networks (CNNs) and generative adversarial networks (GANs), to discern significant visual features within paintings, unveiling distinct stylistic patterns. By embracing machine learning techniques, these models effectively surmount the constraints of traditional methods, yielding results that are both more precise and reliable (Mao 2022; Wenjing and Cai 2023).

Prior research has underscored the efficacy of these models across diverse domains within the social and human sciences, as well as the arts, particularly in the assessment of artistic designs through artificial intelligence and machine learning methodologies (Wenjing and Cai 2023). Noteworthy applications include the utilization of machine learning algorithms to identify and visualize color pigments in cultural heritage artworks (Chen et al. 2022). Additionally, studies have delved into the application of artificial intelligence for safeguarding Dunhuang cultural heritage, involving the creation of a comprehensive dataset in this domain (Yu et al. 2022). Other endeavors have explored the development of a catalog raisonné using artificial intelligence and machine learning (Dobbs et al. 2022).

The integration of machine learning models into the prediction of artistic styles in paintings has emerged as a topic of paramount significance, situated within the interdisciplinary realms of computer vision and artificial intelligence as applied to the analysis of visual art. Within this broader context, the significance of these models in intelligently analyzing art becomes apparent, suggesting that the incorporation of virtual reality and machine learning technologies holds the potential to substantially enhance educational processes within the artistic domain (Mao 2022). Furthermore, research underscores the utility of machine learning models in predicting the evolution of damage in panel paintings, offering a valuable contribution to the preservation of artistic heritage (Califano et al. 2022).

In a broader context, Lürig et al. (2021) underscore the potential of computer vision and machine learning across disciplines like ecology and evolutionary biology, where predictive models have propelled significant advances in phenomics. They posit that applying machine learning models to the analysis of artistic styles in paintings holds the promise of deepening our understanding of art history, the influence of masters on their students, and the evolution of trends over time. This aligns with the work of Sanhudo et al. (2021), who utilized machine learning techniques to classify intricate work activities in the construction sector.

Research on the utilization of machine learning models for predicting artistic styles in paintings reveals certain gaps, warranting bibliometric analysis to address and scrutinize the current state of scientific literature in this domain. A primary gap lies in the necessity to explore and advance machine learning techniques capable of capturing the intricate complexity and diversity inherent in both historical and contemporary artistic styles found in paintings. While existing approaches exist, many tend to focus on specific methodologies or the classification of fragments rather than the comprehensive prediction of the artistic style of an entire work, as noted by Cascone et al. (2023).

Furthermore, there is a need for additional research on the interpretation and explanation of features learned by machine learning models employed for this task. This would foster a deeper understanding of how these models make decisions, ultimately enhancing their reliability and applicability in practical contexts (Li and Zhang 2022). Similarly, despite considerable progress in applying artificial intelligence and machine learning across various fields, such as optimizing industrial operations (Allahloh et al. 2023) and art design (Xu and Nazir 2022), it is imperative to specifically focus on analyzing techniques and approaches applied to predicting artistic styles in paintings. This targeted exploration aims to identify the most promising trends and approaches in this emerging field.

The development of a review study, according to the scientific literature, will allow the compilation, analysis and synthesis of information from different scientific sources, which will provide a holistic vision of the current state of research in deep learning models for predicting artistic styles in paintings. By identifying the most outstanding advances, limitations, and approaches, this literature review will contribute to the development and improvement of new approaches in the field and promote future research in critical areas that require further attention and development (Allahloh et al. 2023; Cascone et al. 2023; Xu and Nazir 2022). In this sense, the aim of this research is to examine the research trends in the use of machine learning for artistic style prediction, with the aim of providing a research agenda for future research. In addition, the following research questions are posed and its motivation:

Initially, among the works developed between the 2000 and 2010 s that related digital technology to art were research studies focused on assessing the aesthetic visual quality of a specific type of visual medium, specifically digital images of paintings. For example, Li and Chen (2009) tackled this challenge as a machine learning problem, aiming to evaluate the aesthetic quality of paintings based on their visual content. They designed a set of methods to extract features representing both global and local char-acteristics of a painting. In another study by Tzeng et al. (2005), the authors introduced a new approach to the volume classification problem that combined machine learning and a painting metaphor to allow more sophisticated classification in an intuitive manner. They trained a system to classify the entire painted volume and represent visual information immediately as the painting progressed in a much larger dimensional space without explicitly specifying the mapping for each dimension used.

One of the significant contributions to research on Machine Learning in painting was a review work where the authors examined whether machine learning and image analysis tools could be used to assist art experts in authenticating unknown or disputed paintings. The authors reexamined some recently successful experiments, showing that variations in image clarity in experimental datasets correlated with authenticity and might have acted as a confounding factor, artificially improving the results (Polatkan et al. 2009). In a different approachFails and Olsen (2003a, b, c) proposed a tool for creating new camera-based interfaces using a simple painting metaphor. They utilized transparent layers to present all necessary information to the designer, adapting traditional machine learning algorithms to suit the rapid response time required by an interactive design tool. In 2005, Yelizaveta et al. (2005) presented an automated approach to analyzing and representing artistic color concepts, such as color temperature, color palette, and color contrasts in the realm of paintings. They achieved this by extracting homogeneous color/texture regions from paintings and employing image processing and machine learning techniques to characterize regions in terms of artistic color concepts.

Moving into the 2010s, deep neural networks began to gain prominence. One of the most relevant works involved generating realistic images using deep neural networks. Liu et al. (2018) focused on investigating the sketch-to-image synthesis problem using conditional generative adversarial networks (cGANs), proposing a model called Auto-Painter that could automatically generate compatible colors given a sketch. Another intriguing work revolved around nonlinear matrix completion (NLMC), ex-tending classical techniques of linear matrix completion to the nonlinear case for recognizing emotions in abstract paintings (Alameda-Pineda et al. 2016). Another interesting contribution was made by Falomir et al. (2018), who presented the QArt-Learn approach for categorizing painting styles based on qualitative color descriptors (QCD), color similarity (SimQCD), and quantitative global features (such as average brightness, hue, saturation, and contrast of luminosity and brightness) to understand the characteristics of Baroque, Impressionist, and Post-Impressionist paintings.

There was also a study that aimed to automatically identify paintings by Vincent van Gogh using a Convolutional Neural Network that extracted discriminative visual patterns directly from images, coupled with a machine learning classifier and a fusion method in the final decision-making process (Folego et al. 2016). Another related interesting work was by Belhi et al. (2018), addressing the challenge of automatically classifying and annotating cultural heritage artifacts using their visual characteristics and available metadata. They presented a multimodal classification approach for cultural heritage artifacts based on a multitask neural network, where a convolutional neural network (CNN) was designed for visual feature learning and a regular neural network for textual feature learning.

For this same decade, specifically in 2019, an article was published that presented the state of the art in artificial intelligence. The article initially defined AI and its relationship with machine learning and deep learning, followed by a brief timeline of AI evaluation, advantages, and challenges in the current world. It then discussed the three fundamental problem-solving techniques: knowledge and reasoning, neural networks, and natural language processing (NLP) (Mondal 2019). However, the author generalized this state of the art and did not delve specifically and deeply into its relation with art, particularly painting.

An intriguing work, despite not having the highest number of citations and published in 2011, focused on an approach to the automatic classification of paintings by artistic genre. The authors classified six genres: realism, impressionism, cubism, fauvism, pointillism, and naive art, demonstrating how automated classification is useful for processing large databases in institutions like museums, and could also find applications in contemporary platforms, including mobile applications (Čuljak et al. 2011). This not only aids in cataloging historical artworks but also sets the stage for how technology can make art more accessible in the contemporary digital era.

Furthermore, in a more in-depth automatic analysis presented by Van Der Maaten and Erdmann (2015), an algorithm capable of conducting canvas-level analysis was introduced. This algorithm not only provided insights into individual threads in the canvas X-ray but also demonstrated the potential for combining traditional art analysis with modern computational techniques. The authors applied this method to a small collection of paintings apparently by Nicholas Poussin and a small collection of paintings by Vincent van Gogh, emphasizing the enduring relevance of historical artworks in the contemporary analytical landscape.

Moving into the 2020s, one of the articles that had a significant impact presented the affective experience triggered by visual artworks. The researchers asked annotators to indicate the dominant emotion they felt for a specific image and, more importantly, to provide a well-founded verbal ex-planation for their emotion. They focused on visual art (e.g., paintings, artistic photographs), demon-strating a series of captioning systems capable of expressing and explaining emotions from visual stimuli (Achlioptas et al. 2021). Another interesting contribution in this decade for painting using machine learning was an algorithm designed to automatically generate reliability evaluations for facial action units (smile, eyebrows, etc.) in European portraits in large historical databases. The authors suggested that increased reliability demonstrations are associated with a higher standard of living (Safra et al. 2020). In 2020, the Self-Organizing Map (SOM), an unsupervised machine learning algorithm based on neural networks, was applied to introduce a novel approach to spectral image data analysis. This method automatically reduced hundreds of thousands of X-ray fluorescence spectra into a handful of distinct groups sharing similar spectra (Kogou et al. 2020).

Moreover, another work conducted in 2020 proposed a method to integrate an artistic style into brushstrokes and the painting process through the collaboration of robotic painting with a human artist. They collected brushstrokes and manual brush movement samples from an artist, then trained a gen-erative model to generate brushstrokes belonging to the artist’s style. They subsequently adjusted a stroke-based representation model to work with their robotic painting setup (Bidgoli et al. 2020). In 2021, some authors proposed using a convolutional graph network and artistic comments instead of paint color to classify the type, school, time period, and author of paintings by implementing natural language processing (NLP) techniques (Zhao et al. 2021). Later, in a work by Tian and Nan (2022), published in 2022, they proposed a multitask convolutional neural network model for the emotion and rating of artworks. They created an artwork appreciation dataset consisting of fifty Chinese paintings and fifty Western oil paintings, recruiting twenty subjects to rate the art appreciation of a hundred artworks in the dataset, encompassing both aesthetic evaluation and emotion evaluation of the painting. Subsequently, using the artwork appreciation dataset, they proposed a convolutional neural network model based on AlexNet to utilize the powerful feature extraction and classification capabilities of neural networks to complete the appreciation of artworks.

Finally, among the most recent works published in the last year, there is the contribution of Spee et al. (2023), who used Machine Learning to probe complex associations between 17 subjective artistic attributes and judgments of creativity in a wide range of artworks. Random Forests regression models applied represented 30% of the variability in judgments of creativity from the set of artistic attributes, revealing that symbolism, emotionality, and imagination are the main attributes influencing judgments about creativity. Another interesting contribution was made by Mengyao and Yu (2023), who conducted a trend analysis in product art design, primarily focusing on industrial product design using machine learning. Throughout time, various review articles have been conducted, focusing on the advancement of artificial intelligence, emotion review, or, like the latter, on the use of artificial intelligence spe-cifically for industrial product design, but they do not focus on predicting artistic styles. A few of the articles included in this literature review, comprising the most cited articles from the 2000 to 2010 s and the most recent ones in the 2020s, are summarized below, as depicted in Table 1.

In order to achieve the objective of the research, a bibliometric analysis is proposed. The methodology followed in this bibliometrics adheres to the guidelines established by the PRISMA 2020 declaration (Page et al. 2021) for conducting literature reviews, thus focusing on the analysis of metadata, which allows obtaining a panoramic and comprehensive vision of the current state. from research on machine learning models applied to the prediction of artistic styles in the field of painting. The PRISMA method is one of the most recent and widely used for literature reviews, allowing greater specificity in establishing inclusion and exclusion criteria for research. This helps reduce bias in the selection and inclusion of documents in the research and literature review. Additionally, the recent update to the PRISMA statement better aligns with the quantitative and qualitative criteria generated for obtaining results, enabling the combination of results from multiple databases and subsequent selection and cleaning of results for analysis.

The bibliometric analysis on the use of machine learning models to predict artistic styles in paintings has allowed us to analyze the volume of annual publications, as shown in Fig. 2, which has revealed a remarkable exponential growth in scientific production. related to this topic, with an impressive increase of 98.49%. This analysis has focused mainly on the years 2020, 2021 and 2022, which have been identified as the periods with the highest number of publications on the subject. These results show a clear upward trend in interest and research in the use of machine learning techniques to address the prediction of artistic styles in paintings, highlighting the potential and relevance of this area in the scientific field. Among the most recent publications, for the year 2023, is the work of Lc (2023), where the authors use visions of climate futures generated by machine learning (stable diffusion, half-trip, etc.) and potential climate adaptations from the use of ChatGPT in a creative space to reconstruct a modern analog scene of ancient cave painting. In addition, the research of (Mengyao and Yu 2023) analyzed the artistic design of smart home products based on machine learning and found that household appliances and lamps will affect the aesthetic feeling of the entire interior design. This addresses questions 1 and 2: What are the years in which there has been more interest in using machine learning to predict artistic styles? and What is the growth in the number of scientific articles on the use of machine learning for predicting artistic styles?

An analysis of the main research references is then presented, broken down by main authors, journals and countries. Regarding the former, as shown in Fig. 3, three groups of prominent authors have been identified. In the first place, there are those with remarkable scientific productivity and impact, among which only Fails JA stands out, with 354 citations for his main work, which, together with other authors, proposes an interactive machine learning model (IML). (IML) that allows users to train, classify/view, and correct images (Fails and Olsen 2003a). Second, there is a group of authors who stand out for their impact on the scientific community despite their low productivity, including Olsen; Li and Chen; and Tzeng, Lum, and Ma. Finally, a group of relevant authors is identified due to their high scientific productivity, although not necessarily due to their number of citations, Zhang X being one of the main references in this category. These results allow a better understanding of the dynamics and contributions of the different groups of authors in the field of deep learning applied to the analysis of artistic styles in paintings. This answers the third question regarding: What are the main research references on the use of machine learning for predicting artistic styles?

In the context of the analysis of the most important journals as part of the research references, two groups of prominent scientific journals were identified, as shown in Fig. 4. On the one hand, journals were found that stand out for their impact on the community, despite a low productivity in terms of publications, among which the “International Conference On Intelligent User Interfaces” and the “IEEE Journal on Select Topics in Signal Processing” stand out. On the other hand, a second group of reference journals was identified due to their high scientific productivity, although not necessarily due to their number of citations, mainly the journal “Proceedings of Spie - The International Society for Optical Engineering”. These results allow a better understanding of the dynamics and approaches of the different journals in the field of deep learning applied to the analysis of artistic styles in paintings.

Two groups of prominent countries were also identified, as shown in Fig. 5. The first group, consisting of China and the United States, is characterized by high scientific productivity and impact on the academic community. The second group, led by Germany, South Korea and India, is characterized by outstanding scientific productivity, although the number of citations is not as relevant as in the first group. These results reflect the leadership and significant contribution of China and the United States in the development of research in the field, while Germany, South Korea and India also emerge as important referents in terms of scientific production related to the subject studied.

The present investigation analyzes the thematic evolution in the literature on the use of machine learning models to predict artistic styles in paintings, as observed in Fig. 6, taking as reference the most used keyword in each year of the study from 2003 to 2023. It was observed that in the first year of the study, 2003, the appearance of terms such as “perceptive user interfaces” stood out. However, in recent years there has been a clear preponderance of topics such as “Artificial Intelligence”, “Deep Learning”, and “Generative Adversarial Networks”, reflecting current trends in research on this topic. These results provide an overview of the progress and changes in the research approach over time, which can be very useful for future work in this area. The terms in this figure were obtained through the mode of key terms from the articles provided by the authors per year. In other words, the summation of key terms from the key articles per year is performed, and the most repeated key term per year identifies the progression of the theme each year. This addresses the fourth question related to: What is the thematic evolution derived from the scientific production on the use of machine learning for predicting artistic styles?

Similarly, the literature review provides an analysis of the keyword co-occurrence network in the context of machine learning models for predicting artistic styles in paintings, where 8 different thematic clusters are identified, as shown in Fig. 7. The red cluster stands out as the most prominent and includes key terms such as “deep learning”, “computer vision”, “cultural heritage”, “generative adversarial network”, and “convolutional neural networks”. Next in importance is the light blue cluster, which includes terms such as “Artificial Intelligence” and “Art Design”. The other color clusters, Green, Yellow, Dark Blue, Orange, Lilac, and Pink, reflect different elements of conceptual affinity in this field of study. This addresses the fifth question related to: What are the main thematic clusters on the use of machine learning for predicting artistic styles?

Finally, an analysis based on a Cartesian plane is proposed to measure the frequency of use of keywords on the X-axis and their validity of use on the Y-axis, which allows a detailed observation of four different quadrants, as observed in Fig. 8. The first quadrant of the Cartesian plane, representing the frequency of usage of keywords against the average year of usage, corresponds to emerging and prominent words. These terms, though recent, have seen a rapid surge in usage and have become prominent elements in contemporary communication. In contrast, the second quadrant houses emerging words, indicating that these terms are relatively new and have not yet achieved widespread adoption. This quadrant suggests a potential for future growth in the popularity of these words, but currently, their presence in conversation is more limited compared to those in the first quadrant.

In quadrant 4, identified as the quadrant of declining concepts, no relevant keywords were found in this particular case. On the other hand, quadrant 2 highlights keywords that have a low frequency of use but a high currency, which classifies them as emerging, including terms such as “Art Design”, “Supervised Machine Learning”, “Art History”, and “Image Generation”. In contrast, established and growing concepts such as “Artificial Intelligence” and “Deep Learning” are located in Quadrant 1. This analytical strategy provides a comprehensive view of the evolution and relevance of keywords in the field of deep learning models for predicting artistic styles in paintings. This addresses the sixth and seventh questions related to: What are the growing and emerging keywords in the research field of the use of machine learning for the prediction of artistic styles? and What issues are positioned as protagonists for the design of a research agenda on the use of machine learning for predicting artistic style?

In this "Discussion" Section, we provide a detailed analysis of the results of the research on the use of machine learning for predicting artistic styles. The practical implications of the results are addressed, considering their relevance and applicability in the artistic field. The limitations of the study are also examined, identifying possible areas for improvement and limitations that could affect the results. A classification of keywords by function is provided to facilitate understanding and organization of the key concepts covered. In addition, the main research gaps identified during the study are highlighted, indicating opportunities for future research in this area. Finally, a research agenda is proposed to guide future work and approaches that can enrich the use of machine learning in artistic style prediction.

The bibliometrics carried out on the use of machine learning models to predict artistic styles in paintings have yielded highly relevant results. A significant growth of interest in this field has been observed in the years 2020, 2021 and 2022, indicating a constant increase of attention and commitment of the scientific community to this field of study. This increase in the production of scientific articles has been exponential, which allows us to conclude that the scientific literature has a sustained growth of research on the subject, positioning it as relevant for the near future.

Regarding the main research references, the author Fails was identified as a leading figure in the literature, indicating his relevance and contribution to the understanding of machine learning models applied to the prediction of artistic styles in paintings. Likewise, it was found that the journals “Conference On Intelligent User Interfaces” and “IEEE Journal on Select Topics in Signal Processing” are leaders in the publication of research on this topic, giving them a fundamental role in the dissemination of knowledge in the field. On the other hand, the most prominent countries in scientific production are the United States and China, reflecting their strong commitment and leadership in research related to this topic.

The analysis of the thematic evolution revealed a transition in the orientation of the studies from a specialization in “Perceptive User Interfaces” to a greater focus on topics of “Artificial Intelligence” and “Deep Learning”. This change allows us to conclude that there is an adaptation of researchers to current trends in research on the use of machine learning models to predict artistic styles in paintings, which contributes to the advancement and enrichment of knowledge in the discipline.

On the other hand, a thematic cluster stands out in the research, composed of terms such as “Deep Learning”, “Computer Vision”, “Cultural Heritage”, “Genera-tive Adversarial Network” and “Convolutional Neural Networks”, pointing out a conceptual affinity between these key themes, which suggests priority focus areas for future research and allows us to conclude on crucial elements about one of the leading concepts of scientific literature such as Deep Learning.

With regard to the keywords that are currently trending, a solid domain of “Artificial Intelligence” and “Deep Learning” has been identified as consolidated concepts and protagonists in the field. Similarly, emerging terms such as “Art Design”, “Supervised Machine Learning”, “Art History”, and “Image Generation” were observed, highlighting the evolution and expansion of the field into new areas of research and applications.

The article primarily focuses on classification and the main journals in the field but covers much additional critical information. It provides a detailed classification of emerging keywords related to machine learning in predicting artistic styles, emphasizing their growth and highlighted approaches. Practical implications range from impacting the scientific community and art to democratizing creativity and preserving cultural heritage. Additionally, it outlines study limitations and offers recommendations to enhance research validity. Lastly, it identifies research gaps and proposes a future agenda spanning from developing advanced algorithms to exploring image generation and classification techniques.

The study on using machine learning for predicting artistic styles in paintings offers a comprehensive overview of the field’s growth and evolution. Through bibliometric analysis, it highlights significant scientific advancements and the evolution of interest in machine learning techniques for this purpose. It identifies emerging themes like “Artificial Intelligence” and “Deep Learning,” guiding future research directions. Additionally, the study emphasizes key references, influential authors, and noteworthy journals, providing a roadmap for researchers to navigate the current landscape of knowledge and potentially enhance the visibility of their work. Moreover, the comparative analysis among countries fosters global collaborations, offering insights into regional contributions and cultural perspectives, while the exploration of thematic evolution and keyword networks inspires innovative research directions and strategies for further exploration of promising concepts.

Finally, the analysis conducted addresses the evolution of the use of machine learning (ML) models in predicting artistic styles, highlighting a shift from user perception approaches towards advanced tools such as deep learning. The emphasis on neural networks, particularly in pattern recognition, underscores the use of design software for image manipulation. Moreover, gaps in existing research are acknowledged, pointing out the need for more comprehensive ML techniques to fully capture the diversity of artistic styles, as well as the importance of interpreting the features learned by these models and further exploring the role of AI in art authentication and preservation.

The review highlights structured questions covering temporal trends, growth patterns, influential research references, thematic evolution, emerging keyword clusters, and key topics shaping the research agenda in this field. The chronological presentation of prominent works from different decades offers a comprehensive view of the advancement of machine learning in predicting artistic styles. In this way, a wide range of literature is synthesized, proposing a comprehensive research agenda based on bibliometric analysis, providing some insights for future interdisciplinary research in this area.