There's rarely time to write about every cool science-y story that comes our way. So this year, we're once again running a special Twelve Days of Christmas series of posts, highlighting one science story that fell through the cracks each day, from December 25 through January 5. Today: How AI is helping shed light on famous paintings.
X-rays are a well-established tool to help analyze and restore valuable paintings because the rays' higher frequency means they pass right through paintings without harming them. X-ray imaging can reveal anything that has been painted over a canvas or where the artist may have altered his (or her) original vision. But the technique has its limitations, and that's where machine learning can prove useful. Two papers this fall illustrated the use of AI to solve specific problems in art analysis and conservation: one to reconstruct an underpainting in greater detail, and the other to make it easier to image two-sided painted panels.
Picasso's The Old Guitarist is one of the best-known works from the artist's so-called "Blue Period." Two decades ago, X-ray and infrared analysis revealed that he had re-used an older canvas (a common practice for struggling artists of the period). There was another painting underneath, of a seated woman, that matched a sketch Picasso had included in a letter to a friend. But the X-ray and infrared images couldn't provide sufficient detail to get a sense of what the original painting really looked like, especially the choice of colors.
In a paper posted to the physics arXiv in September, Anthony Bourached and George Cann of the University College London described how they employed machine learning to reconstruct a full-color image of Picasso's original underpainting—specifically, a technique called neural style transfer, originally developed a few years ago by researchers at the University of Tübingen in Germany. Per Technology Review:
Neural networks consist of layers that analyze an image at different scales. The first layer might recognize broad features like edges, the next layer sees how these edges form simple shapes like circles, the next layer recognizes patterns of shapes, such as two circles close together, and yet another layer might label these pairs of circles as eyes. This kind of network would be able to recognize eyes in paintings in a wide variety of styles, from Leonardo da Vinci to Van Gogh to Picasso. In each case, the eyes form a similar pattern that the machine can pick out.
The network can also be trained to recognize distinctive styles, telling the difference between a Picasso and, say, a Van Gogh painting, for instance. It's also possible to reverse the process: give the neural network an image, and then superimpose a given style onto it.
That's what Bourached and Cann did with the X-ray image of the painting under The Old Guitarist. (They also applied the technique to another Picasso canvas to reconstruct an underpainting done by Spanish painter Santiago Rusinol.) It may or may not be how Picasso chose to paint it, but the process is still useful for gaining a better understanding of subjective human creativity. "Our method of combining original but hidden artwork, subjective human input, and neural style transfer helps to broaden an insight into an artist's creative process," the authors wrote.
Illuminating a medieval masterpiece
AI techniques are also proving useful in art conservation efforts. The Ghent altarpiece—aka the Adoration of the Mystic Lamb—is a 15th-century polyptych attributed to brothers Hubert and Jan van Eyck, displayed in the Saint Bavo Cathedral in Belgium. Originally consisting of 12 panels, the altarpiece features two "wings" of four panels each, painted on both sides. Those wings were opened on church feast days so congregants could view the interior four central panels. The inner upper register features Christ the King, the Virgin Mary, and John the Baptist, flanked by the outer panels depicting angels and the figures of Adam and Eve. The inner lower register depicts John the Baptist and St. John the Evangelist, with a group of saints, sinners, clergy, and soldiers in between.
The Museum of Fine Arts in Ghent began restoring the altarpiece in October 2012, allowing the public to view the process from behind a glass screen as conservators from the Royal Institute for Cultural Heritage worked on individual panels. The restoration also produced a series of high-resolution images of the various panels using different imaging techniques.
Those images in turn formed the basis for an investigation into applying AI analysis to the altarpiece images by researchers at Duke University, the National Gallery, and University College London. The team reported on their work in a paper for Science Advances. "They have one of the most famous paintings in the world, and they are willing to work with us," co-author Ingrid Daubechies of Duke University told Ars. "It's a great situation."
The fact that the wing panels are painted on both sides posed a unique challenge for conventional X-ray analysis. X-rays penetrate so deeply that it can be difficult to determine which content applies to which side of the panel, since "all of the images are visibly overlaid or 'blended' together," the authors wrote. So Daubechies and her colleagues developed a deep neural network algorithm to study mixed X-ray images containing features from the front and back of the painting's double-sided panels. They successfully applied their technique to X-ray images of the Adam and Eve wing panels to deconstruct that data into two clear images—significantly improving on the performance achieved by prior methods.
While art conservationists have made good use of techniques drawn from physics and chemistry, including making X-ray and infrared images of valuable works, they have been slower to incorporate the most state-of-the-art image analysis methods. And while image processing is a vital research area within the electrical engineering and computer science disciplines, according to Daubechies, much of this work has not focused on the unique challenges related to art conservation. There are fewer images available compared to, say, identifying pictures of cats and dogs on Instagram. "Your statistical inference cannot be the same because you have a lot less data," co-author Barak Sober told Ars.
According to Hélène Dubois, head of the conservation project at the Royal Institute for Cultural Heritage, the technique will supply useful information to aid the restoration efforts. "The structural weaknesses of the wooden supports and of the ground and paint layers could be diagnosed with more precision," she said. "These images will also help to understand the brothers Van Eyck's techniques and the changes carried out in the course of the successive execution of this unique masterpiece.
So might this new approach also prove useful for examining underpaintings, as in the case of Picasso's The Old Guitarist? "It's similar yet different, because in the case of this double X-ray of the double painted panel, we had access to high-quality visible images on both sides," Daubechies told Ars. "When you want to find things underneath another painting, the problem is different, in that you don't have access to the one that's hidden." That makes it a harder problem to solve, although, since imaging that hidden painting requires much more sophisticated imaging techniques, the AI would be working with more data.
"This approach demonstrates that artificial intelligence-oriented techniques—powered by deep learning—can be used to potentially solve challenges arising in art investigation," said co-author Miguel Rodrigues of UCL. "We'd like to see the impact that the development of similar AI-oriented approaches will have on our ability to reveal other hidden features in a painting, such as earlier concealed designs."
DOI: Science Advances, 2019. 10.1126/sciadv.aaw7416 (About DOIs).
