Art is a form of expression tracing back thousands of years. The thoughts running through one’s mind painted on canvas, make it one of the most analysed historical works. Critics spend decades trying to find connections between different pieces of art, trying to find parallels between themes, styles and concepts. With a countless number of artistic pieces through the years, it is almost impossible to find connections between them all.
To make this entire process streamlined and quick, a group of researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) partnered with Microsoft to create an algorithm which discovers hidden connections between paintings at the Metropolitan Museum of Art (the Met) and Amsterdam’s Rijksmuseum. The system is called “MosAIc” and it finds pairs between works from different cultures by using deep networks to predict how ‘close’ two images are.
For example – in order to find an art piece of a musical instrument closest to a blue dress, the algorithm retrieves an image of a blue porcelain violin.
“Image retrieval systems let users find images that are semantically similar to a query image, serving as the backbone of reverse image search engines and many product recommendation engines,” says PhD student Mark Hamilton, the lead author on a paper about MosAIc. “Restricting an image retrieval system to particular subsets of images can yield new insights into relationships in the visual world. We aim to encourage a new level of engagement with creative artefacts.”
For many decades, Art and Science were considered to be separate. Two different disciplines, one dealing with beauty and aesthetic, and the other dealing with logic and facts. But over recent years, the relationship between Artificial Intelligence and Art has flourished. The GauGAN project by researchers at MIT, the GenStudio project, the AI-generated artwork that sold at Sotheby’s, all being examples of using Artificial Intelligence to generate art pieces.
MosAIc isn’t meant for ‘creating’ art though. It is more focused on sifting through existing pieces to find connections. All it requires is a single image from two different art forms each, to display the hidden connections between them. But here’s where things get complicated.
Two pieces of art are not only connected by similarity in colour or style, but also by meaning and theme. The meaning of a piece is very subjective to a human mind and is incredibly tough for a computer to decipher. But mosAIc does a pretty good job at it.
The way it achieves this feat is by probing at a deep network’s inner ‘activations’ for each image. The distance between these activations, also called features, is how mosAIc judges two art pieces’ distance in theme. To get even more technical, the team uses a new image-search data structure called “conditional KNN tree” which groups images in a tree-like structure.
When a parent image is an input to the model, the algorithm starts at the ‘trunk’ of the tree and works its way up. It follows the most promising branch until it reaches the closest image. This form of data structure is better than the rest because it quickly morphs itself to a particular query, delivering fast and accurate results.
Testing MosAIc
For the tests, the team wanted to make sure mosAIc not only returned quick results but also provided valuable insights. They evaluated mosAIc based on how closely its results matched with that of human intuition on analogies.
To evaluate this, the team made and released two datasets for testing how good a conditional image retrieving model is. The first dataset challenged algorithms to find images with the same content. The second dataset challenged algorithms to find English letters across different fonts. MosAIc was able to, almost 2/3rd of the time, recover the right image in a single guess out of a cluster of 5000 images.
This is what Hamilton had to say –
“Going forward, we hope this work inspires others to think about how tools from information retrieval can help other fields like the arts, humanities, social science, and medicine. These fields are rich with information that has never been processed with these techniques and can be a source for great inspiration for both computer scientists and domain experts. This work can be expanded in terms of new datasets, new types of queries, and new ways to understand the connections between works.”
The need for computers aiding humans is very much prevalent in today’s world. With the massive amounts of data floating around, the usage of an Artificial Intelligence model is not only helpful but also speeds up the search tremendously. Many hidden connections may even go amiss by the human eye, because of which having an algorithm beside you is mighty beneficial in this age of vast information.
Further Reading –