Introduction and objectives
When you run into a painting on the internet or in a document, how do you verify its authenticity ? Is it the original or is it a copy ? How do you date it ? To answer these questions, art historians usually try to contextualize the painting and look for information on other artworks. Therefore, a tool allowing them to know if the work they are analysing is linked to other ones would be very useful. They could for example know if there exist identical or partial copies of this work, or if there are other paintings with common features or composition, revealing influences and artistic trends. Currently, this recognition can be done only by an expert assessment of specialists; that’s where an automatic recognition using image processing would be of great help for art historians. Also with this automatic retrieval, a rich database could be created, making sure historians do not to miss similar paintings they might not have thought of.
In this project, we will develop some tools to automatically recognize identical patterns in different paintings. The first part will consist in selecting the similarity criteria between the images and create our work database consequently. Then, the second part will be to develop an automatic recognition of similar images based on those criteria. Finally, the objective is to develop an interface which allow, once you have provided a painting, to find the similar ones in the given database.
Today, a lot of algorithms allow us to deal with images and compare them by extracting visual features like colour or textures. These techniques reveal key points on the images and work very well on real pictures. However, when we deal with paintings, the points of interest are not the same as in pictures; we’re more interested in the general composition of the artwork than in its colour or texture which can change totally between two paintings of two different periods or art trends. For example, we can see on the two paintings below that they are clearly copies of each other, but the colours are very distinct, as well as the textures. The landscapes and the faces are totally different too, but we’re not interested in them because they don’t reveal a similar shape between the two paintings. However, we see that the disposition of people, their position and the general shape of the paintings are really similar.
Thus, we need a tool that is able to recognize similar shapes on different paintings. But it would not be sufficient to recognize only the general shape of the paintings, because there exist a lot of partial copies which include only a small part of the original. For example, on the two paintings below, we see that the only common thing is the woman on her knees and we can clearly establish that she has been copied from the original.
Then we need to extract the general shape of the painting as well as particular ones. To do so, the main idea is to deal with points of interest in the images, in order to extract its structure. For example placing a dot on the head of each people in the painting could give us a good idea of its global structure. Regarding the smaller shapes, we could put dots on key points of the body (head, shoulder, elbow …) in order to extract skeletons of each people.
This idea is a first try on this problem but will probably not be sufficient to find all the copies or to always have relevant results. For instance, if we have a skeleton of a person standing, we will found similar skeletons on a lot of paintings which are not necessarily related. Thus, an idea of improvement would be to combine the skeleton technique with image processing and computer vision algorithms that are able to recognize shapes and patterns in images.
The paintings constituting our database belong to the Renaissance period. This period is the most suitable one to begin experimenting with our method and the algorithms used in real picture since people and settings are usually well depicted (they are similar to what we could have on a picture). This is not a trifling choice : imagine we try our methods in Picasso’s or Dali’s pictures for example!
Since we are two groups working on this project, the work will be divided into three tasks: one in common which consists in defining the keypoints on the paintings, and two others about dealing with the general shape and with the particular ones.
For a first version of the project we will mark the points of interest manually on all the paintings in our database. The idea of the first task is to develop an interface allowing the user to provide an image and manually select points of interest on it. An idea is to add attributes to those points, in order to distinguish the general shape from the particular skeletons, and to have a quite precise skeleton on people. An example is shown on the image below. These sets of points will constitute our database and we will use them to compare the images during the recognition phase.
For the part concerning our group, we decided to work on the particular shapes. The objective is to develop a method to automatically find in a painting a skeleton given by the user. To do this, we will compare each skeletons of the database with the one we’re interested in and use some metrics to calculate the similarity between them. One difficulty we will encounter very often, as we see above, will be to deal with non-complete skeletons due to hidden parts of the bodies. We will thus have to take this parameter into account in our metrics, and discuss if we take the skeletons of only for the visible parts of the bodies or if we try to guess the hidden parts.
To help us dealing with these skeletons, we will need to read some literature about it. A widely used system handling with skeletons is the Microsoft Kinect, so it will be a good starting point to see what technique we could apply in our case.
The deliverables of this project will be
- an interface that allows to :
- load a painting to annotate it
- annotate the painting to create the skeletons
- extract the skeletons data to process it
- an algorithm that retrieves replicas or paintings with similar elements to the loaded one