Recognizing copies in paintings – Post 1

Reminder

The project “Recognizing copies in paintings” consists in creating a system that allows to retrieve partial copies of a given painting in a database provided by the DH lab. The method we will use will be based on the comparison of people present in the scene and their position. To compare them, we will consider each person as a skeleton and transform these skeletons into feature vectors in order to compare them using different metrics.

Choice of skeleton’s properties

The idea of using skeletons came from the system used by Kinect, so the first step in our project was to search for information on how this system extracts and uses skeletons. We found out that the Kinect’s system is based on depth clues and uses up to 20 points for one skeleton. [1][2]

IC584844

In our case, using more than 10 key-points for the skeletons would be too much information to process and so too complicated to compare skeletons. It would also probably induce more false results or errors than improve the precision. Concerning the depth map, it is automatically acquired by the Kinect for each image. Then, the Kinect processes the image and the depth map together in order to create skeletons. In our case, we only dispose of the image, we don’t have the depth map so we cannot use the same method as the Kinect skeleton creation.

We decided to create skeletons with six key-points: the head, the abdomen, both hands and both feet. These key-points will allow us to discriminate roughly the position of people and compare them without using an unnecessary large amount of dots.

Interface

We started to develop the Graphical User Interface (GUI) which allows the user to load the image (s)he wants to process and manually annotate the skeletons. Since we decided to use Matlab to process our data, we thought it was more convenient to use Matlab to create the GUI as well. We developed the layout and here is a first version :

Gui2_SHS

This GUI will work the following way :

  • The user loads the painting
  • Places each key-point on one person to create the first skeleton
  • Saves the created skeleton
  • Repeats the process for each person in the painting
  • Extracts all the data and closes the image

This way, we create all the skeletons visible in the image and extract them in order to process them later.

Progress and problems encountered

For the moment, we follow the schedule we defined but we are facing some unexpected problems. The main one is a misunderstanding about the division of the tasks with the other group working on the same project as us. Following a meeting with the assistant in charge of our project, it came out that the work of the two groups is too similar and therefore we need to refocus a bit the goal of our project.

schedule

Next steps

The immediate next step is a meeting for the two groups with Professor Kaplan concerning the orientation of our work. Depending on the conclusion of this meeting, we will either continue in the same direction and focus on the metrics we will use to compare skeletons, or reorientate our project on the automatic extraction of the skeletons using image processing techniques instead of manual annotations for example.

References

  1.  Real-Time Human Pose Recognition in Parts from Single Depth Images
  2. Tracking Users with Kinect Skeletal Tracking