Bilderatlas – Progress Post 3

Current progress

In the past three weeks we have reached the core of the project: analyzing Warburg’s tables through the use of Convolutional Neural Networks.

According to our milestones, by this time we should have analyzed all four tables. We are still slightly behind schedule, as we have only analyzed two tables of the 4, but we are planning on meeting more often in the last period to conclude our project.

There has been an additional change with respect to our original plans. We have been provided with the DH Replica server, which allows us to launch queries in the CNN database. Our database contains several thousands images, including those we have uploaded from Warburg’s tables. Thanks to DH Replica, we can easily select images in the database and launch a query, visualising directly the result  (Figure 1).

Figure 1 - DH Replica server
Figure 1 – DH Replica server

In the DH Replica, the images can be set to positive, which means that the pattern we are looking for is present in the image, or as negative, which obviously implies that the pattern is not present. However, lately the CNN algorithm has been changed, and the research with negative images does not produce reasonable results, probably because the weight of negative images in the algorithm is too high. We thus performed queries with positive images only. We will analyze some interesting queries with negative images once the algorithm is fixed.

We have also completed the bot whose role is to launch queries. It can accomplish the same tasks as the DH Replica. However, since the latter considerably simplifies the visualisation of the results, the purpose of the bot will only be to receive the scores of the  query, together with the annotations relative to the images.

The bot reads from an excel file the IDs of the images to launch in the query. The IDs can be positive or negative, depending on the column in which they are put (Figure 2, red and green columns). It then creates a new excel file where it reports the IDs of the resulting images, together with the correspondent scores and annotations (Figure 2). The bot also gives the option to directly download the images when returning the result and naming them accordingly to their score.

Figure 2 - Excel file reporting the results of a query
Figure 2 – Excel file reporting the results of a query

We will now present our analysis on table 2. Table 45 has only been partially explored, so we will report our considerations in the final report.

Table 2

Table 2 - The cosmos in Greece: the resemblance to humans.
Table 2 – Greece: home of anthropomorphic representation of the cosmos.

The theme of table 2 is “Greece: home of anthropomorphic representation of the cosmos”. We find figures like Apollo (the Sun), the Muses (protectors of poetry and celestial spheres), the Atlas (condemned to hold up the sky for eternity), the story of Perseus (mythological characters as constellations: Andromeda, Cepheus, etc.). This is a rather conceptual table, where the images are connected because they all represent mythological figures, lacking strong visual similarities. There are, however, some subgroups that can be considered visually similar.

In our database we have eight images belonging to this table (Figure 3): four sculptures and four miniatures.

Figure 2 - Images from table 2 present in our database
Figure 3 – Images from table 2 present in our database

By querying all the images, the result is not particularly meaningful. This response was expected because the images are too different, so no common visual pattern is present. No mythological figures appear in the result.

By searching for the four miniatures (7a, 7c, 7d, 7f ), the CNN finds several other miniatures, although inserting (quite low in the rankings) some paintings that are not actually miniatures. It is interesting to point out how the CNN recognized the drawing on paper, which is the only common feature among the found miniatures. However, when launching one of the four miniatures, for example 7c, only one among the other three is found (7d in this case). This is probably due to the similar backgrounds. Indeed, the background of 7c is similar to 7d’s, while that of 7a is similar to 7f’s. Not very many miniatures are found by launching one single miniature, meaning that the CNN needs to compute a common feature (obtained only through multiple images) to find other miniatures.

When querying the Andromeda (7a), surprisingly the Farnese Atlas (6a) shows up, even though it has no visual similarity. The only thing that can explain its finding is the fact that his arms are spread in a similar way as the Andromeda. However, this is not the reason why Warburg put these two images together. By adding Cepheus (7f) to the research (even though his arms are equally spread) the Atlas disappears, suggesting no visual similarities with the two miniatures.

By adding the Farnese Atlas (6a) to the query of the four miniatures, we observe a theme of open arms. By removing the Equus (7d) from the query, the theme becomes evident, meaning that the CNN recognized the path “open arms” (Figure 4).

Figure 3 - Result of the query: 6a, 7a, 7c, 7d, 7f
Figure 4 – Result of the query: 6a, 7a, 7c, 7d, 7f

We tried to launch a query with the Equus (7d) in order to find other paintings containing the theme of the winged horse, and see an evolution of the figure, but no interesting result showed up, probably because of the considerable influence of the background.

Finally, by launching a query containing the four sculptures, the result contains indeed only sculptures, but it is not particularly satisfying as no mythological figure is found.

In conclusion, as expected, the CNN did not succeed in discovering the abstract pattern of mythological figures throughout the images, but indeed proved to be successful in detecting complicated visual patterns, such as that of the miniatures.