The second milestone focuses on modelling and implementing the visualization based on knowledge and pattern identified in the first phase. Our orientation has been changing to “visual story telling”, which improves the readability of a painting in terms of social power and the relationship between representatives of those powers.
What we have done in this milestone?
Particularly, we identify who is represented in the painting according to information gathered from the book Civic Ritual in Renaissance Venice (Muir, 1986) and some other online resources. In the visualization each representative of power is highlighted by a different color and tooltip displaying the name of that character at the time of event. Details of the character is shown if user clicks on that highlighting area, including name, description (Figure 1), and potentially accessories’ details as well as relationships with other characters in the painting (which will be implemented in the next phase).
Figure 1: Visualized story on painting
Here is the demo with full functionality of our visualization:
|Description:||Please click on the title to access the live demo page|
Brief of technology
Object extraction is one of the most highly used techniques in computer vision, which help us to retrieve the object of interest for further processing, analysis and visualization. For our project, we aim to somehow extract the important objects from paintings so that we can map it with the text description and especially to visualize the power system of Venice. Here “important objects” can be understood as people (of different ranks and status) and the accessories carried with them.
Since this task can be very tedious and time-consuming if we manually extract each person from the paintings, determine his location and boundary so we want to combine various computer vision/image processing methods to constitute a semi-automatic system to accurately extract region of interest (ROI).
In order to do this, we have written a simple Matlab script to display the paintings, and then let users to pinpoint to choose the rectangular regions that they want to extract. It will compute the location and save the region automatically for users. These location data will then be used for our web program to highlight the object of interest chosen by visitors (see the live demo). Figure 2 below illustrates few regions (bounded by black rectangles) that are extracted from the paintings.
Figure 2: Extracted regions
Demonstration in details
There are different members of the Confraternity  which are demonstrated in the visualization (Figure 3). Specifically eight trumpeters precede the golden “soler” in the foreground with their campaniles in hands. On the golden soler carried the precious relic of the Holy Cross. The soler is covered by a canopy which is referred to as an umbrella. Both soler and canopy are carried by eight standard-bearers. Following this middle group are six other trumpeters and finally escorted by candle-bearers who carry the white candles of the Doge.
Figure 3: Members of the Confraternity in the foreground
On the far right there have three of the eight comandadori with their banners, followed by six trombe lunghe (also known as musicians). In particular only two of those trombe are actually playing, while the other four are carrying their instruments propped on their shoulders, and with the bells in the air (Figure 4) .
Figure 4: Representatives on the far right
What are the challenges that have been faced for this milestone?
- People representing for roles in the painting are not always easy to identify, since their costumes and accessories vary from samples in the book Civic Ritual in Renaissance Venice (Muir, 1986). The visualization was made with uncertainty about its precision of those representatives. This will be verified with our advisor, Giovanni Colavizza, to find out appropriate solutions.
- The relationships among people are not strong enough to model a particular visualization (e.g. tree graph, time series, sequential graph, etc.). The current approach is closer to story telling which varies from the initial plan (but reveals some interesting trend to proceed further).
- The live demo can not be directly embedded in the blog post due to some technical constraints in WordPress. The team will contact the course TA for this issue.
What will be done for the next phase?
- Identify the relationship of powers among different representatives, visualize those relationships in appropriate means such as hierarchical tree, force graph or time series.
- Extract the object more accurately, i.e. not just the bounding rectangle but the exact boundary of the object. It will better visualize the object on our web program and also, easier for further processing and analysis.
- Collect full information of all representatives and their identifiers in the painting.
- Gather all the pieces together and get prepared for the final report.