Research title: Visual representation of Venetian power: Cultivating the advances of crowdsourcing to facilitate story telling in paintings
Duy Nguyen (firstname.lastname@example.org)
Vu Hiep Doan (email@example.com)
The old paintings are believed to potentially contain bunch of information and meanings related to Venetian political system, encoded in their details such as people’s costumes, accessories and even the pattern of different objects in the scene interacting with each other. Therefore, the process of identifying them and “decode” those information is considered as a high mountain to climb, which requires a considerable degree of interest and extensive knowledge. From that perspective, our project aims to build a framework for visualizing and understanding the Venetian power system through paintings by crowdsourcing the knowledge of various targeted people, such as historians, humanities researchers, anthropologists, and even non-professional public people who are interested in Venetian paintings. In other words, we utilize the power of the crowd, public people, to tell the story in Venetian paintings dynamically through visual representation and with high credibility based on public judgement. In this paper we follow the story of the famous ‘Procession in Piazza San Marco’  of Bellini, which sketches quite clearly the hierarchic, symbolic and paradigmatic structure of Venetian society at that time (Figure 1).
Figure 1: The painting ‘Procession in Piazza San Marco’
Indeed, the painting of the procession is a very useful source of visual information to illustrate the political system since each particular position in the procession reflects a certain ranking and power of a person standing there together with the symbols he was carrying. Conventionally, the procession may consist of three distinct segments, each with different roles and participants. As illustrated in Figure 2 and Figure 3, the first segment, for example, is led by eight standard-bearers followed by commanders walking two by two and six musicians. Then, it continues with the squires of the visiting ambassadors and the ducal squires and cavalier, again followed by another group of instrumentalists. The following position usually belongs to six canons of San Marco, preceding the patriarch, the chaplain and the squire carrying the coronation crown before ending the first segment by the secretaries of the doge and the Senate. For a full description of all three segments consisted in the procession, we refer readers to our first blog post.
Figure 2: Order of procession – page 1
Figure 3: Order of procession – page 2
Motivation for an evolution of story telling
It would be highly potential for us to intensively analyze the images to explore the content and meaning of every object in the paintings. In fact, however, the resources and documentations, which can facilitate the understanding of all the paintings, are quite scarce and usually requires a sufficient prior knowledge on the topic of the paintings in order to fully comprehend the content. During the time of carrying out this project, we have realized an extreme lack of easy-and-open access resources for understanding those paintings. For instance, when working intensively on the paintings in Figure 1, we only found a few resources that we can lean on including the book Civic Ritual in Renaissance Venice (Muir, 1986)  and a brief Wikipedia page in Italian about Venetian power system . This shortage of resources consequently leads to seriously wrong recognitions and misclassifications of objects in the paintings.
That challenging circumstance motivated us to find ways to improve our project quality within the scope of available resources. Coming from the fact that we, as new comers to this research field, were facing various challenges in social science topics, while there could be many people out there who have higher expertise about Venetian painting but might not know how to apply them properly. That leads us to the elaboration of crowdsourcing knowledge in this field by providing a standard framework in which interested people will have an open-access community to share the knowledge, get initial experience and maintain a good “visual documentation” of Venetian paintings, and potentially a much broader scope of old paintings. The framework was initiated with the principle of professional community for expert users such as humanities researchers and historians, but also simple enough to attract non-professional public people.
Our methodology consists of several major steps, involving both social and technical approaches:
- Read and understand Venetian power system in possessions through examples in the book Civic Ritual in Renaissance Venice (Muir, 1986) , combine that knowledge with information obtained from the Wikipedia page in Italian about Venetian power system .
- Collect Venetian paintings from different sources and build a library for all of them.
- Build a sample of annotations for available examples in the book Civic Ritual in Renaissance Venice (Muir, 1986) .
- Adapt voting system and comment functionality to the framework.
- Conduct acceptance testing for the whole system.
Final results – a cutting-edge annotation framework
A live version of our framework can be access here: http://venicepower-app-24pmpxpprk.elasticbeanstalk.com/ (Test account, username: test / password: test)
The framework essentially allows adding annotation for people and items in the available paintings, particularly identifying their position in the painting, size, title and description. There are also various examples as verified references from academic resources  . By allowing users to drag and drop rectangles on the painting canvas, the system automatically records coordinates and size of the annotation, which provides an ease of user interface to non-technical users (Figure 4).
Figure 4: Annotating page
Results of annotation can be viewed and evaluated by other users of the framework. Annotated people and items are represented using different colors, and each of them can be chosen to see further details such as title and description (Figure 5).
Figure 5: Visual overview of the painting
In addition to the overview of painting and annotations on it, users can also interact with each other as well as the annotation owners by comments and votes. Annotations are ordered according to total score by others’ votes, and the top voted annotation of one particular region on the painting would be selected for the display as illustrated in the screenshot in Figure 6.
Figure 6: User interface of voting, adding new annotation and comments.
Thus, as the framework can collect inputs from multiple users, who can be experts, researchers and art-lovers, we could expect a highly qualified annotated painting where all people, especially newcomers, will gain a great deal of initial knowledge and insights on Venetian paintings. The mechanism of voting up/down is very useful as the crowdsourced information filters itself and endlessly improves time after time. And by commenting on others’ annotations everyone has an equal chance to contribute the the painting quality in general.
Moreover, all the details of a specific painting including title, description, list of objects with users’ annotations can also be exported in JSON format, along with an referenced painting image fully annotated accordingly (Figure 7). This functionality allows offline access to the painting annotations without the availability of internet connection or modern browser support.
Figure 7: Offline exported annotations
For now, we restrict the permission of uploading and modifying paintings within registered users, who are created by the super admin. Users who own an account can go to admin page to add new paintings or modify their previously uploaded paintings (this can be tested using test account provided above).
The current version of our annotation framework is just a prototype that can be extended with more features and research values. In particular, the storytelling on paintings would be more effective with the support of relations among entities (both people and physical items). For example, if user highlights a person in the painting he could also see related items that person had as well as the explanation of how the items help to identify the social status of the highlighted person in Venetian society. Also, there are various items that are shared among different people so it could be interesting to observed those items commonly possessed by stakeholders.
Another potential implementation is that we could allow users to model relationships themselves and comment on others’ models. Here the storytelling is not only croudsourced but also approached from different angles (points of view). Users contribute their own knowledge to the community that they also benefit from by viewing, debating, having their annotations corrected and so on. Beside that, the possibility to extend the project to integrate with social media such as Twitter, Facebook, WordPress blogs, etc. is also promising. That will potentially bring more options to the users in terms of social sharing and collaborative learning of Venetian power through paintings, at the same time attracting more audience to involve in the process of popularizing this research area to the public.
Further development could lead us to a real social network for researchers, historical experts and art-lovers who are interested in Venetian paintings. Compared to currently available means such as blogs, Facebook groups, Twitter, etc. our annotation framework potentially brings a “live experience” breakthrough that facilitates those users in sharing their visual and physical feeling, which significantly gain the learning effects and improve the collaborative working process in digital humanities community.
At the end of this project, we have successfully elaborated a potential framework for Venetian paintings, which even has a tremendous perspective and rooms for further development. The horizon of crowdsourcing community is limitless, and it would also endlessly attribute to the successful continuation of our annotation framework for visual representation of Venetian paintings.
The project team would like to sincerely thank Professor Frederic Kaplan for giving us the possibility to propose this project and implement it despite the uncertainty of its outcome. We would further like to thank our advisor Giovanni Colavizza for his valuable advice, criticism and motivation. Finally we would like to give credits to Isabella Di Lenardo and Stefania Coccato who gave us informative resources and great help in annotations of objects in the ‘Procession in Piazza San Marco’ painting.
Previous blog posts