Automatic Recognition of Palaces based on Pictures: Step 1

Introduction

Our project delivers a web-service which acts as a recognition system of Venetian palaces based on a given input image. It provides a platform for a user to identify the palace or the building and the path to reach the place. We exploit the unique architectural features of the buildings to build a model of the learning system.

Approach

We aim to classify the set of images we have on the basis of the era during which they were built. This largely categorizes the Venetian architecture into three major classes. We wish to distinguish between architectures built under Byzantine influence (900-1300), gothic architecture (1300-1500) or during Renaissance period (1500-1600). The Venetian Gothic architecture could be further classified into architectures following Islamic influence, secular gothic or religious gothic. More specifically, we wish to identify the characteristic features of each category and narrow our search to few palaces. This is an important highlight of our approach. Other characteristics that help our quest for identification could be unique colours used for the walls (in cases when image provided by users is coloured), arch shapes, presence of domes, number of windows, relative distances between them, amongst others.

ordine-veneziane1 ordine-veneziane2

Fig.1 Characteristic arch shapes

Step I – The learning task requires a large collection of good quality images of buildings from all the categories. In the first week, we collected two books based on Italian architecture which offered pictures from more than 25 palaces. We captured high quality images of resolution 3400×2400 pixels using overhead camera scanners from the Rolex Learning Center. We cropped these images into smaller sets of images to increase the volume of the dataset which in turn help to generalize the learning model. The figure represents few images that we scanned for the dataset collection. We plan to expand the dataset using other sources like Wikipedia, Google Images, Flickr and among others.

BookcopyCollage3
                         Fig. 2 Palace dataset[2]

Step II – We also reviewed the literature about promising field of deep learning as we had proposed. However, we learnt that application of deep learning requires a huge amount of training images for each category which we lack for our Venetian palaces. As in [1], we started to work on the baseline methods to extract features for architectural scenes. We tried with SIFT and HOG descriptors as shown in figure 3. We plan to explore the shape context features Difference of Gaussian (DoG), HOG descriptors, Dense SIFT, etc. The State of the art classification performance on architectural dataset uses SVM and word mining [1]. We propose to use Random Forest as a learning model which has promised well as a regressor and classifier.

feature
                         Fig. 3 Feature extraction module

 

Web-Application Interface

Our website not only recognizes a palace based on the image, but also gives additional information on the palace from Wikipedia or other sources and how to reach it and any relevant details about the palace. Building the website and integrating the image processing platform will consume more than four weeks of our time. Hence we have decided to do web development parallel with the image processing work. Our plan is to use OpenCV in Python for image processing and Django framework which also uses python for managing the applications. It has been demonstrated that OpenCV can be integrated easily with Django as they are based on Python. The plan over the next two weeks would be to build the basic skeleton of the website and try to write some web applications that can deliver useful information based on the picture uploaded and also test the integration of OpenCV on this framework.

Conclusion

In the past two weeks, we read about various kinds of Venetian Architectures and their uniqueness. We explored about deep learning to remove the ambiguity of choosing learning models among many of them which is suitable for our recognition problem. We also started our work on web application in parallel. In the next phase, we plan to make a working prototype of the recognition system and an integrated web interface.

References

[1]. Goel, Abhinav, Mayank Juneja, and C. V. Jawahar. “Are buildings only instances?: exploration in architectural style categories.” Proceedings of the Eighth Indian Conference on Computer Vision, Graphics and Image Processing. ACM, 2012.
[2]. Rössler, Jan-Christoph. I Palazzi Veneziani. Venezia: Fondazione Giorgio Cini, 2010. Print.
[3]. Fasolo, Andrea. Palazzi Di Venezia. Venezia: Arsenale, 2003. Print.
[4]. Venetian Architecture. (n.d.). Retrieved March 4, 2015, from http://imaginingvenice.com/2013/04/16/venetian-architecture/