Mathias, M. et al. (2011) [38] propose a method to automate the classification process of different architectural styles. In addition, they contribute to the field of architecture by firstly determining if there is an actual facade of a building in an image and, secondly, by rectifying the distorted image. Finally, they introduce a vertical line system to separate multiple facades within the same image. Thereafter, the task of classification of the architectural style is performed, recording the building style as Flemish Renaissance, Haussmannian, or neoclassical using a Naïve Bayes Nearest Neighbor classifier. The observed results clearly distinguish the Haussmannian class from the neoclassical class, whereas many Flemish Renaissance results are classified as background. According to the authors, the proposed model may be used to initialize a building reconstruction process if all the stages are respectfully applied. Mathias et al. noted that little work had been carried out on architectural style identification.

In the same year (2011), Shalunts, G. et al. [39] examined the task of classifying different architectural elements of a building’s facade. The objective of this research is to distinguish the architectural style of facade windows belonging to Romanesque, Gothic, and Renaissance/Baroque periods. However, since the number of arches may differ in the Romanesque windows, eight intra-class types are also proposed. Since each different window is built with specific geometrical rules, certain gradient directions are introduced. As a result, gradient directions can categorize windows from different periods. Accordingly, the features are extracted with the scale-invariant feature transform (SIFT), whereas the bag-of-words algorithm classifies the final extracted result. The database includes 400 images with different resolutions. Experimental results approach a high classification rate of 95.16%.

Shalunts, G. et al. (2012) [40] discuss the need to not only classify buildings based on their architectural style but also on their structural elements. Accordingly, a classification pipeline is proposed to categorize Gothic and Baroque architectural elements. The scale-invariant feature transform (SIFT) algorithm provides invariance to scale and orientation; the bag of visual words (BoW) and the k-means clustering construct a model which classifies and extracts information on gradient directions. Each shape of the tracery, pediment, and balustrade classes includes specific gradient directions. Simultaneously, a data set of 520 images along with their bounding boxes depicts the necessity for optimal training during the learning phase. To conclude, a 96.67% successful classification rate is obtained on a testing data set of 420 images.

Shalunts, G. et al. (2017) [41] propose a methodology for distinguishing faces in human sculptures. Their study includes three experiments. First, they conduct their experiments by establishing the OpenCV Viola–Jones face detector on their custom data set. However, real-world faces cannot be used properly for the current study, owing to the problem of low accuracy. In the second place, new cascade classifiers are trained to detect the sculpted face based on a data set containing 700 photographs with 1608 faces. Augmentation techniques expanded the initial data set. Finally, the last experiment compares the two classifiers on a test data set, resulting in an F-measure of 0.90 using the custom classifier and in an F-measure of 0.73 using the OpenCV face detector.

The contact with the work of Shalunts, G. et al. was instrumental in deciding our research on CH, even though they did not use deep learning algorithms and focused on individual architectural elements. However, their diffusion and success were so influential that they inspired us to experiment in the relevant subject.

In [42], the use of convolutional neural networks is abandoned, as the authors Mercioni, M. A. and Holban, S. (2018) propose data mining techniques to determine the architectural style of an input image. The database consists of 100 images from Timisoara, grouped into five classes: Baroque, eclectic, secession, Moorish, and Byzantine buildings. Content-based image retrieval (CBIR) and local binary pattern (LBP) systems or clustering algorithms aim to classify efficiently architectural buildings. The proposed system is fast and accurate, and its applicability was measured by applying Euclidean and Manhattan distances on the custom data. Moreover, data retrieved with high accuracy tend to share the same texture, shape, and color components that preexist in the mind of culture heritage researchers. However, high performance is achieved when images have the same texture, shape, and color, and the quality or effectiveness of the data set plays a critical role in this system.

Llamas, J. et al. (2016) [43] contribute to the digital documentation of cultural heritage by developing convolutional neural networks for the INCEPTION European project. This project aims to preserve and protect cultural heritage assets by establishing modern-day technologies. In particular, a pre-trained convolutional neural network is configured by replacing the final layer of the network. As a result, the new output and classes are included (10 categories of elements with cultural heritage interest). Subsequently, a data set with over 10,000 images from Flickr trains the network. The authors train it with variations in the number of iterations and the learning rate. Consequently, they achieve accuracy that reaches a 92% rate. The authors claim that the results of the proposed method are very promising, as the time needed to classify assets is shorter compared to the manual methods. For future work, they are looking to introduce new categories of elements (arches, altars, frescos, etc.) and also to train other networks for classifying new kinds of categories, e.g., artistic styles and historical periods; they admit that this new task will require more computational power and a bigger data set.

Schmitz, M. and Mayer, H. (2016) [44] propose a method for an automatic semantic facade segmentation and interpretation. The authors depict the advantages of the transfer learning technique. In particular, when a part of a convolutional neural network already exists, large data sets are unessential. A specific part of the introduced method is based on the AlexNet in order to segment and classify elements of a facade. In addition, the facade, the door, the window, and other elements construct the classes. Moreover, an accurate data set, based on the eTRIM images for training and validation, trains the proposed architecture. Data augmentation techniques enlarge the data set by rescaling the images. During the validation process, a sixfold cross-validation was successfully used.

Pesto, C. (2017) [45] indicates the use of computer vision to automatically understand architectural styles and their wide range of applications by using real-world real-estate listing photography, claiming that it is strong enough to apply to real-life applications. The author also claims that, until then (2017), minimal research has been conducted on this specific area, where convolutional neural networks classify and localize U.S. houses. In particular, three different algorithms including an end-to-end trained model, ResNet-18, and ResNet-34 simplify the aforementioned procedures. The data set of 2500 images from a real estate photography list is personally examined by the author (the images were all collected manually from Zillow.com); it contains at- or near-ground houses, the same houses from different angles or illuminations, and the front facades of the houses. Each one of the five classes is balanced with 500 images of the U.S. houses. The paper indicates that input sizes of 256 × 256 are inadequate for this task. For that reason, several image size experiments finalize the superiority of ResNet-34 with a 512 × 512 dimension. Altogether, a total loss of 0.51 and 0.56 is acquired after a successful evaluation with the test set. The author was able to achieve a correct classification rate of 79.8%, and a 0.710 intersection-over-union localization score on the test set, using ResNet-34 as a feature extractor.

Multiple techniques and optimizations for convolutional neural networks are studied in [46]. In particular, the authors Guo, K. and Li, N. (2017) describe the effects of changes in the architecture of a CNN when trained by a data set with different architectural styles from Wikipedia. Two different data sets describe 10 and 25 different styles, respectively. The number of each category of pictures ranges from 60 to 300; in short, a total of about 5000 architectural style pictures was used as a data set. In this paper, multiple changes are applied in traditional deep learning models including the LeNet-5 framework. Specifically, the classification rate varies depending on dropout implementations and different activation functions, including the sigmoid, the rectified linear unit, tanh, etc., as well as random sampling and drop-connect.

Laupheimer, D. and Haala, N. (2018) [47] aim to contribute to exporting semantic information from 3D urban models. Accordingly, an end-to-end convolutional network is proposed to classify facades by resolving occlusions, illumination, angles, and orientation. The data set includes labeled data with images from Google Street View to train the proposed model by using various CNNs (VGG16, VGG19 (SIMONYAN and ZISSERMAN 2014), Resnet50 (HE et al. 2016), InceptionV3 (SZEGEDY et al. 2016), self-designed networks). However, the authors not only aim to classify images but also want to understand which features are important for the final decision. Consequently, the class activation maps (CAMs) are introduced to visualize the classification criteria in each image. The considered classes include commercial, hybrid, residential, special-use, and under-construction facades of buildings. The research work reaches an overall accuracy of approximately 64%. The residential class undoubtedly has the best accuracy of 98.57, which is encouraging because most of the buildings fall into this category.

Li, Y. et al. (2018) [48] study the availability of methods that estimate a building’s age. With respect to current databases, a new entry containing the age of a building may contribute further to greater cultural heritage awareness, as, according to the authors, building age estimation from images has not been studied sufficiently in the research community. This paper proposes the use of convolutional neural networks such as AlexNet, ResNet, and DenseNet to extract features from the required data set from Google Street View images from the North and West Metropolitan Region (NWMR) in Victoria, Australia. Finally, the support vector regression extracts the building’s age from the input data. Consequently, among the different CNN models, the DenseNet develops the best accuracy for the specific purpose. According to the authors, the complex appearances, materials, components, and styles of buildings allow the deep learning to acquire interesting figures.

Zou, Z et al. (2019) [49] aim to contribute to the protection and the maintenance of ancient buildings by applying deep learning algorithms. Additionally, they propose the use of convolutional neural networks with a manually collected data set and their tuning for inspecting historical buildings. The procedure takes place in the Forbidden City to detect the missing components and consequently abandons the respective human activity. Specifically, the Faster R-CNN architecture is trained and configured for detecting the aforementioned objects in 2D images. This methodology can be applied with high accuracy at the test data, especially when the distance from the camera is characterized as close or median. To conclude, future work may produce a fully automated inspection system based on the principles of machine learning.

Dautov, E. and Astafeva, N. (2021) [50] state that the recognition of architectural styles has not been thoroughly investigated in the modern literature due to their similarity. They introduce the 15 most famous architectural styles including Art Nouveau, Bauhaus, Gothic, Greek revival architecture, Palladian, etc. With respect to the above-mentioned styles, 100,000 images were collected and processed with augmentation techniques to train the neural network. In the second place, the Tensorflow and the Keras libraries established a suitable convolutional neural network by utilizing multiple convolutional and subsampling layers to solve the specific classification problem. Consequently, the accuracy of the proposed classifier was 0.7652 after 50 training epochs.