Multi-Class Defect Detection
Defects in residential building façades affect the structural integrity of buildings and degrade external appearances. Defects in a building façade are typically managed using manpower during maintenance. This approach is time-consuming, yields subjective results, and can lead to accidents or casualties. To address this, we propose a building façade monitoring system that utilizes an object detection method based on deep learning to efficiently manage defects by minimizing the involvement of manpower. The dataset used for training a deep-learning-based network contains actual residential building façade images. Various building designs in these raw images make it difficult to detect defects because of their various types and complex backgrounds.
A building should exhibit good performance in supporting the activities of its occupants. Among the various types of buildings, residential buildings should perform particularly well because occupants spend most of their time in them . Hence, it is crucial to minimize defects to maintain the performance of residential buildings . In particular, building façades are considered essential elements of buildings because they influence the appearance, structural safety, and insulation of the buildings, but also play the role of an exterior shield against weather and pollution . However, continuous exposure to poor environmental conditions during a long service life accelerates aging relatively faster compared with other building components . This phenomenon is eventually manifested in various types of defects on the building façade . If various defects in the building façade are ignored, they may result in shortening the service life, damage to appearance, and increased maintenance costs . Ultimately, it is ideal to prevent all types of defects in the design or construction stages, but this is a very difficult goal to achieve. Thus, there is a need for a method to effectively monitor defects in the maintenance phase and actively respond to the occurrence of the defects .
However, traditional defect management is associated with various issues, such as the subjectivity of the results arising from human-centered inspection, time consumption, occurrence of human causalities, and an increase in labor costs . Therefore, it is necessary to develop a technology that can continually and automatically monitor defects in residential buildings that minimize the dependence on manpower . Furthermore, there are various types of defects in residential buildings , and each defect type in the real world appears in an irregular pattern . To consider the characteristics of these defects, automated defect monitoring technology should be able to simultaneously detect and effectively classify various types of defects in image data.
Deep learning techniques are data-driven methods that do not require rules. The process of building a model only needs to select a suitable network structure, a function to evaluate the model output, and a reasonable optimization algorithm . Deep learning techniques are driving advances in computer vision to tackle the drawbacks of classical defect detection models that allow the automatic capture of intricate structures of large-scale data with models comprising multiple processing layers . Several previous studies have attempted to apply deep-learning methods to detect cracks in various structures and defects in sewer pipes . However, as the residential building façade is designed in various ways, we should be able to simultaneously classify various types of defects that appear in complex backgrounds.
2. Literature Review
As the building ages owing to various factors, it is crucial to continually inspect its various defects. From this perspective, several studies have been conducted to identify an efficient building inspection plan . Kim et al.  presented a probabilistic approach to establish an optimum inspection/repair strategy for Reinforced Concrete structures subjected to pitting corrosion. Liao  proposed a method for the development of inspection strategies for the construction industry. Pires  presented a method to facilitate performance assessment and analyzed the degradation of building envelopes with a focus on painted finishes. Bortolini et al.  presented a building inspection system to evaluate the technical performance of existing buildings based on the consideration of the entire buildings and the interdependence of their parts. Based on a review of literature on these topics, it has been shown that the majority of the publications focused on inspection strategies or maintenance plans. However, as described above, the traditional human resource-oriented inspection has its own limitations. Thus, it is necessary to develop a technology that can continually and automatically monitor the building defects.
To this end, several studies proposed structural health monitoring (SHM) techniques. In general, the SHM system employs a vibration-based structural system identification technique using a numerical method . Rabinovich et al.  developed a robust computational tool based on a combination of the extended finite element method (XFEM) and genetic algorithm (GA) to accurately detect and identify cracks in two-dimensional structures. Chatzi et al.  improved the XFEM-GA detection model by adding a novel genetic algorithm that accelerates the convergence of the scheme, and a generic XFEM formulation of an elliptical hole. Cha et al.  proposed a hybrid multi-objective genetic algorithm as a damage detection method to solve inverse problems to minimize the difference in the modal strain energy (MSE) in each structural element. However, fundamentally, the SHM system has various limitations, such as cost issues, compensation for environmental impacts, and noise signals, because of the installation of multiple sensors. Further, as the SHM system monitors only the structural damage, it has a drawback in that it cannot detect various types of defects, such as cracks, water leakage, detachment, corrosion, and efflorescence .
Therefore, several studies have been conducted on defect detection methods based on image processing techniques (IPT) . Laofor et al.  presented a defect detection and quantification system to augment subjective visual quality inspections in architectural work based on the specification of defect positions and the quantification of defect values. This method is able to use defect feature analysis to quantify the defect value from digital images using a digital image processing technique. Liao et al.  proposed a digital image recognition algorithm that consisted of three different detection techniques: K-means in H, DCDR in RGB, and DCDR in H, to improve the detection accuracy of the rusted areas on steel bridges. Shen et al.  developed a Fourier-transform-based steel bridge coating defect-detection approach (FT-DEDA) that makes use of the fact that the differences between background pixels are not as large as the differences between defect pixels to detect their existence. However, as image data obtained in the real world are quite diversified, IPT using prior knowledge are limited in recognizing defects in image data .
The deep learning technique been has intensively researched in the field of image recognition and can address these issues of IPT. In the construction field, there are several studies that have used deep learning to monitor defects in civil structures . Li et al.  proposed an automated defect detection and classification method from closed-circuit television (CCTV) inspections based on a deep convolutional neural network (DCNN) that takes advantage of the large volume of inspection data. Dung et al.  proposed a vision-based method for concrete crack detection and density evaluation using a deep fully convolutional network (FCN). Yang et al.  developed a transfer learning method based on multiple DCNN knowledge for crack detection. Yin et al.  proposed an automated defect detection system with an object detector based on a convolutional neural network (CNN), commonly known as the YOLOv3 network. There are differences among civil structures and buildings as documented on previous studies depending on the characteristics of the facilities. That is, as the building façade is made up of various shapes, it is critical to recognize specific defects within image data. Furthermore, as the types of defects are diverse and their shapes are irregular, there is a need for a model that can simultaneously classify various types of defects.
Multiple defects occur in various locations in actual buildings. To minimize the negative effects of these defects on the sustainability of buildings, there is a need for a technology that can efficiently monitor defects. In this study, we aimed to simultaneously detect various types of defects in the building façade in the real world using the Faster R-CNN model. The application performance of this model was verified through the collected data, and the average performance of each defect type was approximately 60% based on AP (IoU = 0.5). This is considered to be a meaningful result, justifying the possibility of multiple defect detection using a deep learning model.
In fact, various studies exist that detect defects based on deep learning, but most of them detected a single defect in laboratory conditions or focused on civil structures. However, this study used real-world building defect image data. That is, it is not a dataset built under refined conditions. We used real-world data where irregularities always exist, and various image interferences occur. Furthermore, unlike civil structures, the building façade is composed of various shapes and colors. This means that the background irregularities of image data are quite common. Thus, detecting defects is very challenging.
In general, defect management in a building is a way of dealing with defects that have already occurred. However, this method has limitations in minimizing performance reduction and maintenance cost increase due to defects. To solve these problems, it is necessary to check defects frequently during the maintenance phase, but the existing manpower-oriented defect inspection method is costly. Unmanned defect inspection techniques associated with the model proposed in this study can solve these problems. However, in order to manage the building efficiently, a variety of additional technologies need to be developed along with the deep learning-based MultiDefectNet proposed in this study. In other words, various technologies, such as unmanned aerial vehicle technology, defect location detection technology, and durability assessment technology, need to be linked to each other in order to develop unmanned defect inspection technology.
Also, it is necessary to expand the database (DB) for training and make it well-balanced to increase the accuracy of the model proposed in this study. In addition, it is considered that there is a need for more research on CNN architecture or data preprocessing that can distinguish image backgrounds from defects based on the considerations of the characteristics of buildings that feature various façade shapes.
The entry is from 10.3390/su12229785
- Kim, S.-S.; Yang, I.-H.; Yeo, M.-S.; Kim, K.-W. Development of a housing performance evaluation model for multi-family residential buildings in Korea. Build. Environ. 2005, 40, 1103–1116.
- Meier, J.R.; Russell, J.S. Model process for implementing maintainability. J. Constr. Eng. Manag. 2000, 126, 440–450.
- Ikpo, I.J. Maintainability indices for public building design. J. Build. Apprais. 2009, 4, 321–327.
- Chew, M.Y.L. Maintainability of Facilities: For Building Professionals; World Scientific: Hackensack, NJ, USA, 2010.
- Lee, J.-S. Value engineering for defect prevention on building façade. J. Constr. Eng. Manag. 2018, 144, 04018069.
- Das, S.; Chew, M.Y.L. Generic method of grading building defects using FMECA to improve maintainability decisions. J. Perform. Constr. Facil. 2011, 25, 522–533.
- Guo, J.; Wang, Q.; Li, Y.; Liu, P. Façade defects classification from imbalanced dataset using meta learning-based convolutional neural network. Comput. Aided Civ. Infrastruct. Eng. 2020, 35, 1–16.
- Kim, S.; Frangopol, D.M.; Zhu, B. Probabilistic optimum inspection/repair planning to extend lifetime of deteriorating structures. J. Perform. Constr. Facil. 2010, 25, 534–544.
- Love, P.E.D.; Smith, J. Toward error management in construction: Moving beyond a zero vision. J. Constr. Eng. Manag. 2016, 142, 04016058.
- Liu, Z.; Cao, Y.; Wang, Y.; Wang, W. Computer vision-based concrete crack detection using U-net fully convolutional networks. Autom. Constr. 2019, 104, 129–139.
- Yang, Y.-S.; Yang, C.-M.; Huang, C.-W. Thin crack observation in a reinforced concrete bridge pier test using image processing and analysis. Adv. Eng. Softw. 2015, 83, 99–108.
- Abudayyeh, O.; Al Bataineh, M.; Abdel-Qader, I. An imaging data model for concrete bridge inspection. Adv. Eng. Softw. 2004, 35, 473–480.
- Abdel-Qader, I.; Pashaie-Rad, S.; Abudayyeh, O.; Yehia, S. PCA-based algorithm for unsupervised bridge crack detection. Adv. Eng. Softw. 2006, 37, 771–778.
- Macarulla, M.; Forcada, N.; Casals, M.; Gangolells, M.; Fuertes, A.; Roca, X. Standardizing housing defects: Classification, validation, and benefits. J. Constr. Eng. Manag. 2013, 139, 968–976.
- Pan, W.; Thomas, R. Defects and their influencing factors of posthandover new-build homes. J. Perform. Constr. Facil. 2015, 29, 04014119.
- Cha, Y.-J.; Choi, W.; Büyüköztürk, O. Deep learning-based crack damage detection using convolutional neural networks. Comput. Aided Civ. Infrastruct. Eng. 2017, 32, 361–378.
- Pan, Y.; Zhang, G.; Zhang, L. A spatial-channel hierarchical deep learning network for pixel-level automated crack detection. Autom. Constr. 2020, 119, 103357.
- Li, D.; Cong, A.; Guo, S. Sewer damage detection from imbalanced CCTV inspection data using deep convolutional neural networks with hierarchical classification. Autom. Constr. 2019, 101, 199–208.
- Dung, C.V.; Anh, L.D. Autonomous concrete crack detection using deep fully convolutional neural network. Autom. Constr. 2019, 99, 52–58.
- Yang, Q.; Shi, W.; Chen, J.; Lin, W. Deep convolution neural network-based transfer learning method for civil infrastructure crack detection. Autom. Constr. 2020, 116, 103199.
- Yin, X.; Chen, Y.; Bouferguene, A.; Zaman, H.; Al-Hussein, M.; Kurach, L. A deep learning-based framework for an automated defect detection system for sewer pipes. Autom. Constr. 2020, 109, 102967.
- Liao, C.-W. Optimal Inspection Strategies for Labor Inspection in the Construction Industry. J. Constr. Eng. Manag. 2015, 141, 04014073.
- Pires, R.; de Brito, J.; Amaro, B. Inspection, diagnosis, and rehabilitation system of painted rendered façades. J. Perform. Constr. Facil. 2015, 29, 04014062.
- Bortolini, R.; Forcada, N. Building inspection system for evaluating the technical performance of existing buildings. J. Perform. Constr. Facil. 2018, 32, 04018073.
- Rabinovich, D.; Givoli, D.; Vigdergauz, S. XFEM-based crack detection scheme using a genetic algorithm. Int. J. Numer. Methods Eng. 2007, 71, 1051–1080.
- Chatzi, E.N.; Hiriyur, B.; Waisman, H.; Smyth, A.W. Experimental application and enhancement of the XFEM-GA algorithm for the detection of flaws in structures. Comput. Struct. 2011, 89, 556–570.
- Cha, Y.-J.; Buyukozturk, O. Structural Damage Detection Using Modal Strain Energy and Hybrid Multiobjective Optimization. Comput. Aided Civ. Infrastruct. Eng. 2015, 30, 347–358.
- Laofor, C.; Peansupap, V. Defect detection and quantification system to support subjective visual qualityinspection via a digital image processing: A tiling work case study. Autom. Constr. 2012, 24, 160–174.
- Liao, K.-W.; Lee, Y.-T. Detection of rust defects on steel bridge coatings via digital image recognition. Autom. Constr. 2016, 71, 294–306.
- Shen, H.-K.; Chen, P.-H.; Chang, L.-M. Automated steel bridge coating rust defect recognition method based on color and texture feature. Autom. Constr. 2013, 31, 338–356.