Structural Health Monitoring System Based on Deep Learning: Comparison
View Latest Version
Please note this is a comparison between Version 2 by Wendy Huang and Version 1 by Ayesha Munira Chowdhury.

Concrete stands as the most widely used construction material globally due to its versatility, encompassing applications ranging from pavement, multifloor structures, and bridges to dams. However, these concrete structures endure structural stress and require close monitoring to prevent accidents and ensure sustainability throughout their complete life cycle. Artificial intelligence (AI) and computer vision (CV) have demonstrated considerable potential in diverse applications within construction engineering, including structural health monitoring (SHM) and inspection processes such as crack and damage detection, as well as rebar exposure. While it is undeniable that CV and deep learning models are transforming the construction industry by offering robust solutions for complex scenarios, there remain numerous challenges pertinent to their applications that require attention. This paper aims to systematically and critically review the literature of the past decade on the application of deep learning models in the construction industry for SHM purposes in concrete structures. The review delves into proposed methodologies and technologies while identifying opportunities and challenges associated with these applications in practice. 

  • concrete
  • structural health monitoring (SHM)
  • deep learning
  • damage identification
  • damage quantification

1. Introduction

Concrete is the most important and demanding construction material [1], and concrete structures have been influencing the construction industry for decades [2]. However, an ever-growing number of concrete structures worldwide are entering the aging phase [3]. Due to various factors, such as weather and environmental conditions, chemical reactions, and external and internal stresses, concrete structures are often subject to defects such as cracks, efflorescence, spalling, bar exposure, etc., and fail to meet the expected life cycle, aging earlier than expected [2]. The idea of structural health monitoring (SHM) first emerged in the early 2000s. Although initially, the sole focus of SHM was to monitor concrete bridges, in present times, it is defined as the method of continuously evaluating and assessing the condition and performance of any concrete structures, such as buildings, bridges, dams, pipelines, and other infrastructure, throughout their operational lifespan [4][5][6]. The objective of SHM is to detect any damage, deterioration, or changes in structural properties that could potentially compromise the safety, functionality, or longevity of the concrete structure and is crucial in maintaining structures in optimal condition [2]. The traditional methods for the SHM process primarily involve manual inspection, which is heavily dependent on the expertise of the inspector. However, these methods present various challenges, including time-consuming operation, varying subjectivity, or difficulties in inspecting components at elevated heights in tunnels/road pavement in busy traffic conditions [2][7]. Therefore, there is a pressing need for an innovative and precise inspection approach to effectively monitor the health condition of structures that can overcome the mentioned limitations.
Researchers in the construction engineering field have recognized the immense potential and innovative technological strides resulting from the utilization of deep learning methods [8][9]. Consequently, numerous initiatives have been undertaken to apply deep learning techniques to structural health monitoring (SHM) of concrete infrastructure [10]. In the following sections, deep-learning-based research in the SHM domain is delved into, with a specific focus on two facets: (1) damage identification and (2) concrete condition assessment.

2. Damage Identification

At the heart of any SHM system lies its capacity to conduct damage identification. Damage refers to alterations in a material’s physical characteristics caused by ongoing deterioration or a singular event affecting a structure. Such changes have the potential to compromise the performance and structural integrity of the concrete [11]. One limitation of applying deep learning techniques is that they require a large and annotated database, which is not always available, especially in the concrete research area. However, the application of transfer learning can eliminate this problem, allowing an existing deep learning model to be retrained with smaller amounts of new data [12]; accordingly, an increasing number of applications of deep learning models in concrete research and SHM have been reported.
For example, Gopalakrishnan et al. [13] applied transfer learning to a pretrained VGG-16 model for crack detection in hot-mix asphalt and Portland cement concrete-based pavement. Kolar et al. [14] also applied transfer learning to VGG-16 model to detect safety guardrails to promote on-site safety inspection.
Real-world scenarios often limit the applications of deep learning models at actual construction sites because of lighting and shadow issues. Cha et al. [15] trained a CNN with a large database of 40k images under various lighting conditions and achieved 98% accuracy in detecting concrete cracks. The reseauthorchers later compared the performance of the proposed CNN using Canny and Sobel edge detection methods. Tong et al. [16] integrated three CNNs to perform recognition, localization, and feature extraction tasks, enabling the 3D reconstruction of hidden pavement cracks with images of cracks collected using ground-penetrating radar (GPR). Figure 1 demonstrates the proposed pipeline of 3D reconstruction pavement cracks with GPR data. Gibert et al. [17] combined multiple detectors for automatic inspection of railway tracks.
Constrmater 04 00005 g003
Figure 1.
Pipeline for the 3D reconstruction models of pavement cracks
[16]
.
Assessment of post-disaster damage in concrete to provide valuable insights for necessary follow-up actions is another application of deep learning in the SHM area. Davoudi et al. [18] applied image segmentation to determine the state of the damage in reinforced concrete beams and slabs. Lattanzi et al. [19] also applied image segmentation via the MATLAB Image Processing Toolbox for the extraction of features from images of damaged reinforcement columns to estimate the maximum lateral displacement using a regression model. Spalling, a common type of damage in concrete structures, is another area of application of deep learning in SHM practice. For example, Dawood et al. [20] presented a hybrid model combining image processing and machine learning techniques to identify spalling distress in subway stations. Yeum et al. [21] applied AlexNet to both collapse classification and spalling detection in post-disaster analysis of concrete structures.
Kim and Cho [22] introduced a method utilizing unmanned aerial vehicles (UAVs) and R-CNN to detect cracks in old concrete bridges. They applied transfer learning to R-CNN, using crack images to enhance crack detection, and later, image processing was employed to quantify the identified cracks. Kang and Cha [23] also applied UAV-based damage detection with deep learning; however, they addressed one important issue, which is that UAVs often require a skilled pilot and autonomous flight with GPS in certain complex locations of structures, such as indoors or beneath bridges. The reseauthorchers proposed an ultrasonic beacon for UAV navigation in GPS-incompatible environments.
Xue and Li [24] devised a three-tiered deep learning framework including an FCN, RPN, and position-sensitive region of interest pooling for identification of damage in tunnel linings. Hoang et al. [25] also compared the performance of a CNN with Sobel and Canny edge detection algorithms, as previously reported by Cha et al. [15], for a cyclic survey of pavement cracks. Similarly, Dorafshan et al. [26] compared the performance of four edge detection methods with CNNs in detail for crack detection in concrete. Four common edge detection methods in the spatial domain (Roberts, Prewitt, Sobel, and Laplacian of Gaussian) and two in the frequency domain (Butterworth and Gaussian), as well as the AlexNet model in three modes of training (trained, transfer learning, and without training), were compared, and the reseauthorchers concluded that AlexNet showed superiority over other methods.
AlexNet was used by Wang et al. [27] as well. The reseauthorchers utilized both AlexNet and GoogLeNet for the detection of various types of damage to masonry walls in historic structures, using sliding-window techniques to pinpoint concrete damage. Motivated by the ImageNet Challenge, Gao and Mosalam [28] proposed the concept of Structural ImageNet, with four intended tasks: component identification, spalling detection, damage condition evaluation, and damage type determination in concrete through the application of transfer learning in VGGNet (Visual Geometry Group).
Wu et al. [29] applied transfer learning to VGG16 and ResNet18 to detect two types of prevalent concrete surface defects, namely cracks and corrosion. Zhang et al. [30] proposed Faster R-CNN to determine the spatiotemporal information of the vehicles on bridges in order to determine the stress state and traffic densities. Wang and Cheng [31] proposed DilaSeg-CRF by integrating a CNN with a dense conditional random field (CRF) to improve the segmentation accuracy in sewer pipe defect detection, whereas Li et al. [32] addressed the issue of data imbalance in sewer damage detection by introducing a hierarchical classification approach to supervise the learning process at different levels. Zha et al. [33] applied transfer learning to ResNet (deep residual neural network) for eight types post-disaster concrete damage detection: scenario classification, damage detection, spalling detection, material identification, collapse detection, effected component identification, and damage level and type determination, which were categorized into binary or multiclasses according to the conditions. The reseauthorchers used the 2018 PEER Hub ImageNet Challenge distributed by the Pacific Earthquake Engineering Research Center to evaluate the proposed methodology.
Jang et al. [34] used transfer learning in GoogLeNet with hybrid images, combining vision and infrared thermography images to enhance crack detection in concrete structures. The reseauthorchers suggested the use of a UAV-mounted hybrid system comprising a vision camera, an infrared camera, and a continuous-wave line laser to capture images, particularly for large-scale structures, then used them for inspection of the respective structures. U-Net, which is famous for applications in biomedical image segmentation, was first applied by Liu et al. [35] to concrete crack detection and later compared with FCN using evaluation metrics such as precision and the size of the training set. The reseauthorchers applied U-Net for localization of concrete cracks under various lighting and background conditions. Khani et al. [36] investigated the impact of preprocessing on a concrete crack detection pipeline based on a CNN trained with 700 labelled gas turbine images. The reseauthorchers concluded that bilateral filtering improves the generalization ability of the suggested framework in cases with cracks on complex structures. Zhang et al. [37] argued that two-stage detectors such as Faster R-CNN and ResNet-101 have limited practical applications due to their slow speeds. The reseauthorchers used a single-stage detector (SSD), YOLOv3 (You Only Look Once), to detect multiple types of concrete bridge damage, such as cracks, pop-outs, spalling, exposed rebar, etc.
Liu et al. [38] argued that the motion blur from excessive vibration in UAVs limits the accuracy of crack detection in high-rise buildings. The reseauthorchers introduced a generative adversarial network (GAN) that incorporates the concept of localized skip connections that recognize the correlation between blurred and sharpened crack images. The proposed method was validated through experiments involving the investigation of skip connections in deblurring and compared with a state-of-the-art deblurring model. Kim et al. [39] applied transfer learning to Mask R-CNN for automatic concrete damage detection and localization in four classes—cracks, efflorescence, rebar exposure, and spalling—using an instance segmentation approach.
Mondol et al. [40] applied Faster R-CNN to detect post-disaster damage like surface cracks, exposed rebar, and buckled rebar using image data collected from concrete structures damaged during past earthquakes in Nepal (2015), Taiwan (2016), Ecuador (2016), Erzincan (1992), Duzce (1999), Bingol (2003), Peru (2007), Wenchuan (2008), and Haiti (2010). Deng et al. [41] introduced LinkASSPNet (LinkNet with atrous spatial pyramid pooling) and conducted a performance comparison with U-Net and LinkNet in the context of concrete bridge surface damage detection. Notably, this restudyearch stands out, as the models were trained on a relatively small dataset. It purports to address the challenge of variations in labeling areas among labelers in pixel-wise image segmentation tasks.
Zheng and Zhang [42] proposed a crack detection model for concrete based on image segmentation tasks and the FCN, R-CNN, and RFCN (Richer Fully Convolutional Networks) models. The training included a wide range of image data, including images of buildings, bridges, dams, roads, etc. Karaaslan et al. [43] proposed a combination of an SSD-based VGG-16 model and a modified SegNet, where the former detects regions of interest related to damage, such as cracks or spalling, upon verification by the respective inspector, and the latter then applies segmentation to the damage for further analysis. Miao et al. [44] proposed U-Net-based Damage-Net for semantic segmentation of seismic damage in reinforced concrete structures, where the reseauthorchers adjusted the padding size and stride size to ensure that the input and output size were the same, which is usually not the case in U-Net. The proposed Damage-Net receives its encoder from the convolutional layers of VGG-16, allowing it to adapt transfer learning and to be trained on a comparatively smaller dataset. Based on this architecture, two individual models were proposed: Crack-Net for detecting cracks, and 4Category-Net for identifying four additional damage categories, namely concrete spalling and crushing, reinforcement exposure, buckling, and fracture.
Qiao et al. [45] proposed EMA-DenseNet, a combination of densely connected convolutional networks (DenseNet) integrated with an expected maximum attention (EMA) module in the last pooling layer for the detection of surface damage in concrete bridges in a set of images collected from multiple bridges located in Zhejiang (China). The reseauthorchers claimed that the proposed model performs better than FCN, SegNet, DeepLab v3+, and SDDNet. Huang et al. [46] proposed a software system for damage detection in subway tunnels by integrating four separate functions: image fusion to splice the images acquired by different cameras, image preprocessing to remove background noise and other preprocessing tasks, damage identification performed by the R-CNN model and a data platform for evaluation by the respective personnel. Arya et al. [47] proposed a concrete pavement damage dataset consisting of 26,620 data point from multiple countries and investigated how the demographics of the damage data affect the model performance based on a YOLO-v5/YOLO-v4/cascade R-CNN-based ensemble model. Cui et al. [48] proposed an improved YOLO-v3 model for the detection of erosion damage that achieved up to a 75% mean average precision value.
Pozzer et al. [49] compared the performance of different models, i.e., VGG-16, ResNet-18, ResNet-50, MobileNet-V2, Xception, etc., in detecting concrete defects such as delamination, cracks, spalling, and patches in thermographic and regular images at varying distances and under varying conditions using semantic segmentation. Andrushia et al. [50] implied that most research on damage detection in concrete structures does not consider the complex background or environmental effects and therefore proposed a U-Net with an encoder–decoder framework for thermal damage detection in concrete structures in the event of fires.
Munawar et al. [51] introduced a cycle generative adversarial network (CycleGAN) with 16 convolution layers, providing additional support to refine predictions through guided filtering (GF) and conditional random fields (CRFs). The reseauthorchers applied this model to inspect mid- to high-rise concrete structures constructed during the 2000s using segmentation techniques and drones. Zou et al. [52] proposed a YOLOv4-based approach to the detection of multiple types of damage, including both fine and wide cracks, spalling, exposed and bucking rebars, etc., that was integrated in a graphical user interface (GUI) to streamline the assessment of structural damage in reinforced concrete (RC) buildings following an earthquake. Han et al. [53] proposed the use of a transfer-learning-based AlexNet and threshold segmentation to precisely locate cracks in concrete structures.
Tanveer et al. [54] compared and analyzed the performance of five semantic segmentation models (ENet, CGNet, ESNet, DDRNet-Slim23, and DeepLabV3+ (ResNet-50)). These models were categorized as lightweight and heavyweight based on the parameter count. The evaluation focused on on-site damage detection in concrete structures using edge computing devices such as smartphones, tablets, etc. Bai et al. [55] proposed an EfficientNet-V2-based model for component damage recognition, serving both structural health monitoring (SHM) and post-disaster assessment purposes. They also investigated the relationship between damage type, component damage level, and the structural safety state. Crognale [56] compared four different image processing techniques, namely Otsu-method thresholding, Markov random field segmentation, the RGB color detection technique, and the K-means clustering algorithm, in corrosion and crack detection based on a case study. Chen et al. [57] proposed an AlexNet-based multiclass damage detection method for reinforced concrete bridges in high-speed rail systems.
Wan et al. [58] proposed a BR-DETR model, a concrete bridge damage detection model based on detection transformers (DETR), with deformable Conv2D in place of convolution, as well as with an additional convolutional project attention layer after the self-attention layer. Zhu and Tang [59] introduced a DeepLabV3+ network architecture with Xception as the backbone to automatically estimate detailed crack information in hydraulic concrete structures. Huang et al. [60] proposed a Faster R-CNN with Res-Net101 as the backbone for detection of damage like cracks, spalling, and precipitates in hydraulic concrete structures.

3. Damage Quantification

Damage quantification is the next step after damage identification. Concrete damage quantification aims to determine the extent, severity, and specific characteristics of damage, such as cracks, spalling, corrosion, or other forms of deterioration. Although using deep learning for concrete damage quantification is still a relatively new concept in SHM, researchers are continuously generating new ideas to automate the quantification process, given the inherent challenges associated with this topic.
Kim et al. [61] proposed a UAV-based digital image processing system integrated with imaging and distance-sensing technology to determine the width and length of the cracks in concrete surfaces. Tong et al. [62] proposed a CNN-based method to calculate the mean texture depth (MTD) of pavement surfaces from 3D scan data, which was tested on four different highways in Shanxi, China. Huang et al. [63] studied lining damage in tunnels with a rapid detection and assessment analysis system developed by Nanjing HuoYang Hou Mdt InfoTech Ltd. The system includes a multichannel array of high-speed CCD (charged couple device) cameras to obtaining image data, multiple sensors to mitigate the impact of vehicle vibration on the tunnel, a multilayer lighting system, multiple positioning technology (reference object positioning technology + image positioning technology + mileage positioning technology + infrared laser positioning technology), and a computer vision approach for damage identification and analysis. Tayo et al. [64] presented a device capable of portable crack width calculation in concrete road pavement using pattern recognition based on multiple image processing technologies, such as graying, enhancement, filtering and denoising, binarization, segmentation, etc. Kim and Cho [65] proposed Mask R-CNN+image processing techniques for successful detection and quantification of concrete cracks with widths of 0.3 mm or more. Wei et al. [66] applied the same approach to concrete surface bughole segmentation and diameter measurement.
Beckman [67] applied Faster R-CNN to automatically and simultaneously detect and quantify concrete spalling in multiple locations within the same surface. The reseauthorchers used a depth camera to obtain the volume quantifications of the spalling damage. Park et al. [68] applied YOLO for both concrete crack detection and quantification (i.e., to determine the size of the cracks) in real time. The reseauthorchers used laser beams with integrated distance sensors for accurate measurement of the crack size. Bhowmick et al. [69] applied U-Net-based segmentation for concrete crack localization and binarization to estimate quantitating properties of cracks, like length, width, area, orientation, etc., from video data collected by a camera mounted on a UAV. Flah et al. [70] applied a deep learning technique to identify both structural and durability-related damage in structural members and assess the condition in a short time span by combing a Keras classifier with Otsu image processing. The proposed method can classify cracks; quantify them in terms of length, width, and angular orientation; and evaluate the severity of the damage.
Yuan et al. [71] proposed an inspection robot that transforms the quantification of concrete damage from a 2D plane to 3D space with stereo vision and a Mask R-CNN approach. The robot is based on four different sensors, with a monocular camera as a visual sensor, a stereo camera with a sensor for inertial measurement (IMU) of six degrees of freedom that can be mapped for panoramic image stitching, and a LiDAR sensor to measure the distance between the RC structure and the camera. Miao et al. [72] proposed a GoogLeNet-based transfer learning approach incorporating a novel sliding technique known as neighborhood scanning. This method aims at the detection, segmentation, and quantification of concrete cracks, achieving an average relative error of 14.58% in crack calculation.
Song et al. [73] introduced a deep learning approach for crack segmentation and quantification utilizing MobileNetV1 and ResNet50, along with DeeplabV3+ and U-Net. MobileNetV1 and ResNet50 handle crack classification, while DeeplabV3+ and U-Net manage panoramic crack segmentation (Figure 2). The quantitative information of the crack was subsequently acquired by multiplying the actual physical size corresponding to the unit pixel, assuming the length of a single pixel as the unit length. Kumarapu et al. [74] introduced UAVIC, a system that integrates UAVs with an image processing technique, i.e., digital image correlation. This approach is employed for damage quantification on scaled bridge girders.
Constrmater 04 00005 g004
Figure 2.
Before and after concrete crack segmentation
[73]
.
Bae et al. [75] proposed a computer-vision-based crack quantification algorithm using decision making based on statistical methods for accurate estimation and quantification of damage based on an image dataset of concrete building structures in South Korea. Li et al. [76] proposed a ResNet50-based improved You Only Look At CoefficienTs for Edge devices (YolactEdge) combined with digital image processing techniques for damage identification and quantification in hydraulic tunnels.

4. Suggested Future Frameworks

The number of deep-learning-based applications in concrete research is rapidly growing, especially in the SHM area. Numerous applications have been reported with respect to both concrete damage identification and quantification. The application and integration of stereo cameras and sensors, such as LiDAR and laser sensors, have made deep learning applications for damage quantification. Many researchers have applied various image processing techniques rather than integration with depth cameras or sensors. However, concrete cracks are very fine, so whichever system is adopted must precisely quantify a particular property (either length, width, or diameter). AlexNet, GoogleNet, Faster R-CNN, Mask R-CNN, U-Net, VGG, and YOLO models seem to be popular choices for damage identification. However, compared to other industries, construction falls behind in terms of adopting digitalization; therefore the application of deep-learning- and vision-based systems to monitor concrete health in the SHM area is still not sufficient in real practice, mostly due to the following issues:
  • Data shortage: Although transfer learning has made the adaptation of deep learning easier. Raw data often need to go through many stages of post processing, which is very time-consuming and labor-intensive. Also, there is a need for annotated datasets, which are essential for any deep learning training [77]. Most studies have been conducted using private datasets; making such datasets public would open multiple doors for researchers in the SHM domain for multiple applications. Although data augmentation plays an important role in dataset incrementation, applying various transformations to existing data, such as rotating, scaling, flipping, or cropping images, is insufficient for research in the SHM area. An alternative method involves utilizing generative adversarial networks (GANs), where a deep learning model comprising two distinct networks (namely a generator and a discriminator) is employed to generate synthetic image data instead of relying on real-world camera inputs only, as reported in [38][51]. 
  • Impact of the training data on overfitting: Transfer learning has undeniably simplified the application of deep learning models in structural health monitoring (SHM). However, the persistent challenge of overfitting can arise, particularly in instances where there is a paucity of image data. Deep learning models characterized by multiple layers and millions of parameters demand extensive tuning, as illustrated, for example, by the necessity of adjusting at least 100 million parameters in VGG-16 for crack detection [12]. It is imperative that the training data encompass diverse real-world scenarios, accounting for variations in background, lighting, and weather conditions, to ensure the model’s robustness and applicability.
  • Requirement for high-performance computers: Many deep learning techniques necessitate several days for training due to the extensive calculations involved in computing related training parameters, such as loss functions. Adequate hardware, including high-capacity hard disks, multiple GPUs/CPUs, and substantial memory, is essential for storing these calculations. Researchers should prioritize discovering optimized model structures with fewer parameters, facilitating their seamless adaptation in structural health monitoring (SHM) applications. 
  • Dealing with background noise: On the other hand, in addressing various background noises in images, researchers have implemented different morphological changes in the CNN architecture [31][38][41][44][45][51][58] to increase the detection accuracy. However, the source code is typically not publicly available. Researchers should be encouraged to make their source code publicly accessible, enabling other researchers to enhance the architecture further and, consequently, increase its applicability in actual practice. Due to the image resizing requirement of deep learning models to be trained on computers with average computing capacities, generalization abilities are often lost. For example, stains are a common issue in concrete structures and often incorrectly identified as cracks. To solve this issue, stains and similar could be categorized as another class [78] to improve the generalization abilities.

4. Suggested Future Frameworks

The number of deep-learning-based applications in concrete research is rapidly growing, especially in the SHM area. Numerous applications have been reported with respect to both concrete damage identification and quantification. The application and integration of stereo cameras and sensors, such as LiDAR and laser sensors, have made deep learning applications for damage quantification. Many researchers have applied various image processing techniques rather than integration with depth cameras or sensors. However, concrete cracks are very fine, so whichever system is adopted must precisely quantify a particular property (either length, width, or diameter). AlexNet, GoogleNet, Faster R-CNN, Mask R-CNN, U-Net, VGG, and YOLO models seem to be popular choices for damage identification. However, compared to other industries, construction falls behind in terms of adopting digitalization; therefore the application of deep-learning- and vision-based systems to monitor concrete health in the SHM area is still not sufficient in real practice, mostly due to the following issues:

  1. Data shortage: Although transfer learning has made the adaptation of deep learning easier, . Raw data often need to go through many stages of post processing, which is very time-consuming and labor-intensive. Also, there is a need for annotated datasets, which are essential for any deep learning training [129]. Most studies have been conducted using private datasets; making such datasets public would open multiple doors for researchers in the SHM domain for multiple applications.Although data augmentation plays an important role in dataset incrementation, applying various transformations to existing data, such as rotating, scaling, flipping, or cropping images, is insufficient for research in the SHM area. An alternative method involves utilizing generative adversarial networks (GANs), where a deep learning model comprising two distinct networks (namely a generator and a discriminator) is employed to generate synthetic image data instead of relying on real-world camera inputs only, as reported in [87,100]. 
  2. Impact of the training data on overfitting: Transfer learning has undeniably simplified the application of deep learning models in structural health monitoring (SHM). However, the persistent challenge of overfitting can arise, particularly in instances where there is a paucity of image data. Deep learning models characterized by multiple layers and millions of parameters demand extensive tuning, as illustrated, for example, by the necessity of adjusting at least 100 million parameters in VGG-16 for crack detection [61].  It is imperative that the training data encompass diverse real-world scenarios, accounting for variations in background, lighting, and weather conditions, to ensure the model’s robustness and applicability.
  3. Requirement for high-performance computers: Many deep learning techniques necessitate several days for training due to the extensive calculations involved in computing related training parameters, such as loss functions. Adequate hardware, including high-capacity hard disks, multiple GPUs/CPUs, and substantial memory, is essential for storing these calculations. Researchers should prioritize discovering optimized model structures with fewer parameters, facilitating their seamless adaptation in structural health monitoring (SHM) applications. 
  4. Dealing with background noise: On the other hand, in addressing various background noises in images, researchers have implemented different morphological changes in the CNN architecture [80,87,90,93,94,100,107] to increase the detection accuracy. However, the source code is typically not publicly available. Researchers should be encouraged to make their source code publicly accessible, enabling other researchers to enhance the architecture further and, consequently, increase its applicability in actual practice. Due to the image resizing requirement of deep learning models to be trained on computers with average computing capacities, generalization abilities are often lost. For example, stains are a common issue in concrete structures and often incorrectly identified as cracks. To solve this issue, stains and similar could be categorized as another class [131] to improve the generalization abilities.

Despite the challenges and limitations, the use of deep learning and computer vision technologies holds significant promise in structural health monitoring (SHM) and concrete research. A collaborative effort from researchers, scholars, and engineers in the construction, computer science, and civil engineering domains can establish more effective deep -earning-based SHM inspection systems for both damage identification and quantification, regardless of severity, delicacy of the damage, and the influence of the surrounding environment.

5. Conclusion

The aim of this research was to conduct a systematic review of the utilization of deep learning in the identification and quantification of concrete damage for SHM purposes. This study delved into the concepts and historical development of artificial intelligence (AI), computer vision (CV), and deep learning. With the aim of including the latest advancements in concrete research, the analysis was focused on studies spanning from 2017 to 2023, particularly those addressing vision-based crack identification, categorization, and measurement analysis. Additionally, we addressed four critical issues regarding the application of deep learning in the field of SHM. This research provides helpful insights that can aid in future applications of concrete damage identification and quantification with deep learning and guide researchers and engineers in respective applications. 

 

 

 

 

 

 

 

 

Despite the challenges and limitations, the use of deep learning and computer vision technologies holds significant promise in structural health monitoring (SHM) and concrete research. A collaborative effort from researchers, scholars, and engineers in the construction, computer science, and civil engineering domains can establish more effective deep -earning-based SHM inspection systems for both damage identification and quantification, regardless of severity, delicacy of the damage, and the influence of the surrounding environment.

References

  1. Moein, M.M.; Saradar, A.; Rahmati, K.; Mousavinejad, S.H.G.; Bristow, J.; Aramali, V.; Karakouzian, M. Predictive models for concrete properties using machine learning and deep learning approaches: A review. J. Build. Eng. 2023, 63, 105444.
  2. Wang, W.; Su, C.; Fu, D. Automatic detection of defects in concrete structures based on deep learning. Structures 2022, 43, 192–199.
  3. Pu, R.; Ren, G.; Li, H.; Jiang, W.; Zhang, J.; Qin, H. Autonomous Concrete Crack Semantic Segmentation Using Deep Fully Convolutional Encoder–Decoder Network in Concrete Structures Inspection. Buildings 2022, 12, 2019.
  4. Shibu, M.; Kumar, K.P.; Pillai, V.J.; Murthy, H.; Chandra, S. Structural health monitoring using AI and ML based multimodal sensors data. Meas. Sens. 2023, 27, 100762.
  5. Yuan, F.G.; Zargar, S.A.; Chen, Q.; Wang, S. Machine learning for structural health monitoring: Challenges and opportunities. Sens. Smart Struct. Technol. Civ. Mech. Aerosp. Syst. 2020, 11379, 1137903.
  6. Yoon, J.; Lee, J.; Kim, G.; Ryu, S.; Park, J. Deep neural network-based structural health monitoring technique for real-time crack detection and localization using strain gauge sensors. Sci. Rep. 2022, 12, 20204.
  7. Zhou, Z.; Yan, L.; Zhang, J.; Zheng, Y.; Gong, C.; Yang, H.; Deng, E. Automatic segmentation of tunnel lining defects based on multiscale attention and context information enhancement. Constr. Build. Mater. 2023, 387, 131621.
  8. DeVries, P.M.; Viégas, F.; Wattenberg, M.; Meade, B.J. Deep learning of aftershock patterns following large earthquakes. Nature 2018, 560, 632–634.
  9. Spencer, B.F., Jr.; Hoskere, V.; Narazaki, Y. Advances in computer vision-based civil infrastructure inspection and monitoring. Engineering 2019, 5, 199–222.
  10. Vodrahalli, K.; Bhowmik, A.K. 3D computer vision based on machine learning with deep neural networks: A review. J. Soc. Inf. Disp. 2017, 25, 676–694.
  11. Gomez-Cabrera, A.; Escamilla-Ambrosio, P.J. Review of machine-learning techniques applied to structural health monitoring systems for building and bridge structures. Appl. Sci. 2022, 12, 10754.
  12. Ye, X.W.; Jin, T.; Yun, C.B. A review on deep learning-based structural health monitoring of civil infrastructures. Smart Struct. Syst 2019, 24, 567–585.
  13. Gopalakrishnan, K.; Khaitan, S.K.; Choudhary, A.; Agrawal, A. Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection. Constr. Build. Mater. 2017, 157, 322–330.
  14. Kolar, Z.; Chen, H.; Luo, X. Transfer learning and deep convolutional neural networks for safety guardrail detection in 2d images. Autom. Constr. 2018, 89, 58–70.
  15. Cha, Y.J.; Choi, W.; Suh, G.; Mahmoudkhani, S.; Büyüköztürk, O. Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Comput. Aided Civ. Infrastruct. Eng. 2018, 33, 731–747.
  16. Tong, Z.; Gao, J.; Zhang, H.T. Recognition, location, measurement, and 3D reconstruction of concealed cracks using convolutional neural networks. Constr. Build. Mater. 2017, 146, 775–787.
  17. Gibert, X.; Patel, V.M.; Chellappa, R. Deep multitask learning for railway track inspection. IEEE Trans. Intell. Transp. Syst. 2016, 18, 153–164.
  18. Davoudi, R.; Miller, G.R.; Kutz, J.N. Computer vision based inspection approach to predict damage state and load level for RC members. In Proceedings of the 12th International Workshop on Structural Health Monitoring, Stanford, CA, USA, 10–12 September 2019.
  19. Lattanzi, D.; Miller, G.R.; Eberhard, M.O.; Haraldsson, O.S. Bridge column maximum drift estimation via computer vision. J. Comput. Civ. Eng. 2016, 30, 04015051.
  20. Dawood, T.; Zhu, Z.; Zayed, T. Machine vision-based model for spalling detection and quantification in subway networks. Autom. Constr. 2017, 81, 149–160.
  21. Yeum, C.M.; Dyke, S.J.; Ramirez, J. Visual data classification in post-event building reconnaissance. Eng. Struct. 2018, 155, 16–24.
  22. Kim, B.; Cho, S. Automated vision based detection of cracks on concrete surfaces using a deep learning technique. Sensors 2018, 18, 3452.
  23. Kang, D.; Cha, Y.J. Autonomous UAVs for structural health monitoring using deep learning and an ultrasonic beacon system with geo-tagging. Comput. Aided Civ. Infrastruct. Eng. 2018, 33, 885–902.
  24. Xue, Y.D.; Li, Y.C. A fast detection method via region based fully convolutional neural networks for shield tunnel lining defects. Comput. -Aided Civ. Infrastruct. Eng. 2018, 33, 638–654.
  25. Hoang, N.D.; Nguyen, Q.L.; Tran, V.D. Automatic recognition of asphalt pavement cracks using metaheuristic optimized edge detection algorithms and convolution neural network. Automat. Constr. 2018, 94, 203–213.
  26. Dorafshan, S.; Thomas, R.J.; Maguire, M. Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete. Constr. Build. Mater. 2018, 186, 1031–1045.
  27. Wang, N.; Zhao, Q.; Li, S.; Zhao, X.; Zhao, P. Damage classification for masonry historic structures using convolutional neural networks based on still images. Comput. Aided Civ. Infrastruct. Eng. 2018, 33, 1073–1089.
  28. Gao, Y.; Mosalam, K.M. Deep transfer learning for image-based structural damage recognition. Comput. Aided Civ. Infrastruct. Eng. 2018, 33, 748–768.
  29. Wu, R.T.; Singla, A.; Jahanshahi, M.R.; Bertino, E.; Ko, B.J.; Verma, D. Pruning deep convolutional neural networks for efficient edge computing in condition assessment of infrastructures. Comput. Aided Civ. Infrastruct. Eng. 2019, 34, 774–789.
  30. Zhang, B.; Zhou, L.; Zhang, J. A methodology for obtaining spatiotemporal information of the vehicles on bridges based on computer vision. Comput. Aided Civ. Infrastruct. Eng. 2019, 34, 471–487.
  31. Wang, M.; Cheng, J.C. A unified convolutional neural network integrated with conditional random field for pipe defect segmentation. Comput. Aided Civ. Infrastruct. Eng. 2020, 35, 162–177.
  32. Li, D.; Cong, A.; Guo, S. Sewer damage detection from imbalanced CCTV inspection data using deep convolutional neural networks with hierarchical classification. Autom. Constr. 2019, 101, 199–208.
  33. Zha, B.; Bai, Y.; Yilmaz, A.; Sezen, H. November. Deep Convolutional Neural Networks for Comprehensive Structural Health Monitoring and Damage Detection. In Proceedings of the 12th International Workshop on Structural Health Monitoring, Stanford, CA, USA, 10–12 September 2019.
  34. Zha, B.; Bai, Y.; Yilmaz, A.; Sezen, H. Deep learning–based autonomous concrete crack evaluation through hybrid image scanning. Struct. Health Monit. 2019, 18, 1722–1737.
  35. Liu, Z.; Cao, Y.; Wang, Y.; Wang, W. Computer vision-based concrete crack detection using U-net fully convolutional networks. Autom. Constr. 2019, 104, 129–139.
  36. Mohtasham Khani, M.; Vahidnia, S.; Ghasemzadeh, L.; Ozturk, Y.E.; Yuvalaklioglu, M.; Akin, S.; Ure, N.K. Deep-learning-based crack detection with applications for the structural health monitoring of gas turbines. Struct. Health Monit. 2020, 19, 1440–1452.
  37. Zhang, C.; Chang, C.C.; Jamshidi, M. Concrete bridge surface damage detection using a single-stage detector. Comput.-Aided Civ. Infrastruct. Eng. 2020, 35, 389–409.
  38. Liu, Y.; Yeoh, J.K.; Chua, D.K. Deep learning–based enhancement of motion blurred UAV concrete crack images. J. Comput. Civ. Eng. 2020, 34, 04020028.
  39. Kim, B.; Cho, S. Automated multiple concrete damage detection using instance segmentation deep learning model. Appl. Sci. 2020, 10, 8008.
  40. Ghosh Mondal, T.; Jahanshahi, M.R.; Wu, R.T.; Wu, Z.Y. Deep learning-based multi-class damage detection for autonomous post-disaster reconnaissance. Struct. Control Health Monit. 2020, 27, e2507.
  41. Deng, W.; Mou, Y.; Kashiwa, T.; Escalera, S.; Nagai, K.; Nakayama, K.; Matsuo, Y.; Prendinger, H. Vision based pixel-level bridge structural damage detection using a link ASPP network. Autom. Constr. 2020, 110, 102973.
  42. Zheng, M.; Lei, Z.; Zhang, K. Intelligent detection of building cracks based on deep learning. Image Vis. Comput. 2020, 103, 103987.
  43. Karaaslan, E.; Bagci, U.; Catbas, F.N. Attention-guided analysis of infrastructure damage with semi-supervised deep learning. Autom. Constr. 2021, 125, 103634.
  44. Miao, Z.; Ji, X.; Okazaki, T.; Takahashi, N. Pixel-level multicategory detection of visible seismic damage of reinforced concrete components. Comput. Aided Civ. Infrastruct. Eng. 2021, 36, 620–637.
  45. Qiao, W.; Ma, B.; Liu, Q.; Wu, X.; Li, G. Computer vision-based bridge damage detection using deep convolutional networks with expectation maximum attention module. Sensors 2021, 21, 824.
  46. Huang, Z.; Fu, H.L.; Fan, X.D.; Meng, J.H.; Chen, W.; Zheng, X.J.; Wang, F.; Zhang, J.B. Rapid surface damage detection equipment for subway tunnels based on machine vision system. J. Infrastruct. Syst. 2021, 27, 04020047.
  47. Arya, D.; Maeda, H.; Ghosh, S.K.; Toshniwal, D.; Mraz, A.; Kashiyama, T.; Sekimoto, Y. Deep learning-based road damage detection and classification for multiple countries. Autom. Constr. 2021, 132, 103935.
  48. Cui, X.; Wang, Q.; Dai, J.; Zhang, R.; Li, S. Intelligent recognition of erosion damage to concrete based on improved YOLO-v3. Mater. Lett. 2021, 302, 130363.
  49. Pozzer, S.; Rezazadeh Azar, E.; Dalla Rosa, F.; Chamberlain Pravia, Z.M. Semantic segmentation of defects in infrared thermographic images of highly damaged concrete structures. J. Perform. Constr. Facil. 2021, 35, 04020131.
  50. Andrushia, A.D.N.A.; Lubloy, E. Deep learning based thermal crack detection on structural concrete exposed to elevated temperature. Adv. Struct. Eng. 2021, 24, 1896–1909.
  51. Munawar, H.S.; Ullah, F.; Heravi, A.; Thaheem, M.J.; Maqsoom, A. Inspecting buildings using drones and computer vision: A machine learning approach to detect cracks and damages. Drones 2021, 6, 5.
  52. Zou, D.; Zhang, M.; Bai, Z.; Liu, T.; Zhou, A.; Wang, X.; Cui, W.; Zhang, S. Multicategory damage detection and safety assessment of post-earthquake reinforced concrete structures using deep learning. Comput. Aided Civ. Infrastruct. Eng. 2022, 37, 1188–1204.
  53. Han, X.; Zhao, Z.; Chen, L.; Hu, X.; Tian, Y.; Zhai, C.; Wang, L.; Huang, X. Structural damage-causing concrete cracking detection based on a deep-learning method. Constr. Build. Mater. 2022, 337, 127562.
  54. Tanveer, M.; Kim, B.; Hong, J.; Sim, S.H.; Cho, S. Comparative Study of Lightweight Deep Semantic Segmentation Models for Concrete Damage Detection. Appl. Sci. 2022, 12, 12786.
  55. Bai, Z.; Liu, T.; Zou, D.; Zhang, M.; Zhou, A.; Li, Y. Image-based reinforced concrete component mechanical damage recognition and structural safety rapid assessment using deep learning with frequency information. Autom. Constr. 2023, 150, 104839.
  56. Crognale, M.; De Iuliis, M.; Rinaldi, C.; Gattulli, V. Damage detection with image processing: A comparative study. Earthq. Eng. Eng. Vib. 2023, 22, 333–345.
  57. Chen, L.; Chen, W.; Wang, L.; Zhai, C.; Hu, X.; Sun, L.; Tian, Y.; Huang, X.; Jiang, L. Convolutional neural networks (CNNs)-based multi-category damage detection and recognition of high-speed rail (HSR) reinforced concrete (RC) bridges using test images. Eng. Struct. 2023, 276, 115306.
  58. Wan, H.; Gao, L.; Yuan, Z.; Qu, H.; Sun, Q.; Cheng, H.; Wang, R. A novel transformer model for surface damage detection and cognition of concrete bridges. Expert Syst. Appl. 2023, 213, 119019.
  59. Zhu, Y.; Tang, H. Automatic damage detection and diagnosis for hydraulic structures using drones and artificial intelligence techniques. Remote Sens. 2023, 15, 615.
  60. Huang, B.; Zhao, S.; Kang, F. Image-based automatic multiple-damage detection of concrete dams using region-based convolutional neural networks. J. Civ. Struct. Health Monit. 2023, 13, 413–429.
  61. Kim, H.; Lee, J.; Ahn, E.; Cho, S.; Shin, M.; Sim, S.H. Concrete crack identification using a UAV incorporating hybrid image processing. Sensors 2017, 17, 2052.
  62. Tong, Z.; Gao, J.; Sha, A.; Hu, L.; Li, S. Convolutional neural network for asphalt pavement surface texture analysis. Comput. Aided Civ. Infrastruct. Eng. 2018, 33, 1056–1072.
  63. Huang, Z.; Fu, H.; Chen, W.; Zhang, J.; Huang, H. Damage detection and quantitative analysis of shield tunnel structure. Autom. Constr. 2018, 94, 303–316.
  64. Tayo, C.O.; Linsangan, N.B.; Pellegrino, R.V. Portable crack width calculation of concrete road pavement using machine vision. In Proceedings of the 2019 IEEE 11th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management (HNICEM), Laoag, Philippines, 29 November–1 December 2019; pp. 1–5.
  65. Kim, B.; Cho, S. Image-based concrete crack assessment using mask and region-based convolutional neural network. Struct. Control Health Monit. 2019, 26, e2381.
  66. Wei, F.; Yao, G.; Yang, Y.; Sun, Y. Instance-level recognition and quantification for concrete surface bughole based on deep learning. Autom. Constr. 2019, 107, 102920.
  67. Beckman, G.H.; Polyzois, D.; Cha, Y.J. Deep learning-based automatic volumetric damage quantification using depth camera. Autom. Constr. 2019, 99, 114–124.
  68. Park, S.E.; Eem, S.H.; Jeon, H. Concrete crack detection and quantification using deep learning and structured light. Constr. Build. Mater. 2020, 252, 119096.
  69. Bhowmick, S.; Nagarajaiah, S.; Veeraraghavan, A. Vision and deep learning-based algorithms to detect and quantify cracks on concrete surfaces from UAV videos. Sensors 2020, 20, 6299.
  70. Flah, M.; Suleiman, A.R.; Nehdi, M.L. Classification and quantification of cracks in concrete structures using deep learning image-based techniques. Cem. Concr. Compos. 2020, 114, 103781.
  71. Yuan, C.; Xiong, B.; Li, X.; Sang, X.; Kong, Q. A novel intelligent inspection robot with deep stereo vision for three-dimensional concrete damage detection and quantification. Struct. Health Monit. 2022, 21, 788–802.
  72. Miao, P.; Srimahachota, T. Cost-effective system for detection and quantification of concrete surface cracks by combination of convolutional neural network and image processing techniques. Constr. Build. Mater. 2021, 293, 123549.
  73. Song, L.; Sun, H.; Liu, J.; Yu, Z.; Cui, C. Automatic segmentation and quantification of global cracks in concrete structures based on deep learning. Measurement 2022, 199, 111550.
  74. Kumarapu, K.; Mesapam, S.; Keesara, V.R.; Shukla, A.K.; Manapragada, N.V.S.K.; Javed, B. RCC Structural deformation and damage quantification using unmanned aerial vehicle image correlation technique. Appl. Sci. 2022, 12, 6574.
  75. Bae, H.; An, Y.K. Computer vision-based statistical crack quantification for concrete structures. Measurement 2023, 211, 112632.
  76. Li, Y.; Bao, T.; Huang, X.; Wang, R.; Shu, X.; Xu, B.; Tu, J.; Zhou, Y.; Zhang, K. An integrated underwater structural multi-defects automatic identification and quantification framework for hydraulic tunnel via machine vision and deep learning. Struct. Health Monit. 2023, 22, 2360–2383.
  77. Chowdhury, A.M.; Moon, S. Generating integrated bill of materials using mask R-CNN artificial intelligence model. Autom. Constr. 2023, 145, 104644.
  78. Yokoyama, S.; Matsumoto, T. Development of an automatic detector of cracks in concrete using machine learning. Procedia Eng. 2017, 171, 1250–1255.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/)
ScholarVision Creations
Feedback