Robotic System for Crack Inspection

Version	Summary	Created by	Modification	Content Size	Created at	Operation
1		Md. Al-Masrur Khan	--	2802	2023-07-31 07:24:08	\|
2	layout & references	Sirius Huang	+ 10 word(s)	2812	2023-08-01 02:50:11	\| \|
3	layout	Sirius Huang	Meta information modification	2812	2023-08-01 02:52:18	\|

This entry is adapted from the peer-reviewed paper 10.3390/rs15143573

Crack inspection is important to monitor the structural health of pavement structures and make the repair process easier. Currently, pavement crack inspection is conducted manually, which is inefficient and costly at the same time. To solve the problem, researchers developed robotic systems for automated data collection and analysis in real-time.

crack detection deep learning mobile robotic system

1. Introduction

Cracks in paved roads are one of the most potent indicators of pavement damage. Cracking in the pavement is quite unavoidable, and there are many underlying factors (e.g., exposure to the sun, rain erosion, natural weathering, and long-term driving of vehicles) that accelerate the cracking of the pavement’s surface. If these cracks cannot be localized and repaired in time, they will have a negative impact on the safe driving of vehicles. Consequently, it can cause deadly accidents, as well as expenditure of a huge amount of money for the maintenance and repair of pavements. Therefore, crack detection at an early stage is essential to maintain the structural integrity and serviceability of paved roads. In past decades, manual crack detection was a very common practice for localizing cracks on paved roads. However, the manual method lacks efficiency and accuracy; it is expensive because of the necessary expertise. Moreover, it is considerably tedious, arduous, and time-consuming because experts must monitor the cracks with the naked eye by roaming the roads. Therefore, to lessen the workload of experts and make the system fast and cost-effective, researchers are bringing automation to crack detection. With the advancement of computer vision (CV) technology, various vision-based methodologies have already been developed to perform automatic crack detection. Early implementation of the CV techniques for crack detection was to some extent limited to threshold-based approaches (e.g., pixel intensity was used as the feature) ^[1]^[2], and other hand-crafted feature-based approaches. Some of the prominent hand-crafted feature-extraction techniques are wavelet features ^[3], Local Binary Pattern (LBP) ^[4], Digital Image Co-relation ^[5], Gabor filters ^[6], and so on. But these methods can only extract local patterns instead of global patterns, which pulls the detection results backward. Some research ^[7]^[8]^[9] has used model-based, traditional CV algorithms, which use geometric characteristics of images to perform crack detection globally. The advantages of model-based techniques over feature-based techniques are that model-based techniques can detect cracks in adverse conditions such as noisy environments, poor illumination conditions, and shadow problems. Though these model-based methods can partially solve noise problems and can detect cracks more continuously, their performance is not satisfactory enough when detecting cracks with complex patterns or intensity inhomogeneity.

In recent years, Deep Learning has been extensively applied in CV tasks for its noteworthy representation ability. DL models do not need hand-crafted features; rather, they can extract valuable features (both local and global) automatically from the input data. A few research works have already devoted their efforts to utilizing the properties of deep learning mentioned above to learn robust feature representation and detect cracks with more precision. Zhang et al. introduced a Convolutional Neural Network (CNN) classifier for the first time in 2016 to detect cracks in concrete structures ^[10]. The primary objective of this study was to develop a patch-based classifier to detect cracks in concrete structures. Later on, Cha et al. ^[11] and Eisenbach et al. ^[12] also performed patch-based classification, which can only identify the presence or absence of cracks in a corresponding image patch. Researchers also utilized another deep learning scheme called object detection for localizing the cracks, along with identifying them in an image ^[13]^[14]. However, these models can only classify and localize the cracks in a concrete structure instead of detecting cracks at a pixel level. So to solve this issue, Yang et al. incorporated an image segmentation technique for detecting concrete cracks at pixel level ^[15]. Crack segmentation involves classifying each of the pixels in an image as ’crack’ or ’non-crack’. Instead of detecting the class only in an image, crack segmentation detects an output image, highlighting the pixels containing the cracks, which localizes the cracks and extracts the original shape of the cracks. Moreover, the segmented images can later be used for the important task of extracting length, width, and area of cracks, which provides information about crack severity in concrete structures. Considering the advantages of crack segmentation over crack detection and classification, researchers from all over the world are devoting their efforts to developing crack-segmentation methods and quantifying the cracks to present an automated crack-detection system ^[16]^[17].

2. Robotic System for Crack Inspection

2.1. Traditional Methods

In past decades, various researchers have developed robotic vehicles to automatically inspect cracks. The first study, dating back to 2007 ^[18], designed an automated inspection system for cracks in concrete tunnels using a mobile robot. They collected the images using a CCD camera, which was interfaced with the robot, and stored the images in the robot’s brain. Later, they extracted cracks on a different computer using the Sobel edge-detection algorithm. Oyekola et al. also designed a robotic system for detecting cracks on concrete tank surfaces ^[19]. The authors also first collected the images and later detected cracks using a thresholding algorithm developed using the MATLAB programming language. Li et al. utilized the robotic platform developed by Guimu Robot Co Ltd., Shanghai, China for detecting cracks on pavement structures ^[20]. The authors developed an unsupervised algorithm named the Multiscale Fusion Crack Detection (MFCD) for inspecting the cracks. However, in this research, the cracks are also not detected by the onboard computer. La et al. developed a wall-climbing robot for detecting cracks on steel bridges ^[21]. The robot was equipped with several sensors and a camera. Navigating through the steel bridges, it collected data and passed them in real-time to the ground station for further processing and for detecting cracks using a Hessian-matrix-based filter. In another work, La et al. used the Seekur mobile robot platform and modified it by installing several NDE sensors (e.g., GPR, USW, ER, IE) and a camera for the monitoring of a concrete bridge deck ^[22]. The authors collected the images and passed them to the remote computer to extract the cracks using a Gradient-based algorithm. They also presented the delamination maps of the cracking using NDE data. The robot could localize itself and maneuver automatically on the bridge deck. However, this robotic system needed multiple onboard computers for navigating and processing everything. In the studies ^[18]^[19]^[20]^[21]^[22], the researchers relied on conventional methods for crack assessment, employing image-processing algorithms and threshold-based approaches. While these methods provided initial insights, their weaknesses lie in accurate detection as well as in the lack of post-processing techniques to obtain comprehensive geometric information about the cracks. Additionally, the reliance on external computers for crack detection introduced potential delays and limitations in real-time assessment.

2.2. Learning-Based Methods

Hendrik et al. developed a legged robot named ANYmal for inspecting the crack conditions in underground tunnels ^[23]. The researchers considered the tactile sensory system instead of the vision data because of the presence of noise, water, etc., on the surface. They collected signals from the footsteps of the robot and classified the crack conditions using the Support Vector Machine (SVM) algorithm. The authors classified five types of crack conditions, including good, satisfactory, fair, critical, and failure, to provide information about the severity. Le et al. developed a mobile robotic system for the in-line inspection of the pipes ^[24]. The authors integrated multiple sensors (e.g., LIDAR, optic sensors) on the robot and classified these combined sensory data using the SVM algorithm for detecting cracks in pipes. Lei et al. developed a low-cost unmanned aerial vehicle for inspecting cracks in concrete structures ^[25]. They collected images using their UAV and classified cracks using the SVM classifier running on the onboard computer. Pan et al. utilized low-altitude images collected from a UAV to detect cracks on asphalt pavements ^[26]. The researchers collected centimeter-level spatial resolution images and utilized a hybrid model (ANN+SVM) to inspect the cracks. In ^[23]^[24]^[25]^[26], machine learning techniques were introduced for crack assessment, improving upon the conventional methods. By using legged robots, mobile robots, and UAVs, these studies leveraged tactile sensory systems, the fusion of camera data with other sensors, and image analysis models such as Support Vector Machine (SVM), Random Forest (RF), and Artificial Neural Network (ANN). The focus was on classifying cracks and achieving enhanced crack detection using machine learning algorithms. However, weaknesses persisted in terms of limited classification capabilities and the reliance on specific sensor data fusion. The studies focused on crack classification rather than providing detailed geometric information, and the machine learning models were not fully capable of accurately quantifying crack properties such as depth and length. Montero et al. developed a mobile robotic system for detecting cracks in concrete tunnels ^[27]. They designed the mobile robot with an adjustable crane and a robotic arm. The adjustable crane carried the vision sensor and the robotic arm, while the robotic arm carried an ultrasonic sensor. They designed the crane to be adjustable so that it could reach different heights and directions for collecting data accurately. They collected images and passed them to the host computer, which analyzed the images using CNN, and they also contacted the ultrasonic sensor with the tunnel wall to analyze the cracks. Li et al. developed a quadrotor flying robot for detecting cracks in concrete bridges and tunnels ^[28]. The authors focused on reconstructing 3D metrics to determine the location of the defects and severity information using a visual-inertial fusion approach. They proposed a novel Deep Learning model named AdaNet to detect cracks using their own crafted dataset named Concrete Structure Spalling and Cracking (CSSC). Gui et al. developed a robotic system using the ARIR robotic platform for detecting cracks on airport pavement ^[29]. They utilized one vision camera and a GPR sensor for collecting surface and subsurface data. The data were passed to an analysis center to process the collected data. They employed an intensity-based algorithm to detect cracks from images and a voting-based CNN to predict the GPR data. Finally, the authors stitched the collected data together to present a continuous grid for visualization. Ramalingam et al. developed a mobile robotic platform named Panthera for segmenting cracks and detecting garbage on the road ^[30]. The authors adopted SegNet for the segmenting task and a DCNN-based object detector for detecting garbage. Furthermore, they also utilized the Mobile Mapping System (MMS) for localizing the defects. He et al. developed an unmanned surface vessel (USV) for inspecting cracks on the bottom part of bridges or urban culverts ^[31]. The researchers installed both RGB cameras and LIDAR to collect information from the environment. The authors proposed a novel Deep Learning model named CenWholeNet for detecting cracks. The USV was controlled from a ground station module, where the LIDAR and video information was also transmitted from the robot’s brain (Intel NUc mini pc). In ^[27]^[28]^[29]^[30]^[31], advancements were made in crack assessment with the integration of deep learning techniques. These studies utilized CCD cameras, UAVs, and mobile robots to collect images and employ deep-learning models such as CNN and Adanet. The emphasis was on detecting cracks and estimating crack location. However, limitations were observed in terms of limited crack parameter estimation and the need for additional computational resources to handle the complex deep-learning algorithms. Moreover, comprehensive datasets for training and evaluating deep learning models were not extensively developed.

Yang et al. developed a wall-climbing robot for detecting cracks and spalling on concrete structures ^[32]. The researchers collected data using an RGB-D camera and estimated the cracks on the images by utilizing a novel deep learning model named InspectionNet deployed in Intel Nuc Mini PC. They also developed a map-fusion module for their work to highlight the detected cracks. Yuan et al. developed a mobile robotic platform for detecting cracks in reinforced concrete structures ^[33]. The authors proposed a Mask-RCNN-based model for segmenting the cracks on the images collected from a stereo camera. They utilized an NVIDIA Jetson Xavier device to implement the edge computing technique and pass the predicted frames to the host computer through the WebSocket protocol. They designed a UI for successfully controlling the robot and collecting the frames. Another important feature of this research is that after quantifying the damages, the researchers presented information about the actual size in a 3D cloud point reconstruction of the inspected structures. In ^[32]^[33], the research focused on real-time crack assessment using robotic systems, incorporating RGB-D cameras, stereo cameras, and LIDAR sensors. The integration of deep learning models on onboard computers demonstrated promising results. However, weaknesses include the need for fine-tuning algorithms to handle variations in lighting conditions and challenges associated with accurately reconstructing actual crack sizes. Additionally, limited attention was given to automated crack quantification and severity mapping. Table 1 presents a summary of the robotic platforms for crack inspection. Although there have been numerous remarkable research works in the field of automatically detecting pavement cracks, there is still a vast scope for improving existing methods.

Table 1. Summary of robotic platforms for crack inspection.

Researchers	Inspected Structure	Robot Platform	Deep Learning	Remarks
Yu et al. ^[18]	Concrete Tunnel	Mobile robot	No	Images were collected by the robotic system. An image-processing algorithm was utilized in an external computer for detecting cracks and crack information.
Oyekola et al. ^[19]	Concrete Tank	Mobile robot	No	Images were collected by the robotic system. A threshold-based algorithm was used in another computer for detecting the cracks.
Li et al. ^[20]	Concrete pavement	Guimi robot co ltd.	No	Detected crack using an unsupervised learning algorithm named MFCD. Detection was not performed in the onboard computer
La et al. ^[21]	Steel bridge	Wall climbing robot	No	Images were collected and passed to the ground station in real-time. Cracks were detected using the Hessian-matrix algorithm.
La et al. ^[22]	Bridge deck	Seekur robot	No	Combined visual sensor and NDE sensors for crack inspection. Presented stitched images after crack detection and a delamination map.
Hendrik et al. ^[23]	Concrete sewers	ANYmal (legged robot)	Yes (Machine learning)	Tactile sensory system were used to collect time-series signals from the footstep of ANYmal, and Support Vector Machine (SVM) was used to classify types of cracks.
Le et al. ^[24]	Concrete pipe	Mobile robot	Yes (Machine Learning)	Data from the camera and other sensors were fused to classify using SVM for detecting cracks.
Pan et al. ^[26]	Asphalt pavement	UAV	Yes (Machine learning)	Collected images using the UAV and the cracks were detected using Random Forest (RF), SVM, and Artificial Neural Network (ANN) models.
Montero et al. ^[27]	Concrete Tunnel	Mobile robot	Yes	Collected RGB images using a camera and ultrasound data by an ultrasonic sensor. A CNN model was used for detecting cracks from the images, and a traditional method was used for estimating crack depth from the ultrasonic data.
Li et al. ^[28]	Concrete Bridge	Flying robot	Yes	A deep learning model named Adanet was developed for detecting cracks. Crack location and severity information was provided as well.
Gui et al. ^[29]	Airport pavement	ARIR robot	Yes	Both surface and subsurface data were collected by a camera and a GPR interfaced into the robotic system. An intensity-based algorithm and voting-based CNN were applied for processing image and GPR data.
Ramalingam et al. ^[30]	Concrete pavement	Panthera robot	Yes	A SegNet-based model was developed to detect cracks and garbage using the onboard computer. A Mobile Mapping System was also utilized to localize the cracks.
He et al. ^[31]	Concrete Bridge	USV	Yes	A USV with an onboard computer was applied to detect cracks in the bottom of a concrete bridge using a model named cenWholeNet.
Yang et al. ^[32]	Concrete wall	Climbing robot	Yes	A network named InspectionNet was used for detecting the cracks from the RGB-D camera on the onboard computer of a robotic system. A map-fusion module was also proposed to highlight the cracks.
Yuan et al. ^[33]	Reinforced concrete	Mobile robot	Yes	This robotic system used a stereo camera for collecting pictures and utilized a Mask RCNN model on the onboard computer to detect cracks. A 3D point cloud was reconstructed from the actual size of the cracks.

Researchers developed a robotic system for automated data collection and analysis in real-time. The robotic system navigates the pavement and collects visual images from the surface. A deep-learning-based semantic segmentation framework named RCDNet was proposed. The RCDNet was implemented on the onboard computer of the robot to identify cracks from the visual images. The encoder-decoder architecture was utilized as the base framework of the proposed RCDNet. The RCDNet comprises a dual-channel encoder and a decoder module. The encoder and decoder parts contain a context-embedded channel attention (CECA) module and a global attention module (GAM), respectively. Simulation results show that the deep learning model obtained 96.29% accuracy for predicting the images. The proposed robotic system was tested in both indoor and outdoor environments. The robot was observed to complete the inspection of a 3 m × 2 m grid within 10 min and a 2.5 m × 1 m grid within 6 min. This outcome shows that the proposed robotic method can drastically reduce the time of manual inspection. Furthermore, a severity map was generated using the visual image results. This map highlights areas that require greater attention for repair in the test grid.

References

Chen, C.; Seo, H.; Jun, C.; Zhao, Y. A potential crack region method to detect crack using image processing of multiple thresholding. Signal Image Video Process. 2022, 16, 1673–1681.
Akagic, A.; Buza, E.; Omanovic, S.; Karabegovic, A. Pavement crack detection using otsu thresholding for image segmentation. In Proceedings of the 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), Opatija, Croatia, 21–25 May 2018.
Nigam, R.; Singh, S.K. Crack detection in a beam using wavelet transform and photographic measurements. Structures 2020, 25, 436–447.
Zoubir, H.; Rguig, M.; Aroussi, M.E.; Chehri, A.; Saadane, R. Concrete Bridge Crack Image Classification Using Histograms of Oriented Gradients, Uniform Local Binary Patterns, and Kernel Principal Component Analysis. Electronics 2022, 11, 3357.
Gehri, N.; Mata-Falcón, J.; Kaufmann, W. Automated crack detection and measurement based on digital image correlation. Constr. Build. Mater. 2020, 256, 119383.
Medina, R.; Llamas, J.; Gómez-García-Bermejo, J.; Zalama, E.; Segarra, M.J. Crack Detection in Concrete Tunnels Using a Gabor Filter Invariant to Rotation. Sensors 2017, 17, 1670.
Nguyen, H.-N.; Kam, T.-Y.; Cheng, P.-Y. An Automatic Approach for Accurate Edge Detection of Concrete Crack Utilizing 2D Geometric Features of Crack. J. Signal Process. Syst. 2013, 77, 221–240.
Chun, P.; Izumi, S.; Yamane, T. Automatic detection method of cracks from concrete surface imagery using two-step light gradient boosting machine. Comput.-Aided Civ. Infrastruct. Eng. 2020, 36, 61–72.
Vedrtnam, A.; Kumar, S.; Barluenga, G.; Chaturvedi, S. Early crack detection using modified spectral clustering method assisted with FE analysis for distress anticipation in cement-based composites. Sci. Rep. 2021, 11, 1–19.
Zhang, L.; Yang, F.; Zhang, Y.D.; Zhu, Y.J. Road crack detection using deep convolutional neural network. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016.
Cha, Y.-J.; Choi, W.; Büyüköztürk, O. Deep Learning-Based Crack Damage Detection Using Convolutional Neural Networks. Comput. Civ. Infrastruct. Eng. 2017, 32, 361–378.
Eisenbach, M.; Stricker, R.; Seichter, D.; Amende, K.; Debes, K.; Sesselmann, M.; Ebersbach, D.; Stoeckert, U.; Gross, H.-M. How to get pavement distress detection ready for deep learning? A systematic approach. In Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, USA, 14–19 May 2017; pp. 2039–2047.
Li, Y.; Han, Z.; Xu, H.; Liu, L.; Li, X.; Zhang, K. YOLOv3-Lite: A Lightweight Crack Detection Network for Aircraft Structure Based on Depthwise Separable Convolutions. Appl. Sci. 2019, 9, 3781.
Li, L.; Zheng, S.; Wang, C.; Zhao, S.; Chai, X.; Peng, L.; Tong, Q.; Wang, J. Crack Detection Method of Sleeper Based on Cascade Convolutional Neural Network. J. Adv. Transp. 2022, 2022, 1–14.
Yang, X.; Li, H.; Yu, Y.; Luo, X.; Huang, T.; Yang, X. Automatic Pixel-Level Crack Detection and Measurement Using Fully Convolutional Network. Comput.-Aided Civil Infrastruct. Eng. 2018, 33, 1090–1109.
Polovnikov, V.; Alekseev, D.; Vinogradov, I.; Lashkia, G.V. DAUNet: Deep Augmented Neural Network for Pavement Crack Segmentation. IEEE Access 2021, 9, 125714–125723.
Yong, P.; Wang, N. RIIAnet: A Real-Time Segmentation Network Integrated with Multi-Type Features of Different Depths for Pavement Cracks. Appl. Sci. 2022, 12, 7066.
Yu, S.N.; Jang, J.-H.; Han, C.-S. Auto inspection system using a mobile robot for detecting concrete cracks in a tunnel. Autom. Constr. 2007, 16, 255–261.
Oyekola, P.; Mohamed, A.; Pumwa, J. Robotic model for unmanned crack and corrosion inspection. Int. J. Innov. Technol. Explor. Eng. 2019, 9, 862–867.
Li, H.; Song, D.; Liu, Y.; Li, B. Automatic Pavement Crack Detection by Multi-Scale Image Fusion. IEEE Trans. Intell. Transp. Syst. 2018, 20, 2025–2036.
La, H.M.; Dinh, T.H.; Pham, N.H.; Ha, Q.P.; Pham, A.Q. Automated robotic monitoring and inspection of steel structures and bridges. Robotica 2018, 37, 947–967.
La, H.M.; Gucunski, N.; Dana, K.; Kee, S.-H. Development of an autonomous bridge deck inspection robotic system. J. Field Robot. 2017, 34, 1489–1504.
Kolvenbach, H.; Valsecchi, G.; Grandia, R.; Ruiz, A.; Jenelten, F.; Hutter, M. Tactile inspection of concrete deterioration in sewers with legged robots. In Proceedings of the 12th Conference on Field and Service Robotics (FSR 2019), Tokyo, Japan, 29–31 August 2019.
Le, D.V.K.; Chen, Z.; Rajkumar, R. Multi-sensors in-line inspection robot for pipe flaws detection. IET Sci. Meas. Technol. 2020, 14, 71–82.
Lei, B.; Ren, Y.; Wang, N.; Huo, L.; Song, G. Design of a new low-cost unmanned aerial vehicle and vision-based concrete crack inspection method. Struct. Heal. Monit. 2020, 19, 1871–1883.
Pan, Y.; Zhang, X.; Cervone, G.; Yang, L. Detection of Asphalt Pavement Potholes and Cracks Based on the Unmanned Aerial Vehicle Multispectral Imagery. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2018, 11, 3701–3712.
Montero, R.; Menendez, E.; Victores, J.G.; Balaguer, C. Intelligent robotic system for autonomous crack detection and caracterization in concrete tunnels. In Proceedings of the 2017 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), Coimbra, Portugal, 26–28 April 2017.
Yang, L.; Li, B.; Li, W.; Brand, H.; Jiang, B.; Xiao, J. Concrete defects inspection and 3D mapping using CityFlyer quadrotor robot. IEEE/CAA J. Autom. Sin. 2020, 7, 991–1002.
Gui, Z.; Li, H. Automated Defect Detection and Visualization for the Robotic Airport Runway Inspection. IEEE Access 2020, 8, 76100–76107.
Ramalingam, B.; Hayat, A.A.; Elara, M.R.; Gómez, B.F.; Yi, L.; Pathmakumar, T.; Rayguru, M.M.; Subramanian, S. Deep Learning Based Pavement Inspection Using Self-Reconfigurable Robot. Sensors 2021, 21, 2595.
He, Z.; Jiang, S.; Zhang, J.; Wu, G. Automatic damage detection using anchor-free method and unmanned surface vessel. Autom. Constr. 2021, 133, 104017.
Yang, L.; Li, B.; Feng, J.; Yang, G.; Chang, Y.; Jiang, B.; Xiao, J. Automated wall-climbing robot for concrete construction inspection. J. Field Robot. 2022, 40, 110–129.
Yuan, C.; Xiong, B.; Li, X.; Sang, X.; Kong, Q. A novel intelligent inspection robot with deep stereo vision for three-dimensional concrete damage detection and quantification. Struct. Health Monit. 2021, 21, 788–802.

© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.

Upload a video for this entry

Information

Subjects: Engineering, Civil

Contributors MDPI registered users' name will be linked to their SciProfiles pages. To register with us, please refer to https://encyclopedia.pub/register :

Md. Al-Masrur Khan

Regidestyoko Wasistha Harseno

Seong-Hoon Kee

Abdullah-Al Nahid

View Times: 302

Update Date: 01 Aug 2023

Table of Contents

Video Upload Options

Confirm