Encyclopedia
Please note this is a comparison between Version 1 by Ricardo Rüther and Version 2 by Beatrix Zheng.
With the ever-increasing capacity and size of utility-scale photovoltaic (PV) power plants, reaching the scales of gigawatts and hundreds of hectares, automation is increasingly becoming a matter not only of scientific interest, but also of economic importance. Therefore, the autonomous procedure and classification of faults task must still be explored to enhance the accuracy and applicability of the aerial infrared thermography (aIRT) method.
- aerial infrared thermography (aIRT)
- PV power plant
- PV monitoring
- deep learning
- automatic fault detection
- PV reliability
1. Applications of Automatization Algorithms
1.1. Automatic Path Planning
In an automatic aerial infrared thermography (aIRT) mission, the unmanned aerial vehicle (UAV)UAV flies over a set of waypoints that cover all modules of the photovoltaic (PV)PV plant. Therefore, an optimized path-planning algorithm aiming at an optimal path for time and battery efficiency is essential [1][51]. Figure 12 depicts the coverage area by a UAV based on the field of view (FoV) and resolution of the camera on board as well as the essential parameters used for path planning [1][51].
Available market software packages already provide an automatic flight based on a so-called “lawn mower” flight pattern. However, they do not always provide the most efficient flight and do not guarantee a centralized view to the PV arrays, especially in power plants installed over complex topography.
Studies aiming at optimizing path planning include different approaches to the problem. In the researchtudy developed by Salahat et al. [2][52], the traveling salesman shortest path algorithm was used to generate a path that includes a randomly selected set of modules that represent the entire PV plant, allowing an optimization of the battery use. Ding et al. [3][53] based their method on density clustering, boustrophedon path planning and Bezier curves. Luo et al. [4][54] also based their algorithm for path planning optimization on Bezier curves in a joint approach with particle swarm optimization (PSO), taking into consideration the flight attitude, gimbal limitation and path length.
Image stitching and DIP techniques were used by Henry et al. [5][55] to find contours of the power plant and generate a “lawn mower” path over it. A similar approach, using DL, was adopted by Moradi Sizkouhi et al. [1][6][51,56], which was also complemented with a dynamic path planning, which deviates the previous flight plan to take closer photos when faults are detected. Pérez-González et al. [7][57] also used DL to detect the area of the PV plant and then used different algorithms to determine the best flight path, wherein exact cellular decomposition boustrophedon and grid-based wavefront coverage algorithms produced the best results.
Other real-time algorithms have been proposed that calculate the optimized path of the UAV during the flight. In Roggi et al. [8][58], the UAV corrected the pre-planned “lawn mower” path according to the images that it acquired and the image processing techniques that are applied. A vision-based flight control was also proposed by Xi et al. [9][59], which performs a real-time direction and velocity correction.
1.2. Detection of PV Systems
Although not always related to aerial inspections, the detection of PV systems in aerial imagery (UAV or satellite imagery) has been proposed by many authors and even used for path planning before the aIRT flight [6][7][56,57]. Table 12 presents a summary of methods used in the literature for detecting PV systems in aerial imagery. The table presents the best metrics obtained in each study, the type of image data used as input and the type of detection output obtained with each method. The detection output was classified into three categories:
- (a)Boxes: the output is given by the coordinates of a box or polygon placed around the PV system;
- (b)Mask: the output is a binary image where the pixels corresponding to the segment of the PV system are represented by the value 1 and the rest of the image is represented by 0;
- (c)Binary: for each image, the presence or absence of a PV system is the result of the algorithm (1 or 0).
Figure 23 shows two examples of two types of detections, by the coordinates of boxes around the detected PV panels (left) and masks of the segment of the PV system (right). Besides developing an algorithm for the detection of PV arrays, Wu et al. [10][60] also matched them to their string identifiers.
1.3. Detection of PV Modules
The detection of the area of the modules is an important step in the image processing task, which is required for both detecting modules with defects and locating them in the power plant.
Table 12.
Summary of methods for detecting PV systems using aerial imagery.
| [Ref]/Year | Algorithm | Best Results | Output Type | Images | |
|---|---|---|---|---|---|
| [12] 2016 | [38] 2016 | RF and DL | Pr: 90% | Mask | Aerial imagery |
| [10] 2017 | [60] 2017 | Feature description vector according to PV modules’ different colors | - | Boxes | UAV |
| [13] 2017 | [62] 2017 | Adaptive clustering method based on k- means | Loss rate is lower than 5% | Mask | Aerial imagery |
| [14] 2017 | [63] 2017 | GLCM algorithm | Pr: 93.16% | Mask | aIRT |
| F1: 77.8% | |||||
| [15] 2018 | [64] 2018 | DIP and k-means classifier | Pr > 99% | Boxes | Aerial imagery |
31][78] and Wu et al. [32][79]. Uma et al. [33][80] did the same using a k-means clustering algorithm. In 2021, Xie et al. [34][81] used a Segnet, a CNN used for segmentation.
For aerial images, Tribak and Zaz [35][82], Salamanca et al. [36][83] and Perez et al. [37][84] published results on detecting PV modules in aerial visual images, and many studies used different techniques to detect and segment PV module boundaries in aIRT data. Table 23 shows the studies related to DIP and DL algorithms. In this case, the detection result of the algorithm can be given by a box or polygon around the PV module, a mask of the segment of the module or lines that mark the borders of the modules in a PV panel.
The best metric among the studies was obtained with a combination of many algorithms (DIP, SVM and DL) [38][85]. On the other hand, the worst metrics were obtained with simple DIP filters [39][86]. It is important to note that not all studies are comparable, because not all have presented metrics for their performance, and they have different dataset sizes, which make the comparison difficult. The studies described by Carletti et al. [40][87] and Bommes et al. [41][88], besides detecting the PV module, also developed ways to track the modules in subsequent frames of a video. A tracking system was also developed by Xie et al. [42][89] using an AlexNet CNN.
Table 23.
Summary of methods for detecting PV modules in aIRT using DIP and DL algorithms.
| [Ref]/Year | Algorithm | Best Results | Output Type | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| [43] 2016 | [48] 2016 | DIP (normalization and thresholding) | F1: 92.76% | Boxes | ||||||
| [44] 2017 | [90] 2017 | DIP (edge extraction by Hough transform) | - | Boxes | ||||||
| [45] 2017 | [91] 2017 | DIP (thresholding) | Pr: 96.9% | Mask | ||||||
| [46] 2017 | [92] 2017 | RANSAC (random sample consensus) algorithm | - | Boxes | ||||||
| [47] 2017 | [93] 2017 | DIP (not detailed) | Pr: 82% | Boxes | ||||||
| [48] 2017 | [94] 2017 | DIP (thresholding in HSV color space) | - | Mask | ||||||
| [16] 2018 | [65] 2018 | DL (Segnet) | Pr: 90% | Mask | ||||||
| [ | Aerial imagery | |||||||||
| 49] 2018 | [95] 2018 | DIP (template matching algorithm) | F1: 83.0% | Boxes | [17] 2018 | [66] 2018 | DL (PolyCNN) | |||
| [40] 2020 | [87 | IoU: 79.5% | ] 2020 | Mask | Google Earth | |||||
| DIP (canny edge and morphological filters) | F1: 87% | Boxes | [18] 2019 | [67] 2019 | ||||||
| [39] 2020 | DL (Faster R-CNN, based on the classifier ResNet-50) | [ | Pr: 92.9% | Boxes | Google Earth | |||||
| 86] 2020 | DIP (ACM LS and filtering by area, Hough transform) | Re: 70% | Boxes | [19] 2019 | [49] 2019 | DL (Res-UNet) | Ac: 97.11% | |||
| [50] 2020 | [96 | Mask | ] 2020 | DIP (thresholding in HSV color space and MSER algorithm) | System IRT images | |||||
| Ac: 98% | Boxes | [20] 2020 | [68] 2020 | DL (Mask R-CNN and VGG16) | Ac: 96.99% | Mask | UAV | |||
| [51] 2020 | [97] 2020 | DL (YOLOv3) | F1: 95% | Boxes | [21] 2020 | [47] 2020 | ||||
| [38 | DL (U-net) | ] | F1: 82% | Mask | Google Earth | |||||
| 2020 | [85] 2020 | DIP + support vector machine (SVM) and DL (Mask R-CNN) | F1: 98.9% | Boxes | [22] 2020 | [69] 2020 | DL | F1: 92.2% | Binary | Satellite imagery |
| [52] 2020 | [98] 2020 | DIP (Hough line detection, Sobel operator) | - | Lines | [23] 2020 | [70] 2020 | DL | Pr: 92.66% | Mask | Google Earth |
| Re: 97.43% | ||||||||||
| [24 | ||||||||||
| [53] 2020 | [99] 2020 | DIP (Sobel and canny operator, HoughPLine) | Pr: 90.91% | Lines | ||||||
| [54] 2020 | [100] 2020 | DIP (LSD algorithm and k-means clustering) | Pr: 77.3% | Mask | ] 2020 | [71] 2020 | DL (CNN for semantic segmentation) | Average error of 5.75% | Mask | UAV |
| F1: 86.3% | [25] 2020 | [72] 2020 | k-means, SVM and CNN | |||||||
| [55] 2020 | [101 | MCC: 0.17 | ] 2020 | Mask | Identify solar on rooftops | |||||
| DIP (k-means clustering and thresholding) | Ac: 98.46% | Mask | [11] 2020 | [61] 2020 | ||||||
| [41] 2021 | DIP (edge detection) and DL (R-CNN) | [ | Pr: 92.25% | Mask | Panels in aIRT images | |||||
| 88] 2021 | DL (Mask R-CNN) | Pr: 90.01% | Mask | [26] 2021 | [73] 2021 | DL algorithms | F1: 95.38% | Mask | Aerial imagery | |
| F1: 90.51% | [27] 2021 | [74] 2021 | DIP (transform invariant low-rank textures (TILT) algorithm for orthographic view and Otsu’s method for segmentation) | - | Mask | Panels in aIRT images | ||||
| [28 | ||||||||||
| [] 2021 | [75] 2021 | Unsupervised segmentation parameter optimization (USPO) and RF classifier | F1: 98.7% | Mask | UAV | |||||
| 56] | [7] 2021 | [57] 2021 | DL server | - | Mask | UAV | ||||
| [1] 2022 | [51] 2022 | Mask R-CNN structure | Ac: 96.99% | Mask | UAV |
The first attempts to automatize the segmentation of the PV modules in IRT images appear to have been made in 2015, with images obtained with a moving cart, using simple DIP methods [29][76]. Other attempts were described in Menéndez et al. [30][77], Montanez et al. [
| 2022 |
| [ |
| 102 |
| ] 2022 |
| DIP (geometry coercion, clustering and angularity-based segment filtering) |
| - |
| Mask |
1.4. Orthomosaicking
The localization of faults within a power plant is a challenging issue that can be addressed by creating an orthomosaic of the PV power plant [57][103]. Image mosaicking, also known as image stitching, is a computational technique that detects overlapping key points in spatially subsequent photos and uses them to create a so-called panorama picture [48][94].
To improve the sight perspective and enable an expanded view of the localization of faults in both visual and thermal images of PV power plants, some researchers used commercially available software packages to create orthomaps with aerial imagery. Lee and Park [58][104] and Zefri et al. [59][105] used the software Pix4D capture to process thermal and visual images and create orthographic images with temperature information. Oliveira et al. [57][103] compared the use of two software packages (DroneDeploy and ContextCapture) to create an orthomosaic of a 1 MW PV power plant. Higuchi and Babasaki [60][106] used the software OpenDroneMap to generate the orthographic image of a 2 MW PV power plant.
Grimaccia et al. [48][94], Aghaei et al. [61][27] and Ismail et al. [62][107] have proposed methods for the orthomosaicking of visual images of PV power plants using DIP techniques. Tsanakas et al. [63][108] used the method of aerial triangulation, and Lafkih and Zaz [64][109] and Zefri [56][65][102,110] used the SIFT technique to perform the task. To optimize the mosaicking of visual PV images, Qi et al. [66][111] used a Faster R-CNN to detect key points in aerial sequence images in the world coordinate system, so it avoids redundant information generated by traditional methods. López-Fernández et al. [46][92] developed a tool that creates a 5D point cloud of the power plant, where each coordinate point has a temperature and an intensity value associated with it. After segmenting the modules in a dataset, Costa et al. [26][73] used a sliding window algorithm with overlapping pixels, combining frames side by side to reconstruct orthomosaics of power plants.
1.5. Soiling
A common cause of hot spots in PV power plants is soiling and shadow over the modules, which hinders the evaluation of results since they are not considered real defects of the PV modules [67][6]. Cipriani et al. [68][112] approached this issue by using a CNN to differentiate hot spots caused by faults from soiling, obtaining an accuracy of up to 98%.
Another solution to the problem is the analysis of the visual images that are normally taken together with the IRT images in the UAV, which enables the operator to discard hot spots caused by soiling. Automation of the task of detecting soiling in individual modules was proposed by Yang et al. [69][113], Pivem et al. [70][114] and Qasem et al. [71][115] using DIP techniques. Similar techniques were employed by Wen et al. [72][116], and by Karaköse and Firildak [73][117] to detect shadows over PV systems. Hanafy et al. [74][43] compared different ML algorithms (KNN, NN, RF and SVM) to classify modules in different categories of cleanliness and obtained an accuracy of over 90% using an SVM algorithm. Mehta et al. [75][37] proposed a method that uses a weakly supervised CNN-based classification network to predict power loss, detect soiling and categorize it given a PV module image. This method obtained an accuracy of about 87% on a test dataset of about 50 images.
1.6. Detection and Classification of Faults
The manual assessment of aIRT imagery is a time- and computing-consuming task; therefore, its automation is the most explored part of the aIRT framework in the literature. This detection can either be processed on board, during the UAV flight, as shown in the example in Figure 34, or in a computer software, after the acquisition of images has been carried out by the UAV (Figure 45). Both Figure 34 and Figure 45 show the procedure of the inspections for the two different approaches, including all tasks being automatized in each case.
In 2002, Pilla et al. [76][118] used the Sobel and canny edge operators to detect cracks in IRT PV cell images. In 2003, Wang et al. [77][119] used thresholding and a fuzzy classifier to detect faults in IRT images. After that, in 2011, Vergura and Falcone [78][120] used DIP techniques to analyze IRT images for faults. Since then, many other studies have used mostly DIP techniques to segment faults in IRT images.
1.7. Other Applications
Imaging techniques have been employed in some other applications to facilitate the analysis of PV modules. An example is the detection of blurred images that was addressed by Tribak and Zaz [79][176] with image processing techniques in order to filter frames of videos before employing mosaicking techniques. Similar techniques were used by Shen et al. [80][177] to correct the angle distortion of IRT images.
2. Current Insights
This research has shown that different automatization algorithms, including DIP, DL and classification techniques, have been employed for automating different tasks of the aIRT procedure for inspecting PV power plants. Among the conclusions, this resviearchw showed that only a few among the selected studies have assessed two important aspects of the autonomous inspection procedure, namely, the optimization of the flight path (nine papers), and the detection of soiling (eight papers). These two topics are of great importance to increase time efficiency in aIRT and therefore should be further investigated. The latter goal of detecting soiling over PV modules and differentiating it from actual faults of the modules was investigated by some authors, e.g., Dunderdale et al. [81][170] and Arosh et al. [82][159], together with the detection and classification of other faults.
For the task of performing the orthomosaicking of aIRT images to facilitate the localization of the faults in the field, four papers employed existing software to perform the task, while ten studies approached the development of algorithms to create the orthomosaic of the PV plant. However, most of the proposed methods are based on DIP techniques; therefore, the resulting mosaic consists of a simple image, without additional GPS information. The correlation of orthomosaic images with GPS coordinates and the identification of modules and strings according to the site nomenclature are areas that require further investigation.
Another approach to the challenge was developed by Wu et al. [10][60], with the development of an algorithm that detects PV arrays in power plants and performs the automatic correlation with their string identifiers. This is a promising strategy that could also be used to facilitate the localization of detected faults in the field through aIRT. Besides the study carried out by Wu et al. [10][60], another 20 studies among the selected literature focused on the development of algorithms to detect PV systems and panels in aerial imagery. However, only three of these studies focused on aerial IRT images of the PV plants, obtaining up to 93.16% precision in the results [14][63]. On the one hand, 18 papers presented the results of developed algorithms for the detection of individual PV modules in aIRT images, of which three of them applied DL techniques. Although the methods are hardly comparable given their different structures for results (i.e., mask, box or line), their different dataset sizes and the different evaluation metrics used, a method that combined many algorithms (DIP, SVM and DL) for detecting PV modules in aIRT images and obtained an F1 score of 98.4% can be highlighted [38][85]. On the other hand, the worst metrics were obtained with simple DIP filters [39][86], which although providing fast results with small datasets required for training, are characterized by a lack in generalization. This is important for the replication of the algorithm in images acquired in different conditions and with a different quality. The algorithms proposed by Carletti et al. [40][87], Xie et al. [42][89] and Bommes et al. [41][88] also performed the tracking of the modules in subsequent frames of an aIRT video. This task is of utter importance for the cross-correlation of detected modules and faults, as well as their location in PV plants.
Most of the selected studies have assessed autonomous fault detection and classification in PV plants through visual (12 papers), IRT (22 papers) and aIRT images (43 papers). Among these studies, 35% used DL techniques for the detection or classification of PV faults, with an increase in developed algorithms using CNNs in recent years. Still, DIP-based algorithms also presented high accuracy results, even though most of them use smaller datasets, and therefore their replication in other sets of data is possibly not feasible. The combination of DL or DIP techniques with classifier algorithms was a promising approach in recent studies. In the field, fault detection can either be processed on board, during the UAV flight, or subsequently through a post-processing procedure after the flight. For the first case, the high computational requirements and the processing time of DL are still a challenge, as even in high-performing computers, the processing of a set of images of a large-scale PV power plant (that consists of some gigabytes of data) can take hours when using a DL algorithm. In the same way as in the detection of PV systems and modules, many types of outputs for the algorithms are possible, namely the segmentation of the faults, the detection of damaged modules or even the classification of faults in separate classes. The classes also differ among authors, and these differences represent a great challenge not only for the comparison between studies, but also for the exchange of data, experiences and algorithms among researchers in PV community, which hinders the advancements in this area. The exchange of data to enable the development of larger and more generalized datasets that consider different environmental conditions is also deaccelerated by data protection clauses.
Besides the different result types, the different evaluation metrics (or the lack of them), dataset sizes and image resolutions of the inputs also make the comparison between studies difficult. However, in general, the algorithms with the highest metrics are the ones dedicated to detecting and classifying a few types of faults compared to those that carry out the classification of many classes of faults. This proves that detection and classification of multiple faults is a complex task and further investigation is required. On this subject, the algorithm developed by Bommes et al. [41][88] can be highlighted for its encouraging results, with an accuracy of 90% in the detection and classification of faults in ten different anomaly types. In summary, to achieve the goal of an entirely autonomous aIRT procedure, advances in some of the tasks related to the technique must be achieved. Even tasks that were already the focus of many research studies, such as the detection and classification of faults, should be further explored to contemplate different types of datasets and conditions. The exchange of data and academic collaborations are fundamental to allow for a fully automatic procedure that not only detects modules and faults on PV modules but also provides information about the type and location of the faults, in a simple and accessible manner, to enable quick remediation measures.
