Unmanned Aerial Vehicle Search Target Recognition Techniques: History
View Latest Version
Please note this is an old version of this entry, which may differ significantly from the current revision.
Contributor:
Ing-Chau Chang
,
Chin-En Yen
,
Hao-Fu Chang
,
Yi-Wei Chen
,
Ming-Tsung Hsu
,
Wen-Fu Wang
,
Da-Yi Yang
,
Yu-Hsuan Hsieh

The traditional method of finding missing people involves deploying fixed cameras in some hotspots to capture images and using humans to identify targets from these images. However, in this approach, high costs are incurred in deploying sufficient cameras in order to avoid blind spots, and a great deal of time and human effort is wasted in identifying possible targets. Further, most AI-based search systems focus on how to improve the human body recognition model, without considering how to speed up the search in order to shorten the search time and improve search efficiency. As the technology of the unmanned aerial vehicle (UAV) has seen significant progress, a number of applications have been proposed for it due to its unique characteristics, such as higher mobility and more flexible integration with different equipment, such as sensors and cameras, etc.

  • unmanned aerial vehicle
  • hierarchical human-weight-first path planning
  • artificial intelligence image recognition

1. Introduction

As the technology of the unmanned aerial vehicle (UAV) has seen significant progress in recent years, a number of applications have been proposed for it [1][2] due to its unique characteristics, such as higher mobility and more flexible integration with different equipment, such as sensors and cameras, etc. [3][4]. The researchers of [5] explore algorithms for the formation movement of UAV swarms, with the objective of facilitating simultaneous adjustments to the formation shape while the UAV swarm is in motion. Signal transmission is another highly significant topic in UAV control. The research in [6] proposes automatic modulation classification utilizing deep learning in this context. The study in [7] addresses improvements to existing GNSS systems, such as GPS positioning, tackling issues related to inaccuracies. The researchers propose a time-differenced carrier phase (TDCP) derivation-controlled GNSS/IMU integration scheme to successfully acquire vehicle information such as the relative position and heading. Real-world tests demonstrate that this method exhibits higher accuracy compared to traditional algorithms. In recent years, the increasing integration of UAVs with various interdisciplinary domains has also been observed. Koopman operators are mathematical tools used to describe the evolution of nonlinear dynamic systems. The work of [8] proposes robust tube-based model predictive control with Koopman operators, while [9] integrates Koopman operators with the control of UAVs. Furthermore, there exist various UAV path planning problems and related studies, such as the capacitated arc routing problem (CARP). The objective of CARP is to find the shortest path in a mixed graph with undirected edges and directed arcs, minimizing the distance of the path while considering capacity constraints for objects moving on the graph. In [10], the study introduces a memetic algorithm based on Two_Arch2 (MATA), which simultaneously considers multiple optimization objectives for the path planning problem, including the total cost, makespan, carbon emissions, and load utilization rate.
Recently, UAVs have been used for search and rescue (SAR) missions to find missing persons at the scene of a natural disaster or when an emergency event occurs [11][12][13]. The issue of missing persons is a challenging societal problem, particularly when involving minors. Children, due to their smaller stature, are susceptible to disappearance within large crowds, especially in crowded places such as amusement parks, making it difficult to notice their absence. Unfortunately, they generally exhibit a lower level of vigilance towards unfamiliar individuals, rendering them vulnerable to abduction. As the duration of a missing person’s search is prolonged, the probability of encountering a perilous situation escalates, imposing significant psychological distress upon parents.
However, there is a limited amount of research aimed at identifying specific individuals, such as missing persons, and researchers have primarily relied on fixed cameras installed in specific areas. This limitation prevents the continuous tracking of targets, leading to difficulties in inferring their actual positions due to the limited perspective and potential blind spots. Furthermore, most of the existing works on search and rescue adopt unmanned aerial vehicles (UAVs) [14][15][16] and employ indiscriminate search algorithms, without prioritizing the areas where the search target may be located, resulting in inefficient search operations and excessive UAV power consumption.

2. Traditional Unmanned Aerial Vehicle Path Planning Methods for Search and Rescue Operations

Several search and rescue methods have been proposed recently [14][15][16]. In [14], the sweep line search method conducts a thorough search from left to right, as illustrated in Figure 1. Meanwhile, ref. [15] introduces the spiral search, which navigates the designated search area in a spiral pattern, as depicted in Figure 2. Both methods are uncomplicated and exhibit algorithms with linear time complexity in relation to the search area. Differing from these two methods, refs. [16][17] introduce block-based methods. These approaches offer the advantage of categorizing the whole search area into blocks with and without search targets. Figure 3 demonstrates the relationship between the UAV’s perspective and the altitude concerning the search blocks when the whole search area is partitioned [17]. Through the traveling salesman problem (TSP) [18] approach, the shortest path that does not require the visiting of all blocks is computed if all blocks with search targets have been recognized in advance. However, the four methods mentioned above do not prioritize the block searching sequence in proximity to the search target, which results in inadequate search efficiency. Therefore, taking inspiration from block-based approaches, this research assigns priority to all blocks based on the likelihood of the blocks containing potential targets, which are automatically recognized in real time using the YOLOv5 model. In contrast to [16], which primarily focuses on finding the shortest path, this research emphasizes improving the search efficiency to yield the shortest search time by searching in the block with the highest priority first.
Figure 1. Sweep line search.
Figure 2. Spiral search.
Figure 3. The relationship between the altitude of the UAV and the partitioned search area.

3. Search Target Recognition Techniques

3.1. Color Space Exchange

The RGB color space is the most widely used color space, where RGB denotes red, green, and blue. It is similar to the well-known concept of the primary colors of light, where mixing these colors yields various levels of brightness and chromaticity. However, the RGB color space has a strong dependence on the lighting conditions, meaning that the color of an object can change with variations in brightness. In addition, the three elements in the RGB color space are highly correlated, indicating that a change in one element will result in a corresponding change in the perceived color. Therefore, using the RGB color space for the color extraction of objects is not ideal [19]. In contrast, the HSV color space [20] is more intuitive and easily understood compared to the RGB color space. It separates the brightness value (V) from the color chrominance, which can be further divided into hue (H) and saturation (S). Because these elements in HSV have a relatively weak correlation with each other, it is highly suitable for use in feature color extraction. In comparison to RGB, one of the advantages of the HSV color space is its weak inter-element correlation, making it easy to control. In applications involving color recognition, we can convert the detected images from the RGB color space to the HSV color space with Equation (1).
H = cos − 1 ( 1 2 [ ( R − G ) + ( R − B ) ] [ ( R − G ) 2 + ( R − B ) ( G − B ) 2 ] ) S = 1 − 3 [ min ( R , G , B ) ] R + G + B V = max ( R , G , B ) 255

3.2. Extracting Feature Colors of Image

The feature color extraction process in [20] involves first segmenting the elements of an image’s HSV color space, followed by the conversion of each element (H, S, V) into a histogram of oriented gradient (HOG). Since the HOG divides each element into several element intervals, the segmentation proportions for each element can be determined. Then, selecting the interval with the highest proportion for each element, we can obtain their respective numerical values (H, S, V). These values represent the HSV feature colors for the image.

3.3. Transformation of Color Space

After experimenting, it has been observed that certain issues exist when directly calculating color distances in the HSV color space. Specifically, when the saturation (S) is low, it often leads to the k-nearest neighbors (KNN) [21] decision result being mistakenly classified as gray, regardless of how the hue (H) changes. To address this, the extracted feature colors in HSV are transformed into the RGB color space using Equation (2) [20]. This transformation involves mapping the hue (h) range to ℎ𝑖, and calculating variables p, q, t based on the hue (ℎ𝑖) range to determine which combination of RGB attributes (p, q, t, v) applies. The calculated RGB values (𝑟0, 𝑔0, 𝑏0) are then subjected to Euclidean distance computation [22] against pre-established RGB color table values (𝑟1, 𝑔1, 𝑏1) to determine the color distance (d), as illustrated in Equation (3). Subsequently, the KNN algorithm is employed to identify the color of the clothing based on this computed distance. 
h i = ⌊ h 60 ⌋ f = h 60 − h i p = v × ( 1 − s ) q = v × ( 1 − f × s ) t = v × ( 1 − ( 1 − f ) × s ) ( r , g , b ) = { ( v , t , p ) , i f   h i = 0 ( q , v , p ) , i f   h i = 1 ( p , v , t ) , i f   h i = 2 ( p , q , v ) , i f   h i = 3 ( t , p , v ) , i f   h i = 4 ( v , p , q ) , i f   h i = 5
d = ( r 1 − r 0 ) 2 + ( g 1 − g 0 ) 2 + ( b 1 − b 0 ) 2 ,

3.4. K-Nearest Neighbors (KNN) Color Classification

K-nearest neighbors (KNN) [21] is a fundamental classification and regression algorithm. After obtaining the HSV feature colors of an image and calculating the color distances using Equation (3), these distances are compared to a pre-established RGB color table. After sorting the color distances for each color, K colors with the closest distances are then selected. Followed by a voting process among neighboring colors, the color with the most votes is determined as the final color result selected by the KNN algorithm.

3.5. UAV Systems for Human Detection

The work in [23] proposes an approach utilizing an automated human detection system on UAVs to identify human bodies, discussing the hardware configuration of UAVs and real-time human recognition capabilities. Ref. [24] presents a comprehensive human activity recognition algorithm, where the UAV first identifies whether the object is a person and subsequently recognizes various human activities, such as throwing, walking, and digging. Additionally, the study introduces various image stabilization techniques. The research of [15] focuses on achieving human body recognition using a CNN. Due to the difficulty in acquiring datasets, data augmentation is employed to enhance the training outcomes. The study compares the training outcomes using various architectures and outlines the algorithm’s path planning as a spiral search. The focus of the study in [25] lies in the application of UAVs for commercial transportation, aiming to achieve successful human body recognition using UAVs. The research encompasses the design of five distinct scenarios, revealing that the distance variation between the UAV and the human body has a more significant impact on the recognition success compared to the quality of the camera. In the context of search and rescue operations for swimmers, ref. [26] proposes a methodology that integrates global navigation satellite system (GNSS) techniques with computer vision algorithms to locate individuals in distress. Refs. [27][28] primarily focus on the training of human detection models. Ref. [27] introduces a modified YOLOv8 architecture by incorporating the SC3T module into the final layer and training the model using images captured from a UAV perspective. The emphasis of the study lies in the recognition performance. The experimental results are evaluated using confusion matrices and the mean average precision. The findings reveal that, across the precision rate, recall rate, and mAP, the modified YOLOv8 outperforms both the original YOLOv5 and YOLOv8 models. Ref. [28] primarily utilizes YOLOv5 for human detection and further employs a Haar cascade classifier to identify specific body parts (head, upper body, lower body). The final results indicate that YOLOv5 achieves 98% average precision (AP), while the Haar cascade classifier attains approximately 78% AP. Table 1 presents a comparison of related studies on human detection using UAVs. It can be found that most of the related methods focus on how to improve the human body recognition model, without considering how to speed up the search in order to shorten the search time and search efficiency.
Table 1. Comparison of related studies of UAV human detection.
 

Human Body Recognition Model

Dataset Used

Recognition of Human Clothing Types and Colors

Segmentation of the Search Area

Dynamic Route Planning for Search

Integration of Human Body and Clothing/Pant Color Recognition with Dynamic Route Planning

[23]

Motion detection outputs a score of human confidence

No

No

No

No

No

[24]

CNN

UCF-ARG dataset

No, proposes human activity classification algorithm

No

No

No

[15]

CNN

Self-developed captured dataset

No

No

No, spiral search

No

[25]

DNN with MobileNet V2 SSDLite

COCO dataset

No

No

Yes, estimates the person and moves in his direction with GPS

  

[26]

CNN with Tiny YOLOv3

COCO dataset + self-developed swimmers dataset

No

No

No

No

[27]

CNN with modified YOLOv8

Self-developed UAV view real-world dataset

No

No

No

No

[28]

CNN with YOLOv5 and Haar Cascade classifier

VisDrone dataset + COC0128 dataset

No, proposes a human body region classification algorithm

No

No

No

HWF

CNN with YOLOv5

VisDrone dataset + self-developed drone-clothing dataset

Yes, uses KNN color recognition

Yes

Yes, proposes the hierarchical human-weight-first (HWF) path planning algorithm

Yes,

Proposes the integrated YOLOv5 and HWF framework

This entry is adapted from the peer-reviewed paper 10.3390/machines12010065

References

  1. Sahingoz, O.K. Networking models in flying ad-hoc networks (FANETs): Concepts and Challenges. J. Intell. Robot. Syst. 2014, 74, 513–527.
  2. Menouar, H.; Guvenc, I.; Akkaya, K.; Uluagac, A.S.; Kadri, A.; Tuncer, A. UAV-enabled intelligent transportation systems for the smart city: Applications and challenges. IEEE Commun. Mag. 2017, 55, 22–28.
  3. Aasen, H. UAV spectroscopy: Current sensors, processing techniques and theoretical concepts for data interpretation. In Proceedings of the IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain, 22–27 July 2018; pp. 8809–8812.
  4. Ezequiel, C.A.F.; Cua, M.; Libatique, N.C.; Tangonan, G.L.; Alampay, R.; Labuguen, R.T.; Favila, C.M.; Honrado, J.L.E.; Canos, V.; Devaney, C.; et al. UAV aerial imaging applications for post-disaster assessment, environmental management and infrastructure development. In Proceedings of the International Conference on Unmanned Aircraft Systems, Orlando, FL, USA, 27–30 May 2017; pp. 274–283.
  5. Zhang, Y.; Li, S.; Wang, S.; Wang, X.; Duan, H. Distributed bearing-based formation maneuver control of fixed-wing UAVs by finite-time orientation estimation. Aerosp. Sci. Technol. 2023, 136, 108241.
  6. Zheng, Q.; Zhao, P.; Li, Y.; Wang, H.; Yang, Y. Spectrum interference-based two-level data augmentation method in deep learning for automatic modulation classification. Neural Comput. Applic. 2021, 33, 7723–7745.
  7. Mao, Y.; Sun, R.; Wang, J.; Cheng, Q.; Kiong, L.C.; Ochieng, W.Y. New time-differenced carrier phase approach to GNSS/INS integration. GPS Solut. 2022, 26, 122.
  8. Zhang, X.; Pan, W.; Scattolini, R.; Yu, S.; Xu, X. Robust tube-based model predictive control with Koopman operators. Automatica 2022, 137, 110114.
  9. Narayanan, S.S.K.S.; Tellez-Castro, D.; Sutavani, S.; Vaidya, U. SE(3) (Koopman-MPC: Data-driven learning and control of quadrotor UAVs. IFAC-PapersOnLine 2023, 56, 607–612.
  10. Cao, B.; Zhang, W.; Wang, X.; Zhao, J.; Gu, Y.; Zhang, Y. A memetic algorithm based on two_Arch2 for multi-depot heterogeneous-vehicle capacitated arc routing problem. Swarm Evol. Comput. 2021, 63, 100864.
  11. Erdelj, M.; Natalizio, E. UAV-assisted disaster management: Applications and open issues. In Proceedings of the International Conference on Computing, Networking and Communications, Kauai, HI, USA, 15–18 February 2016; pp. 1–5.
  12. Mukherjee, A.; De, D.; Dey, N.; Crespo, R.G.; Herrera-Viedma, E. DisastDrone: A Disaster Aware Consumer Internet of Drone Things System in Ultra-Low Latent 6G Network. IEEE Trans. Consum. Electron. 2023, 69, 38–48.
  13. Pasandideh, F.; da Costa, J.P.J.; Kunst, R.; Islam, N.; Hardjawana, W.; Pignaton de Freitas, E. A Review of Flying Ad Hoc Networks: Key Characteristics, Applications, and Wireless Technologies. Remote Sens. 2022, 14, 4459.
  14. Majeed, A.; Hwang, S.O. A Multi-Objective Coverage Path Planning Algorithm for UAVs to Cover Spatially Distributed Regions in Urban Environments. Aerospace 2021, 8, 343.
  15. Das, L.B.; Das, L.B.; Lijiya, A.; Jagadanand, G.; Aadith, A.; Gautham, S.; Mohan, V.; Reuben, S.; George, G. Human Target Search and Detection using Autonomous UAV and Deep Learning. In Proceedings of the IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT), Bali, Indonesia, 7–8 July 2020; pp. 55–61.
  16. Bandeira, T.W.; Coutinho, W.P.; Brito, A.V.; Subramanian, A. Analysis of Path Planning Algorithms Based on Travelling Salesman Problem Embedded in UAVs. In Proceedings of the Brazilian Symposium on Computing Systems Engineering (SBESC), Fortaleza, Porto Alegre, Brazil, 3–6 November 2015; pp. 70–75.
  17. Cabreira, T.; Brisolara, L.; Ferreira, P.R., Jr. Survey on Coverage Path Planning with Unmanned Aerial Vehicles. Drones 2019, 3, 4.
  18. Jünger, M.; Reinelt, G.; Rinaldi, G. The Traveling Salesman Problem. In Handbooks in Operations Research and Management Science; Elsevier B.V.: Amsterdam, The Netherlands, 1995; Volume 7, pp. 225–330.
  19. Ali, M.; Md Rashid, N.K.A.; Mustafah, Y.M. Performance Comparison between RGB and HSV Color Segmentations for Road Signs Detection. Appl. Mech. Mater. 2013, 393, 550–555.
  20. Haritha, D.; Bhagavathi, C. Distance Measures in RGB and HSV Color Spaces. In Proceedings of the 20th International Conference on Computers and Their Applications (CATA 2005), New Orleans, LA, USA, 16–18 March 2005.
  21. Pooja, K.S.; Shreya, R.N.; Lakshmi, M.S.; Yashika, B.C.; Rekha, B.N. Color Recognition using K-Nearest Neighbors Machine Learning Classification Algorithm Trained with Color Histogram Features. Int. Res. J. Eng. Technol. (IRJET) 2021, 8, 1935–1936.
  22. Pradeep, A.G.; Gnanapriya, M. Novel Contrast Enhancement Algorithm Using HSV Color Space. Int. J. Innov. Technol. Res. 2016, 4, 5073–5074.
  23. Krishna, S.L.; Chaitanya, G.S.R.; Reddy, A.S.H.; Naidu, A.M.; Poorna, S.S.; Anuraj, K. Autonomous Human Detection System Mounted on a Drone. In Proceedings of the 2019 International Conference on Wireless Communications Signal Processing and Networking (WiSPNET), Chennai, India, 21–23 March 2019; pp. 335–338.
  24. Mliki, H.; Bouhlel, F.; Hammami, H. Human activity recognition from UAV-captured video sequences. Pattern Recognit. 2020, 100, 107140.
  25. Safadinho, D.; Ramos, J.; Ribeiro, R.; Filipe, V.; Barroso, J.; Pereira, A. UAV Landing Using Computer Vision Techniques for Human Detection. Sensors 2020, 20, 613.
  26. Lygouras, E.; Santavas, N.; Taitzoglou, A.; Tarchanidis, K.; Mitropoulos, A.; Gasteratos, A. Unsupervised Human Detection with an Embedded Vision System on a Fully Autonomous UAV for Search and Rescue Operations. Sensors 2019, 19, 3542.
  27. Do, M.-T.; Ha, M.-H.; Nguyen, D.-C.; Thai, K.; Ba, Q.-H.D. Human Detection Based Yolo Backbones-Transformer in UAVs. In Proceedings of the International Conference on System Science and Engineering (ICSSE), Ho Chi Minh, Vietnam, 27–28 July 2023; pp. 576–580.
  28. Wijesundara, D.; Gunawardena, L.; Premachandra, C. Human Recognition from High-altitude UAV Camera Images by AI based Body Region Detection. In Proceedings of the Joint 12th International Conference on Soft Computing and Intelligent Systems and 23rd International Symposium on Advanced Intelligent Systems (SCIS & ISIS), Ise, Japan, 29 November—2 December 2022; pp. 1–4.
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.
This entry is offline, you can click here to edit this entry!
Video Production Service
Feedback