Precision spraying can significantly reduce herbicide input for turf weed management. A major challenge for autonomous precision herbicide spraying is to accurately and reliably detect weeds growing in turf. Deep convolutional neural networks (DCNNs), an important artificial intelligent tool, demonstrated extraordinary capability to learn complex features from images. The feasibility of using DCNNs, including various image classification or object detection neural networks, has been investigated to detect weeds growing in turf.
1. Introduction
Turf is the predominant vegetation cover in urban landscapes, golf courses, residential lawns, and sports fields. In the United States, it was estimated that the total turf area covers 163,812 km
2 with a lower and upper 95% confidence interval bounds of ±35,850 km
2 [1]. According to the National Golf Foundation, there are over 15,000 golf courses, with an average of 50 to 73 ha per golf course, in the United States
[2]. Turf offers many benefits, such as providing evaporative cooling in an urban area, remediating contaminated soil, absorbing atmospheric pollutants, and increasing the aesthetic value of residential and non-residential areas
[3]. Nevertheless, weeds are a challenging issue for turf management. Weeds compete with turf for sunlight, nutrients, and water resources and may significantly reduce turf aesthetics and functionalities
[4][5][6].
Turf managers predominately rely on synthetic herbicides for controlling weeds
[7][8][9]. Unfortunately, for controlling certain weeds growing in turf, the present control programs relying on synthetic herbicides are not cost-efficient
[6][10]. For example, repeat applications of sulfonylurea herbicides thiencarbazone + foramsulfuron + halosulfuron in combination with amicarbazone at 0.25 kg ai ha
−1 adequately controlled tropical signalgrass (
Urochloa distachya (L.) T.Q. Nguyen] in bermudagrass (
Cynodon dactylon (L.) Pers.)
[11]. However, repeated application of this herbicide program is expensive since a single application of amicarbazone at 0.25 kg ai ha
−1 would cost approximately 1500 U.S. dollars.
Moreover, some synthetic herbicides used in turf are suspected of polluting environments
[12]. Possible adverse impacts include, but are not limited to, the damaging effect on non-target organisms, water pollution, and harmful impact on humans
[13][14][15]. In the United States, it was reported that nearly 80% of stream samples in urban/suburban contained at least five pesticides
[16]. Atrazine is one of the most commonly used herbicides in warm-season turfgrasses
[17]; however, it is frequently detected in underground water
[16][18]. Monosodium methylarsenate (MSMA) is a highly effective broad-spectrum herbicide against a number of difficult-to-control weed species, including dallisgrass (
Paspalum dilatatum Poir.), but pollutes underground water
[19]. Following application, MSMA is converted to a more toxic form of inorganic arsenic that may contaminate water through soil runoff
[20]. In the United States, only spot-treatment of MSMA is permitted to be sprayed on established golf courses
[21].
Deep learning, a subset of machine learning technology, has emerged as successful applications in various scientific domains, including computer vision
[22][23][24]. Deep convolutional neural networks (DCNNs) demonstrated extraordinary capability to extract complex features from images
[25] and are utilized as a tool to detect weeds and perform precision herbicide spraying
[26][27][28][29][30][31]. For example, See & Spray
®, an autonomous smart sprayer utilizing DCNNs for weed detection, has been developed for precision herbicide application in agronomic crops
[32]. Detection of weeds growing in turf needs to consider weeds (e.g., weed growth stage, weed species, and biotypes) and turf factors (e.g., turf quality, mowing height, dormant vs. non-dormant stages). DCNNs recognize weeds based on plant morphological features, leaf texture, and color
[33][34][35][36]. Therefore, it is logical to assume that the detection of weeds growing in dormant turfgrass is easier than in actively growing turfgrass; the detection of large-leaved weeds is easier than small-leaved weeds; and the detection of broadleaf weeds is easier than grasses or grass-like weeds growing in turfgrass (
Figure 1)
[37][38].
Figure 1. Presumed difficulty of using DCNNs for detecting weeds growing in turfgrass.
2. Detection of Weeds Growing in Turf
2.1. Image Classification versus Object Detection
As shown in
Figure 2, weeds grow either scatteringly or in relatively large patches in turf. The preparation of training datasets for object detection neural networks involves drawing bounding boxes on the training images. For this reason, object detectors are used to detect scattered weeds growing in turf
[37][39][40]. However, to detect inconspicuous weeds, such as common lespedeza (
Kummerowia striata L.) and spotted spurge (
Euphorbia maculata L.), labeling the ground-truth locations within images for individual weeds is rather painstaking and laborious. Moreover, when detecting weeds in relatively large patches, a large number of weeds per image need to be labeled prior to training the object detectors.
Figure 2. Dandelion (Taraxacum officinale F.H. Wigg.) scatteringly grows in perennial ryegrass (Lolium perenne L.) turf (A). Smooth crabgrass (Digitaria ischaemum (Schreb.) Muhl) grows in a relatively large patch in bermudagrass turf (B).
Compared to object detectors, the training of image classification neural networks takes less time because it does not need to draw the bounding boxes. The grid cells (sub-images) could be created on the input images. Subsequently, the developed image classification neural networks could be used to detect if the grid cells contain weeds
[41]. The image classification neural networks could be employed to detect either scattered or relatively large-patched weeds in turf. When using the image classification neural networks as the machine vision decision system, the spray outputs of the smart sprayers need to be the same or slightly larger than the size of the sub-images in order to fully cover the sub-images containing the target weeds
[42].
2.2. Detection of Weeds in Dormant Turfgrass
Yu et al.
[37] evaluated DetectNet, GoogLeNet, and VGGNet for detecting annual bluegrass (
Poa annua L.) or annual bluegrass growing in proximity to various broadleaf weeds, such as common chickweed (
Stellaria media (L.) Vill.), dandelion, and white clover (
Trifolium repens L.). The authors reported that DetectNet was the most effective, while GoogLeNet was the least effective among the neural networks evaluated for detecting annual bluegrass in dormant bermudagrass. DetectNet achieved high precision and recall values with the highest F
1 score (≥0.99) at detecting annual bluegrass growing in dormant bermudagrass. In another study, Yu et al.
[38] reported that VGGNet achieved high F
1 scores with high recall values (1.00) for detecting various broadleaf weeds, including common chickweed [
Stellaria media (L.) Vill.], dandelion, henbit (
Lamium amplexicaule L.), purple deadnettle (
Lamium purpureum L.), and white clover (
Trifolium repens L.) in dormant bermudagrass turf.
2.3. Detection of Broadleaf Weeds in Actively Growing Turfgrass
Yu et al.
[43] compared DetectNet, GoogLeNet, and VGGNet to detect dandelion, ground ivy (
Glechoma hederacea L.), and spotted spurge growing in actively growing perennial ryegrass and reported that VGGNet was more effective than AlexNet and GoogLeNet in detecting these weeds. When the neural networks were trained with 15,486 negative (images without weeds) and 17,600 positive images (6500 images contain spotted spurge, 4600 images contain ground ivy, and 6500 images contain dandelion), VGGNet achieved high F
1 scores (≥0.9345) with high recall values (≥0.9952) to detect these weeds; the F
1 scores of AlexNet and GoogLeNet did not exceed 0.9103, while DetectNet was highly effective and achieved high F
1 scores (≥0.9843) to detect dandelion growing in perennial ryegrass.
2.4. Detection of Grass or Grass-Like Weeds in Actively Growing Turfgrasss
It was assumed that machine vision-based detection of grass or grass-like weeds in turfgrass is especially challenging due to the similarity in plant morphology
[37][38][44]. Yu et al.
[44] evaluated the use of image classification neural networks, including AlexNet, GoogLeNet, and VGGNet, for the detection of smooth crabgrass (
Digitaria ischaemum L.), dallisgrass, doveweed [
Murdannia nudiflora (L.) Brenan], and tropical signalgrass [
Urochloa distachya (L.) T.Q. Nguyen] growing in bermudagrass with erratic turf surface conditions (i.e., varying mowing heights and surface qualities). The authors found that VGGNet achieved excellent performances for detecting these weed species with high F
1 scores (≥0.93) and recall values (1.00). Although AlexNet and GoogLeNet achieved high recall, they exhibited low precision
[44]. The low precision indicates that the neural networks are more likely to misclassify turfgrass as weeds, leading to herbicide applications in turf where weeds do not occur.
2.5. Weed Localization
Object detectors, such as Faster R-CNN
[45], YOLO (You Only Look Once)
[46], and SSD (Single Shot Detector)
[47], generate bounding box outputs but do not determine the exact location of weeds on the images. Mask R-CNN, a segmentation network, can address this issue because it can achieve finer image segmentation for object detection
[48]. Nevertheless, this neural network requires pixel-wise precise ground truth labeling, which is time-consuming. Xie et al.
[49] developed an algorithm to generate synthetic data and constructed a nutsedge (
Cyperus spp.) skeleton-based probabilistic map as the neural network input to reduce the dependence on pixel-wise precise labeling. This approach effectively overcame the effect of insufficient training images and reduced the labeling time by 95%, and meanwhile, it outperformed the original Mask R-CNN approach for weed detection.
Despite all the successes described in previous paragraphs, detecting weeds growing in turf with image classification neural networks faces challenges
[37][50]. While previous researchers reported that image classification neural networks could detect and discriminate the sub-images containing weeds, they did not attempt to identify the location of weeds on the images
[37][38][50]. When using the image classification neural networks for weed detection, the exact location of the sub-images containing weeds on the input images needs to be determined to realize precision herbicide application with the smart sprayers. To address this issue, Yu and Jin
[41] developed a software that can integrate image classification neural networks and OpenCV-Python to create the grid cells on the input images. This software can crop the testing image (1920 × 1080 pixels) into a total of 40 equal size grid cells. The software marks the grid cells as “spray” if the inference of the developed neural networks indicates that they contained weeds and marks as “non-spray” if the inference indicates that they did not contain weeds. The x, y coordinates of the grid cells containing weeds are located with the developed software when used in conjunction with the image classification neural networks. Using this software, Jin et al.
[41] found that EfficientNetV2 was reliably inferred if the grid cells contained the target weeds with high F
1 scores (≥0.980) and noted that DenseNet, EfficientNetV2, ResNet, RegNet, and VGGNet reliably detected and discriminated the grid cells contained in dandelion, dallisgrass, purple nutsedge, and white clover.
2.6. Detection of Weeds Growing in Various Turfgrass Surface Conditions
Image classification and object detection neural networks can detect weeds growing in various turf surface conditions (
Table 1)
[37][50]. When the neural networks were trained with images taken at athletic fields, institutional lawns, and various golf course management zones (i.e., fairways, tees, putting greens, and rough), VGGNet demonstrated high F
1 score values (≥0.95) and effectively detected dollar weeds (
Hydrocotyle spp.), old world diamond-flower (
Hedyotis cormybosa L. Lam.), and Florida pusley (
Richardia scabra L.) in actively growing bermudagrass turf
[37].
Table 1. A summary of published reports on the use of DCNNs for detecting weeds growing in turf.