Deep Learning for Accurate Real-Time Weed Detection: History
View Latest Version
Please note this is an old version of this entry, which may differ significantly from the current revision.
Subjects: Computer Science, Artificial Intelligence
Contributor:
ildar rakhmatulin

This article discusses the possibility of accurately detecting the position of weeds in real-time in real conditions. Presented detailed recommendations for solving the problem with scene density, considered ways for increasing accuracy, and FPS.

  • deep learning in agriculture
  • precision agriculture
  • weed detection
  • robotic weed control
  • machine vision for weed control

1. Introduction

Weeds constitute one of the most devastating constraints for crop production, and efficient weed control is a prerequisite for increasing crop yield and food production for a growing world population [1]. However, weed control may negatively affect the environment [2]. The application of herbicides may result in pollution of the environment because, in most cases, only a tiny proportion of the applied chemicals hits the targets while most herbicides hit the ground, and a part of them may drift away [2][3]. Mechanical weed control may result in erosion and harm beneficial organisms such as earthworms in the soil and spiders on the soil surface [4][5]. Other weed control methods have other disadvantages and often affect the environment negatively. Sustainable weed control methods need to be designed only to affect the weed plants and interfere as little as possible with the surroundings. Weed control could be improved and be more sustainable if weeds were identified and located in real-time before applying any control methods.

2. Datasets and Image Pre-Processing

2.1. Datasets for Training Neural Networks

A dataset is needed to train neural networks, and image annotation of datasets is one of the main tasks for developing a computer vision system. Neural networks can be trained via multi-dimensional data and have the potential to model and extract meaning from a wide range of input information to address complex and challenging image classification and object detection problems. However, image datasets targeted for training neural networks should contain a significant number of images and enough variation in the object classes. When tracking an object with an accuracy of a centimeter, the dataset must be marked up appropriately, which is a highly labor-intensive task in terms of the time and effort required. At the same time, whichever neural network architecture is used, it is advisable to initialize the neurons using weights from a trained model that was trained using similar/related objects/images (i.e., the transfer learning technique). Transfer learning is a way to transfer the understanding (trained weights) of a model trained especially on the large dataset for initialization on a new dataset with similar objects, or on a smaller training set.

2.2. Image Preprocessing

Image preprocessing (e.g., color space transform, rescaling, flipping, rotation, and translation) is a stage of image preparation that focuses on creating accurate data in the format needed for the efficient use of image processing algorithms. The advantages are that these methods do not require high computing power and these processes can be implemented in real-time on single-board computers. Faisal et al. [6] presented a good example of a typical preprocessing process, exploring the use of a support vector machine (SVM) and a Bayesian classifier as an ML algorithm for the classifying of pigweed (Amaranthus retroflexus L.) in image classification processes. With sufficiently bright color contrast, this method accurately recognizes the position of the object (Figure 1).
Remotesensing 13 04486 g004 550
Figure 1. Image processing of pigweed (A. retroflexus): (a) original image, (b) grey-scale image, (c) after sharpening, (d) with noise filter [6].
Image preprocessing has a broad scope with many different possible filters and different options for transforming an image. Dyrmann [7] presented a review of various filters for preprocessing but did not give enough examples with images. Preprocessing can also be used in real-time. Pajares et al. [8] provided guidance on choosing a vision system for optimal performance given adverse outdoor conditions with large light variability, uneven terrain, and different plant growth conditions. Calibration was used for color balance, especially when the illumination changed (Figure 2). This kind of system can be used in conjunction with DL. Since DL can be trained to show only one illumination, this will reduce the number of images required to train the model.
Remotesensing 13 04486 g005 550
Figure 2. Example of auto-calibration process: (a) original image, (b) corrected image [8].

2.3. Available Weed Detection Systems

The real application of machine vision systems is closely related to robotics. Controlling weeds with robotic systems is a practice that has recently gained increasing interest. In robotic systems, mechanical aspects usually require higher precision; hence, detecting weeds from machine vision becomes even more challenging. Qiu et al. [9] and Ren et al. [10] used popular models of neural networks, such as R-CNN and VGG16, for weed detection with a robotics system. Asha et al. [11] created a robot for weed control, but realized the challenge of accurate weed detection after deployment in the field due to the minor differences in size between the weed and the crop (Figure 3). When the weed was close to a cultivated crop and its leaves (e.g., when they covered each other), the robot had a problem identifying the weeds. After color segmentation, the weed was perceived as a single object together with the crop.
Remotesensing 13 04486 g003 550
Figure 3. Examples of artificial images generated by LASSR [12].
Chang et al. [13] and Shinde et al. [14] used neural networks for image segmentation and the subsequent identification of weeds by finding the shapes that represent the weed’s structure (Figure 4).
Remotesensing 13 04486 g006 550
Figure 4. Example of a robot for removing weeds [14].

3. Deep Learning for Weed Detection

2.1. The Curse of Dense Scenes in Agricultural Fields

Dense scenes are the ones where there is dense vegetation, including both crops and weeds, and large occlusion between weeds and crops. Zhang et al. [15] presented a review of DL for dense scenes in agriculture. The purpose of this study is illustrated in Figure 5a, presenting a cluster of fruits on a dense background (including trees from other rows). As a result, the authors give several general recommendations such as increasing the dataset, generating synthetic data, and setting hyper-parameters. They concluded that it is advisable to continue increasing the depth of neural networks to solve problems with dense scenes. Assad et al. [16] focused on weeds (Figure 5b,c). The authors used semantic segmentation when conducting DL with a ResNet-50-based SegNet model. As a result, they attained a Frequency-Weighted Intersection Over Union (FWIOU) value of 0.9869, but the authors did not consider situations where the weed and the crop came into contact; in such cases, this method will not be efficient.
Remotesensing 13 04486 g007 550
Figure 5. Detection of objects within dense scenes: (a) for apples [15], (b) canola and weeds (initial image) [16], (c) weeds after semantic segmentation [16].
Proper annotation of the training dataset was crucial to detect the objects precisely. Bounding box-based detection methods used several IoU (intersection over union) and NMS (non-maximum suppression) values for model training and inference. Some segmentation methods can be used as well, with blob detection and thresholding techniques for the identification of individual objects in clusters. Identifying individual objects in clusters is a problem in various fields of activity, and much research has been devoted to solving it. Shawky et al. [17] and Zhuoyao et al. [18] considered the problem of dense scenes and concluded that each situation is unique, and it is advisable to combine various convolutional neural networks with each other. However, it is very difficult to predict the result in advance because such a method may take a lot of time. Dyrmann et al. [7] considered an object as a combination of different objects, depending on their position, which made the model a little heavier but, at the same time, increased the tracking accuracy. This method is one of the most promising.

2.2. State-of-the-Art Methods in Weed Detection

Many scientists have used DL to detect a single feature on a specific background. Rehman et al. [19] considered the role of CV only in fruits and vegetables among various horticultural products in agricultural areas with statistical machine learning technology. Similarly, Osorio et al. [20] considered a DL approach for weed detection in lettuce crops, while Ferreira et al. [21] considered weed detection in soybean crops. It is critical that the authors only used a limited dataset from one field. Therefore, the results of the neural network presented in the work have high accuracy, but at the slightest change in the detection condition (e.g., a new field or a change of season), the accuracy may change. As a rule, the results of such works can be used only under consistent conditions, which very rarely happens.
Dokic et al. [22] concluded that DL neural methods were better than classical ML methods for plant classification in images. The analysis of deep neural networks in the context of agriculture is not fully disclosed because problems with dense scenes have not been solved. A review by Tian et al. [23] considered practical applications of machine vision in agriculture. Their analysis showed that computer vision could help in developing agricultural automation for small fields in order to achieve benefits such as low cost, high efficiency, and accuracy. However, more emphasis was placed on the automation process than on machine vision. Khaki et al. [24] proposed the use of a CNN to detect and count corn kernels in images. The authors trained different models for the detection of objects at high speed under various conditions, lighting, and angles. The authors applied the standard sliding window approach to detecting the kernel, obtaining high accuracy due to the correct annotation of the dataset (Figure 6).
Remotesensing 13 04486 g008 550
Figure 6. Counting of corn kernels processed in images with CNN [24].

This entry is adapted from the peer-reviewed paper 10.3390/rs13214486

References

  1. FAO. NSP-Weeds. Available online: http://www.fao.org/agriculture/crops/thematic-sitemap/theme/biodiversity/weeds/en/ (accessed on 18 August 2021).
  2. Kudsk, P.; Streibig, J.S. Herbicides and two edge-sword. Weed Res. 2003, 43, 90–102.
  3. Harrison, J.L. Pesticide Drift and the Pursuit of Environmental Justice; MIT Press: Cambridge, MA, USA; London, UK, 2011; Available online: https://www.jstor.org/stable/j.ctt5hhd79 (accessed on 5 November 2021).
  4. Lemtiri, A.; Colinet, G.; Alabi, T.; Cluzeau, D.; Zirbes, L.; Haubruge, E.; Francis, F. Impacts of earthworms on soil components and dynamics. A review. Biotechnol. Agron. Soc. Environ. 2014, 18, 121–133. Available online: https://popups.uliege.be/1780-4507/index.php?id=16826&file=1&pid=10881 (accessed on 18 August 2021).
  5. Pannacci, E.; Farneselli, M.; Guiducci, M.; Tei, F. Mechanical weed control in onion seed production. Crop. Prot. 2020, 135, 105221.
  6. Faisal, F.; Hossain, B.; Emam, H. Performance Analysis of Support Vector Machine and Bayesian Classifier for Crop and Weed Classification from Digital Images. World Appl. Sci. 2011, 12, 432–440. Available online: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.390.9311&rep=rep1&type=pdf (accessed on 18 August 2021).
  7. Dyrmann, M. Automatic Detection and Classification of Weed Seedlings under Natural Light Conditions. Det Tekniske Fakultet.University of Southern Denmark. 2017. Available online: https://pure.au.dk/portal/files/114969776/MadsDyrmannAfhandlingMedOmslag.pdf (accessed on 18 August 2021).
  8. Pajares, G.; Garcia-Santillam, I.; Campos, Y.; Montalo, M. Machine-vision systems selection for agricultural vehicles: A guide. Imaging 2016, 2, 34.
  9. Qiu, Q.; Fan, Z.; Meng, Z.; Zhang, Q.; Cong, Y.; Li, B.; Wang, N.; Zhao, C. Extended Ackerman Steering Principle for the coordinated movement control of a four wheel drive agricultural mobile robot. Comput. Electron. Agric. 2018, 152, 40–50.
  10. Ren, G.; Lin, T.; Ying, Y.; Chowdhary, G.; Ting, K.C. Agricultural robotics research applicable to poultry production: A review. Comput. Electron. Agric. 2020, 169, 105216.
  11. Asha, R.; Aman, M.; Pankaj, M.; Singh, A. Robotics-automation and sensor based approaches in weed detection and control: A review. Intern. J. Chem. Stud. 2020, 8, 542–550.
  12. Cap, Q.H.; Tani, H.; Uga, H.; Kagiwada, S.; Lyatomi, H. LASSR: Effective Super-Resolution Method for Plant Disease Diagnosis. arXiv 2020, arXiv:2010.06499.
  13. Chang, C.; Lin, K. Smart Agricultural Machine with a Computer Vision Based Weeding and Variable-Rate Irrigation Scheme. Robotics 2018, 7, 38.
  14. Shinde, A.; Shukla, M. Crop detection by machine vision for weed management. Intern. J. Adv. Eng. Technol. 2014, 7, 818–826. Available online: https://www.academia.edu/38850273/CROP_DETECTION_BY_MACHINE_VISION_FOR_WEED_MANAGEMENT (accessed on 18 August 2021).
  15. Zhang, Q.; Liu, Y.; Gong, C.; Chen, Y.; Yu, H. Applications of deep learning for dense scenes, analysis in agriculture: A review. Sensors 2020, 20, 1520.
  16. Asad, M.; Bais, A. Weed Detection in Canola Fields Using Maximum Likelihood Classification and Deep Convolutional Neural Network. Inform. Process. Agric. 2020, 7, 535–545.
  17. Shawky, O.; Hagag, A.; Dahshan, E.; Ismail, M. Remote sensing image scene classification using CNN-MLP with data augmentation. Optik 2020, 221, 165356.
  18. Zhuoyao, Z.; Lei, S.; Qiang, H. Improved localization accuracy by LocNet for Faster R-CNN based text detection in natural scene images. Pattern Recognit. 2019, 96, 106986.
  19. Rehman, T.U.; Mahmud, M.S.; Chang, Y.K.; Jin, J.; Shin, J. Current and future applications of statistical machine learning algorithms for agricultural machine vision systems. Comput. Electron. Agricult. 2019, 156, 585–605.
  20. Osorio, K.; Puerto, A.; Pedraza, C.; Jamaica, D.; Rodríguez, L. A Deep Learning Approach for Weed Detection in Lettuce Crops Using Multispectral Images. AgriEngineering 2020, 2, 32.
  21. Ferreira, A.S.; Freitas, D.M.; Gonçalves da Silva, G.; Pistori, H.; Folhes, M.T. Weed detection in soybean crops using ConvNets. Comput. Electron. Agric. 2017, 143, 314–324.
  22. Dokic, K.; Blaskovic, L.; Mandusic, D. From machine learning to deep learning in agriculture—The quantitative review of trends. IOP Conf. Ser. Earth Environ. Sci. 2020, 614, 012138. Available online: https://iopscience.iop.org/article/10.1088/1755-1315/614/1/012138 (accessed on 18 August 2021).
  23. Tian, H.; Wang, T.; Yadong, Y.; Qiao, X.; Li, Y. Computer vision technology in agricultural automation —A review. Inform. Process. Agric. 2020, 7, 1–19.
  24. Khaki, S.; Pham, H.; Han, Y.; Kuhl, A. Convolutional Neural Networks for Image-Based Corn Kernel Detection and Counting. arXiv 2020, arXiv:2003.12025v2. Available online: https://arxiv.org/pdf/2003.12025.pdf (accessed on 18 August 2021).
More
© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.
This entry is offline, you can click here to edit this entry!
Video Production Service
Feedback