Given the speed of sensor development and miniaturization, sensor technologies are commercially available that produce reliable data in an on-farm environment rather than in a clean laboratory. Moving sensors on-farm and in-field enable growers to strategically manage grain quality before the product is sold (post-farmgate) (Figure 1). The most common commercially available sensor that determines grain composition is the benchtop NIR. Growers subsample harvested grain and use these systems to determine moisture and protein prior to sale or storage.
1.1. Options for On-Farm Sensing Grain Quality
Within dryland cropping systems, proximal and remote sensing technologies are being used to capture and manage temporal and spatial variability that cause variation in crop growth, yield and quality across the landscape
[34][35][34,35]. Remote sensing can operate at multiple scales to assess crop growth and stress, including using hand-held instruments at the field scale, airborne platforms at the field and farm scales and satellites at scales from field to regional levels (
Figure 1). For example, a crop reflectance index, referred to as the Normalized Difference Vegetation Index (NDVI), measures the difference between near infrared (vegetation strongly reflects) and red light (vegetation absorbs). This index measures crop canopy cover and can correlate with plant biomass enabling the monitoring of vegetation systems
[36]. In one long-term study which assessed the response of winter wheat to heat stress on the North China Plain, canopy reflectance measured from satellite platforms
[37] was used to assess crop phenology and senescence rate (spatially and temporally), providing insight into climate change impacts on broadscale production potential. Canopy reflectance information using targeted spectral indices from the visible and near infrared spectral regions has also been correlated with plant nitrogen status in wheat
[38], leading to the development of indices including the Canopy Content Chlorophyll Index (CCCI) and the Canopy Nitrogen Index (CNI) at the experimental plot scale. Further testing of these indices across different dryland growing environments in Australia and Italy confirmed their utility, with good agreement between these indices and canopy nitrogen concentration (
r2 = 0.97) early in the growing season
[18]. In this example, such canopy-based remotely sensed indices provide growers with a decision support tool to guide in-season nitrogen fertilizer application, enabling spatial management of fertilizer rate across the paddock, thus optimizing the match between plant nitrogen demand and application.
More recently, crop canopy reflectance characteristics have also been used to monitor frost damage in wheat grown in southern Australia
[19][22][19,22], where rapid estimation of frost damage to crops on a spatial basis supports the timely management decisions by growers to reduce the economic impact of frost. In these studies, where hyperspectral reflectance and active light fluorescence were assessed, the reflectance indices Photochemical Response Index (PRI), Normalized Difference Red-Edge Index (NDRE), NDVI and the fluorescence-based index FLAV (Near Infrared Fluorescence excited with Red/Infrared Fluorescence excited with UV (375 nm) correlated well with frost damage experimentally imposed to field grown wheat at flowering. These principles are now evolving to a more widespread application within the grains industry, with service providers now offering seasonal satellite imagery and interpretation in the context of crop performance, spatial variation (topography) and soil characteristics. These data can be useful to inform the management of distinct parts of a paddock or farm (as management zones) for nutritional, abiotic, and biotic disorders to maximize yields and ensure high grain quality.
Research undertaken on-farm relates monitoring of canopy reflectance to crop growth and yield. However, this re
svie
archw highlights the importance and value of predicting the quality of the end-product, i.e., the grain, for growers on-farm. To predict grain-yield, a study that employed hand-held hyperspectral devices assessed variation in frost-affected lentil crops and confirmed that frost damage could be effectively detected in-season using remote sensing and that the stress response corresponds to both yield and quality outcomes
[4][39][4,39]. Similarly, the use of the active light fluorometer index FLAV in chickpeas and the spectral reflectance-based Anthocyanin Release Index 1 in faba beans have shown utility for the detection of the disease Ascochyta blight at the leaf-level
[40]. Given the potential for Ascochyta blight, a fungal disease in pulse grains caused by
Ascochyta rabei, to affect grain quality, this offers the opportunity for improved control of diseases through spatial management of foliar fungicides
[40], potentially reducing associated quality downgrades and seed carryover. More broadly, if variation in crop canopy reflectance (NIR) based on environmental stresses could be linked to grain quality, this would allow in-season mapping to predict spatial variation in grain quality prior to harvest. With this knowledge, the harvesting program could be tactically designed to harvest zones based on predicted yield and limit economic losses to environmental effects on grain quality within the grains industry.
Monitoring crop canopies using spectral analysis has the potential to provide growers with the tools to better manage grain quality. For this technology to be applied on-farm, further research is required to identify remote sensing measures that correlate with grain quality outcomes for key crop species. The logistics of employing such techniques on-farm must also be considered; the timeliness of collecting and interpreting spatial data to inform harvest zoning and the physical infrastructure and logistics to accommodate the various quality grades are two such examples. Developing prediction models for grain quality based on sensor data will inform harvest zones for quality and maximize financial return.
1.2. Data Acquisition Systems and Sample Handling
In addition to monitoring and managing grain quality prior to harvest, there are opportunities to directly monitor grain quality at various points in the supply chain from harvest through to sale. In this case, sensors could be installed within harvesting or grain handling machinery
[26], in grain storage infrastructure
[41], or in a ‘benchtop’ or laboratory setting similar to grain receival points
[32]. Compared with in-season quality monitoring, assessment of grain quality at harvest or during movement to storage offers the advantage of avoiding the need for mapping and interpreting quality data prior to harvest, saving an additional operation. It may also allow for the identification of quality defects not otherwise evident through spectral analysis of the crop canopy; for example, instances where weeds or other contaminants (e.g., snails) arise closer to crop maturity.
Monitoring of grain quality at- or post-harvest can also be undertaken spatially, enabling proactive management in future seasons. Commercially available grain protein monitors are one such example
[26][42][26,42], enabling segregation based on grain quality at harvest or during movement into or out of storage while providing information to inform future decisions relating to nitrogen fertilizer. In addition to segregating grain based on quality grade, monitoring quality at- or post-harvest may enable growers to blend grain of various quality levels in specific proportions prior to entering storage
[42]. This strategy can be used to ‘lift’ low-quality grain from one grade to the next, helping to achieve the highest aggregate quality and price for grain produced across a farm business.
In addition to monitoring grain quality at-harvest or during movement into storage, directly monitoring grain quality could also be undertaken during storage. Studies such as
[43][44][43,44] have shown that grain quality can change over time. While optimizing storage conditions through aeration, temperature control and fumigation can all help to maintain grain quality for longer, monitoring of stored grain in situ offers the potential to target marketing decisions based on projected commodity prices and prediction of grain quality over time.
In considering the on-farm application of grain quality monitoring systems, it is important to understand accuracy requirements. The ultimate success of quality monitoring is dependent on its ability to correctly classify a known volume of grain into its respective quality grade to trade. The accuracy required will depend on the crop species and the quality grade being targeted. This is further complicated where the quality of a given grain parcel varies across different target traits, e.g., contamination, color, and size, requiring more comprehensive monitoring systems with multiple algorithms (and perhaps different technologies) to account for the different traits within the grade. Typically, the utility of a monitoring system depends on the accuracy of both the instrument and the sampling process. Analyzing a sample that is representative of a given parcel of grain is vital to ensure the accuracy of the monitoring system
[45] and the ability to apply it in an on-farm situation.
2. Sensor Technologies Used in the Agricultural Production System
Key sensor technologies employed post-farmgate to objectively measure grain quality include RGB imagery, NIR spectroscopy and multispectral spectroscopy. Hyperspectral spectroscopy will become more widely adopted for commercial applications on farms as new cheaper sensors are developed and manufactured. These technologies and grain quality applications are summarized in
Table 1, and their advantages and limitations are outlined below.
Table 1. List of sensor technologies and examples of their applications in evaluating grain quality associated with measuring traits important for on-farm applications.
2.1. Digital RGB Camera
Digital cameras are one of the most widely used sensors with broad applications ranging from industrial quality control and robotics to capturing photographs of scenes and objects. These sensors can provide data-rich information and are used to analyze a wide range of visual traits, key classifying grain quality
[63]. Digital (RGB) cameras are typically equipped with light-sensitive Complementary Metal–Oxide–Semiconductor (CMOS) or Charge-Coupled Device (CCD) sensors to acquire colored images of scenes and objects
[32][63][32,63]. Color filter arrays (CFAs) arranged in a mosaic pattern on top of the sensor selectively filter RGB light, and the intensity of the light in each color channel at different spatial points on the sensor is measured and used to compose an RGB image
[63]. RGB cameras are commercially sold as DSLR (Digital Single Lens Reflex), small-scale machine vision cameras, industrial cameras, or as a device component, for example, in smartphones. The specific application of a camera depends on the resolution of the captured images, optics, light and color sensitivity, and frame rate, among a range of other factors. Factors including the signal-to-noise ratio and size of the sensor, optics, illumination and processing of the image (e.g., interpolation of pixels) determine the raw image data. The resolution of a static photo is usually expressed in megapixels, i.e., the number of pixels (length × width) in the image.
Digital cameras equipped with CMOS sensors are used for process-control and monitoring automated systems. Their adoption is mostly driven by reduced costs, improved power efficiency of the CMOS chips, and advancements in machine vision techniques for the analysis of the captured data. CMOS cameras can operate with both passive (sunlight) and active lighting, be deployed on vehicles or UAVs for remote monitoring, and the relatively small size of the generated data (e.g., compared to in-laboratory hyperspectral imaging) enables images to be processed in real-time.
Computer Vision (CV) is the scientific field in which computational methods are developed to derive meaningful information from camera data (images and videos). Computer vision includes computational methods for object and motion tracking, scene understanding, edge detection, segmentation, color and texture analysis
[64], object detection, classification, and counting, among some of the applications
[65].
Many indicators of food and plant quality, including ripeness of fruit, symptoms of diseases, nutrient deficiencies, damaged plants, weeds and plant species in crop fields, and others, manifest visually and thus assessed according to visual criteria; these features are suitable targets for detection by RGB cameras
[32][63][66][67][68][69][70][32,63,66,67,68,69,70]. Throughout the agricultural industry, the assessment of morphological and phenotypic features usually requires a visual inspection, a labor-intensive process prone to subjectivity and errors. Digital image analysis can improve methods of grain and plant classification according to objective visual criteria. Applications of RGB imaging in grain quality are outlined in
Table 1.
The advantage of using RGB cameras is that images are readily interpretable because the information captured is how a human perceives the appearance of the grain sample. Regarding the limitations of applying machine vision, these are similar to those encountered in human vision in that they both operate within the visible range of the light spectrum. Quantifying color using RGB imaging systems is complex as it can be difficult to correct or calibrate between systems due to differences in the camera optics, sample illumination, consistent presentation of the sample for image capture and different ways images can be processed and compressed for storage. The use of consistent lighting across samples and the inclusion of calibration panels within images is critical to robustly quantify grain characteristics. Digital imaging captures the external view of samples, where the internal grain structure and chemical information is not detected in RGB images. Therefore, to analyze sub-surface grain features, other spectroscopic techniques would need to be engaged, including NIR, Raman, NMR, UV, X-ray and fluorescence. Another limitation is the sample orientation. RGB imaging captures one viewpoint of a sample; therefore, traits, such as surface area, volume, length, width and diameter of samples, may not be representative depending on sample presentation. Also, certain complex processing traits have been analyzed using image analysis, such as grain milling, the error can be high and the range in laboratory values limited, compromising the accuracy and precision of the technique
[30][71][30,71].
2.2. Stereo Cameras
Stereo vision and Structure from Motion (SfM) are two methods used for three-dimensional (3D) imaging with conventional digital cameras
[72]. Binocular stereo vision uses images captured from two cameras at different angles and triangulation to compute the depth of scenes, analogous to the human binocular depth perception. In SfM, a single moving camera is used to obtain multiple images from different locations and angles, and the images are processed to obtain the 3D information. Stereo vision enables the estimation of the volume of objects and their spatial arrangements. Most of the applications of stereo cameras in agriculture are field-based and have been to generate 3D field maps
[73] for biomass estimation
[74], determine morphological features of wheat crops
[52] and crop status, including growth, height, shape, nutrition and health
[72]. There are limited applications of SfM for grain quality analysis; these include stereo imaging systems with two viewpoints, which have enabled the prediction of grain length, width, thickness and crease depth in wheat
[75].
As with the two-dimensional (2D) or single imaging RGB systems, similar limitations are found with stereo vision systems: the information collected is within the visible range (RGB), sample color, differences between imaging systems, sample lighting, internal grain structure and chemical information is not detected, and sample orientation may not be visible. The benefit of using a stereo imaging system is that multiple positions of the sample are captured, and, therefore, the data is not limited to one viewpoint, allowing a complete view of the sample and thereby reducing the error in estimating traits such as surface area and sample volume.
2.3. Near Infrared Spectroscopy
The near infrared (NIR) region of the electromagnetic spectrum encompasses the 700–2500 nm wavelength range and covers the overtone and combination bands involving the C–H, O–H, and N–H functional groups, all of which are prevalent in organic molecules
[54][76][54,76]. NIR spectroscopy is widely applied to the analysis of materials in agriculture, biomedicine, pharmaceutics, and petrochemistry. In the agriculture industry, NIR spectroscopy is used for determining the physicochemical properties of forages, grains and grain products, oilseeds, coffee, fruits and vegetables, meat and dairy, among many other agricultural products
[54][76][54,76].
The instrumentation used in NIR spectroscopy consists of a light source (typically an incandescent lamp with broadband NIR radiation), a dispersive element (commonly a diffraction grating) to produce monochromatic light at different wavelengths, and a detector that records the intensity of the reflected or transmitted light at each wavelength
[76]. An alternative method of obtaining the same information is with Fourier Transform near infrared (FT-NIR) spectroscopy, where the spectrum is reconstructed from interference patterns produced by a Michelson interferometer (interferogram) within the instrument
[77]. Compared to dispersive NIR, FT-NIR systems can collect spectra at higher spectral resolutions; however, unlike gases, this advantage is not significant in the analysis of liquids and solid samples, where the spectral bands are broad (>2 nm). For whole-grain analysis (e.g., protein and moisture constituency), the predictive performance of both types of instruments (FT and dispersive) is similar, indicating no advantage of either method over the other
[78].
NIR sensor readings are referenced with ‘white’ (reference) and ‘dark’ scans obtained from highly reflective (assumed to be 100% reflective) flat and homogenous materials such as fluoropolymers like Spectralon
®, and dark current (no light) signals, respectively. The reference panel provides calibration to reflectance, which is the physical measure of light from the surface of an object, and the dark measurement quantifies sensor noise. The most common referencing methods assume a linear response for sample reflectance R, given by
, where I is the intensity of the signal measured by the sensor and the dark (noise) from the sensor is removed from the sample and reference. Alternatively, sample reflectance may assume a non-linear relation, in which case the reflectance is typically modeled with a higher order polynomial equation, calibrated using a set of reflectance standards (e.g., Spectralon
® doped with graded amounts of carbon black) whose reflectance span the ~0–100% reflective range.
NIR spectroscopy is widely adopted, is relatively affordable ($US5000–$50,000) depending on the application and spectral sensitivity needed and is available as both desktop and portable low-power instruments. NIR can provide accurate measures of the chemical constituency of a sample, including the
w/
w% of nitrogen concentration (and thus protein), moisture, carbohydrates, and oils, among other organic compounds. Because the NIR radiation can penetrate a sample, it can be used to investigate its chemical composition. Furthermore, in densely packed bulk grain where objects overlap and occlude one another, unlike in image analysis, the NIR device is operable and not sensitive to the orientation and careful arrangement of the individual grains and can be used to measure whole grain sample properties.
Traditional NIR spectroscopy sensors, unlike digital imaging, capture the average spectrum of a sample of grain packed within a measuring cell. Digital images capture two-dimensional information, and within the image features can measure grain size distribution, whereas NIR spectroscopy, typically in homogenous samples, quantifies an average (of the analyzed sample) quantity of all the individual grains within the sensor field of view. Complexities in sample composition, mixing of spectral components (different parts of the grains, shadows, contaminants, etc.), and low concentration of analytes can limit the accuracy of traits measured with a NIR instrument.
Applications for NIR spectroscopy include rapid determination of oil, protein, starch, and moisture in a range of grains and their products
[54], including forages and food products
[79]. Other applications include the identification of wheat varieties and seed health
[53], fungal contamination
[46] and prediction of protein and moisture concentration
[54][79][54,79]. NIR has been widely adopted to measure protein and moisture concentration which is then used to determine the value (grade of the grain) and processing quality of the grain, i.e., baking quality in wheat and malting quality in barley.
2.4. Multispectral Imaging
Multispectral imaging acquires reflectance data at (often narrow) discrete bands (up to about 20) spanning the ultraviolet (UV), visible, and near infrared (NIR) regions of the electromagnetic spectrum. In contrast, RGB color images only provide data at three (broad wavelength) channels (R, G, and B) within the visible spectrum. Regardless of the modality (RGB, multispectral, or in-lab hyperspectral), the data is organized in three-dimensional numerical arrays (i.e., data cubes) where the first two dimensions (X and Y) correspond to the spatial information, and the third dimension (λ) stores the spectral information. There are three main methods for the acquisition of data in spectral imaging systems, named after the sequence of data acquisition along each of the X, Y and λ directions: (i) point-scanning (whiskbroom), (ii) line-scanning (push-broom) and (iii) area scanning methods. Multispectral imaging has similar applications to RGB imaging; however, as the spectral bands can extend beyond the visible region, multispectral imagers have been used to identify wheat varieties, detect black point disease or fungal contamination
[53][57][53,57], track desiccation of seeds
[55], seed authentication
[56] and identify the histological origin of wheat grain
[58]. Point-scanning involves the acquisition of a complete spectrum at each spatial point (X, Y), and the data is stored in band-interleaved-by-pixel (BIP) format. Because the spectrum at each pixel is acquired one at a time, this system is typically used in microscopy (e.g., atomic force microscopy) where acquisition speed is not a priority (because the object is not moving). In line-scanners, data is recorded line by line (y, λ) as the target sample moves along the X-direction. The data is stored in band-interleaved-by-line (BIL) format. This configuration is typically used in industrial scanners where samples are scanned during their movement on a conveyor under the imaging system. This is also typical when the sensor itself is moving across a stationary target, such as when deployed from an aircraft. In area-scanners, an entire 2D image is acquired at each λ, which results in a band-sequential (BSQ) data format. This method requires a rotating filter wheel or a tunable filter (e.g., Liquid crystal tenable filter, LCTF, or Acousto-optic tunable filters, AOTF) to target the wavelengths of interest at each scan and is generally not suitable for moving samples, unless movement is minimal with a high degree of overlap
[76]. Other imaging systems use LEDs of different emission wavelengths (UV—NIR) to sequentially illuminate objects placed in a dark enclosure to capture greyscale images, which are then multiplexed along the λ direction to form the multispectral data. Variations in illumination (due to lighting geometry and setup), sensor sensitivity, imaging method, and environmental conditions (e.g., temperature and humidity) can affect the data quality acquired by spectral imaging systems, hence calibrations of these systems are very crucial and important for their function
[30]. In remote sensing applications, the set of calibrations are often referred to as radiometric calibrations, which additionally account for the effects of altitude, weather, and other atmospheric conditions
[80].
Low-frequency NIR wavelengths or UV have been used in multispectral imagers that can penetrate objects to capture information beyond the surface images of standard RGB cameras. Therefore, features ‘invisible’ to RGB imaging can be used to determine the chemical composition of samples, albeit with limited accuracy, depending on the number and frequency of the spectral bands. Multispectral systems are limited in their capacity to measure the chemical composition of samples effectively because only a limited number of wavebands tend to be utilized to ensure the instrument is low-cost. Hyperspectral imaging and spectroscopy methods are suited for this purpose. Furthermore, multispectral cameras are more expensive than digital RGB cameras.