Spectral and machine vision technology have become the main choices for crop N status diagnosis. As an optical sensor, it enables rapid and periodic assessment of crop N status ^[1][36]. As shown in Figure 12, for the spectral technology, we can use it to obtain the spectral information of crop leaves and canopy, respectively, and then use the data pre-processing method to obtain the characteristic bands. For machine vision technology, we can use it to obtain the image information of crop leaves and canopy, and then obtain the feature information through image segmentation and feature extraction. Among them, the spectral information or image information of canopy can not only reflect the N status of crop canopy, but also reflect the N status of the whole plant. Then the destructive methods (such as the Kjeldahl method) are used to obtain the actual N status of the crop. Based on the correlation analysis between spectral feature band or image feature information and crop actual N status, the estimation model of crop leaf, canopy, and whole plant N status are established. However, the quality of the N status estimation model is often affected by the external environment factors, such as external light, soil background, etc.

Figure 12. Basic principles of N non-destructive diagnosis.

1. Spectral Technology

 Basic principles of  N    non-destructive diagnosis.

1. Spectral Technology

Spectral reflectance is a promising and convenient index for continuous sampling and narrow wave selection, which can sensitively reflect the specific physiological variables of crops ^[2][3][37,38]. Within the range of 350–1300 nm, the accurate measurement of leaf spectral reflectance depends on the interaction between light and crops and its influence on the spectral characteristics of green leaves, which enable accurate quantification of the crop N status ^[4][39]. Among them, we usually use hand-held or near ground spectral sensors to collect the crop leaf spectral information and unmanned aerial vehicle (UAV) or remote sensing technology to collect the spectral information of large-scale crops canopy in the visible and near-infrared spectral band range (Figure 23). Then, the data pre-processing method is used to select the characteristic band, and the appropriate estimation method is selected to construct the N status estimation model.

Figure 23.

Basic principles of spectral technology.

1.1. Application of Spectral System Selection to the Diagnosis of Crop N Status

 1.1. Application of Spectral System Selection to the Diagnosis of  Crop    N Status

Recently, a number of remote sensing systems have been proposed for the assessment of crop N status, such as the compact airborne spectrographic imager system [6], the hyperspectral LiDAR(HSL) ^[4][5][6][39,40,41], remote sensing with hyperspectral system ^[7][42], the QuickBird satellite with multi-spectral system ^[8][43], and others ^[9][10][11][44,45,46]. Hyperspectral sensors describe the reflectance of crop canopy in more information than multispectral sensors ^[12][47], which mainly includes two forms: non-imaging and imaging ^[13][14][48,49]. The non-imaging measurement can only obtain a small amount of spectral information of sampling points, and it cannot achieve the rapid and efficient information acquisition of large-area crops ^[14][49]. The imaging measurement such as satellite-borne hyperspectral imaging technology can obtain a wide range of spectral information, but also has some problems, such as long revisit period, limited breadth, low spatial resolution (generally less than 30 m), etc. ^[15][50]. Airborne hyperspectral imaging technology has the characteristics of mobility and flexibility. However, it is subject to air traffic control and requires high light conditions ^[16][51]. Consequently, the data collection cost based on this method is high, and its popularization and application in precision agriculture are limited. Compared with the aforementioned satellite-borne hyperspectral imaging technology and airborne hyperspectral imaging technology, UAV-based remote sensing measurement technology has the characteristics of low flight control, efficient use, flexible, and low operating cost ^[17][52], which can offer particular advantages with a high spatial resolution, an appropriate revisit time, and a spectral resolution adjusted for a specific task ^[12][18][19][47,53,54]. Besides, the combination of other methods and spectral technology such as the combination of spectroscopy and SPAD ^[20][55], the combination of Ground-Based Hyperspectral and UAV-Based Multispectral Imagery for crop N status assessment in rice ^[12][47], and the combination of physical optics approach and UAV-based hyperspectral imagery has great potential for assessing canopy N density (CND) in winter wheat ^[21][56].

1.2. Application of Spectral Data Processing to the Diagnosis of Crop N Status

 1.2. Application of Spectral Data Processing to the Diagnosis of  Crop    N Status

The data obtained by spectral technology have collinearity, high redundancy, sometimes noise and spectral autocorrelation ^[12][47], the estimation ability, and calculation efficiency of the model can be reduced ^[22][57]. Thus, the commonly used spectral preprocessing methods are multiplicative scatter correction (MSC), Savitzky–Golay smoothing (SGS), first-derivative (1-Der), second-derivative (2-Der), standard normal variable (SNV), etc. ^[13][48]. Besides, there are other methods to extract the spectral information of the target, including continuum removal methods, hyperspectral vegetation indices, multivariable statistical methods, and differential technique ^[23][24][58,59]. Therefore, it is of great significance to select the most critical spectral bands for crop N status estimation. At present, there are many spectral indices or vegetation indices for diagnosis of crop N status ^[25][26][60,61]. Therefore, we selected core and novel indexes to summarize.

       The diagnosis of crop N status at leaf scales is the basis of crop population N status diagnosis. Some studies reported that there is a good correlation between crop N status and leaf spectral data ^{[27][28][29][30][31][32]}[62,63,64,65,66,67], and different spectral indices are suitable for different crops and their growing stages ^[33][34][68,69]. Several spectral indices (SIs) such as the ratio index (RI), normalized difference spectral index (NDSI) ^{[35][36][37][38][39][40][41]}[70,71,72,73,74,75,76], and others ^{[42][43][44][45][46][47]}[78,79,80,81,82,83] are used for leaf N content (LNC) estimation based on leaf reflectance. However, most of these indices focus on two or three bands only. Because different crops have different sensitive bands in different growth stages of N diagnosis, which has certain limitations in the realization of comprehensive and accurate diagnosis of crop N status. Therefore, it is difficult to establish a unified index to evaluate LNC in the different growth periods, varieties, and sites. However, the optimum multiple narrow band reflectance (OMNBR) models was proposed by ^[25][60] significantly increase the accuracy for estimating the LNC (R2 = 0.67 0.71) and plant N concentration (PNC) (R2 = 0.57 0.78) with six bands. Except for that mentioned above, some NDVI-like indices derived from different diagnostic wavelengths have been proposed for monitoring N status ^{[48][49][50][51][52]}[84,85,86,87,88]. However, due to the interference of soil background, NDVI values will change and reach saturation at moderate-to-high vegetation densities. Some indices such as the leaf N content spectral indices (LNCSI) ^[53][89], which has a good effect on the quantitative inversion of LNC in wheat leaves. However, the latter only evaluated the N content of wheat leaves at flag leaf stage and flowering stage, but did not consider other growth stages. The CIred-edge ^[54][90] can provide a more accurate and stable estimation of the LNC in maize, which can accurately reflect to the dynamic changes of leaf N status during different growth stages of maize. Besides, for the diagnosis of leaf N status in different varieties, sites, and phenological characteristics, a new index named dual peak area normalized difference (NDDA) was proposed by Feng et al. ^[29][64], which has the advantages of good stability and strong monitoring ability for this problem. Therefore, there are many challenges and problems in the diagnosis of leaf N status on different crops and growth stages.

       The diagnosis of crop N status at canopy level is an effective method for accurate diagnosis of crop population N status. The spectral characteristics of the canopy can represent the growth information of vegetation canopy ^[55][15,91]. However, leaf reflectance is relatively high in the near-infrared regions due to the multi scattering and low chlorophyll absorption and relatively low in the visible wavelengths due to the high chlorophyll absorption ^[56][92] Monitoring leaf nitrogen status with hyperspectral reflectance in wheat]. To address this issue, some chlorophyll red-edge and plant N spectral indices, such as SDr/SDb, DIDA, and RSI(D740, D522) for assessing canopy N status have been developed ^[15][57][58][50,93,94]. Besides, canopy chlorophyll content index (CCCI) based on the normalized difference red edge (NDRE) and NDVI was developed ^[59][95]. When the canopy cover is above 30%, as an effective indicator, NDRE can be used to estimate crop N status. However, due to the influence of leaf characteristics, canopy structure, atmospheric conditions, and soil background, the obtained canopy spectra are mixed spectral information ^[51][23,86].

       First, the vegetation index is effective in reducing the influence of different backgrounds and can improve the reflectance sensitivity to crop N status. For example, the SAVI can minimize the soil background interference through a soil regulating parameter L, which is widely used in the measured and simulated data. Meanwhile, it has been proved that it can effectively decrease the impact of soil background and correct normalized difference spectral index (NDSI) for better diagnostic performance ^[60][96]. However, SAVI must know the density distribution or coverage percentage of the underlying vegetation in advance, so it is only suitable for extracting the vegetation information of the underlying vegetation in a small area with small vegetation coverage change. Besides, a multi-angular vegetation index (MAVISR), two red-edge-based indices, and red edge chlorophyll index (CIRE) have also proved to have a good performance in crop canopy N status diagnostics ^[36][61][71,97].

       Second, regarding water absorption of fresh leaves, a water removal technology was proposed ^[62][98], whose main idea is to remove the influence of water absorption and improve the diagnosis of N status ^[63][64][99,100] and N-P ratio ^[65][101]. By water removal technology and combining continuous wavelet analysis (CWA) in the SWIR, a better effect can be achieved. Additionally, the Datt index, Medium Resolution Imaging Spectrometer (MERIS) terrestrial chlorophyll indices (MTCI), a water resistance N index (WRNI) was also proposed to increase the accuracy of the LNC estimation model by minimizing the influence of water stress ^[66][102].

       Third, regarding the influence of canopy structure change, using ratio vegetation index (RVI) can slow down the expansion of saturation under dense canopies, which is still sensitive to the change of vegetation state after canopy closure ^[67][103]. However, the vegetation coverage affects RVI. When the vegetation coverage is high, RVI is very sensitive to vegetation. When the vegetation coverage is less than 50%, the sensitivity decreases significantly. Therefore, Li et al. ^[68][104] conducted a study on rice of different years, varieties, and growth stages, and found that RVI could be used to estimate the N status of over fertilized winter wheat before heading. Meanwhile, the RVI is affected by atmospheric conditions, which greatly reduces the sensitivity of vegetation detection, so atmospheric correction or reflectance calculation of RVI is needed before calculation.

       Finally, the relationship between the index and N status is often inconsistent, due to the change of canopy background and growth status in different stages of crop. GNDVI has been proposed as the most suitable spectral index to estimate the leaf N content in each growth stage of the corn, while SAVI performs better at the beginning of the season ^[69][105]. However, the canopy structure of different crops is different. Therefore, a different vegetation index or spectral index should be proposed for different crops in the later research to estimate the N status of crops.

       Whether it is leaf or canopy scale, the increase of computation and the massive data characteristics easily cause complex problems such as overfitting, which affects the estimation of the model. There are some methods to obtain the most relevant sensitive bands from high-dimensional data samples. For instance, the combination of principal component analysis (PCA) with a genetic algorithm ^[70][106], the partial least squares regression (PLSR) ^[71][107], and the Gaussian process regression (GPR) ^[72][108] can reduce the dimensionality of the original data, thereby decreasing redundant information in the data and obviously increasing the data validity.

1.3. Application of Estimation Methods Based on Spectral Data to Crop N Status Diagnosis

 1.3. Application of Estimation Methods Based on Spectral Data to Crop  N    Status Diagnosis

Although the accuracy of classification or regression can be significantly enhanced by increasing the number of wavelengths in the calculation process ^[5][40], the correct use of estimation methods is very important to improve the accuracy of crop N status diagnosis. Most studies used linear or multiple nonlinear regression models to construct the relationship between the spectral index and N status ^[73][74][109,110]. However, when the data contains a large number of characteristic dimensions, the correlation between these spectral indices and leaf N status are usually not very high, the estimation model is prone to overfitting and losing the accuracy of estimation. To solve the problems of multicollinearity and overfitting ^[10][45], the PLSR method can decrease extensive collinear variables to a little non-correlated factor and reduce the influence of background effects on model accuracy ^[75][76][77][111,112,113]. Meanwhile, PLSR usually stresses contiguous data, full-spectrum and efforts to identify and subset related spectral features are always ignored. The genetic algorithm with PLSR can realize the latter goals, but studies using this method are far fewer than that using PLSR alone ^[78][114]. Furthermore, based on the SAIL canopy model and the N-based PROSPECT model, an N-PROSAIL model was established and used for estimating crop N content both at canopy and leaf scales and which was proved to have great potential for crop N status diagnosis in wheat ^[79][115].

       Machine learning in crop N status diagnosis was reported in many recent studies ^[14][80][81][12,49,116,117]. For example, the artificial neural network (ANN) [12], the error backpropagation artificial neural network (BP-ANN) ^[10][45], the support vector machine (SVM) ^[82][83][118,119], SVM-PLS ^[84][120], and wavelet transform ^[85][121]. After comparing the stepwise multiple linear regression (SMLR) and ANN models of mangroves, it was found that the use of the ANN method for N status estimation produces satisfactory results [12]. ANNs also has many advantages in nonlinear modeling, because of its robustness and estimation ability under incomplete or noisy data ^[86][122]. However, there are some drawbacks such as complex input–response relationships in the use of ANNs for nonlinear modeling, which may not conform to physical or biological models [34]. The support vector regression (SVR) based radial basis function (RBF) kernel is better than the SMLR in canopy N content (CNC) estimation ^[81][117]. However, compared with general regression neural networks, SVR, and band ratio polynomial regression, the GPR has higher estimation accuracy ^[87][123]. As discussed by Verrelst et al. ^[88][124], GPR is more flexible for choosing kernel type than SVM and easier to train than the neural network ^[89][125]. Furthermore, some studies have demonstrated that the combination of other methods based on SVM has a higher advantage in the assessment of N status, such as least squares support vector machines (LS-SVM) and Savitzky–Golay support vector machines (SG-SVM) ^[13][48]. With the development of deep learning, it has been widely used in the field of agricultural research. However, there are few applications in spectral data processing and estimation. Deep learning can automatically combine and transform the low-order features of input data to get high-order features, which saves the manual work of constructing high-order features. Therefore, the feature extraction process based on deep learning is more accurate and faster, and we can use deep learning to extract features and establish models in order to achieve a better model estimation effect.

       In summary, although there are many measurement systems, data processing methods, and modeling algorithms mentioned above in the field of crop N status diagnosis, there are huge differences in the selection of spectral system, data pre-processing methods, and estimation methods due to the complex and variable crop growth environment and the influence of many factors. Therefore, it is difficult to find a unified measurement system, data processing method, and modeling algorithm to deal with the non-destructive diagnosis task of different agricultural scenes for the study of crop N status. Therefore, the future research should be devoted to make up for this problem.

2. Machine Vision Technology

2. Machine Vision Technology

       The machine vision technology can visually evaluate the N status by the shape, color, and texture of crops, and determine the N stress of seedlings by building a machine vision system to extract the object area of canopy ^[90][126]. Color image processing has been successfully applied to the diagnosis of crop N status and growth analysis ^[91][92][93][127,128,129], which can use digital cameras to get images in the visible light range that reflect the characteristics of the crop or soil background based on the R, G, and B spectral information (Figure 34). Then we do further segmentation and feature extraction of the image, and select the appropriate estimation method to construct the N status estimation model. If the segmentation effect is not good, the original R, G, and B bands can also be transformed into normalized color components, hue-saturation-intensity (HSI) space to improve the estimation accuracy in vegetation analysis ^[94][130].

Figure 34.

Basic principles of the machine vision technology.

2.1. Application of Machine Vision System Selection to the Diagnosis of Crop N Status

 2.1. Application of Machine Vision System Selection to the Diagnosis of  Crop    N Status

A digital camera is one of the main components of most machine vision systems, which is also used as a remote evaluation tool to monitor crop growth and N status by capturing crop images ^[49][95][34,85,131]. Moreover, hyperspectral and airborne miniaturized multispectral cameras have also been used to extract spectral and 3D features ^[96][132]. An artificial vision system (AVS) (HP Scanjet 3800) was developed for interpretation and analysis of images, which can acquire high quality images and estimate nutrient deficiency at different growth stages of crop, especially at the beginning of growth period, and may be helpful for early diagnosis and correction in the same growth cycle [24]. However, it cannot move flexibly. Tewari et al. ^[97][133] designed a manually operated four-wheel test trolley, which can flexibly acquire an outdoor color image feature of the crop under controlled illumination to estimate crop N status successfully in the field. Furthermore, some studies have proved that the combination of multiple diagnostic methods based on machine vision has a higher advantage in the assessment of N status, such as the combination of SPAD and machine vision ^[98][99][134,135] and the combination of spectroscopy and machine vision ^[100][101][136,137].

2.2. Application of Visual Data Processing to the Diagnosis of Crop N Status

 2.2. Application of Visual Data Processing to the Diagnosis of  Crop    N Status

The effective processing of visual data plays an important role in avoiding noise interference in the natural environment, such as soil, weeds, stones, dried, and semi-dried leaves in the image. One of the important steps of the image processing is to segment out various necessary regions and take it as the region of interest for decision-making of the crop N status ^[98][102][134,138]. There are some segmentation algorithms for automatic image segmentation, such as spatially varying mean intensity values, mathematical morphology, nonlinear spatial filtering, YCbCr color and grayscale morphology, which can be used to separate the plant from the background ^[49][85]. In the canopy image segmentation, the magnitude and distribution of the difference value can be obtained by subtracting the red channel value from the green channel, and which can be set for segmentation, and then the relationship between the characteristic parameters and N content can be established ^[102][138]. At present, there are still many problems in image processing. On the one hand, the external light condition is always changing in the process of image acquisition, which makes it a challenging task in image processing [4]. The image segmentation method based on the neural network can remove unnecessary components from plant images and keep the leaves as the region of interest, which can effectively avoid the influence of light intensity on image acquisition ^[103][139]. On the other hand, the complexity of the images obtained in the field makes it difficult for the traditional RGB color system to obtain the segmentation results accurately. However, the Lab color system has robust illumination variations and large color ranges, providing better performance than RGB and other color systems ^[97][133]. Therefore, we can transform RGB images into the Lab system to extract the channel of L, a, and b, then used the Otsu method combined with morphological processing and median filtering to obtain a binary image. Among them, L-channel can be used to segment objects from other crops using luminance differences, to remove scattered pixels by using morphological operations and median filtering, then to obtain the final segmentation results ^[104][140]. 

       Based on the segmented image, the RGB, Lab, HSI, and RLI channel information of the image is usually extracted for feature acquisition ^[97][101][133,137]. For example, a greenness index ^[105][141], some color feature parameters including dark green color index (DGCI), value (V) and hue (H) ^[99][135], the spatial and temporal distributions of the color index of the canopy such as G, G/R, G/B, NRI, NGI, NBI  [22] can also be used to acquire the crop N status. Among them, the NRI was regarded as a valid indicator that can better reflect on the N status of rice ^[98][134] and maize ^[106][142]. Besides, the total N status can influence the leaf color. A novel indicator named Growth Status (GS) was developed to reflect the crop growth conditions, which mainly includes GSMER and GSMCC versions, more precision results can be obtained by combining this indicator with the color factors (color characteristics of leaf surface) ^[104][140].

       Furthermore, previous studies ^[107][146] have also demonstrated that texture and color are the main visual features related to maize N status. There have some methods such as Gabor Wavelet (GW), Volumetric Fractal Dimension (VFD), and VFD with canonical analysis (VFDCA) for the texture analysis [24]. For instance, some non-destructive methods were proposed to extract 11 crop features from digital images, including a morphological feature (top projected canopy area), color features (the value of R, G, B, H, S, I), and textural features (entropy, contrast, homogeneity, and energy) ^[101][108][137,147]. Besides, as the shape and color of leaves dynamically change with the amount of fertilizer applied, some new feature parameters such as shape features (etiolation degree (ED), etiolation area (EA)), color features (normalized red or green index, etc.), and morphological features (perimeter, area) were proposed and used to assess the process of leaf change, which has a good potential in crop N status estimation ^[109][110][148,149]. However, sometimes, color and texture features will be misjudged due to the influence of external light. Therefore, this is a situation that needs to be considered to ensure the accuracy of the estimation model.

2.3. Application of Estimation Methods Based on Vision Data to Crop N Status Diagnosis

 2.3. Application of Estimation Methods Based on Vision Data to Crop  N    Status Diagnosis

It is a critical task to select an appropriate estimation method for establishing a robust estimation model, which can assess crop N status. Compared with the statistical method, ANN has good potential to process data. In particular, when the image feature is multifarious, and the original data do not follow a similar distribution pattern ^{[111][112][113]}[150,151,152]. For example, based on R, G, B channels of the color image obtained from a digital camera, a linear regression model and ANNs model named the multilayer perception neural network (MLPNN) was established and the result showed that the MLPNN model has better accuracy than the linear regression model ^[114][153]. As a popular algorithm, PLSR was used when processing multivariable data, and many studies have demonstrated that this method is powerful for acquiring key variables and establishing an accurate regression model ^[115][116][154,155], which is effective in estimating water and N status of winter wheat ^[109][145]. Moreover, combinations of algorithms are also common. The random forest (RF) was used as the estimator for crop N status and biomass estimation, and simple linear regression (SLR) was used for validating the consistency of the results of RF ^[96][132]. However, when there is a large amount of data, the above methods have some limitations in calculation efficiency and model accuracy. In this case, it is difficult to ensure the accuracy and real-time of crop N status assessment. With the development of deep learning in image processing, an ensemble of deep learning multilayer perceptron was proposed by using committee machines, which can be used for color normalization and image segmentation, and combine with a genetic algorithm (an optimization algorithm) to fine-tune the color normalization and achieve a good result in crop N estimation ^[117][156]. Among them, the principle of image processing and N status evaluation of this method is shown in Figure 5. Compared with linear regression, non-linear regression, and neural network, the deep learning method has higher training accuracy, but it requires a lot of data. When the amount of data is small, it is likely that the training results have been fitted. Therefore, in the later process of algorithm selection, we should choose whether to use a deep learning method according to the size of the data. 

Figure 5. Method based on deep learning for on-field N status estimation in plants.

       In summary, vision technology has been widely used in agricultural research due to its advantages of low cost and high precision, such as crop N status diagnosis and N stress research. However, the phenotypic phenomenon of leaves is not very obvious in the early stage of N deficiency, and it is difficult to catch the early symptoms of N deficiency by using visual technology. However, when N deficiency occurs seriously, although visual technology can realize low-cost identification, but the crop has been under serious stress, this affects crop yield and quality. In addition, most of the data involved in the visual technology are image data, so the traditional methods have some limitations, such as low efficiency and large amount of calculation. Although the deep learning method has a good effect on image data, it has certain requirements on the amount of data, and it is difficult to obtain large-scale image data in N diagnosis. With the development of crop phenotype technology, the research and application of phenotype platform will be able to solve this problem, and it can be used to obtain the image data of the whole growth stage of crops for the evaluation of crop N status.