1. Introduction

Precision viticulture aims at maximizing grape yield and quality by minimizing input costs. Grape harvesting is the most important viticulture operation since the choice of the harvest time determines the desired quality of the yield. Identifying the maturity levels in vineyards could enhance the efficiency of harvesting operations ^[1]; especially in wine production, where the optimal harvest time, associated with specific concentrations of certain compounds, e.g., anthocyanins, is strongly related to the desired wine quality ^[2]. The precise time for grape harvest depends on the location, the duration of the growing season, the grape variety, the vine tree load, and the intended use of grapes, i.e., eating, or wine production. Environmental conditions also affect the ripening process [3,4]^[3][4]. Therefore, estimation of the exact harvest time is rather challenging; however, grape ripeness estimation is a less complex process and is performed regularly during veraison.

In this context, automated solutions for grape ripeness estimation are to be sought. Lately, research is focused on developing non-destructive, cost-effective, and environmentally friendly techniques. Machine vision is currently used excessively for agricultural-related tasks [7]^[5]. The technological improvement in hardware provides sensors that combine high performance and reasonable pricing, while innovative software design provides algorithms that can support effective real-time artificial vision systems. Towards this end, machine vision has been introduced to in-field applications for grape ripeness estimation [7,8]^[5][6]. Reported results reveal that image analysis can be used as a quick, efficient, and attractive alternative to chemical analysis, due to its simplicity, flexibility and low cost.

This work aims to comprehensively survey current applications of machine vision techniques for grape ripening estimation. The techniques reported here are recent and cover a wide range of image analysis applications. Each technique’s potential is analyzed; performance results, prediction models, input data, and pre-processing needs are reported. Suggestions of the most effective methods for specific applications and their limitations are also highlighted. The current and potential integration of reported methods in agricultural robots, namely agrobots, is examined, and future trends are discussed. This work critically reviews the most leading-edge methods in machine vision-based grape ripening estimation and, therefore, experts can use it as a complete guide to help them select the appropriate methodology to best fit their application.

The rest of the paper is structured as follows. Section 2 summarizes the peculiarities of grapes compared to other fruits regarding ripeness estimation. The most commonly used indices related to grape maturity are presented in Section 3 . Section 4 reviews machine vision methods for grape ripeness estimation. Limitations and perspectives are discussed in Section 5 . The integration of the reviewed algorithms in grape harvesting robots is examined in Section 6 . Finally, Section 7 concludes the paper.

2. Grape Ripeness Peculiarities

The physiological maturity of fruit occurs before the harvest maturity. When the fruit quality is acceptable to the customer, it reaches commercial maturity ^[1]. Physiological and commercial maturity need to be distinguished; commercial maturity is achieved when the development of the fruit is over even if the ripening process is not fulfilled, while physiological maturity is achieved when both maximum growth and maturity have occurred. In general, fruit maturity is estimated by using several maturity indices, summarized in the following section.

At this point, it should be noted that fruits are divided into two broader categories: climacteric and non-climacteric . Climacteric is a stage of fruit ripening related to increased production of ethylene, the required hormone for ripening, and a rise in cellular respiration. Climacteric fruits can produce ethylene even when they are detached from the crop and, thus, continue to ripen autonomously and change in taste, color, and texture. Apples, melons, bananas, and tomatoes are climacteric fruits. Non-climacteric fruits do not change in color and taste after being harvested. Citrus, grapes, and strawberries are non-climacteric fruits.

All varieties of grapes are non-climacteric. This means that grapes picked early in one day may taste different than those picked in the next day, and will not ripen any further so that they would all come to the same degree of maturity. Therefore, for grapes, it is important to keep sampling/tasting until the grapes are uniformly ripened and harvest all grapes of the same maturity level at the same time. After being harvested, grapes are sensitive. A general rule is that the more mature the grape is, the shorter is its post-harvest life. For grapes that need to be transported to distant markets/wineries, harvest must occur as soon as possible after reaching the desired maturity level and refrigerator tractors are required. If grapes are not harvested on time, the grape berries may shatter, become rotten, or be damaged by animals, i.e., birds, insects, etc., which severely affects yield quantity and quality.

Towards this end, the time of grape harvest is one of the most important and challenging viticultural decisions for grape producers due to: (1) the difficulty of assessing grape maturity in the vineyard after exhausting sampling, (2) harvesting all grapes at the same maturity level by organizing on standby human resources to harvest, and (3) maintaining and transporting the harvested product in time. Therefore, grape harvesting based on ripeness estimation could increase the sustainable production of grapes by improving the quality of harvested grapes due to homogenous ripened and equally fresh fruit. In this way, the post-harvest waste along the supply chain reduces due to less rotter/damaged grapes, with an additional reduction of the production costs and human labor due to sustainable resources management.

3. Grape Ripeness Estimation Indices

Each grape cultivar displays a different refractometric index that is related to maturity; table grapes are considered ripened at 16 o Brix, Sauvignon Blanc at 20–22 °Brix, Merlot and Cabernet at 21–23 °Brix, etc. Thus, it is obvious that the proper harvesting time is not related to a standard value, but to a desired value depending on the harvested variety. Moreover, the post-harvest application of viticulture practices is strongly related to harvest at optimal maturity; in the wine industry, the maturity level of harvested grapes determines the exact procedure, diffusional, enzymatic, or biochemical, to be subsequently applied [9]^[7], while for table grapes, the refractometric index is combined with the sugar/acid ratio in order to determine harvest time that reflects the consumer acceptability.

The most valuable quality indicators for grape maturity among those in Table 1 , are the SSC, pH, and TA ^[1], especially when combined. However, these common maturity parameters that constitute the definition of ripeness, may vary between different cultivars. For the latter indicators in the wine industry, regardless of the grape variety, the limits that collectively indicate a ripened grape are those summarized in Table 2 . It should be noted that the limits presented here are general and indicative and are intended to specify a wide range of values for each index as resulted from the bibliography [10]^[8]. It is well-known that it is not feasible for a single set of numbers to define ripeness for one or more grape varieties; ripeness can only be defined by the individual [11]^[9].

Subjective maturation criteria are just as important and are used in addition to and in conjunction with the objective criteria. The latter include sensory characteristics that can be discriminative among samples and related to both chemical measurements and consumer liking. A list of sensory attributes that are used as maturity indices is included in Table 3 . Color, size, and taste are the three main subjective attributes to determine grape maturity.

Grapes change color from green to red, dark blue, yellow, or white, depending on the variety. Color is the most important indicator of maturity. Upon the change of color are based all machine vision algorithms toward harvest automation. However, external grape color is not always a reliable indicator since many cultivars change color prior to ripening. The color of grape seeds is more discriminative; seeds in all cultivars turn from green to brown [12]^[10]. However, the latter investigation suggests the destruction/crush of grape berries, which is an invasive approach. Grape size is another pointer of the ripening of grapes. When grapes are ripened, they appear full in size and less firm when being touched. Taste is the most important sensory attribute to ascertain the ripeness level. This is the reason why chemical samplings usually are accompanied by taste samplings. Grapes are tasted regularly while ripening until they are as sweet as needed for their intended use.

4. Machine Vision Methods for Grape Ripeness Estimation

Figure 1 illustrates a tendency; over time, grape ripeness estimation techniques once could perform exclusively in the laboratory, then transferred in the vineyard initially due to the advent of portable sensors, and finally due to the rapid development of machine vision algorithms which is the current trend. However, machine vision techniques have also been applied in the laboratory and have been combined with portable sensors [13,14,15]^[11][12][13].

A set of 100 RGB images per sample for 150 samples was used to define the phenolic maturity of grape seeds in [15]^[13]. In total, 21 polyphenols were determined and correlated to CIELAB color channel values and morphological variables obtained from the images. Results revealed a high correlation coefficient for predicting the maturity stage of grapes. In [14]^[12], RGB images of grape berries and seeds were related to chemical phenolic compositions and classified as ripened or immature based on the browning index and morphological features by applying discriminant analysis models. A classification method that classifies grape bunches on-site in mature or undeveloped was suggested in [22]^[14]. First, the grape bunches were segmented and then classified based on texture and color features from HSV and RGB representation of the images.

In [26]^[15], visual inspection of grape seeds took place for grape ripening estimation by the Dirichlet Mixture Model (DMM), without the performance of chemical analyses. DΜM allowed modeling the color histogram of grape seeds to estimate ripening class memberships. A method for quality evaluation of table grapes was presented in [27]^[16]. Image analysis and machine learning techniques were employed to analyze color images and classify them in the predefined five quality classes. In [28]^[17], Convolutional Neural Networks (CNN) and Support Vector Machine (SVM) were used for the classification of grapes into unripen or ripen. Morphological features along with RGB and HSV values were used as inputs of the classification models. In [29]^[18], color histograms derived from RGB images were represented by Intervals’ Numbers (INs). Previous INs were fed to the NN in order to predict future INs, and thus, the grape harvest time. A CNN model for ripeness classification in eight classes was employed in [30]^[19]. RGB images were acquired under varying illumination and only texture features were extracted and considered as parameters for the model.

A hyperspectral imaging technique was proposed in [31]^[20] for the prediction of physico-chemical and sensory indices of table grapes. The reflectance spectra of berries were acquired and afterwards the berries were analyzed to compute pH, TA, and SSA. A Partial Least Square Regressor (PLSR) was employed to search for connections between physico-chemical indices and spectra information. Images of the grape berries were taken by an in-lab hyperspectral imaging system inside a dark room under a halogen light source. In [32]^[21], hyperspectral images were used to construct the spectrum of grape berries. The spectrum was then converted to an enological parameter. Simultaneous determination of pH, sugars, and anthocyanins took place by a Neural Network (NN).