Introduction

Grapevine is globally considered one of the most important crops for its economic relevance (Buonassisi et al., 2017). The occurrence of diseases caused by fungi, bacteria and viruses is the cause of severe economic losses due to the reduced quantity and quality of production. Among grapevine diseases, downy mildew is considered of primary importance (Buonassisi et al., 2017) due to the significant yield losses that it causes and the costs of the protection measures. Downy mildew is caused by the biotrophic oomycete Plasmopara viticola, an obligate parasite of grapevine. Under favourable weather conditions, the pathogen can infect all leaves and bunches of the plant if it is not adequately controlled (Toffolatti et al., 2018). As a result, grapevine downy mildew has become a pressing issue that has deserved attention worldwide.

The assessment of plant disease can be expressed in terms of incidence, i.e., the proportion of the plant units that are diseased in a defined population or sample (Madden et al., 2017), or severity, which is the proportion of the plant exhibiting visible disease symptoms or sporulation, usually expressed as a percentage (Madden et al., 2017). To perform the disease evaluation, personnel with appropriate competence and expertise must be selected (Mokhtar et al., 2015). The traditional phenotyping techniques, in fact, rely on visual ratings of the disease symptoms in grapevine leaves and bunches or the sporulation extent in experimentally inoculated leaves. More precisely, visual estimation is carried out by assigning a value to the severity of symptoms or sporulation perceived by the human eye. It requires a considerable time and can be subjective, i.e., influenced by the operator's ability to discriminate the disease symptoms/pathogen sporulation and associated with errors (Bock et al., 2010). The application of new sensing technologies gives the opportunity to remove the inconsistencies linked to the operator-based visual assessment and to improve the speed, accuracy, reproducibility and versatility of the disease assessment in several crops (Bock et al., 2020).

The application of new and emerging technologies has allowed the development of automatic and semi-automatic methods for plant disease assessment. Visualisation of the plant infections at different symptomatic stages can be achieved using several sensing technologies such as computer vision (Barbedo, 2013, 2014), hyperspectral imaging (Lowe et al., 2017), thermography (Stoll et al., 2008) and chlorophyll fluorescence (Cséfalvay et al., 2009). Methods based on the chlorophyll fluorescence, hyperspectral, and thermal images require expensive equipment and sophisticated analysis methods. In contrast, computer vision can be applied to RGB images that can be acquired using a large number of very accessible, economically-friendly devices (Lin et al., 2019).

Several reports have shown the usefulness of image analysis for the objective assessment of disease severity in different plant–pathogen interactions, including grapevine (Barbedo, 2014; Barbedo, 2018b; Bock et al., 2008; Boso et al., 2004; Corkidi et al., 2006; Stoll et al., 2008; Tucker et al., 2001; Wijekoon et al., 2008). Moreover, the assessment of disease severity throughout image analysis allows one to obtain high-throughput phenotyping data in a non-destructive and simple way and could offer new opportunities for a more precise evaluation of plant-pathogen interaction in leaf disc bioassays (Peressotti et al., 2011). Diseased visible symptoms can be assessed and distinguished from the infected area using computer vision techniques (Barbedo, 2013; Barbedo, 2014; Stewart and McDonald, 2014). Infected area segmentation has witnessed various algorithms, such as the segmentation of infected pixels based on grey levels (Price et al., 1993). Segmentation is used for partitioning the image to find regions of interest and it aims to separate the region having abnormalities. Images with disease spots on the leaves are considered and converted into a binary image separating background and leaf pixels. Leaf pixels are also divided into the diseased area and the rest of the leaf. Once the total disease area and total leaf area are known, the severity of the disease is estimated by calculating the percentage of the infected region (number of diseased pixels) over the entire leaf region (Mukherjee, 2020). All image analysis studies reported in the literature that are based on automatic or semi-automatic methods for the assessment of the disease severity have in common that digital evaluation is performed by detecting pixels with special features that define each particular disease symptom (Bock et al., 2010; Bock et al., 2020). Image segmentation is one of the most utilised techniques to classify the infected area from the entire plant organ region. Grey levels and histograms of intensities have been used for image segmentation (Barbedo, 2014; Camargo and Smith, 2009). According to some works (Barbedo, 2013; Zhang et al., 2020), thresholding can be a useful tool for segmenting images, detecting disease symptoms and quantifying disease infection in plants.

Fuzzy logic can be combined with computer vision techniques to extract information that humans could perceive from regions of interest in images. Recently, this approach has been suggested to evaluate plant diseases (Mukherjee, 2020; Sibiya and Sumbwanyambe, 2019) and enhance image segmentation to locate diseased areas (Sekulska-Nalewajko and Goclawski, 2011). The application of a fuzzy threshold in segmentation has been suggested to improve disease quantification algorithms (Nagi and Tripathy, 2020; Nagi and Tripathy, 2021). These works pointed to the diversity of the use of fuzzy logic and the closeness of the results provided by the fuzzy logic to the expert’s evaluation.

The aim of this work was to develop and validate a new method to automatically assess the severity of downy mildew disease in grapevine leaves by combining fuzzy logic and computer vision techniques.

Materials and methods

For the evaluation of the severity of downy mildew disease in grapevine leaves under laboratory conditions, a new method was developed combining computer vision techniques with fuzzy logic. The process is summarised in Figure 1. The first stage of the process was the preparation of the plant material and the RGB image acquisition. The second stage involved the use of computer vision techniques to find leaf discs in the image and to highlight the disease severity in each leaf disc. Finally, for validation, the downy mildew severity obtained from the sporulation located in each image was compared against the scores obtained by a visual assessment performed by experts.

Figure 1. Diagram of the methodology followed for the development of a method to assess the downy mildew severity on grapevine leaves. Grapevine leaf discs were prepared and imaged with an RGB (red, green and blue) camera. Leaf discs were evaluated by computer vision and fuzzy logic techniques. Finally, these evaluations were compared with an expert visual assessment for validation.

Save View full size Expand inline Collapse inline

2. Computer vision algorithm for downy mildew assessment

The new computer vision algorithm for assessing downy mildew in grapevine leaves consisted of four main steps: i) Leaf disc location in the Petri dishes, ii) image pre-processing, iii) segmentation of pre-processed images to separate the pixels representing downy mildew infection from the rest and iv) estimation of downy mildew severity for each leaf disc.

2.1. Leaf disc location on Petri dishes

Each leaf disc of the Petri dishes was located using Hough's transform (Yuen et al., 1990), obtaining the location of its centre on the image and its radius. This method segmented the discs ignoring the background and the Petri dish, regardless of potential differences in disc sizes.

To achieve this disc localisation, a preliminary step was required involving common colour correction (Figure 2). Likewise, the images belonging to Set-2 were adapted so that the approximate diameter of the discs was similar to those of Set-1 (from 400 pixels in diameter to 264 pixels), resizing the images in this set from 6000 × 4000 pixels to 3960 × 2640 pixels. Then, the colour space was changed from RGB to HLS to separate the colour (hue), lightness and saturation of each image and to use the saturation values to highlight the leaf discs from the rest of the image, avoiding the identification of the Petri dish due to its circular shape (Figure 2b). This HLS image was then smoothed with the median filter used in the initial stage of the mean shift segmentation (Comaniciu and Meer, 2002) and with the median blur filter to reduce image noise (Figure 2c). Finally, the saturation component of the blurred images was used for the localisation of the leaf discs with the Hough Transform from the OpenCV 4.5.1.48 library applying the Hough gradient method, with a minimum distance of 190 pixels between detected centres, an upper threshold of 100 for the internal Canny edge detector, a minimum radius to be detected of 115 pixels, a maximum radius to be detected of 145 pixels and an inverse ratio of the resolution of 1 (Figure 2d). Furthermore, the threshold for centre detection was automatically searched (varying the threshold between 11 and 25) to find eight circles in each image from Set-1 and nine circles in each image from Set-2.

Figure 2. Disc location steps: transforming the colour space from the original RGB (red, green and blue) image (a) to hue, lightness, saturation (HLS), applying the median filter of the mean shift segmentation (b), applying the median blur filter to the saturation component of the HLS colour space (c) and locating the discs using the Hough’s Transform (d). The located discs are highlighted with a green circumference (d).

Save View full size Expand inline Collapse inline

In most cases, leaf discs were not represented by a perfect circle, so the Hough Transform had problems detecting the borders of the discs with full accuracy, as seen in Figure 2. To improve this detection, a radius reduction of 20 pixels was applied, discarding part of the background close to the borders that could be detected as part of the sporulation.

2.2. Image pre-processing

Image pre-processing was carried out to achieve the best possible differentiation between P. viticola sporulation and the rest of the leaf. Based on the images in the HLS colour space used for the disc location, the saturation channel of these images was selected to find the pixels that represented pathogen sporulation. Then, a median blur filter with a kernel of 3 × 3 pixels was used in the image that represented the saturation component to blur the image. Finally, Contrast Limited Adaptive Histogram Equalization (CLAHE; Pizer et al., 1987) was applied to improve the histogram contrast of all images, preventing some images from having significantly high or low lighting.

2.3. Segmentation of pre-processed images

To obtain a clear separation of the pixels corresponding to P. viticola sporulation and those representing the rest of the leaf, one independent image was prepared for each leaf disc (Figure 3a). A pre-processed image of each leaf disc was obtained by the segmentation of each leaf disc with the same height and width as the disc diameter (Figure 3b).

Figure 3. Segmentation of a grapevine leaf disc. Original leaf disc (a), pre-processed image (grayscale) using the saturation component of hue, lightness, saturation (HLS) colour space (b), the segmentation obtained after applying the multi-threshold variant of the Otsu method (c) and histogram created with the saturation component and separated with segmentation (d). The separation of the red, blue and green regions (c) was made with the Otsu method.

Save View full size Expand inline Collapse inline

Segmentation of pre-processed disc images was carried out using a threshold that separated the pixels between pathogen sporulation and the rest of the leaf in each disc (Figure 3c). Since leaves exhibited different saturation values, the multi-threshold variant of the Otsu method (Liao et al., 2001) was applied to leaf disc images to obtain a precise segmentation, which allowed several thresholds to be found to divide an image based on the histogram of the image (Figure 3d). For each image, three thresholds were found, which separated background pixels (previously labelled with the value zero to focus the analysis on the localised discs), P. viticola sporulation, light areas of the leaf and darker areas of the leaf (Figure 3). Considering that the second threshold separated the pathogen sporulation from the rest of the leaf, this threshold was used to select the most appropriate threshold for all discs.

A crisp threshold (Cui et al., 2010; Gutiérrez et al., 2021), dividing the pixels of the discs into disease infection and the rest of the leaf using a fixed value, was used. Furthermore, a fuzzy threshold based on Nagi and Tripathy (2020), assigning a degree of infection to the pixels depending on the value of the pixels using membership functions within a range, was also used due to the variability of saturation values that represent the sporulation. For this purpose, the distribution of the thresholds obtained from all the discs was used to select three representative threshold ranges: from the first quartile to the median (th1, between 92 and 103), from the first quartile to the third quartile (interquartile range, th2, between 92 and 119) and from the minimum to the third quartile (th3, between 58 and 119).

The degree of downy mildew infection was then evaluated by assigning a membership value to each pixel of the leaf disc image according to its saturation value. The membership value using the fuzzy threshold was calculated with some of the best-known membership functions in fuzzy logic (Robinson, 2003; Sambariya and Prasad, 2017; Warner et al., 2019): gaussian, S-shape, sigmoidal, trapezoidal and generalised bell. For each disc image, each pixel (p) was evaluated with one of the threshold ranges obtained with the Otsu method, according to the application of the membership function (Ross, 2005). Therefore, with fuzzy thresholding, the values of the pixels with infection vary between zero and one (the likelihood of a pixel corresponding to infection). In the case of crisp thresholding, the mask resulting from the thresholding represents the pixels with downy mildew infection as ones and the rest as zeros.

Due to the similarity between leaf nerves and sporulation saturation values, nerves were mistakenly detected as part of the infection. To reduce the negative impact of this detection, the shape of each region labelled as part of the infection was analysed on each disc to ignore areas that were similar to lines representing leaf nerves. Nerves were defined as regions with an eccentricity greater than 0.95 to exclude regions similar to circumferences; a minor axis length less than 20 pixels to account for shapes with a small width; and a major axis length greater than 5 pixels, to discard small regions. Besides, to give greater importance to the pixels detected as downy mildew infection, the morphological operation of dilation with a structuring element of a size of 5 × 5 pixels was applied (Figure 4), thus joining and enlarging small areas of pixels labelled as sporulation, which in the human eye would be detected as single areas, instead of separate areas that would result in much less severity.

Figure 4. Processing of infection mask extracted from the disc image (a). The infection mask represents its regions with different colours (b), lines detected as nerves (c) and the final mask ignoring nerves and applying dilation morphology transformation (d).

Save View full size Expand inline Collapse inline

2.4. Downy mildew severity calculation

After obtaining a mask for each leaf disc (with the values of the pixels between 0 and 1), the sporulation was evaluated, and the severity was computed. Once the discs were located in each Petri dish image and the mask with the degree of infection of the pixels was obtained, the infection severity (s) in each disc was calculated considering the percentage of infection represented by the value of its pixels, according to the following equation:

$\text{s}\text{=}\frac{\sum _{i=0}^n{v}_i}{\text{n}}100\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{ }\left(10\right)$

where v_i represents the pixel value according to the degree of downy mildew infection detected by the membership function and n represents the total number of pixels assigned to the area of the disc which, depending on the radius obtained automatically by the Hough Transform, follows the equation $\text{n}\text{=}\text{π}{r}^2$.

3. Visual rating and validation

A visual rating of downy mildew severity was performed to validate the infection values yielded by the method developed using computer vision and fuzzy logic. Downy mildew severity in all leaf disc images was visually evaluated by eleven experts. Each panellist evaluated each leaf disc using a 0–100 scale (Bock et al., 2010), reflecting the percentage of downy mildew infection over the total leaf surface.

Prior to the visual evaluation of all leaf discs, the panellists were intensively trained. During the training, each panellist was provided with a downy mildew assessment scale composed of 20 leaf disc RGB images, covering a range of infection between 0 % and 100 %. Once the panellist got familiar with the scale, a training set of 36 leaf discs was provided together with a self-evaluation spreadsheet. Should the difference between the infection rating given by the panellist and that provided by the method differ more than 15 %, then re-training of the panellist was carried out. Only when the average difference between the panellist’s ratings and those yielded by the method was lower than 15 % then the panellist could start the evaluation of the leaf discs involved in the modelling.

During the visual inspection and rating of the leaf discs, each panellist assessed 370 leaf discs in two different sessions to avoid fatigue. To compare the infection ratings provided by the experts with those obtained by the new method, a value representative of all panellists was used. For this purpose, the evaluations of each disc were analysed using the interquartile range (IQR), the first quartile (Q₁) and the third quartile (Q₃) from the distribution of these evaluations to discard the extreme outliers (Chandola et al., 2009). Values outside the range between $Q_1-\left(3IQR\right)$ and $Q_3+\left(3IQR\right)$ were considered outliers. Once the extreme values of the evaluations were discarded, the average evaluation of each disc was considered as a representative value of the panellist visual rating.

The downy mildew severity values for each leaf disc obtained by the computer vision method were then compared with the average expert visual evaluation values (reference method). A simple linear regression model was applied to approximate the values obtained with the new method to the experts’ assessment. The determination coefficient (R²) and root mean square error (RMSE) were computed to compare the values obtained with the linear regression model and the expert assessment using 10-fold cross-validation.

Moreover, infection ratings were also analysed discretely, taking into account severity as levels (Nagi and Tripathy, 2020). To do this, different levels of infection severity were defined: low, including ratings between 0 and 25 %; middle, comprising severity values between 26 and 50 %; and high, including severity ratings between 51 and 100 %. Some examples of these assignments and the number of discs of each level available in each set can be seen in Table 1. The severity levels were compared using the accuracy, F1 score, recall and precision metrics. The definitions of these classification metrics are the following:

$Accuracy=\frac{T_L+T_M+T_H}{N} \left(13\right)$


$Precision=\frac{TP}{TP+FP} \left(14\right)$


$Recall=\frac{TP}{TP+FN} \left(15\right)$
    

$F1=2\mathrm{ }\frac{Precision\bullet Recall}{Precision\mathrm{ }+\mathrm{ }Recall} \left(16\right)$

where T_X in accuracy metric reflects the number of discs correctly classified with level x, representing x as L for the low level, M for the middle level and H for the high level, and N represents the number of discs evaluated. The last three metrics were used on each of the levels independently and their mean values were considered, with TP (true positives) being the number of hits for that level, FP (false positives) refers to the number of discs misclassified as that level, and FN (false negatives) corresponds to the number of discs misclassified as another level. A confusion matrix was also built to determine the number of discs correctly classified or designated as another level of severity.

Table 1. Examples of grapevine leaf discs showing downy mildew sporulation, severity level, percentage of downy mildew sporulation and number of leaf discs for each dataset.

View popup Expand inline Collapse inline

Grapevine leaf discs		Downy mildew severity level	Percentage of downy mildew sporulation (%)	Number of leaf discs
				Dataset 1	Dataset 2	All
		Low	0–25	30	214	244
		Middle	26–50	34	39	73
		High	51–100	45	8	53
		All	0–100	109	261	370

Results and discussion

The method developed provided information for each leaf disc in the Petri dish image (Figure 5), where the areas detected as infection (with blue values if the sporulation has a low intensity and pinker colours if the sporulation has a high intensity) and the percentage of infection calculated from the sporulation can be seen. This way, for each disc, the disease severity was evaluated through the evaluation of the level of downy mildew sporulation. Crisp thresholding (Figure 5b) detected the clearest pathogen sporulation. On the other hand, fuzzy thresholding also detected less clear sporulation, giving different values for areas of clear sporulation and areas of more intense sporulation when calculating severity (Figure 5c). Fuzzy logic allowed to provide an assessment of the severity of downy mildew close to the experts, treating regions with lower symptom intensity differently from others with higher symptom intensity. This is reflected in the visualisation of the symptoms (Figure 5c,f), where the mask representing the infection not only indicates the infected area but also provides visual information on the intensity of the symptoms in each part of the infected area. Apart from that, as seen in other works (Barbedo, 2014), when using pixel values to detect disease, it was necessary to discard leaf nerves that could be detected as having similar values to disease symptoms. This was achieved by the algorithm analysing the shape of the regions detected as sporulation, making these nerves minimally influential in the estimation of disease severity.

Figure 5. Original Petri dishes (a,d) with downy mildew estimation in each grapevine leaf disc using crisp (b,e) and fuzzy threshold (using the s-shape membership function) (c,f) in Set-1 (a–c) and Set-2 (d–f) were showed.

Save View full size Expand inline Collapse inline

Comparing the evaluation of the discs calculated by the new method with the visual evaluation provided by eleven trained experts (summarised in Figure 6), a positive linear relationship was observed in all the cases, reaching an R² value of 0.88 using a fuzzy threshold to segment all the discs (Figure 6e,f) and an R² of 0.84 for individual sets (Figure 6a–d). Despite the similarity of the results between the use of a crisp or fuzzy threshold, in all the graphs, it can be seen in Figure 6 that the evaluations calculated with the fuzzy threshold are more similar to the experts' evaluation. Although images from the two datasets presented differences in image capture, infection conditions and grapevine variety, the similar results from the same method in the two sets suggest that the developed method is consistent and robust against significant changes in image acquisition. This is also reflected in the comparison of the linear relationships using all the discs and those of each set individually, where using all the discs gives better results. Figure 6 also shows that the infection values provided by the method showed small differences with the experts’ evaluation (mostly by slightly underestimating the disease incidence), so a linear regression was used to adapt them.

Figure 6. The linear relationship between downy mildew severity, evaluated using computer vision techniques, and expert visual assessment. Computer vision evaluations performed by crisp threshold (a,c,e) and fuzzy threshold using the s-shape membership function (b,d,f) in Set-1 (a,b), Set-2 (c,d) and both datasets jointly (e,f) are shown. The dashed lines indicate a perfect linear relationship and the continuous lines indicate the real linear relationship.

Save View full size Expand inline Collapse inline

Adapting the severity estimation provided by the method to the experts' evaluation with linear regression, the proposed algorithm was evaluated using a 10-fold cross-validation, and the results are presented in Tables 2, 3 and 4, comparing severity values and levels. To see how P. viticola sporulation was best reflected by image saturation values, six types of thresholds and three ranges of saturation values were combined. Saturation ranges were estimated by applying the multi-threshold variant of the Otsu method to discs where sporulation differed from the rest of the leaf, considering the good performance of this method for separating the main areas of the images (Lin et al., 2019).

The results provided for both sets together (Table 2) indicate that the ranges chosen to separate sporulation from the rest of the leaf achieved a severity estimate similar to that provided by the experts, with a minimum R² value of 0.85 and a maximum RMSE value of 8.42 %, reaching an R² score of 0.87 and an RMSE of 7.61 % with a fuzzy threshold, using the s-shape membership function (in the range between the saturation value 92 and 103). The classification of leaf discs into three severity levels provided by the method and the carried out by the experts exhibited a good degree of coincidence, obtaining an accuracy of 86 %, an F1 score of 0.79, a precision of 0.83 and a recall of 0.76 with a fuzzy threshold, using the sigmoid membership function (in the range between the saturation value 58 and 119). The human evaluation of the level of infection in leaves can become a delicate work yielding very significant differences between assessments. This has been previously exposed by (Mukherjee, 2020), where fuzzy logic was used to achieve an appraisal that mimicked the subjectivity of the experts. The high accuracy values in the present work reflect the correct classification of most of the discs from the low level of infection, which has significantly more discs than the two other levels (as it can be shown in Table 1). This is particularly relevant as human inspection may disregard low levels of severity depending on the visual capability and acuity of each individual, which will not be overlooked by the method developed. Overall, the obtained results seem to indicate that the method developed offers improved performance in regression and classification using the fuzzy threshold, which allows yielding a quantitative estimation of the severity of infection close to that of the experts.

Considering the best performing thresholds in regression and classification and their equivalent crisp thresholds, the two different datasets were analysed individually (Tables 3 and 4). Table 3 shows the individual regression and classification results, displaying a large difference between the two datasets, especially in the classification results. Set-1 showed high levels of classification accuracy (77–83 %), while Set-2 was characterised by a high level of accuracy (88–89 %) but low F1 score, precision and recall scores (close to 0.50, Table 3). The number of discs correctly classified or classified as other levels can be seen in detail in Table 4, along with the confusion matrices associated with the methods that used the best performing thresholds (results split by datasets). In Set-1, the infection levels are correctly classified in most cases, in contrast to Set-2, where the classification is notable for the correct classification of the low severity level (correctly classifying 99 %), with most medium severity discs being classified as having low severity (51–54 %) and 75 % of high severity discs being classified as having medium severity. This occurs as the variability of the data was low, and the classification focused on lower severities due to this unbalance. As can be seen in Figure 6a,b, Set-1 contained severities varying between 0 and 100, making its estimation results more representative.

This new method has been developed using two large and different datasets: Leaves from two grapevine varieties and with image acquisition carried out under different indoor conditions (operators, artificial illumination and digital cameras). The new method was validated by expert visual assessment, as eleven trained experts evaluated each leaf disc from both datasets using a reference method in plant pathology. The strong relation between the expert visual assessments and the downy mildew severity computed by the algorithm suggests that fuzzy logic could assist adequately for automated disease evaluation very close to expert human assessment.

The developed method could be applied in numerous studies for the phenotypization of P. viticola traits, such as fungicide resistance and fitness (Corio-Costet et al., 2011; Massi et al., 2021; Toffolatti et al., 2015) or the evaluation of pathogen resistance in grapevine (Bellin et al., 2009; Possamai et al., 2020; Toffolatti et al., 2012) and efficacy of antifungal compounds (Colombo et al., 2020). The performance of the method could be improved in the future by trying to solve the errors due to the detection of nerves in the leaves, water droplets or light reflections by performing a segmentation with deep learning, which has been shown to have a high accuracy in similar problems (Lin et al., 2019). The effective application of deep learning requires a large and varied dataset (Barbedo, 2018a, 2018b), so to perform this type of segmentation and avoid overfitting more varied images of leaf discs would be needed, using different illumination, leaf discs with different sizes, different grapevine varieties, and, especially, leaf discs with severity percentages between 0 and 100 % and avoiding an imbalance of severity levels.

The main advantages of the new method are the following: Objective, quantifiable, rapid, automated and trained personnel is not required. The outcomes of this work agree with previous studies in demonstrating that the use of RGB images with computer vision techniques for the estimation of disease severity in plants provides advantages over expert estimation (Mukherjee, 2020; Nagi and Tripathy, 2020; Peressotti et al., 2011) as the latter involves some degree of subjectivity and require trained experts. In the same way, RGB and computer vision approaches could be preferred over the utilisation of other types of images, such as hyperspectral images (Bock et al., 2020), due to the lower cost of equipment and operator expertise required in the former, as well as the speed with which information is obtained from the plant images once the method is developed (Bock et al., 2020). With our new method, it is possible to estimate the disease severity of 6131 discs per hour, as opposed to manual assessment, where each trained expert assesses approximately 185 discs per hour. This implies an improvement in time compared to the expert evaluation, which, together with the objectivity that allows having the same method to analyse all leaf discs, and the automation of the evaluation, achieves an accurate and useful method for the evaluation of downy mildew in grapevine leaves.

This new method can be applied to other crops and plant diseases in agriculture, collecting leaves from the field and taking images with a digital camera under indoor conditions with artificial illumination. Fuzzy logic could help to treat differently the colours of the spots that represent the symptoms in grapevine leaves. Recently, Gutiérrez et al. (2021) used computer vision and deep learning to differentiate downy mildew and other pests using images taken under day-light conditions in commercial vineyards. These results suggest that the developed method can be applied to leaf images taken in both field and laboratory conditions.

In this work, computer vision and fuzzy logic techniques have enabled the estimation of the severity of downy mildew in the leaves of different grapevine varieties under laboratory conditions with high performance, automatically and objectively. Fuzzy logic has been commonly used in agricultural applications (when compared to other computer vision and machine learning approaches), but these results point out that it can be a powerful tool either as a core component or included in a larger automated identification system, opening several ways of applying fuzzy logic to more agricultural-themed solutions involving computing or electronics. The next steps would involve the adaptation of this method to be applied in the estimation of downy mildew or other fungal diseases, such as powdery mildew or botrytis, in field conditions, grapevines or in other crops. Improved monitoring of crop disease incidence at early stages would contribute to better tracking the infection and actively reacting (either through spraying or another strategy) in a more optimised way.

Conclusions

A new method, based on the combined application of computer vision and fuzzy logic, has been developed and validated for the automated evaluation of downy mildew severity in grapevine leaves. Fuzzy logic contributed to the assessment of disease severity according to sporulation intensities and provided a detailed visualisation of the different sporulation intensities. The estimation of downy mildew severity provided by this new method was strongly related to the visual assessment of the infection percentage rated by eleven trained experts. However, the new method was objective, cheap and fast enough to be automated and serve as decision support in the control of fungal crop diseases, such as downy mildew.

The developed method demonstrated its effectiveness with images of different grapevine varieties taken under different conditions, achieving an estimate of downy mildew severity similar to that provided by experts. This demonstrates the robustness of the method, based on fuzzy logic, and its possible application to other grapevine diseases or its adaptation to analyse leaves collected in the field.

Acknowledgements

This work has been developed as part of the project NoPest (Novel Pesticides for a Sustainable Agriculture), which received funding from the European Union Horizon 2020 FET Open program under Grant agreement ID 828940. It was also supported by the Spanish Ministry of Economy and Competitiveness under Grant PID2020-119478GB-I00. Inés Hernández would like to acknowledge the research funding FPI grant 1150/2020 by Universidad de La Rioja and Gobierno de La Rioja.

References

• Barbedo, J. G. A. (2013). Digital image processing techniques for detecting, quantifying and classifying plant diseases. SpringerPlus, 2(1), 660. https://doi.org/10.1186/2193-1801-2-660
• Barbedo, J. G. A. (2014). An automatic method to detect and measure leaf disease symptoms using digital image processing. Plant Disease, 98(12), 1709–1716. https://doi.org/10.1094/PDIS-03-14-0290-RE
• Barbedo, J. G. A. (2018a). Factors influencing the use of deep learning for plant disease recognition. Biosystems Engineering, 172, 84–91. https://doi.org/10.1016/j.biosystemseng.2018.05.013
• Barbedo, J. G. A. (2018b). Impact of dataset size and variety on the effectiveness of deep learning and transfer learning for plant disease classification. Computers and Electronics in Agriculture, 153(August), 46–53. https://doi.org/10.1016/j.compag.2018.08.013
• Bellin, D., Peressotti, E., Merdinoglu, D., Wiedemann-Merdinoglu, S., Adam-Blondon, A. F., Cipriani, G., Morgante, M., Testolin, R., and Di Gaspero, G. (2009). Resistance to Plasmopara viticola in grapevine “Bianca” is controlled by a major dominant gene causing localised necrosis at the infection site. Theoretical and Applied Genetics, 120(1), 163–176. https://doi.org/10.1007/s00122-009-1167-2
• Bock, C. H., Barbedo, J. G. A., Del Ponte, E. M., Bohnenkamp, D., and Mahlein, A.-K. (2020). From visual estimates to fully automated sensor-based measurements of plant disease severity: status and challenges for improving accuracy. Phytopathology Research, 2(1), 9. https://doi.org/10.1186/s42483-020-00049-8
• Bock, C. H., Parker, P. E., Cook, A. Z., and Gottwald, T. R. (2008). Visual rating and the use of image analysis for assessing different symptoms of citrus canker on grapefruit leaves. Plant Disease, 92(4), 530–541. https://doi.org/10.1094/PDIS-92-4-0530
• Bock, C. H., Poole, G. H., Parker, P. E., and Gottwald, T. R. (2010). Plant disease severity estimated visually, by digital photography and image analysis, and by hyperspectral imaging. Critical Reviews in Plant Sciences, 29(2), 59–107. https://doi.org/10.1080/07352681003617285
• Boso, S., Santiago, J. L., and Martínez, M. C. (2004). Resistance of eight different clones of the grape cultivar Albariño to Plasmopara viticola. Plant Disease, 88(7), 741–744. https://doi.org/10.1094/PDIS.2004.88.7.741
• Buonassisi, D., Colombo, M., Migliaro, D., Dolzani, C., Peressotti, E., Mizzotti, C., Velasco, R., Masiero, S., Perazzolli, M., and Vezzulli, S. (2017). Breeding for grapevine downy mildew resistance: a review of “omics” approaches. Euphytica, 213(5), 103. https://doi.org/10.1007/s10681-017-1882-8
• Camargo, A., and Smith, J. S. (2009). An image-processing based algorithm to automatically identify plant disease visual symptoms. Biosystems Engineering, 102(1), 9–21. https://doi.org/10.1016/j.biosystemseng.2008.09.030
• Chandola, V., Banerjee, A., and Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3), 1–58. https://doi.org/10.1145/1541880.1541882
• Colombo, M., Masiero, S., Rosa, S., Caporali, E., Toffolatti, S. L., Mizzotti, C., Tadini, L., Rossi, F., Pellegrino, S., Musetti, R., Velasco, R., Perazzolli, M., Vezzulli, S., and Pesaresi, P. (2020). NoPv1: a synthetic antimicrobial peptide aptamer targeting the causal agents of grapevine downy mildew and potato late blight. Scientific Reports, 10(1), 17574. https://doi.org/10.1038/s41598-020-73027-x
• Comaniciu, D., and Meer, P. (2002). Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5), 603–619. https://doi.org/10.1109/34.1000236
• Corio-Costet, M. F., Dufour, M. C., Cigna, J., Abadie, P., and Chen, W. J. (2011). Diversity and fitness of Plasmopara viticola isolates resistant to QoI fungicides. European Journal of Plant Pathology, 129(2), 315–329. https://doi.org/10.1007/s10658-010-9711-0
• Corkidi, G., Balderas-Ruíz, K. A., Taboada, B., Serrano-Carreón, L., and Galindo, E. (2006). Assessing mango anthracnose using a new three-dimensional image-analysis technique to quantify lesions on fruit. Plant Pathology, 55(2), 250–257. https://doi.org/10.1111/j.1365-3059.2005.01321.x
• Cséfalvay, L., Gaspero, G. Di, Matouš, K., Bellin, D., Ruperti, B., and Olejníčková, J. (2009). Pre-symptomatic detection of Plasmopara viticola infection in grapevine leaves using chlorophyll fluorescence imaging. European Journal of Plant Pathology, 125(2), 291–302. https://doi.org/10.1007/s10658-009-9482-7
• Cui, D., Zhang, Q., Li, M., Hartman, G. L., & Zhao, Y. (2010). Image processing methods for quantitatively detecting soybean rust from multispectral images. Biosystems Engineering, 107(3), 186–193. https://doi.org/10.1016/j.biosystemseng.2010.06.004
• Gutiérrez, S., Hernández, I., Ceballos, S., Barrio, I., Díez-Navajas, A. M., & Tardaguila, J. (2021). Deep learning for the differentiation of downy mildew and spider mite in grapevine under field conditions. Computers and Electronics in Agriculture, 182, 105991. https://doi.org/10.1016/j.compag.2021.105991
• Liao, P. S., Chen, T. S., and Chung, P. C. (2001). A fast algorithm for multilevel thresholding. Journal of Information Science and Engineering, 17(5), 713–727. https://doi.org/10.6688/JISE.2001.17.5.1
• Lin, K., Gong, L., Huang, Y., Liu, C., and Pan, J. (2019). Deep learning-based segmentation and quantification of cucumber powdery mildew using convolutional neural network. Frontiers in Plant Science, 10, 155. https://doi.org/10.3389/fpls.2019.00155
• Lowe, A., Harrison, N., and French, A. P. (2017). Hyperspectral image analysis techniques for the detection and classification of the early onset of plant disease and stress. Plant Methods, 13(1), 80. https://doi.org/10.1186/s13007-017-0233-z
• Madden, L. V., Hughes, G., and van den Bosch, F. (2017). The study of plant disease epidemics. In The Study of Plant Disease Epidemics. The American Phytopathological Society. https://doi.org/10.1094/9780890545058
• Massi, F., Torriani, S. F. F., Borghi, L., and Toffolatti, S. L. (2021). Fungicide resistance evolution and detection in plant pathogens: Plasmopara viticola as a case study. Microorganisms, 9(1), 119. https://doi.org/10.3390/microorganisms9010119
• Mokhtar, U., El Bendary, N., Hassenian, A. E., Emary, E., Mahmoud, M. A., Hefny, H., and Tolba, M. F. (2015). SVM-based detection of tomato leaves diseases. In Intelligent Systems’2014. Advances in Intelligent Systems and Computing (Vol. 323, pp. 641–652). Springer, Cham. https://doi.org/10.1007/978-3-319-11310-4_55
• Mukherjee, A. (2020). Analysis of diseased leaf images using digital image processing techniques and SVM classifier and disease severity measurements using fuzzy logic. International Journal of Scientific & Engineering Research, 11(8), 1905–1912. https://doi.org/10.14299/ijser.2020.08.12
• Nagi, R., and Tripathy, S. S. (2020). Infected area segmentation and severity estimation of grapevine using fuzzy logic. Advances in Computational Intelligence. Advances in Intelligent Systems and Computing, 988, 57–67. https://doi.org/10.1007/978-981-13-8222-2_5
• Nagi, R., and Tripathy, S. S. (2021). Severity estimation of grapevine diseases from leaf images using fuzzy inference system. Agricultural Research, 1–11. https://doi.org/10.1007/s40003-021-00540-4
• Peressotti, E., Duchêne, E., Merdinoglu, D., and Mestre, P. (2011). A semi-automatic non-destructive method to quantify grapevine downy mildew sporulation. Journal of Microbiological Methods, 84(2), 265–271. https://doi.org/10.1016/j.mimet.2010.12.009
• Pizer, S. M., Amburn, E. P., Austin, J. D., Cromartie, R., Geselowitz, A., Greer, T., ter Haar Romeny, B., Zimmerman, J. B., and Zuiderveld, K. (1987). Adaptive histogram equalization and its variations. Computer Vision, Graphics, and Image Processing, 39(3), 355–368. https://doi.org/10.1016/S0734-189X(87)80186-X
• Possamai, T., Migliaro, D., Gardiman, M., Velasco, R., and De Nardi, B. (2020). Rpv mediated defense responses in grapevine offspring resistant to Plasmopara viticola. Plants, 9(6), 781. https://doi.org/10.3390/plants9060781
• Price, T., Gross, R., Ho, W. J., and Osborne, C. (1993). A comparison of visual and digital image-processing methods in quantifying the severity of coffee leaf rust (Hemileia vastatrix). Australian Journal of Experimental Agriculture, 33(1), 97–101. https://doi.org/10.1071/EA9930097
• Robinson, V. B. (2003). A perspective on the fundamentals of fuzzy sets and their use in geographic information systems. Transactions in GIS, 7(1), 3–30. https://doi.org/10.1111/1467-9671.00127
• Sambariya, D. K., and Prasad, R. (2017). Selection of membership functions based on fuzzy rules to design an efficient power system stabilizer. International Journal of Fuzzy Systems, 19(3), 813–828. https://doi.org/10.1007/s40815-016-0197-6
• Sekulska-Nalewajko, J., and Goclawski, J. (2011). A semi-automatic method for the discrimination of diseased regions in detached leaf images using fuzzy c-means clustering. Perspective Technologies and Methods in MEMS Design, 172–175.
• Sibiya, M., and Sumbwanyambe, M. (2019). An algorithm for severity estimation of plant leaf diseases by the use of colour threshold image segmentation and fuzzy logic inference: A proposed algorithm to update a “Leaf Doctor” application. AgriEngineering, 1(2), 205–219. https://doi.org/10.3390/agriengineering1020015
• Stewart, E. L., and McDonald, B. A. (2014). Measuring quantitative virulence in the wheat pathogen Zymoseptoria tritici using high-throughput automated image analysis. Phytopathology, 104(9), 985–992. https://doi.org/10.1094/PHYTO-11-13-0328-R
• Stoll, M., Schultz, H. R., and Berkelmann-Loehnertz, B. (2008). Exploring the sensitivity of thermal imaging for Plasmopara viticola pathogen detection in grapevines under different water status. Functional Plant Biology, 35(4), 281–288. https://doi.org/10.1071/FP07204
• Toffolatti, S. L., Maddalena, G., Salomoni, D., Maghradze, D., Bianco, P. A., and Failla, O. (2016). Evidence of resistance to the downy mildew agent Plasmopara viticola in the Georgian Vitis vinifera germplasm. Vitis - Journal of Grapevine Research, 55(3), 121–128. https://doi.org/10.5073/vitis.2016.55.121-128
• Toffolatti, S. L., Russo, G., Campia, P., Bianco, P. A., Borsa, P., Coatti, M., Torriani, S. F. F., and Sierotzki, H. (2018). A time-course investigation of resistance to the carboxylic acid amide mandipropamid in field populations of Plasmopara viticola treated with anti-resistance strategies. Pest Management Science, 74(12), 2822–2834. https://doi.org/10.1002/ps.5072
• Toffolatti, S. L., Venturini, G., Campia, P., Cirio, L., Bellotto, D., and Vercesi, A. (2015). Sensitivity to cymoxanil in Italian populations of Plasmopara viticola oospores. Pest Management Science, 71(8), 1182–1188. https://doi.org/10.1002/ps.3906
• Toffolatti, S. L., Venturini, G., Maffi, D., and Vercesi, A. (2012). Phenotypic and histochemical traits of the interaction between Plasmopara viticola and resistant or susceptible grapevine varieties. BMC Plant Biology, 12(1), 124. https://doi.org/10.1186/1471-2229-12-124
• Tucker, C. C., Chakraborty, S., and Wilson, P. A. (2001). Configuration of “ASSESS” to measure rust severity on sugarcane and sunflower. Acta Horticulturae, 562, 397–400. https://doi.org/10.17660/ActaHortic.2001.562.47
• Warner, J., Sexauer, J., scikit-fuzzy, twmeggs, alexsavio, Unnikrishnan, A., Castelão, G., Pontes, F. A., Uelwer, T., pd2f, laurazh, Batista, F., alexbuy, Broeck, W. Van den, Song, W., Badger, T. G., Pérez, R. A. M., Power, J. F., Mishra, H., … 99991. (2019). JDWarner/scikit-fuzzy: Scikit-Fuzzy version 0.4.2 (v0.4.2). Zenodo. https://doi.org/10.5281/ZENODO.3541386
• Wijekoon, C. P., Goodwin, P. H., and Hsiang, T. (2008). Quantifying fungal infection of plant leaves by digital image analysis using Scion Image software. Journal of Microbiological Methods, 74(2–3), 94–101. https://doi.org/10.1016/j.mimet.2008.03.008
• Yuen, H., Princen, J., Illingworth, J., and Kittler, J. (1990). Comparative study of Hough Transform methods for circle finding. Image and Vision Computing, 8(1), 71–77. https://doi.org/10.1016/0262-8856(90)90059-E
• Zhang, N., Yang, G., Pan, Y., Yang, X., Chen, L., and Zhao, C. (2020). A review of advanced technologies and development for hyperspectral-based plant disease detection in the past three decades. Remote Sensing, 12(19), 3188. https://doi.org/10.3390/rs12193188