Segmentation and recognition of filed sweet pepper based on improved self-attention convolutional neural networks

Segmentation and recognition of filed sweet pepper based on improved self-attention convolutional neural networks | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Segmentation and recognition of filed sweet pepper based on improved self-attention convolutional neural networks Weidong Zhu, Jun Sun, Simin Wang, Kaifeng Yang, Jifeng Shen, Xin Zhou This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-1484696/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 7 You are reading this latest preprint version Abstract Automatic and accurate recognition of the parts to be picked is the key to realize the intelligent picking of sweet pepper. However, pepper fruits are always covered by other organs, and small objects like stems and shoots are difficult to be recognized by machines or cameras under certain extreme conditions. To accurately segment and recognize all kinds of objects in sweet pepper images captured at night, three experiments were performed in this paper, and an enhanced model based on convolutional neural networks (CNNs) was eventually achieved. In experiment I, several semantic segmentation networks were trained on a small dataset, and the full-resolution residual network (FRRN) was taken as a primary network. Then, the impact of resolution of input images on the segmentation effect was investigated in experiment II. In order to strengthen the feature presentation of inconspicuous objects, the position attention module (PAM) was appended on top of the FRRN in experiment III. This architecture was further trained to provide more precise segmentation results. The experimental result shows that the mean intersection over union (IoU) is 78.88%, which is at least 1.94 percentage points higher than other models, and the mean pixel accuracy (PA) is 97.94% on the test set. The proposed method has higher generalization performance when facing unforeseen picking situations, meanwhile it is generic and can be applied to other fruits and vegetables. Sweet pepper Semantic segmentation Recognition Convolutional neural networks Machine vision Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 1 Introduction Sweet pepper is one of the most favored vegetables in the world, which has rich nutritive value and edible value [ 1 ]. According to data from internet, the annual production of sweet pepper is more than 10 million tons worldwide [ 2 ]. Since the economic benefits of sweet peppers are substantial, it is essential to maximize production efficiency [ 3 ] and product quality. And now, the automated harvesting of sweet peppers [ 4 , 5 ] is emerging as a key instrument to achieve high production efficiency in precision agriculture. In practical production, most of the sweet peppers are cultivated in greenhouses, and the high temperature and humidity in greenhouses make it difficult to work in them. Replacing human beings with machines for picking work has been the major trend in modern precision agriculture [ 6 ]. With the fast development of robot technology, the research of greenhouse fruit and vegetable picking robot [ 7 ] has become a hot spot. In recent decades, picking robot technology has developed rapidly, and many countries have made a profound study on this technology and made some achievements [ 8 ]. However, the problems of weak adaptability, slow picking speed, and low recognition accuracy in the greenhouse environment still need to be solved. In particular, image recognition technology is essential for robot to identify targets quickly and accurately. At present, researchers have made numerous researches on image recognition of fruits and vegetables. The main methods are based on the combination of machine learning algorithms and machine vision techniques. Wang et al. [ 9 ] used an improved Otsu segmentation algorithm to locate the cotton peach region and sampled RGB values of the pixels in different regions. Thus, the problem of image segmentation was transformed into the problem of pixel classification. As a result, they trained the ELM classification model and realized accurate segmentation of the cotton peach images. In the work of Dhingra et al. [ 10 ], the recognition of leaf diseases was studied. An extended segmentation technique based on neutron logic was used to extract the region of interest (ROI) of leaves. Then several features, including texture, color, and histogram, were employed to detect diseased leaves. Nine different classifiers were constructed, and the best classification accuracy of 98.4% was eventually obtained. To recognize and locate cucumber accurately, Bao et al. [ 11 ] proposed a multi-template matching method and established a multi-template matching library containing 65 cucumber images. The robot vision system can utilize the matching library to calculate the normalized correlation coefficient matrix of the target image one by one, and judge whether there is a target cucumber in the image. The aforementioned researches prove that image recognition technology has been widely used in the agricultural field to identify and locate vegetables accurately. However, most of the above methods adopt traditional machine learning, which mainly depends on multiple feature selection and application under specific background. It is difficult to play a role in the environment of the greenhouse. With the development of high-performance computing systems and the sharing of massive datasets, deep learning has gradually become one of the most important technologies in the field of intelligent agriculture. The basic deep learning tool used in this work is convolutional neural networks (CNNs), which have a strong feature extraction capability and are suitable for processing a large amount of input images in parallel. Therefore, CNN is one of the most powerful technologies in the field of image recognition, which is widely used in image classification [ 12 , 13 ], object detection [ 14 , 15 ] and image segmentation [ 16 , 17 ]. In recent years, CNN technology has been introduced into the field of fruit and vegetable image recognition. Zhao et al. [ 18 ] proposed an apple location method based on YOLOv3, which realized apple detection in a complex environment. This work provided a theoretical basis for the research and development of apple harvesting robot. Fu et al. [ 19 ] studied the problem of fast identification of multiple clusters of Kiwifruit in the field. They trained LeNet as a classification model to identify occluded fruits, overlapping fruits, adjacent fruits, and independent fruits, and finally achieved an average recognition rate of more than 85%. In the work presented by Yang et al. [ 20 ] based on the Mask-RCNN, Resnet-50 was adopted as a backbone network. On this basis, the feature pyramid network (FPN) architecture was integrated to construct the feature extraction network. Their method accurately located strawberry picking points, which was helpful to improve the working performance of strawberry picking robot. The aforementioned literatures demonstrate that it is feasible to recognize fruit and vegetable by convolutional neural networks instead of traditional machine learning. In this paper, an improved CNN architecture based on the FRRN was designed and verified, which was applied to segment sweet pepper images captured at night greenhouse. Specifically, PAM was appended on top of the FRRN to improve the feature representation, which contributes to more precise segmentation results. Furthermore, the influence of image resolution on segmentation results was explored. This study aimed to provide a theoretical basis for the development of image recognition performance of picking robot in precision agriculture. The rest of this paper is organized as follows. The dataset and experiment arrangements were introduced in Section 2 . Section 3 describes the model architecture of the FRRN combined with the PAM in detail. The training details of the model are listed in Section 4 . The experimental results are analyzed and discussed in Section 5 . The conclusions are drawn in Section 6 . 2 Materials And Experimental Scheme 2.1 Dataset description The sweet pepper dataset, used in this paper, was obtained from 4TU. 10 500 greenhouse sweet pepper images and their corresponding ground truth labels can be obtained at ResearchData ( https://data.4tu.nl ). These synthetic images were rendered through Blender based on 21 empirically measured plant properties. They highly simulated empirical images, which obtained by sweet pepper harvest robot from different angles at night greenhouse [ 21 ]. Besides, the sweet pepper dataset also contains 50 empirical images of the crop obtained from a high-tech commercial greenhouse. These empirical images were used in experiment III. The sweet pepper dataset was publicly released with the intention of comparing the performance of agricultural computer vision methods. There are 8 classes of segmentation objects in the picture, including background, leaves, peppers, peduncles, stems, shoots, wires, and cuts for picking pepper. In the labeled images, 8 classes were annotated on a per-pixel level, corresponding to the color of black, blue, yellow, red, pink, green, white, and light blue. In Fig. 1 , examples of synthetic images are shown. 2.2 Overview of experiment arrangement To obtain the optimal segmentation model of the sweet pepper image, three main experiments were designed and carried out in this study. Overview of performed main experiments I through III are shown in Table 1 . Experiments I: A suitable CNN architecture was picked out for sweet pepper segmentation. Since the same network structure may perform differently on different datasets, in a specific task, it is necessary to select the appropriate network according to the dataset. Existing semantic segmentation network models are mostly applied to segmentation of street scenes, and rarely used in agricultural scene. In this experiment, several model structures were applied on the same sweet pepper dataset for training and testing, and the differences in segmentation performance of different models were analyzed. Each network structure is verified with the same number of images, 400 images were used as training set, and 100 images were used as validation set and test set respectively. Since different network configurations has different requirements for the size of the input images, the size of the input images was set to 384×384 to balance the training time and model performance. Experiments II: The effect of image resolution on segmentation results was investigated. After experiment I, a suitable model architecture for the sweet pepper segmentation task was obtained. Then the impact of the input image size on the segmentation performance of the model was further explored. Images with different sizes were input into the model for training. Unlike experiment I, experiment II was performed on a larger dataset. In this experiment, 10 500 sweet pepper images were divided into training set, validation set and test set according to 3:1:1, specifically, 6 300 images in training set, 2 100 images in validation set and test set respectively. Experiments III: The self-attention mechanism was introduced into the network to improve the segmentation performance. The self-attention mechanism can effectively establish dependencies between remote features, and make the model focus on extracting some key features. In our work, the PAM was used to capture global dependencies in the spatial dimensions. It was added on top of the FRR. Afterwards, the enhanced model was trained and tested on our dataset. The size of the input image and the distribution of the dataset in experiment III were the same as that in experiment II. Finally, the model with the best segmentation performance was further evaluated on empirical (real) images. Table 1 Overview of performed main experiments I through III Experiment Methods/Architecture Input size/pixel Training set Validation set Test set I-A ~ I-E Several CNN-based models 384×384 400 100 100 II-A FRRN 384×384 6 300 2 100 2 100 II-B FRRN 512×512 6 300 2 100 2 100 III-A FRRN + PAM 384×384 6 300 2 100 2 100 III-B FRRN + PAM 512×512 6 300 2 100 2 100 3 Segmentation And Recognition Model 3.1 Semantic segmentation network architecture Semantic segmentation method was used to segment and recognize sweet pepper in this paper. The goal of semantic segmentation is to segment and parse an image into different image regions associated with semantic categories (e.g. leaf, sweet pepper, shoot). Semantic segmentation models based on convolutional neural network have attracted extensive attention since FCNs [ 22 ] was proposed, and have made great progress. The existing depth-based semantic segmentation methods are mainly divided into two kinds: one is based on regional classification represented by Mask R-CNN [ 23 ], the other is based on pixel classification represented by FCNs. The former method is based on target detection. Firstly, the image is divided into different image patches, then each pixel in the image block is semantically classified. Finally, semantics segmentation is realized. This kind of method first appeared in R-CNN and gradually evolved into improved algorithms such as Faster R-CNN [ 24 ] and Mask R-CNN. The advantage of this method is that it can accomplish both tasks of target detection and semantic segmentation at the same time. However, due to the lack of global information of the image, this method is not effective for small-scale objects and small areas. Therefore, this method was not considered in our experiments. The semantic segmentation method based on pixel classification is favored by many researchers and has been widely used. In particular, dilation convolution, as well as atrous spatial pyramid pooling was adopted in the Deeplab series [ 25 , 26 ] to embed contextual features of different scales. Moreover, the encoder-decoder structures can effectively fuse mid-level and high-level semantic features. In experiment I, since the methods based on pixel classification performed better in fine segmentation task, we mainly adopted them. 3.2 Full-Resolution Residual Networks This section briefly introduces the FRRN, which is the backbone of the improved model in this paper. FRRN is a novel network structure similar to ResNet [ 27 ]. It does not rely on the pre-trained networks and can start training from scratch. In addition, since FRRN combines two distinct processing streams, it has strong recognition performance and precise localization capabilities. As shown in Fig. 2 , one stream is pooling stream: semantic information is captured through a sequence of convolution and pooling operations for identification. The other stream is a residual stream: feature maps at the full image resolution are carried to achieve precise boundary adherence. Meanwhile, FRRN consists of a series of full resolution residual units (FRRUs). The structure of FRRU is given in Fig. 3 . Each unit includes two inputs and outputs, because they are applied to both streams at the same time. If \({z}_{n-1}\) is the residual input of the n th FRRU and \({y}_{n-1}\) is pooling input. Then the outputs are computed as: \({z}_{n}={z}_{n-1}+H\left({y}_{n-1},{z}_{n-1};{W}_{n}\right)\) (1) \({y}_{n}=G\left({y}_{n-1},{z}_{n-1};{W}_{n}\right)\) (2) where H( · ) and G( · ) represent residual calculation and pooling operation respectively. \({W}_{n}\) is weight matrix. 3.3 Position attention module The attention mechanism in deep learning is generated by imitating human selective visual attention mechanism. It has been widely used in image processing [ 28 ], speech recognition [ 29 ] and natural language [ 30 ] in recent years. The essence of attention mechanism lies in the automatic and efficient allocation of attention resources, which contributes to the acquisition of the most critical information to solve the current task. There are many variants of the attention model, including the soft attention model, the global attention model [ 31 ], and the key-value attention model [ 32 ]. PAM can adaptively aggregate long-range context information to improve the feature representation of semantic segmentation. The implementation details of PAM are shown in Fig. 4 . A represents a local feature, where A \(\in {R}^{C\times H\times W}\) . It is first fed into a convolution layer to generate two new feature maps B and C , respectively, where {B,C}∈R C×H×W . Then they are reshaped to \({ R}^{C\times N}\) , where \(N=H\times W\) is the number of pixels. After that, a matrix multiplication is performed between transposed B and C , and a softmax layer is applied to calculate the spatial attention map M \(\in {R}^{N\times N}\) : (3) where M ji measures \({i}^{th}\) position’s impact on \({j}^{th}\) position. The more similar feature representations of two positions contribute to a greater correlation between them. Meanwhile, feature A is also fed into a convolution layer to generate a new feature map D \(\in {R}^{C\times H\times W}\) and is reshaped to \({ R}^{C\times N}\) . Then a matrix multiplication is performed between D and M , and the result is reshaped to \({ R}^{C\times H\times W}\) . Finally, it is multiplied by a scale parameter \(\alpha\) , and an element-wise sum operation is performed between it and features A to obtain the final output Q \(\in {R}^{C\times H\times W}\) as follows: (4) where \(\alpha\) is initialized as 0 and gradually assigned more weight. 3.4 Attention module embedding with full-resolution residual networks It can be inferred from Eq. 4 that the resulting feature Q at each position is a weighted sum of the features across all positions and original features. Accordingly, the PAM provides a global contextual view and selectively aggregates contexts according to the spatial attention map. In the sweet pepper images, some ‘stem’ and ‘shoot’ are inconspicuous or incomplete objects because of the influence of lighting and view. Those dominated salient objects (e.g. leaf, sweet pepper) would harm those inconspicuous object labeling. Since the PAM can selectively aggregate the similar features of inconspicuous objects to highlight their feature representations and avoid the influence of salient objects, we appended the PAM on top of the FRRN. The proposed FRRN-based network structure for semantic segmentation of the sweet pepper image is shown in Fig. 5 . As illustrated in Fig. 5 , FRRN was employed as the backbone. The whole network structure mainly refers to Encoder-Decoder [ 33 ] structure. When the network starts working, the input feature maps are down-sampled through four pooling layers to extract the deep features in the images. After that, 4 up-sample layers are used to restore the size of the feature maps. And the mapping relationship between the original image and the label is obtained. Then the features from the FRRN would be fed into the PAM. Specifically, PAM is cascaded behind the concatenate (fusion) layer. The output features of the attention module are transformed by the convolution layer and input into the last convolution layer to generate the final prediction map. Besides, the notations RU and FRRU in Fig. 5 refer to the residual unit and the full-resolution residual unit, and the numbers on the lower right corner represent the number of convolution kernels in the unit, respectively. Also, the number of convolution kernels of each convolution layer in Fig. 5 is 48, 32, and 8 (number of classes to predict). The sizes of convolution kernels are 5×5, 1×1, and 1×1, respectively. The model architecture is proposed to contribute to the field of agricultural segmentation task. 4 Model Training 4.1 Experiment platform The experiments in this work were conducted on the Ubuntu 16.10 LTS 64bit system. All the methods were implemented based on Tensorflow, and the TF-Slim library provided main functions for building model framework. The computer is equipped with 16 GB RAM and Intel Core i7-7700K CPU. Meanwhile, NVIDIA GTX1080Ti GPU acceleration technology was applied to improve the training speed. 4.2 Implementation Details The RMSProp optimization algorithm was adopted during the process of training models. For the experiment I, the base learning rate was set to 1e-4, and the weight decay coefficient was set to 0.995. For the remaining experiments, the base learning rate and attenuation coefficient were set to 1e-5 and 0.995, respectively. Batch size was only set to 1 for all datasets because of the limitation of GPU memory and computing power. The number of iterations was set to 300 epochs for the experiment I and 200 epochs for the other experiments. The loss function used in our study was the softmax cross-entropy function. Besides, the data augmentation technology was not adopted in the work of this paper. The mean IoU is the most important segmentation evaluation metric, which is computed as follows: \(\text{M}\text{I}\text{o}\text{U}=\frac{1}{k+1} \sum _{i=0}^{k}\frac{{p}_{ii}}{{\sum }_{j=0}^{k}{p}_{ij}+{\sum }_{j=0}^{k}{p}_{ji}-{p}_{ii}}\) (5) where \(k\) represents different classes, \({p}_{ii}\) is the number of pixels correctly predicted, \({\sum }_{j=0}^{k}{p}_{ij}\) is the number of pixels in which class i is predicted to be class j . The precision, recall, and F 1 score were introduced as additional evaluation indexes to evaluate the segmentation effect of sweet pepper images more objectively. In general, the higher and closer the precision and recall, the better the classification recognition effect. The F 1 score is computed based on the mean per-class precision/recall results as follows: \(F1= 2\times \frac{precision\times recall}{precision+recall}\) (6) 5 Results And Discussion 5.1 Evaluation of experiment I In experiment I, several existing semantic segmentation methods were evaluated and compared on a small sweet pepper dataset. The mean pixel accuracy (PA) and mean intersection over union (IoU) of the trained models on the test set are shown in Table 2 . These model architectures achieved good segmentation results when used for sweet pepper segmentation in this paper. It is worth noting that most of these methods were training from scratch, rather than based on the pre-trained models. The approach based on pre-trained models was also adopted in our experiments. The results showed that although the training time of this method was less than other means, the segmentation effect was far inferior to those methods training from scratch. Consequently, the results of approach based on pre-trained models were not shown and compared in this paper. As shown in Table 2 , the mean PA and mean IoU of FRRN are 94.3% and 69.18%, respectively. In particular, its recognition accuracy for sweet peppers reached 93.2%. The segmentation and recognition effect of FRRN is better than that of other methods. From the perspective of pixel classification, the mean PAs of the five methods are similar, and all of them exceed 90%. It illustrates that the five networks have achieved good performance in pixel classification, especially for salient objects (e.g. leaf, sweet pepper). Meanwhile, the mean PAs of sub-experiments I-A, I-B, and I-E are slightly higher than that of the other two methods. From Table 2 , it can be seen that the main difference is caused by the small objects, including peduncles, stems, shoots, and wires. The PA of wires in experiment I-C was 6.30%, while that of peduncle in experiment I-D was only 6.00%. FRRN performs better than other methods, which verifies that it is more suitable for small-scale object recognition and fine segmentation. Moreover, the effects of several methods can be visualized in Fig. 6 . These visualizations show that the FRRN achieved better segmentation results, both numerically and qualitatively. However, the segmentation effect of FRRN for sweet pepper needs to be improved, especially for the pixel-level recognition of inconspicuous objects. Hence, we took experiment II and experiment III. Table 2 Results of each category on a small sweet pepper test set Methods Pixel accuracy/% Mean IoU Mean PA Background Leaf Pepper Peduncle Stem Shoot Wire Cut I-A-MobileUNet [ 34 ] 93.6 93.8 97.5 93.5 46.4 81.1 64.5 73.5 59.6 67.82 I-B-SegNet [ 33 ] 93.7 93.8 96.9 94.1 50.3 80.3 63.5 73.2 63.7 67.33 I-C-BiseNet [ 35 ] 90.7 90.9 96.2 90.8 44.1 73.2 40.9 6.30 54.5 53.78 I-D-ED [ 33 ] 92.0 92.3 96.7 88.2 6.00 73.6 55.7 55.5 50.6 55.80 I-E-FRRN 94.3 93.7 97.6 93.2 53.1 81.2 68.3 69.5 67.5 69.18 5.2 Evaluation of experiment II Semantic segmentation belongs to pixel-level tasks, which need to provide pixel by pixel output. Consequently, semantic segmentation requires higher resolution and more detailed information. It is supposed that the higher resolution of the input images, the richer feature information that the convolutional neural network could capture. High-resolution images help to improve the recognition accuracy and segmentation performance of the model. Especially for fine agriculture segmentation, higher image resolution is beneficial. However, if the resolution is extremely high, maybe it causes the problem of slow training speed. In this study, the resolution of the original images in the dataset is 800×600, but the general model cannot directly perform feature analysis and extraction on such images. To balance the relationships between segmentation effect and computational complexity, the original image was cropped to 384×384 and 512×512 in experiment II, and train FRRN on images of different sizes. With the purpose of verifying the effect of input image resolution on segmentation results, we conducted experiment II. As shown in Table 3 , the higher segmentation performance was achieved by the same model after inputting high-resolution images. Compared with the input size of 384×384, employing an input image size of 512×512 yielded a result of 76.81% in mean IoU, which bringed 3.14% improvement. This shows that the increase in resolution contributes to the refinement of segmentation results. Besides, when the size of the input image is 512×512, the mean PA and the F 1 score further improves to 95.47% and 95.57%, respectively. Meanwhile, although the improvement effect of pixel classification is not significant, the precision and recall have been kept at a high level and close to each other, which indicates that the classification effect of the model has improved. Results show that, in a certain range, the higher the resolution of the input images, the better the segmentation performance of FRRN. Table 3 Evaluation and test results of different methods Method(model) Size of input image/pixel Mean PA/% Mean IoU/% Precision/% Recall/% F1 score/% FRRN 384×384 95.10 73.67 95.55 95.10 95.22 FRRN 512×512 95.47 76.81 95.97 95.47 95.57 FRRN + PAM 384×384 97.38 76.45 97.64 97.38 97.51 FRRN + PAM 512×512 97.94 78.88 98.20 97.94 98.07 Barth et al. [ 36 ] 300×300 - 55.00 - - - 5.3 Evaluation of experiment III The PAMs were employed on top of the FRRN to capture long-distance dependencies for better feature presentation. Similarly, two different resolution images were used as input to train the improved network. Various evaluation indexes achieved by the model on the test set are shown in Table 3 . The segmentation performance of PAM on two different resolution images is shown in Table 3 . It is observed that the best performance was achieved at an input image resolution of 512×512. Compared with the baseline FRRN, the employment of PAM yielded a result of 78.88% in mean IoU, which brought an improvement of 2.07 percentage points. Also, mean PA increased by 2.47 percentage points. Meanwhile, the precision, recall, and F 1 score have also improved. In conclusion, results show that attention modules bring benefits to sweet pepper segmentation. Quantitative evaluation is not enough to verify the importance of PAM, thus the effects of PAMs were visualized in Fig. 7 . As expected in Section 3.4 , the PAM helps to strengthen the feature presentation of some inconspicuous objects. It can be observed from Fig. 7 that the employment of PAM makes some segmentation details and object boundaries clearer. As shown by the red circles in the first row, it is challenging to segment small-scale objects in detail, but under the action of PAM, the segmentation effect of the third column is significantly better than that of the second column. Specifically, in the second row, the FRRN model with PAM is better than the original FRRN model for the segmentation of small objects such as leaf stems. Therefore, Fig. 7 demonstrates that selective fusion over local features enhances the discrimination of details and refines the segmentation results. Besides, compared with the research of Barth et al. [ 36 ], the accuracy of pepper segmentation has been improved obviously. To explore the generalization performance exhibited by a synthetically trained model when faced with similar data in the same domain, the best performing model (model of experiment III-B) was evaluated on 20 empirical (real) images. The segmentation results of the proposed model on empirical data are shown in Fig. 8 . The model failed to discriminate and segment all the classes properly. This result confirms our hypothesis that the synthetically bootstrapped model could not generalize accurately to empirical data without fine-tuning. Note that these empirical images were not used for training or validation. In order to prove our hypothesis, 30 empirical images were used to fine-tune the proposed model, and the generalization performance of the model was tested on 20 empirical photos. As illustrated in Table 4 , this practice resulted in increased performance, with a mean IoU of 59.81%. The result implies fine-tuning on a synthetically trained network can generalize to similar data in the same domain (empirical images). Note that, for the empirical dataset, due to the influence of dark regions, some objects that were not manually annotated in the ground truth would be detected in the images and have a certain impact on the segmented images. Hence, although these parts were true positive, they were still evaluated as false positives. This annotation bias resulted in a lower segmentation performance of empirical images. In short, the results of experiment III show that the proposed optimization model of FRRN combined with PAM is feasible and effective for the recognition and segmentation of sweet pepper. Table 4 The evaluation results of the proposed model on empirical (real) images model fine-tuning Mean PA/% Mean IoU/% FRRN + PAM 73.42 51.63 FRRN + PAM \(\surd\) 80.56 59.81 5.4 Results compared with other models We further compared our method with existing methods on the sweet pepper testing set. The results are shown in Table 5 . The FRRN + PAM model performs better than other segmentation methods. It is supposed that these networks are designed for street scene segmentation tasks, so they maybe not perform well in agricultural segmentation tasks. Among these methods, in order to achieve real-time performance, only ENet refrained from using a pre-trained network when designing the structure, so it did not get high scores. Specifically, a pre-trained network was abandoned by our network either, but it outperformed ENet by a large margin. Moreover, it also surpassed DANet, which uses a complex backbone ResNet-101. Table 5 IoU scores on sweet pepper test set Method Mean IoU/% ENet [ 37 ] 65.78 BiseNet [ 35 ] 69.75 DANet [ 38 ] 76.94 Our method (FRRN + PAM) 78.88 6 Conclusions In this paper, CNN was applied to the semantic segmentation of the sweet pepper images. The FRRN model was improved by combining with attention mechanism. Specifically, PAMs were embedded at the top of the network to capture global dependencies in the spatial dimensions. This new architecture is clean, requires no additional post-processing, and can be trained from scratch. The proposed network (FRRN + PAM) was trained to segment sweet pepper images captured at night. Relevant experimental results show that PAMs contribute to providing more precise segmentation results. Meanwhile, our experiments indicate that proper image resolution benefits for segmentation performance. Besides, the synthetically trained model combined with fine-tuning can also be well generalized to the similar data in the same domain (empirical dataset). Our method achieves good performance on the test set of sweet pepper, with a mean IoU of 78.88%. The proposed method is generic and can be applied for other fruits and vegetables. It is hoped that this study can provide a theoretical basis for the development of image recognition in precision agriculture. Declarations Acknowledgements This work is partly supported by Project of Agricultural Equipment Department of Jiangsu University (NZXB20210210), a project funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD) and Jiangsu University undergraduate scientific research project (20AB00). CONFLICT OF INTEREST The authors declare no conflicts of interest. References López-Marín, J., Gálvez, A.: Selecting vegetative/generative/dwarfing rootstocks for improving fruit yield and quality in water stressed sweet peppers. Sci. Hortic. 214 , 9–17 (2017) Gilanie, G., Nasir, N., Bajwa, U.I. et al.: RiceNet: convolutional neural networks-based model to classify Pakistani grown rice seed types. Multimed. Syst. 27 , 867–875 (2021) Ghimire, S., Shakya, S.M., Srivastava, A.: Effects of organic manures and their combination with urea on sweet pepper production in the mid-hills. J. Agric. Environ. 14 , 23 (2018) Bac, C.W., Hemming, J., Van Tuijl, J., Barth, R., Wais, E., Van Henten, E.J.: Performance evaluation of a harvesting robot for sweet pepper. J. Field Robot. 34 (6), 1123–1139 (2017) Lehnert, C., English, A., Mccool, C., Tow, A.W., Perez, T.: Autonomous sweet pepper harvesting for protected cropping systems. IEEE Robot. Autom. Lett. 2 (2), 872–879 (2017) Luo, Y.Q., Sun, J., Shen, J.F., Wu, X.H., Wang, L., Zhu, W.D.: Apple leaf disease recognition and sub-class categorization based on improved multi-scale feature fusion network. IEEE Access 9 , 95517–95527 (2021) Ding, X.: Pepper picking robot in greenhouse. Agric. Eng. Technol. 39 (1), 78–82 (2019). Sun, J., He, X.F., Ge, X., Wu, X.H., Shen, J.F., Song, Y.Y.: Detection of key organs in tomato based on deep migration learning in a complex background. Agriculture-Basel 8 , 196–210 (2018) Wang, J., Zhou, Q., Yin, A.: Self-adaptive segmentation method of cotton in natural scene by combining improved Otsu with ELM algorithm. Trans. Chin. Soc. Agric. Eng. 34 (14), 173–180 (2018) Dhingra, G., Kumar, V., Joshi, H.D.: A novel computer vision based neutrosophic approach for leaf disease identification and classification. Meas. 135 , 782–794 (2019) Bao, G., Cai, S., Qi, L., Xun, Y., Zhang, L., Yang, Q.: Multi-template matching algorithm for cucumber recognition in natural environment. Comput. Electron. Agric. 127 , 754–762 (2016) Liu, C., Cao, Y., Luo, Y., Chen, G.L., Vokkarane, V., Ma, Y.S.: DeepFood: deep learning-based food image recognition for computer-aided dietary assessment. Lect. Notes Comput. Sci. 2016, 37–48 (2016) Sun, J., Tan, W., Mao, H.: Recognition of multiple plant leaf diseases based on improved convolutional neural network. Trans. Chin. Soc. Agric. Eng. 33 (19), 209–215 (2017) Gidaris, S., Komodakis, N.: Object detection via a multi-region and semantic segmentation-aware CNN model. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1134–1142 (2015) Sun, J., Yang, Y., He, X.F., Wu, X.H.: Northern maize leaf blight detection under complex field environment based on deep learning. IEEE Access 8 , 33679–33688 (2020) He, D., Xie, C.: Semantic image segmentation algorithm in a deep learning computer network. Multimed. Syst. (2020) Olimov, B., Sanjar, K., Din, S. et al.: FU-Net: fast biomedical image segmentation model based on bottleneck convolution layers. Multimed. Syst. 27 , 637–650 (2021) Zhao, D., Wu, R., Liu, X.: Apple positioning based on YOLO deep convolutional neural network for picking robot in complex background. Trans. Chin. Soc. Agric. Eng. 35 (3), 172–181 (2019) Fu, L., Feng, Y., Elkamil, T.: Image recognition method of multi-cluster kiwifruit in field based on convolutional neural networks. Trans. Chin. Soc. Agric. Eng. 34 (2), 205–211 (2018) Yang, Y., Kailiang, Z., Li, Y.: Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN. Comput. Electron. Agric. 163 , 104846 (2019) Barth, R., Ijsselmuiden, J., Hemming, J., Henten, E.J.V.: Data synthesis methods for semantic segmentation in agriculture: a capsicum annuum dataset. Comput. Electron. Agric. 144 , 284–296 (2018) Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39 (4), 640–651 (2014) He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, Venice, Italy, pp. 2980–2988 (2017) Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39 (6), 1137–1149 (2017) Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. 40 (4), 834–848 (2016) Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Honolulu, HI, USA (2017) He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Las Vegas, NV, USA, pp. 770–778 (2016) Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Honolulu, HI, USA (2017) Bahdanau, D., Chorowski, J., Serdyuk, D.: End-to-end attention-based large vocabulary speech recognition. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, Shanghai, China, pp. 4945–4949 (2016) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N.: Attention is all you need. In: Advances in Neural Information Processing Systems (NIPS), MIT Press, Long Beach, CA, USA, pp. 5998–6008 (2017) Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations (ICLR), San Diego, CA, USA (2015) Michał, D., Tim, R., Johannes, W.: Frustratingly short attention spans in neural language modeling. In: 5th International Conference on Learning Representations (ICLR), Toulon, France (2017) Badrinarayanan, V., Kendall, A., Cipolla, R.: SegNet: A deep convolutional encoder-decoder architecture for scene segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39 (12), 2481–2495 (2017) Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T.: Mobilenets: efficient convolutional neural networks for mobile vision applications. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: Bisenet: bilateral segmentation network for real-time semantic segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), Springer, Munich, Germany, pp. 334–349 (2018) Barth, R., Ijsselmuiden, J., Hemming, J., Van Henten, E. J.: Synthetic bootstrapping of convolutional neural networks for semantic plant part segmentation. Comput. Electron. Agric. 161 , 291–304 (2017) Paszke, A., Chaurasia, A., Kim, S.: ENet: A deep neural network architecture for real-time semantic segmentation. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (2016), IEEE, Las Vegas, NV, USA (2016) Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z.: Dual attention network for scene segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Long Beach, CA, USA, pp. 3146–3154 (2019) Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Major revision 27 Jul, 2022 Reviews received at journal 09 May, 2022 Reviewers agreed at journal 05 May, 2022 Reviewers invited by journal 05 May, 2022 Editor assigned by journal 09 Apr, 2022 Submission checks completed at journal 24 Mar, 2022 First submitted to journal 24 Mar, 2022 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-1484696","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":93285174,"identity":"06f9a2e9-e7f1-4372-9d99-92f7b18e04e3","order_by":0,"name":"Weidong Zhu","email":"","orcid":"","institution":"of Jiangsu University","correspondingAuthor":false,"prefix":"","firstName":"Weidong","middleName":"","lastName":"Zhu","suffix":""},{"id":93285175,"identity":"17510382-7bb5-4d74-831d-a371d0a3db41","order_by":1,"name":"Jun Sun","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA10lEQVRIiWNgGAWjYHACxgcJBhLMDAzMBxgSiNTCbPChwgaohS2BaC1skjPOpAFpHgPi1MvPyD0gzdt2mN3gds/nDw932DHwt3fjt8zgRl6CMVALs8Gds9skEs8kM0icObsBvxaJHINksJYbudsYEtuYgSK5+LXIz8gxOAzRkvP4Q2JbPWEtDDdyDBuB3gdpYZBIbDtMWIvBmTfGDKBAlryRZgbUcpyHoF/k23PMfwCjMpnvRvLjjz/bquX423sJOAwKkmEMHqKUg4Ad0SpHwSgYBaNg5AEAyWRJGW8EaBUAAAAASUVORK5CYII=","orcid":"","institution":"of Jiangsu University","correspondingAuthor":true,"prefix":"","firstName":"Jun","middleName":"","lastName":"Sun","suffix":""},{"id":93285176,"identity":"f3046dad-997a-4d24-906e-16076c5d47c6","order_by":2,"name":"Simin Wang","email":"","orcid":"","institution":"of Jiangsu University","correspondingAuthor":false,"prefix":"","firstName":"Simin","middleName":"","lastName":"Wang","suffix":""},{"id":93285177,"identity":"1f685cd6-9aea-4318-a98e-967d75c54e4d","order_by":3,"name":"Kaifeng Yang","email":"","orcid":"","institution":"of Jiangsu University","correspondingAuthor":false,"prefix":"","firstName":"Kaifeng","middleName":"","lastName":"Yang","suffix":""},{"id":93285179,"identity":"0e755411-b32a-4b98-994a-ef323f68bdbe","order_by":4,"name":"Jifeng Shen","email":"","orcid":"","institution":"of Jiangsu University","correspondingAuthor":false,"prefix":"","firstName":"Jifeng","middleName":"","lastName":"Shen","suffix":""},{"id":93285181,"identity":"b4f27a1f-11aa-4a17-8cec-ebd9eb2e2413","order_by":5,"name":"Xin Zhou","email":"","orcid":"","institution":"of Jiangsu University","correspondingAuthor":false,"prefix":"","firstName":"Xin","middleName":"","lastName":"Zhou","suffix":""}],"badges":[],"createdAt":"2022-03-24 09:14:23","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-1484696/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-1484696/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":19796288,"identity":"f1308875-5177-4f3e-8212-45fd20bb11af","added_by":"auto","created_at":"2022-03-30 20:21:56","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":270907,"visible":true,"origin":"","legend":"\u003cp\u003e\tExamples of sweet pepper images and their corresponding ground truth labels.\u003c/p\u003e\u003cp\u003e\u003cbr\u003e\u003c/p\u003e","description":"","filename":"fig1.png","url":"https://assets-eu.researchsquare.com/files/rs-1484696/v1/c99e112a9f6c2499e00bb0cc.png"},{"id":19796285,"identity":"c0d9c50f-e4a3-4b68-a777-eed55ce8bce5","added_by":"auto","created_at":"2022-03-30 20:21:56","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":77375,"visible":true,"origin":"","legend":"\u003cp\u003eThe structure of full-resolution residual networks (FRRN)\u003c/p\u003e\u003cp\u003e\u003cbr\u003e\u003c/p\u003e","description":"","filename":"fig2.png","url":"https://assets-eu.researchsquare.com/files/rs-1484696/v1/be8a3a284e32535caa2119c5.png"},{"id":19796283,"identity":"c5ec07c3-fd7d-4b46-807a-962e668b6901","added_by":"auto","created_at":"2022-03-30 20:21:56","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":42274,"visible":true,"origin":"","legend":"\u003cp\u003eThe structure of full-resolution residual unit (FRRU)\u003c/p\u003e\u003cp\u003e\u003cbr\u003e\u003c/p\u003e","description":"","filename":"fig3.png","url":"https://assets-eu.researchsquare.com/files/rs-1484696/v1/83547f2901d86d6b66feefa3.png"},{"id":19796284,"identity":"771f64bc-0f48-4b66-ad78-8bd7c587d271","added_by":"auto","created_at":"2022-03-30 20:21:56","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":54388,"visible":true,"origin":"","legend":"\u003cp\u003e\tThe details of position attention module\u003c/p\u003e\u003cp\u003e\u003cbr\u003e\u003c/p\u003e","description":"","filename":"fig4.png","url":"https://assets-eu.researchsquare.com/files/rs-1484696/v1/18c618961b31c7cd38d3ada6.png"},{"id":19796576,"identity":"7d891069-1b8c-488c-adee-94f20959bdd1","added_by":"auto","created_at":"2022-03-30 20:26:56","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":92377,"visible":true,"origin":"","legend":"\u003cp\u003eOverview of FRRN+PAM architecture\u003c/p\u003e\u003cp\u003e\u003cbr\u003e\u003c/p\u003e","description":"","filename":"fig5.png","url":"https://assets-eu.researchsquare.com/files/rs-1484696/v1/0813c4d5310b687707cc95d8.png"},{"id":19796286,"identity":"ab003ba6-a0e5-4f74-9014-b081064224e4","added_by":"auto","created_at":"2022-03-30 20:21:56","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":497349,"visible":true,"origin":"","legend":"\u003cp\u003e\tQualitative comparison on the test set\u003c/p\u003e\u003cp\u003e\u003cbr\u003e\u003c/p\u003e","description":"","filename":"fig6.png","url":"https://assets-eu.researchsquare.com/files/rs-1484696/v1/03b06ef410affd2a48aacbe7.png"},{"id":19796289,"identity":"53505b01-4bd2-43af-b7be-b90601713fa9","added_by":"auto","created_at":"2022-03-30 20:21:56","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":301515,"visible":true,"origin":"","legend":"\u003cp\u003eVisualization results of position attention module on test set\u003c/p\u003e","description":"","filename":"fig7.png","url":"https://assets-eu.researchsquare.com/files/rs-1484696/v1/3609226d6aca5a95132a100c.png"},{"id":19796577,"identity":"85b0f08f-f78f-4fae-94f8-d5f2f131eb02","added_by":"auto","created_at":"2022-03-30 20:26:56","extension":"png","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":157347,"visible":true,"origin":"","legend":"\u003cp\u003eExample segmentation results of empirical test set without fine-tuning (left column) and results of empirical test set with fine-tuning (right column). Color images (top row), ground truth (middle row) and classification segmentation (bottom row) are shown\u003c/p\u003e","description":"","filename":"fig8.png","url":"https://assets-eu.researchsquare.com/files/rs-1484696/v1/af2c7b073c110a6c6e4c3202.png"},{"id":19796578,"identity":"f4045aa9-8de6-4c8d-971e-1b1f5486165f","added_by":"auto","created_at":"2022-03-30 20:27:00","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":971611,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-1484696/v1/a84fd260-cef4-4be8-a802-e53928a3c475.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Segmentation and recognition of filed sweet pepper based on improved self-attention convolutional neural networks","fulltext":[{"header":"1 Introduction","content":"\u003cp\u003eSweet pepper is one of the most favored vegetables in the world, which has rich nutritive value and edible value [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. According to data from internet, the annual production of sweet pepper is more than 10\u0026nbsp;million tons worldwide [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. Since the economic benefits of sweet peppers are substantial, it is essential to maximize production efficiency [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e] and product quality. And now, the automated harvesting of sweet peppers [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] is emerging as a key instrument to achieve high production efficiency in precision agriculture.\u003c/p\u003e \u003cp\u003eIn practical production, most of the sweet peppers are cultivated in greenhouses, and the high temperature and humidity in greenhouses make it difficult to work in them. Replacing human beings with machines for picking work has been the major trend in modern precision agriculture [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. With the fast development of robot technology, the research of greenhouse fruit and vegetable picking robot [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e] has become a hot spot. In recent decades, picking robot technology has developed rapidly, and many countries have made a profound study on this technology and made some achievements [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. However, the problems of weak adaptability, slow picking speed, and low recognition accuracy in the greenhouse environment still need to be solved. In particular, image recognition technology is essential for robot to identify targets quickly and accurately.\u003c/p\u003e \u003cp\u003eAt present, researchers have made numerous researches on image recognition of fruits and vegetables. The main methods are based on the combination of machine learning algorithms and machine vision techniques. Wang et al. [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e] used an improved Otsu segmentation algorithm to locate the cotton peach region and sampled RGB values of the pixels in different regions. Thus, the problem of image segmentation was transformed into the problem of pixel classification. As a result, they trained the ELM classification model and realized accurate segmentation of the cotton peach images. In the work of Dhingra et al. [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e], the recognition of leaf diseases was studied. An extended segmentation technique based on neutron logic was used to extract the region of interest (ROI) of leaves. Then several features, including texture, color, and histogram, were employed to detect diseased leaves. Nine different classifiers were constructed, and the best classification accuracy of 98.4% was eventually obtained. To recognize and locate cucumber accurately, Bao et al. [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e] proposed a multi-template matching method and established a multi-template matching library containing 65 cucumber images. The robot vision system can utilize the matching library to calculate the normalized correlation coefficient matrix of the target image one by one, and judge whether there is a target cucumber in the image. The aforementioned researches prove that image recognition technology has been widely used in the agricultural field to identify and locate vegetables accurately. However, most of the above methods adopt traditional machine learning, which mainly depends on multiple feature selection and application under specific background. It is difficult to play a role in the environment of the greenhouse.\u003c/p\u003e \u003cp\u003eWith the development of high-performance computing systems and the sharing of massive datasets, deep learning has gradually become one of the most important technologies in the field of intelligent agriculture. The basic deep learning tool used in this work is convolutional neural networks (CNNs), which have a strong feature extraction capability and are suitable for processing a large amount of input images in parallel. Therefore, CNN is one of the most powerful technologies in the field of image recognition, which is widely used in image classification [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e], object detection [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e] and image segmentation [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. In recent years, CNN technology has been introduced into the field of fruit and vegetable image recognition. Zhao et al. [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e] proposed an apple location method based on YOLOv3, which realized apple detection in a complex environment. This work provided a theoretical basis for the research and development of apple harvesting robot. Fu et al. [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e] studied the problem of fast identification of multiple clusters of Kiwifruit in the field. They trained LeNet as a classification model to identify occluded fruits, overlapping fruits, adjacent fruits, and independent fruits, and finally achieved an average recognition rate of more than 85%. In the work presented by Yang et al. [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e] based on the Mask-RCNN, Resnet-50 was adopted as a backbone network. On this basis, the feature pyramid network (FPN) architecture was integrated to construct the feature extraction network. Their method accurately located strawberry picking points, which was helpful to improve the working performance of strawberry picking robot. The aforementioned literatures demonstrate that it is feasible to recognize fruit and vegetable by convolutional neural networks instead of traditional machine learning.\u003c/p\u003e \u003cp\u003eIn this paper, an improved CNN architecture based on the FRRN was designed and verified, which was applied to segment sweet pepper images captured at night greenhouse. Specifically, PAM was appended on top of the FRRN to improve the feature representation, which contributes to more precise segmentation results. Furthermore, the influence of image resolution on segmentation results was explored. This study aimed to provide a theoretical basis for the development of image recognition performance of picking robot in precision agriculture.\u003c/p\u003e \u003cp\u003eThe rest of this paper is organized as follows. The dataset and experiment arrangements were introduced in Section \u003cspan refid=\"Sec2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. Section \u003cspan refid=\"Sec5\" class=\"InternalRef\"\u003e3\u003c/span\u003e describes the model architecture of the FRRN combined with the PAM in detail. The training details of the model are listed in Section \u003cspan refid=\"Sec10\" class=\"InternalRef\"\u003e4\u003c/span\u003e. The experimental results are analyzed and discussed in Section \u003cspan refid=\"Sec13\" class=\"InternalRef\"\u003e5\u003c/span\u003e. The conclusions are drawn in Section \u003cspan refid=\"Sec18\" class=\"InternalRef\"\u003e6\u003c/span\u003e.\u003c/p\u003e"},{"header":"2 Materials And Experimental Scheme","content":"\u003cdiv class=\"Section2\" id=\"Sec3\"\u003e\n \u003ch2\u003e2.1 Dataset description\u003c/h2\u003e\n \u003cp\u003eThe sweet pepper dataset, used in this paper, was obtained from 4TU. 10 500 greenhouse sweet pepper images and their corresponding ground truth labels can be obtained at ResearchData (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://data.4tu.nl\u003c/span\u003e\u003c/span\u003e). These synthetic images were rendered through Blender based on 21 empirically measured plant properties. They highly simulated empirical images, which obtained by sweet pepper harvest robot from different angles at night greenhouse [\u003cspan class=\"CitationRef\"\u003e21\u003c/span\u003e]. Besides, the sweet pepper dataset also contains 50 empirical images of the crop obtained from a high-tech commercial greenhouse. These empirical images were used in experiment III. The sweet pepper dataset was publicly released with the intention of comparing the performance of agricultural computer vision methods. There are 8 classes of segmentation objects in the picture, including background, leaves, peppers, peduncles, stems, shoots, wires, and cuts for picking pepper. In the labeled images, 8 classes were annotated on a per-pixel level, corresponding to the color of black, blue, yellow, red, pink, green, white, and light blue. In Fig. \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e, examples of synthetic images are shown.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"Section2\" id=\"Sec4\"\u003e\n \u003ch2\u003e2.2 Overview of experiment arrangement\u003c/h2\u003e\n \u003cp\u003eTo obtain the optimal segmentation model of the sweet pepper image, three main experiments were designed and carried out in this study. Overview of performed main experiments I through III are shown in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e\n \u003cp\u003eExperiments I: A suitable CNN architecture was picked out for sweet pepper segmentation. Since the same network structure may perform differently on different datasets, in a specific task, it is necessary to select the appropriate network according to the dataset. Existing semantic segmentation network models are mostly applied to segmentation of street scenes, and rarely used in agricultural scene. In this experiment, several model structures were applied on the same sweet pepper dataset for training and testing, and the differences in segmentation performance of different models were analyzed. Each network structure is verified with the same number of images, 400 images were used as training set, and 100 images were used as validation set and test set respectively. Since different network configurations has different requirements for the size of the input images, the size of the input images was set to 384\u0026times;384 to balance the training time and model performance.\u003c/p\u003e\n \u003cp\u003eExperiments II: The effect of image resolution on segmentation results was investigated. After experiment I, a suitable model architecture for the sweet pepper segmentation task was obtained. Then the impact of the input image size on the segmentation performance of the model was further explored. Images with different sizes were input into the model for training. Unlike experiment I, experiment II was performed on a larger dataset. In this experiment, 10 500 sweet pepper images were divided into training set, validation set and test set according to 3:1:1, specifically, 6 300 images in training set, 2 100 images in validation set and test set respectively.\u003c/p\u003e\n \u003cp\u003eExperiments III: The self-attention mechanism was introduced into the network to improve the segmentation performance. The self-attention mechanism can effectively establish dependencies between remote features, and make the model focus on extracting some key features. In our work, the PAM was used to capture global dependencies in the spatial dimensions. It was added on top of the FRR. Afterwards, the enhanced model was trained and tested on our dataset. The size of the input image and the distribution of the dataset in experiment III were the same as that in experiment II. Finally, the model with the best segmentation performance was further evaluated on empirical (real) images.\u003c/p\u003e\n\u003ctable border=\"1\" id=\"Tab1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eOverview of performed main experiments I through III\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eExperiment\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMethods/Architecture\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eInput size/pixel\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eTraining set\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eValidation set\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eTest set\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eI-A\u0026thinsp;~\u0026thinsp;I-E\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eSeveral CNN-based models\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e384\u0026times;384\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e400\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e100\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e100\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eII-A\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eFRRN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e384\u0026times;384\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6 300\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 100\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 100\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eII-B\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eFRRN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e512\u0026times;512\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6 300\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 100\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 100\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eIII-A\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eFRRN\u0026thinsp;+\u0026thinsp;PAM\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e384\u0026times;384\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6 300\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 100\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 100\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eIII-B\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eFRRN\u0026thinsp;+\u0026thinsp;PAM\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e512\u0026times;512\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6 300\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 100\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 100\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e"},{"header":"3 Segmentation And Recognition Model","content":"\u003cdiv class=\"Section2\" id=\"Sec6\"\u003e\n \u003ch2\u003e3.1 Semantic segmentation network architecture\u003c/h2\u003e\n \u003cp\u003eSemantic segmentation method was used to segment and recognize sweet pepper in this paper. The goal of semantic segmentation is to segment and parse an image into different image regions associated with semantic categories (e.g. leaf, sweet pepper, shoot). Semantic segmentation models based on convolutional neural network have attracted extensive attention since FCNs [\u003cspan class=\"CitationRef\"\u003e22\u003c/span\u003e] was proposed, and have made great progress. The existing depth-based semantic segmentation methods are mainly divided into two kinds: one is based on regional classification represented by Mask R-CNN [\u003cspan class=\"CitationRef\"\u003e23\u003c/span\u003e], the other is based on pixel classification represented by FCNs. The former method is based on target detection. Firstly, the image is divided into different image patches, then each pixel in the image block is semantically classified. Finally, semantics segmentation is realized. This kind of method first appeared in R-CNN and gradually evolved into improved algorithms such as Faster R-CNN [\u003cspan class=\"CitationRef\"\u003e24\u003c/span\u003e] and Mask R-CNN. The advantage of this method is that it can accomplish both tasks of target detection and semantic segmentation at the same time. However, due to the lack of global information of the image, this method is not effective for small-scale objects and small areas. Therefore, this method was not considered in our experiments. The semantic segmentation method based on pixel classification is favored by many researchers and has been widely used. In particular, dilation convolution, as well as atrous spatial pyramid pooling was adopted in the Deeplab series [\u003cspan class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e26\u003c/span\u003e] to embed contextual features of different scales. Moreover, the encoder-decoder structures can effectively fuse mid-level and high-level semantic features. In experiment I, since the methods based on pixel classification performed better in fine segmentation task, we mainly adopted them.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"Section2\" id=\"Sec7\"\u003e\n \u003ch2\u003e3.2 Full-Resolution Residual Networks\u003c/h2\u003e\n \u003cp\u003eThis section briefly introduces the FRRN, which is the backbone of the improved model in this paper. FRRN is a novel network structure similar to ResNet [\u003cspan class=\"CitationRef\"\u003e27\u003c/span\u003e]. It does not rely on the pre-trained networks and can start training from scratch. In addition, since FRRN combines two distinct processing streams, it has strong recognition performance and precise localization capabilities. As shown in Fig. \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e, one stream is pooling stream: semantic information is captured through a sequence of convolution and pooling operations for identification. The other stream is a residual stream: feature maps at the full image resolution are carried to achieve precise boundary adherence. Meanwhile, FRRN consists of a series of full resolution residual units (FRRUs). The structure of FRRU is given in Fig. \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e. Each unit includes two inputs and outputs, because they are applied to both streams at the same time. If \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({z}_{n-1}\\)\u003c/span\u003e\u003c/span\u003e is the residual input of the \u003cem\u003en\u003c/em\u003eth FRRU and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({y}_{n-1}\\)\u003c/span\u003e\u003c/span\u003e is pooling input. Then the outputs are computed as:\u0026nbsp;\u003c/p\u003e\n \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({z}_{n}={z}_{n-1}+H\\left({y}_{n-1},{z}_{n-1};{W}_{n}\\right)\\)\u003c/span\u003e\u003c/span\u003e(1)\u003c/p\u003e\n \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({y}_{n}=G\\left({y}_{n-1},{z}_{n-1};{W}_{n}\\right)\\)\u003c/span\u003e\u003c/span\u003e(2)\u003c/p\u003ewhere \u003cem\u003eH(\u003c/em\u003e\u003cstrong\u003e\u0026middot;\u003c/strong\u003e\u003cem\u003e)\u003c/em\u003e and \u003cem\u003eG(\u003c/em\u003e\u003cstrong\u003e\u0026middot;\u003c/strong\u003e\u003cem\u003e)\u003c/em\u003e represent residual calculation and pooling operation respectively. \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({W}_{n}\\)\u003c/span\u003e\u003c/span\u003eis weight matrix.\n\u003c/div\u003e\n\u003cdiv class=\"Section2\" id=\"Sec8\"\u003e\n \u003ch2\u003e3.3 Position attention module\u003c/h2\u003e\n \u003cp\u003eThe attention mechanism in deep learning is generated by imitating human selective visual attention mechanism. It has been widely used in image processing [\u003cspan class=\"CitationRef\"\u003e28\u003c/span\u003e], speech recognition [\u003cspan class=\"CitationRef\"\u003e29\u003c/span\u003e] and natural language [\u003cspan class=\"CitationRef\"\u003e30\u003c/span\u003e] in recent years. The essence of attention mechanism lies in the automatic and efficient allocation of attention resources, which contributes to the acquisition of the most critical information to solve the current task. There are many variants of the attention model, including the soft attention model, the global attention model [\u003cspan class=\"CitationRef\"\u003e31\u003c/span\u003e], and the key-value attention model [\u003cspan class=\"CitationRef\"\u003e32\u003c/span\u003e]. PAM can adaptively aggregate long-range context information to improve the feature representation of semantic segmentation. The implementation details of PAM are shown in Fig. \u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e. \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eA\u003c/span\u003e represents a local feature, where \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eA\u003c/span\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\in {R}^{C\\times H\\times W}\\)\u003c/span\u003e\u003c/span\u003e. It is first fed into a convolution layer to generate two new feature maps \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eB\u003c/span\u003e and \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eC\u003c/span\u003e, respectively, where \u0026nbsp;{B,C}\u0026isin;R\u003csup\u003eC\u0026times;H\u0026times;W\u003c/sup\u003e. Then they are reshaped to\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({ R}^{C\\times N}\\)\u003c/span\u003e\u003c/span\u003e, where \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(N=H\\times W\\)\u003c/span\u003e\u003c/span\u003e is the number of pixels. After that, a matrix multiplication is performed between transposed \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eB\u003c/span\u003e and \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eC\u003c/span\u003e, and a softmax layer is applied to calculate the spatial attention map \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eM\u003c/span\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\in {R}^{N\\times N}\\)\u003c/span\u003e\u003c/span\u003e:\u003c/p\u003e\n \u003cp\u003e\u003cimg src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAANwAAABFCAYAAAAhM4gKAAAPEElEQVR4nO2dz2sc5R/H3/nyvWo7racSPOyTg6GVFp2koGkhgp2tBfFQmBShFw9lcxBUNLjBg6RaJtRLkHa9eakbUVGETborWHA2xcRFdkHppTMUKZ4mWe0/8HhIP9NnZmdmZ3dnZjfJ84KBZH48++yz83l+fJ7PjzHOOYdEIsmE/w27AhLJQUIKnESSIf8fdgUk2WHbNq5fv46dnR2srq4OVNb29jbu3bsXev2pp57CyZMnB/qMpKnX63jnnXfw8ccf47XXXhtOJbjkQFCpVLiiKLxYLHLHcQYur1wucwBc0zSuqioHwFVV5ZqmcQC8UCgkUOvkMU2TK4oytPpJgTsANJtNDoAbhpFYmcVi0S2vUqlwANyyLM4554VCgZdKpcQ+K2mazebQhE4K3AGAMcY1TUu0zHK57I6UhmFwRVHca6ZpusI3qtAIXalUMv3cMc7ltsB+ZnV1FZcuXYJpmpiZmUnlM/L5PA4fPjzwujBrpqam0G63cf/+/cw+UwrcPmdiYgIAIl+qtbU1rKyshF7P5XI4e/Ys5ubmAq+PjY2hVCrhypUrg1U2JrZt46effsKdO3fwzz//uHU8efJkT3XIojPqINPxVJIptHYrFotd7yXFBx4rPEzT5KZpcl3X3fOqqnY8Z5omB8CbzWYaX6GDYrHo1scwDG6apntOnNbGwbKszBU8UuD2MYZhcADcNM2u95J2EQHrGjofVFapVOJZ9dtip1AulzvqqOt6z2UyxnoW1EGQG9/7mJ9//hkAMDk52fVeccr53HPPxf6MO3fuQNO03ivXI4uLi2g0GgCAYrHYMb01TROffvppz+Wqqop2uw3bthOpZ1cyE+0DSLlc5rquc03TeKFQ6NDcWZbFC4UC1zTNM3I4juOedxyHO47DDcPgmqZ5VPulUsktO2hvjTEWa/RxHMcdOfy9PU1Lg67RnhZjLFWtJE396EhiH5HoZRaQBFLgUqDZbHJVVTljjJfLZXefSlTNW5bFNU3jlUqFM8Y4Y8y9RlMnxhh3HIdrmuYKL71wdE7cgPaDkHWXH6offNMyx3HcuiiK0rFOKxaLXNM0ty5pIa7bkl5vkcAluUcZhRS4hHEcx+31xZ6Y1kj+ezl/8qNblsWLxSJnjPFKpeK+4P77dF33vPxBZXPOQwXRD5VLQk5CROeSsk4RKZfLXNO0WAodGqkRsHYbFFL6ZCVw0pYyYT766CO02238+OOPOHr0qHv+/v37UFXVcy9df+mllwAA169fR61Ww+bmpudZ+vv3338HALzxxhuBdor1er0v9Tat9QDg/fffx/Hjx/Hw4UNsb2+j0WigVqvh3Xff7bncKP766y9Uq9Wu99m2Dcuy3P+np6cTrUfmZCLWBwRaC4nTQ8uy3KlglFUDQqZtIjRy+hl0hIOwPhIJm2omAY2q3epHIxC1Ty9YlsUNw4hcX2Y9wkktZYKQ9fzOzg7y+TwmJibAGAMANJvNUAv17e1tKIqC6enpUAt727bRbreh63rHtWq16n5Or7RaLfdv/wj89NNPu39//fXXoc/n83lPOXG4ePEiTNOEYRixn+l1dNva2sLCwgIOHTrU03NpIqeUCXL37l0AQKFQwPnz53Hs2DHkcrnIZ7a3t5HP58EYi5xibW1tAQBefvllz/m1tTUACBREKj+KX3/91f17amoq9L4wgb537x6q1Spu3boV+Tl+crlc17YZlPHxcRiG4Zme+6HfLCvkCJcgzz77LADg/PnzmJmZ8bxQy8vLHfeTsE1NTeG9994DsLsOC+KXX34B4B11AGBlZQWKouCtt97qeIYx5u5dhSGOTP7R9auvvnL/PnfuXODz09PTME0z8qUehGPHjkVeX15exuLiYuD5u3fvuuvjbsS9b1CkwCUITXnW19fdcyRUDx486Dh3+vRpAMDS0lLHs34B/e233wAAjx49cs/Nz8+jWq3ixo0bgaMF2VFGjXKiwfHx48c952/evAlgd6q5tLTU8ezc3Bzm5+fx8OHD0PIHJZfLuaN3tVr1bFCvrq5iYWEh8LkHDx5gYWHB015BkCKqm2AnRiYrxQME7YuRM6aiKB2+YaSMYIx1qPcVReGqqnrU36SMUVWVK4rCNU1z9+6iFDHdNnVFO0k8VmBQ2XispDAMI3RLgL5H2i4ujuN47CXFOoZ9P1KGdNuQpzbNCuktkAJi+IEgNb1t2/j7778xOTnpmYrRc/6139raGi5cuIByuYxXX301smyRVquFU6dOoVgs4pNPPgmtYxBx1p/1eh1nzpyB4zipTSlF/HWOquMXX3yBDz/8EDs7O6Hl2bYNxhgKhQJu3LiReH0DyUy0JX0jboz3it+KJUlKpVIsS5ZhoOt6160Mmo1kZdbFudwW2BPQxnQ/Wr2rV6/CsqxQZcwgfP/995GazWFSq9XwwgsvRN7z2WefgTGWnS8c0DnCiWY+dARtCopGrRDWAJLkofbt16wpjRALnHO3Ts1mM/NQBVHQuxk1cmW1/vQTOKUU/Y7CBE60tYNgaCtJFsdxXGfQfqc+aQQR4vyJ5Qt5NYwK3Xz0Ri6IkF+Y/BUTzW3oGOUoTZLkw+Rxvms6leX6Jy6apoWu30YyTJ6iKB6VsX86QuprUTCzcrGX9A/53yVtFzkqGIbBdV0PtUk1TZOrqjrU6W+HwJGzn7iWEwWO5r6k4UEfRqUSSRoYhsFLpdJIh+jrsKUkm70TJ06AMQbLstxzAPD222+DMeYxMUrTZSKfz8dy4xDhcmvxQPLBBx8Muwpd6dgWIJu906dPu6ZB7XYbwK4pjWVZ+PLLL7GxseE+88orr6RWwdu3b4PvjsSxD4lkZPEPeRQagHOv8oRzr3pZvDaKC+co4FP4yEMeWRyc+9ZwZLNHGhz/Og54Yu3gLyiMoHgXacfAkEhGFc+UcnNzEwBw9uxZ+KlWqygWi8jlch6rBb/Tokir1UK1WsX4+Lh7zrZtVKvVDjeTMPL5PMbGxno6JJJRxaM0+eOPPwA8UYKcOHHCvaYoihvXQnTaC/OTAnYdAE3T9MRFPHToUMe5KG7fvh3rPolkL+AZ4b755hsAT2z2xFHo2rVrrkW4GHTG74FMLC4u4s0338T6+rr73PLycsc5yd7Htm3Mz8+H5h7Y79TrdUxNTbne95HQ3FIMGEOWCLQnJ1qb+61MwixMgsyJms0mZ4zFCo026pDvmPj9KNbkQVqjJm3B0mw23TYMOkb13YlrwQLOOyPbRlkiIED7EtbQQHCc+lEydO0Xyokmfn+KmOx3LN2vpGGjaRgGV1WVm6bp2kSWSiVumiZnjGUWXasf4thopuYPF+RxSz/QKBm69ku5XHa9kMXRzDCMke2FkyYNLwRd191tJr8RsmEYI99Zd0v0mJrAGYbR4fgYdG4YiGZpcQ9/z1ooFHilUnFDKRC6rh+I6WQWzpuFQmFkHVyjEPey/aQWYiEoK2Y+n0cul8PS0hK+/fbbzBL4hdWP4jn6Ix0TFEIAAAzD8JgOTUxMoFarYWtrC5cuXYJlWcjlcjhy5AgajUbqIeCGTZxEj6RMCePw4cOYnZ3FxYsXA9t/YmICuq53hIdIi0wSPaYl5QhQqJAlt6qqQ1/jWJblrsGipoCFQqFjhLMsy+3ByFiAjGZHYQRPm14SPYqJOGhtZpqmx6hCUZTAzELIcL2fVaLHVAQuLCsmOVOOyhqO1giImBoFhcKuVCqextR13Y20tV9dX0R6SfEkCpZfQEXzQP+UnbTmWbwrWSZ6TCWmyeeffw7GWEdg0aNHj2JmZmZk9uCuXLnixjx8/fXXA+M3BsW72NjY8Hy3y5cvo9Fo4IcffsDs7Gx6FR4Rekn0KMbjfP7552N/xsbGBhhjqb8rmSd67LtbCKBYLLpx/oY9ZYwLpZdCDz0ZY6yjd6cysjDk3iuJHjn3jh7+d4LaLOiaXxmVBsNI9JiowDWbzT3nOcC5d9O/25qB1nT+tRrN99NkryV6jMqsSu2IgGlc1LUkGUaiRxmX8jH00gUt4EWoU/F3LI7jpDqq78VEj/5UU2QtIs4ogtpMbONevbdFS5VuDCPRoxS4x/QztcwS6vX9gs4YCx1t6Ecn65ewKRN1NkFuVIHTopgCJypMCoUCN02TVyoVT+eWdCclCnkU/ulk0mEZpMDFIMhyZBTYq4kew5yUo6aagxJX4AZJ9BhH2x4mcDLy8mNs28bNmzdRKBRGzup9LyZ6BOCJhSNqNEXNY7vdTjQq9OTkJEzThGmasZ/pNSbP5uYmzpw5g3///bfX6smEjISu6zhy5EhgWqYgyAol0JogYfZiosdWq+XGwumm3g9KFdVv+9LWU5o8evQImqZF/gZhiR7lCIddP71Go4Hvvvsu9r7Pn3/+CaB7BpsoyPQpyvwJ2JuJHsUsN/6oAKK5H2Ms8MVNon2j6DfRY71ex/j4eOxsOx2JHnuavO5Dsk6qzjn37H/F+Wxa4IuWGqTCF9XZdI4UKY7jdDzr/yzaKhDXgaSgCVvLUr2j1jBiIGG/TyQpp4a9XyvWUVSa0JZIkOkatVe3dT6V7VfGHGiBcxwnlouJf2OYXph+hdSyLPcF95cdxl5K9CiazIl1FjfBgzbsiUHbNy6DJHrs1lGEJXo80AJHxtRRKmF6Kf3Gy/4fJCjfgv8I+gHjChzVJSqpB8X694889Jz/e4pRtLuVLdLNeFlMPuI/4oxoQe2bJv6EKVHvA3V8UWRuvLwX6OYoyPkTUyi/wJFwJUEvApc0o5roMcn2TRoyjYsiylfwQCpNWq2Wq6i4cOFCaLi9Z555xk0sL7K+vg5N07KuduKMaqLHUW7fWq3WNdJ4VKLHAylw165dc1XWcRG1TbVaDS+++KLner1e7xovM42XcxBou0DUGsZlbm4OjDFcvXo16WoFtu8oYNs2LMvyhI/0s7a2hkajgZWVlcDrqXl872fGxsZQLpfd//vdKLdtG+fOncPExARu3bqVqduSP0F9P+r3VquFU6dOdXjDD0pS7Zs05MntOE7gb9VqtTA7O4u5ubnwbYN0Zrr7G9K2DRIsCAFKlb2YsjmNRI9JtG8aRMVY6SlMnkQyCPs90aMYsi9IydZLokc5pZRIukABsS5fvhxqtxoXKXASSYYcSC2lRDIspMBJJBkiBU4iyRApcBJJhkiBk0gy5D+iShtZOVPVPwAAAABJRU5ErkJggg==\"\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e \u003c/span\u003e\u003c/span\u003e(3)\u003c/p\u003e\u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003eM\u003csub\u003eji\u003c/sub\u003e\u003c/span\u003e\u003c/span\u003emeasures \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({i}^{th}\\)\u003c/span\u003e\u003c/span\u003eposition\u0026rsquo;s impact on \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({j}^{th}\\)\u003c/span\u003e\u003c/span\u003eposition. The more similar feature representations of two positions contribute to a greater correlation between them.\u003c/p\u003e\u003cp\u003eMeanwhile, feature \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eA\u003c/span\u003e is also fed into a convolution layer to generate a new feature map \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eD\u003c/span\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\in {R}^{C\\times H\\times W}\\)\u003c/span\u003e\u003c/span\u003eand is reshaped to\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({ R}^{C\\times N}\\)\u003c/span\u003e\u003c/span\u003e. Then a matrix multiplication is performed between \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eD\u003c/span\u003e and \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eM\u003c/span\u003e, and the result is reshaped to\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({ R}^{C\\times H\\times W}\\)\u003c/span\u003e\u003c/span\u003e. Finally, it is multiplied by a scale parameter\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e, and an element-wise sum operation is performed between it and features \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eA\u003c/span\u003e to obtain the final output \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eQ\u003c/span\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\in {R}^{C\\times H\\times W}\\)\u003c/span\u003e\u003c/span\u003eas follows:\u0026nbsp;\u003cp\u003e\u003cimg src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAOIAAABMCAYAAABwBmtJAAANF0lEQVR4nO2d3YsbVR/Hv/vgrbqz8WrpVUahxZYt7uwW7CpUsJOqeONLFltEaEGyhV4odjFRvHDbbXLhhdDdtSjsjXQWLZXSjTsVKjSxsLupJKD0QicXIl5lnNV/YJ6LPuc8M8kkmZczzWT394FAMpM5LzPzPa+/3zkjtm3bIAhioPxn0AkgCAJ4ZNAJ2I0UCgXcu3cPo6Oj0DSNH5+dncXOzg4mJydx4cKFAaaQSBpUI8bAiRMnsLW1hbW1NZTLZX782LFj0HUdR48eHWDqiCRCQoyB8fFxTE9PQ1EU3Lx5kx9/+umnoSgKXnrppQGmjkgiJMQY2NrawuTkJI4fPw5N02CaJgDg119/xdTU1IBTRyQREmIM3LlzB4cOHcLp06dhWRZ++OEHAMCPP/6I559/fsCpI5IICTEGbt26henpaaTTaSiKgu+++w4AUKvVMD09PeDUEUmEhCgY1gxNp9MAgDNnzmBtbQ2NRgN///03P04QTkiIgtnc3ISiKPz366+/DgBYXFzE8ePHB5UsIuGQEAVz8+ZNPPPMM/x3KpVCNpvF2tqa6zhBOCEhCkTTNCwvL+PSpUtoNBr8+Ntvvw0AePbZZweVNCLhjJCtqTiazSb++usvAMCBAweQSqX4uWq1ipmZmUEljUg4JESCSADUNCWIBEBCJIgEQEIkiARAQgxAtVrFyMhIrJ9SqTTobBIDgIQYgJmZGWSzWf5bkiTU63XYth3602q1kMvlBpgrIgmQEANy+fJlyLIMALAsC2fOnOFmbWFIpVJYWlqCqqqikkgMISTEgKRSKVy7do3/rtVq+OyzzyKH+84770QOgxheSIghmJiYwMrKCv998eJFlyd+GMgrY29DQgzJu+++62pOnjp1Cs1mM3R45JWxtyHLmgiYpomnnnoKlmUBAFRVxcbGxoBTRQwjVCNGIJVK4caNG/y3rus0/UCEgoQYkZmZGRSLRf57fn7e5XkRB81mE3Nzc5idnY01nqRQrVYxNTUVuR+eaGxCCKqq2gBsALYsy3ar1YolnvX1dVuSJDufz8cWRxKpVCq2JEl2LpcbdFJigYQoCMMwbEmSuBiz2azwOOr1ug3ALhaLwsMeBur1+q4VIwlRIOvr61yIAOyrV68KDV+WZVtVVaFhDhtXr161Adjr6+uDTopQSIiCyefzXIiSJNmGYQgJl72AlUpFSHjDjKIotizLg06GUAIJ0TAMO5fL2bIsu/pDxWJxT/VX+qEoCr8/iqIICVOWZV8vnzNu50eWZTubzYaupYvFYkeYXk1k1nx2fkTX4kktlCqViivfQfD9b+eDYMJrtVpclKJeuN0A68uw+5XP5yOH5zccr5eh1WrZKysrkYXRLnIvIToHreIauDIMwwaQuL6is4KKRYjO5lb7y+AUqOg+0TDDSm32iVJ6s3vsJwynENsF53yOYQZ82kXWLoT2QgCAvbKyEjgeP8iybEuSFPp60YNeXi2GQOnp9wfnzfUq3Zznk1ZCDZpcLufqL4atGZgA/FzvrPnaC832ZxkUSZLsbDbbVeiqqtqSJLkEW6/XA8fjB5aOsH1wkUJstVq2JEm2oiiuvAeh74T+wsIC//7BBx+4ViYDgD///JN/j2Jr+TBhE+JjY2MYGRnB3NwcTNOEaZoYGxvDk08+KSSeTz/9lC82bFkWTp48GSqc33//HQA67r0XTmOCQ4cOuc45V5EzDCNQGprNJizL6ro2a7lchq7rWFpagq7rAB74a05MTASKxy8sHWzVvEHy8ccfw7IsfPnll67jQdzjegqx0Wjwmwr8f9VqJ3/88YfvyPySyWQCe7b7pdFoQFEUbG9vo1arodVqYXt7G99++y2++uorWJaF1dVVIflIpVKuh+O8l0EwDMO1engvbt26xb8fOHAgVHxebG1tAQAOHjzI/THZMQA4d+4cZFnGY489xo89DI+Su3fvxh5HLxqNBpaXl1EsFjsKnfv37/sOp6cQnS+OqqqeJfLPP//Mv4vyINjY2Ajs6e4H0zRx7NgxjI2NYWNjA+l0GqlUCm+88QauX7+OK1euoFgsCl1/dGJiAvl8HgAiOf/6qQ1N0+Q1nVdtVK1WQ8d/584dAMCRI0d4i4EZu2uaBsMwsLq6ip9++olf88ILL4SOrx9JWax5fn4esizj9OnTkcLpKUSnyCYnJz3/4yyB42qGiILVeAsLCx0vNit0zp8/LzTOZrOJ5eVlSJKEr7/+WmjY7WxubvLvXrWRsxvBajW/bG9vQ5ZlzwLho48+gqqqmJmZwb179/jxpIglLlhzfHV11VdB2YueQtzZ2eHf2/sbwINqmZWKAPDiiy92DUvTNGQymTBpFMaVK1cAoKuxtKgmqZNsNgvLsnDjxo3ID6sfv/zyC//uVRuxWg2A54Y4jUYDmUymw2jdNE3UajV+jTPsTCYDwzCwtLQEwN2K8tOyyGQy0DSt77GwlEqlrl2Z+fl5z3N+Ww7nzp0DADz33HP8Wj/dD6/8+fa+2LdvX8exL774gn9XFKVn05TtEeiHuPqIhmF4Ng9v377NS3SRFAoF1Go15PP5h7Lc/u3bt/n3gwcPus6Zpul6+G+99VbH9ffv34eu6x3PmtW0Xpus6rqOfD6PdDrteoH99GnZGIQzvmazCV3XXX3NKJw/f75rV6ZYLHqe8/OsSqUSDMOAYRiua/t1P7zyDACPhM1g+4N9//33e/7/ww8/9B12HM613Uq5arUKXdeFL95ULpdx8eJFKIqCCxcuRA7PzwicszQ+cuSI6xxrlgMPammvl216ehqVSqWj5mY1LWvuOkUuSRLee+89AO6BEz9b0O3btw+VSsU1qPT44493HPNikIM0pmni0qVLyOVygcdFvPIM9KkRnf3Cf//913Xu7NmzrgfbrbnHmjvz8/OBEiya8fHxjmOmaWJhYcH3iKRfTNPEqVOnIEkS1tbWIocnyzJqtVrP/zibk+19uUajwe+/oii4fPlyx/Wzs7OYm5tz9SMZ33zzDYD/D8Y5a6vFxUUel7NGPnr0aM/0FgoFnDx5Et9//z2/vlQqdRzrxyD6oWy64pVXXun5v/bCwivPnF6TjE7XHlVVuVmbc6JaUZSeE83O/w8aZo3RarVswzBsVVXter1uq6rK85HP5yMbarNJXVEeAn4m9J1WM8wFyzAMl8UHe4ZeMM+R9jQ7PUrYtczEzGkU0G5V08+ixsulq16v27Is+zLlG9SEvtNiqv1eOU0+4WFQ0cuNra862IvqvMn4n6WI34wUi8VE2KIy51L2srKHyKxRFEWJbEjMXvyo9qVeYXZLm9Oapv0jy7Kdy+X6FgpMSE6hMsG1C9wLr7j7WQJ5vcx+CzBFUR66iVt7YdMef7utaZD8+a6m2tUexHRJVdU9Yf7GHpToQieI0XdYVlZWHmphye6Vs0Zj+ewnYBFG32FrxCh45Znhe9SUbUHNWFxcBPBgsKPftISu654jbrsJ0zTx6quvQpKkDlOnqExMTECWZSH9zW5cv34dU1NTsYXfzt27dyHLsmuwQ9f1rnOVTphFj9fIr19s2xY+Z9wPrzw7E+Sb9uUg2KeX1wUrBeIy/k0KrPkelwdK3D54LO31ev2heL+rqtrR1GUtJ+a21Y1hdQz2yjMj8AjK+vo6b6LKstz3oRWLxUht+WGA9eHibn7HuVSGJEk8/Ifh5A2PAZ1sNsu9GLoV3N0GlYYBrzzzc3FH3qsU2A2wfk2/0WORccXRtzEM46F5vHdrJbVaLbtSqXS9j8O8eFS/lmGsQmQvzm51GGYDWJIkCWl6symIXuyG5RSz2WzgpuWwL6fYL8+xCJFNeQzzjfMDm8sSVdCwfmY/2NpBw9bSyOfzfNohSMFVqVRsRVGGsjnqN8+xCJE1MUStYJZE2NydSDHAw+t9N1Gv1xO34FPc+M0zbUITgkajgcOHD0OWZWxubgrxqiiXy3j55ZdpI5s9Cu19ERDTNPHaa68BAK5duyZEhM1mE5988knkcIjhJbT3xV7l7Nmz3Av+8OHDwsMfHR0VHiaRfKhGDICmabFatwDoujgTsbshIQYgDg9+ggBox2CCSARUIxJEAiAhEkQCICESRAIgIUagUCigUCgMOhnELoCEGAERUxmmaaJUKqFcLgtIETGs0IR+BFZXVz1Xh/OLpmmYm5uDZVmoVCoCU0YMGyTEEJTLZXz++ecAwq/Bqmka/vnnH/z222944oknRCaPGEJIiCHYv38/RkdHO7Y2KxQKrr0f2pmcnOSLDXdbB5bYm5AQQ5BOp7Gzs9OxmvWbb76JEydOdL3u0UcfjTtpxJBCQgzJ1tYW34SEkfTdsIjkQkIMQbVahWVZ2L9/v+t4kKYpQTghIYbAuT5lqVTi62OGaZqyzWW89pwg9hAxrxSwK3Eu0R92vRrnuj5w7E2xWxfaInpD3hchqVarGB8fD71duWmannusRwmTGF5IiASRAMjEjSASAAmRIBIACZEgEsB/AaqQ97RnqeM9AAAAAElFTkSuQmCC\"\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e \u003c/span\u003e\u003c/span\u003e(4)\u003c/p\u003e\u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003eis initialized as 0 and gradually assigned more weight.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"Section2\" id=\"Sec9\"\u003e\n \u003ch2\u003e3.4 Attention module embedding with full-resolution residual networks\u003c/h2\u003e\n \u003cp\u003eIt can be inferred from Eq. 4 that the resulting feature \u003cspan class=\"BoldItalic\" name=\"Emphasis\" type=\"BoldItalic\"\u003eQ\u003c/span\u003e at each position is a weighted sum of the features across all positions and original features. Accordingly, the PAM provides a global contextual view and selectively aggregates contexts according to the spatial attention map. In the sweet pepper images, some \u0026lsquo;stem\u0026rsquo; and \u0026lsquo;shoot\u0026rsquo; are inconspicuous or incomplete objects because of the influence of lighting and view. Those dominated salient objects (e.g. leaf, sweet pepper) would harm those inconspicuous object labeling. Since the PAM can selectively aggregate the similar features of inconspicuous objects to highlight their feature representations and avoid the influence of salient objects, we appended the PAM on top of the FRRN. The proposed FRRN-based network structure for semantic segmentation of the sweet pepper image is shown in Fig. \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e.\u003c/p\u003e\n \u003cp\u003eAs illustrated in Fig. \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e, FRRN was employed as the backbone. The whole network structure mainly refers to Encoder-Decoder [\u003cspan class=\"CitationRef\"\u003e33\u003c/span\u003e] structure. When the network starts working, the input feature maps are down-sampled through four pooling layers to extract the deep features in the images. After that, 4 up-sample layers are used to restore the size of the feature maps. And the mapping relationship between the original image and the label is obtained. Then the features from the FRRN would be fed into the PAM. Specifically, PAM is cascaded behind the concatenate (fusion) layer. The output features of the attention module are transformed by the convolution layer and input into the last convolution layer to generate the final prediction map. Besides, the notations RU and FRRU in Fig. \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e refer to the residual unit and the full-resolution residual unit, and the numbers on the lower right corner represent the number of convolution kernels in the unit, respectively. Also, the number of convolution kernels of each convolution layer in Fig. \u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e is 48, 32, and 8 (number of classes to predict). The sizes of convolution kernels are 5\u0026times;5, 1\u0026times;1, and 1\u0026times;1, respectively. The model architecture is proposed to contribute to the field of agricultural segmentation task.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"4 Model Training","content":"\u003cdiv class=\"Section2\" id=\"Sec11\"\u003e\n \u003ch2\u003e4.1 Experiment platform\u003c/h2\u003e\n \u003cp\u003eThe experiments in this work were conducted on the Ubuntu 16.10 LTS 64bit system. All the methods were implemented based on Tensorflow, and the TF-Slim library provided main functions for building model framework. The computer is equipped with 16 GB RAM and Intel Core i7-7700K CPU. Meanwhile, NVIDIA GTX1080Ti GPU acceleration technology was applied to improve the training speed.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"Section2\" id=\"Sec12\"\u003e\n \u003ch2\u003e4.2 Implementation Details\u003c/h2\u003e\n \u003cp\u003eThe RMSProp optimization algorithm was adopted during the process of training models. For the experiment I, the base learning rate was set to 1e-4, and the weight decay coefficient was set to 0.995. For the remaining experiments, the base learning rate and attenuation coefficient were set to 1e-5 and 0.995, respectively. Batch size was only set to 1 for all datasets because of the limitation of GPU memory and computing power. The number of iterations was set to 300 epochs for the experiment I and 200 epochs for the other experiments. The loss function used in our study was the softmax cross-entropy function. Besides, the data augmentation technology was not adopted in the work of this paper.\u003c/p\u003e\n \u003cp\u003eThe mean IoU is the most important segmentation evaluation metric, which is computed as follows:\u003c/p\u003e\n \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\text{M}\\text{I}\\text{o}\\text{U}=\\frac{1}{k+1} \\sum _{i=0}^{k}\\frac{{p}_{ii}}{{\\sum }_{j=0}^{k}{p}_{ij}+{\\sum }_{j=0}^{k}{p}_{ji}-{p}_{ii}}\\) \u003c/span\u003e\u003c/span\u003e(5)\u003c/p\u003e\n \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(k\\)\u003c/span\u003e\u003c/span\u003e represents different classes, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({p}_{ii}\\)\u003c/span\u003e\u003c/span\u003eis the number of pixels correctly predicted, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\sum }_{j=0}^{k}{p}_{ij}\\)\u003c/span\u003e\u003c/span\u003e is the number of pixels in which class \u003cem\u003ei\u003c/em\u003e is predicted to be class \u003cem\u003ej\u003c/em\u003e.\u003c/p\u003e\n \u003cp\u003eThe precision, recall, and \u003cem\u003eF\u003c/em\u003e1 score were introduced as additional evaluation indexes to evaluate the segmentation effect of sweet pepper images more objectively. In general, the higher and closer the precision and recall, the better the classification recognition effect. The \u003cem\u003eF\u003c/em\u003e1 score is computed based on the mean per-class precision/recall results as follows:\u003c/p\u003e\n \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(F1= 2\\times \\frac{precision\\times recall}{precision+recall}\\) \u003c/span\u003e\u003c/span\u003e(6)\u003c/p\u003e\n\u003c/div\u003e"},{"header":"5 Results And Discussion","content":"\u003cdiv class=\"Section2\" id=\"Sec14\"\u003e\n \u003ch2\u003e5.1 Evaluation of experiment I\u003c/h2\u003e\n \u003cp\u003eIn experiment I, several existing semantic segmentation methods were evaluated and compared on a small sweet pepper dataset. The mean pixel accuracy (PA) and mean intersection over union (IoU) of the trained models on the test set are shown in Table \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e. These model architectures achieved good segmentation results when used for sweet pepper segmentation in this paper. It is worth noting that most of these methods were training from scratch, rather than based on the pre-trained models. The approach based on pre-trained models was also adopted in our experiments. The results showed that although the training time of this method was less than other means, the segmentation effect was far inferior to those methods training from scratch. Consequently, the results of approach based on pre-trained models were not shown and compared in this paper.\u003c/p\u003e\n \u003cp\u003eAs shown in Table \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e, the mean PA and mean IoU of FRRN are 94.3% and 69.18%, respectively. In particular, its recognition accuracy for sweet peppers reached 93.2%. The segmentation and recognition effect of FRRN is better than that of other methods. From the perspective of pixel classification, the mean PAs of the five methods are similar, and all of them exceed 90%. It illustrates that the five networks have achieved good performance in pixel classification, especially for salient objects (e.g. leaf, sweet pepper). Meanwhile, the mean PAs of sub-experiments I-A, I-B, and I-E are slightly higher than that of the other two methods. From Table \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e, it can be seen that the main difference is caused by the small objects, including peduncles, stems, shoots, and wires. The PA of wires in experiment I-C was 6.30%, while that of peduncle in experiment I-D was only 6.00%. FRRN performs better than other methods, which verifies that it is more suitable for small-scale object recognition and fine segmentation. Moreover, the effects of several methods can be visualized in Fig. \u003cspan class=\"InternalRef\"\u003e6\u003c/span\u003e. These visualizations show that the FRRN achieved better segmentation results, both numerically and qualitatively. However, the segmentation effect of FRRN for sweet pepper needs to be improved, especially for the pixel-level recognition of inconspicuous objects. Hence, we took experiment II and experiment III.\u003c/p\u003e\n\u003ctable border=\"1\" id=\"Tab2\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eResults of each category on a small sweet pepper test set\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\" rowspan=\"2\"\u003e\n \u003cp\u003eMethods\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colspan=\"9\"\u003e\n \u003cp\u003ePixel accuracy/%\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" rowspan=\"2\"\u003e\n \u003cp\u003eMean IoU\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMean PA\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eBackground\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eLeaf\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003ePepper\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003ePeduncle\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eStem\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eShoot\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eWire\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eCut\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eI-A-MobileUNet [\u003cspan class=\"CitationRef\"\u003e34\u003c/span\u003e]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e93.6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e93.8\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e97.5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e93.5\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e46.4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e81.1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e64.5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e73.5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e59.6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e67.82\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eI-B-SegNet [\u003cspan class=\"CitationRef\"\u003e33\u003c/span\u003e]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e93.7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e93.8\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e96.9\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e94.1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e50.3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e80.3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e63.5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e73.2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e63.7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e67.33\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eI-C-BiseNet [\u003cspan class=\"CitationRef\"\u003e35\u003c/span\u003e]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e90.7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e90.9\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e96.2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e90.8\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e44.1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e73.2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e40.9\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e6.30\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e54.5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e53.78\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eI-D-ED [\u003cspan class=\"CitationRef\"\u003e33\u003c/span\u003e]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e92.0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e92.3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e96.7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e88.2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e6.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e73.6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e55.7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e55.5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e50.6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e55.80\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eI-E-FRRN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e94.3\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e93.7\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e97.6\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e93.2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e53.1\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e81.2\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e68.3\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e69.5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e67.5\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e\u003cstrong\u003e69.18\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cdiv class=\"Section2\" id=\"Sec15\"\u003e\n \u003ch2\u003e5.2 Evaluation of experiment II\u003c/h2\u003e\n \u003cp\u003eSemantic segmentation belongs to pixel-level tasks, which need to provide pixel by pixel output. Consequently, semantic segmentation requires higher resolution and more detailed information. It is supposed that the higher resolution of the input images, the richer feature information that the convolutional neural network could capture. High-resolution images help to improve the recognition accuracy and segmentation performance of the model. Especially for fine agriculture segmentation, higher image resolution is beneficial. However, if the resolution is extremely high, maybe it causes the problem of slow training speed. In this study, the resolution of the original images in the dataset is 800\u0026times;600, but the general model cannot directly perform feature analysis and extraction on such images. To balance the relationships between segmentation effect and computational complexity, the original image was cropped to 384\u0026times;384 and 512\u0026times;512 in experiment II, and train FRRN on images of different sizes.\u003c/p\u003e\n \u003cp\u003eWith the purpose of verifying the effect of input image resolution on segmentation results, we conducted experiment II. As shown in Table \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e, the higher segmentation performance was achieved by the same model after inputting high-resolution images. Compared with the input size of 384\u0026times;384, employing an input image size of 512\u0026times;512 yielded a result of 76.81% in mean IoU, which bringed 3.14% improvement. This shows that the increase in resolution contributes to the refinement of segmentation results. Besides, when the size of the input image is 512\u0026times;512, the mean PA and the \u003cem\u003eF\u003c/em\u003e1 score further improves to 95.47% and 95.57%, respectively. Meanwhile, although the improvement effect of pixel classification is not significant, the precision and recall have been kept at a high level and close to each other, which indicates that the classification effect of the model has improved. Results show that, in a certain range, the higher the resolution of the input images, the better the segmentation performance of FRRN.\u003c/p\u003e\n\u003ctable border=\"1\" id=\"Tab3\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eEvaluation and test results of different methods\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMethod(model)\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSize of input image/pixel\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMean PA/%\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMean IoU/%\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003ePrecision/%\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eRecall/%\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eF1 score/%\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eFRRN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e384\u0026times;384\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e95.10\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e73.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e95.55\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e95.10\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e95.22\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eFRRN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e512\u0026times;512\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e95.47\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e76.81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e95.97\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e95.47\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e95.57\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eFRRN\u0026thinsp;+\u0026thinsp;PAM\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e384\u0026times;384\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e97.38\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e76.45\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e97.64\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e97.38\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e97.51\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eFRRN\u0026thinsp;+\u0026thinsp;PAM\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e512\u0026times;512\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e97.94\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e78.88\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e98.20\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e97.94\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e98.07\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eBarth et al. [\u003cspan class=\"CitationRef\"\u003e36\u003c/span\u003e]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e300\u0026times;300\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e55.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cdiv class=\"Section2\" id=\"Sec16\"\u003e\n \u003ch2\u003e5.3 Evaluation of experiment III\u003c/h2\u003e\n \u003cp\u003eThe PAMs were employed on top of the FRRN to capture long-distance dependencies for better feature presentation. Similarly, two different resolution images were used as input to train the improved network. Various evaluation indexes achieved by the model on the test set are shown in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e. The segmentation performance of PAM on two different resolution images is shown in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e. It is observed that the best performance was achieved at an input image resolution of 512\u0026times;512. Compared with the baseline FRRN, the employment of PAM yielded a result of 78.88% in mean IoU, which brought an improvement of 2.07 percentage points. Also, mean PA increased by 2.47 percentage points. Meanwhile, the precision, recall, and \u003cem\u003eF\u003c/em\u003e1 score have also improved. In conclusion, results show that attention modules bring benefits to sweet pepper segmentation.\u003c/p\u003e\n \u003cp\u003eQuantitative evaluation is not enough to verify the importance of PAM, thus the effects of PAMs were visualized in Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e. As expected in Section \u003cspan class=\"InternalRef\"\u003e3.4\u003c/span\u003e, the PAM helps to strengthen the feature presentation of some inconspicuous objects. It can be observed from Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e that the employment of PAM makes some segmentation details and object boundaries clearer. As shown by the red circles in the first row, it is challenging to segment small-scale objects in detail, but under the action of PAM, the segmentation effect of the third column is significantly better than that of the second column. Specifically, in the second row, the FRRN model with PAM is better than the original FRRN model for the segmentation of small objects such as leaf stems. Therefore, Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e demonstrates that selective fusion over local features enhances the discrimination of details and refines the segmentation results. Besides, compared with the research of Barth et al. [\u003cspan class=\"CitationRef\"\u003e36\u003c/span\u003e], the accuracy of pepper segmentation has been improved obviously.\u003c/p\u003e\n \u003cp\u003eTo explore the generalization performance exhibited by a synthetically trained model when faced with similar data in the same domain, the best performing model (model of experiment III-B) was evaluated on 20 empirical (real) images. The segmentation results of the proposed model on empirical data are shown in Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e8\u003c/span\u003e. The model failed to discriminate and segment all the classes properly. This result confirms our hypothesis that the synthetically bootstrapped model could not generalize accurately to empirical data without fine-tuning. Note that these empirical images were not used for training or validation. In order to prove our hypothesis, 30 empirical images were used to fine-tune the proposed model, and the generalization performance of the model was tested on 20 empirical photos. As illustrated in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e, this practice resulted in increased performance, with a mean IoU of 59.81%. The result implies fine-tuning on a synthetically trained network can generalize to similar data in the same domain (empirical images).\u003c/p\u003e\n \u003cp\u003eNote that, for the empirical dataset, due to the influence of dark regions, some objects that were not manually annotated in the ground truth would be detected in the images and have a certain impact on the segmented images. Hence, although these parts were true positive, they were still evaluated as false positives. This annotation bias resulted in a lower segmentation performance of empirical images. In short, the results of experiment III show that the proposed optimization model of FRRN combined with PAM is feasible and effective for the recognition and segmentation of sweet pepper.\u003c/p\u003e\n\u003ctable border=\"1\" id=\"Tab4\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eThe evaluation results of the proposed model on empirical (real) images\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"4\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003emodel\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003efine-tuning\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMean PA/%\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMean IoU/%\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eFRRN\u0026thinsp;+\u0026thinsp;PAM\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e73.42\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e51.63\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eFRRN\u0026thinsp;+\u0026thinsp;PAM\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\surd\\)\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e80.56\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e59.81\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv class=\"Section2\" id=\"Sec17\"\u003e\n \u003ch2\u003e5.4 Results compared with other models\u003c/h2\u003e\n \u003cp\u003eWe further compared our method with existing methods on the sweet pepper testing set. The results are shown in Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e5\u003c/span\u003e. The FRRN\u0026thinsp;+\u0026thinsp;PAM model performs better than other segmentation methods. It is supposed that these networks are designed for street scene segmentation tasks, so they maybe not perform well in agricultural segmentation tasks. Among these methods, in order to achieve real-time performance, only ENet refrained from using a pre-trained network when designing the structure, so it did not get high scores. Specifically, a pre-trained network was abandoned by our network either, but it outperformed ENet by a large margin. Moreover, it also surpassed DANet, which uses a complex backbone ResNet-101.\u003c/p\u003e\n\u003ctable border=\"1\" id=\"Tab5\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eIoU scores on sweet pepper test set\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"2\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMethod\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eMean IoU/%\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eENet [\u003cspan class=\"CitationRef\"\u003e37\u003c/span\u003e]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e65.78\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eBiseNet [\u003cspan class=\"CitationRef\"\u003e35\u003c/span\u003e]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e69.75\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eDANet [\u003cspan class=\"CitationRef\"\u003e38\u003c/span\u003e]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e76.94\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eOur method (FRRN\u0026thinsp;+\u0026thinsp;PAM)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"char\"\u003e\n \u003cp\u003e78.88\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003c/div\u003e"},{"header":"6 Conclusions","content":"\u003cp\u003eIn this paper, CNN was applied to the semantic segmentation of the sweet pepper images. The FRRN model was improved by combining with attention mechanism. Specifically, PAMs were embedded at the top of the network to capture global dependencies in the spatial dimensions. This new architecture is clean, requires no additional post-processing, and can be trained from scratch. The proposed network (FRRN\u0026thinsp;+\u0026thinsp;PAM) was trained to segment sweet pepper images captured at night. Relevant experimental results show that PAMs contribute to providing more precise segmentation results. Meanwhile, our experiments indicate that proper image resolution benefits for segmentation performance. Besides, the synthetically trained model combined with fine-tuning can also be well generalized to the similar data in the same domain (empirical dataset). Our method achieves good performance on the test set of sweet pepper, with a mean IoU of 78.88%. The proposed method is generic and can be applied for other fruits and vegetables. It is hoped that this study can provide a theoretical basis for the development of image recognition in precision agriculture.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003eThis work is partly supported by Project of Agricultural Equipment Department of Jiangsu University (NZXB20210210), a project funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD) and Jiangsu University undergraduate scientific research project (20AB00).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCONFLICT OF INTEREST\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare no conflicts of interest.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eL\u0026oacute;pez-Mar\u0026iacute;n, J., G\u0026aacute;lvez, A.: Selecting vegetative/generative/dwarfing rootstocks for improving fruit yield and quality in water stressed sweet peppers. Sci. Hortic. \u003cb\u003e214\u003c/b\u003e, 9\u0026ndash;17 (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGilanie, G., Nasir, N., Bajwa, U.I. et al.: RiceNet: convolutional neural networks-based model to classify Pakistani grown rice seed types. Multimed. Syst. \u003cb\u003e27\u003c/b\u003e, 867\u0026ndash;875 (2021)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGhimire, S., Shakya, S.M., Srivastava, A.: Effects of organic manures and their combination with urea on sweet pepper production in the mid-hills. J. Agric. Environ. \u003cb\u003e14\u003c/b\u003e, 23 (2018)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBac, C.W., Hemming, J., Van Tuijl, J., Barth, R., Wais, E., Van Henten, E.J.: Performance evaluation of a harvesting robot for sweet pepper. J. Field Robot. \u003cb\u003e34\u003c/b\u003e(6), 1123\u0026ndash;1139 (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLehnert, C., English, A., Mccool, C., Tow, A.W., Perez, T.: Autonomous sweet pepper harvesting for protected cropping systems. IEEE Robot. Autom. Lett. \u003cb\u003e2\u003c/b\u003e(2), 872\u0026ndash;879 (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLuo, Y.Q., Sun, J., Shen, J.F., Wu, X.H., Wang, L., Zhu, W.D.: Apple leaf disease recognition and sub-class categorization based on improved multi-scale feature fusion network. IEEE Access \u003cb\u003e9\u003c/b\u003e, 95517\u0026ndash;95527 (2021)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDing, X.: Pepper picking robot in greenhouse. Agric. Eng. Technol. \u003cb\u003e39\u003c/b\u003e(1), 78\u0026ndash;82 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSun, J., He, X.F., Ge, X., Wu, X.H., Shen, J.F., Song, Y.Y.: Detection of key organs in tomato based on deep migration learning in a complex background. Agriculture-Basel \u003cb\u003e8\u003c/b\u003e, 196\u0026ndash;210 (2018)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang, J., Zhou, Q., Yin, A.: Self-adaptive segmentation method of cotton in natural scene by combining improved Otsu with ELM algorithm. Trans. Chin. Soc. Agric. Eng. \u003cb\u003e34\u003c/b\u003e(14), 173\u0026ndash;180 (2018)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDhingra, G., Kumar, V., Joshi, H.D.: A novel computer vision based neutrosophic approach for leaf disease identification and classification. Meas. \u003cb\u003e135\u003c/b\u003e, 782\u0026ndash;794 (2019)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBao, G., Cai, S., Qi, L., Xun, Y., Zhang, L., Yang, Q.: Multi-template matching algorithm for cucumber recognition in natural environment. Comput. Electron. Agric. \u003cb\u003e127\u003c/b\u003e, 754\u0026ndash;762 (2016)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu, C., Cao, Y., Luo, Y., Chen, G.L., Vokkarane, V., Ma, Y.S.: DeepFood: deep learning-based food image recognition for computer-aided dietary assessment. Lect. Notes Comput. Sci. 2016, 37\u0026ndash;48 (2016)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSun, J., Tan, W., Mao, H.: Recognition of multiple plant leaf diseases based on improved convolutional neural network. Trans. Chin. Soc. Agric. Eng. \u003cb\u003e33\u003c/b\u003e(19), 209\u0026ndash;215 (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGidaris, S., Komodakis, N.: Object detection via a multi-region and semantic segmentation-aware CNN model. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp.\u0026nbsp;1134\u0026ndash;1142 (2015)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSun, J., Yang, Y., He, X.F., Wu, X.H.: Northern maize leaf blight detection under complex field environment based on deep learning. IEEE Access \u003cb\u003e8\u003c/b\u003e, 33679\u0026ndash;33688 (2020)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHe, D., Xie, C.: Semantic image segmentation algorithm in a deep learning computer network. Multimed. Syst. (2020)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOlimov, B., Sanjar, K., Din, S. et al.: FU-Net: fast biomedical image segmentation model based on bottleneck convolution layers. Multimed. Syst. \u003cb\u003e27\u003c/b\u003e, 637\u0026ndash;650 (2021)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhao, D., Wu, R., Liu, X.: Apple positioning based on YOLO deep convolutional neural network for picking robot in complex background. Trans. Chin. Soc. Agric. Eng. \u003cb\u003e35\u003c/b\u003e(3), 172\u0026ndash;181 (2019)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFu, L., Feng, Y., Elkamil, T.: Image recognition method of multi-cluster kiwifruit in field based on convolutional neural networks. Trans. Chin. Soc. Agric. Eng. \u003cb\u003e34\u003c/b\u003e(2), 205\u0026ndash;211 (2018)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang, Y., Kailiang, Z., Li, Y.: Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN. Comput. Electron. Agric. \u003cb\u003e163\u003c/b\u003e, 104846 (2019)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBarth, R., Ijsselmuiden, J., Hemming, J., Henten, E.J.V.: Data synthesis methods for semantic segmentation in agriculture: a capsicum annuum dataset. Comput. Electron. Agric. \u003cb\u003e144\u003c/b\u003e, 284\u0026ndash;296 (2018)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLong, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. \u003cb\u003e39\u003c/b\u003e(4), 640\u0026ndash;651 (2014)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHe, K., Gkioxari, G., Doll\u0026aacute;r, P., Girshick, R.: Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, Venice, Italy, pp.\u0026nbsp;2980\u0026ndash;2988 (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRen, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. \u003cb\u003e39\u003c/b\u003e(6), 1137\u0026ndash;1149 (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. \u003cb\u003e40\u003c/b\u003e(4), 834\u0026ndash;848 (2016)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Honolulu, HI, USA (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHe, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Las Vegas, NV, USA, pp.\u0026nbsp;770\u0026ndash;778 (2016)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Honolulu, HI, USA (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBahdanau, D., Chorowski, J., Serdyuk, D.: End-to-end attention-based large vocabulary speech recognition. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, Shanghai, China, pp.\u0026nbsp;4945\u0026ndash;4949 (2016)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N.: Attention is all you need. In: Advances in Neural Information Processing Systems (NIPS), MIT Press, Long Beach, CA, USA, pp.\u0026nbsp;5998\u0026ndash;6008 (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations (ICLR), San Diego, CA, USA (2015)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMichał, D., Tim, R., Johannes, W.: Frustratingly short attention spans in neural language modeling. In: 5th International Conference on Learning Representations (ICLR), Toulon, France (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBadrinarayanan, V., Kendall, A., Cipolla, R.: SegNet: A deep convolutional encoder-decoder architecture for scene segmentation. IEEE Trans. Pattern Anal. Mach. Intell. \u003cb\u003e39\u003c/b\u003e(12), 2481\u0026ndash;2495 (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHoward, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T.: Mobilenets: efficient convolutional neural networks for mobile vision applications. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: Bisenet: bilateral segmentation network for real-time semantic segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), Springer, Munich, Germany, pp.\u0026nbsp;334\u0026ndash;349 (2018)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBarth, R., Ijsselmuiden, J., Hemming, J., Van Henten, E. J.: Synthetic bootstrapping of convolutional neural networks for semantic plant part segmentation. Comput. Electron. Agric. \u003cb\u003e161\u003c/b\u003e, 291\u0026ndash;304 (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePaszke, A., Chaurasia, A., Kim, S.: ENet: A deep neural network architecture for real-time semantic segmentation. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (2016), IEEE, Las Vegas, NV, USA (2016)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z.: Dual attention network for scene segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Long Beach, CA, USA, pp.\u0026nbsp;3146\u0026ndash;3154 (2019)\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"multimedia-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"mmsj","sideBox":"Learn more about [Multimedia Systems](http://link.springer.com/journal/530)","snPcode":"530","submissionUrl":"https://submission.nature.com/new-submission/530/3","title":"Multimedia Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Sweet pepper, Semantic segmentation, Recognition, Convolutional neural networks, Machine vision","lastPublishedDoi":"10.21203/rs.3.rs-1484696/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-1484696/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAutomatic and accurate recognition of the parts to be picked is the key to realize the intelligent picking of sweet pepper. However, pepper fruits are always covered by other organs, and small objects like stems and shoots are difficult to be recognized by machines or cameras under certain extreme conditions. To accurately segment and recognize all kinds of objects in sweet pepper images captured at night, three experiments were performed in this paper, and an enhanced model based on convolutional neural networks (CNNs) was eventually achieved. In experiment I, several semantic segmentation networks were trained on a small dataset, and the full-resolution residual network (FRRN) was taken as a primary network. Then, the impact of resolution of input images on the segmentation effect was investigated in experiment II. In order to strengthen the feature presentation of inconspicuous objects, the position attention module (PAM) was appended on top of the FRRN in experiment III. This architecture was further trained to provide more precise segmentation results. The experimental result shows that the mean intersection over union (IoU) is 78.88%, which is at least 1.94 percentage points higher than other models, and the mean pixel accuracy (PA) is 97.94% on the test set. The proposed method has higher generalization performance when facing unforeseen picking situations, meanwhile it is generic and can be applied to other fruits and vegetables.\u003c/p\u003e","manuscriptTitle":"Segmentation and recognition of filed sweet pepper based on improved self-attention convolutional neural networks","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2022-03-30 20:21:54","doi":"10.21203/rs.3.rs-1484696/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Major revision","date":"2022-07-27T10:04:10+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2022-05-10T02:28:11+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"e8ed5b65-eb43-403d-b70b-3e0de298076e","date":"2022-05-05T06:18:17+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2022-05-05T05:29:27+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2022-04-10T02:10:01+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2022-03-24T11:42:27+00:00","index":"","fulltext":""},{"type":"submitted","content":"Multimedia Systems","date":"2022-03-24T09:09:33+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"multimedia-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"mmsj","sideBox":"Learn more about [Multimedia Systems](http://link.springer.com/journal/530)","snPcode":"530","submissionUrl":"https://submission.nature.com/new-submission/530/3","title":"Multimedia Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"c14b6cd9-0a1f-4926-9d74-12f8e8e5bfaf","owner":[],"postedDate":"March 30th, 2022","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2022-08-06T04:29:06+00:00","versionOfRecord":[],"versionCreatedAt":"2022-03-30 20:21:54","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-1484696","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-1484696","identity":"rs-1484696","version":["v1"]},"buildId":"J0_U0BvcaRcwD8yVFaRlm","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}