A real-time semantic segmentation model using iteratively shared features in multiple sub-encoders

In this study, we propose a segmentation model based on convolutional neural networks (CNNs) to address image segmentation challenges in computer vision. Prior to designing the model, the activation function and other modules of the convolutional neural network were optimized to meet specific requirements. The segmentation task was transformed into binary classification problem to simplify network calculations and improve efficiency. Additionally, the model utilized a mask map obtained from the semantic segmentation model to aid in instance segmentation. Class activation technology was introduced to extract feature mapping maps. The corresponding thermal maps were obtained to achieve target instance segmentation. To further validate the effectiveness of the segmentation model, simulation experiments were conducted on semantic segmentation and instance segmentation respectively. The results show that the accuracy of the basic semantic segmentation model reached 87.58%, while the average accuracy of the entire class of the optimized instance segmentation model reached 97.9%. Therefore, the research and design of image segmentation models demonstrate high accuracy and good robustness.

Object detection and semantic segmentation are two fundamental techniques of various applications in the fields of Intelligent Vehicles (IV) and Advanced Driving Assistance System (ADAS). Early studies separately handle these two problems. In this paper, inspired by some recent works, we propose a deep neural network model for joint object detection and semantic segmentation. Given an image, an encoder-decoder convolution network extracts a set of feature maps, these feature maps are shared by the detection branch and the segmentation branch to jointly carry out the object detection and semantic segmentation. In the detection branch, we design a PriorBox initialization mechanism to propose more object candidates. In the segmentation branch, we use the multi-scale atrous convolution to explore the global and local semantic information in traffic scenes. Benefiting from the PriorBox Initialization Mechanism (PBIM) and Multi-Scale Atrous Convolution (MSAC), our model presents the competitive performance. In the experiments, we widely compare with several recently-proposed methods on the public Cityscapes dataset, achieving the highest accuracy. In addition, to verify the robustness and generalization of our model, the extension experiments are also conducted on the well-known VOC2012 dataset.

Recent advances in convolutional neural networks have led to considerably accurate semantic segmentation tasks. However, previous works use a pre-trained backbone borrowed from image classification tasks with considerable computational complexity, resulting in large latencies. Many methods have been proposed to reduce the latency of the segmentation network without loss of accuracy. These methods have reported varying experimental results with different computing devices, acceleration techniques, and input image sizes. Although most studies claim that their results are state-of-the-art, it is necessary to reconsider whether they are being compared under the same conditions. We propose a performance evaluation method of real-time semantic segmentation models to compare the performance under the same conditions fairly. In addition, we carry out an empirical study to evaluate the performance of recent real-time semantic segmentation networks and make a comparative analysis between them. We train the segmentation models using the same input data and data augmentation method. Then, the performance of the segmentation methods is analyzed regarding accuracy and speed. In contrast to most studies that exclude the time required for the pre-processing and post-processing steps, we measured the actual processing time needed to perform semantic segmentation with a real dataset. Further, we measured the processing speed and power consumption of the segmentation models in embedded devices in which real-time segmentation is applied, unlike previous studies that measured performance on a PC. Experimental results showed that the real-time semantic segmentation methods could not run in real-time on embedded devices when considering the pre-processing and post-processing steps. By comprehensively considering the inference speed, energy consumption, and processing time of semantic segmentation models, the experimental results show that FasterSeg-S is suited for embedded devices.

Pine wilt disease (PWD) is the most dangerous biohazard of pine species and poses a serious threat to forest resources. Coupling satellite remote sensing technology and deep learning technology for the accurate monitoring of PWD is an important tool for the efficient prevention and control of PWD. We used Gaofen-2 remote sensing images to construct a dataset of discolored standing tree samples of PWD and selected three semantic segmentation models—DeepLabv3+, HRNet, and DANet—for training and to compare their performance. To build a GAN-based semi-supervised semantic segmentation model for semi-supervised learning training, the best model was chosen as the generator of generative adversarial networks (GANs). The model was then optimized for structural adjustment and hyperparameter adjustment. Aimed at the characteristics of Gaofen-2 images and discolored standing trees with PWD, this paper adopts three strategies—swelling prediction, raster vectorization, and forest floor mask extraction—to optimize the image identification process and results and conducts an application demonstration study in Nanping city, Fujian Province. The results show that among the three semantic segmentation models, HRNet was the optimal conventional semantic segmentation model for identifying discolored standing trees of PWD based on Gaofen-2 images and that its MIoU value was 68.36%. Additionally, the GAN-based semi-supervised semantic segmentation model GAN_HRNet_Semi improved the MIoU value by 3.10%, and its recognition segmentation accuracy was better than the traditional semantic segmentation model. The recall rate of PWD discolored standing tree monitoring in the demonstration area reached 80.09%. The combination of semi-supervised semantic segmentation technology and high-resolution satellite remote sensing technology provides new technical methods for the accurate wide-scale monitoring, prevention, and control of PWD.

Research on image classification sparked the latest deep-learning boom. Many downstream tasks, including semantic segmentation, benefit from it. The state-of-the-art semantic segmentation models are all based on deep learning, and they sometimes make some semantic mistakes. In a semantic segmentation dataset with a small number of categories, images are often collected from a single scene, and there is a close semantic connection between any two categories. However, in the semantic segmentation dataset collected from multiple scenes, two categories may be irrelevant. The probability of objects in one category appearing next to objects in other categories is different, which is the basis of the paper. Semantic segmentation methods need to solve two problems of positioning and classification. This paper is dedicated to correcting those clearly wrong classifications that are contrary to reality. Specifically, we first calculate the relevancy between different class pairs. Then, based on this knowledge, we infer the category of a connected component according to the relationships of this connected component with its surrounding connected components and correct the obviously wrong classifications made by a deep learning semantic segmentation model. Several well-performing deep learning models are experimented on two challenging public datasets in the field of semantic image segmentation. Our proposed method improves the performance of UPerNet, OCRNet and SETR from 40.7%, 43% and 48.64% to 42.07%, 44.09% and 49.09% mean IoU on the ADE20K validation set, and the performance of PSPNet, DeepLabV3 and OCRNet from 37.26%, 37.3% and 39.5% to 38.93%, 38.95% and 40.63% mean IoU on the COCO-Stuff dataset, which shows the effectiveness of the method.

Maize stem–leaf segmentation framework based on deformable point clouds

Deep learning (DL)-based approaches are notable for their ability to establish feature associations without relying on physical constraints, unlike traditional strategies that are complex and dependent on expert experience. However, three main challenges hinder the versatility of semantic segmentation models. First, the targets in these images are dense and exist at varying spatial scales, which imposes higher demands on the model for accurate segmentation across scales. Second, the segmentation of small targets in the images is often overlooked, leading to a compromise between fine segmentation and model efficiency. Lastly, the data-intensive nature of remote sensing images and the resource-intensive operations of large-scale networks impose significant communication and computation burdens on edge devices, which may not have sufficient resources to handle them effectively. To address these challenges, this paper proposes a lightweight semantic segmentation method for remote sensing images to achieve high-precision segmentation for multi-scale targets while maintaining low computational complexity. The main components include: (1) embedding the inverted residual block structure to minimize the number of model parameters and computational costs; (2) introducing the parallel irregular space pyramid pooling module to efficiently aggregate multi-scale contextual information for fine-grained recognition of small targets; and (3) embedding transfer learning into the encoder-decoder structure to speed up the convergence rate and improve multi-scale feature fusion capability, thereby reducing semantic information loss. The proposed lightweight method has been extensively tested on real-world high-resolution remote sensing datasets. It achieved PA, MPA, MIoU, and FWIoU scores of 87.90%, 75.76%, 66.29%, and 78.81% on the Vaihingen dataset; 87.03%, 85.31%, 74.85%, and 77.54% on the Potsdam dataset; and 95.37%, 83.33%, 75.70%, and 91.31% on the Aeroscapes dataset. Compared to other popular semantic segmentation models, the proposed method achieved the highest values in all four evaluation indicators, demonstrating its effectiveness and superiority.

In order to improve the accuracy of image segmentation without changing the structure of original semantic segmentation models, an approach to train semantic segmentation models by using generative adversarial network (SS-GAN) is proposed. Using adversarial network to distinguish the source of segmented images, the model can learn the high-order relationship between pixels to enhance the spatial continuity of pixels in the segmented image. There are three aspects related to the work: constructing the generative model of fully convolutional network (FCN) structure by using VGG, and segment image preliminarily; constructing the adversarial model and training it by combining the original images, fake segmented images and real segmented images; modifying the loss function, adding the anti-loss to assist segmentation model training, encouraging generative network to learn the inter-pixel relationship independently. Experiments on PASCAL VOC and Cityscapes datasets show that the proposed method achieves better performance than the existing advanced methods, and improves IoU by 1.56% and 1.93%, respectively.

In the task of image classification of machine tool machining, the coolant water column will block the tool and interfere with the image classification. To solve this problem, firstly, the coolant water column is removed from the image by semantic segmentation. Because the number of samples in the data set is insufficient, the effect of semantic segmentation model is not very good. These results are taken as the input number of image classification model According to, it will directly affect the accuracy of classification, but it is impossible to modify the parameters of semantic segmentation model during classification training. In this case, the image after semantic segmentation processing is combined with the original image according to certain rules, combined with the information of some original images, and then classified. This method can effectively reduce the interference to the classification model when the semantic segmentation model is wrong.

The purpose of semantic segmentation is to classify the pixels within the target contour. Edge detection is another major basic vision task in machine vision. Today’s most effective semantic segmentation models and contour edge detection models are isolated networks. The edge of the output of the semantic segmentation model is coarse and cannot be directly used. And the output of the edge detection network cannot output the classification information of the pixels inside the contour. In view of the above shortcomings of the existing network, we propose a semantic segmentation model based on edge constraint optimization, so that the output of the semantic segmentation model has more delicate edge information, and the network directly outputs accurate contour edge graphs. The edge information output by the network can be directly used for tasks such as corner detection and center point detection. Experiments show that the mIOU statistics obtained by our model on the validation set of PASCAL VOC2012 can reach 83.9%. At the same time, more detailed edge details can be obtained. This algorithm has high engineering and theoretical research value.

Semantic image segmentation is an interesting problem in Computer Vision with many potential applications. The DeepLab model is combined with two other networks: Resnet and Conditional Random Field networks, making the DeepLab model a fairly deep network structure to increase semantic segmentation performance. Many previous studies argued that there are some limits on the deep learning model's depth as the deep structure may lead to vanishing/exploding gradient, which the model's performance. This paper presents an experimental study to compare the effect of several ImageNet pre-trained Resnet variant models with different network layers used as feature extractor in DeepLab model to solve semantic image segmentation task. In this study, three Resnet34, Resnet50, and Resnet101 models as network extractor of DeepLabV3were explored. The experiment found that semantic image segmentation model performance measured by the best accuracy and average accuracies of DeepLabV3- Resnet34, DeepLabV3-Resnet50, and DeepLabV3-Resnet101 are (0.87, 0.86) (0.86, 0.84), and (0.92, 0.88) respectively. Based on the experiment, DeepLabV3-Resnet101 achieved the best semantic segmentation performance than the other models

The yield and quality of fresh lettuce can be determined from the growth rate and color of individual plants. Manual assessment and phenotyping for hundreds of varieties of lettuce is very time consuming and labor intensive. In this study, we utilized a “Sensor-to-Plant” greenhouse phenotyping platform to periodically capture top-view images of lettuce, and datasets of over 2000 plants from 500 lettuce varieties were thus captured at eight time points during vegetative growth. Here, we present a novel object detection–semantic segmentation–phenotyping method based on convolutional neural networks (CNNs) to conduct non-invasive and high-throughput phenotyping of the growth and development status of multiple lettuce varieties. Multistage CNN models for object detection and semantic segmentation were integrated to bridge the gap between image capture and plant phenotyping. An object detection model was used to detect and identify each pot from the sequence of images with 99.82% accuracy, semantic segmentation model was utilized to segment and identify each lettuce plant with a 97.65% F1 score, and a phenotyping pipeline was utilized to extract a total of 15 static traits (related to geometry and color) of each lettuce plant. Furthermore, the dynamic traits (growth and accumulation rates) were calculated based on the changing curves of static traits at eight growth points. The correlation and descriptive ability of these static and dynamic traits were carefully evaluated for the interpretability of traits related to digital biomass and quality of lettuce, and the observed accumulation rates of static straits more accurately reflected the growth status of lettuce plants. Finally, we validated the application of image-based high-throughput phenotyping through geometric measurement and color grading for a wide range of lettuce varieties. The proposed method can be extended to crops such as maize, wheat, and soybean as a non-invasive means of phenotype evaluation and identification.

Methods and datasets on semantic segmentation for Unmanned Aerial Vehicle remote sensing images: A review

Semantic segmentation has always been a very challenging research topic in computer vision and deep learning and has extensive applications in real-life scenarios. With the development of computing hardware and deep learning technology, researchers have a higher research enthusiasm for semantic segmentation. This work briefly introduces several semantic segmentation models, datasets, and the main issues of traditional semantic segmentation. Afterward, the current mainstream semantic segmentation algorithm and the experimental results were compared. Finally, future research directions of semantic segmentation are presented and discussed.

Automated data-driven method for creating digital building models from dense point clouds and images through semantic segmentation and parametric model fitting