Augmenting Data Using Gaussian Mixture Embedding For Improving Land Cover Segmentation

Medical image segmentation has played an important role in medical analysis and widely developed for many clinical applications. Deep learning-based approaches have achieved high performance in semantic segmentation but they are limited to pixel-wise setting and imbalanced classes data problem. In this paper, we tackle those limitations by developing a new deep learning-based model which takes into account both higher feature level i.e. region inside contour, intermediate feature level i.e. offset curves around the contour and lower feature level i.e. contour. Our proposed Offset Curves (OsC) loss consists of three main fitting terms. The first fitting term focuses on pixel-wise level segmentation whereas the second fitting term acts as attention model which pays attention to the area around the boundaries (offset curves). The third terms plays a role as regularization term which takes the length of boundaries into account. We evaluate our proposed OsC loss on both 2D network and 3D network. Two common medical datasets, i.e. retina DRIVE and brain tumor BRATS 2018 datasets are used to benchmark our proposed loss performance. The experiments have shown that our proposed OsC loss function outperforms other mainstream loss functions such as Cross-Entropy, Dice, Focal on the most common segmentation networks Unet, FCN.

Accurate organ-level segmentation is essential for achieving high-throughput, non-destructive, and automated plant phenotyping. To address the challenge of intelligent acquisition of phenotypic parameters in tomato plants, we propose TSINet, an end-to-end dual-task segmentation network designed for effective and precise semantic labeling and instance recognition of tomato point clouds, based on the Pheno4D dataset. TSINet adopts an encoder–decoder architecture, where a shared encoder incorporates four Geometry-Aware Adaptive Feature Extraction Blocks (GAFEBs) to effectively capture local structures and geometric relationships in raw point clouds. Two parallel decoder branches are employed to independently decode shared high-level features for the respective segmentation tasks. Additionally, a Dual Attention-Based Feature Enhancement Module (DAFEM) is introduced to further enrich feature representations. The experimental results demonstrate that TSINet achieves superior performance in both semantic and instance segmentation, particularly excelling in challenging categories such as stems and large-scale instances. Specifically, TSINet achieves 97.00% mean precision, 96.17% recall, 96.57% F1-score, and 93.43% IoU in semantic segmentation and 81.54% mPrec, 81.69% mRec, 81.60% mCov, and 86.40% mWCov in instance segmentation. Compared with state-of-the-art methods, TSINet achieves balanced improvements across all metrics, significantly reducing false positives and false negatives while enhancing spatial completeness and segmentation accuracy. Furthermore, we conducted ablation studies and generalization tests to systematically validate the effectiveness of each TSINet component and the overall robustness of the model. This study provides an effective technological approach for high-throughput automated phenotyping of tomato plants, contributing to the advancement of intelligent agricultural management.

Semantic segmentation is an object classification such as a perceptual grouping in image analysis tasks. And semantic segmentation methods have been proposed to improve the pixel-wise classification accuracies in the research field of computer vision. Recently, the methods achieve high accuracies by using convolutional neural networks (CNN). These CNN-based methods generally need large number and variety of training images to assure a generalization performance without fine tuning. However, it is difficult to prepare the large-scale image datasets which were given reliable semantic labels. In order to overcome this problem, we propose an efficient method for feature transformation to improve the generalization performance in semantic segmentation. Our method is formulated as a variant of neighborhood preserving embedding (NPE) incorporating the within-class coherency and showed best generalization performances compared with other feature transformation methods.

Understanding how landscape composition relates to aesthetic perception remains a challenge in landscape design research. While semantic segmentation methods have achieved strong performance in scene understanding, they are rarely used to analyze design structure and aesthetic interpretation in garden landscapes. This study introduces a computational framework that integrates semantic segmentation with compositional design descriptors and social aesthetic indicators for the analysis of garden design language. A dataset containing more than 2,000 garden images was constructed with pixel-level semantic annotations and associated aesthetic labels. Based on the segmentation outputs, a set of quantitative descriptors was derived to represent spatial composition of landscape elements, including spatial distribution, proportional composition, and element adjacency relationships. These compositional descriptors were further combined with social aesthetic indicators obtained from public surveys and textual analysis to explore relationships between landscape structure and perceived aesthetic qualities. Experimental results show that the proposed framework achieves competitive performance in semantic segmentation while providing interpretable representations of landscape design composition. The analysis further reveals consistent associations between certain compositional patterns and aesthetic perception. The proposed approach demonstrates the potential of integrating semantic scene understanding with design-aware feature representation for data-driven analysis of landscape aesthetics and garden design language.

Fire and smoke recognition has a wide range of applications in fire warning, anti-jamming, battlefield situational awareness and other fields. Using the method of semantic segmentation for fire and smoke recognition can obtain the fire and smoke area of the image more accurately and comprehensively. Although the DeepLabV3+ network based on the Encoder-decoder structure has achieved excellent performance in semantic segmentation in many fields, it still has some shortcomings, such as slow fitting, blurred edge segmentation, and holes in the feature map. In order to address the deficiencies of DeepLabV3+,In this paper we propose a novel method of introducing parallel dual attention mechanism network(DAMN). We designed a parallel connection between DAMN and atous spatial pyramid pooling(ASPP). DAMN and ASPP process the feature map generated by the network in parallel, and then fuse the feature map information generated by DAMN and ASPP module. Meanwhile, we built a multi-scene fire and smoke dataset named “Fire-Smoke” for comprehensive assessment in fire and smoke image recognition. Fire-Smoke contains over 8,000 hand-labeled smoke labels. We achieve better segmentation performance on Fire-Smoke dataset. In particular, it achieves an average IoU score of 85.77 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">%</sup> on the Fire-Smoke test set, an improvement of 2.18% over the DeepLabV3+.

Chapter 12 - Semantic scene segmentation for robotics

Introduction: Synchrotron radiation micro-computed tomography (SRμCT) has been used as a non-invasive technique to examine the microstructure and tissue integration of biodegradable bone implants. To be able to characterize parameters regarding the disintegration and osseointegration of such materials quantitatively, the three-dimensional (3D) image data provided by SRμCT needs to be processed by means of semantic segmentation. However, accurate image segmentation is challenging using traditional automated techniques. This study investigates the effectiveness of deep learning approaches for semantic segmentation of SRμCT volumes of Mg-based implants in sheep bone ex vivo.Methodology: For this purpose different convolutional neural networks (CNNs), including U-Net, HR-Net, U²-Net, from the TomoSeg framework, the Scaled U-Net framework, and 2D/3D U-Net from the nnU-Net framework were trained and validated. The image data used in this work was part of a previous study where biodegradable screws were surgically implanted in sheep tibiae and imaged using SRμCT after different healing periods. The comparative analysis of CNN models considers their performance in semantic segmentation and subsequent calculation of degradation and osseointegration parameters. The models’ performance is evaluated using the intersection over union (IoU) metric, and their generalization ability is tested on unseen datasets.Results and discussion: This work shows that the 2D nnU-Net achieves better generalization performance, with the degradation layer being the most challenging label to segment for all models.

Several large scale datasets, coupled with advances in deep neural network architectures have been greatly successful in pushing the boundaries of performance in semantic segmentation in recent years. However, the scale and magnitude of such datasets prohibits ubiquitous use and widespread adoption of such models, especially in settings with serious hardware and software resource constraints. Through this work, we propose two simple variants of the recently proposed IDD dataset, namely IDD-mini and IDD-lite, for scene understanding in unstructured environments. Our main objective is to enable research and benchmarking in training segmentation models. We believe that this will enable quick prototyping useful in applications like optimum parameter and architecture search, and encourage deployment on low resource hardware such as Raspberry Pi. We show qualitatively and quantitatively that with only 1 h of training on 4 GB GPU memory, we can achieve satisfactory semantic segmentation performance on the proposed datasets. KeywordsSemantic segmentationNeural architecture search

Augmenting RGB data with measured depth has been shown to improve the performance of a range of tasks in computer vision, including object detection and semantic segmentation. Although depth sensors such as the Microsoft Kinect have facilitated easy acquisition of such depth information, the vast majority of images used in vision tasks do not contain depth information. In this paper, we show that augmenting RGB images with estimated depth can also improve the accuracy of both object detection and semantic segmentation. Specifically, we first exploit the recent success of depth estimation from monocular images and learn a deep depth estimation model. Then, we learn deep depth features from the estimated depth and combine with RGB features for object detection and semantic segmentation. In addition, we propose an RGB-D semantic segmentation method, which applies a multi-task training scheme: semantic label prediction and depth value regression. We test our methods on several data sets and demonstrate that incorporating information from estimated depth improves the performance of object detection and semantic segmentation remarkably.

Rotated target recognition is a challenge for Convolutional Neural Networks (CNN), and the current solution is to make CNN rotational invariant through data augmentation. However, data augmentation makes CNN easy to overfit small scale sonar image datasets, and increases its numbers of parameters and training time. This paper proposes to recognize rotated targets of sonar images using a novel CNN with Rotated Inputs (RICNN), which doesn’t need data augmentation. During training, RICNN was trained with sonar images of targets only at one orientation, which avoid it to learn multiple rotated versions of the same targets, and reduces both number of parameters and training time of CNN. During testing, RICNN calculated classification scores for each test image and its all-possible rotated versions. The max of these classification scores were used to simultaneously estimate the category and orientation of each target. Besides, to improve the generalization of RICNN on imbalanced sonar datasets, this paper also designs an imbalanced data sampler. Experiments on a self-made small, imbalanced sonar image rotated target recognition dataset show that the improved RICNN achieves 4.25% higher classification accuracy than data augmentation, and reduces the number of parameters and training time to 2.25% and 19.2% of that of data augmentation method. Moreover, RICNN achieves comparable orientation estimation accuracy with a CNN orientation regressor trained with data augmentation. Codes, dataset are publicly available.

Preparation of high-quality datasets for the urban scene understanding is a labor-intensive task, especially, for datasets designed for the autonomous driving applications. The application of the coarse ground truth (GT) annotations of these datasets without detriment to the accuracy of semantic image segmentation (by the mean intersection over union - mIoU) could simplify and speedup the dataset preparation and model fine tuning before its practical application. Here the results of the comparative analysis for semantic segmentation accuracy obtained by PSPNet deep learning architecture are presented for fine and coarse annotated images from Cityscapes dataset. Two scenarios were investigated: scenario 1 - the fine GT images for training and prediction, and scenario 2 - the fine GT images for training and the coarse GT images for prediction. The obtained results demonstrated that for the most important classes the mean accuracy values of semantic image segmentation for coarse GT annotations are higher than for the fine GT ones, and the standard deviation values are vice versa. It means that for some applications some unimportant classes can be excluded and the model can be tuned further for some classes and specific regions on the coarse GT dataset without loss of the accuracy even. Moreover, this opens the perspectives to use deep neural networks for the preparation of such coarse GT datasets.

Single-view height estimation and semantic segmentation have received increasing attention in recent years and play an important role in the photogrammetry and remote sensing communities. The height information and semantic information of images are correlated, and some recent works have shown that multi-task learning methods can achieve complementation of task-related features and improve the prediction results of the multiple tasks. Although much progress has been made in recent works, how to effectively extract and fuse height features and semantic features is still an open issue. In this paper, a self- and cross-enhancement network (SCE-Net) is proposed to jointly perform height estimation and semantic segmentation on single aerial images. A feature separation–fusion module is constructed to effectively separate and fuse height features and semantic features based on an attention mechanism for feature representation enhancement across tasks. In addition, a height-guided feature distance loss and a semantic-guided feature distance loss are designed based on deep metric learning to achieve task-aware feature representation enhancement. Extensive experiments are conducted on the Vaihingen dataset and the Potsdam dataset to verify the effectiveness of the proposed method. The experimental results demonstrate that the proposed SCE-Net could outperform the state-of-the-art methods and achieve better performance in both height estimation and semantic segmentation.

Convolutional Neural Networks (CNN) consist of various hyper-parameters which need to be specified or can be altered when defining a deep learning architecture. There are numerous studies which have tested different types of networks (e.g. U-Net, DeepLabv3+) or created new architectures, benchmarked against well-known test datasets. However, there is a lack of real-world mapping applications demonstrating the effects of changing network hyper-parameters on model performance for land use and land cover (LULC) semantic segmentation. In this paper, we analysed the effects on training time and classification accuracy by altering parameters such as the number of initial convolutional filters, kernel size, network depth, kernel initialiser and activation functions, loss and loss optimiser functions, and learning rate. We achieved this using a well-known top performing architecture, the U-Net, in conjunction with LULC training data and two multispectral aerial images from North Queensland, Australia. A 2018 image was used to train and test CNN models with different parameters and a 2015 image was used for assessing the optimised parameters. We found more complex models with a larger number of filters and larger kernel size produce classifications of higher accuracy but take longer to train. Using an accuracy-time ranking formula, we found using 56 initial filters with kernel size of 5 × 5 provide the best compromise between training time and accuracy. When fully training a model using these parameters and testing on the 2015 image, we achieved a kappa score of 0.84. This compares to the original U-Net parameters which achieved a kappa score of 0.73.

Semantic image segmentation is an interesting problem in Computer Vision with many potential applications. The DeepLab model is combined with two other networks: Resnet and Conditional Random Field networks, making the DeepLab model a fairly deep network structure to increase semantic segmentation performance. Many previous studies argued that there are some limits on the deep learning model&#x0027;s depth as the deep structure may lead to vanishing/exploding gradient, which the model&#x0027;s performance. This paper presents an experimental study to compare the effect of several ImageNet pre-trained Resnet variant models with different network layers used as feature extractor in DeepLab model to solve semantic image segmentation task. In this study, three Resnet34, Resnet50, and Resnet101 models as network extractor of DeepLabV3were explored. The experiment found that semantic image segmentation model performance measured by the best accuracy and average accuracies of DeepLabV3- Resnet34, DeepLabV3-Resnet50, and DeepLabV3-Resnet101 are (0.87, 0.86) (0.86, 0.84), and (0.92, 0.88) respectively. Based on the experiment, DeepLabV3-Resnet101 achieved the best semantic segmentation performance than the other models

The semantic segmentation of remote sensing images faces two major challenges: high inter-class similarity and interference from ubiquitous shadows. In order to address these issues, we develop a novel edge loss reinforced semantic segmentation network (ERN) that leverages the spatial boundary context to reduce the semantic ambiguity. The main contributions of this paper are as follows: (1) we propose a novel end-to-end semantic segmentation network for remote sensing, which involves multiple weighted edge supervisions to retain spatial boundary information; (2) the main representations of the network are shared between the edge loss reinforced structures and semantic segmentation, which means that the ERN simultaneously achieves semantic segmentation and edge detection without significantly increasing the model complexity; and (3) we explore and discuss different ERN schemes to guide the design of future networks. Extensive experimental results on two remote sensing datasets demonstrate the effectiveness of our approach both in quantitative and qualitative evaluation. Specifically, the semantic segmentation performance in shadow-affected regions is significantly improved.