Coherency Preserving Feature Transformation for Semantic Segmentation

Introduction: Synchrotron radiation micro-computed tomography (SRμCT) has been used as a non-invasive technique to examine the microstructure and tissue integration of biodegradable bone implants. To be able to characterize parameters regarding the disintegration and osseointegration of such materials quantitatively, the three-dimensional (3D) image data provided by SRμCT needs to be processed by means of semantic segmentation. However, accurate image segmentation is challenging using traditional automated techniques. This study investigates the effectiveness of deep learning approaches for semantic segmentation of SRμCT volumes of Mg-based implants in sheep bone ex vivo.Methodology: For this purpose different convolutional neural networks (CNNs), including U-Net, HR-Net, U²-Net, from the TomoSeg framework, the Scaled U-Net framework, and 2D/3D U-Net from the nnU-Net framework were trained and validated. The image data used in this work was part of a previous study where biodegradable screws were surgically implanted in sheep tibiae and imaged using SRμCT after different healing periods. The comparative analysis of CNN models considers their performance in semantic segmentation and subsequent calculation of degradation and osseointegration parameters. The models’ performance is evaluated using the intersection over union (IoU) metric, and their generalization ability is tested on unseen datasets.Results and discussion: This work shows that the 2D nnU-Net achieves better generalization performance, with the degradation layer being the most challenging label to segment for all models.

Understanding how landscape composition relates to aesthetic perception remains a challenge in landscape design research. While semantic segmentation methods have achieved strong performance in scene understanding, they are rarely used to analyze design structure and aesthetic interpretation in garden landscapes. This study introduces a computational framework that integrates semantic segmentation with compositional design descriptors and social aesthetic indicators for the analysis of garden design language. A dataset containing more than 2,000 garden images was constructed with pixel-level semantic annotations and associated aesthetic labels. Based on the segmentation outputs, a set of quantitative descriptors was derived to represent spatial composition of landscape elements, including spatial distribution, proportional composition, and element adjacency relationships. These compositional descriptors were further combined with social aesthetic indicators obtained from public surveys and textual analysis to explore relationships between landscape structure and perceived aesthetic qualities. Experimental results show that the proposed framework achieves competitive performance in semantic segmentation while providing interpretable representations of landscape design composition. The analysis further reveals consistent associations between certain compositional patterns and aesthetic perception. The proposed approach demonstrates the potential of integrating semantic scene understanding with design-aware feature representation for data-driven analysis of landscape aesthetics and garden design language.

Convolutional Neural Network (CNN) is widely used for semantic segmentation and land-use and land-cover (LULC) mapping of very high-resolution (VHR) remote sensing images. The convolution operation is a powerful method for VHR classification, but the loss of high-frequency detail information caused during its operation decreases the classification accuracy, particularly in the boundary. Thus, it is necessary to supply additional boundary information to the CNN for alleviating this situation. In the classification task (and in LULC mapping), providing more effective information generates a better classification result. Current methods regard the boundary of images as the same category object and process it uniformly, which loses a notable amount of useful information because of the different properties, such as ambiguity and transition, between remote sensing images and their boundaries. Thus, a semantic segmentation method with category boundary for LULC mapping is proposed in this paper. First, a multi-task CNN called the category boundary detection network (CBDN) is designed to extract the boundary information of different category objects. Second, this category boundary and VHR images are used for initial semantic segmentation. Finally, the category boundary and the initial semantic segmentation result (ISSR) are fused to obtain the final LULC map by a two-step strategy, including the explicit fusion and the boundary attention loss function. To verify whether category boundary improved the classification accuracy, a set of comparative experiments were conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam datasets. The method in this paper was compared with a semantic segmentation method with no boundary information and a semantic segmentation method with global boundary. The results showed that the proposed method in this paper achieved good performances in the Vaihingen (overall accuracy (OA) = 0.924, Kappa coefficient (K) = 0.898, mean F1 score (mF1) = 0.896 and mean Intersection over Union (mIoU) = 0.817) and Potsdam datasets (OA = 0.890, K = 0.857, mF1 = 0.923, and mIoU = 0.860) based on the eroded labels.

Image semantic segmentation is a challenging problem for low‐level computer vision. Recently, deep convolutional neural networks (DCNNs) have been proved to achieve outstanding performance in image semantic segmentation. Most current methods still have some problems in segmenting the object edges and the integrity of objects. In this study, the authors first construct the difference‐pooling module in the DCNNs to extract the object edge gradients and get finer boundary in segmentation results. Then the combination of the pyramid pooling module and the atrous spatial pyramid pooling extracts the image global features and the context structure information by building long‐distance dependency between pixels, which is just like a simple fully connected conditional random field (CRF). Different from other methods, the proposed method does not need extra pre‐processing and post‐processing steps, such as extracting gradient features by the traditional algorithm and building context relationships by CRF. Finally, the experimental results on the PASCAL VOC2012 benchmark indicate that the proposed model can obtain the finer boundaries and more complete parts.

Remote sensing images contain a wealth of geospatial information. To accurately identify different geospatial categories and extract relevant data, image semantic segmentation plays a crucial role. In recent years, deep learning technology has brought significant breakthroughs to semantic segmentation of remote sensing images, significantly enhancing its performance. This paper investigates the application of deep learning technologies in remote sensing image semantic segmentation, based on Convolutional Neural Networks (CNN) and Transformer-based semantic segmentation methods. It conducts an in-depth comparison of their structural characteristics and applicable scenarios, summarizes the achievements and shortcomings of existing research, and provides technical references and theoretical support for future studies, thereby contributing to the further development of deep learning technology in the field of remote sensing. Research results indicate that CNN-based semantic segmentation methods still hold advantages in extracting local features and achieving efficient segmentation, whereas Transformers address CNN's limitations in global context modeling and long-range dependency capture. Therefore, the collaborative integration of CNN and Transformers will become an important research direction for enhancing model performance in the future.

Research on image classification sparked the latest deep-learning boom. Many downstream tasks, including semantic segmentation, benefit from it. The state-of-the-art semantic segmentation models are all based on deep learning, and they sometimes make some semantic mistakes. In a semantic segmentation dataset with a small number of categories, images are often collected from a single scene, and there is a close semantic connection between any two categories. However, in the semantic segmentation dataset collected from multiple scenes, two categories may be irrelevant. The probability of objects in one category appearing next to objects in other categories is different, which is the basis of the paper. Semantic segmentation methods need to solve two problems of positioning and classification. This paper is dedicated to correcting those clearly wrong classifications that are contrary to reality. Specifically, we first calculate the relevancy between different class pairs. Then, based on this knowledge, we infer the category of a connected component according to the relationships of this connected component with its surrounding connected components and correct the obviously wrong classifications made by a deep learning semantic segmentation model. Several well-performing deep learning models are experimented on two challenging public datasets in the field of semantic image segmentation. Our proposed method improves the performance of UPerNet, OCRNet and SETR from 40.7%, 43% and 48.64% to 42.07%, 44.09% and 49.09% mean IoU on the ADE20K validation set, and the performance of PSPNet, DeepLabV3 and OCRNet from 37.26%, 37.3% and 39.5% to 38.93%, 38.95% and 40.63% mean IoU on the COCO-Stuff dataset, which shows the effectiveness of the method.

Fire and smoke recognition has a wide range of applications in fire warning, anti-jamming, battlefield situational awareness and other fields. Using the method of semantic segmentation for fire and smoke recognition can obtain the fire and smoke area of the image more accurately and comprehensively. Although the DeepLabV3+ network based on the Encoder-decoder structure has achieved excellent performance in semantic segmentation in many fields, it still has some shortcomings, such as slow fitting, blurred edge segmentation, and holes in the feature map. In order to address the deficiencies of DeepLabV3+,In this paper we propose a novel method of introducing parallel dual attention mechanism network(DAMN). We designed a parallel connection between DAMN and atous spatial pyramid pooling(ASPP). DAMN and ASPP process the feature map generated by the network in parallel, and then fuse the feature map information generated by DAMN and ASPP module. Meanwhile, we built a multi-scene fire and smoke dataset named “Fire-Smoke” for comprehensive assessment in fire and smoke image recognition. Fire-Smoke contains over 8,000 hand-labeled smoke labels. We achieve better segmentation performance on Fire-Smoke dataset. In particular, it achieves an average IoU score of 85.77 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">%</sup> on the Fire-Smoke test set, an improvement of 2.18% over the DeepLabV3+.

Accurate organ-level segmentation is essential for achieving high-throughput, non-destructive, and automated plant phenotyping. To address the challenge of intelligent acquisition of phenotypic parameters in tomato plants, we propose TSINet, an end-to-end dual-task segmentation network designed for effective and precise semantic labeling and instance recognition of tomato point clouds, based on the Pheno4D dataset. TSINet adopts an encoder–decoder architecture, where a shared encoder incorporates four Geometry-Aware Adaptive Feature Extraction Blocks (GAFEBs) to effectively capture local structures and geometric relationships in raw point clouds. Two parallel decoder branches are employed to independently decode shared high-level features for the respective segmentation tasks. Additionally, a Dual Attention-Based Feature Enhancement Module (DAFEM) is introduced to further enrich feature representations. The experimental results demonstrate that TSINet achieves superior performance in both semantic and instance segmentation, particularly excelling in challenging categories such as stems and large-scale instances. Specifically, TSINet achieves 97.00% mean precision, 96.17% recall, 96.57% F1-score, and 93.43% IoU in semantic segmentation and 81.54% mPrec, 81.69% mRec, 81.60% mCov, and 86.40% mWCov in instance segmentation. Compared with state-of-the-art methods, TSINet achieves balanced improvements across all metrics, significantly reducing false positives and false negatives while enhancing spatial completeness and segmentation accuracy. Furthermore, we conducted ablation studies and generalization tests to systematically validate the effectiveness of each TSINet component and the overall robustness of the model. This study provides an effective technological approach for high-throughput automated phenotyping of tomato plants, contributing to the advancement of intelligent agricultural management.

One of the most common tasks in medical imaging is semantic segmentation. Achieving this segmentation automatically has been an active area of research, but the task has been proven very challenging due to the large variation of anatomy across different patients. However, recent advances in deep learning have made it possible to significantly improve the performance of image recognition and semantic segmentation methods in the field of computer vision. Due to the data driven approaches of hierarchical feature learning in deep learning frameworks, these advances can be translated to medical images without much difficulty. Several variations of deep convolutional neural networks have been successfully applied to medical images. Especially fully convolutional architectures have been proven efficient for segmentation of 3D medical images. In this article, we describe how to build a 3D fully convolutional network (FCN) that can process 3D images in order to produce automatic semantic segmentations. The model is trained and evaluated on a clinical computed tomography (CT) dataset and shows state-of-the-art performance in multi-organ segmentation.

In recent years, the convolutional neural network (CNN) has made remarkable achievements in semantic segmentation. The method of semantic segmentation has a desirable application prospect. Nowadays, the methods mostly use an encoder-decoder architecture as a way of generating pixel by pixel segmentation prediction. The encoder is for extracting feature maps and decoder for recovering feature map resolution. An improved semantic segmentation method on the basis of the encoder-decoder architecture is proposed. We can get better segmentation accuracy on several hard classes and reduce the computational complexity significantly. This is possible by modifying the backbone and some refining techniques. Finally, after some processing, the framework has achieved good performance in many datasets. In comparison with the traditional architecture, our architecture does not need additional decoding layer and further reuses the encoder weight, thus reducing the complete quantity of parameters needed for processing. In this paper, a modified focal loss function is also put forward, as a replacement for the cross-entropy function to achieve a better treatment of the imbalance problem of the training data. In addition, more context information is added to the decode module as a way of improving the segmentation results. Experiments prove that the presented method can get better segmentation results. As an integral part of a smart city, multimedia information plays an important role. Semantic segmentation is an important basic technology for building a smart city.

Weakly supervised semantic segmentation based on EM algorithm with localization clues

Augmenting RGB data with measured depth has been shown to improve the performance of a range of tasks in computer vision, including object detection and semantic segmentation. Although depth sensors such as the Microsoft Kinect have facilitated easy acquisition of such depth information, the vast majority of images used in vision tasks do not contain depth information. In this paper, we show that augmenting RGB images with estimated depth can also improve the accuracy of both object detection and semantic segmentation. Specifically, we first exploit the recent success of depth estimation from monocular images and learn a deep depth estimation model. Then, we learn deep depth features from the estimated depth and combine with RGB features for object detection and semantic segmentation. In addition, we propose an RGB-D semantic segmentation method, which applies a multi-task training scheme: semantic label prediction and depth value regression. We test our methods on several data sets and demonstrate that incorporating information from estimated depth improves the performance of object detection and semantic segmentation remarkably.

Models based on deep convolutional neural networks (CNN) have significantly improved the performance of semantic segmentation. However, learning these models requires a large amount of training images with pixel-level labels, which are very costly and time-consuming to collect. In this paper, we propose a method for learning CNNbased semantic segmentation models from images with several types of annotations that are available for various computer vision tasks, including image-level labels for classification, box-level labels for object detection and pixel-level labels for semantic segmentation. The proposed method is flexible and can be used together with any existing CNNbased semantic segmentation networks. Experimental evaluation on the challenging PASCAL VOC 2012 and SIFTflow benchmarks demonstrate that the proposed method can effectively make use of diverse training data to improve the performance of the learned models.

Semantic segmentation is considered to be one of the basic steps in understanding image content. For semantic segmentation, if multi-spectral images are used together with color images, more successful results are obtained due to complementary information obtained from multi-spectral images. In this paper, a semantic segmentation method was developed in which the images obtained from CCD and thermal sensors were used together. In the proposed method, convolutional neural networks were used in encoder-decoder architecture. The experiments carried out show that the developed method produces better numerical and visual results than the works published in the literature.

Several large scale datasets, coupled with advances in deep neural network architectures have been greatly successful in pushing the boundaries of performance in semantic segmentation in recent years. However, the scale and magnitude of such datasets prohibits ubiquitous use and widespread adoption of such models, especially in settings with serious hardware and software resource constraints. Through this work, we propose two simple variants of the recently proposed IDD dataset, namely IDD-mini and IDD-lite, for scene understanding in unstructured environments. Our main objective is to enable research and benchmarking in training segmentation models. We believe that this will enable quick prototyping useful in applications like optimum parameter and architecture search, and encourage deployment on low resource hardware such as Raspberry Pi. We show qualitatively and quantitatively that with only 1 h of training on 4 GB GPU memory, we can achieve satisfactory semantic segmentation performance on the proposed datasets. KeywordsSemantic segmentationNeural architecture search