DSIS-DPR:Structured Instance Segmentation and Diffusion Prior Refinement for Dental Anatomy Learning

Few-shot instance segmentation (FSIS) conjoins the few-shot learning paradigm with general instance segmentation, which provides a possible way of tackling instance segmentation in the lack of abundant labeled data for training. This paper presents a Fully Guided Network (FGN) for few-shot instance segmentation. FGN perceives FSIS as a guided model where a so-called support set is encoded and utilized to guide the predictions of a base instance segmentation network (i.e., Mask R-CNN), critical to which is the guidance mechanism. In this view, FGN introduces different guidance mechanisms into the various key components in Mask R-CNN, including Attention-Guided RPN, Relation-Guided Detector, and Attention-Guided FCN, in order to make full use of the guidance effect from the support set and adapt better to the inter-class generalization. Experiments on public datasets demonstrate that our proposed FGN can outperform the state-of-the-art methods.

Instance segmentation in high-resolution (HR) remote sensing imagery is one of the most challenging tasks and is more difficult than object detection and semantic segmentation tasks. It aims to predict class labels and pixel-wise instance masks to locate instances in an image. However, there are rare methods currently suitable for instance segmentation in the HR remote sensing images. Meanwhile, it is more difficult to implement instance segmentation due to the complex background of remote sensing images. In this article, a novel instance segmentation approach of HR remote sensing imagery based on Cascade Mask R-CNN is proposed, which is called a high-quality instance segmentation network (HQ-ISNet). In this scheme, the HQ-ISNet exploits a HR feature pyramid network (HRFPN) to fully utilize multi-level feature maps and maintain HR feature maps for remote sensing images’ instance segmentation. Next, to refine mask information flow between mask branches, the instance segmentation network version 2 (ISNetV2) is proposed to promote further improvements in mask prediction accuracy. Then, we construct a new, more challenging dataset based on the synthetic aperture radar (SAR) ship detection dataset (SSDD) and the Northwestern Polytechnical University very-high-resolution 10-class geospatial object detection dataset (NWPU VHR-10) for remote sensing images instance segmentation which can be used as a benchmark for evaluating instance segmentation algorithms in the high-resolution remote sensing images. Finally, extensive experimental analyses and comparisons on the SSDD and the NWPU VHR-10 dataset show that (1) the HRFPN makes the predicted instance masks more accurate, which can effectively enhance the instance segmentation performance of the high-resolution remote sensing imagery; (2) the ISNetV2 is effective and promotes further improvements in mask prediction accuracy; (3) our proposed framework HQ-ISNet is effective and more accurate for instance segmentation in the remote sensing imagery than the existing algorithms.

Panoptic Segmentation aims to provide an understanding of background (stuff) and instances of objects (things) at a pixel level. It combines the separate tasks of semantic segmentation (pixel level classification) and instance segmentation to build a single unified scene understanding task. Typically, panoptic segmentation is derived by combining semantic and instance segmentation tasks that are learned separately or jointly (multi-task networks). In general, instance segmentation networks are built by adding a foreground mask estimation layer on top of object detectors or using instance clustering methods that assign a pixel to an instance center. In this work, we present a fully convolution neural network that learns instance segmentation from semantic segmentation and instance contours (boundaries of things). Instance contours along with semantic segmentation yield a boundary aware semantic segmentation of things. Connected component labeling on these results produces instance segmentation. We merge semantic and instance segmentation results to output panoptic segmentation. We evaluate our proposed method on the CityScapes dataset to demonstrate qualitative and quantitative performances along with several ablation studies. Our overview video can be accessed from https://youtu.be/wBtcxRhG3e0.

Motivated by how humans perceive geometry and color to recognize objects, we propose a boundary embedded attentional convolution (BEACon) network for point cloud instance segmentation. At the core of BEACon, we introduce the attentional weight in the convolution layer to adjust the neighboring features, with the weight being adapted to the relationship between geometry and color changes. As a result, BEACon makes use of both geometry and color information, takes instance boundary as an important feature, and thus learns a more discriminative feature representation in the neighborhood. Experimental results show that BEACon outperforms the state-of-the-art by a large margin. Ablation studies are also provided to prove the large benefit of incorporating both geometry and color into attention weight for instance segmentation.

Bounding boxes have been widely implemented into aerial object detection for its simplicity. They perform instance-level location with the coordinates and orientation for each target. But the defects such as coarse edge information impede semantic interpretation in earth observation. Besides, in terms of the aerial imaging instruments, it's essential to recognize the exterior appearance and contour of the objects. In this paper, we propose a novel aerial instance segmentation method termed adaptive RoI extraction network (ARE-Net) which bridges the gap of accurately delineating instances under the complex back-ground of aerial images. To exert instance segmentation under the proprietary property, e.g., complex background, densely distributed instances, of aerial images, RoIs are pooled from multi-level feature maps and integral region proposals. On this basis, global attention RoI extractor (GA-RoIE) and perceptual RoI extractor (PRoIE) are respectively introduced for detection branch and mask branch to perform adaptive RoI extraction for aerial images. Meanwhile, to reconcile the probability distribution regional distribution of pixel-wise prediction in aerial images, we present the Adaptive Compound loss function to improve the integrating degree of the predicted binary mask to ground truth mask. Additionally, we adopt RegNetx with Deformable Convolution to optimize ARE-Net, and name it as R-ARE-Net. Despite implementing pixel-wise prediction, comprehensive experiments on iSAID and NWPU VHR-10 instance segmentation dataset still have verified the effectiveness and efficiency of ARE-Net and R-ARE-Net. Experimental results indicate that our proposed methods receive the highest AP value (38.0% AP on iSAID and 64.2% AP on NWPU VHR-10 instance segmentation dataset) and lowest FLOPs and Parameters consumption (~46% reduced FLOPs and 61.5% reduced Parameters than SCNet) among the mainstream methods. Besides, the false alarms, missing segmentations, poorly predicted masks, and under-segmentations that appeared in the mainstream methods can be avoided to some extend for R-ARE-Net.

Extracting building instances from remote sensing images has various applications. However, existing instance segmentation methods have difficulty maintaining regular and angular building boundaries, corrupting the mask quality. This letter proposes a boundary-optimized instance segmentation method based on the direction field to cope with the low boundary quality. The proposed method develops a DF-Mask head to improve the mask quality of the primary instance segmentation network (Mask R-CNN) through boundary optimization. Within the DF-Mask head, first, a gated directional context-aware module (GDCAM) improves the mask features through the directional context and a designed gating mechanism. Then a direction field rectification module (DFRM) predicts the direction field and iteratively rectifies the mask. To the best of our knowledge, this is the first time that direction field rectification has been developed to the instance segmentation method and applied to building instance extraction. Experimental results on the Vaihingen and Massachusetts datasets suggest that our method outperforms the eight state-of-the-art methods. Compared with Mask R-CNN, our method improves APs by 4.7% and 2.2% on the two datasets, respectively.

Depth hoar (DH) forms when there is a strong temperature gradient in the snowpack. In the Antarctic ice sheet (AIS), DH mostly forms during summer insolation. The seasonal regularity provides an age marker for each internal snow/firn layer, which is necessary for estimating surface mass balance (SMB) using ice-penetrating radar (IPR) and ice core. However, little is known about DH inside the AIS because ice drill and snow pit observations are inefficient and sparse. The deployment of the Operation IceBridge (OIB) airborne snow radar has significantly enhanced the field observation dataset. So far, the spatial distribution of DH remains unclear due to the lack of DH extraction over the AIS from OIB snow radar. Based on instance segmentation SOLOv2 and self-attention mechanism Swin Transformer, a DH layer automatic extraction algorithm ST-SOLOv2 is proposed, with AP50 of 0.9 and F1-score of 0.83, outperforms other commonly used instance segmentation networks (SOLOv2, Mask R-CNN, BlendMask, YOLACT, and CondInst). After the proposed preprocessing pipeline, we conduct the ST-SOLOv2 to locate each DH near the surface. Our results suggest DH number increases with elevation and decreases with slope angle from coast to inland. And DH number decreases with surface melting days as melting/freezing cycles obscure snow layering. We present a method for efficiently monitoring the DH distribution that can be used in further studies of snow radar SMB estimation.

Automatic coarse-to-fine joint detection and segmentation of underwater non-structural live crabs for precise feeding

Instance segmentation of images is an important tool for automated scene understanding. Neural networks are usually trained to optimize their overall performance in terms of accuracy. Meanwhile, in applications such as automated driving, an overlooked pedestrian seems more harmful than a falsely detected one. In this work, we present a false negative detection method for image sequences based on inconsistencies in time series of tracked instances given the availability of image sequences in online applications. As the number of instances can be greatly increased by this algorithm, we apply a false positive pruning using uncertainty estimates aggregated over instances. To this end, instance-wise metrics are constructed which characterize uncertainty and geometry of a given instance or are predicated on depth estimation. The proposed method serves as a post-processing step applicable to any neural network that can also be trained on single frames only. In our tests, we obtain an improved trade-off between false negative and false positive instances by our fused detection approach in comparison to the use of an ordinary score value provided by the instance segmentation network during inference.

Tissue clearing and light-sheet microscopy technologies offer new opportunities to quantify the three-dimensional (3D) neural structure at a cellular or even sub-cellular resolution. Although many efforts have been made to recognize nuclei in 3D using deep learning techniques, current state-of-the-art approaches often work in a two-step manner, i.e., first segment nucleus regions within a 2D optical slice and then assemble the regions into the 3D instance of a nucleus. Due to the poor inter-slice resolution in many volumetric microscopy images and lack of contextual information across image slices, the current two-step approaches yield less accurate instance segmentation results. To address these limitations, a novel neural network for 3D nucleus instance segmentation (NIS) is proposed, called NIS-Net, which jointly segments and assembles the 3D instances of nuclei. Specifically, a pretext task is designed to predict the image appearance of the to-be-processed slice using the learned context from the processed slices, where the well-characterized contextual information is leveraged to guide the assembly of 3D nuclei instances. Since our NIS-Net progressively identifies nuclei instances by sliding over the entire image stack, our method is capable of segmenting nuclei instances for the whole mouse brain. Experimental results show that our proposed NIS-Net achieves higher accuracy and more reasonable nuclei instances than the current counterpart methods.KeywordsNucleus instance segmentationLight-sheet microscopy imagesSelf-supervised learningVariational autoencoderDeep learning

The primary premise of autonomous railway inspection using unmanned aerial vehicles is achieving autonomous flight along the railway. In our previous work, fitted centerline-based unmanned aerial vehicle (UAV) navigation is proven to be an effective method to guide UAV autonomous flying. However, the empirical parameters utilized in the fitting procedure lacked a theoretical basis and the fitted curves were also not coherent nor smooth. To address these problems, this paper proposes a skeleton detection method, called the dynamic-weight parallel instance and skeleton network, to directly extract the centerlines that can be viewed as skeletons. This multi-task branch network for skeleton detection and instance segmentation can be trained end to end. Our method reformulates a fused loss function with dynamic weights to control the dominant branch. During training, the sum of the weights always remains constant and the branch with a higher weight changes from instance to skeleton gradually. Experiments show that our model yields 93.98% mean average precision (mAP) for instance segmentation, a 51.9% F-measure score (F-score) for skeleton detection, and 60.32% weighted mean metrics for the entire network based on our own railway skeleton and instance dataset which comprises 3235 labeled overhead-view images taken in various environments. Our method can achieve more accurate railway skeletons and is useful to guide the autonomous flight of a UAV in railway inspection.

Maritime traffic monitoring systems are particularly important in Mediterranean ports, as they provide more comprehensive data collection compared to traditional systems such as the Automatic Identification System (AIS), which is not mandatory for all vessels. This paper improves the existing real-time maritime traffic monitoring systems by introducing a distance estimation algorithm for monocular cameras, which aims to provide high quality maritime traffic metadata collection for traffic density analysis. Two distance estimation methods based on a pinhole camera model are presented: the Vessel-Focused Distance Estimation (VFDE) and the novel Vessel Object-Focused Distance Estimation (VOFDE). While VFDE uses the predefined height of a vessel for distance estimation, VOFDE uses standardized dimensions of objects on the vessel, detected with a Convolutional Neural Network (CNN) for instance segmentation to enhance estimation accuracy. Our evaluation covers distances up to 414 m, which is significantly beyond the scope of previous studies. When compared to the distances measured with a precise instrument, VOFDE achieves a Percentage Deviation Index (PDI) of 1.34% to 9.45%. This advance holds significant potential for improving maritime surveillance with monocular cameras and is also applicable in other areas, such as low-cost maritime vehicles equipped with single cameras.

Medical image analysis is critical yet challenged by the need of jointly segmenting organs or tissues, and numerous instances for anatomical structures and tumor microenvironment analysis. Existing studies typically formulated different segmentation tasks in isolation, which overlooks the fundamental interdependencies between these tasks, leading to suboptimal segmentation performance and insufficient medical image understanding. To address this issue, we propose a Co-Seg++ framework for versatile medical segmentation. Specifically, we introduce a novel co-segmentation paradigm, allowing semantic and instance segmentation tasks to mutually enhance each other. We first devise a spatio-sequential prompt encoder (SSP-Encoder) to capture long-range spatial and sequential relationships between segmentation regions and image embeddings as prior spatial constraints. Moreover, we devise a multi-task collaborative decoder (MTC-Decoder) that leverages cross-guidance to strengthen the contextual consistency of both tasks, jointly computing semantic and instance segmentation masks. Extensive experiments on diverse CT and histopathology datasets demonstrate that the proposed Co-Seg++ outperforms state-of-the-arts in the semantic, instance, and panoptic segmentation of dental anatomical structures, histopathology tissues, and nuclei instances. The source code is available at https://github.com/xq141839/Co-Seg-Plus.

Instance segmentation and panoptic segmentation is being paid more and more attention in recent years. In comparison with bounding box based object detection and semantic segmentation, instance segmentation can provide more analytical results at pixel level. Given the insight that pixels belonging to one instance have one or more common attributes of current instance, we bring up an one-stage instance segmentation network named Common Attribute Support Network (CASNet), which realizes instance segmentation by predicting and clustering common attributes. CASNet is designed in the manner of fully convolutional and can implement training and inference from end to end. And CASNet manages predicting the instance without overlaps and holes, which problem exists in most of current instance segmentation algorithms. Furthermore, it can be easily extended to panoptic segmentation through minor modifications with little computation overhead. CASNet builds a bridge between semantic and instance segmentation from finding pixel class ID to obtaining class and instance ID by operations on common attribute. Through experiment for instance and panoptic segmentation, CASNet gets mAP 32.8% and PQ 59.0% on Cityscapes validation dataset by joint training, and mAP 36.3% and PQ 66.1% by separated training mode. For panoptic segmentation, CASNet gets state-of-the-art performance on the Cityscapes validation dataset.

Tracking the myotendinous junction (MTJ) in consecutive ultrasound images is crucial for understanding the mechanics and pathological conditions of the muscle-tendon unit. However, the lack of reliable and efficient identification of MTJ due to poor image quality and boundary ambiguity restricts its application in motion analysis. In recent years, with the rapid development of deep learning, the region-based convolution neural network (RCNN) has shown great potential in the field of simultaneous objection detection and instance segmentation in medical images. This article proposes a region-adaptive network (RAN) to localize MTJ region and to segment it in a single shot. Our model learns about the salient information of MTJ with the help of a composite architecture. Herein, a region-based multitask learning network explores the region containing MTJ, while a parallel end-to-end U-shaped path extracts the MTJ structure from the adaptively selected region for combating data imbalance and boundary ambiguity. By demonstrating the ultrasound images of the gastrocnemius, we showed that the RAN achieves superior segmentation performance when compared with the state-of-the-art Mask RCNN method with an average Dice score of 80.1%. Our proposed method is robust and reliable for advanced muscle and tendon function examinations obtained by ultrasound imaging.