DF-Mask R-CNN: Direction Field-Based Optimized Instance Segmentation Network for Building Instance Extraction

Few-shot instance segmentation (FSIS) conjoins the few-shot learning paradigm with general instance segmentation, which provides a possible way of tackling instance segmentation in the lack of abundant labeled data for training. This paper presents a Fully Guided Network (FGN) for few-shot instance segmentation. FGN perceives FSIS as a guided model where a so-called support set is encoded and utilized to guide the predictions of a base instance segmentation network (i.e., Mask R-CNN), critical to which is the guidance mechanism. In this view, FGN introduces different guidance mechanisms into the various key components in Mask R-CNN, including Attention-Guided RPN, Relation-Guided Detector, and Attention-Guided FCN, in order to make full use of the guidance effect from the support set and adapt better to the inter-class generalization. Experiments on public datasets demonstrate that our proposed FGN can outperform the state-of-the-art methods.

To solve the complex background problems (e.g. Noise interference, object overlap, and different illumination) that affect the classification performance on plant leaf images, this paper proposes an instance segmentation and classification method for plant leaf images based on IFPN SNMS CFFI-Mask R-CNN (ISC-MRCNN) and ACPSOSVM-Dual Channels Convolutional Neural Network (APS-DCCNN). To obtain the foreground of plant leaf images, the lateral connection structure of the feature map pyramid in ISC-MRCNN fuses the feature maps of different depths, so that the network learns more detailed features. Then, the Soft Non-Maximum Suppression Algorithm is employed to improve the detection performance of overlapping objects. Next, the pooling method of integrating the continuous function can reduce the precision loss during the alignment of the mapping between the feature map and the original image. Finally, by constructing a mask filter layer, complex backgrounds are masked. To distinguish the similarity between plant leaf images, APS-DCCNN is used to classify the foreground images. In this process, the Support Vector Machine is used to replace softmax and then an Adaptive Chaotic Particle Swarm Algorithm is employed to optimize it. The experimental results show that compared with Mask R-CNN, the average precision of ISC-MRCNN has increased by 1.89% under different thresholds. The proposed method is suitable for the object detection and instance segmentation problems with complex background. Besides, compared with traditional CNN, the average precision of the classification results obtained by APS-DCCNN has improved by 1.59%. This has shown that the proposed method is suitable for the classification of plant leaves.

Instance segmentation needs to locate all instances in an image correctly and segment each instance precisely. Currently, the most dominant methods for instance segmentation take object detection as a pre-task. However, they rely on the accuracy of object detection incredibly. If the pre-task cannot predict an accurate bounding box, the performance of instance segmentation will degenerate. In this paper, we present a novel method for instance segmentation to solve this problem, which is called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</b> egmenting <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</b> eyond the <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</b> ounding <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</b> ox ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S3B-Net</b> ). Our S3B-Net designs a sub-network to help instance segmentation methods based on object detection to segment the part of an instance beyond the bounding box. Specifically, the sub-network first predicts a two-dimensional pixel embedding for each pixel. Then, the Gaussian function is employed to calculate a pixel’s probability belongs to a corresponding instance according to the two-dimensional pixel embedding. Finally, the output of the sub-network combines with the output of instance segmentation based on object detection to generate a more precise instance mask. Our sub-network can easily extend on the existing instance segmentation method based on object detection to segment instance beyond the bounding box. We do our experiments on dominant instance segmentation datasets, such as the COCO dataset and Cityscapes dataset. The results show that our method can achieve 6.8 points gain compared with the baseline Mask R-CNN with ResNet-50-FPN in Cityscapes datasets, and 1.7 points gain with ResNet-101-FPN-DCN in COCO datasets. Our S3B-Net outperforms the previous state-of-the-art instance segmentation method, which proves our method is competitive. The source code of our method will be made available.

Instance segmentation in medical imaging plays a crucial role in clinical diagnostic tasks, and have shown promising performance in practical applications. In this paper, we discuss a more fine-grained instance segmentation task: dental structured instance segmentation based on panoramic radiographs. However, direct segmentation of tooth structures encounters inherent challenges. Traditional instance segmentation networks often fall short in capturing intricate internal features, and exacerbated by the frequent blurring found in medical imaging, which can result in the deficiency of anatomical details. To deal with these problems, we propose a novel framework called DSISDPR, which combines a dental structured instance segmentation (DSIS) network with an enhanced diffusion prior refinement (DPR) method. Specifically, our innovatively designed structureaware network leverages fine-grained feature fusion, acquiring a richer representation of internal anatomical structures. With the integration of adversarial learning, the model is primed to deliver holistic and subtle predictions of tooth structures. Furthermore, taking inspiration from dentists’ inherent ability to utilize prior knowledge, such as understanding dental structures to label invisible anatomical structures, we propose a diffusion inpainting to refine the results of DSIS without additional annotations. Equipped with built-in structure learning, DPR is capable of modifying anomalies within each predicted segmentation, resulting in a more robust and complete structured segmentation result. Meanwhile, we ensure rigorous oversight over the reconstruction of areas affected by abnormalities, ensuring that any introduced adjustments minimally disrupt the wellpredicted structured segmentation results. Extensive experiments have demonstrated that our DSIS-DPR outperforms all existing classical instance segmentation networks. The collected dataset is available: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Zzz512/TSD</uri> .

Instance segmentation in high-resolution (HR) remote sensing imagery is one of the most challenging tasks and is more difficult than object detection and semantic segmentation tasks. It aims to predict class labels and pixel-wise instance masks to locate instances in an image. However, there are rare methods currently suitable for instance segmentation in the HR remote sensing images. Meanwhile, it is more difficult to implement instance segmentation due to the complex background of remote sensing images. In this article, a novel instance segmentation approach of HR remote sensing imagery based on Cascade Mask R-CNN is proposed, which is called a high-quality instance segmentation network (HQ-ISNet). In this scheme, the HQ-ISNet exploits a HR feature pyramid network (HRFPN) to fully utilize multi-level feature maps and maintain HR feature maps for remote sensing images’ instance segmentation. Next, to refine mask information flow between mask branches, the instance segmentation network version 2 (ISNetV2) is proposed to promote further improvements in mask prediction accuracy. Then, we construct a new, more challenging dataset based on the synthetic aperture radar (SAR) ship detection dataset (SSDD) and the Northwestern Polytechnical University very-high-resolution 10-class geospatial object detection dataset (NWPU VHR-10) for remote sensing images instance segmentation which can be used as a benchmark for evaluating instance segmentation algorithms in the high-resolution remote sensing images. Finally, extensive experimental analyses and comparisons on the SSDD and the NWPU VHR-10 dataset show that (1) the HRFPN makes the predicted instance masks more accurate, which can effectively enhance the instance segmentation performance of the high-resolution remote sensing imagery; (2) the ISNetV2 is effective and promotes further improvements in mask prediction accuracy; (3) our proposed framework HQ-ISNet is effective and more accurate for instance segmentation in the remote sensing imagery than the existing algorithms.

<b><sc>Abstract.</sc></b> Deep learning has achieved great success in computer vision. However, most of the existing methods are designed and tested on public datasets without severe occlusions, which may lead to low recall or even algorithm failure in real-world animal industry where occlusions commonly exist. Therefore, whether deep learning methods are still applicable in scenes with frequently occurring occlusions remains to be verified. This study assessed the applicability and capacity limit of deep learning methods on scenarios of pigs in farrowing pens as a case study. We collected 1,100 images from a farrowing pen where farrowing crate was adopted, and 94.8% of images have at least one piglet under occlusion. Six state-of-the-art deep learning methods in three major computer vision tasks, including object detection (Faster RCNN and Yolov4), semantic segmentation (FCN and Unet), and instance segmentation (Mask RCNN and Yolact++) were evaluated, respectively. To further challenge these methods, we randomly and gradually added virtual occlusions in ten cases on the images and tested their capacity limitation under extreme occlusion. The result showed that all the six deep learning methods achieved satisfactory performance on the original baseline case while the performance fell when occlusion became heavier. Object detection methods were less affected by occlusion than instance segmentation methods due to their rougher output. Semantic segmentation method Unet demonstrated the least effect by occlusion due to its pixel-wise classification characters, while the instance segmentation method Mask RCNN was most affected by occlusion. We suggest further study on segmentation method in animal industry.

Depth hoar (DH) forms when there is a strong temperature gradient in the snowpack. In the Antarctic ice sheet (AIS), DH mostly forms during summer insolation. The seasonal regularity provides an age marker for each internal snow/firn layer, which is necessary for estimating surface mass balance (SMB) using ice-penetrating radar (IPR) and ice core. However, little is known about DH inside the AIS because ice drill and snow pit observations are inefficient and sparse. The deployment of the Operation IceBridge (OIB) airborne snow radar has significantly enhanced the field observation dataset. So far, the spatial distribution of DH remains unclear due to the lack of DH extraction over the AIS from OIB snow radar. Based on instance segmentation SOLOv2 and self-attention mechanism Swin Transformer, a DH layer automatic extraction algorithm ST-SOLOv2 is proposed, with AP50 of 0.9 and F1-score of 0.83, outperforms other commonly used instance segmentation networks (SOLOv2, Mask R-CNN, BlendMask, YOLACT, and CondInst). After the proposed preprocessing pipeline, we conduct the ST-SOLOv2 to locate each DH near the surface. Our results suggest DH number increases with elevation and decreases with slope angle from coast to inland. And DH number decreases with surface melting days as melting/freezing cycles obscure snow layering. We present a method for efficiently monitoring the DH distribution that can be used in further studies of snow radar SMB estimation.

Automatic coarse-to-fine joint detection and segmentation of underwater non-structural live crabs for precise feeding

We present a new, embarrassingly simple approach to instance segmentation. Compared to many other dense prediction tasks, e.g., semantic segmentation, it is the arbitrary number of instances that have made instance segmentation much more challenging. In order to predict a mask for each instance, mainstream approaches either follow the “detect-then-segment” strategy (e.g., Mask R-CNN), or predict embedding vectors first then use clustering techniques to group pixels into individual instances. We view the task of instance segmentation from a completely new perspective by introducing the notion of “instance categories”, which assigns categories to each pixel within an instance according to the instance’s location and size, thus nicely converting instance segmentation into a single-shot classification-solvable problem. We demonstrate a much simpler and flexible instance segmentation framework with strong performance, achieving on par accuracy with Mask R-CNN and outperforming recent single-shot instance segmenters in accuracy. We hope that this simple and strong framework can serve as a baseline for many instance-level recognition tasks besides instance segmentation. Code is available at https://git.io/AdelaiDet .

The current instance segmentation method can achieve satisfactory results in common scenarios. However, under the overlap or partial occlusion between targets caused by the complex scenes, accurate segmentation of pigs remains a challenging task. To address the problem, the authors propose an instance segmentation method based on Mask Scoring region‐based convolutional neural networks (R‐CNN) (MS R‐CNN), which creates the adversarial network called MaskDis in the head branch of MS R‐CNN. The MaskDis is trained as a discriminator using a generative adversarial network, and the MS R‐CNN model is used as a generator during model training. The adversarial training enables the generator to learn context information and features at the pixel level, which effectively improves the segmentation quality under pigs’ overlapping or dense occlusions scenes. Experimental conducted on the pig object segmentation dataset show that the proposed approach achieves a precision of 92.03%, a recall of 92.18%, and an F1 score of 0.9210. Compared with the basic MS R‐CNN model, the approach achieved a 2.25% improvement in precision and 1.18% improvement in F1 score. Furthermore, the improved approach outperformed advanced instance segmentation methods such as YOLACT, Swin Transformer, YOLOv5‐seg, and SOLOv2 on COCO evaluation metrics. These experimental results demonstrate the effectiveness of the proposed approach in instance segmentation of pigs in complex scenes, providing technical support for non‐contact pig automatic management.

Highlights Mask R-CNN ResNet-101 was analyzed to have the highest detection accuracy for grape berries. Grape cluster compactness is one of the important factors that determines the quality of grapes. Indicator DGC was used to intuitively calculate the compactness of grape cluster. An average error of 1.6 mm occurred with the verification of diameter estimation algorithm. Abstract. Consumption and cultivation of grapes are constantly increasing because they are known as alkaline foods, which relieve fatigue by accelerating carbohydrate metabolism in the human body. Among these grapes, a Shine Muscat doesn’t have the bitter taste of ordinary grapes, has low acidity, and has a crunchy texture, which results in growing interest in modern society. As the consumption of Shine Muscat increases, an automated process to save labor and time becomes essential. Particularly, one challenge that needs to be solved is the determination of grape quality based solely on visual evaluations by experts, which can result in inconsistent pricing for consumers. In this study, an algorithm was adopted to detect the grape berries from a bunch of single grapes by acquiring RGB images of the harvested Shine Muscat. Mask R-CNN, a convolutional neural network-based image segmentation technique, was used with various backbones to compare and evaluate the performance of each model. Results showed that Mask R-CNN ResNet 101 had the highest AP (Average Precision) value of 0.961 among all models. In addition, indexes such as the size (diameter) of each grape berry, the area of the grape on the image, and the area of the empty space that are required to find the compactness of the grape cluster are obtained. In particular, it was analyzed that the average error value (mm) and percent error (%) of the diameter estimation algorithm developed in this study were 1.60 mm and 4.79%, respectively. Keywords: Keywords., Density of Grape Cluster (DGC), Diameter estimation, Grape berry, Grape cluster compactness, Mask R-CNN, Object detection.

As a concise and classic framework for object detection and instance segmentation, Mask R-CNN achieves promising performance in both two tasks.However, considering stronger feature representation for Mask R-CNN fashion framework, there is room for improvement from two aspects. On the one hand, performing multi-task prediction needs more credible feature extraction and multi-scale features integration to handle objects with varied scales. In this paper, we address this problem by using a novel neck module called SA-FPN (Scale Aware Feature Pyramid Networks). With the enhanced feature representations, our model can accurately detect and segment the objects of multiple scales. On the other hand, in Mask R-CNN framework, isolation between parallel detection branch and instance segmentation branch exists, causing the gap between training and testing processes. To narrow this gap, we propose a unified head module named EJ-Head (Effective Joint Head) to combine two branches into one head, not only realizing the interaction between two tasks, but also enhancing the effectiveness of multi-task learning. Comprehensive experiments show that our proposed methods bring noticeable gains for object detection and instance segmentation. In particular, our model outperforms the original Mask R-CNN by 1~2 percent AP in both object detection and instance segmentation task on MS-COCO benchmark. Code will be available soon.

Building segmentation is of great importance in the task of remote sensing imagery interpretation. However, the existing semantic segmentation and instance segmentation methods often lead to segmentation masks with blurred boundaries. In this paper, we propose a novel instance segmentation network for building segmentation in high-resolution remote sensing images. More specifically, we consider segmenting an individual building as detecting several keypoints. The detected keypoints are subsequently reformulated as a closed polygon, which is the semantic boundary of the building. By doing so, the sharp boundary of the building could be preserved. Experiments are conducted on selected Aerial Imagery for Roof Segmentation (AIRS) dataset, and our method achieves better performance in both quantitative and qualitative results with comparison to the state-of-the-art methods. Our network is a bottom-up instance segmentation method that could well preserve geometric details.

The automated classification of fish, such as herring and salmon, in multi-frequency echograms is important for ecosystems monitoring. This paper implements a novel approach to instance segmentation: a hybrid of deep-learning and heuristic methods. This approach implements semantic segmentation by a U-Net to detect fish, which are converted to instances of fish-schools derived from candidate components within a defined linking distance. In addition to four frequency channels of echogram data (67.5, 125, 200, 455 kHz), two simulated channels (water depth and solar elevation angle) are included to encode spatial and temporal information, which leads to substantial improvement in model performance. The model is shown to out-perform recent experiments that have used a Mask R-CNN architecture. This approach demonstrates the ability to classify sparsely distributed objects in a way that is not possible with state-of-the-art instance segmentation methods.

Lotus seedpod maturity detection and segmentation in pond environments play a significant role in yield prediction and picking pose estimation for lotus seedpods. However, it is a great challenge to accurately detect and segment lotus seedpods due to insignificant phenotypic differences between the adjacent maturity, changing illumination, overlap, and occlusion of lotus seedpods. The existing research pays attention to lotus seedpod detection while ignoring maturity detection and segmentation problems. Therefore, a semantic segmentation dataset of lotus seedpods was created, where a copy-and-paste data augmentation tool was employed to eliminate the class-imbalanced problem and improve model generalization ability. Afterwards, an improved YOLOv8-seg model was proposed to detect and segment the maturity of lotus seedpods. In the model, the convolutional block attention module (CBAM) was embedded in the neck network to extract distinguished features of different maturity stages with negligible computation cost. Wise-Intersection over Union (WIoU) regression loss function was adopted to refine the regression inference bias and improve the bounding box prediction accuracy. The experimental results showed that the proposed YOLOv8-seg model provides an effective method for “ripe” and “overripe” lotus seedpod detection and instance segmentation, where the mean average precision of segmentation mask (mAPmask) reaches 97.4% and 98.6%, respectively. In addition, the improved YOLOv8-seg exhibits high robustness and adaptability to complex illumination in a challenging environment. Comparative experiments were conducted using the proposed YOLOv8-seg and other state-of-the-art instance segmentation methods. The results showed that the improved model is superior to the Mask R-CNN and YOLACT models, with recall, precision, mAPbox and mAPmask being 96.5%, 94.3%, 97.8%, and 98%, respectively. The average running time and weight size of the proposed model are 25.9 ms and 7.4 M, respectively. The proposed model obtained the highest mAP for lotus seedpod maturity detection and segmentation while maintaining an appropriate model size and speed. Furthermore, based on the obtained segmentation model, 3D visualization of the lotus pond scene is performed, and cloud point of lotus seedpods is generated, which provides a theoretical foundation for robot harvesting in the lotus pond.