SideGuide:A Large-scale Sidewalk Dataset for Guiding Impaired People

Weakly supervised instance segmentation using multi-prior fusion

Given the wide diffusion of deep neural network architectures for computer vision tasks, several new applications are nowadays more and more feasible. Among them, a particular attention has been recently given to instance segmentation, by exploiting the results achievable by two-stage networks (such as Mask R-CNN or Faster R-CNN), derived from R-CNN. In these complex architectures, a crucial role is played by the Region of Interest (RoI) extraction layer, devoted to extracting a coherent subset of features from a single Feature Pyramid Network (FPN) layer attached on top of a backbone. This paper is motivated by the need to overcome the limitations of existing RoI extractors which select only one (the best) layer from FPN. Our intuition is that all the layers of FPN retain useful information. Therefore, the proposed layer (called Generic RoI Extractor - GRoIE) introduces non-local building blocks and attention mechanisms to boost the performance. A comprehensive ablation study at component level is conducted to find the best set of algorithms and parameters for the GRoIE layer. Moreover, GRoIE can be integrated seamlessly with every two-stage architecture for both object detection and instance segmentation tasks. Therefore, the improvements brought about by the use of GRoIE in different state-of-the-art architectures are also evaluated. The proposed layer leads up to gain a 1.1% AP improvement on bounding box detection and 1.7% AP improvement on instance segmentation. The code is publicly available on GitHub repository at https://github.com/IMPLabUniPr/mmdetection/tree/groie_dev

As a concise and classic framework for object detection and instance segmentation, Mask R-CNN achieves promising performance in both two tasks.However, considering stronger feature representation for Mask R-CNN fashion framework, there is room for improvement from two aspects. On the one hand, performing multi-task prediction needs more credible feature extraction and multi-scale features integration to handle objects with varied scales. In this paper, we address this problem by using a novel neck module called SA-FPN (Scale Aware Feature Pyramid Networks). With the enhanced feature representations, our model can accurately detect and segment the objects of multiple scales. On the other hand, in Mask R-CNN framework, isolation between parallel detection branch and instance segmentation branch exists, causing the gap between training and testing processes. To narrow this gap, we propose a unified head module named EJ-Head (Effective Joint Head) to combine two branches into one head, not only realizing the interaction between two tasks, but also enhancing the effectiveness of multi-task learning. Comprehensive experiments show that our proposed methods bring noticeable gains for object detection and instance segmentation. In particular, our model outperforms the original Mask R-CNN by 1~2 percent AP in both object detection and instance segmentation task on MS-COCO benchmark. Code will be available soon.

Instance segmentation needs to locate all instances in an image correctly and segment each instance precisely. Currently, the most dominant methods for instance segmentation take object detection as a pre-task. However, they rely on the accuracy of object detection incredibly. If the pre-task cannot predict an accurate bounding box, the performance of instance segmentation will degenerate. In this paper, we present a novel method for instance segmentation to solve this problem, which is called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</b> egmenting <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</b> eyond the <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</b> ounding <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</b> ox ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S3B-Net</b> ). Our S3B-Net designs a sub-network to help instance segmentation methods based on object detection to segment the part of an instance beyond the bounding box. Specifically, the sub-network first predicts a two-dimensional pixel embedding for each pixel. Then, the Gaussian function is employed to calculate a pixel’s probability belongs to a corresponding instance according to the two-dimensional pixel embedding. Finally, the output of the sub-network combines with the output of instance segmentation based on object detection to generate a more precise instance mask. Our sub-network can easily extend on the existing instance segmentation method based on object detection to segment instance beyond the bounding box. We do our experiments on dominant instance segmentation datasets, such as the COCO dataset and Cityscapes dataset. The results show that our method can achieve 6.8 points gain compared with the baseline Mask R-CNN with ResNet-50-FPN in Cityscapes datasets, and 1.7 points gain with ResNet-101-FPN-DCN in COCO datasets. Our S3B-Net outperforms the previous state-of-the-art instance segmentation method, which proves our method is competitive. The source code of our method will be made available.

Welding is one of the most common machining methods in the industrial field, and weld grinding is a key task in the industrial manufacturing process. Although several weld-image datasets exist, most provide only coarse annotations and have limited scale and diversity. To address this gap, we constructed INWELD, a comprehensive multi-category weld dataset captured under real-world production conditions, providing both single-label and multi-label annotations. The dataset covers various types of welds and is evenly divided according to production needs. The proposed multi-category annotation method can predict the weld geometry and welding method without additional calculation and is applied to object detection and instance segmentation tasks. To evaluate the applicability of this dataset, we utilized the mainstream algorithms CenterNet and YOLOv7 for object detection, as well as Mask R-CNN, Deep Snake, and YOLACT for instance segmentation. The experimental results show that in single-category annotation, the AP50 of CenterNet and YOLOv7 is close to 90%, and the AP50 of Mask R-CNN and Deep Snake is greater than 80%. In multi-category annotation, the AP50 of CenterNet and YOLOv7 is greater than 80%, and the AP50 of Deep Snake and YOLACT is nearly 70%. The INWELD dataset constructed in this paper fills the gap in industrial weld surface images, lays the theoretical foundation for the intelligent research of welds, and provides data support and research direction for the development of automatic grinding and polishing of welds.

Weakly supervised semantic instance segmentation with only image-level supervision, instead of relying on expensive pixel-wise masks or bounding box annotations, is an important problem to alleviate the data-hungry nature of deep learning. In this article, we tackle this challenging problem by aggregating the image-level information of all training images into a large knowledge graph and exploiting semantic relationships from this graph. Specifically, our effort starts with some generic segment-based object proposals (SOP) without category priors. We propose a multiple instance learning (MIL) framework, which can be trained in an end-to-end manner using training images with image-level labels. For each proposal, this MIL framework can simultaneously compute probability distributions and category-aware semantic features, with which we can formulate a large undirected graph. The category of background is also included in this graph to remove the massive noisy object proposals. An optimal multi-way cut of this graph can thus assign a reliable category label to each proposal. The denoised SOP with assigned category labels can be viewed as pseudo instance segmentation of training images, which are used to train fully supervised models. The proposed approach achieves state-of-the-art performance for both weakly supervised instance segmentation and semantic segmentation. The code is available at https://github.com/yun-liu/LIID.

Autonomous mobile robots for industrial inspection can reduce cost for digitalization of existing plants by performing autonomous routine inspections. A frequent task is reading of analog gauges to monitor the health of the facility. Automating this process involves capturing image data with a camera sensor and processing the data to read the value. Detection algorithms deployed on a mobile robot have to deal with increased uncertainty regarding localization and environmental influences. This imposes increased requirements regarding robustness to viewing angle, lighting and scale variation on detection and reading. Current approaches based on conventional computer vision require high quality images or prior knowledge. We address these limitations by leveraging the advances of neural networks in the task of object detection and instance segmentation in a two-stage pipeline. Our method robustly detects and reads manometers without prior knowledge of object location or exact object type. In our evaluation we show that our approach can detect and read manometers from a distance of up to 3m and a viewing angle of up to 60° in different lighting conditions with needle angle estimation errors of ±2.2°. We publish the validation split of our training dataset for manometer and needle detection at https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/2881.

Panoptic Segmentation aims to provide an understanding of background (stuff) and instances of objects (things) at a pixel level. It combines the separate tasks of semantic segmentation (pixel level classification) and instance segmentation to build a single unified scene understanding task. Typically, panoptic segmentation is derived by combining semantic and instance segmentation tasks that are learned separately or jointly (multi-task networks). In general, instance segmentation networks are built by adding a foreground mask estimation layer on top of object detectors or using instance clustering methods that assign a pixel to an instance center. In this work, we present a fully convolution neural network that learns instance segmentation from semantic segmentation and instance contours (boundaries of things). Instance contours along with semantic segmentation yield a boundary aware semantic segmentation of things. Connected component labeling on these results produces instance segmentation. We merge semantic and instance segmentation results to output panoptic segmentation. We evaluate our proposed method on the CityScapes dataset to demonstrate qualitative and quantitative performances along with several ablation studies. Our overview video can be accessed from https://youtu.be/wBtcxRhG3e0.

For few-shot object detection has attracted wide attention. Plateau wildlife detection is a typical problem of few-shot object detection. The solution of this problem can reduce the manpower and material resources of collecting and marking large-scale wildlife data sets. Deep neural network can significantly improve the efficiency of wildlife detection, and the key step is to obtain a large amount of data with bounding box annotations. However, there are few plateau wildlife pictures with labeling, which is insufficient to support the training of deep neural network. In this paper, we propose a target detection method based on few-shot learning to detect plateau wildlife images, where each animal category has only a small number of annotated bounding boxes (no more than 10). we use two-stage training method to solve the task of small samples detection task. Based on the two-stage target detector Faster R-CNN, we established a few shot target detection model. The training is divided into two stages. The first stage is to train the model on the basic set, and the second stage is to adjust the balance set of the new class and the base class on the last layer. In order to further improve the detection effect, the two-stage training adopts data enhancement methods to expand the sample data. Experimental results show that this model can achieve better detection effect on plateau wildlife pictures with a small number of labeled samples.

Attention mechanisms have been widely used to address the common multi-scale challenges in remote sensing imagery for tasks such as object detection and instance segmentation. However, they often require the design of task-specific network architectures tailored to different data targets. To better accommodate objects of varying scales and downstream task requirements, we propose a dynamic search framework based on a Mixture-of-Experts model. In this framework, each module is treated as an expert, and their combinations are flexibly adjusted to ensure that the pyramid structure adapts to the needs of different-scale tasks. Specifically, we design a Scalable Bi-Directional Feature Pyramid Network (SBFPN), which incorporates various hybrid attention mechanisms to dynamically enhance feature fusion across different layers. This approach not only captures long-range dependencies within the image but also suppresses noise and interference from complex backgrounds. The scalable feature pyramid structure we construct utilizes sparse feature fusion within the same layers, while also computing self-attention weights between nodes and retaining or trimming skip connections across layers. We conduct experiments on object detection, instance segmentation, and panoptic segmentation tasks. This structure can be transferred to YOLO-based and R-CNN-based networks, achieving superior detection and segmentation performance. On the Airbus Ship dataset, the mAP for detection and segmentation increased from 71.3% and 62.4% to 82.7% and 71.1%, respectively, demonstrating the effectiveness of our proposed method. Code is available at https://github.com/chaibosong/SAFPN.

Instance segmentation in high-resolution (HR) remote sensing imagery is one of the most challenging tasks and is more difficult than object detection and semantic segmentation tasks. It aims to predict class labels and pixel-wise instance masks to locate instances in an image. However, there are rare methods currently suitable for instance segmentation in the HR remote sensing images. Meanwhile, it is more difficult to implement instance segmentation due to the complex background of remote sensing images. In this article, a novel instance segmentation approach of HR remote sensing imagery based on Cascade Mask R-CNN is proposed, which is called a high-quality instance segmentation network (HQ-ISNet). In this scheme, the HQ-ISNet exploits a HR feature pyramid network (HRFPN) to fully utilize multi-level feature maps and maintain HR feature maps for remote sensing images’ instance segmentation. Next, to refine mask information flow between mask branches, the instance segmentation network version 2 (ISNetV2) is proposed to promote further improvements in mask prediction accuracy. Then, we construct a new, more challenging dataset based on the synthetic aperture radar (SAR) ship detection dataset (SSDD) and the Northwestern Polytechnical University very-high-resolution 10-class geospatial object detection dataset (NWPU VHR-10) for remote sensing images instance segmentation which can be used as a benchmark for evaluating instance segmentation algorithms in the high-resolution remote sensing images. Finally, extensive experimental analyses and comparisons on the SSDD and the NWPU VHR-10 dataset show that (1) the HRFPN makes the predicted instance masks more accurate, which can effectively enhance the instance segmentation performance of the high-resolution remote sensing imagery; (2) the ISNetV2 is effective and promotes further improvements in mask prediction accuracy; (3) our proposed framework HQ-ISNet is effective and more accurate for instance segmentation in the remote sensing imagery than the existing algorithms.

&amp;lt;p&amp;gt;Nowadays, the exploitation artificial intelligence approached is overwhelming in several domains. As well, geomatic data are becoming more and more complex and heterogeneous, as they are collected with multi-source data collection techniques. Remotely sensed data, point cloud, thermal images and more are just few examples of complex data which requires growing computational capabilities, but, foremost, powerful tools of processing and interpretation.&amp;lt;/p&amp;gt;&amp;lt;p&amp;gt;The applications of modern AI-based algorithms for the processing of geomatics data offer opportunities that wouldn't be affordable up to few years ago. For geospatial domains, fundamental questions include how AI can be specifically applied to or has to be specifically created for geomatics data. This change is also having a significant impact on geospatial data. Machine Learning (ML) has been a core component of spatial analysis for classification, clustering, and prediction. In addition, Deep Learning (DL) is being integrated to automatically extract useful information with the task of classification, object detection, semantic and instance segmentation, etc. The integration of AI, ML, and DL in geomatics has developed into the concept of Geospatial Artificial Intelligence (GeoAI) that is a new paradigm for geographic knowledge discovery and beyond.&amp;amp;#160;&amp;lt;/p&amp;gt;&amp;lt;p&amp;gt;This talk aims at giving a sight on the emergent discipline called GeoAI, a novel research field in which cutting edge learning based methods are applied to enhance the knowledge and improve the ability of humans to manage complex information. Beside providing a picture of the latest achievements in the filed (outlining AI-based techniques for the analysis and the interpretation of complex geomatics data), this lecture will provide several examples of researches and applications, demonstrating opportunities, challenges and limitations with practical examples.&amp;amp;#160;&amp;lt;/p&amp;gt;&amp;lt;p&amp;gt;Bearing in mind that, for the upcoming future, the &amp;quot;man on the loop&amp;quot; will always assess unpredictable outcomes from the automatization process, it will be demonstrated, at different scales of representation and facing research challenges in different domains (e.g. environmental challenges, forestry, cultural heritage, tourism just to mention some), AI outperforms manual operations in terms of both cost effectiveness and reliability.&amp;amp;#160;&amp;lt;/p&amp;gt;

THINGS-data reflects three large-scale neuroimaging and behavioral datasets of object processing in humans, comprising densely sampled functional MRI and magnetoencephalographic recordings, as well as 4.70 million similarity judgments in response to thousands of photographic images for up to 1854 objects.

LPCL: Localized prominence contrastive learning for self-supervised dense visual pre-training

In pathology images, different stains highlight different glomerular structures, so a supervised deep learning-based glomerular instance segmentation model trained on individual stains performs poorly on other stains. However, it is difficult to obtain a training set with multiple stains because the labeling of pathology images is very time-consuming and tedious. Therefore, in this paper, we proposed an unsupervised stain augmentation-based method for segmentation of glomerular instances. In this study, we successfully realized the conversion between different staining methods such as PAS, MT and PASM by contrastive unpaired translation (CUT), thus improving the staining diversity of the training set. Moreover, we replaced the backbone of mask R-CNN with swin transformer to further improve the efficiency of feature extraction and thus achieve better performance in instance segmentation task. To validate the method presented in this paper, we constructed a dataset from 216 WSIs of the three stains in this study. After conducting in-depth experiments, we verified that the instance segmentation method based on stain augmentation outperforms existing methods across all metrics for PAS, PASM, and MT stains. Furthermore, ablation experiments are performed in this paper to further demonstrate the effectiveness of the proposed module. This study successfully demonstrated the potential of unsupervised stain augmentation to improve glomerular segmentation in pathology analysis. Future research could extend this approach to other complex segmentation tasks in the pathology image domain to further explore the potential of applying stain augmentation techniques in different domains of pathology image analysis.