LPCL: Localized prominence contrastive learning for self-supervised dense visual pre-training

Instance segmentation in high-resolution (HR) remote sensing imagery is one of the most challenging tasks and is more difficult than object detection and semantic segmentation tasks. It aims to predict class labels and pixel-wise instance masks to locate instances in an image. However, there are rare methods currently suitable for instance segmentation in the HR remote sensing images. Meanwhile, it is more difficult to implement instance segmentation due to the complex background of remote sensing images. In this article, a novel instance segmentation approach of HR remote sensing imagery based on Cascade Mask R-CNN is proposed, which is called a high-quality instance segmentation network (HQ-ISNet). In this scheme, the HQ-ISNet exploits a HR feature pyramid network (HRFPN) to fully utilize multi-level feature maps and maintain HR feature maps for remote sensing images’ instance segmentation. Next, to refine mask information flow between mask branches, the instance segmentation network version 2 (ISNetV2) is proposed to promote further improvements in mask prediction accuracy. Then, we construct a new, more challenging dataset based on the synthetic aperture radar (SAR) ship detection dataset (SSDD) and the Northwestern Polytechnical University very-high-resolution 10-class geospatial object detection dataset (NWPU VHR-10) for remote sensing images instance segmentation which can be used as a benchmark for evaluating instance segmentation algorithms in the high-resolution remote sensing images. Finally, extensive experimental analyses and comparisons on the SSDD and the NWPU VHR-10 dataset show that (1) the HRFPN makes the predicted instance masks more accurate, which can effectively enhance the instance segmentation performance of the high-resolution remote sensing imagery; (2) the ISNetV2 is effective and promotes further improvements in mask prediction accuracy; (3) our proposed framework HQ-ISNet is effective and more accurate for instance segmentation in the remote sensing imagery than the existing algorithms.

<p>Nowadays, the exploitation artificial intelligence approached is overwhelming in several domains. As well, geomatic data are becoming more and more complex and heterogeneous, as they are collected with multi-source data collection techniques. Remotely sensed data, point cloud, thermal images and more are just few examples of complex data which requires growing computational capabilities, but, foremost, powerful tools of processing and interpretation.</p><p>The applications of modern AI-based algorithms for the processing of geomatics data offer opportunities that wouldn't be affordable up to few years ago. For geospatial domains, fundamental questions include how AI can be specifically applied to or has to be specifically created for geomatics data. This change is also having a significant impact on geospatial data. Machine Learning (ML) has been a core component of spatial analysis for classification, clustering, and prediction. In addition, Deep Learning (DL) is being integrated to automatically extract useful information with the task of classification, object detection, semantic and instance segmentation, etc. The integration of AI, ML, and DL in geomatics has developed into the concept of Geospatial Artificial Intelligence (GeoAI) that is a new paradigm for geographic knowledge discovery and beyond. </p><p>This talk aims at giving a sight on the emergent discipline called GeoAI, a novel research field in which cutting edge learning based methods are applied to enhance the knowledge and improve the ability of humans to manage complex information. Beside providing a picture of the latest achievements in the filed (outlining AI-based techniques for the analysis and the interpretation of complex geomatics data), this lecture will provide several examples of researches and applications, demonstrating opportunities, challenges and limitations with practical examples. </p><p>Bearing in mind that, for the upcoming future, the "man on the loop" will always assess unpredictable outcomes from the automatization process, it will be demonstrated, at different scales of representation and facing research challenges in different domains (e.g. environmental challenges, forestry, cultural heritage, tourism just to mention some), AI outperforms manual operations in terms of both cost effectiveness and reliability. </p>

We present a new approach to instill 4D dynamic object priors into learned 3D representations by unsupervised pre-training. We observe that dynamic movement of an object through an environment provides important cues about its objectness, and thus propose to imbue learned 3D representations with such dynamic understanding, that can then be effectively transferred to improved performance in downstream 3D semantic scene understanding tasks. We propose a new data augmentation scheme leveraging synthetic 3D shapes moving in static 3D environments, and employ contrastive learning under 3D-4D constraints that encode 4D invariances into the learned 3D representations. Experiments demonstrate that our unsupervised representation learning results in improvement in downstream 3D semantic segmentation, object detection, and instance segmentation tasks, and moreover, notably improves performance in data-scarce scenarios. Our results show that our 4D pre-training method improves downstream tasks such as object detection mAP@0.5 by 5.5%/6.5% over training from scratch on ScanNet/SUN RGB-D while involving no additional run-time overhead at test time.

As a concise and classic framework for object detection and instance segmentation, Mask R-CNN achieves promising performance in both two tasks.However, considering stronger feature representation for Mask R-CNN fashion framework, there is room for improvement from two aspects. On the one hand, performing multi-task prediction needs more credible feature extraction and multi-scale features integration to handle objects with varied scales. In this paper, we address this problem by using a novel neck module called SA-FPN (Scale Aware Feature Pyramid Networks). With the enhanced feature representations, our model can accurately detect and segment the objects of multiple scales. On the other hand, in Mask R-CNN framework, isolation between parallel detection branch and instance segmentation branch exists, causing the gap between training and testing processes. To narrow this gap, we propose a unified head module named EJ-Head (Effective Joint Head) to combine two branches into one head, not only realizing the interaction between two tasks, but also enhancing the effectiveness of multi-task learning. Comprehensive experiments show that our proposed methods bring noticeable gains for object detection and instance segmentation. In particular, our model outperforms the original Mask R-CNN by 1~2 percent AP in both object detection and instance segmentation task on MS-COCO benchmark. Code will be available soon.

Given the wide diffusion of deep neural network architectures for computer vision tasks, several new applications are nowadays more and more feasible. Among them, a particular attention has been recently given to instance segmentation, by exploiting the results achievable by two-stage networks (such as Mask R-CNN or Faster R-CNN), derived from R-CNN. In these complex architectures, a crucial role is played by the Region of Interest (RoI) extraction layer, devoted to extracting a coherent subset of features from a single Feature Pyramid Network (FPN) layer attached on top of a backbone. This paper is motivated by the need to overcome the limitations of existing RoI extractors which select only one (the best) layer from FPN. Our intuition is that all the layers of FPN retain useful information. Therefore, the proposed layer (called Generic RoI Extractor - GRoIE) introduces non-local building blocks and attention mechanisms to boost the performance. A comprehensive ablation study at component level is conducted to find the best set of algorithms and parameters for the GRoIE layer. Moreover, GRoIE can be integrated seamlessly with every two-stage architecture for both object detection and instance segmentation tasks. Therefore, the improvements brought about by the use of GRoIE in different state-of-the-art architectures are also evaluated. The proposed layer leads up to gain a 1.1% AP improvement on bounding box detection and 1.7% AP improvement on instance segmentation. The code is publicly available on GitHub repository at https://github.com/IMPLabUniPr/mmdetection/tree/groie_dev

Generally, most approaches using methods such as cropping, rotating, and flipping achieve more data to train models for improving the accuracy of detection and segmentation. However, due to the difficulties of labeling such data especially semantic segmentation data, those traditional data augmentation methodologies cannot help a lot when the training set is really limited. In this paper, a model named OFA-Net (One For All Network) is proposed to combine object detection and semantic segmentation tasks. Meanwhile, using a strategy called “1-N Alternation” to train the OFA-Net model, which can make a fusion of features from detection and segmentation data. The results show that object detection data can be recruited to better the segmentation accuracy performance, and furthermore, segmentation data assist a lot to enhance the confidence of predictions for object detection. Finally, the OFA-Net model is trained without traditional data augmentation methodologies and tested on the KITTI test server. The model works well on the KITTI Road Segmentation challenge and can do a good job on the object detection task.

Welding is one of the most common machining methods in the industrial field, and weld grinding is a key task in the industrial manufacturing process. Although several weld-image datasets exist, most provide only coarse annotations and have limited scale and diversity. To address this gap, we constructed INWELD, a comprehensive multi-category weld dataset captured under real-world production conditions, providing both single-label and multi-label annotations. The dataset covers various types of welds and is evenly divided according to production needs. The proposed multi-category annotation method can predict the weld geometry and welding method without additional calculation and is applied to object detection and instance segmentation tasks. To evaluate the applicability of this dataset, we utilized the mainstream algorithms CenterNet and YOLOv7 for object detection, as well as Mask R-CNN, Deep Snake, and YOLACT for instance segmentation. The experimental results show that in single-category annotation, the AP50 of CenterNet and YOLOv7 is close to 90%, and the AP50 of Mask R-CNN and Deep Snake is greater than 80%. In multi-category annotation, the AP50 of CenterNet and YOLOv7 is greater than 80%, and the AP50 of Deep Snake and YOLACT is nearly 70%. The INWELD dataset constructed in this paper fills the gap in industrial weld surface images, lays the theoretical foundation for the intelligent research of welds, and provides data support and research direction for the development of automatic grinding and polishing of welds.

Attention mechanisms have been widely used to address the common multi-scale challenges in remote sensing imagery for tasks such as object detection and instance segmentation. However, they often require the design of task-specific network architectures tailored to different data targets. To better accommodate objects of varying scales and downstream task requirements, we propose a dynamic search framework based on a Mixture-of-Experts model. In this framework, each module is treated as an expert, and their combinations are flexibly adjusted to ensure that the pyramid structure adapts to the needs of different-scale tasks. Specifically, we design a Scalable Bi-Directional Feature Pyramid Network (SBFPN), which incorporates various hybrid attention mechanisms to dynamically enhance feature fusion across different layers. This approach not only captures long-range dependencies within the image but also suppresses noise and interference from complex backgrounds. The scalable feature pyramid structure we construct utilizes sparse feature fusion within the same layers, while also computing self-attention weights between nodes and retaining or trimming skip connections across layers. We conduct experiments on object detection, instance segmentation, and panoptic segmentation tasks. This structure can be transferred to YOLO-based and R-CNN-based networks, achieving superior detection and segmentation performance. On the Airbus Ship dataset, the mAP for detection and segmentation increased from 71.3% and 62.4% to 82.7% and 71.1%, respectively, demonstrating the effectiveness of our proposed method. Code is available at https://github.com/chaibosong/SAFPN.

Panoptic Segmentation aims to provide an understanding of background (stuff) and instances of objects (things) at a pixel level. It combines the separate tasks of semantic segmentation (pixel level classification) and instance segmentation to build a single unified scene understanding task. Typically, panoptic segmentation is derived by combining semantic and instance segmentation tasks that are learned separately or jointly (multi-task networks). In general, instance segmentation networks are built by adding a foreground mask estimation layer on top of object detectors or using instance clustering methods that assign a pixel to an instance center. In this work, we present a fully convolution neural network that learns instance segmentation from semantic segmentation and instance contours (boundaries of things). Instance contours along with semantic segmentation yield a boundary aware semantic segmentation of things. Connected component labeling on these results produces instance segmentation. We merge semantic and instance segmentation results to output panoptic segmentation. We evaluate our proposed method on the CityScapes dataset to demonstrate qualitative and quantitative performances along with several ablation studies. Our overview video can be accessed from https://youtu.be/wBtcxRhG3e0.

Deep learning models have revealed outstanding performance on image classification and object detection tasks. However, there is a crucial drop in performance when they are subject to learn new data incrementally in the absence of previous training data. They suffer from catastrophic forgetting—abrupt drop in performance. This phenomenon affects the implementation of artificial intelligence in practical scenarios. To overcome catastrophic forgetting, the previous method has either saved previous data in memory or generated the previous data. However, these methods are computationally complex and infeasible for real-time applications. In this article, we proposed the YOLOv3 as an object detection framework for incremental learning. A knowledge distillation loss is introduced for the prediction of previously learned knowledge without utilizing previous training data. Consequently, these predictions are updated while learning the current model. Experimental results on the Pascal VOC2007 indicate that the proposed method significantly improved the mean average precision up to 74% for two classes in comparison to the state-of-the-art methods.

Object region proposal has a wide application on various domains, such as object detection, object tracking, robot navigation, and anomaly detection. Widely used region proposal methods are based on either grouping superpixels or sliding windows. Previous studies have been done by grouping either convolution features, superpixels, or windows. Moreover, histogram of gradients (HoGs) of lower-level features (pixels) can be used for the region proposal task. In this study, both the concepts of HoGs and convolution features are used for proposing object(s) region(s) in an image. Region proposal is done by HoGs of convolution features and statistical application on the gradient data of different feature maps. First, a sized image is passed through one convolution layer to generate reduced feature maps. Then, a window slides over each feature map and HoG is applied over each window. Thereafter, the histogram is analysed using the concept of Coefficient of Variation (CV) and angle (slope) applied on histogram data values. A window is considered as the object region if the CV and angle of its histogram data exceed some thresholds. For all feature maps, we have obtained several widows that represent approximated object regions. These windows are grouped to obtain the possible region(s) proposal. The effectiveness of the proposed method is demonstrated over one of the benchmark datasets, called `PASCAL VOC'. Our method is found superior to two other state-of-the-art, namely region-based segmentation (RBS), and combining efficient object localization and image classification (CEOLIC) in terms of recall.

Segmenting or detecting objects in sparse Lidar point clouds are two important tasks in autonomous driving to allow a vehicle to act safely in its 3D environment. The best performing methods in 3D semantic segmentation or object detection rely on a large amount of annotated data. Yet annotating 3D Lidar data for these tasks is tedious and costly. In this context, we propose a self-supervised pretraining method for 3D perception models that is tailored to autonomous driving data. Specifically, we leverage the availability of synchronized and calibrated image and Lidar sensors in autonomous driving setups for distilling self-supervised pre-trained image representations into 3D models. Hence, our method does not require any point cloud nor image annotations. The keyingredient of our method is the use of superpixels which are used to pool 3D point features and 2D pixel features in visually similar regions. We then train a 3D network on the self-supervised task of matching these pooled point features with the corresponding pooled image pixel features. The advantages of contrasting regions obtained by superpixels are that: (1) grouping together pixels and points of visually coherent regions leads to a more meaningful contrastive task that produces features well adapted to 3D semantic segmentation and 3D object detection; (2) all the different regions have the same weight in the contrastive loss regardless of the number of 3D points sampled in these regions; (3) it mitigates the noise produced by incorrect matching of points and pixels due to occlusions between the different sensors. Extensive experiments on autonomous driving datasets demonstrate the ability of our image-to-Lidar distillation strategy to produce 3D representations that transfer well on semantic segmentation and object detection tasks.

This paper proposes a learnable line encoding technique for bounding boxes commonly used in the object detection task. A bounding box is simply encoded using two main points: the top-left corner and the bottom-right corner of the bounding box; then, a lightweight convolutional neural network (CNN) is employed to learn the lines and propose high-resolution line masks for each category of classes using a pixel-shuffle operation. Post-processing is applied to the predicted line masks to filtrate them and estimate clear lines based on a progressive probabilistic Hough transform. The proposed method was trained and evaluated on two common object detection benchmarks: Pascal VOC2007 and MS-COCO2017. The proposed model attains high mean average precision (mAP) values (78.8% for VOC2007 and 48.1% for COCO2017) while processing each frame in a few milliseconds (37 ms for PASCAL VOC and 47 ms for COCO). The strength of the proposed method lies in its simplicity and ease of implementation unlike the recent state-of-the-art methods in object detection, which include complex processing pipelines.

Advanced Driver Assistance System (ADAS) is a very important part of an up to date vehicle. For achieving highlevel objectives in such ADAS functionality like LKA (lane keeping assistance), LDW (lane departure warning system), FCW (forward collision warning) the quality of the algorithms under the hood must be extremely high. In the last few years, it is common that these algorithms are based on DNNs (deep neural networks) applied to the tasks of semantic and instance segmentation, 2D/3D object detection and visual object tracking. Recent state-of-the-art DNN models as usual solve only one single task from the listed above and running several neural networks is rather computationally expensive and even impossible due to the lack of the GPU memory. One of the approaches used to overcome such a problem is a shared backbone (also called feature extractor or encoder). The backbone consumes most of the computing resources thus the model with a shared backbone achieves better inference performance. Unfortunately, the training procedure for a shared backbone model has several difficulties. The first one is the lack of datasets with all the required and uniform annotation types. The second problem is a more sophisticated backpropagation procedure. In this paper, we consider several methods for multi-task neural network training and present the results of such training procedures on several public datasets with dissimilar annotation types. The shared backbone is applied to the following three tasks performed simultaneously on the road scene: semantic segmentation, 2D object detection and 3D object detection. While the performance of the DNNs with shared backbone increased significantly, we obtained the quality evaluation results, which are quite close to the original separate state-of-the-art DNNs and even outperforms them in some evaluation indices.

Having profound implications for autonomous systems, intelligent surveillance, and human-computer interaction, object detection and classification of images are one of the principal challenges in computer vision. Using the vast benchmark datasets of PASCAL VOC and ImageNet, this study introduces a deep learning system for object recognition and classification. The model employs a hybrid structure combining Vision Transformers (ViT) and ResNet-50 for extracting global context and spatial features. On the more diversified and complex PASCAL VOC dataset, the pre-trained weights of ImageNet are transferred for the purpose of generalization and speed of convergence. By thorough testing on PASCAL VOC 2007 and 2012 datasets, the model outperformed baseline models like Faster R-CNN and YOLOv5 by 2.7% and 1.9%, respectively, in terms of mean Average Precision (mAP) at 83.4% on VOC 2007 and 82.1% on VOC 2012. With 78.9% top-1 classification accuracy, ImageNet-based training significantly improved the system discriminative ability for 1,000 object categories. Greater robustness to occlusion and class imbalance were achieved with increased data augmentation, optimized focus loss, and improved attention modules.