CGMGM: A Cross-Gaussian Mixture Generative Model for Few-Shot Semantic Segmentation

Few-shot semantic segmentation, aiming to segment query images with a few annotated support samples, has drawn increasing attention. Most existing few-shot methods leverage the single prototype obtained from global average pooling to represent all support information and further use the extracted prototype to segment the query images in a matching manner. Although promising results for natural images have been reported, these methods cannot be directly applied on aerial images. The main reason comes from that the extracted single support prototype can only provide a coarse guidance for matching between query and support images and could not handle the large variance of objects’ appearances and scales. To deal with these challenges on aerial images, we propose a scale-aware few-shot semantic segmentation network to perform detailed matching with multiple prototypes. More specifically, the detailed matching module is first constructed to compute the pixel-level similarity between the query features and the extracted multiple support prototypes for providing more accurate parsing guidance. Subsequently, to address the problem of scale imbalance, the scale-aware focal loss is designed to dynamically down-weight the loss assigned to large well-parsed objects and focus training on tiny hard-parsed objects. To facilitate the reproducible research on the task of few-shot semantic segmentation in aerial images, we further provide a few-shot segmentation benchmark iSAID- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5^{\mathrm {i}}$ </tex-math></inline-formula> constructed from the large-scale iSAID dataset <xref ref-type="bibr" rid="ref1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[1]</xref> . Comprehensive experiments and comparisons with the state-of-the-art few-shot segmentation methods on the iSAID- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5^{\mathrm {i}}$ </tex-math></inline-formula> dataset clearly demonstrate the superiority of our proposed method. The code and dataset are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/caoql98/SDM</uri> .

Detection of articulated objects in images, including location and state, is an important and challenging task in many object tracking applications. Image edges have proven to be a key feature, although their quality is influenced by many factors. In this paper, we propose a novel edge gradient-based template matching method for object detection. In contrast to other methods, ours does not perform any binarization or discretization during the online matching. This is facilitated by a new continuous edge gradient similarity measure. Its main components are a novel edge gradient operator, which is applied to query and template images, and the formulation as a convolution, which can be computed very efficiently in Fourier space. Our method consists of a preprocessing stage for the template images, a simple preprocessing of the query image, and our similarity measure between template and query image, which yields a confidence map for the query image. The resulting confidence map can be used for the final object localization. We compared our method to a state-of-the-art chamfer-based matching method. The results demonstrate that our method is much more robust against weak edge response and yields much better confidence maps with fewer maxima that are also more significant. In addition, our method lends itself well to efficient implementation on GPUs: at a query image resolution of 320× 256 and a template resolution of 80× 80 we can generate about 330 confidence maps per second.

Few-shot segmentation(FSS), which aims to extract never learned classes of objects from query images with a few annotated support samples, is a challenging problem especially in the cases that the appearance of objects in the support and the query images is significant different. Therefore, we propose a deep network called Pyramid Co-Attention Compare Network (PCCNet) to narrow the gap between them by introducing a Pyramid Co-attention Module (PCAM). PCAM acts as a task-specific transformer to transform the features of corresponding objects in query and support images into a space in which they are much closer by taking advantage of the underlying relation between query and support images. We also introduce a Prototypical Guide Module (PGM) which uses non-parametric metric learning to guide parametric metric learning so as to combine the advantages of them. In addition, a Superpixel Refine Module(SRM) is proposed to optimize the final output segmentation masks. Experiments conducted on Pascal-5i shows that our PCCNet achieves a mean Intersection-over-Union(mIoU) score of 63.01% for 1-shot segmentation and 64.57% for 5-shot segmentation, outperforming state-of-the-art methods by margin of 2.2% and 1.6%, respectively.

In this study, we propose a segmentation model based on convolutional neural networks (CNNs) to address image segmentation challenges in computer vision. Prior to designing the model, the activation function and other modules of the convolutional neural network were optimized to meet specific requirements. The segmentation task was transformed into binary classification problem to simplify network calculations and improve efficiency. Additionally, the model utilized a mask map obtained from the semantic segmentation model to aid in instance segmentation. Class activation technology was introduced to extract feature mapping maps. The corresponding thermal maps were obtained to achieve target instance segmentation. To further validate the effectiveness of the segmentation model, simulation experiments were conducted on semantic segmentation and instance segmentation respectively. The results show that the accuracy of the basic semantic segmentation model reached 87.58%, while the average accuracy of the entire class of the optimized instance segmentation model reached 97.9%. Therefore, the research and design of image segmentation models demonstrate high accuracy and good robustness.

Bag of Features (BoF) has been successfully exploited in content-based image retrieval for several years. Due to its performance and popularity, several extensions have been proposed that involve feature description, dictionary building, feature encoding and post-query process, etc. This paper proposes a dynamically weighting scheme for BoF-based image retrieval based on feedback. It involves two contributions: (i) analyzing the statistical distribution characteristic of similar BoF representations and (ii) computing weights dynamically based on the feedback obtained from different initial query results. We quantitatively evaluate the proposed method on two different databases. Experiments confirm that the proposed weighting scheme has better performance than the baseline of BoF-based image retrieval systems. Meanwhile, the results demonstrate the effectiveness of the weighting scheme in terms of the precision of top-N returned images.

Research on image classification sparked the latest deep-learning boom. Many downstream tasks, including semantic segmentation, benefit from it. The state-of-the-art semantic segmentation models are all based on deep learning, and they sometimes make some semantic mistakes. In a semantic segmentation dataset with a small number of categories, images are often collected from a single scene, and there is a close semantic connection between any two categories. However, in the semantic segmentation dataset collected from multiple scenes, two categories may be irrelevant. The probability of objects in one category appearing next to objects in other categories is different, which is the basis of the paper. Semantic segmentation methods need to solve two problems of positioning and classification. This paper is dedicated to correcting those clearly wrong classifications that are contrary to reality. Specifically, we first calculate the relevancy between different class pairs. Then, based on this knowledge, we infer the category of a connected component according to the relationships of this connected component with its surrounding connected components and correct the obviously wrong classifications made by a deep learning semantic segmentation model. Several well-performing deep learning models are experimented on two challenging public datasets in the field of semantic image segmentation. Our proposed method improves the performance of UPerNet, OCRNet and SETR from 40.7%, 43% and 48.64% to 42.07%, 44.09% and 49.09% mean IoU on the ADE20K validation set, and the performance of PSPNet, DeepLabV3 and OCRNet from 37.26%, 37.3% and 39.5% to 38.93%, 38.95% and 40.63% mean IoU on the COCO-Stuff dataset, which shows the effectiveness of the method.

Object detection and semantic segmentation are two fundamental techniques of various applications in the fields of Intelligent Vehicles (IV) and Advanced Driving Assistance System (ADAS). Early studies separately handle these two problems. In this paper, inspired by some recent works, we propose a deep neural network model for joint object detection and semantic segmentation. Given an image, an encoder-decoder convolution network extracts a set of feature maps, these feature maps are shared by the detection branch and the segmentation branch to jointly carry out the object detection and semantic segmentation. In the detection branch, we design a PriorBox initialization mechanism to propose more object candidates. In the segmentation branch, we use the multi-scale atrous convolution to explore the global and local semantic information in traffic scenes. Benefiting from the PriorBox Initialization Mechanism (PBIM) and Multi-Scale Atrous Convolution (MSAC), our model presents the competitive performance. In the experiments, we widely compare with several recently-proposed methods on the public Cityscapes dataset, achieving the highest accuracy. In addition, to verify the robustness and generalization of our model, the extension experiments are also conducted on the well-known VOC2012 dataset.

Recent advances in convolutional neural networks have led to considerably accurate semantic segmentation tasks. However, previous works use a pre-trained backbone borrowed from image classification tasks with considerable computational complexity, resulting in large latencies. Many methods have been proposed to reduce the latency of the segmentation network without loss of accuracy. These methods have reported varying experimental results with different computing devices, acceleration techniques, and input image sizes. Although most studies claim that their results are state-of-the-art, it is necessary to reconsider whether they are being compared under the same conditions. We propose a performance evaluation method of real-time semantic segmentation models to compare the performance under the same conditions fairly. In addition, we carry out an empirical study to evaluate the performance of recent real-time semantic segmentation networks and make a comparative analysis between them. We train the segmentation models using the same input data and data augmentation method. Then, the performance of the segmentation methods is analyzed regarding accuracy and speed. In contrast to most studies that exclude the time required for the pre-processing and post-processing steps, we measured the actual processing time needed to perform semantic segmentation with a real dataset. Further, we measured the processing speed and power consumption of the segmentation models in embedded devices in which real-time segmentation is applied, unlike previous studies that measured performance on a PC. Experimental results showed that the real-time semantic segmentation methods could not run in real-time on embedded devices when considering the pre-processing and post-processing steps. By comprehensively considering the inference speed, energy consumption, and processing time of semantic segmentation models, the experimental results show that FasterSeg-S is suited for embedded devices.

Maize stem–leaf segmentation framework based on deformable point clouds

In order to improve the accuracy of image segmentation without changing the structure of original semantic segmentation models, an approach to train semantic segmentation models by using generative adversarial network (SS-GAN) is proposed. Using adversarial network to distinguish the source of segmented images, the model can learn the high-order relationship between pixels to enhance the spatial continuity of pixels in the segmented image. There are three aspects related to the work: constructing the generative model of fully convolutional network (FCN) structure by using VGG, and segment image preliminarily; constructing the adversarial model and training it by combining the original images, fake segmented images and real segmented images; modifying the loss function, adding the anti-loss to assist segmentation model training, encouraging generative network to learn the inter-pixel relationship independently. Experiments on PASCAL VOC and Cityscapes datasets show that the proposed method achieves better performance than the existing advanced methods, and improves IoU by 1.56% and 1.93%, respectively.

In rarefied gas flows, discontinuity phenomena such as velocity slip and temperature jump commonly appear in the gas layer adjacent to a solid boundary. Due to the physical complexity of the interactions at the gas-solid interface, particularly in the case of systems with local nonequilibrium state, boundary models with limited number of parameters cannot completely describe the reflection of gas molecules at the boundary. In this work, the Gaussian mixture (GM) model, which is an unsupervised machine learning technique, is employed to construct a statistical gas-solid surface scattering model based on the collisional data obtained from molecular dynamics (MD) simulations. The GM model is applied to study Couette flow for different inert gases (Ar and He) confined between two parallel infinite gold walls at different temperatures. A direct comparison between the results obtained from the GM model and the Cercignani-Lampis-Lord (CLL) scattering kernel against the MD collisional data in terms of the distribution of the predicted postcollisional velocities, and accommodation coefficients has shown that the results from the GM model are an excellent match with the MD results outperforming the CLL scattering kernel. As an example, for He gas, while the predicted energy accommodation coefficient by the CLL model is more than two times higher than the MD predictions, the value computed by the GM model is in excellent agreement with the MD results. This superior performance of the GM model confirms its high potential to derive a generalized boundary condition in systems encountered with highly nonequilibrium and complex gas flow conditions.

Change detection, or anomaly detection, from street-view images acquired by an autonomous robot at multiple different times, is a major problem in robotic mapping and autonomous driving. Formulation as an image comparison task, which operates on a given pair of query and reference images is common to many existing approaches to this problem. Unfortunately, providing relevant reference images is not straightforward. In this paper, we propose a novel formulation for change detection, termed compressive change retrieval, which can operate on a query image and similar reference images retrieved from the web. Compared to previous formulations, there are two sources of difficulty. First, the retrieved reference images may frequently contain non-relevant reference images, because even state-of-the-art place-recognition techniques suffer from retrieval noise. Second, image comparison needs to be conducted in a compressed domain to minimize the storage cost of large collections of street-view images. To address the above issues, we also present a practical change detection algorithm that uses compressed bag-of-words (BoW) image representation as a scalable solution. The results of experiments conducted on a practical change detection task, "moving object detection (MOD)," using the publicly available Malaga dataset validate the effectiveness of the proposed approach.

Indexing and retrieval of images by spatial constraints

How to accurately and efficiently localize objects in images is a challenging computer vision problem. In this article, a novel top-down fine-grained saliency object localization method based on deep-learned features is proposed, which can localize the same object in input image as the query image. The query image and its three subsample images are used as top-down cues to guide saliency detection. We ameliorate Convolutional Neural Network (CNN) using the fast VGG network (VGG-f) and retrained on the Pascal VOC 2012 dataset. Experiment on the FiFA dataset demonstrates that the proposed algorithm can effectively localize the saliency region and find the same object (human face) as the query. Experiments on the David1 and Face1 sequences conclusively prove that the proposed algorithm is able to effectively deal with different challenging factors including appearance and scale variations, shape deformation and partial occlusion.

The large-scale variation issue in high-resolution aerial images significantly lowers the accuracy of segmenting small objects. For a deep-learning-based semantic segmentation model, the main reason is that the deeper layers generate high-level semantics over considerably large receptive fields, thus improving the accuracy for large objects but ignoring small objects. Although the low-level features extracted by shallow layers contain small-object information, large-object information has predominant effects. When the model, using low-level features, is trained, the large objects push the small objects aside. This observation motivates us to propose a novel reverse difference mechanism (RDM). The RDM eliminates the predominant effects of large objects and highlights small objects from low-level features. Based on the RDM, a novel semantic segmentation method called the reverse difference network (RDNet) is designed. In the RDNet, a detailed stream is proposed to produce small-object semantics by enhancing the output of RDM. A contextual stream for generating high-level semantics is designed by fully accumulating contextual information to ensure the accuracy of the segmentation of large objects. Both high-level and small-object semantics are concatenated when the RDNet performs predictions. Thus, both small- and large-object information is depicted well. Two semantic segmentation benchmarks containing vital small objects are used to fully evaluate the performance of the RDNet. Compared with existing methods that exhibit good performance in segmenting small objects, the RDNet has lower computational complexity and achieves 3.9–18.9% higher accuracy in segmenting small objects.