Interaction Feedback Network With Salient-Region-Aware Augmentation for Fine-Grained Visual Categorization

Fine-grained visual categorization aims to distinguish objects in subordinate classes instead of basic class, and is a challenge visual task due to the high correlation between subordinated classes and large intra-class variation (e.g. different object poses). Although, deep convolutional neural network (DCNN) has brought dramatic success on generic object classification, detection and segmentation with the availability of the large-scale training samples, direct application of DCNN on fine-grained visual categorization, where only decades or at most hundreds of training samples for each subordinate class are available in most public finegrained image datasets, cannot lead to satisfactory classification results due to small number of training samples. This study explores the transfer learning strategy for finegrained dog breed categorization based on the learned CNN models with the large-scale image dataset: ImageNet, and prove promising performance with two DCNN models: AlexNet and VGG-16. Furthermore, we argue that different DCNN architecture may extract the representation of different image aspects due to the previously defined CNN kernel sizes, number and various operations in the model learning procedure, and thus result in different performance for visual categorization. This study proposes to fusion multiple CNN architectures for combining different aspect representations to give more accurate performance. We compressively study the fusion of different layers such as Fc6 and Fc7 in AlexNet and VGG-16, and manifest 2.88% improvement of the fusion architecture over the best performance of the only one DCNN model: VGG-16 from 81.2% to 84.08%.

Fine-Grained Visual Categorization (FGVC) is challenging due to the large intra-class and small inter-class variance, and classes which are prone to be confused with one or more similar classes are called as confusing classes. Hierarchical visual data structures are widely used for image classification with a large number of classes so that the sophisticated classification problem can be processed from coarse to fine. However, hierarchical structures are accompanied with error propagation. In order to reduce the error propagation caused by the hierarchical structure, we propose the concept of minimal confusion set and its metrics. Minimal confusion set is the minimal set of classes that are highly confused with other classes in the set and much less confused with classes outside the set. Based on minimal confusion sets, a flat two-layer hierarchical structure can be formed, which reduces the error propagation. An effective mining method of minimal confusion set is also proposed, which can also be used for hierarchical learning. The mining method of minimal confusion set is based on confusion matrix, but it is different from the traditional methods using the confusion matrix obtained from test data. This method is independent on test data, and experiments on multiple datasets demonstrate the effectiveness of the method.

Feature relocation network for fine-grained image classification

Fine-grained visual categorization (FGVC) that aims to recognize objects from subcategories with very subtle differences remains a challenging task due to the large intra-class and small inter-class variation caused by, e.g., deformation, occlusion, illumination, background clutter, etc. A great deal of recent work tackles this problem by forcing the network to focus on partial discriminable features using attention mechanisms or part-based methods. However, these methods neglect the point that the network may learn to discriminate objects from identity-unrelated features, for instance, when backgrounds are discriminable in training samples, degrading the network’s generalization ability. In this paper, for the first time, we use disentangled representation learning to disentangle the fine-grained visual feature into two parts: the identity-related feature and the identity-unrelated feature. Only the identity-related feature is used for the final classification. Since identity-unrelated information is neglected in classification, intra-class variation is reduced while inter-class variation is amplified through the disentanglement, improving the classification performance as a result. Experimental results on three standard fine-grained visual categorization datasets, i.e., CUB-200-2011 (CUB), Stanford Cars (CAR) and FGVC-Aircraft (AIR), demonstrate the effectiveness of our method and show that we achieve state-of-the-art performance on the benchmarks.

Gait recognition systems often suffer from the challenges when query gaits are under the coupled effects of unknown view angles and large intra-class variations (e.g., wearing a coat). In this paper, we deem it as a two-stage classification problem, namely, view detection and fixed-view gait recognition. First, we propose two simple yet effective feature types (i.e., global features and local features) for view detection. By using the detected view information, the corresponding gallery (i.e., enrolled gait) for the detected view can be adaptively selected to perform the fixed-view gait recognition. For fixed-view gait recognition, since the inter-class variations for training are normally small, whereas the query gait usually has large intra-class variations, random subspace method are adopted. We evaluate our approach on the largest multi-view gait database CASIA-B dataset. The avoidance of searching whole multi-view database as well as the competitive performance indicate that our proposed method is practical for gait recognition in real world surveillance scenarios.

This paper presents the use of description logics (DL) in the definition and development of a Structured Query Language (SQL) domain ontology for a multi-agent based pre-assessment system. Description logics is a knowledge representation language for defining terms or classes, the relationships between classes, their instances, including individuals and literals. In a formal school curriculum, modules of learning are inter-dependent. So, teaching and learning follows an ordered sequence of learning from lower-level module(s) to higher-level ones. This process enables students to gain mastery of lower-level materials before moving up the ladder to higher-level learning. To describe an SQL ontology and its representation for a multi-agent based system application, this paper uses a description logic language to present the organization of learning modules into DesiredConcept $$ $$ , PrerequisiteConcept $$ $$ and LeafNodes $$ $$ as well as their associated relationships, namely, hasPrerequisite and hasKB between the learning modules. The paper thus presents a TBox and an Abox of a DL ontology and further transformation into a first-order predicate for a multi-agent based system that was implemented in Jason.

Remote sensing images are usually characterized by complex backgrounds, scale and orientation variations, and large intraclass variance. General semantic segmentation methods usually fail to fully investigate the above issues, and thus their performances on remote sensing image segmentation are limited. In this article, we propose our LOGCAN++, a semantic segmentation model customized for remote sensing images, which is made up of a global class-aware (GCA) module and several local class-aware (LCA) modules. The GCA module captures global representations for class-level context modeling to reduce the interference of background noise. The LCA module generates local class representations as intermediate perceptual elements to indirectly associate pixels with the global class representations, targeting dealing with the large intraclass variance problem. In particular, we introduce affine transformations in the LCA module for adaptive extraction of local class representations to effectively tolerate scale and orientation variations in remote sensing images. Extensive experiments on three benchmark datasets show that our LOGCAN++ outperforms current mainstream general and remote sensing semantic segmentation methods and achieves a better trade-off between speed and accuracy.

ABSTRACTCardiovascular disease is one of the leading causes of death worldwide. Early and accurate detection of abnormal cardiac activity can be an effective way to prevent serious cardiovascular events. Electrocardiogram (ECG) and phonocardiogram (PCG) signals provide an objective evaluation of the heart's electrical and acoustic functions, enabling medical professionals to make an accurate diagnosis. Therefore, the cardiologists often use them to make a preliminary diagnosis of abnormal cardiac activity in clinical practice. For this reason, many diagnostic models have been proposed. However, these models fail to utilize the interaction information within and between the signals to aid the diagnosis of disease. To address this issue, we designed a residual dual‐attention network (ResAN) for the detection of abnormal cardiac activity using synchronized ECG and PCG signals. First, ResAN uses a feature learning module with two parallel residual networks, for example, ECG‐ResNet and PCG‐ResNet to automatically learn the deep modal‐specific features from the ECG and PCG sequences, respectively. Second, to fully utilize the available information of different modal signals, ResAN uses a dual‐attention fusion module to capture the salient features of the integrated ECG and PCG features learned by the feature learning module, as well as the alternating features between them based on the attention mechanisms. Finally, these fused features are merged and fed to the classification module to detect abnormal cardiac activity. Our model achieves an accuracy of 96.1%, surpassing the performances of comparison models by 1.0% to 9.9% when using synchronized ECG and PCG signals. Furthermore, the ablation study confirmed the efficacy of the components in ResAN and also showed that ResAN performs better with synchronized ECG and PCG signals compared to using single‐modal signals. Overall, ResAN provides a valid solution for the early detection of abnormal cardiac activity using ECG and PCG signals.

Efficient and reliable fingerprint matching is crucial for many civilian and forensic applications. Fingerprints are characterized by large intra-class variations (i.e. variability in different impressions from the same finger). This variability manifests itself by the missing or the displacement of genuine minutiae and the detection of spurious minutiae. Displaced, missing and spurious minutiae make fingerprint matching a very challenging pattern recognition problem.Although significant improvements have been made in minutiae-based matching, most of minutiae matching algorithms lack of robustness with respect to displaced, missing and spurious minutiae [4]. This paper introduces EDT-C, a new fingerprint matcher based on minutiae triplets. The proposed matcher uses an extended form of Delaunay Triangulation that allows to take into account, not only genuine minutiae, but also spurious, displaced and missing ones. EDT-C characterizes minutiae triplets by a set of innovative geometric features that help in tolerating linear and non-linear distortions. Finally, EDT-C includes some optimizations that allows to quickly consolidate local matchings and filter non-matching minutia triplets. Experiments show that EDT-C has a reasonable computational cost and is far more accurate than its main competitors.

In this paper, we propose a face detection method with multi-scale deep representation learning. While existing face detection methods have achieved good performance, they fail to consider the large intra-class variations between faces in the prediction stage and the relation between multi-scale proposals in the proposal stage. Our method encourages the network to learn compact features which simultaneously minimizes the intra-class variations and enlarges the inter-class variations. We can make more reliable binary classification between face regions and non-face ones. To obtain better proposals, we predict and select hard proposals according to the sizes of faces. Our method achieves competitive results on the widely used FDDB and WIDER face datasets, which demonstrates the effectiveness of our approach.

Due to great challenges such as tremendous intra-class variations and low image resolution, context information has been playing a more and more important role for accurate and robust event recognition in surveillance videos. The context information can generally be divided into the feature level context, the semantic level context, and the prior level context. These three levels of context provide crucial bottom-up, middle level, and top down information that can benefit the recognition task itself. Unlike existing researches that generally integrate the context information at one of the three levels, we propose a hierarchical context model that simultaneously exploits contexts at all three levels and systematically incorporate them into event recognition. To tackle the learning and inference challenges brought in by the model hierarchy, we develop complete learning and inference algorithms for the proposed hierarchical context model based on variational Bayes method. Experiments on VIRAT 1.0 and 2.0 Ground Datasets demonstrate the effectiveness of the proposed hierarchical context model for improving the event recognition performance even under great challenges like large intra-class variations and low image resolution.

Texture classification is a challenging task due to the wide range of natural texture types and large intra-class variations in texture images, such as different rotations, scales, positions and lighting conditions. Many existing methods for extracting texture features are designed carefully by user for specific applications. The extracted texture features are then used as input to various classification methods, such as support vector machines, to classify the textures. The system performance greatly depends on the feature extractor. Unfortunately, there is no systematic approach for feature extractor design. In this paper, we propose a method called extreme learning machine with multi-scale local receptive fields (ELM-MSLRF) to achieve feature learning and classification simultaneously for texture classification. In contrast to traditional methods, the proposed method learns the features by the network itself and can be applied to more general applications. Additionally, it is fast and requires few computations. Experiments on the ALOT texture dataset demonstrate the attractive performance of ELM-MSLRF even compared with the state-of-the-art algorithms. Moreover, the proposed ELM-MSLRF achieves the best performance on the NORB dataset.

BackgroundThe images of different flower species had small inter-class variations across different classes as well as large intra-class variations within a class. Flower classification techniques are mainly based on the features of color, shape and texture, however, the procedure always involves too many heuristics as well as manual labor to tweak parameters, which often leads to datasets with poor qualitative and quantitative measures. The current study proposed a deep architecture of convolutional neural network (CNN) for the purposes of improving the accuracy of identifying the white flowers of Fragaria × ananassa from other three wild flower species of Androsace umbellata (Lour.) Merr., Bidens pilosa L. and Trifolium repens L. in fields.ResultsThe explored CNN architecture consisted of eightfolds of learnable weights including 5 convolutional layers and 3 fully connected layers, which received a true color 227 × 227 pixels flower image as its input. The developed CNN detector was able to classify the instances of flowers at overall average accuracies of 99.2 and 95.0% in the training and test procedure, respectively. The state-of-the-art CNN model was compared with the classical models of the scale-invariant feature transform (SIFT) features and the pyramid histogram of orientated gradient (PHOG) features combined with the multi-class support vector machine (SVM) algorithm. The proposed model turned out to be much more accurate than the traditional models of SIFT + SVM at overall average accuracies of 82.9 and 55.6% in the training and test procedure and PHOG + SVM at overall average accuracies of 78.3 and 63.1%, respectively.ConclusionsThe proposed state-of-the-art CNN method demonstrates that artificial intelligence is capable of precise classification of the white flower images, whose accuracy is comparable to traditional algorithms. The presented algorithm can be further used for the discrimination of white wild flowers in fields.

Recognizing human activities in partially observed videos is a challenging problem and has many practical applications. When the unobserved subsequence is at the end of the video, the problem is reduced to activity prediction from unfinished activity streaming, which has been studied by many researchers. However, in the general case, an unobserved subsequence may occur at any time by yielding a temporal gap in the video. In this paper, we propose a new method that can recognize human activities from partially observed videos in the general case. Specifically, we formulate the problem into a probabilistic framework: 1) dividing each activity into multiple ordered temporal segments, 2) using spatiotemporal features of the training video samples in each segment as bases and applying sparse coding (SC) to derive the activity likelihood of the test video sample at each segment, and 3) finally combining the likelihood at each segment to achieve a global posterior for the activities. We further extend the proposed method to include more bases that correspond to a mixture of segments with different temporal lengths (MSSC), which can better represent the activities with large intra-class variations. We evaluate the proposed methods (SC and MSSC) on various real videos. We also evaluate the proposed methods on two special cases: 1) activity prediction where the unobserved subsequence is at the end of the video, and 2) human activity recognition on fully observed videos. Experimental results show that the proposed methods outperform existing state-of-the-art comparison methods.

Cross-modal retrieval is such a challenging topic that traditional global representations would fail to bridge the semantic gap between images and texts to a satisfactory level. Using local features from images and words from documents directly can be more robust for the scenario with large intraclass variations and small interclass discrepancies. In this paper, we propose a novel unsupervised binary coding algorithm called binary set embedding (BSE) to obtain meaningful hash codes for local features from the image domain and words from text domain. Understanding image features with the word vectors learned from the human language instead of the provided documents from data sets, BSE can map samples into a common Hamming space effectively and efficiently where each sample is represented by the sets of local feature descriptors from image and text domains. In particular, BSE explores relationship among local features in both feature level and image (text) level, which can balance the sensitivity of each other. Furthermore, a recursive orthogonalization procedure is applied to reduce the redundancy of codes. Extensive experiments demonstrate the superior performance of BSE compared with state-of-the-art cross-modal hashing methods using either image or text queries.