Weakly Supervised Collective Feature Learning From Curated Media

Audio description from image by modal translation network

Target detection (TD) is one of the major tasks in hyperspectral image (HSI) processing, and its performance is greatly affected by the background. Feature extraction (FE) has been an effective way to mine discriminative information, especially FE based on deep learning, which can learn the intrinsic properties of data to further improve the detection performance. Unlike supervised networks, unsupervised stacked sparse autoencoders (SSAEs) can learn deep and nonlinear features without any labeled data. However, SSAEs usually require a supervised fine-tuned model to obtain better discrimination, which is not feasible for TD, since the prior information is generally insufficient. In this letter, we introduce a distance constraint that is added to the SSAE to form a new distance constrained SSAE (DCSSAE) network. Specifically, the distance constraint maximizes the distinction between the target pixels and other background pixels in the feature space. Then, using the discriminative features learned from the DCSSAE, a simple detector using radial basis function kernel is derived for background suppression. Experiments on two HSIs demonstrate that the deep spectral features learned from the DCSSAE are more distinguishable, and our proposed detector, namely, the DCSSAE detector, outperforms several popular detectors, especially in background suppression.

PMANet: Progressive multi-stage attention networks for skin disease classification

With the rapid development of various satellite sensors, automatic and advanced scene classification technique is urgently needed to process a huge amount of satellite image data. Recently, a few of research works start to implant the sparse coding for feature learning in aerial scene classification. However, these previous research works use the single-layer sparse coding in their system and their performances are highly related with multiple low-level features, such as scale-invariant feature transform (SIFT) and saliency. Motivated by the importance of feature learning through multiple layers, we propose a new unsupervised feature learning approach for scene classification on very high resolution satellite imagery. The proposed unsupervised feature learning utilizes multipath sparse coding architecture in order to capture multiple aspects of discriminative structures within complex satellite scene images. In addition, the dense low-level features are extracted from the raw satellite data by using different image patches with varying size at different layers, and this approach is not limited to a particularly designed feature descriptors compared with the other related works. The proposed technique has been evaluated on two challenging high-resolution datasets, including the UC Merced dataset containing 21 different aerial scene categories with a 1 foot resolution and the Singapore dataset containing 5 land-use categories with a 0.5m spatial resolution. Experimental results show that it outperforms the state-of-the-art that uses the single-layer sparse coding. The major contributions of this proposed technique include (1) a new unsupervised feature learning approach to generate feature representation for very high-resolution satellite imagery, (2) the first multipath sparse coding that is used for scene classification in very high-resolution satellite imagery, (3) a simple low-level feature descriptor instead of many particularly designed low-level descriptor, such as SIFT descriptors and saliency, (4) evaluation on two satellite image datasets that come from different sensor sources.

Sparse coding is a popular technique for achieving compact data representation and has been used in many applications. However, the instability issue often causes degeneration in practice and thus attracts a lot of studies. While the traditional graph sparse coding preserves the neighborhood structure of the data, this study integrates the low-rank representation(LRR) to fix the inconsistency of sparse coding by holding the subspace structures of the high-dimensional observations. The proposed method is dubbed low-rank graph regularized sparse coding (LogSC), which learns sparse codes and low-rank representations jointly rather than the traditional two-step approach. Since the two data representations share a dictionary matrix, the resulted sparse representation on this dictionary could be benefited from LRR. We solved the optimization problem of LogSC by using the linearized alternating direction method with adaptive penalty. Experimental results show the proposed method is discriminative in feature learning and robust to various noises. This work provides a one-step approach to integrating graph embedding in representation learning.

The co-occurrence features learned through mining methods have more discriminative power to separate images from other categories than individual low-level features. However, the pattern explosion problem involved in mining process prevents its application in many visual tasks. In this paper, we propose a novel scheme to learn discriminative features based on a mined optimal model. The proposed method deals with the pattern explosion problem from two aspects, 1 it uses selected weak semantic patches instead of grid patches to substantially reduce the database to mine; 2 the adopted optimal model can produce compact and representative patterns which make the resulted image code more effective and discriminative for classification. In our work, we apply the minimal description length MDL to mine the optimal model. We evaluate the proposed method on two publicly available datasets 15-Scenes and Oxford-Flowers17 and the experimental results demonstrate its effectiveness.

Network traffic classification plays an important role in many network monitoring and security tasks. More recently, with the development of deep learning techniques, the performance of network traffic classification has been significantly improved due to the powerful feature representations learned by deep neural networks. Despite the great success that has been achieved, the problems of within-class diversity and between-class similarity are still big challenges. In this paper, we propose to train a CNN model by optimizing a new discriminative objective function, where apart from minimizing the empirical risk, a metric learning regularization term is also imposed on the learned features. This metric learning regularization term enforces the CNN model to learn more discriminative features in the mapped feature space, where the instances from the same class are closer together while the instances of different classes are farther apart. We conduct extensive experiments to evaluate the proposed method on three traffic datasets. The experimental results demonstrate that our proposed method outperforms the existing baseline methods and obtains state-of-the-art results on all the three datasets.

Pseudo-label learning could effectively explore additional signals of unlabeled point clouds. The existing methods mainly utilize category hints of pseudo-label points to assist the training of model. However, they overlook other implicit hints of pseudo-label points, thus leading to limited performance gains. To explore and make full use of these implicit hints, we propose a novel deep pseudo-label-guided network (DPLG-Net) for weakly-supervised semantic segmentation of airborne lidar scanning (ALS) point clouds. Specifically, we first utilize cardinality hints of different categories of pseudo-label points to construct weighted cross-entropy (WCE) loss. This minor design could alleviate the class-imbalance issue of data. Then, we introduce a weighted entropy minimization module which is guided by predictive probability hints of pseudo-label points. It takes the generated pseudo-label points as anchor points, and the predictive probabilities of anchor points are served as weights to calculate weighted entropy of their neighbors, thus providing the direction of entropy minimization. Moreover, we propose a feature learning module for unlabeled points which is guided by semantic feature hints of pseudo-label points. In this module, semantic features of pseudo-label points are employed to produce prototype features of all the categories. By querying the semantic similarity between unlabeled points and category prototype features, the feature learning of unlabeled points is guided towards the most similar category while keeping away from other categories. Experimental results under 0.1% weak labels indicate DPLG-Net achieves state-of-the-art average F1 (71.8%) and overall accuracy (84.1%) on ISPRS dataset; and achieves state-of-the-art average F1 (78.6%) and competitive overall accuracy (87.3%) on LASDU dataset. In addition, we conduct ALS experiment and present a new ALS point cloud dataset named HBUCVC. Experimental results on HBUCVC dataset also validate the effectiveness of DPLG-Net.

This thesis presents a clear conceptual basis for theoretically studying machine learning problems. Machine learning methods afford means to automate the discovery of relationships in data sets. A relationship between quantities X and Y allows the prediction of one quantity given information of the other. It is these relationships that we make the central object of study. We call these relationships transitions. A transition from a set X to a set Y is a function from X into the probability distributions on Y. Beginning with this simple notion, the thesis proceeds as follows: • Utilizing tools from statistical decision theory, we develop an abstract language for quantifying the information present in a transition. • We attack the problem of generalized supervision. Generalized supervision is the learning of classifiers from non-ideal data. An important example of this is the learning of classifiers from noisily labelled data. We demonstrate the virtues of our abstract treatment by producing generic methods for solving these problems, as well as producing generic upper bounds for our methods as well as lower bounds for any method that attempts to solve these problems. • As a result of our study in generalized supervision, we produce means to define procedures that are robust to certain forms of corruption. We explore, in detail, procedures for learning classifiers that are robust to the effects of symmetric label noise. The result is a classification algorithm that is easier to understand, implement and parallelize than standard kernel based classification schemes, such as the support vector machine and logistic regression. Furthermore, we demonstrate the uniqueness of this method. • Finally, we show how many feature learning schemes can be understood via our language. We present well motivated objectives for the task of learning features from unlabelled data, before showing how many standard feature learning methods (such as PCA, sparse coding, auto-encoders and so on) can be seen as minimizing surrogates to our objective functions.

Image matting in the perception granular deep learning

This paper proposes a new neural network learning method to improve the performance for action recognition in video. Most human action recognition methods use a clip-level training strategy, which divides the video into multiple clips and trains the feature learning network by minimizing the loss function of clip classification. The video category is predicted by the voting of clips from the same video. In order to obtain more effective action feature, a new video-level feature learning method is proposed to train 3D CNN to boost the action recognition performance. Different with clip-level training which uses clips as input, video-level learning network uses the entire video as the input. Consistent constraint loss is defined to minimize the distance between clips of the same video in voting space. Further, a video-level loss function is defined to compute the video classification error. The experimental results show that the proposed video-level training is a more effective action feature learning approach compared with the clip-level training. And this paper has achieved the state-of-the-art performance on UCF101 and HMDB51 datasets without using pre-trained models of other large-scale datasets. Our code and final model are available at https://github.com/hqu-cst-mmc/VLL.

In this paper, we present a deep coupled framework to address the problem of matching sketch image against a gallery of mugshots. Face sketches have the essential information about the spatial topology and geometric details of faces while missing some important facial attributes such as ethnicity, hair, eye, and skin color. We propose a coupled deep neural network architecture which utilizes facial attributes in order to improve the sketch-photo recognition performance. The proposed Attribute-Assisted Deep Convolutional Neural Network (AADCNN) method exploits the facial attributes and leverages the loss functions from the facial attributes identification and face verification tasks in order to learn rich discriminative features in a common embedding subspace. The facial attribute identification task increases the inter-personal variations by pushing apart the embedded features extracted from individuals with different facial attributes, while the verification task reduces the intra-personal variations by pulling together all the features that are related to one person. The learned discriminative features can be well generalized to new identities not seen in the training data. The proposed architecture is able to make full use of the sketch and complementary facial attribute information to train a deep model compared to the conventional sketch-photo recognition methods. Extensive experiments are performed on composite (E-PRIP) and semi-forensic (IIIT-D semi-forensic) datasets. The results show the superiority of our method compared to the state-of-the-art models in sketch-photo recognition algorithms.

With the rapid development of information technologies, more and more data are collected from multiple sources, which contain different perspectives of the data. To accurately explore the shared information among multiple views, K-means based multi-view clustering methods are designed and widely used in various applications for their simplicity and efficiency. However, all of these methods cluster data in the original high-dimensional feature space which is extremely time-consuming and sensitive to outliers, or cluster data in the embedded feature space for each view, which is hard to find the optimal reduced dimensionality. To solve these problems, we propose a robust discriminative multi-view K-means clustering with feature selection and group sparsity learning. Compared to the state-of-the-arts, the proposed algorithm has two advantages: 1) Discriminative K-means clustering and feature learning are integrated jointly into a single framework, where robust and accurate clustering results are obtained in the embedded feature space with an l2, 1-norm based loss function. 2) Group sparsity constraints are imposed to select the most relevant features and the most important views. We apply the proposed algorithm to serval kinds of multimedia understanding applications. Experimental results demonstrate the effectiveness of the proposed algorithm.

Various levels of feature maps extracted from the convolutional neural network models can capture multiple degrees of semantic cues in representation. However, learning the relationship between different semantic features across numerous layers can be challenging. Furthermore, existing representations cannot effectively capture the orientation cues. This paper proposes a representation method, the multi-layer orientation histogram, to address these problems. The main highlights are: (1) An iterative multi-layer integration method to combine the feature maps of various levels is suggested in this study. This method can provide discriminative characteristics based on spatial relationships. (2) An effective approach is suggested to apply Gabor filtering for detecting orientation cues. It can amplify the most dominant orientation cues and is convenient for efficiently using them in subsequent implementations. (3) The proposed representation directly captures orientation cues from each learned feature map. It can incorporate the learned deep features and orientation cues to create a more discriminative representation. Comparative experiments demonstrate that the proposed method exhibits a highly competitive performance on several benchmark datasets in terms of mean average precision. Moreover, this method can provide an effective architecture for learning discriminative global features.

We propose a novel discriminative feature learning method via Max-Min Ratio Analysis (MMRA) for exclusively dealing with the long-standing "worst-case class separation" problem. Existing technologies simply consider maximizing the minimal pairwise distance on all class pairs in the low-dimensional subspace, which is unable to separate overlapped classes entirely especially when the distribution of samples within same class is diverging. We propose a new criterion, i.e., Max-Min Ratio Analysis (MMRA) that focuses on maximizing the minimal ratio value of between-class and within-class scatter to extremely enlarge the separability on the overlapped pairwise classes. Furthermore, we develop two novel discriminative feature learning models for dimensionality reduction and metric learning based on our MMRA criterion. However, solving such a non-smooth non-convex max-min ratio problem is challenging. As an important theoretical contribution in this paper, we systematically derive an alternative iterative algorithm based on a general max-min ratio optimization framework to solve a general max-min ratio problem with rigorous proofs of convergence. More importantly, we also present another solver based on bisection search strategy to solve the SDP problem efficiently. To evaluate the effectiveness of proposed methods, we conduct extensive pattern classification and image retrieval experiments on several artificial datasets and real-world ScRNA-seq datasets, and experimental results demonstrate the effectiveness of proposed methods.