State representation learning for control: An overview

We propose novel deep speaker representation learning that considers perceptual similarity among speakers for multi-speaker generative modeling. Following its success in accurate discriminative modeling of speaker individuality, knowledge of deep speaker representation learning (i.e., speaker representation learning using deep neural networks) has been introduced to multi-speaker generative modeling. However, the conventional discriminative algorithm does not necessarily learn speaker embeddings suitable for such generative modeling, which may result in lower quality and less controllability of synthetic speech. We propose three representation learning algorithms that utilize a perceptual speaker similarity matrix obtained by large-scale perceptual scoring of speaker-pair similarity. The algorithms train a speaker encoder to learn speaker embeddings with three different representations of the matrix: a set of vectors, the Gram matrix, and a graph. Furthermore, we propose an active learning algorithm that iterates the perceptual scoring and speaker encoder training. To obtain accurate embeddings while reducing costs of scoring and training, the algorithm selects unscored speaker-pairs to be scored next on the basis of the sequentially-trained speaker encoder's similarity prediction results. Experimental evaluation results show that 1) the proposed representation learning algorithms learn speaker embeddings strongly correlated with perceptual speaker-pair similarity, 2) the embeddings improve synthetic speech quality in speech autoencoding tasks better than conventional d-vectors learned by discriminative modeling, 3) the proposed active learning algorithm achieves higher synthetic speech quality while reducing costs of scoring and training, and 4) among the proposed similarity {vector, matrix, graph} embedding algorithms, the first achieves the best speaker similarity for synthetic speech and the third gives the most improvement in the synthetic speech naturalness.

Not having access to compact and meaningful representations is known to significantly increase the complexity of reinforcement learning (RL). For this reason, it can be useful to perform state representation learning (SRL) before tackling RL tasks. However, obtaining a good state representation can only be done if a large diversity of transitions is observed, which can require a difficult exploration, especially if the environment is initially reward-free. To solve the problems of exploration and SRL in parallel, we propose a new approach called XSRL (eXploratory State Representation Learning). On one hand, it jointly learns compact state representations and a state transition estimator which is used to remove unexploitable information from the representations. On the other hand, it continuously trains an inverse model, and adds to the prediction error of this model a k-step learning progress bonus to form the maximization objective of a discovery policy. This results in a policy that seeks complex transitions from which the trained models can effectively learn. Our experimental results show that the approach leads to efficient exploration in challenging environments with image observations, and to state representations that significantly accelerate learning in RL tasks.

Multi-Agent Deterministic Policy Gradient (MADDPG) is a very useful algorithm in multi-agent domains. We analyze the algorithm and find that MADDPG uses deep neural networks (DNNs) as a Q function model. One advantage of DNNs is that they can build very complex processing functions to handle high-dimensional input. However, the disadvantage of this end-to-end learning is that it usually requires a lot of data, which is not always available for real-world control applications. In this paper, a new algorithm, State Representation Learning Multi-Agent Deep Deterministic Policy Gradient (SRL-MADPPG), is proposed that combines MADDPG with state representation learning which uses DNNs as a function fitting. i.e., model learning network is used to pre-train the first layer of the actor and critic networks, then the actor and critic learn from the state representation instead of the raw observations. Simulation result shows that the SRL-MADDPG algorithm improves the final performance in comparison with the end-to-end learning.

Through well-designed optimization paradigm and deep neural networks as feature extractor, deep reinforcement learning (DRL) algorithms learn optimal policy on discrete and continuous action space. However, such capability is restricted by the low sampling efficiency. By inspecting the importance of feature extraction in DRL, we find that state feature learning is one of the key obstacles for sampling efficiently. To this end, we propose a new state representation learning scheme with adjacent state consistency loss (ASC loss). The loss is based on the hypothesis that the distance between adjacent states is smaller than that of far apart ones since scenes in videos generally evolve smoothly. We exploit ASC loss as an assistant of RL loss in the training phase to boost the state feature learning, and make evaluation on existing DRL algorithms as well as behavioral cloning algorithm. Experiments on Atari games and MuJoCo continuous control tasks demonstrate the effectiveness of our scheme.

Camera Selection Decisions (CSD) are highly useful for several applications in a multi-camera network. For example, CSD benefit multi-camera target tracking by reducing the number of candidate cameras to look for the target’s next location. The correct candidate cameras, decreases the number of false Re-ID queries as well as the computation time. Also, in multi-camera trajectory forecasting (MCTF) to predict where a person will re-appear in the camera network along with the transition time. These applications require a large amount of annotated data for training. In this paper, we use state-representation learning with a reinforcement learning based policy to effectively and efficiently make camera selection decisions. We further demonstrate that by using learned state representations, as opposed to hand-crafted state variables, we are able to achieve state-of-the-art results on camera selection, while reducing the training time for the RL policy. Along with this, we use a reward function that helps to reduce the amount of supervision in training the policy in a semi-supervised way. We report our results on four datasets: NLPR_MCT, DukeMTMC, CityFlow, and WNMF dataset. We show that an RL policy reduces unnecessary Re-ID queries and therefore the false alarms, scales well to larger camera networks, and is target-agnostic.

A meta-learning framework using representation learning to predict drug-drug interaction

In offline reinforcement learning-based recommender systems (RLRS), learning effective state representations is crucial for capturing user preferences that directly impact long-term rewards. However, raw state representations often contain high-dimensional, noisy information and components that are not causally relevant to the reward. Additionally, missing transitions in offline data make it challenging to accurately identify features that are most relevant to user satisfaction. To address these challenges, we propose Policy-Guided Causal Representation (PGCR), a novel two-stage framework for causal feature selection and state representation learning in offline RLRS. In the first stage, we learn a causal feature selection policy that generates modified states by isolating and retaining only the causally relevant components (CRCs) while altering irrelevant components. This policy is guided by a reward function based on the Wasserstein distance, which measures the causal effect of state components on the reward and encourages the preservation of CRCs that directly influence user interests. In the second stage, we train an encoder to learn compact state representations by minimizing the mean squared error (MSE) loss between the latent representations of the original and modified states, ensuring that the representations focus on CRCs. We provide a theoretical analysis proving the identifiability of causal effects from interventions, validating the ability of PGCR to isolate critical state components for decision-making. Extensive experiments demonstrate that PGCR significantly improves recommendation performance, confirming its effectiveness for offline RL-based recommender systems.

This paper presents the Bayesian Hierarchical Words Representation (BHWR) learning algorithm. BHWR facilitates Variational Bayes word representation learning combined with semantic taxonomy modeling via hierarchical priors. By propagating relevant information between related words, BHWR utilizes the taxonomy to improve the quality of such representations. Evaluation of several linguistic datasets demonstrates the advantages of BHWR over suitable alternatives that facilitate Bayesian modeling with or without semantic priors. Finally, we further show that BHWR produces better representations for rare words.

Quorum-sensing peptides (QSPs) are the signal molecules that are closely associated with diverse cellular processes, such as cell-cell communication, and gene expression regulation in Gram-positive bacteria. It is therefore of great importance to identify QSPs for better understanding and in-depth revealing of their functional mechanisms in physiological processes. Machine learning algorithms have been developed for this purpose, showing the great potential for the reliable prediction of QSPs. In this study, several sequence-based feature descriptors for peptide representation and machine learning algorithms are comprehensively reviewed, evaluated and compared. To effectively use existing feature descriptors, we used a feature representation learning strategy that automatically learns the most discriminative features from existing feature descriptors in a supervised way. Our results demonstrate that this strategy is capable of effectively capturing the sequence determinants to represent the characteristics of QSPs, thereby contributing to the improved predictive performance. Furthermore, wrapping this feature representation learning strategy, we developed a powerful predictor named QSPred-FL for the detection of QSPs in large-scale proteomic data. Benchmarking results with 10-fold cross validation showed that QSPred-FL is able to achieve better performance as compared to the state-of-the-art predictors. In addition, we have established a user-friendly webserver that implements QSPred-FL, which is currently available at http://server.malab.cn/QSPred-FL. We expect that this tool will be useful for the high-throughput prediction of QSPs and the discovery of important functional mechanisms of QSPs.

Ecoacoustics is an emerging science that seeks to understand the role of sound in ecological processes. Passive acoustic monitoring is being used to collect vast quantities of soundscape audio recordings to study variations in acoustic community and monitor biodiversity. However, extracting relevant information from soundscape recordings is non-trivial. Recent approaches to machine-learned acoustic features appear promising but are limited by at least three issues: inductive biases, lack of interpretability and crude temporal integration. In this paper we introduce a novel self-supervised representation learning algorithm for ecoacoustics - a convolutional Variational Auto-encoder (VAE) - and directly address these shortcomings. Firstly, we train the network on soundscape recordings from temperate and tropical field sites along a gradient of ecological degradation to provide a more relevant inductive bias than prior approaches. Secondly, we present a new method that allows interpretation of the latent space for the first time, giving insight into the basis of classification. Thirdly, we advance existing methods for temporal aggregation of learned embeddings by encoding latent features as a distribution over time. Under our approach to increase interpretability, we provide insight into how learned features drive habitat classification for the first time: inspection of latent space confirms that varying combinations of biophony, geophony and anthrophony are used to infer sites along a degradation gradient. Our novel temporal encoding method increases sensitivity to periodic signals and improves on previous research that uses time-averaged representations for site classification. This approach also reveals the contribution of hardware-specific frequency response that create a potential bias; we demonstrate how a simple linear transformation can be used to mitigate the effect of hardware variance on the learned representation under our approach. Our novel approach paves the way for development of a new class of deep neural networks that afford more interpretable learned ecoacoustic representations to advance both fundamental and applied science and support global conservation efforts.

Image annotation is a time-consuming and costly task. Previously, we published MorphoCluster as a novel image annotation tool to address problems of conventional, classifier-based image annotation approaches: their limited efficiency, training set bias and lack of novelty detection. MorphoCluster uses clustering and similarity search to enable efficient, computer-assisted image annotation. In this work, we provide a deeper analysis of this approach. We simulate the actions of a MorphoCluster user to avoid extensive manual annotation runs. This simulation is used to test supervised, unsupervised and transfer representation learning approaches. Furthermore, shrunken k-means and partially labeled k-means, two new clustering algorithms that are tailored specifically for the MorphoCluster approach, are compared to the previously used HDBSCAN*. We find that labeled training data improve the image representations, that unsupervised learning beats transfer learning and that all three clustering algorithms are viable options, depending on whether completeness, efficiency or runtime is the priority. The simulation results support our earlier finding that MorphoCluster is very efficient and precise. Within the simulation, more than five objects per simulated click are being annotated with 95% precision.

The deep computation model has been proved to be effective for big data hierarchical feature and representation learning in the tensor space. However, it requires expensively computational resources including high-performance computing units and large memory to train a deep computation model with a large number of parameters, limiting its effectiveness and efficiency for industry informatics big data feature learning. In this paper, a tensor-train deep computation model is presented for industry informatics big data feature learning. Specially, the tensor-train network is used to compress the parameters significantly by converting the dense weight tensors into the tensor-train format. Furthermore, a learning algorithm is implemented based on gradient descent and back-propagation to train the parameters of the presented tensor-train deep computation model. Extensive experiments are carried on STL-10, CUAVE, and SNAE2 to evaluate the presented model in terms of the approximation error, classification accuracy drop, parameters reduction, and speedup. Results demonstrate that the presented model can improve the training efficiency and save the memory space greatly for the deep computation model with small accuracy drops, proving its potential for industry informatics big data feature learning.

Representation Learning in a heterogeneous space with mixed variables of numerical and categorical types has interesting challenges due to its complex feature manifold. Moreover, feature learning in an unsupervised setup, without class labels and a suitable learning loss function, adds to the problem complexity. Further, the learned representation and subsequent predictions should not reflect discriminatory behavior towards certain sensitive groups or attributes. The proposed feature map should preserve maximum variations present in the data and needs to be fair with respect to the sensitive variables. We propose, in the first phase of our work, an efficient encoder-decoder framework to capture the mixed-domain information. The second phase of our work focuses on de-biasing the mixed space representations by adding relevant fairness constraints. This ensures minimal information loss between the representations before and after the fairness-preserving projections. Both the information content and the fairness aspect of the final representation learned has been validated through several metrics where it shows excellent performance. Our work (FairMixRep) addresses the problem of Mixed Space Fair Representation learning from an unsupervised perspective and learns a Universal representation which is timely, unique and a novel research contribution. 11This paper is accepted at ICDM'2020 DLC Workshop.

This work presents a video compression method based on sparse representation and dictionary learning algorithms. The proposed scheme achieves superb rate-distortion performance and decent subjective quality, compared to modern standards, especially at low bit-rates. Different from similar works, sparse representation is employed here for both intra-frame and block matching inter-frame motion information. Dividing video frames to reference and current frames, motion vectors and motion compensation residuals of current frames are estimated in regard to reference frames. The sparse codes of reference frames and motion compensation residuals are obtained using learned dictionaries, entropy-coded, and stored or sent to the receiver along with the coded motion field. In the receiver, after decoding the sparse codes and motion vectors, the reference frames and residuals are reconstructed employing the same learned dictionary and the current frames are recovered using the reference frames and motion fields. In the proposed scheme, the Iterative Least Square Dictionary Learning Algorithm (ILS-DLA) and K-SVD dictionary building methods are employed in the DCT domain. The compression rate and quality of the method based on the two dictionary learning algorithms are compared to each other and to H.264/AVC and HEVC modern standards. The results based on PSNR and SSIM criteria show that the proposed approach presents superior performance respect to H.264/AVC and even HEVC for higher bit-rates of QCIF video format, and the K-SVD learning algorithm performs slightly better than the ILS-DLA for the purpose.

Sparse representation has been widely used in machine learning, signal processing and communications. K-SVD, which generalizes k-means clustering, is one of the most famous algorithms for sparse representation and dictionary learning. K-SVD is an iterative method that alternates between encoding the data sparsely by using the current dictionary and updating the dictionary based on the sparsely represented data. In this paper, we introduce a single-pass K-SVD method. In this method, the previous input data are first summarized as a condensed representation of weighted samples. Then, we developed a weighted K-SVD algorithm to learn a dictionary from the union of this representation and the newly input data. Experimental results show that our approach can approximate K-SVD’s performance well by consuming considerably less storage resource.