Bayesian Hierarchical Words Representation Learning

In order to satisfy safety requirements of plant-wide processes, distributed process monitoring methods are often used. However, few of them consider the problem on building multi-level knowledge blocks and the associations between different blocks, all of which are beneficial for plant-wide process monitoring in avoiding information conflict and even to improve monitoring accuracy. To handle these issues, a plant-wide process monitoring method is proposed, which is based on hierarchical graph representation learning with differentiable pooling by using multi-level knowledge graph (MLKG). Specifically, MLKG consists of devices level, subprocess level, process level, etc. Each level has numerous blocks (nodes) which is firstly constructed by priori-knowledge on monitoring variables to calculate the status of key components, such as the Hotelling's statistics. And then normalized mutual information (NMI) is used to obtain the associations between monitoring variables, and the status of each block on each level can be updated. Based on this method, MLKG can be completely constructed. In order to consider the association information of hierarchical representations of MLKG, the hierarchical graph representation learning is used to achieve plant-wide process monitoring. Results of case study on practical cobalt and nickel removal from zinc solution demonstrate the effectiveness and applicability of this method.

This study carries out a systematic intrinsic evaluation of the semantic representations learned by state-of-the-art pre-trained multimodal Transformers. These representations are claimed to be task-agnostic and shown to help on many downstream language-and-vision tasks. However, the extent to which they align with human semantic intuitions remains unclear. We experiment with various models and obtain static word representations from the contextualized ones they learn. We then evaluate them against the semantic judgments provided by human speakers. In line with previous evidence, we observe a generalized advantage of multimodal representations over language- only ones on concrete word pairs, but not on abstract ones. On the one hand, this confirms the effectiveness of these models to align language and vision, which results in better semantic representations for concepts that are grounded in images. On the other hand, models are shown to follow different representation learning patterns, which sheds some light on how and when they perform multimodal integration.

Word embedding methods revolve around learning continuous distributed vector representations of words with neural networks, which can capture semantic and/or syntactic cues, and in turn be used to induce similarity measures among words, sentences and documents in context. Celebrated methods can be categorized as prediction-based and count-based methods according to the training objectives and model architectures. Their pros and cons have been extensively analyzed and evaluated in recent studies, but there is relatively less work continuing the line of research to develop an enhanced learning method that brings together the advantages of the two model families. In addition, the interpretation of the learned word representations still remains somewhat opaque. Motivated by the observations and considering the pressing need, this paper presents a novel method for learning the word representations, which not only inherits the advantages of classic word embedding methods but also offers a clearer and more rigorous interpretation of the learned word representations. Built upon the proposed word embedding method, we further formulate a translation-based language modeling framework for the extractive speech summarization task. A series of empirical evaluations demonstrate the effectiveness of the proposed word representation learning and language modeling techniques in extractive speech summarization.

The deep computation model has been proved to be effective for big data hierarchical feature and representation learning in the tensor space. However, it requires expensively computational resources including high-performance computing units and large memory to train a deep computation model with a large number of parameters, limiting its effectiveness and efficiency for industry informatics big data feature learning. In this paper, a tensor-train deep computation model is presented for industry informatics big data feature learning. Specially, the tensor-train network is used to compress the parameters significantly by converting the dense weight tensors into the tensor-train format. Furthermore, a learning algorithm is implemented based on gradient descent and back-propagation to train the parameters of the presented tensor-train deep computation model. Extensive experiments are carried on STL-10, CUAVE, and SNAE2 to evaluate the presented model in terms of the approximation error, classification accuracy drop, parameters reduction, and speedup. Results demonstrate that the presented model can improve the training efficiency and save the memory space greatly for the deep computation model with small accuracy drops, proving its potential for industry informatics big data feature learning.

Military vehicle object detection technology in complex environments is the basis for the implementation of reconnaissance and tracking tasks for weapons and equipment, and is of great significance for information and intelligent combat. In response to the poor performance of traditional detection algorithms in military vehicle detection, we propose a military vehicle detection method based on hierarchical feature representation and reinforcement learning refinement localization, referred to as MVODM. First, for the military vehicle detection task, we construct a reliable dataset MVD. Second, we design two strategies, hierarchical feature representation and reinforcement learning-based refinement localization, to improve the detector. The hierarchical feature representation strategy can help the detector select the feature representation layer suitable for the object scale, and the reinforcement learning-based refinement localization strategy can improve the accuracy of the object localization boxes. The combination of these two strategies can effectively improve the performance of the detector. Finally, the experimental results on the homemade dataset show that our proposed MVODM has excellent detection performance and can better accomplish the detection task of military vehicles.

Risk prediction, usually achieved by learning representations from patient’s physiological sequence or user’s behavioral sequence data, and has been widely applied in healthcare and finance. Despite that, some recent time-aware deep learning methods have led to superior performances in such sequence representation learning tasks, such improvement is limited due to a lack of guidance from hierarchical global view. To address this issue, we propose a novel end-to-end H ierarchical G lobal V iew-guided (HGV) sequence representation learning framework. Specifically, the Global Graph Embedding (GGE) module is proposed to learn sequential clip-aware representations from temporal correlation graph (TCG) at instance level. Furthermore, following the way of key-query attention, the harmonic β-attention (β-Attn) is also developed for making a global tradeoff between time-aware decay and observation significance at channel level adaptively. Moreover, the hierarchical representations at both instance level and channel level can be coordinated by the heterogeneous information aggregation under the guidance of global view. Experimental results on both healthcare risk prediction benchmark and SMEs credit overdue risk prediction task from the real-world industrial scenario in MYBank, Ant Group, have illustrated that the proposed model can achieve competitive prediction performance compared with other known baselines. The code has been released public available at: https://github.com/LiYouru0228/HGV.

We propose novel deep speaker representation learning that considers perceptual similarity among speakers for multi-speaker generative modeling. Following its success in accurate discriminative modeling of speaker individuality, knowledge of deep speaker representation learning (i.e., speaker representation learning using deep neural networks) has been introduced to multi-speaker generative modeling. However, the conventional discriminative algorithm does not necessarily learn speaker embeddings suitable for such generative modeling, which may result in lower quality and less controllability of synthetic speech. We propose three representation learning algorithms that utilize a perceptual speaker similarity matrix obtained by large-scale perceptual scoring of speaker-pair similarity. The algorithms train a speaker encoder to learn speaker embeddings with three different representations of the matrix: a set of vectors, the Gram matrix, and a graph. Furthermore, we propose an active learning algorithm that iterates the perceptual scoring and speaker encoder training. To obtain accurate embeddings while reducing costs of scoring and training, the algorithm selects unscored speaker-pairs to be scored next on the basis of the sequentially-trained speaker encoder's similarity prediction results. Experimental evaluation results show that 1) the proposed representation learning algorithms learn speaker embeddings strongly correlated with perceptual speaker-pair similarity, 2) the embeddings improve synthetic speech quality in speech autoencoding tasks better than conventional d-vectors learned by discriminative modeling, 3) the proposed active learning algorithm achieves higher synthetic speech quality while reducing costs of scoring and training, and 4) among the proposed similarity {vector, matrix, graph} embedding algorithms, the first achieves the best speaker similarity for synthetic speech and the third gives the most improvement in the synthetic speech naturalness.

State representation learning for control: An overview

Pre-trained distributed word representations have been proven to be useful in various natural language processing (NLP) tasks. However, the geometric basis of word representations and their relations to the representations of word's contexts has not been carefully studied yet. In this study, we first investigate such geometric relationship under a general framework, which is abstracted from some typical word representation learning approaches, and find out that only the directions of word representations are well associated to their context vector representations while the magnitudes are not. In order to make better use of the information contained in the magnitudes of word representations, we propose a hierarchical Gaussian model combined with maximum a posteriori estimation to learn word representations, and extend it to represent polysemous words. Our word representations have been evaluated on multiple NLP tasks, and the experimental results show that the proposed model achieved promising results, comparing to several popular word representations.

Graph Neural Networks (GNNs), which generalize deep neural networks to graph-structured data, have drawn considerable attention and achieved state-of-the-art performance in numerous graph related tasks. However, existing GNN models mainly focus on designing graph convolution operations. The graph pooling (or downsampling) operations, that play an important role in learning hierarchical representations, are usually overlooked. In this paper, we proposed a novel multi-view graph pooling operator dubbed as MVPool, which ranks nodes across different views with different contextual graph information. Meanwhile, attention mechanism is utilized to promote the collaboration of different views for generating robust node rankings. Then the pooling operation adaptively selects a subset of nodes to form an induced subgraph based on the ranking list. To preserve the underlying graph topological information, we further introduce a structure learning mechanism to learn a refined graph structure for the pooled graph at each layer. The proposed MVPool operator is a general strategy that can be integrated into various graph neural network architectures. By combining MVPool operator with graph neural networks, we perform hierarchical representation learning for both node and graph level classification as well as clustering tasks. Experimental results on nine widely used benchmarks demonstrate the effectiveness of our proposed model.

Most of the existing automated compliance checking (ACC) systems are unable to fully automatically convert building-code requirements, especially requirements that have hierarchically complex semantic and syntactic structures, into computer-processable forms. The state-of-the-art rule-based ACC methods that are able to deal with complex requirements are based on information extraction and transformation rules, which are inflexible when applied to different types of regulatory documents. More research is thus needed to develop a flexible method to automatically process and understand requirements to support the downstream tasks in ACC systems, such as information matching and compliance reasoning. To address this need, this paper proposes (1) a new representation of requirements, the requirement hierarchy, and (2) a deep learning-based method to automatically extract semantic relations between words from building-code sentences, which are used to transform the sentences into such hierarchies. The proposed method was evaluated using a corpus of sentences from multiple regulatory documents. It achieved high semantic relation and requirement hierarchy extraction performance.

The theoretical basis and operation of LEBEX, a machine-learning system for jet-engine performance monitoring, are described. The behavior of the engine is modeled in terms of four parameters (the rotational speeds of the high- and low-speed sections and the exhaust and combustion temperatures), and parameter variations indicating malfunction are transformed into structural representations involving instances and events. LEBEX extracts descriptors from a set of training data on normal and faulty engines, represents them hierarchically in a knowledge base, and uses them to diagnose and predict faults on a real-time basis. Diagrams of the system architecture and printouts of typical results are shown.

Text is composed of words and phrases. In the bag‐of‐words model, phrases in text are split into words. This may discard the semantics of phrases, which, in turn, may give an inconsistent relatedness score between 2 texts. Our objective is to apply phrase relatedness in conjunction with word relatedness on the text relatedness task to improve text relatedness performance. We adopt 2 existing word relatedness measures based on Google n‐gram and Global Vectors for Word Representation, respectively, and incorporate them differently with an existing Google n‐gram–based phrase relatedness method to compute text relatedness. The combination of Google n‐gram–based word and phrase relatedness performs better than Google n‐gram–based word relatedness alone, by achieving the higher weighted mean of Pearson's r, ie, 0.639 and 0.619, respectively, on the 14 data sets from the series of Semantic Evaluation workshops SemEval‐2012, SemEval‐2013, and SemEval‐2015. Similarly, the combination of GloVe‐based word relatedness and Google n‐gram–based phrase relatedness performs better than GloVe‐based word relatedness alone, by achieving the higher weighted mean of Pearson's r, ie, 0.619 and 0.605, respectively, on the same 14 data sets. On the SemEval‐2012, SemEval‐2013, and SemEval‐2015 data sets, the text relatedness results obtained from the combination of Google n‐gram–based word and phrase relatedness ranked 24, 3, and 31 out of 89, 90, and 73 text relatedness systems, respectively.

One of the workhorses in studying spoken word production is the picture-word interference paradigm (PWI). In this paradigm, speakers are instructed to ignore superimposed written distractor words while fulfilling the picture-naming task. An essential finding in picture - word interference paradigm is that people spend more time in naming pictures when distractor word is of the same category as the picture naming (semantic related word) than when the distractor is unrelated. That effect is often referred to as semantic interference effect (SIE). A reversed finding is that pictures naming latency is shorter in condition of related words than in condition of unrelated words. That effect is referred to as semantic facilitation At present, there are several theories about the nature and level of effects in picture - word interference paradigm. Lexical selection by competition hypothesis assumes that SIE is the result of competition between the representations of related distractor word and the target word and then arises at lexical level. Conceptual selection model interprets the effects as the result of competition between the representations of distractor conception and the target conception. As a result, effect should arise at level. Response selection account proposes that there is a single-channel output buffer in word production and the non –target response would occupy the buffer earlier than the target. It will take some time to exclude the non-target response from the buffer and induces SIE. So SIE arises at the post-lexical stage in word production. Each account has its experimental evidence and each has its difficulty in explaining some phenomenon. In the present study, 2 experiments were conducted to investigate the universality of current theories and to explore the roles of similarity and category size on effect in PWI. In Experiment 1, subjects were asked to name pictures accompanied by words with high similarity or words with low similarity at either basic level or at category level. Results showed a shorter picture - naming latency in condition of high similarity distractor words than in condition of low similarity words, leading to significant similarity effect. Nevertheless, similarity effect occurred in basic - level naming rather than in category - level naming and the effect was stronger for low familiar pictures than for high familiar ones. Experiment 1 also showed that latency of category - level naming was shorter than that of basic-level naming and high familiar pictures were named faster than low familiar ones. Combining previous studies, these results suggested that the intensity of relationship between pictures and words would affect the direction of effect in the PWI. Semantic facilitation effect will be foundwhen there is strong relationship between pictures and words. Otherwise, interference will be found. In Experiment 2, subjects were asked to name pictures accompanied by related words or by unrelated words. Thirty pictures from large size of category and another thirty pictures from small size of category were chosen for this experiment. Results showed that related words had rather different influences on pictures naming in different condition. They would promote pictures category-level naming, thus giving rise to facilitation effect. However, they would interference pictures basic-level naming, thus causing interference effect. Category size had a prominent influence on these effects. Semantic interference effect would be stronger for large size of category pictures in basic-level naming and facilitation effect also would be stronger for big size of category pictures in category-level naming. Combining previous studies, the experiment suggested that the range of relationship between pictures and words affects the size of effect. When there is wide range of relationship, the effect will became stronger. Otherwise, the effect will be weaker. Single current theory of effect couldn’t give a proper explanation for the findings in this study. Results in this study were discussed with reference to the equilibrium between conceptual priming and lexical competition. If the conceptual priming overweight the lexical competition, facilitation effect would be found, otherwise interference would be found.

Sentiment classification is a fundamental task in many natural language processing applications. Neural networks have achieved great successes on the sentiment classification task in recent years, since recurrent neural networks and long-short-term memory networks have the ability to deal with sequences of different lengths and to capture contextual semantic information. However, the effectiveness of these methods is limited when used to extract contextual information from relatively long texts. Therefore, in our model, we apply bidirectional gated recurrent units to capture contextual information as far as possible when learning word representations, which may effectively reduce the noise compared to other methods. We also propose a novel loss function namely drop loss (DL) which makes the model focus on the hard examples - examples which are easily classified incorrectly - in order to improve the accuracy of the model. We experiment on four commonly used datasets, and the results show that the proposed method has a good performance on four datasets, and needs fewer parameters compared with recent benchmarks, such as CoVe, ULMFiT, embeddings from language models, and bidirectional encoder representations from transformers. Furthermore, we demonstrate that the classification performance of existing shallow network models can be significantly improved by using DL. In particular, the accuracy of the CNN+LSTM model improves 9% on the IMDB-10 dataset.