Core2Vec: A Core-Preserving Feature Learning Framework for Networks

It is widely accepted that traditional word embedding models, which rely on distributional semantics hypothesis, are relatively limited for contrast meaning problem. Distributional semantics hypothesis indicates that words lying in similar contexts have similar representations in vector space. Nevertheless, synonyms and antonyms often locate in similar contexts, which means they appear close to each other in vector space. Hence, it is of great difficulty to distinguish antonyms from synonyms. To address this challenge, we propose an optimization model, named Lexicon-based Word Embedding Tuning (LWET) model. The goal of LWET is to incorporate reliable semantic lexicons to tune the distributions of pre-trained word embeddings in the vector space so as to improve their ability of distinguishing antonyms from synonyms. To speed up the training process of LWET, we propose two approximation algorithms, including positive sampling and quasi-hierarchical softmax. Compared with quasi-hierarchical softmax, positive sampling is faster, however, at the cost of worse performance. In experiments, LWET and other state-of-the-art models are tested on antonyms recognition, distinguishing antonyms from synonyms and word similarity. The results of the first two experiments show that LWET significantly improves the ability of word embeddings to detect antonyms, thus achieving the state-of-the-art performance. On word similarity, LWET gets slightly better performance than the state-of-the-art models. It means that LWET can remain and strengthen the semantic structure rather than destroy it when tuning word distributions in vector space. In general, compared with related work, LWET can not only achieve similar or even better performance, but also speed up the training process.

Drug repositioning is a promising approach to discover new therapeutic applications with existing drugs. However, existing Graph Neural Networks (GNNs)-based methods suffer from two limitations: (1) the sparsity of labeled drug-disease associations and high-quality node representations, despite the continuous emergence of biomedical knowledge from multiple modalities; and (2) the over-smoothing issue in GNNs, which causes representations to become indistinguishable, thereby degrading model performance. To address these issues, we propose Panacea, a graph learning framework that leverages multimodal information to enhance drug repositioning performance. Panacea introduces an automated knowledge acquisition and refinement pipeline that collects and encodes multimodal information, including drug molecular structures (SMILES strings), clinical symptom descriptions of diseases, and both homogeneous and heterogeneous biomedical graphs. The resulting multimodal embeddings are aligned via a learnable projection layer, providing strong initial node representations for graph learning. In the subsequent hierarchical graph learning stage, we improve the Graph Isomorphism Network (GIN) with gated mechanisms and residual connections to enhance the network's representation capacity and avoid over-smoothing. The multimodal embeddings and the graph learning module are jointly optimized in an end-toend manner, enabling node representations to fuse graph structure. Experimental results demonstrate that Panacea outperforms state-of-the-art methods, achieving significant improvements in drug repositioning tasks, especially in scenarios with sparse data and insufficient information representation. Code is available at here.

Meta-path infomax joint structure enhancement for multiplex network representation learning

Network representation aims to learn low-dimensional vector representations of network nodes while preserving the inherent properties of the network. For all its popularity, majority of the existing methods focus on exploitation of diverse information, including network topology and semantic information on nodes of network, and ignore their implicit semantics. For example, we all know the saying that birds of a feather flock together. More concretely, semantic information of one node can be influenced by its neighbors' semantic information. Furthermore, even two nodes are not directly connected, they may have similar implicit semantic information (i.e., high-order semantic proximity). Thus, they should be close in the represented vector space. To this end, we propose a Deep Semantic Network Representation approach (DSNR) in the self-translation framework from sequence to sequence. To excavate the implicit semantic information of nodes and capture the high-order semantic proximity, three key components make our approach effective, i.e., aggregation of nodes neighbors' semantic information and enhancement to the semantic feature representations of nodes by a deep autoencoder, integration of nodes semantic information in node identity sequence to generate node semantic sequence, and translation from node semantic sequence to node identity sequence to capture the high-order semantic proximity in an attention-enhanced seq2seq framework. Extensive experiments based on three real-world datasets have verified the effectiveness of our proposed approach11Code is available at https://github.com/DASE4/DSNR.

Low-dimensional vector representations of network nodes have proven successful to feed graph data to machine learning algorithms and to improve performance across diverse tasks. Most of the embedding techniques, however, have been developed with the goal of achieving dense, low-dimensional encoding of network structure and patterns. Here, we present a node embedding technique aimed at providing low-dimensional feature vectors that are informative of dynamical processes occurring over temporal networks – rather than of the network structure itself – with the goal of enabling prediction tasks related to the evolution and outcome of these processes. We achieve this by using a lossless modified supra-adjacency representation of temporal networks and building on standard embedding techniques for static graphs based on random walks. We show that the resulting embedding vectors are useful for prediction tasks related to paradigmatic dynamical processes, namely epidemic spreading over empirical temporal networks. In particular, we illustrate the performance of our approach for the prediction of nodes’ epidemic states in single instances of a spreading process. We show how framing this task as a supervised multi-label classification task on the embedding vectors allows us to estimate the temporal evolution of the entire system from a partial sampling of nodes at random times, with potential impact for nowcasting infectious disease dynamics.

Drug repositioning (also called drug repurposing) is a strategy for identifying new therapeutic targets for existing drugs. This approach is of great importance in pharmacology as it is a faster and cheaper way to develop new medical treatments. In this paper, we present, to our knowledge, the first application of multiplex-heterogeneous network embedding to drug repositioning. Network embedding learns the vector representations of nodes, opening the whole machine learning toolbox for a wide variety of applications including link prediction, node labelling or clustering. So far, the application of network embedding for drug repositioning focused on heterogeneous networks. Our approach for drug repositioning is based on multiplex-heterogeneous network embedding. Such method allows the richness and complexity of multiplex and heterogeneous networks to be projected in the same vector space. In other words, multiplex-heterogeneous networks aggregate different multi-omics data in the same network representation. We validate the approach on a task of link prediction and on a case study for SARS-CoV2 drug repositioning. Experimental results show that our approach is highly robust and effective for finding new drug-target associations.KeywordsNetwork embeddingMultiplex-heterogeneous networkMulti-layer networkDrug repositioningGraph representation learningSARS-CoV2COVID-19

Query expansion is a classical topic in the field of information retrieval, which is proposed to bridge the gap between searchers' information intents and their queries. Previous researches usually expand queries based on document collections, or some external resources such as WordNet and Wikipedia [1, 2, 3, 4, 5]. However, it seems that independently using one of these resources has some defects, document collections lack semantic information of words, while WordNet and Wikipedia may not include domain-specific knowledge in certain document collection. Our work aims to combine these two kinds of resources to establish an expansion model which represents not only domain-specific information but also semantic information. In our preliminary experiments, we construct a two-layer word graph and use Random-Walk algorithm to calculate the weights of each term in pseudo-relevance feedback documents, then select the highest weighted term to expand original query. The first layer of the word graph contains terms in related documents, while the second layer contains semantic senses corresponding to these terms. These terms and semantic senses are treated as vertices of the graph and connected with each other by all possible relationships, such as mutual information and semantic similarities. We utilized mutual information, semantic similarity and uniform distribution as the weight of term-term relation, sense-sense relation and word-sense relation respectively. Though these experiments show that our expansion outperform original queries, we are troubled with some difficult problems. Given the framework of semantic graph model, we need more effort to find out an optimal graph to represent the relationships between terms and their semantic senses. We utilized a two-layer graph model in our preliminary research, where terms from different documents are treated equally. Maybe we can introduce the document as a third layer in future work, where we can differ the same terms in different documents according to document relevance and context. Then we need appropriately represent initial weights of this words, senses and relationships. Various measures for weights of terms and term relations have been proved effective in other information retrieval tasks, such as TFIDF, mutual information (MI), but there is little research on weights for semantic senses and their relations. For polysemous words, we add all of their semantic senses to the graph and assume that these senses are uniformly distributed. Actually, it is not precise for a word in a special document and query. As we know, a polysemous word may have only one or two senses in a document, and they are not uniformly distributed. Give a word, what we should do is to determine its word senses in a relevant document and estimate the distribution of these senses. Word sense disambiguation may help us in this problem. Then, there are many methods to compute word similarity according to WordNet, which we use to represent the weights of relationships between word senses. Varelas et al implemented some popular methods to compute semantic similarity by mapping terms to an ontology and examining their relationships in that ontology [4]. We also need to know which algorithm for semantic similarity is most suitable for our model. Additional, WordNet is suitable to calculate word similarity but not suitable to measure word relevance. The inner hyperlinks of Wikipedia could help us to calculate word relevance. We wish to find an effective way to combine the similarity measure from WordNet and relevance measure from Wikipedia, which may completely reflect word relationships.

Representation of large data sets became a key question of many scientific disciplines in the last decade. Several approaches for network visualization, data ordering and coarse-graining accomplished this goal. However, there was no underlying theoretical framework linking these problems. Here we show an elegant, information theoretic data representation approach as a unified solution of network visualization, data ordering and coarse-graining. The optimal representation is the hardest to distinguish from the original data matrix, measured by the relative entropy. The representation of network nodes as probability distributions provides an efficient visualization method and, in one dimension, an ordering of network nodes and edges. Coarse-grained representations of the input network enable both efficient data compression and hierarchical visualization to achieve high quality representations of larger data sets. Our unified data representation theory will help the analysis of extensive data sets, by revealing the large-scale structure of complex networks in a comprehensible form.

The virtual power plant (VPP) is an important technology for realizing smart grids. VPPs are not limited by geographical location, which greatly aggregates the distributed energy resources. To divide a VPP across regions, we no longer limit it to power lines. This paper proposes a three-stage strategy for the classification of VPPs that includes graph model construction, communication capacity assessment and cross-regional classification. To construct the VPP graph model, complex network theory and communication matrices are used. In addition, a network representation learning and clustering algorithm is used to classify the VPP decision areas. Then, the classification results are evaluated using modularity and network communication sequence entropy to identify VPP boundaries. Finally, the validity of the method is verified by simulating the test network and the real network.

Social networks are often analyzed through a graph model of the network. The dot product model assumes that two individuals are connected in the social network if their attributes or opinions are similar. In the model, a d-dimensional vector av represents the extent to which individual v has each of a set of d attributes or opinions. Then two individuals u and v are assumed to be friends, that is, they are connected in the graph model, if and only if au · av ≥ t, for some fixed, positive threshold t. The resulting graph is called a d-dot product graph..We consider two measures for diversity and clustering in social networks by using a d-dot product graph model for the network. Diversity is measured through the size of the largest independent set of the graph, and clustering is measured through the size of the largest clique. We obtain a tight result for the diversity problem, namely that it is polynomial-time solvable for d = 2, but NP-complete for d ≥ 3. We show that the clustering problem is polynomial-time solvable for d = 2. To our knowledge, these results are also the first on the computational complexity of combinatorial optimization problems on dot product graphs.We also consider the situation when two individuals are connected if their preferences are not opposite. This leads to a variant of the standard dot product graph model by taking the threshold t to be zero. We prove in this case that the diversity problem is polynomial-time solvable for any fixed d.KeywordsSocial NetworkMaximum CliqueStructural ResultProduct GraphInterval GraphThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Advances in exponential random graph ( p*) models applied to a large social network

Cross-lingual word embeddings display words from different languages in the same vector space. They provide reasoning about semantics, compare the meaning of words across languages and word meaning in multilingual contexts, necessary to bilingual lexicon induction, machine translation, and cross-lingual information retrieval. This paper proposes an efficient approach to learn bilingual transform mapping between monolingual word embeddings in language pairs. We choose ten different languages from three different language families and downloaded their last update Wikipedia dumps1 1. https://dumps.wikimedia.org. Then, with some pre-processing steps and using word2vec, we produce word embeddings for them. We select seven language pairs from chosen languages. Since the selected languages are relative, they have thousands of identical words with similar meanings. With these identical dictation words and word embedding models of each language, we create training, validation and, test sets for the language pairs. We then use a generative adversarial network (GAN) to learn the transform mapping between word embeddings of source and target languages. The average accuracy of our proposed method in all language pairs is 71.34%. The highest accuracy is achieved for the Turkish-Azerbaijani language pair with the accuracy 78.32%., which is noticeably higher than prior methods.

Recently, neural machine translation (NMT) has made a remarkable achievement. However, performance of NMT is highly influenced by the size of training parallel text. The required amount of parallel text is not available for low-resource languages. The issue of low-resource NMT exacerbated if at least one of paired languages is morphologically rich. Inspired by the capabilities of pre-trained word embedding models, we propose simple yet effective word embedding based parallel text augmentation technique. The proposed technique is vital to enlarge size of limited parallel text for low-resource NMT. To this end, we adopt publicly available pre-trained word embedding models that favor morphologically rich languages. Accordingly, we replace each word in the original text with the n most similar words in the vector space (in this study n=3). For experimental analysis, we simulate low-resource languages by using publicly available parallel texts for German↔English, Turkish↔English and Finnish↔English language pairs. We use the original parallel texts as a seed for generating synthetic parallel texts and for training character-level NMT that deemed as baseline models. The obtained results show that our augmentation techniques improve performance of baseline and dummy source sentence based NMT models by up to 5.9 CHRF scores.

Embeddings in Natural Language Processing: Theory and Advances in Vector Representations of Meaning

Problems involving multiple networks are prevalent in many scientific and other domains. In particular, network alignment, or the task of identifying corresponding nodes in different networks, has applications across the social and natural sciences. Motivated by recent advancements in node representation learning for single-graph tasks, we propose REGAL (REpresentation learning-based Graph ALignment), a framework that leverages the power of automatically-learned node representations to match nodes across different graphs. Within REGAL we devise xNetMF, an elegant and principled node embedding formulation that uniquely generalizes to multi-network problems. Our results demonstrate the utility and promise of unsupervised representation learning-based network alignment in terms of both speed and accuracy. REGAL runs up to 30x faster in the representation learning stage than comparable methods, outperforms existing network alignment methods by 20 to 30% accuracy on average, and scales to networks with millions of nodes each.