Towards Qualitative Word Embeddings Evaluation: Measuring Neighbors Variation

We demonstrate several ways to use morphological word analogies to examine the representation of complex words in semantic vector spaces. We present a set of morphological relations, each of which can be used to generate many word analogies. 1. We show that the difference-vectors for pairs which have the same relation to each other are similarly aligned. 2. We suggest that addition of difference-vectors is a useful phrase-building operator. 3. We propose that pairs in the same relation may have similar relative frequencies. 4. We suggest that homographs, which necessarily have the same semantic vectors, can sometimes be separated into different vectors for different senses, using frequency estimates and alignment constraints obtained from word analogies. 5. We observe that some of our analogies seem to be parallel, and might be combined. We use Arabic words as a case study, because Arabic orthography includes verb conjugations, object pronouns, definitive articles, possessive pronouns, and some prepositions in single word-forms. Therefore, a number of short phrases, built up of easily perceived constituents, are already present in stock semantic spaces for Arabic available on the web. Similar phrases in English would require including bigrams or trigrams as lemmas in the word embedding, although English derivational morphology allows for other relationships in standard semantic spaces which Arabic does not, for example negation. We make our corpus of morphological relations available to other researchers.

This study examines the applicability of the Cloze test, a widely used tool for assessing text comprehension proficiency, while highlighting its challenges in large-scale implementation. To address these limitations, an automated correction approach was proposed, utilizing Natural Language Processing (NLP) techniques, particularly word embeddings (WE) models, to assess semantic similarity between expected and provided answers. Using data from Cloze tests administered to students in Brazil, WE models for Brazilian Portuguese (PT-BR) were employed to measure the semantic similarity of the responses. The results were validated through an experimental setup involving twelve judges who classified the students’ answers. A comparative analysis between the WE models’ scores and the judges’ evaluations revealed that GloVe was the most effective model, demonstrating the highest correlation with the judges’ assessments. This study underscores the utility of WE models in evaluating semantic similarity and their potential to enhance large-scale Cloze test assessments. Furthermore, it contributes to educational assessment methodologies by offering a more efficient approach to evaluating reading proficiency.

Vector representations of words learned by the statistical information of word-context matrix encode the semantic meaning of words in semantic space. Knowledge mapping shows the development process and structure of scientific knowledge, using a series of techniques, such as data mining and data visualization. This paper did research on word vectors and proposed a knowledge mapping construction method based on word vectors. Compared with traditional way, the proposed method uses a large amount of contextual information to get the semantic vector representations of keywords. Semantic relation between words can be reflected in the distance between corresponding vectors. Experiments were carried out on the corpus of computer science subject. It is showed that the proposed method was able to mine the field structure of disciplines and to serve as a powerful reference for promoting discipline construction and development. In addition, Comparison with results of traditional method also proved that knowledge mapping constructed by word vectors was indeed more meaningful and unified.

In this paper, we conduct in‐depth research and analysis on the intelligent recognition and teaching of English fuzzy text through parallel projection and region expansion. Multisense Soft Cluster Vector (MSCVec), a multisense word vector model based on nonnegative matrix decomposition and sparse soft clustering, is constructed. The MSCVec model is a monolingual word vector model, which uses nonnegative matrix decomposition of positive point mutual information between words and contexts to extract low‐rank expressions of mixed semantics of multisense words and then uses sparse. It uses the nonnegative matrix decomposition of the positive pointwise mutual information between words and contexts to extract the low‐rank expressions of the mixed semantics of the polysemous words and then uses the sparse soft clustering algorithm to partition the multiple word senses of the polysemous words and also obtains the global sense of the polysemous word affiliation distribution; the specific polysemous word cluster classes are determined based on the negative mean log‐likelihood of the global affiliation between the contextual semantics and the polysemous words, and finally, the polysemous word vectors are learned using the Fast text model under the extended dictionary word set. The advantage of the MSCVec model is that it is an unsupervised learning process without any knowledge base, and the substring representation in the model ensures the generation of unregistered word vectors; in addition, the global affiliation of the MSCVec model can also expect polysemantic word vectors to single word vectors. Compared with the traditional static word vectors, MSCVec shows excellent results in both word similarity and downstream text classification task experiments. The two sets of features are then fused and extended into new semantic features, and similarity classification experiments and stack generalization experiments are designed for comparison. In the cross‐lingual sentence‐level similarity detection task, SCLVec cross‐lingual word vector lexical‐level features outperform MSCVec multisense word vector features as the input embedding layer; deep semantic sentence‐level features trained by twin recurrent neural networks outperform the semantic features of twin convolutional neural networks; extensions of traditional statistical features can effectively improve cross‐lingual similarity detection performance, especially cross‐lingual topic model (BL‐LDA); the stack generalization integration approach maximizes the error rate of the underlying classifier and improves the detection accuracy.

Word embeddings and convolutional neural networks (CNN) have attracted extensive attention in various classification tasks for Twitter, e.g. sentiment classification. However, the effect of the configuration used to train and generate the word embeddings on the classification performance has not been studied in the existing literature. In this paper, using a Twitter election classification task that aims to detect election-related tweets, we investigate the impact of the background dataset used to train the embedding models, the context window size and the dimensionality of word embeddings on the classification performance. By comparing the classification results of two word embedding models, which are trained using different background corpora (e.g. Wikipedia articles and Twitter microposts), we show that the background data type should align with the Twitter classification dataset to achieve a better performance. Moreover, by evaluating the results of word embeddings models trained using various context window sizes and dimensionalities, we found that large context window and dimension sizes are preferable to improve the performance. Our experimental results also show that using word embeddings and CNN leads to statistically significant improvements over various baselines such as random, SVM with TF-IDF and SVM with word embeddings.

How to jointly represent images and texts is important for tasks involving both modalities. Visual-semantic embedding models have been recently proposed and shown to be effective. The key idea is that by learning a mapping from images into a semantic text space, the algorithm is able to learn a compact and effective joint representation. However, existing approaches simply map each text concept to a single point in the semantic space. Mapping instead to a density distribution provides many interesting advantages, including better capturing uncertainty about each text concept, and enabling better geometric interpretation of concepts such as inclusion, intersection, etc. In this work, we present a novel Gaussian Visual-Semantic Embedding (GVSE) model, which leverages the visual information to model text concepts as Gaussian distributions in semantic space. Experiments in two tasks, image classification and text-based image retrieval on the large scale MIT Places205 dataset, have demonstrated the superiority of our method over existing approaches, with higher accuracy and better robustness.

Effects of rake angle of chisel plough on soil cutting factors and power requirements: A computer simulation

We present a clustering-based language model using word embeddings for text readability prediction. Presumably, an Euclidean semantic space hypothesis holds true for word embeddings whose training is done by observing word co-occurrences. We argue that clustering with word embeddings in the metric space should yield feature representations in a higher semantic space appropriate for text regression. Also, by representing features in terms of histograms, our approach can naturally address documents of varying lengths. An empirical evaluation using the Common Core Standards corpus reveals that the features formed on our clustering-based language model significantly improve the previously known results for the same corpus in readability prediction. We also evaluate the task of sentence matching based on semantic relatedness using the Wiki-SimpleWiki corpus and find that our features lead to superior matching performance.

Word embedding models typically learn two types of vectors: target word vectors and context word vectors. These vectors are normally learned such that they are predictive of some word co-occurrence statistic, but they are otherwise unconstrained. However, the words from a given language can be organized in various natural groupings, such as syntactic word classes (e.g. nouns, adjectives, verbs) and semantic themes (e.g. sports, politics, sentiment). Our hypothesis in this paper is that embedding models can be improved by explicitly imposing a cluster structure on the set of context word vectors. To this end, our model relies on the assumption that context word vectors are drawn from a mixture of von Mises-Fisher (vMF) distributions, where the parameters of this mixture distribution are jointly optimized with the word vectors. We show that this results in word vectors which are qualitatively different from those obtained with existing word embedding models. We furthermore show that our embedding model can also be used to learn high-quality document representations.

Zero shot learning (ZSL) provides a solution to recognising unseen classes without class labelled data for model learning. Most ZSL methods aim to learn a mapping from a visual feature space to a semantic embedding space, e.g. attribute or word vector spaces. The use of word vector space is particularly attractive as compared to attribute, it offers vast auxiliary classes with free parts embedding without human annotation. However, using the word vector embedding often provides weaker discriminative power than manually labelled attributes of the auxiliary classes. This is compounded further in zero-shot action recognition due to richer content variations among action classes. In this work we propose to explore a broader semantic contextual information in the text domain to enrich the word vector representation of action classes. We show through extensive experiments that this method improves significantly the performance of a number of existing word vector embedding ZSL methods. Moreover, it also outperforms attribute embedding ZSL with human annotation.

Word embeddings are widely used in Natural Language Processing (NLP) for a vast range of applications. However, it has been consistently proven that these embeddings reflect the same human biases that exist in the data used to train them. Most of the introduced bias indicators to reveal word embeddings’ bias are average-based indicators based on the cosine similarity measure. In this study, we examine the impacts of different similarity measures as well as other descriptive techniques than averaging in measuring the biases of contextual and non-contextual word embeddings. We show that the extent of revealed biases in word embeddings depends on the descriptive statistics and similarity measures used to measure the bias. We found that over the ten categories of word embedding association tests, Mahalanobis distance reveals the smallest bias, and Euclidean distance reveals the largest bias in word embeddings. In addition, the contextual models reveal less severe biases than the non-contextual word embedding models.

AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP

Aims and objectives: This study aims to redress the paucity of research on the semantics of loanwords, by extending and empirically testing Backus’s ((2001). The role of semantic specificity in insertional codeswitching: Evidence from Dutch-Turkish. Jacobson, Rodolfo (Hg): Codeswitching Worldwide. Bd, 2, 125–154) Specificity Hypothesis – ‘Embedded language elements in code-switching have a high degree of semantic specificity’ (p. 128). Approach: Adopting a concept-based approach to examine loanwords in a large, reliable corpus, the study pursues the following question: Do loanwords have a high degree of semantic specificity relative to their receiving-language equivalents? Specificity is operationalized as an entropy measure of the target word’s environment, the assumption being that more specific words have less variety in their surrounding context. Data and analysis: To test this hypothesis, Anglicisms in a 24-million-word newspaper corpus of Argentine Spanish were processed in three stages: detecting loanwords, selecting semantic equivalents, and measuring specificity. Findings/conclusions: A Wilcoxon Signed-Rank Test revealed that loanwords receive significantly lower entropy scores, that is, they are more specific than their Spanish equivalents. The results suggest a possible motive for adopting loanwords when terms already exist in the source language, namely, to utilize words that provide more nuanced meaning. Originality: Methodologically, this study offers innovative applications of computational methods to loanword research, employing a distributional model to measure entropy. Theoretically, it addresses an underrepresented aspect of loanword adoption, semantics, by extending Backus’s hypothesis to loanwords and increasing its scope to data often viewed as ‘monolingual’. Significance/implications: The conclusions offer novel perspectives on loanwords with existing semantic equivalents, often viewed as ‘unnecessary’ when compared to loanwords that introduce new concepts into the recipient language (e.g. blog). With the notion of specificity, we may understand these loanwords as disruptors to the semantic system of the recipient language, dividing up the semantic space formerly occupied solely by the native equivalent, thus increasing the level of nuance expressed in the original concept.

Recent advances in the field of computational linguistics have led to the development of various prediction-based models of semantics. These models seek to infer word representations from large text collections by predicting target words from neighbouring words (or vice versa). The resulting representations are vectors in a continuous space, collectively called word embeddings. Although psychological plausibility was not a primary concern for the developers of predictive models, it has been the topic of several recent studies in the field of psycholinguistics. That is, word embeddings have been linked to similarity ratings, word associations, semantic priming, word recognition latencies, and so on. Here, we build on this work by investigating category structure. Throughout seven experiments, we sought to predict human typicality judgements from two languages, Dutch and English, using different semantic spaces. More specifically, we extracted a number of predictor variables, and evaluated how well they could capture the typicality gradient of common categories (e.g., birds, fruit, vehicles, etc.). Overall, the performance of predictive models was rather modest and did not compare favourably with that of an older count-based model. These results are somewhat disappointing given the enthusiasm surrounding predictive models. Possible explanations and future directions are discussed.

In this paper, we investigate whether a semantic representation of patent documents provides added value for a multi-dimensional visual exploration of a patent landscape compared to traditional approaches that use tf–idf (term frequency–inverse document frequency). Word embeddings from a pre-trained word2vec model created from patent text are used to calculate pairwise similarities in order to represent each document in the semantic space. Then, a hierarchical clustering method is applied to create several semantic aggregation levels for a collection of patent documents. For visual exploration, we have seamlessly integrated multiple interaction metaphors that combine semantics and additional metadata for improving hierarchical exploration of large document collections.