Word embedding and cognitive linguistic models in text classification tasks

In recent years, supervised video summarization has achieved promising progress with various recurrent neural networks (RNNs) based methods, which treats video summarization as a sequence-to-sequence learning problem to exploit temporal dependency among video frames across variable ranges. However, RNN has limitations in modelling the long-term temporal dependency for summarizing videos with thousands of frames due to the restricted memory storage unit. Therefore, in this paper we propose a stacked memory network called SMN to explicitly model the long dependency among video frames so that redundancy could be minimized in the video summaries produced. Our proposed SMN consists of two key components: Long Short-Term Memory (LSTM) layer and memory layer, where each LSTM layer is augmented with an external memory layer. In particular, we stack multiple LSTM layers and memory layers hierarchically to integrate the learned representation from prior layers. By combining the hidden states of the LSTM layers and the read representations of the memory layers, our SMN is able to derive more accurate video summaries for individual video frames. Compared with the existing RNN based methods, our SMN is particularly good at capturing long temporal dependency among frames with few additional training parameters. Experimental results on two widely used public benchmark datasets: SumMe and TVsum, demonstrate that our proposed model is able to clearly outperform a number of state-of-the-art ones under various settings.

This study examines the applicability of the Cloze test, a widely used tool for assessing text comprehension proficiency, while highlighting its challenges in large-scale implementation. To address these limitations, an automated correction approach was proposed, utilizing Natural Language Processing (NLP) techniques, particularly word embeddings (WE) models, to assess semantic similarity between expected and provided answers. Using data from Cloze tests administered to students in Brazil, WE models for Brazilian Portuguese (PT-BR) were employed to measure the semantic similarity of the responses. The results were validated through an experimental setup involving twelve judges who classified the students’ answers. A comparative analysis between the WE models’ scores and the judges’ evaluations revealed that GloVe was the most effective model, demonstrating the highest correlation with the judges’ assessments. This study underscores the utility of WE models in evaluating semantic similarity and their potential to enhance large-scale Cloze test assessments. Furthermore, it contributes to educational assessment methodologies by offering a more efficient approach to evaluating reading proficiency.

Due to the continuous growth and advancement of technology, an enormous volume of text data is generated daily across various sources including social media platforms, websites, search engines, healthcare records, and news articles. Extracting meaningful patterns from text data, such as viewpoints, related theories, journal distribution, facts, and the development of online news text, is a challenging task due to the varying lengths of the texts. One issue arises from the length of the text data itself, and another challenge lies in extracting valuable features, especially in news articles. In the deep learning models, the convolutional neural networks (CNNs) are capable of capturing local features in text data, but unable to capture the structural information or semantic relationships between words. Consequently, a sole CNN network often yields poor performance in text classification tasks, whereas the Gated Recurrent Unit (GRU) is adept at effectively extracting semantic information and understanding the global structural relationships present in textual data. This paper presents a solution to the problem by introducing a new text classification that integrates the strengths of CNN and GRU. The proposed hybrid models incorporate word vectorization and word dispersion in parallel. Initially, the model trains word vectors using the Word2vec model and then leverages the GRU model to capture semantic information from text sentences. Subsequently, the CNN method is employed to capture crucial semantic features, leading to classification using the SoftMax layer. Experimental findings demonstrated that the proposed hybrid GRU_CNN model outperformed and achieved accuracy 97.73% as compared to individual CNN, LSTM, and GRU models in terms of classification effectiveness and accuracy.

Artificial Intelligence (AI) has been used widely to extract people's opinions from social media websites. However, most of the existing works focus on eliciting the features from English text. In this paper, we describe an Arabic text sentiment analysis approach using a Deep Neural network, namely Long Short-Term Memory Recurrent Neural Network (LSTM-RNN). In this research, we investigate how the different pre-trained Word Embedding (WE) models affect our model's accuracy. The dataset includes Arabic corpus collected from Twitter. The results show significant improvement in Arabic text classification.

Word embeddings have been successfully exploited in systems for NLP tasks, such as parsing and text classification. It is intuitive that word embeddings created from a larger corpus would provide a better coverage of vocabulary. Meanwhile, word embeddings trained on a corpus related to the given task or target domain would more effectively represent the semantics of terms. However, in some emerging domains (e.g. bio-surveillance using social media data), it may be difficult to find a domain corpus that is large enough for creating effective word embeddings. To deal with this problem, we propose novel approaches that use both word embeddings created from generic and target domain corpora. Our experimental results on sentence classification tasks show that our approaches significantly improve the performance of an existing convolutional neural network that achieved state-of-the-art performances on several text classification tasks.

Word embeddings and convolutional neural networks (CNN) have attracted extensive attention in various classification tasks for Twitter, e.g. sentiment classification. However, the effect of the configuration used to train and generate the word embeddings on the classification performance has not been studied in the existing literature. In this paper, using a Twitter election classification task that aims to detect election-related tweets, we investigate the impact of the background dataset used to train the embedding models, the context window size and the dimensionality of word embeddings on the classification performance. By comparing the classification results of two word embedding models, which are trained using different background corpora (e.g. Wikipedia articles and Twitter microposts), we show that the background data type should align with the Twitter classification dataset to achieve a better performance. Moreover, by evaluating the results of word embeddings models trained using various context window sizes and dimensionalities, we found that large context window and dimension sizes are preferable to improve the performance. Our experimental results also show that using word embeddings and CNN leads to statistically significant improvements over various baselines such as random, SVM with TF-IDF and SVM with word embeddings.

Many human activities consist of physical gestures that tend to be performed in certain sequences. Wearable inertial sensor data have as a consequence been employed to automatically detect human activities, lately predominantly with deep learning methods. This article focuses on the necessity of recurrent layers—more specifically Long Short-Term Memory (LSTM) layers—in common Deep Learning architectures for Human Activity Recognition (HAR). Our experimental pipeline investigates the effects of employing none, one, or two LSTM layers, as well as different layers' sizes, within the popular DeepConvLSTM architecture. We evaluate the architecture's performance on five well-known activity recognition datasets and provide an in-depth analysis of the per-class results, showing trends which type of activities or datasets profit the most from the removal of LSTM layers. For 4 out of 5 datasets, an altered architecture with one LSTM layer produces the best prediction results. In our previous work we already investigated the impact of a 2-layered LSTM when dealing with sequential activity data. Extending upon this, we now propose a metric, rGP, which aims to measure the effectiveness of learned temporal patterns for a dataset and can be used as a decision metric whether to include recurrent layers into a network at all. Even for datasets including activities without explicit temporal processes, the rGP can be high, suggesting that temporal patterns were learned, and consequently convolutional networks are being outperformed by networks including recurrent layers. We conclude this article by putting forward the question to what degree popular HAR datasets contain unwanted temporal dependencies, which if not taken care of, can benefit networks in achieving high benchmark scores and give a false sense of overall generability to a real-world setting.

Task-specific dependency-based word embedding methods

Word embedding models typically learn two types of vectors: target word vectors and context word vectors. These vectors are normally learned such that they are predictive of some word co-occurrence statistic, but they are otherwise unconstrained. However, the words from a given language can be organized in various natural groupings, such as syntactic word classes (e.g. nouns, adjectives, verbs) and semantic themes (e.g. sports, politics, sentiment). Our hypothesis in this paper is that embedding models can be improved by explicitly imposing a cluster structure on the set of context word vectors. To this end, our model relies on the assumption that context word vectors are drawn from a mixture of von Mises-Fisher (vMF) distributions, where the parameters of this mixture distribution are jointly optimized with the word vectors. We show that this results in word vectors which are qualitatively different from those obtained with existing word embedding models. We furthermore show that our embedding model can also be used to learn high-quality document representations.

In this paper we show how word embeddings can be used to evaluate semantically the topic detection process in social networks. We propose to create and train a word embeddings with word2vec model to be used for text classification process. Then when the documents are classified, we use a pre-trained word embeddings and two similarity measures for semantic evaluation of the classification process. In particular, we perform experiments with two datasets of Twitter, using both bag-of-words with conventional classification algorithms and word embeddings with deep learning-based classification algorithms. Finally, we perform a benchmark and make some inferences about results.

Text classification often faces the problem of imbalanced training data. This is true in sentiment analysis and particularly prominent in emotion classification where multiple emotion categories are very likely to produce naturally skewed training data. Different sampling methods have been proposed to improve classification performance by reducing the imbalance ratio between training classes. However, data sparseness and the small disjunct problem remain obstacles in generating new samples for minority classes when the data are skewed and limited. Methods to produce meaningful samples for smaller classes rather than simple duplication are essential in overcoming this problem. In this paper, we present an oversampling method based on word embedding compositionality which produces meaningful balanced training data. We first use a large corpus to train a continuous skip-gram model to form a word embedding model maintaining the syntactic and semantic integrity of the word features. Then, a compositional algorithm based on recursive neural tensor networks is used to construct sentence vectors based on the word embedding model. Finally, we use the SMOTE algorithm as an oversampling method to generate samples for the minority classes and produce a fully balanced training set. Evaluation results on two quite different tasks show that the feature composition method and the oversampling method are both important in obtaining improved classification results. Our method effectively addresses the data imbalance issue and consequently achieves improved results for both sentiment and emotion classification.

In the realm of contemporary soft computing practices, analysis of public perceptions and opinion mining (OM) have received considerable attention due to the easy availability of colossal data in the form of unstructured text generated by social media, e-commerce portals, blogs, and other similar web resources. The year 2020 witnessed the gravest epidemic in the history of mankind, and in the present year, we stand amidst a global, massive and exhaustive vaccination movement. Since the inception of the COVID-19 vaccines and their applications, people across the globe, from the ordinary public to celebrities and VIPs have been expressing their fears, doubts, experiences, expectations, dilemmas and perceptions about the current COVID-19 vaccination program. Being very popular among a large class of modern human society, the Twitter platform has been chosen in this research to study public perceptions about this global vaccination drive. More than 112 thousand Tweets from users of different countries around the globe are extracted based on hashtags related to the affairs of the COVID-19 vaccine. A three-tier framework is being proposed in which raw Tweets are extracted and cleaned first, visualized and converted into numerical vectors through word embedding and N-gram models next, and finally analyzed through a few machine learning classifiers with the standard performance metrics, accuracy, precision, recall, and F1-measure. The Logistic Regression (LR) and Adaptive Boosting (AdaBoost) classifiers attended the highest accuracies of 87% and 89% with the Bag of Words (BoW) and Term Frequency-Inverse Document Frequency (TF-IDF) word embedding models respectively. Overall, the BoW model achieved slightly better average classification accuracy (78.33%) than that of the TF-IDF model (77.89%). Moreover, the experimental results show that most of the people have a neutral attitude towards the current COVID-19 vaccination drive and people favoring the COVID-19 vaccination program are greater in number than those who doubt it and its consequences.KeywordsVaccineCOVID-19 vaccination programSentiments analysis Twitter Machine learningN-gram

A short text topic modeling method based on integrating Gaussian and Logistic coding networks with pre-trained word embeddings

Question Classification (QC) is the fundamental task for Question Answering Systems (QASs) implementation, and is a vital task, as it helps in identifying the question category. It plays a big role in predicting the answer to a question while building a QAS. However, classifying medical questions is still a challenging task due to the complexity of medical terms. Many researchers have proposed different techniques to solve these problems, but some of these problems remain partially solved or unsolved. With the help of deep learning technology, various text-processing problems have become much easier to solve. In this paper, an improved deep learning-based model for Medical Forum Question Classification (MFQC) is proposed to classify medical questions. In the proposed model, feature representation is performed using Word2Vec, which is a word embedding model. Additionally, the features are extracted from the word embedding layer based on Convolutional Neural Networks (CNNs). Finally, a Bidirectional Long Short Term Memory (BiLSTM) network is used to classify the extracted features. The BiLSTM model analyzes the target information of the representation and then outputs the question category via a SoftMax layer. Our model achieves state-of-the-art performance by effectively capturing semantic and syntactic features from the input questions. We evaluate the proposed CNN-BiLSTM model on two benchmark datasets and compare its performance with existing methods, demonstrating its superiority in accurately categorizing medical forum questions.

Text classification is a basic task in natural language processing, but the small character perturbations in words can greatly decrease the effectiveness of text classification models, which is called character-level adversarial example attack. There are two main challenges in character-level adversarial examples defense, which are out-of-vocabulary words in word embedding model and the distribution difference between training and inference. Both of these two challenges make the character-level adversarial examples difficult to defend. In this paper, we propose a framework which jointly uses the character embedding and the adversarial stability training to overcome these two challenges. Our experimental results on five text classification data sets show that the models based on our framework can effectively defend character-level adversarial examples, and our models can defend 93.19% gradient-based adversarial examples and 94.83% natural adversarial examples, which outperforms the state-of-the-art defense models.