TreeLSTM with tag-aware hypernetwork for sentence representation

Natural Language Processing (NLP) involves the use of algorithms and models and various computational techniques to analyze, process, and generate natural language data, including speech and text. NLP helps computers interact with humans in a more natural way, which has become increasingly important as more humancomputer interactions take place. NLP allows machines to process and analyze voluminous unstructured data, including social media posts, newspaper articles, reviews from customers, emails, and others. It helps organizations extract insights, automate tasks, and improve decision-making by enabling machines to understand and generate human-like language. A linguistic background is essential for understanding NLP. Linguistic theories and models help in developing NLU systems, as NLP specialists need to understand the structure and rules of language. NLU systems are organized into different components, including language modelling, parsing, and semantic analysis. NLU systems may be assessed through the use of metrics that includes measures like precision and recall, as well as indicators that convey meaningful information that include F1 score and others. Semantics and knowledge representation are central to NLU, as they involve understanding the meaning of words and sentences and representing this information in a way that machines can use. Approaches to knowledge representation include semantic networks, ontologies, and vector embeddings. Language modelling is an essential step in NLP that sees usage in applications like speech recognition, text generation, and text completion and also in areas such as machine translation. Ambiguity Resolution remains a major challenge in NLP, as language is often ambiguous and context-dependent. Some common applications of NLP include sentiment analysis, chatbots, virtual assistants, machine translation, speech recognition, text classification, text summarization, and information extraction. In this chapter, we show the applicability of a popular unsupervised learning technique, viz., clustering through K-Means. The efficiency provided by the K-Means algorithm can be improved through the use of an optimization loop. The prospects for NLP are promising, with an increasing demand for AI-powered language technologies in various industries, including healthcare, finance, and e-commerce. There is also a growing need for ethical and responsible AI systems that are transparent and accountable.

Representing a sentence in a high dimensional space is fundamental for most natural language processing (NLP) tasks at present. These representations depend on the underlying structures upon which they are built. Two scenarios are possible: one is to view the sentence as a sequence of words and another is to consider its inherent grammatical structure. It is possible to equip the first way with some external grammatical knowledge, but to capture a proper syntax would be close to impossible. Therefore, we investigate the second one by extending the design of an existing dependency tree transformer (DT-Transformer). We propose adding a novel edge encoding mechanism to this prior architecture. Experiments show that in sentence encoding, having access to information about the relationships between “head” words and their “dependent” words and how the heads are influenced by the dependent words achieves better sentence representation. Evaluation on the four tasks shows noteworthy results compared to the existing DT-Transformer, standard Transformer, LSTM-based models, and tree- structured LSTMs. Extensive experimentation with representing the edge embeddings as different distributions (mean and standard deviation), encoding the edges in different ways, and an ablation study to find where to place each module in the architecture and which modules to use in the design is also provided.

Word vector is an important tool for natural language processing (NLP) tasks such as text classification. However, existing static language models such as Word2vec cannot solve the polysemy problem, leading to a decline in text classification performance. To solve this problem, this paper proposes a method for making Chinese word vector dynamic (MCWVD). The part of speech (POS) is used to solve the ambiguity problem caused by different POS. The POS structure graph is constructed and the syntactic structure information of POS features is extracted by GCN (Graph Convolutional Network). POS vector and word vector are concatenated into PW (POS-Word) vector. Parametric matrix is added to improve the fusion effect of POS and word features. Multilayer attention is used to distinguish the importance of different features and further update the vector expression of word vectors about the current context. Experiments on Chinese datasets THUCNews and SogouNews show that MCWVD effectively improves the accuracy of text classification and achieves better performance than CoVe (Context Vectors) and ELMo (Embeddings from Language Models). MCWVD also achieves similar performance to BERT and GPT-1 (Generative Pre-Training), but with a much lower computational cost and only 4% of BERT parameters.

This research studies the problem of email overload and proposes a system that automatically detects whether the email is read or do. The goal of our research is to test if we could automate both the features extraction and sentence classification phases by using word embedding. We propose to achieve this goal using a simple feature-based adaptation approach, where email's sentences are represented as dense numeric vectors of reduced dimensionality using either word embeddings or sentence encodings. Given that several types of word embeddings and sentence encodings exist, we compare email's sentence representations corresponding to different word embeddings and sentence encodings with the goal of understanding what embeddings/encodings are more suitable for use in the task of detecting the intent of an email. Our experimental results using three different types of embeddings: context-free word embeddings (word2vec and GloVe), contextual word embeddings (ELMo and BERT), and sentence embeddings (DAN-based Universal Sentence Encoder and Transformer-based Universal Sentence Encoder) suggest that the email's sentences representations based on ELMo embeddings produce better results than the representations that use other embeddings. We achieved an accuracy of 90.10%, comparing with word2vec (82.02%), BERT (58.08%), DAN-based USE (86.66%), and Transformer-based USE (88.16%).

In the text classification, The similarity between the text need to be calculated, but the existing classification methods only consider the similarity between feature words and categories and does not involve the semantic similarity between feature words. In this paper, a new classification model LDA (Latent Dirichlet Allocation) - KNN (K-Nearest Neighbor) is proposed. LDA is used to solve the problem of semantic similarity measurement in traditional text categorization. The sample space is modeled and selected by this model. In the reduced feature space, KNN classifier is used to classify the sample. The experiment was based on the Matlab software platform, and the data set was obtained from the Chinese corpus of Fudan University, and the high precision classification result was obtained with the average value of 0.933. LDA-KNN model is compared with MI(Mutual Information)-KNN model and LSI(Latent Semantic Index)-KNN model. The results show that LDA-KNN model has superior classification performance in automatic text categorization.

Over the last years, Natural Language Processing (NLP) for Arabic language has obtained increasing importance due to the massive textual information available online in an unstructured text format, and its capability in facilitating and making information retrieval easier. One of the widely used NLP task is “Text Classification”. Its goal is to employ machine learning technics to automatically classify the text documents into one or more predefined categories. An important step in machine learning is to find suitable and large data for training and testing an algorithm. Moreover, Deep Learning (DL), the trending machine learning research, requires a lot of data and needs to be trained with several different and challenging datasets to perform to its best. Currently, there are few available corpora used in Arabic text categorization research. These corpora are small and some of them are unbalanced or contains redundant data. In this paper, a new voluminous Arabic corpus is proposed. This corpus is collected from 16 Arabic online news portals using an automated web crawling process. Two versions are available: the first is imbalanced and contains 3252934 articles distributed into 8 predefined categories. This version can be used to generate Arabic word embedding; the second is balanced and contains 720000 articles also distributed into 8 predefined categories with 90000 each. It can be used in Arabic text classification research. The corpus can be made available for research purpose upon request. Two experiments were conducted to show the impact of dataset size and the use of word2vec pre-trained word embedding on the performance of Arabic text classification using deep learning model.

Text classification is a requirement for every text processing application because the web contains a vast amount of text data. Intent detection, information extraction, sentiment analysis, and spam detection involves text categorization. Since text classification uses idioms, metaphors, and polysemic words, intent detection can be difficult. It is challenging to automatically identify idioms in Natural Language Processing applications such as Information Retrieval, Machine Translation, and chatbots. In all these applications, automatic idiom recognition is crucial. In this work, idiomatic and literals sentences are being classified. Idioms are typical expressions with new meanings. This research proposes an ensemble model using pretrained deep learning models to make model with more predictive nature. The models are trained and tested using in-house dataset. Moreover, an in-house dataset that contains 1040 idiomatic and literal sentences is suggested. The experimental results demonstrate the effectiveness of the proposed approach, achieving an accuracy of 86% on the test dataset.

1 IntroductionOrganizations focus on automate their processes in order to improve efficiency, reduce costs, and/or reduce human beings' errors in an easy and rapid manner. Business process management (BPM) methods provide a solution to this issue. In this context, information systems like CRM, ERP, SCM, etc. have known an increasing demand. The main problem consists of the length of the business process specifications. If new regulations appear, these specifications must be adapted. Manually extraction of visual models is time consuming. During time, a series of solutions were proposed. The literature shows a crowd of approaches that extracts data models [1], [2], [3], and process models [4], [5] from Natural Language (NL) text. This paper aims to analyze the Natural Language Processing (NLP) techniques and tools used in order to provide different types of visual representations.On the last few years, approaches based on NLP have been developed in order to automate process conversion from NL text. NLP plays an important role in NL text analysis as NLP tries to understand speech and text as humans beings would do. Colloquialism, abbreviations or typos make this task a challenging one. NLP has the origins in 1950s when Alan Turing proposed the Turing test [6], by introducing the imitation game. Since then, the literature shows a plethora of NLP tools [7], [8], [9], [10] using several machine learning techniques, our focus being on those applied on data models and process models discovery from NL text, starting with the first language parser [11] to the actual ones like NLTk [7], [8], ANTLR1, etc.Linguistic analysis is closely tied to NLP. Liddy [12] highlights 7 levels of linguistic analysis: a) Phonetic or Phonological level: how words are pronounced, b) Morphological level: prefixes, suffixes and roots analysis, c) Lexical level: word level analysis including lexical meaning and Part-Of-Speech (POS) analysis, d) Syntactic level: grammatical analysis of words in a sentence, e) Semantic level: determining the possible meanings of sentences, f) Discourse level: interpreting structure and meaning for texts larger than a sentence, g) Pragmatic level: understanding the purpose of a language.Some of the problems approached by NLP are: POS tagging, parsing, Named Entity Recognition (NER), chunking, Semantic Role Labeling (SRL). Anaphora resolution [13] refers to the interpretation of the link between the anaphor and its antecedents.The remainder of the paper is organized as follows: Section 2 briefly outlines the NLP domain, describing the levels of linguistic analysis and the main NLP approaches of analyzing NL requirements. Section 3 focuses on data and process models extraction from NL text. This section makes an introduction to Object Oriented Analysis and Business Process Modeling and analyses the existing tools that discover data and process models from text. Subsequently, Section 4 summarizes the results of this work and the conclusions are drawn in Section 5.2NLPA detailed review on NLP is given in [14] and in [15]. Jones [14] divides the history of NLP into four phases: the first starts at the beginning of the 1940s and lasts to the late 1960s, the second begins from the end of 60s and lasts to the end of 70s, the third is represented by late 80s, where the fourth phase starts in the late of 90s. Next, we will detail each phase as they were defined in [14] and [15]. First phase treated machine translation issues, while the second focused on artificial intelligence. The third phase can be called grammatico-logical phase, which is followed by the lexical phase. A fifth phase is proposed in [16] where formal theories and statistical data are combined. Since this study was published first in 1994 and then it was re-organized in 2001 we can add the sixth phase: from 2000 until present where NLP techniques are combined in order to contribute to visual models extraction.Software requirements are usually written in NL which is asymmetric and irregular [17]. …

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

This chapter discusses the concept of natural language processing (NLP), specifically how the NLP is applied. The chapter further outlines the typical functions of the NLP. In addition, it explores the NLP in strategy and strategy implementation.The NLP is a branch of computer science, AI and linguistics that focuses on the interaction between computer programs and human language. There are three functions of the NLP, namely, machine translation (MT), text summarisation and sentiment analysis (SA). MT is a branch of computer linguistics that analyses the use of computerised tools to translate the context derived from this, from one language to the next. The language that is being translated is the human language.Automatic text summarisation refers to a process of bringing about a succinct and meaningful summary of the text. SA is a way of finding out the polarity or strength of the opinion that is expressed in written text. The opinion could be either positive or negative. MT has three major approaches, namely, rule-based MT, statistical MT and neural MT. A rule-based system would typically require the expert to know both the source and the target languages. This knowledge is then used to develop syntactic, semantic and morphological rules to achieve the translation. The statistical approach uses statistical models based on the analysis of bilingual text corpora. The neural MT approach uses neural networks to achieve machine translation. There are two broad categories of text summarisation approaches, namely, extractive summarisation and abstractive summarisation. Furthermore, there are four popular types of SA, namely, the fine-grained SA, emotion detection, aspect-based SA and multilingual SA. The firm that deploys the NLP and ML in Big Data Analytics would gain crucial and better insights, setting the firm apart from its competitors.

Evaluation of machine translation (MT) into morphologically rich languages has not been well studied despite its importance. This paper proposes a classifier, that is, a deep learning (DL) schema for MT evaluation, based on different categories of information (linguistic features, natural language processing (NLP) metrics and embeddings), by using a model for machine learning based on noisy and small datasets. The linguistic features are string based for the language pairs English (EN)–Greek (EL) and EN–Italian (IT). The paper also explores the linguistic differences that affect evaluation accuracy between different kinds of corpora. A comparative study between using a simple embedding layer (mathematically calculated) and pre-trained embeddings is conducted. Moreover, an analysis of the impact of feature selection and dimensionality reduction on classification accuracy has been conducted. Results show that using a neural network (NN) model with different input representations produces results that clearly outperform the state-of-the-art for MT evaluation for EN–EL and EN–IT, by an increase of almost 0.40 points in correlation with human judgments on pairwise MT evaluation. It is observed that the proposed algorithm achieved better results on noisy and small datasets. In addition, for a more integrated analysis of the accuracy results, a qualitative linguistic analysis has been carried out in order to address complex linguistic phenomena.

A Heterogeneous Directed Graph Attention Network for inductive text classification using multilevel semantic embeddings

The development of machine translation technology closely follows that of information science. It is a breakthrough in natural language processing and a big step forward in the field of artificial intelligence. The purpose of this paper is to study the application of machine translation technology based on natural language processing. This paper introduces the relationship between natural language processing and machine translation technology, expounds the research importance of machine translation technology in natural language processing and the research status at home and abroad. The application of neural network in natural language processing machine translation system is introduced. At the same time, for the existing machine translation system, a neural network machine translation system based on reinforcement learning is proposed. And analyze the machine translation of different models through case studies. Finally, the Python software is used to implement the program, and the accuracy of the Transformer-PG-NMT model is obtained.

Text extraction is an important part of natural language processing (NLP) tasks. Most NLP tasks like text classification, machine translation, text-to-speech, text-based language identification, text summarization, and named-entity recognition involve the use of textual data. Such data is limited for low-resourced languages making it difficult to experiment advanced NLP techniques on these languages. This paper presents a Python-based toolkit for text analysis and text extraction from different types of images, documents, and audio files. The toolkit is built as a library that has functions that can be imported and utilized for text extraction.

Aiming at the problems of short text length, less information contained in text and poor classification performance caused by easy interference of deep learning model, combined with the advantages of CNN and BiLSTM, a text classification method based on fusion semantics, attention mechanism and BiLSTM-CNN model is proposed. Firstly, the Skip-gram model in word2vec is used to map the words in the data and convert them into vectors with fixed dimensions as the input of the model. The local features of the text vector are extracted by CNN and the global features related to the text context are extracted by BiLSTM. The features extracted by the two complementary models are fused, It solves the problem that the single convolution neural network model ignores the semantic and grammatical information of words in the context, and effectively avoids the gradient disappearance or gradient dispersion of the traditional cyclic neural network. This paper makes an experimental analysis on R8 and Ohsumed datasets. The results show that the proposed feature fusion model effectively improves the accuracy of text classification.