Text classification by CEFR levels using machine learning methods and BERT language model

Using information technology such as search engine for collecting and monitoring of network public opinion is a practical and effective method. This paper puts forward an improved algorithm of SVM classifier based on incremental learning online, and then implements a topic crawler system for network public opinion and to grab the public opinion.

In this paper we present a novel method for the automatic classification of multi-label text documents. In principle, automatic classification of text is usually tackled by supervised Machine Learning techniques like Support Vector Machines (SVM), that typically achieve state-of-the-art accuracy in several domains. Nevertheless, SVM can not handle multi-labeled documents, thus a specific preprocessing of the data is needed. In this paper we present a novel technique for the transformation of multi-label data into mono-label that is able to maintain all the information, allowing the use of standard approaches like SVM. We then evaluate our system using JRC-Acquis-it, a large dataset of italian legislation that has been manually annotated according to EuroVoc, demonstrating the potential of our approach compared to the current state of the art.

Modelling short text is challenging due to the small number of word co-occurrence and insufficient semantic information that affects downstream Natural Language Processing (NLP) tasks, for example, text classification. Gathering information from external sources is expensive and may increase noise. For efficient short text classification without depending on external knowledge sources, we propose Expressive Short text Classification (EStC). EStC consists of a novel document context-aware semantically enriched topic model called the Short text Topic Model (StTM) that captures words, topics and documents semantics in a joint learning framework. In StTM, the probability of predicting a context word involves the topic distribution of word embeddings and the document vector as the global context, which obtains by weighted averaging of word embeddings on the fly simultaneously with the topic distribution of words without requiring an additional inference method for the document embedding. EStC represents documents in an expressive (number of topics × number of word embedding features) embedding space and uses a linear support vector machine (SVM) classifier for their classification. Experimental results demonstrate that EStC outperforms many state-of-the-art language models in short text classification using several publicly available short text data sets.

Machine learning methods, including Deep learnings are popular for data processing. Deep learning methods have shown great promise in applied in natural language processing (NLP) tasks. Text classification is a supervised machine learning task that involves labelled documents to train classifier. Previous works involving machine learning and deep learning methods for text classification have been tested with relatively small- sized data instances. In this paper, we compared the performance of the machine leaning and deep learning algorithms in text classification task. This paper also studied, explored and compared the scalability of these methods with respect to bigger data instances. We used support vector machines (SVM), Logistic regression, Random forest and Naive Bayes Machine leaning algorithms and convolutional neural network (CNN) deep learning method. The task involved a multi-class classification problem involving six (6) classes consisting of six thousand (6,000) data instances with an average of 20 sentences in each data instance. The CNN deep leaning algorithm outperformed all the machine learning algorithms, achieving an accuracy of over 85%. This is because the filter weights are leaning are updated in backward propagation in each epoch, hence, this result in better result compared to the traditional methods.

1 IntroductionOrganizations focus on automate their processes in order to improve efficiency, reduce costs, and/or reduce human beings' errors in an easy and rapid manner. Business process management (BPM) methods provide a solution to this issue. In this context, information systems like CRM, ERP, SCM, etc. have known an increasing demand. The main problem consists of the length of the business process specifications. If new regulations appear, these specifications must be adapted. Manually extraction of visual models is time consuming. During time, a series of solutions were proposed. The literature shows a crowd of approaches that extracts data models [1], [2], [3], and process models [4], [5] from Natural Language (NL) text. This paper aims to analyze the Natural Language Processing (NLP) techniques and tools used in order to provide different types of visual representations.On the last few years, approaches based on NLP have been developed in order to automate process conversion from NL text. NLP plays an important role in NL text analysis as NLP tries to understand speech and text as humans beings would do. Colloquialism, abbreviations or typos make this task a challenging one. NLP has the origins in 1950s when Alan Turing proposed the Turing test [6], by introducing the imitation game. Since then, the literature shows a plethora of NLP tools [7], [8], [9], [10] using several machine learning techniques, our focus being on those applied on data models and process models discovery from NL text, starting with the first language parser [11] to the actual ones like NLTk [7], [8], ANTLR1, etc.Linguistic analysis is closely tied to NLP. Liddy [12] highlights 7 levels of linguistic analysis: a) Phonetic or Phonological level: how words are pronounced, b) Morphological level: prefixes, suffixes and roots analysis, c) Lexical level: word level analysis including lexical meaning and Part-Of-Speech (POS) analysis, d) Syntactic level: grammatical analysis of words in a sentence, e) Semantic level: determining the possible meanings of sentences, f) Discourse level: interpreting structure and meaning for texts larger than a sentence, g) Pragmatic level: understanding the purpose of a language.Some of the problems approached by NLP are: POS tagging, parsing, Named Entity Recognition (NER), chunking, Semantic Role Labeling (SRL). Anaphora resolution [13] refers to the interpretation of the link between the anaphor and its antecedents.The remainder of the paper is organized as follows: Section 2 briefly outlines the NLP domain, describing the levels of linguistic analysis and the main NLP approaches of analyzing NL requirements. Section 3 focuses on data and process models extraction from NL text. This section makes an introduction to Object Oriented Analysis and Business Process Modeling and analyses the existing tools that discover data and process models from text. Subsequently, Section 4 summarizes the results of this work and the conclusions are drawn in Section 5.2NLPA detailed review on NLP is given in [14] and in [15]. Jones [14] divides the history of NLP into four phases: the first starts at the beginning of the 1940s and lasts to the late 1960s, the second begins from the end of 60s and lasts to the end of 70s, the third is represented by late 80s, where the fourth phase starts in the late of 90s. Next, we will detail each phase as they were defined in [14] and [15]. First phase treated machine translation issues, while the second focused on artificial intelligence. The third phase can be called grammatico-logical phase, which is followed by the lexical phase. A fifth phase is proposed in [16] where formal theories and statistical data are combined. Since this study was published first in 1994 and then it was re-organized in 2001 we can add the sixth phase: from 2000 until present where NLP techniques are combined in order to contribute to visual models extraction.Software requirements are usually written in NL which is asymmetric and irregular [17]. …

The security of information is among the greatest challenges facing organizations and institutions. Cybercrime has risen in frequency and magnitude in recent years, with new ways to steal, change and destroy information or disable information systems appearing every day. Among the types of penetration into the information systems where confidential information is processed is malware. An attacker injects malware into a computer system, after which he has full or partial access to critical information in the information system. This paper proposes an ensemble classification-based methodology for malware detection. The first-stage classification is performed by a stacked ensemble of dense (fully connected) and convolutional neural networks (CNN), while the final stage classification is performed by a meta-learner. For a meta-learner, we explore and compare 14 classifiers. For a baseline comparison, 13 machine learning methods are used: K-Nearest Neighbors, Linear Support Vector Machine (SVM), Radial basis function (RBF) SVM, Random Forest, AdaBoost, Decision Tree, ExtraTrees, Linear Discriminant Analysis, Logistic, Neural Net, Passive Classifier, Ridge Classifier and Stochastic Gradient Descent classifier. We present the results of experiments performed on the Classification of Malware with PE headers (ClaMP) dataset. The best performance is achieved by an ensemble of five dense and CNN neural networks, and the ExtraTrees classifier as a meta-learner.

Integrated phenology and climate in rice yields prediction using machine learning methods

The tremendous growth of information emerging from diverse sources has resulted in digital information overwork. The vast information availability has initiated the need for automatic text classification for managing, organizing massive data, and knowledge discovery. Most of these data is in an unstructured form. The main hindrance in achieving good classification accuracy is the natural form of data, i.e., unstructured form. Such issues can be handled by applying preprocessing steps so that unstructured data can be transformed into structured data. In this paper, we have applied different preprocessing steps like bag of words, stemming, and lemmatization on raw data to study the impact of preprocessing in the text classification process. We have verified this by providing preprocessed data to various classifiers like multinominal Naive Bayes, logistic regression, stochastic gradient descent classifier, and k-nearest neighbors and found improvement in classification accuracy results. We have carried out experimentation on the 20-newsgroup dataset.

This paper presents a study of the problem of automatic classification of short coherent texts (essays) in English according to the levels of the international CEFR scale. Determining the level of text in a natural language is an important component of assessing a student’s knowledge, including checking open tasks in e-learning systems. To solve this problem, vector text models are considered based on the stylometric numerical features of the character, word, and sentence structure levels. The obtained vectors are classified by the standard machine learning classifiers. This article presents the results of the three most successful ones: Support Vector Classifier, Stochastic Gradient Descent Classifier, and LogisticRegression. Precision, comprehensiveness, and the F-measure served as the quality measures. Two open text corpora, CEFR Levelled English Texts and BEA-2019, are chosen for the experiments. The best classification results for six CEFR levels and sublevels from A1 to C2 are shown by the Support Vector Classifier with an F-score of 67% for the CEFR Levelled English Texts. This approach is compared with the application of the BERT language model (six different variants). The best model, bert-base-cased, provided an F-score value of 69%. The analysis of classification errors shows that most of them are between neighboring levels, which is quite understandable from the point of view of the domain. In addition, the quality of classification strongly depends on the text corpus, which demonstrates a significant difference in F-scores during the application of the same text models for different corpora. In general, the results obtained show the effectiveness of automatic text level determination and the possibility of its practical application.

Opinion mining, also referred to as sentiment analysis, has been an important research topic for identifying and analyzing opinions in natural language text. Computational linguistics and information retrieval are combined to extract and assess subjective information from textual data in this interdisciplinary field. In this work, I present a Bidirectional Deep Recurrent Neural Network (BDRNN) based framework for sentiment polarity detection. In this work, we propose the Sentiment Analysis using BDRNN (SA-BDRNN) system, which aims to overcome the challenges in extracting unbiased opinions from text corpora. A dataset of labeled sentiments is used for training and evaluation to assure good model performance. Then the framework aims to mine sentiments effectively from source documents and improve the understanding of important contextual information. Finally, the SA-BDRNN system is benchmarked against state of the art methods, including Convolutional Neural Networks (CNN), Support Vector Machines (SVM), and an optimized SVM using Particle Swarm Optimization (SVM-PSO). Experimental results show that the proposed SA-BDRNN framework outperforms the best previous solutions in terms of sentiment classification accuracy and robustness, and hence can be a promising solution to advanced opinion mining applications.

Biomarkers are known to be the key driver behind targeted cancer therapies by either stratifying the patients into risk categories or identifying patient subgroups most likely to benefit. However, the ability of a biomarker to stratify patients relies heavily on the type of clinical endpoint data being collected. Of particular interest is the scenario when the biomarker involved is a continuous one where the challenge is often to identify cut-offs or thresholds that would stratify the population according to the level of clinical outcome or treatment benefit. On the other hand, there are well-established Machine Learning (ML) methods such as the Support Vector Machines (SVM) that classify data, both linear as well as non-linear, into subgroups in an optimal way. SVMs have proven to be immensely useful in data-centric engineering and recently researchers have also sought its applications in healthcare. Despite their wide applicability, SVMs are not yet in the mainstream of toolkits to be utilised in observational clinical studies or in clinical trials. This research investigates the very role of SVMs in stratifying the patient population based on a continuous biomarker across a variety of datasets. Based on the mathematical framework underlying SVMs, we formulate and fit algorithms in the context of biomarker stratified cancer datasets to evaluate their merits. The analysis reveals their superior performance for certain data-types when compared to other ML methods suggesting that SVMs may have the potential to provide a robust yet simplistic solution to stratify real cancer patients based on continuous biomarkers, and hence accelerate the identification of subgroups for improved clinical outcomes or guide targeted cancer therapies.

Due to the wide proliferation of the Internet and telecommunication, huge amount of information has been produced as digital data format. It is impossible to classify this information with one's own hand one by one in many realistic problems, so that the research on automatic text classification has been grown. Machine learning technologies have applied in text classification. However, the traditional statistic machine learning technologies require large number of labeled training examples to learn accurately. To obtain enough training examples, we have to label on these huge training examples by hand. This paper presents a supervised learning algorithm based on support vector machine (SVM) to classify text documents more accurately by using unlabeled documents to augment available labeled training examples. Experimental results indicate that the classification with unlabeled examples using SVM is superior to the conventional classification,with labeled examples.

Artificial intelligence has been applied in wildfire science and management since the 1990s, with early applications including neural networks and expert systems. Since then, the field has rapidly progressed congruently with the wide adoption of machine learning (ML) methods in the environmental sciences. Here, we present a scoping review of ML applications in wildfire science and management. Our overall objective is to improve awareness of ML methods among wildfire researchers and managers, as well as illustrate the diverse and challenging range of problems in wildfire science available to ML data scientists. To that end, we first present an overview of popular ML approaches used in wildfire science to date and then review the use of ML in wildfire science as broadly categorized into six problem domains, including (i) fuels characterization, fire detection, and mapping; (ii) fire weather and climate change; (iii) fire occurrence, susceptibility, and risk; (iv) fire behavior prediction; (v) fire effects; and (vi) fire management. Furthermore, we discuss the advantages and limitations of various ML approaches relating to data size, computational requirements, generalizability, and interpretability, as well as identify opportunities for future advances in the science and management of wildfires within a data science context. In total, to the end of 2019, we identified 300 relevant publications in which the most frequently used ML methods across problem domains included random forests, MaxEnt, artificial neural networks, decision trees, support vector machines, and genetic algorithms. As such, there exists opportunities to apply more current ML methods — including deep learning and agent-based learning — in the wildfire sciences, especially in instances involving very large multivariate datasets. We must recognize, however, that despite the ability of ML models to learn on their own, expertise in wildfire science is necessary to ensure realistic modelling of fire processes across multiple scales, while the complexity of some ML methods such as deep learning requires a dedicated and sophisticated knowledge of their application. Finally, we stress that the wildfire research and management communities play an active role in providing relevant, high-quality, and freely available wildfire data for use by practitioners of ML methods.

SVOIS: Support Vector Oriented Instance Selection for text classification

Text classification is the process of setting records into classes that have already been set up based on what they say. It automatically puts texts in natural languages into categories that have already been set up. Text classification is the most crucial part of text retrieval systems, which find texts based on what the user requests, and text understanding systems, which change the text in some way, like by making summaries, answering questions, or pulling out data. Existing algorithms that use supervised learning to classify text automatically need enough examples to learn well. The algorithms for data mining are used to classify texts, as well as a review of the work that has been done on classifying texts. Design/Methodology/Approach: Data mining algorithms that are used to classify texts were talked about, and studies that looked at how these algorithms were used to classify texts were looked at, with a focus on comparative studies. Findings: No classifier can always do the best job because different datasets and situations lead to different classification accuracy. Implications for Real Life: When using data mining algorithms to classify text documents, it's important to keep in mind that the conditions of the data will affect how well the documents are classified. For this reason, the data should be well organized.