Text grouping: a comprehensive guide

In multi-label text classification, the central aim is to associate an array of descriptive labels for a better understanding of the text. There are three main challenges in doing multi-label text classification (i) a large number of text (input) features, (ii) the underlying implicit relationship between input features and output labels, and (iii) an implicit inter-label dependency. In traditional approaches to multi-label classification, these problems are not being addressed collectively. A feature selection strategy that inherently uses local features to discriminate a class and similarly global features that can distinctly separate classes can be very effective for multi-label classification. In this research, we perform a feature selection and ranking strategy based on local and global features. A Naïve Bayes classifier is being used using a combination of these two -feature sets, it is compared with the baseline implemented with the term frequency-inverse document frequency (TF-IDF). A series of experiments have been carried out on standard multi-label text datasets, using evaluation metrics like Hamming loss, Subset Accuracy and Micro/Macro F1 scores, and encouraging results are obtained.

Malay social media text is a text written on social media networks like Twitter. Commonly, this text comprises non-standard words, filled with dialects, foreign languages, word abbreviations, grammatical neglect, spelling errors, and many more. It is well known that this type of text is difficult to process due to its high noise and distinct text structure. Such problems can be resolved using rigorous text normalization, which is critical before any technique can be implemented and evaluated on social media text. In this paper, an improved normalization method towards Malay social media text was proposed by converting non-standard Malay words using a rule-based model. The method normalizes common language words often used by Malaysian users, such as non-standard Malay (like dialect and slangs), Romanized Arabic, and English words. Thus, a Malay text normalizer was proposed using a set of rules that extend across different domains of natural language processing (NLP) and is expected to address the challenges of processing Malay social media text. This study implements the proposed Malay text normalizer in a Part-of-Speech (POS) tagging application to evaluate the normalizer’s performance. The implementation demonstrates a substantial improvement in the POS tagging efficiency over several pre-processing stages, with an improvement of accuracy up to 31.8%. The increase of accuracy in the POS tagging indicates two main points. First, the Malay text normalizer’s rules improve the performance of a Malay text normalizer on social media text. Second, our proposed Malay text normalizer has successfully improved the POS tagging percentage and demonstrates the importance of normalized pre-processing in any NLP application.

Prediction of the stage of cancer plays an important role in planning the course of treatment and has been largely reliant on imaging tools which do not capture molecular events that cause cancer progression. Gene-expression data-based analyses are able to identify these events, allowing RNA-sequence and microarray cancer data to be used for cancer analyses. Breast cancer is the most common cancer worldwide, and is classified into four stages - stages 1, 2, 3, and 4 [2]. While machine learning models have previously been explored to perform stage classification with limited success, multi-class stage classification has not had significant progress. There is a need for improved multi-class classification models, such as by investigating deep learning models. Gene-expression-based cancer data is characterised by the small size of available datasets, class imbalance, and high dimensionality. Class balancing methods must be applied to the dataset. Since all the genes are not necessary for stage prediction, retaining only the necessary genes can improve classification accuracy. The breast cancer samples are to be classified into 4 classes of stages 1 to 4. Invasive ductal carcinoma breast cancer samples are obtained from The Cancer Genome Atlas (TCGA) and Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) datasets and combined. Two class balancing techniques are explored, synthetic minority oversampling technique (SMOTE) and SMOTE followed by random undersampling. A hybrid feature selection pipeline is proposed, with three pipelines explored involving combinations of filter and embedded feature selection methods: Pipeline 1 - minimum-redundancy maximum-relevancy (mRMR) and correlation feature selection (CFS), Pipeline 2 - mRMR, mutual information (MI) and CFS, and Pipeline 3 - mRMR and support vector machine-recursive feature elimination (SVM-RFE). The classification is done using deep learning models, namely deep neural network, convolutional neural network, recurrent neural network, a modified deep neural network, and an AutoKeras generated model. Classification performance post class-balancing and various feature selection techniques show marked improvement over classification prior to feature selection. The best multiclass classification was found to be by a deep neural network post SMOTE and random undersampling, and feature selection using mRMR and recursive feature elimination, with a Cohen-Kappa score of 0.303 and a classification accuracy of 53.1%. For binary classification into early and late-stage cancer, the best performance is obtained by a modified deep neural network (DNN) post SMOTE and random undersampling, and feature selection using mRMR and recursive feature elimination, with an accuracy of 81.0% and a Cohen-Kappa score (CKS) of 0.280. This pipeline also showed improved multiclass classification performance on neuroblastoma cancer data, with a best area under the receiver operating characteristic (auROC) curve score of 0.872, as compared to 0.71 obtained in previous work, an improvement of 22.81%. The results and analysis reveal that feature selection techniques play a vital role in gene-expression data-based classification, and the proposed hybrid feature selection pipeline improves classification performance. Multi-class classification is possible using deep learning models, though further improvement particularly in late-stage classification is necessary and should be explored further.

Automatic text summarization come under the domains of natural language processing, machine learning and information retrieval. As the abundance of textual information grows so does the need for summarising it. Here we consider a method for single document summarization using natural language processing tools and an extractive approach based on sentence similarity and document context. The technique uses a weighted undirected graph based scoring on paragraphs and a word frequency based scoring system on the entire document to obtain summaries. The proposed method is validated through experiments and the results are promising.

The domain of Natural Language Processing covers various tasks, such as classification, text generation, and language model. The data processed using word embeddings, or vectorizers, is then trained using Machine Learning and Deep Learning algorithms. In order to observe the tradeoff between both these types of algorithms, with respect to data available, accuracy obtained and other factors, a binary classification is undertaken to distinguish between insincere and regular questions on Quora. A dataset called Quora Insincere Questions Classification was used to train various machine learning and deep learning models. A Bidirectional-Long Short Term Network (LSTM) was trained, with the text processed using Global Vectors for Word Representation (GloVe). Machine Learning algorithms such as Extreme Gradient Boosting classifier, Gaussian Naive Bayes, and Support Vector Classifier (SVC), by using the TF-IDF vectorizer to process the text. This paper also presents an evaluation of the above algorithms on the basis of precision, recall, f1 score metrics.

This article presents a novel conversational artificial intelligence (CAI)-enabled active ideation system as a creative idea generation tool to assist novice product designers in mitigating the initial latency and ideation bottlenecks that are commonly observed. It is a dynamic, interactive, and contextually responsive approach, actively involving a large language model (LLM) from the domain of natural language processing (NLP) in artificial intelligence (AI) to produce multiple statements of potential ideas for different design problems. Integrating such AI models with ideation creates what we refer to as an active ideation scenario, which helps foster continuous dialog-based interaction, context-sensitive conversation, and prolific idea generation. An empirical study was conducted with 30 novice product designers to generate multiple ideas for given problems using traditional methods and the new CAI-based interface. The ideas generated by both methods were qualitatively evaluated by a panel of experts. The findings demonstrated the relative superiority of the proposed tool for generating prolific, meaningful, novel, and diverse ideas. The interface was enhanced by incorporating a prompt-engineered structured dialog style for each ideation stage to make it uniform and more convenient for the product designers. A pilot study was conducted and the resulting responses of such a structured CAI interface were found to be more succinct and aligned toward the subsequent design stage. The article thus established the rich potential of using generative AI (Gen-AI) for the early ill-structured phase of the creative product design process.

Text categorization serves as a fundamental objective within the domain of Natural Language Processing (NLP). Text data often contains rich structural and contextual information, such as correlations between words, logical relationships between sentences, and diverse contexts, which makes it especially critical to effectively capture and process full-text information. In this paper, a model named BBGC is proposed to improve the capture and processing of full-text information. The model extracts contextual and structural information of text by combining BERT-BiLSTM and Graph Convolutional Network (GCN), respectively, and employs a co-attention mechanism and a gate mechanism to fuse features from different perspectives of the GCN and BERT-BiLSTM, which enhances the attention to key features and integrates features from different sources more effectively. In addition, we conducted experimental validation on four datasets, namely R52, R8, Ohsumed, and MR, with accuracies of 96.65%, 98.72%, 73.83%, and 90.41%, respectively. The experimental results demonstrate the effectiveness of the proposed method in improving the accuracy of text categorization and outperform the existing methods.

Deep Learning techniques have significant progress in the last few years. Different designs and methods of the model have experimented to enhance the performance in various domains of Natural Language Processing (NLP). Machine Translation is a domain of NLP. In this paper, we have reviewed different Encoding-Decoding methods for Neural Machine Translation(NMT). We also, compare these various methods and put forward a past, present and future of Neural Machine Translation.

Stemming vulnerabilities out of a smart contract prior to its deployment is essential to ensure the security of decentralized applications. As such, numerous tools and machine-learning-based methods have been proposed to help detect vulnerabilities in smart contracts. Furthermore, various ways of encoding the smart contracts for analysis have also been proposed. However, the impact of these input methods has not been systematically studied, which is the primary goal of this paper. In this preliminary study, we experimented with four common types of input, including Word2Vec, FastText, Bag-of-Words (BoW), and Term Frequency–Inverse Document Frequency (TF-IDF). To focus on the comparison of these input types, we used the same deep-learning model, i.e., convolutional neural networks, in all experiments. Using a public dataset, we compared the vulnerability detection performance of the four input types both in the binary classification scenarios and the multiclass classification scenario. Our findings show that TF-IDF is the best overall input type among the four. TF-IDF has excellent detection performance in all scenarios: (1) it has the best F1 score and accuracy in binary classifications for all vulnerability types except for the delegate vulnerability where TF-IDF comes in a close second, and (2) it comes in a very close second behind BoW (within 0.8%) in the multiclass classification.

This article addresses the task of automatically assigning peer reviewers based on historical data from previously submitted and reviewed manuscripts. In conventional editorial practice, reviewer selection relies heavily on the subjective judgment of editors, which can lead to delays and inconsistencies in the quality of expert evaluation. The purpose of this study is to demonstrate that simple natural language processing (NLP) models can be used to automate this process in an efficient and transparent manner. The dataset used in this research consists of both published and rejected articles submitted to the Almaz-Antey Air and Space Defense Corporation Journal, enriched with information about the reviewers assigned to each manuscript. Methodologically, the approach relies on basic text preprocessing, including lemmatization, removal of stop words and punctuation, followed by vectorization using bag-of-words (BoW) and Term Frequency-Inverse Document Frequency (TF-IDF) models. Text similarity is calculated via cosine distance between vectorized representations. The core assumption is that a newly submitted manuscript is most similar to an already reviewed one and, therefore, can be assigned to the same reviewers. The results indicate that simple frequency-based models (BoW, TF-IDF) achieve higher accuracy in reviewer assignment (up to 99%) compared to neural network approaches such as Doc2Vec, especially when enhanced with a reviewer co-review graph. The proposed method remains interpretable, requires minimal computational resources, and is fully compatible with office-level computing environments. The model has been shown to perform reliably under class imbalance and is applicable even to relatively small datasets, starting from around 30 manuscripts. However, its generalization to multi-journal editorial systems would require local adaptation, and the task of predicting publication outcomes calls for significantly larger corpora and the use of deep learning architectures. This approach can be seamlessly integrated into digital editorial platforms, contributing to faster decision-making, increased transparency in peer review, and reduced workload for journal staff.

Recurrent Neural Networks (RNNs) have been shown to be valuable for constructing Intrusion Detection Systems (IDSs) for network data. They allow determining if a flow is malicious or not already before it is over, making it possible to take action immediately. However, considering the large number of packets that has to be inspected, for example in cloud/fog and edge computing, the question of computational efficiency arises. We show that by using a novel Reinforcement Learning (RL)-based approach called SparseIDS, we can reduce the number of consumed packets by more than three fourths while keeping classification accuracy high. To minimize the computational expenses of the RL-based sampling we show that a shared neural network can be used for both the classifier and the RL logic. Thus, no additional resources are consumed by the sampling in deployment. Comparing to various other sampling techniques, SparseIDS consistently achieves higher classification accuracy by learning to sample only relevant packets. A major novelty of our RL-based approach is that it can not only skip up to a predefined maximum number of samples like other approaches proposed in the domain of Natural Language Processing but can even skip arbitrarily many packets in one step. This enables saving even more computational resources for long sequences. Inspecting SparseIDS's behavior of choosing packets shows that it adopts different sampling strategies for different attack types and network flows. Finally we build an automatic steering mechanism that can guide SparseIDS in deployment to achieve a desired level of sparsity.

Robots are becoming everyday devices, increasing their interaction with humans. To make human-machine interaction more natural, cognitive features like Visual Voice Activity Detection (VVAD), which can detect whether a person is speaking or not, given visual input of a camera, need to be implemented. Neural networks are state of the art for tasks in Image Processing, Time Series Prediction, Natural Language Processing and other domains. Those Networks require large quantities of labeled data. Currently there are not many datasets for the task of VVAD. In this work we created a large scale dataset called the VVAD-LRS3 dataset, derived by automatic annotations from the LRS3 dataset. The VVAD-LRS3 dataset contains over 44K samples, over three times the next competitive dataset (WildVVAD). We evaluate different baselines on four kinds of features: facial and lip images, and facial and lip landmark features. With a Convolutional Neural Network Long Short Term Memory (CNN LSTM) on facial images an accuracy of 92% was reached on the test set. A study with humans showed that they reach an accuracy of 87.93% on the test set.

Boosting ICD multi-label classification of health records with contextual embeddings and label-granularity

Summary Objectives : Analyze papers published in 2019 within the medical natural language processing (NLP) domain in order to select the best works of the field. Methods : We performed an automatic and manual pre-selection of papers to be reviewed and finally selected the best NLP papers of the year. We also propose an analysis of the content of NLP publications in 2019. Results : Three best papers have been selected this year including the generation of synthetic record texts in Chinese, a method to identify contradictions in the literature, and the BioBERT word representation. Conclusions : The year 2019 was very rich and various NLP issues and topics were addressed by research teams. This shows the will and capacity of researchers to move towards robust and reproducible results. Researchers also prove to be creative in addressing original issues with relevant approaches.

Neural Networks (NNs) have provided state-of-the-art results for many challenging machine learning tasks such as detection, regression and classification across the domains of computer vision, speech recognition and natural language processing. Despite their success, they are often implemented in a frequentist scheme, meaning they are unable to reason about uncertainty in their predictions. This article introduces Bayesian Neural Networks (BNNs) and the seminal research regarding their implementation. Different approximate inference methods are compared, and used to highlight where future research can improve on current methods.