Deep Active-Self Learning Applied to Named Entity Recognition

Document classification is a prevalent task in Natural Language Processing (NLP), which has an extensive range of applications in the biomedical domains such as biomedical literature indexing, automatic diagnosis codes assignment, tweets classification for public health topics, and patient safety reports classification. Nevertheless, manual classification of biomedical articles published every year into specific predefined categories becomes a cumbersome task. Hence, building an automatic document classification for biomedical databases emerges as a significant task among the scientific community. In recent years, Deep Learning (DL) models like Deep Neural Networks (DNN), Convolution Neural Networks (CNN), Recurrent Neural Networks (RNN), and Ensemble Deep Learning models are widely used in the area of text document classification for better classification performance compared to Machine Learning (ML) algorithms. The major advantage of using DL models in document classification is that it provides rich semantic and grammatical information for document representation through pre-trained word embedding. Hence, this paper investigates the deployment of the various state-of-the-art DL based classification models in automatic classification of benchmark biomedical datasets. Finally, the performance of all the aforementioned constitutional classifiers is compared and evaluated through the well-defined performance evaluation metrics such as accuracy, precision, recall, and f1-measure.

The advances in deep learning have led to great achievements in many Natural Language Processing (NLP) tasks. With the nature of language, i.e., sequential data, most NLP tasks can be framed into the sequence learning framework, such as text generation. As one of the most important foundations for modern NLP techniques, autoregressive generation models have achieved dominant performance in a great deal of NLP tasks. Therefore, this thesis emphasizes improving the autoregressive generation model for different NLP tasks. While many tasks can naturally fit into the sequence learning framework, some of them, e.g., building discourse parsing tree, require sophisticated designs to fit into neural models. Therefore, this thesis firstly emphasizes a novel unified framework for discourse parsing, which builds a discourse tree in a top-down depth-first manner, and it frames the task as an autoregressive generation task with the goal of each step being the prediction of the node position given a piece of text. The proposed approach is proven effective with extensive empirical experiments. In addition, I extend the above framework by proposing a hierarchical decoder, which leverages the information from parents and siblings of the nodes that are currently processed. The proposed decoder utilizes the nature of the tree structure and further improves the experiment performance on both discourse parsing and dependency parsing tasks. On the other hand, the de facto strategies, i.e., cross entropy loss and teacher forcing, for training the autoregressive generation models have been shown problematic in certain aspects. For example, cross entropy loss, which is one of the widely leveraged training objective functions, often leads to text degeneration in text generation, and teacher forcing suffers from the exposure bias problem, where there exists a mismatch between the training and testing setup. For text degeneration, I introduce a class of diminishing attentions, which enforces the submodularity of the coverage calculated by cross attention in the sequence-to-sequence model. The proposed diminishing attentions achieve notable improvement on several neural text generation tasks, including text summarization, machine translation, and image paragraph generation. Further, I propose a novel training objective, ScaleGrad, to replace cross entropy, which significantly reduces the degeneration problem in different text generation tasks. In fact, ScaleGrad can be extended to problems beyond text degeneration. It provides wide flexibility to inject different inductive biases into the text generation model by directly modifying the gradient information in the output layer. Next, for the exposure bias problem, this thesis introduces a novel type of scheduled sampling based on training accuracy, which requires only minimal hyper-parameter tuning compared to existing scheduled sampling methods. Additionally, a novel imitation loss is proposed to further enforce the model’s generative behavior to match the teacher-forced behavior. Moreover, this thesis demonstrates that reducing exposure bias can improve the robustness of language models against repetition and toxic errors.

Many tasks in natural language processing like offensive, toxic, or emotional text classification are subjective by nature. Humans tend to perceive textual content in their own individual way. Existing methods commonly rely on the agreed output values, the same for all consumers. Here, we propose personalized solutions to subjective tasks. Our four new deep learning models take into account not only the content but also the specificity of a given human. The models represent different approaches to learning the representation and processing data about text readers. The experiments were carried out on four datasets: Wikipedia discussion texts labelled with attack, aggression, and toxicity, as well as opinions annotated with ten numerical emotional categories. Emotional data was considered as multivariate regression (multitask), whereas Wikipedia data as independent classifications. All our models based on human biases and their representations significantly improve the prediction quality in subjective tasks evaluated from the individual’s perspective.

Machine learning methods, including Deep learnings are popular for data processing. Deep learning methods have shown great promise in applied in natural language processing (NLP) tasks. Text classification is a supervised machine learning task that involves labelled documents to train classifier. Previous works involving machine learning and deep learning methods for text classification have been tested with relatively small- sized data instances. In this paper, we compared the performance of the machine leaning and deep learning algorithms in text classification task. This paper also studied, explored and compared the scalability of these methods with respect to bigger data instances. We used support vector machines (SVM), Logistic regression, Random forest and Naive Bayes Machine leaning algorithms and convolutional neural network (CNN) deep learning method. The task involved a multi-class classification problem involving six (6) classes consisting of six thousand (6,000) data instances with an average of 20 sentences in each data instance. The CNN deep leaning algorithm outperformed all the machine learning algorithms, achieving an accuracy of over 85%. This is because the filter weights are leaning are updated in backward propagation in each epoch, hence, this result in better result compared to the traditional methods.

Text-based Question Answering (QA) is an essential task in natural language processing that aims to provide relevant and accurate answers to users’ queries. Traditional approaches to QA relied on rule-based systems and hand-crafted features. However, recent advancements in information retrieval and deep learning have shown promising results in improving the performance of QA systems. The combination of IR and deep learning techniques has led to significant improvements in QA performance. Hybrid models, such as the bi-encoder and tri-encoder architectures, have been proposed to leverage the strengths of both IR and deep learning models. Additionally, pre-training techniques, such as BERT and RoBERTa, have been shown to improve the performance of QA systems by providing pre-trained contextualized word embeddings. This study provides a brief overview of the QA system and its different domains and subdomains. This study also provides review of different proposed models and different datasets available for the task and evaluate their performance using some performance metrics. A comparison between various techniques are employed using the obtained result.

Sentiment analysis, a critical task in natural language processing, aims to automatically identify and classify the sentiment expressed in textual data. Aspect-level sentiment analysis focuses on determining sentiment at a more granular level, targeting specific aspects or features within a piece of text. In this paper, we explore various techniques for sentiment analysis, including traditional machine learning approaches and state-of-the-art deep learning models. Additionally, deep learning techniques has been utilized to identifying and extracting specific aspects from text, addressing aspect-level ambiguity, and capturing nuanced sentiments for each aspect. These datasets are valuable for conducting aspect-level sentiment analysis. In this article, we explore a language model based on pre-trained deep neural networks. This model can analyze sequences of text to classify sentiments as positive, negative, or neutral without explicit human labeling. To evaluate these models, data from Twitter's US airlines sentiment database was utilized. Experiments on this dataset reveal that the BERT, RoBERTA and DistilBERT model outperforms than the ML based model in accuracy and is more efficient in terms of training time. Notably, our findings showcase significant advancements over previous state-of-the-art methods that rely on supervised feature learning, bridging existing gaps in sentiment analysis methodologies. Our findings shed light on the advancements and challenges in sentiment analysis, offering insights for future research directions and practical applications in areas such as customer feedback analysis, social media monitoring, and opinion mining.

• Figurative language styles are analyzed at lexical, syntactic, semantic, discourse-level, and pragmatic NLP levels • Discourse-level styles dominate due to sarcasm’s prevalence in social media • Arabic ranks second in dataset count, but lacks stylistic diversity coverage • Embedding-based features lead performance, especially in semantic tasks • Publicly available datasets remain scarce for pragmatic and syntactic styles Figurative language detection has emerged as a critical task in natural language processing (NLP), enabling machines to comprehend non-literal expressions such as metaphor, irony, and sarcasm. This study presents a systematic literature review with a multilevel analytical framework, examining figurative language across lexical, syntactic, semantic, discourse, and pragmatic levels. We investigate the interplay between feature engineering, model architectures, and annotation strategies across different languages, analyzing datasets, linguistic resources, and evaluation metrics. Special attention is given to morphologically rich and low-resource languages, where deep learning dominates but rule-based and hybrid approaches remain relevant. Our findings indicate that deep learning models–especially transformer-based architectures like BERT and RoBERTa–consistently outperform other approaches, particularly in semantic and discourse-level tasks, due to their ability to capture context-rich and abstract patterns. However, these models often lack interpretability, raising concerns about transparency. Additional challenges include inconsistencies in annotation practices, class imbalance between figurative and literal instances, and limited data coverage for under-resourced languages. The absence of standardized evaluation metrics further complicates cross-study comparison, especially when diverse figurative language styles are involved. By structuring our analysis through linguistic and computational dimensions, this review aims to facilitate the development of more robust, inclusive, and explainable figurative language detection systems.

Deep learning algorithms can be used to design systems that report data on patients and deliver warnings to medical applications or electronic health information if there are changes in the patient's health. These systems could be created using deep Learning. This helps verify that patients get the proper effective care at the proper time for each specific patient. A healthcare decision support system was presented using the Internet of Things and deep learning methods. In the proposed system, we examined the capability of integrating deep learning technology into automatic diagnosis and IoT capabilities for faster message exchange over the Internet. We have selected the suitable Neural Network structure (number of best-hidden layers and activation function classes) to construct the e-health system. In addition, the e-health system relied on data from doctors to understand the Neural Network. In the validation method, the total evaluation of the proposed healthcare system for diagnostics provides dependability under various patient conditions. Based on evaluation and simulation findings, a dual hidden layer of feed-forward NN and its neurons store the tanh function more effectively than other NN. To overcome challenges, this study will integrate artificial intelligence with IoT. This study aims to determine the NN's optimal layer counts and activation function variations.

What's New in Spine Surgery.

Objectives: Language detection is the process of identifying a language associated with a text. The proposed system aims to detect the Dravidian language that is associated with the given text using different machine learning and deep learning algorithms. The paper presents an empirical analysis of the results obtained using the different models. It also aims to evaluate the performance of a language agnostic model for the purpose of language detection. Method: An empirical analysis of Dravidian language identification in social media text using machine learning and deep learning approaches with k-fold cross validation has been implemented. The identification of Dravidian languages, including Tamil, Malayalam, Tamil Code Mix, and Malayalam Code Mix, is performed using both machine learning (ML) and deep learning algorithms. The machine learning algorithms used for language detection are Naive Bayes (NB), Multinomial Logistic Regression (MLR), Support Vector Machine (SVM), and Random Forest (RF). The supervised Deep Learning (DL) models used include BERT, mBERT and language agnostic models. Findings: The language agnostic model outperform all other models considering the task of language detection in Dravidian languages. The results of both the ML and DL models are analyzed empirically with performance measures like accuracy, precision, recall, and f1-score. The accuracy associated with different machine learning algorithms varies from 85% to 89%. It is evident from the experimental result that the deep learning model outperformed with an accuracy of 98%. Novelty: The proposed system emphasizes on the use of the language agnostic model to implement the process of detecting Dravidian languages associated with the given text which provides a promising result of 98% accuracy which is higher than the existing methodologies. Keywords: Language, Machine learning, Deep learning, Transformer model, Encoder, Decoder

Recent technological advancements allow deep learning to be employed in practically every aspect of life. Because deep learning techniques are so precise, they can be used in medicine to classify and detect various diseases. The coronavirus (SARSCoV2) epidemic has recently affected global health systems. One of the ways to diagnose SARSCoV2 is with a chest X-ray. This paper proposes a deep learning technique to distinguish SARSCoV2 positive and normal cases. In this study, we fine-tuned the deep learning models and hyperparameters, and the fine-tuned deep learning models performed significantly better. To classify X-ray images, we developed a system based on deep learning algorithms that includes five models: Xception, VGG19, ResNet50, DenseNet121, and Inception. We offer deep learning models and algorithms that have been trained and evaluated to support medical efforts and reduce medical staff workload when dealing with SARSCoV2. In addition, the classification model that was proposed yields positive results because it makes use of accurate classification of the SARSCoV2 disease based on medical images. Additionally, the performance of our proposed CNN classification method for medical imaging was evaluated using various edge-based neural networks. The accuracy of tertiary classification with CNN will decrease as the number of classes in the training network grows. In tertiary classification, which includes normal and SARSCoV2 positive images, the proposed model achieved a 0.9897 accuracy. The proposed algorithm achieves a high level of classification accuracy when using the DenseNet121 model for binary classification.

This paper has conducted a comprehensive review of more than 30 machine learning (ML) and deep learning (DL) algorithms across four main categories: supervised, unsupervised, reinforcement and deep learning. The discussion focused on the description of each algorithm, the application of mathematical foundations in their implementation, and their relevance to various practical applications. Through this analysis, the critical importance of underlying mathematical and statistical concepts, such as optimization, probability theory, and geometry, in the development of ML and DL models was highlighted A key conclusion of the paper is the diversity and adaptability of ML and DL algorithms in a wide range of fields, including computer vision, natural language processing, robotics, and medicine. This analysis underscores how rapidly the field is advancing, marked by the evolution of complex models that have transformed machine learning and adaptive capabilities.However, the work also recognizes the challenges and limitations that exist in the design and implementation of these algorithms, including overfitting, interpretability, and computational consumption. These challenges underscore the need for continued research and development to optimize and create new techniques that overcome these barriers.Looking to the future, the work suggests a focus on developing even more generative models.

Background Machine learning (ML) algorithm support vector machine (SVM) performed better than Thrombolysis in Myocardial Infarction (TIMI) score for ASIAN STEMI patients. However, Deep Learning (DL) effectiveness in the multiethnic ASIAN population has yet to be determined. DL has automatic learning of the feature from a given dataset without the need to conduct feature selection. However, the selected features by the algorithm is black box. Identifying features associated with mortality is essential to recognize characteristics of patients with high risk for better patient management. Purpose To develop a DL algorithm for in-hospital mortality in multiethnic STEMI patients using predictors identified from the SVM algorithm. To investigate DL performance constructed using predictors from SVM feature extraction and expert-recommended predictors. Methods We constructed four algorithms; a) DL and SVM algorithms with predictors identified from the SVM variable importance b) DL and SVM using predictors based on expert recommendation. We used registry data from the National Cardiovascular Disease Database of 11397 patient's. Fifty parameters including demographics, cardiovascular risk, medications and clinical variables were considered. The Area under the curve (AUC) is the performance evaluation metric. Algorithms were validated against the TIMI and tested using the same validation data. SVM variable importance with backward elimination was used to select and rank important variables. Results DL algorithms outperform SVM and TIMI on the validation dataset; i) DL with SVM selected predictors (15 predictors, AUC = 0.97), ii) DL with expert-recommended predictors (16 predictors, AUC = 0.96), iii) SVM with selected predictors (15 predictors, AUC = 0.92), iv) SVM with expert-recommended predictors (AUC = 0.89) and TIMI (AUC = 0.82). Common predictors across SVM feature selection, expert-recommendation and TIMI are: age, heart rate, Killip class, fasting blood glucose, systolic blood pressure, comorbid diseases and ST-elevation. SVM feature selection also identified diuretics, PCI and pharmacotherapy drugs as predictors that improve mortality prediction in STEMI patients. Our findings suggest that the TIMI score underestimates patients risk of mortality. DL algorithm using selected predictors classified 35% of nonsurvival patients as high risk (risk probabilities >50%) compared to only 12.7% nonsurvival patients by TIMI (score >5) (Figure below). Conclusions In the ASIAN population, patients with STEMI can be better classified using the DL algorithm compared to the ML and TIMI score. Combining ML feature selection with DL allows the identification of distinct factors in a unique ASIAN population for better mortality prediction than relying solely on an expert recommendation as it is a very subjective approach. Continuous validation on population-specific algorithms using DL and ML is needed before implementing in a real clinical setting. Funding Acknowledgement Type of funding sources: Public grant(s) – National budget only. Main funding source(s): Technology Development Fund 1 TIMI performance on validation setDL performance on validation set

Text error correction is an essential task in natural language processing (NLP) that focuses on automatically identifying and correcting errors in written text. With the increasing amount of digital text in both Chinese and English, errors such as typos, grammatical mistakes, and contextual ambiguities have become more prominent, affecting readability and communication. Over the years, various models and methodologies have been developed to tackle these challenges, evolving from traditional rule- based systems to more advanced statistical and machine learning-based approaches. This paper presents an overview of the current research status in Chinese and English text error corrections. By analyzing several papers involving Chinese and English error correction models, the article points out the types of errors such as spelling, grammatical and semantic errors included in text error correction and their basic elements. It discusses the advantages and limitations of the latest approaches including rule-based models, statistical and deep learning methods. The development potential of deep learning and big modeling techniques in the field of text error correction is shown. These findings help to advance automated error correction systems and their facilitation in the real world.

Speech emotion recognition is a challenging task in natural language processing. It relies heavily on the effectiveness of speech features and acoustic models. However, existing acoustic models may not handle speech emotion recognition efficiently for their built-in limitations. In this work, a novel deep-learning acoustic model called attention-based skip convolution bi-directional long short-term memory, abbreviated as SCBAMM, is proposed to recognize speech emotion. It has eight hidden layers, namely, two dense layers, convolutional layer, skip layer, mask layer, Bi-LSTM layer, attention layer, and pooling layer. SCBAMM makes better use of spatiotemporal information and captures emotion-related features more effectively. In addition, it solves the problems of gradient exploding and gradient vanishing in deep learning to some extent. On the databases EMO-DB and CASIA, the proposed model SCBAMM achieves an accuracy rate of 94.58% and 72.50%, respectively. As far as we know, compared with peer models, this is the best accuracy rate.