Enhancing Grammar Error Correction with Continuous Knowledge Distillation and Reinforcement Learning

LLMCL-GEC: Advancing grammatical error correction with LLM-driven curriculum learning

The correction of grammatical errors in natural language processing is a crucial task as it aims to enhance the accuracy and intelligibility of written language. However, developing a grammatical error correction (GEC) framework for low-resource languages presents significant challenges due to the lack of available training data. This article proposes a novel GEC framework for low-resource languages, using Arabic as a case study. To generate more training data, we propose a semi-supervised confusion method called the equal distribution of synthetic errors (EDSE), which generates a wide range of parallel training data. Additionally, this article addresses two limitations of the classical seq2seq GEC model, which are unbalanced outputs due to the unidirectional decoder and exposure bias during inference. To overcome these limitations, we apply a knowledge distillation technique from neural machine translation. This method utilizes two decoders, a forward decoder right-to-left and a backward decoder left-to-right, and measures their agreement using Kullback-Leibler divergence as a regularization term. The experimental results on two benchmarks demonstrate that our proposed framework outperforms the Transformer baseline and two widely used bidirectional decoding techniques, namely asynchronous and synchronous bidirectional decoding. Furthermore, the proposed framework reported the highest F1 score, and generating synthetic data using the equal distribution technique for syntactic errors resulted in a significant improvement in performance. These findings demonstrate the effectiveness of the proposed framework for improving grammatical error correction for low-resource languages, particularly for the Arabic language.

The advances in deep learning have led to great achievements in many Natural Language Processing (NLP) tasks. With the nature of language, i.e., sequential data, most NLP tasks can be framed into the sequence learning framework, such as text generation. As one of the most important foundations for modern NLP techniques, autoregressive generation models have achieved dominant performance in a great deal of NLP tasks. Therefore, this thesis emphasizes improving the autoregressive generation model for different NLP tasks. While many tasks can naturally fit into the sequence learning framework, some of them, e.g., building discourse parsing tree, require sophisticated designs to fit into neural models. Therefore, this thesis firstly emphasizes a novel unified framework for discourse parsing, which builds a discourse tree in a top-down depth-first manner, and it frames the task as an autoregressive generation task with the goal of each step being the prediction of the node position given a piece of text. The proposed approach is proven effective with extensive empirical experiments. In addition, I extend the above framework by proposing a hierarchical decoder, which leverages the information from parents and siblings of the nodes that are currently processed. The proposed decoder utilizes the nature of the tree structure and further improves the experiment performance on both discourse parsing and dependency parsing tasks. On the other hand, the de facto strategies, i.e., cross entropy loss and teacher forcing, for training the autoregressive generation models have been shown problematic in certain aspects. For example, cross entropy loss, which is one of the widely leveraged training objective functions, often leads to text degeneration in text generation, and teacher forcing suffers from the exposure bias problem, where there exists a mismatch between the training and testing setup. For text degeneration, I introduce a class of diminishing attentions, which enforces the submodularity of the coverage calculated by cross attention in the sequence-to-sequence model. The proposed diminishing attentions achieve notable improvement on several neural text generation tasks, including text summarization, machine translation, and image paragraph generation. Further, I propose a novel training objective, ScaleGrad, to replace cross entropy, which significantly reduces the degeneration problem in different text generation tasks. In fact, ScaleGrad can be extended to problems beyond text degeneration. It provides wide flexibility to inject different inductive biases into the text generation model by directly modifying the gradient information in the output layer. Next, for the exposure bias problem, this thesis introduces a novel type of scheduled sampling based on training accuracy, which requires only minimal hyper-parameter tuning compared to existing scheduled sampling methods. Additionally, a novel imitation loss is proposed to further enforce the model’s generative behavior to match the teacher-forced behavior. Moreover, this thesis demonstrates that reducing exposure bias can improve the robustness of language models against repetition and toxic errors.

Text classification is a basic task in natural language processing. In 2018, the BERT(Bidirectional Encoder Representation from Transformers) was proposed. This model greatly improves the effect of natural language processing related tasks. However, the model size of the pre-trained language model is large, and because of its huge network structure, the predict time is long. In order to solve these problems, an improved model that uses knowledge distillation and adversarial perturbation is proposed. In the training phase, RoBERTa-wwm-ext is used as the teacher model, the joint model of Text-CNN and Text-RCNN is used as a student model, combined with label smoothing and adversarial perturbation method to improve the classification accuracy of the student model, compared with Text-CNN on two datasets, is improved by 1.91% and 1.21% respectively. Using the student model to classify texts has the advantages of easy for deployment and taking less prediction time while obtaining ideal accuracy.

Text extraction is an important part of natural language processing (NLP) tasks. Most NLP tasks like text classification, machine translation, text-to-speech, text-based language identification, text summarization, and named-entity recognition involve the use of textual data. Such data is limited for low-resourced languages making it difficult to experiment advanced NLP techniques on these languages. This paper presents a Python-based toolkit for text analysis and text extraction from different types of images, documents, and audio files. The toolkit is built as a library that has functions that can be imported and utilized for text extraction.

Long Short-Term Memory (LSTM) networks are unique to exercise data in its memory cell with long-term memory as Natural Language Processing (NLP) tasks have inklings of intensive time and computational power due to their complex structures like magnitude language model Transformer required to pre-train and learn billions of data performing different NLP tasks. In this paper, a dynamic chaotic model is proposed for the objective of transforming neurons states in network with neural dynamic characteristics by restructuring LSTM as Chaotic Neural Oscillatory-Long-Short Term Memory (CNO-LSTM), where neurons in LSTM memory cells are weighed in substitutes by oscillatory neurons to speed up computational training of language model and improve text classification accuracy for real-world applications. From the implementation perspective, five popular datasets of general text classification including binary, multi classification and multi-label classification are used to compare with mainstream baseline models on NLP tasks. Results showed that the performance of CNO-LSTM, a simplified model structure and oscillatory neurons state in exercising different types of text classification tasks are above baseline models in terms of evaluation index such as Accuracy, Precision, Recall and F1. The main contributions are time reduction and improved accuracy. It achieved approximately 46.76% of the highest reduction training time and 2.55% accuracy compared with vanilla LSTM model. Further, it achieved approximately 35.86% in time reduction compared with attention model without oscillatory indicating that the model restructure has reduced GPU dependency to improve training accuracy.

Towards a Novel Weakly Supervised Joint Approach of Named Entity Recognition and Normalization for Noisy Text

Generative large language models (LLMs) are a subset of transformers-based neural network architecture models. LLMs have successfully leveraged a combination of an increased number of parameters, improvements in computational efficiency, and large pre-training datasets to perform a wide spectrum of natural language processing (NLP) tasks. Using a few examples (few-shot) or no examples (zero-shot) for prompt-tuning has enabled LLMs to achieve state-of-the-art performance in a broad range of NLP applications. This article by the American Medical Informatics Association (AMIA) NLP Working Group characterizes the opportunities, challenges, and best practices for our community to leverage and advance the integration of LLMs in downstream NLP applications effectively. This can be accomplished through a variety of approaches, including augmented prompting, instruction prompt tuning, and reinforcement learning from human feedback (RLHF). Our focus is on making LLMs accessible to the broader biomedical informatics community, including clinicians and researchers who may be unfamiliar with NLP. Additionally, NLP practitioners may gain insight from the described best practices. We focus on 3 broad categories of NLP tasks, namely natural language understanding, natural language inferencing, and natural language generation. We review the emerging trends in prompt tuning, instruction fine-tuning, and evaluation metrics used for LLMs while drawing attention to several issues that impact biomedical NLP applications, including falsehoods in generated text (confabulation/hallucinations), toxicity, and dataset contamination leading to overfitting. We also review potential approaches to address some of these current challenges in LLMs, such as chain of thought prompting, and the phenomena of emergent capabilities observed in LLMs that can be leveraged to address complex NLP challenge in biomedical applications.

This study aims to explore Korean students’ pragmatic and grammatical awareness with attention to whether there are any differences between college and university students in their level of pragmatic and grammatical awareness. For this purpose, this study employs a language awareness task, which covers three sub-tasks: error recognition, error severity ratings, and error correction. Correlation analyses show that there was a positive correlation between the college group’s grammatical error correction and their pragmatic error correction and between the university group’s pragmatic error recognition and correction. The results also show that the college group recognized pragmatic errors better than grammatical ones and that the university group outperformed the college group at grammatical error recognition. The analysis also reveals that both college and university groups rated pragmatic errors as more severe than grammatical ones. However, the error correction task reveals the opposite, showing that the two learner groups performed grammatical error correction much better than pragmatic error correction. On the basis of the results, pedagogical implications are provided to help learners promote their pragmatic and grammatical awareness.

Chinese text error correction (CTEC), encompassing both Spelling Error Correction (CSC) and Grammar Error Correction (CGED), represents a core task in natural language processing (NLP). When applied to vertical domains like the legal field and multi-instruction scenarios, existing models frequently encounter significant challenges including low correction accuracy, prevalent overcorrection, and unintended semantic distortion. Building upon an analysis of 100 legal text samples and the CGED2020 dataset, this study pinpoints two primary failure mechanisms plaguing current approaches. Firstly, these models often violate fundamental correction requirements such as semantic preservation, similar-shaped or homophonic character recognition, and proper noun standardization, resulting in a notably low baseline accuracy of merely 17.00%. Secondly, the phenomenon of "error stacking" emerges due to the unordered execution of multiple correction instructions. To mitigate these issues, this paper introduces a "single-input–dual-model progressive correction" framework. Within this architecture, Model 1 acts as a filter for explicit errors, whereas Model 2 performs more in-depth correction through meticulous comparison and verification. Experimental evaluations demonstrate that the proposed framework achieves superior performance compared to common large models like ChatGLM3-6B and Qwen-4B. Specifically, on legal texts, it elevates the F0.5 score by 28.3% while successfully constraining the overcorrection rate below 5%.

Text error correction is an essential task in natural language processing (NLP) that focuses on automatically identifying and correcting errors in written text. With the increasing amount of digital text in both Chinese and English, errors such as typos, grammatical mistakes, and contextual ambiguities have become more prominent, affecting readability and communication. Over the years, various models and methodologies have been developed to tackle these challenges, evolving from traditional rule- based systems to more advanced statistical and machine learning-based approaches. This paper presents an overview of the current research status in Chinese and English text error corrections. By analyzing several papers involving Chinese and English error correction models, the article points out the types of errors such as spelling, grammatical and semantic errors included in text error correction and their basic elements. It discusses the advantages and limitations of the latest approaches including rule-based models, statistical and deep learning methods. The development potential of deep learning and big modeling techniques in the field of text error correction is shown. These findings help to advance automated error correction systems and their facilitation in the real world.

In this paper, we present the preliminary results on the analysis of deep learning terms used for natural language processing (NLP) tasks. We propose a statistical analysis of papers published from 2012 to 2015 in the main ACL conferences. Our aim is investigating which DL term, and consequently which DL method, is mostly used for each specific NLP task, since its introduction in the field. In order to do this, our first contribution is the development of two terminological lists, referring respectively to DL methods for text analysis and NLP tasks. The list of deep learning terms contains 41 terms and acronyms, as well as a NLP term list contains 145 terms and acronyms. From our corpus, the frequencies of various terms have been extracted with respect to the ACL conference and the publication year. After the preliminary data analysis, we decided to restrict the extraction process to abstract texts. We applied multivariate techniques called correspondence analysis in order to visualize and evaluate the joint behavior of our variables.

In this study, we explore the potential of machine learning (ML) models after different text representation methods on the balanced IMDB dataset, which is widely regarded as a gold standard in sentiment analysis, one of the Natural Language processing (NLP) tasks. On the open source IMDB movie reviews dataset, we first undertake data cleaning and text representation with data preprocessing steps. Then, we apply sentiment classification using different ML models. In order to evaluate the models, we used precision (P), recall (R), F1-score (F1), and area under curve (AUC), as well as receiver operating characteristic (ROC). It is worth noting that text feature extraction with Bidirectional Encoder Representations from Transformers (BERT) provided the highest performance in all models, with the SVM model offering particularly promising results. In this model, we observed the following results: ACC 0.9033, F1 0.9308, R 0.9015, R 0.9015, P 0.9072, AUC 0.9638, and ROC 0.96. These findings suggest that NLP techniques and, in particular, machine learning models that employ BERT may offer high levels of accuracy and reliability in text classification problems. It would be beneficial for future studies to validate these findings using BERT on different NLP tasks. This would help to evaluate the effectiveness and applicability of the models in practice.

For second language learners, a reliable and effective Grammatical Error Correction (GEC) system is imperative, since it can be used as an auxiliary assistant for errors correction and helps learners improve their writing ability. Researchers have paid more emphasis on this task with deep learning methods. Better results were achieved on the standard benchmark datasets compared to traditional rule based approaches. We treat GEC as a special translation problem which translates wrong sentences into correct ones like other former works. In this paper, we propose a novel correction system based on sequence to sequence (Seq2Seq) architecture with residual connection and semantically conditioned LSTM (SC-LSTM), incorporating edit operations as special semantic information. Our model further improves the performance of neural machine translation model for GEC and achieves state-of-the-art \(F _{0.5}\)-score on standard test data named CoNLL-2014 compared with other methods that without any re-rank approach.

Work orders, equipment information, technical records and best practices documents contain within them a wealth of insights related to Equipment Maintenance which can be unlocked with Natural Language Processing tasks like classification, clustering, named entity recognition or part of speech tagging. But obtaining large enough labelled data sets in Equipment Maintenance domain manually is prohibitive and very expensive. This lack of labeled Equipment Maintenance data can be overcome with Large Language Models (LLMs) such as GPT-3, BERT that are pretrained transformer networks, considered state-of-the-art when it comes to Natural Language Processing (NLP) tasks. However, the vocabulary understood by these LLMs are mostly from English Language and need to be fine-tuned to understand industry and organization specific vocabulary and acronyms. This paper explores the potential of a domain specific LLM model for oil and gas industries. Data that are of good quality and that provide a comprehensive overview of industry are collected. This corpus of text contains documents like work orders, equipment data and technical documents. A custom tokenizer is trained on this data to identify domain specific terminology. A comparative study is done with other off-the-shelf tokenizers: BERT and RoBERTa, to compare the effectiveness of the tokenization. With millions of work orders and equipment documents, training pipelines had to parallelized so that training can occur on multiple GPUs. A comprehensive study of multiple training methods is done in this paper. Model and tokenizer developed were packaged and archived to be consumed in machine learning pipelines to specific use-cases across the organization. For an organization adopting digital transformation, the availability of an organization specific LLM is an enabler to extract insights from millions of documents containing free text. The applicability of such models spans across multiple disciplines like Maintenance, Reliability, Safety etc. and streamlines the development of highly accurate and robust text analytics.