Truth Discovery in Sequence Labels from Crowds

Towards a Novel Weakly Supervised Joint Approach of Named Entity Recognition and Normalization for Noisy Text

Position-aware self-attention based neural sequence labeling

Towards a Novel Weakly Supervised Joint Approach of Named Entity Recognition and Normalization for Noisy Text

Like any other science, research in natural language processing (NLP) depends on the ability to draw correct conclusions from experiments. A key tool for this is statistical significance testing: We use it to judge whether a result provides meaningful, generalizable findings or should be taken with a pinch of salt. When comparing new methods against others, performance metrics often differ by only small amounts, so researchers turn to significance tests to show that improved models are genuinely better. Unfortunately, this reasoning often fails because we choose inappropriate significance tests or carry them out incorrectly, making their outcomes meaningless. Or, the test we use may fail to indicate a significant result when a more appropriate test would find one. NLP researchers must avoid these pitfalls to ensure that their evaluations are sound and ultimately avoid wasting time and money through incorrect conclusions.This book guides NLP researchers through the whole process of significance testing, making it easy to select the right kind of test by matching canonical NLP tasks to specific significance testing procedures. As well as being a handbook for researchers, the book provides theoretical background on significance testing, includes new methods that solve problems with significance tests in the world of deep learning and multidataset benchmarks, and describes the open research problems of significance testing for NLP.The book focuses on the task of comparing one algorithm with another. At the core of this is the p-value, the probability that a difference at least as extreme as the one we observed could occur by chance. If the p-value falls below a predetermined threshold, the result is declared significant. Leaving aside the fundamental limitation of turning the validity of results into a binary question with an arbitrary threshold, to be a valid statistical significance test, the p-value must be computed in the right way. The book describes the two crucial properties of an appropriate significance test: The test must be both valid and powerful. Validity refers to the avoidance of type 1 errors, in which the result is incorrectly declared significant. Common mistakes that lead to type 1 errors include deploying tests that make incorrect assumptions, such as independence between data points. The power of a test refers to its ability to detect a significant result and therefore to avoid type 2 errors. Here, knowledge of the data and experiment must be used to choose a test that makes the correct assumptions. There is a trade-off between validity and power, but for the most common NLP tasks (language modeling, sequence labeling, translation, etc.), there are clear choices of tests that provide a good balance.Beginning with a detailed background on significance testing, the book then shows the reader how to carry out tests for specific NLP tasks. There is a mix of styles, with the first four chapters providing reference material that will be extremely useful to both new and experienced researchers. Here, it is easy to find the material related to a given NLP task. The next two chapters discuss more recent research into the application of significance tests to deep neural networks and for testing across multiple datasets. Alongside open research questions, these later chapters provide clear guidelines on how to apply the proposed methods. It is this mix of background material and reference guidelines that I believe makes this book so compelling and nicely self-contained.The introduction in Chapter 1 motivates the need for a comprehensive textbook and outlines challenges that the later chapters address more deeply. The theoretical background material begins in Chapter 2, which introduces core concepts, including hypothesis testing, type 1 and type 2 errors, validity and power, and p-values. The reader does not need to have any prior knowledge of statistical significance tests to follow this part. However, experienced readers could still benefit from reading this chapter, as concepts such as p-values are widely misunderstood and misused (Amrhein, Greenland, and McShane 2019).The significance tests themselves are introduced in Chapter 3, categorized into parametric and nonparametric tests. The chapter begins with the intuitively simple paired z-test, then builds up to more commonly-applied techniques, showing the connections and assumptions that each test makes. Step-by-step algorithms help the reader to implement each test. Although the chapter does cite uses of tests in NLP research, the main purpose is to present the theory behind each test and point out their differences.Chapter 4 provides perhaps the most handy part of the book for reference: a correspondence between common NLP tasks and statistical tests. Each task is discussed in terms of the evaluation metrics used, then a decision tree is introduced to guide the reader toward a choice between a parametric test, bootstrap or randomization test, or sampling-free nonparametric test. Section 4.3 then links each NLP evaluation measure to a specific significance test, presenting a large table that helps readers identify which test they need for a specific task. Particular considerations for each task are also pointed out to provide more detail about the appropriate options. The final part of this chapter describes the issue of p-hacking, in which dataset sizes are increased until a significance threshold is reached without consideration for biases in the data (discussed, for example, in Hofmann [2015]). The chapter proposes a simple solution to ensure robust significance testing with large datasets.Where Chapter 4 presents well-established methods, Chapter 5 introduces the current research question of how best to apply statistical significance testing to deep learning. Non-convex loss functions, stochastic optimization, random initialization, and a multitude of hyperparameters limit the conclusions we can draw from a single test run of a deep neural network (DNN). This chapter, which is based on the authors’ ACL paper (Dror, Shlomov, and Reichart 2019), explains how the comparison process can be overhauled to provide more meaningful evaluations. Beginning by explaining the difficulties of evaluating DNNs, the chapter then introduces criteria for a comparison framework, then discusses the limitations of current methods. Reimers and Gurevych (2018) have previously tackled this problem, but their approach has limited power and does not provide a confidence score. Other works, such as Clark et al. (2011), compare DNNs using a collection of statistics, such as the mean or standard deviation of performance across runs. This book shows how such an approach violates the assumptions of the significance tests. The authors propose almost stochastic dominance as the basis for a better alternative. The chapter explains how to use the proposed method, evaluates it in an empirical case study, and finally analyzes the errors made by each testing approach.Large NLP models are often tested across a range of datasets, which presents another problem for standard significance testing. Chapter 6 discusses the challenges of assessing two questions: (1) On how many datasets does algorithm A outperform algorithm B? (2) On which datasets does A outperform B? Applying standard significance tests individually to each dataset and counting the number of significant results is likely to overestimate the total number of significant results, as this chapter explains. The authors then present a new framework for replicability analysis, based on partial conjunction testing, and discuss two variants (Bonferroni and Fisher) for when the datasets are independent or dependent. They introduce a method based on Benjamini and Heller (2008) to count the number of datasets where one method outperforms another, then show how to use the Holm procedure (Holm 1979) to identify which datasets these are. Chapter 6 provides a lot of detailed background on the proposed replicability analysis framework, and the later sections again link the process to specific NLP case studies, and step-by-step summaries help the reader to apply the methodology. Extensive empirical results illustrate the very substantial differences in outcomes between the proposed approach and standard procedures.The final two chapters present open problems and conclude, showing that the topic has many interesting research questions of its own, such as problems when performing cross-validation, and the limited statistical power of replicability analysis.Overall, I highly recommend this book to a wide range of NLP researchers, from new students to seasoned experts who wish to ensure that they compare methods effectively. The book is excellent as both an introduction to the topic of significance testing and as a reference to use when evaluating your results. For anyone with further interest in the topic, it also points the way to future work. If one could level any criticism at this book at all, it is that it does not deeply discuss the basic flaws of significance testing or what the alternatives might be. For now, though, significance testing is an integral part of NLP research and this book provides a great resource for researchers who wish to perform it correctly and painlessly.

Learning the inherent meaning of a word in Natural Language Processing (NLP) has motivated researchers to represent a word at various levels of abstraction, namely character-level, morpheme-level, and subword-level vector representations. Syllable-Aware Word Embedding (SAWE) can effectively handle agglutinative and fusion-based NLP tasks. However, research attempts on assessing the SAWE on such extrinsic NLP tasks has been scanty, especially for low-resource languages in the context of code-mixing with English. A model to learn SAWE to extract semantics at fine-grained subunits of a word is proposed in this article, and the representative ability of the embeddings is assessed through sentiment analysis of code-mixed Telugu-English review corpora. Multilingual societies and advancements in communication technologies have accounted for the prolific usage of mixed data, which renders the State-of-the-Art (SOTA) sentiment analysis models developed based on monolingual data ineffective. Social media users in the Indian subcontinent exhibit a tendency to mix English and their respective native language (using the phonetic form of English) in expressing their opinions or sentiments. A code-mixing scenario provides flexibility to borrow words from a foreign language, usage of shorthand notations, elongation of vowels, and usage of words without following syntactic/grammatical rules, which renders the sentiment analysis of code-mixed data challenging to perform. Deep neural architectures like Long Short-Term Memory and Gated Recurrent Unit networks have been shown to be effective in solving several NLP tasks, such as sequence labeling, named entity recognition, and machine translation. In this article, a framework to perform sentiment analysis on a code-mixed Telugu-English review corpus is implemented. Both word embedding and SAWE are input to a unified deep neural network that contains a two-level Bidirectional Long Short-Term Memory/Gated Recurrent Unit network with Softmax as the output layer. The proposed model leverages the advantages of both word embedding and SAWE, which enable the proposed model to outperform existing SOTA code-mixed sentiment analysis models on a Telugu-English code-mixed dataset of the International Institute of Information Technology–Hyderabad and a dataset curated by the authors. The improvement realized by the proposed model on these datasets is [3% increase in F1-score and 2% increase in accuracy] and [7% increase in F1-score and 5% in accuracy], respectively, in comparison with the best-performing SOTA model.

Automatic Parts of Speech(POS) tagging is one of the most fundamental tasks for a language in Natural Language Processing(NLP), which acts as a feature for solving advanced NLP tasks. Named Entity Recognition(NER) is another essential task of NLP for information retrieval. Researchers could not find up to the mark solution yet on these two tasks for Bangla language compared to other languages, for instance, English, Ger-man. Moreover, many solutions heavily depend on handcrafted features that require strong linguistic expertise. As these two sequence labeling tasks are similar, In this work, two different datasets of POS tagging and NER were prepared, and different deep neural network approaches studied for solving these two tasks separately. All of the approaches were end to end and did not need any handcrafted feature like word suffixes or affixes, gazetteers, dictionary. This study came up with an end to end solution using deep neural network-based model consisting of Bi-directional Long short-term memory(BLSTM), Convolutional Neural Network(CNN) and Conditional Random Field(CRF). The proposed model trained on respected datasets achieved an accuracy of 93.86% on POS tagging and a strict f1 score of 0.6285 on NER on prepared datasets, respectively.

Named Entity Recognition (NER) is a vital task in natural language processing (NLP) that involves identifying and classifying named entities in text. This paper focuses on the comparison of four different algorithms of NER techniques in the context of India. The four different algorithms are CRF (Conditional Random Fields), Stanza, Spacy, and Nameparser. The accuracy will be compared with the expected accuracy of all four methods of NER. Accuracy will be obtained after running the method on a dataset or text. To calculate the accuracy of four algorithms a large dataset of Indian names will be used. A comparison graph was obtained to display the accuracy graph of all four algorithms. "NLP-Named Entity Recognition in India" which focuses on detecting person names, states, and districts, and suggesting names based on religion. Named Entity Recognition (NER) is a Natural Language Processing (NLP) task that identifies and classifies named entities in text. Named entities are typically proper nouns, such as person names, organization names, and location names. NER is a challenging task for Indian languages, due to the diversity of Indian languages and the lack of resources for training NER systems. The proposed system can be used to improve the accuracy and efficiency of a variety of NLP applications in India.

<title>Abstract</title> The Tenyidie language, a.k.a Angami language, is a Tibeto-Burman Language of Nagaland in the north-eastern part of India. Tenyidie is a low-resource language spoken by the Tenyimia Community and is considered a major language in Nagaland. Tenyidie is a tonal, Subject-Object-Verb, and highly agglutinative language. Among the many Natural Language Processing (NLP) tasks, part-of-speech tagging is one of the primary tasks which is used in building many other NLP tasks, such as dependency parsing, named entity recognition, machine translation, etc. The main aim of this paper is to develop the Part-of-speech (POS) Tagset and Tagger for the Tenyidie Language. To the best of our knowledge, there is no reported work on POS Tagset and POS Tagger for the Tenyidie language. The main contributions of this research are: creation of POS tagset for the Tenyidielanguage, creation of POS annotated dataset (11,540 sentences), and the verification of the proposed POS tagset and annotated dataset using the Machine Learning/Deep Learning techniques such as CRF, BLSTM, etc., including the state-of-the-art model such as BERT.

Work orders, equipment information, technical records and best practices documents contain within them a wealth of insights related to Equipment Maintenance which can be unlocked with Natural Language Processing tasks like classification, clustering, named entity recognition or part of speech tagging. But obtaining large enough labelled data sets in Equipment Maintenance domain manually is prohibitive and very expensive. This lack of labeled Equipment Maintenance data can be overcome with Large Language Models (LLMs) such as GPT-3, BERT that are pretrained transformer networks, considered state-of-the-art when it comes to Natural Language Processing (NLP) tasks. However, the vocabulary understood by these LLMs are mostly from English Language and need to be fine-tuned to understand industry and organization specific vocabulary and acronyms. This paper explores the potential of a domain specific LLM model for oil and gas industries. Data that are of good quality and that provide a comprehensive overview of industry are collected. This corpus of text contains documents like work orders, equipment data and technical documents. A custom tokenizer is trained on this data to identify domain specific terminology. A comparative study is done with other off-the-shelf tokenizers: BERT and RoBERTa, to compare the effectiveness of the tokenization. With millions of work orders and equipment documents, training pipelines had to parallelized so that training can occur on multiple GPUs. A comprehensive study of multiple training methods is done in this paper. Model and tokenizer developed were packaged and archived to be consumed in machine learning pipelines to specific use-cases across the organization. For an organization adopting digital transformation, the availability of an organization specific LLM is an enabler to extract insights from millions of documents containing free text. The applicability of such models spans across multiple disciplines like Maintenance, Reliability, Safety etc. and streamlines the development of highly accurate and robust text analytics.

This chapter presents a brief introduction to entropy guided transformation learning (ETL), a machine learning algorithm for classification tasks. ETL generalizes transformation based learning (TBL) by automatically solving the TBL bottleneck: the construction of good template sets. The main advantage of ETL is its easy applicability to natural language processing (NLP) tasks. This introductory chapter presents the motivation behind ETL and summarizes our experimental results. In Sect. 1.1, we first briefly detail TBL and explain its bottleneck. Next, we briefly present ETL and list some of its advantages. In Sect. 1.2, we first list some related works on the use of ETL for different NLP tasks. Next, we report a summary of our experimental results on the application of ETL to four language independent NLP tasks: part-of-speech tagging, phrase chunking, named entity recognition and semantic role labeling. Finally, in Sect. 1.3, we detail the structure of the book.KeywordsMachine learningEntropy guided transformation learningETL committeeTransformation based learningNatural language processingPart-of-speech taggingPhrase chunkingNamed entity recognitionSemantic role labeling

Extreme Multi-Label Text Classification (XMTC) is a crucial task in natural language processing, aiming to assign the most relevant subset of labels to an input text from an extremely large label set. Existing deep learning models often neglect the correlation information among labels when addressing XMTC, resulting in limited prediction performance. This paper proposes the Label Correlation Enhancement Network (LCENet), a pluggable modular architecture capable of effectively capturing and utilizing inter-label correlation knowledge without modifying the original model structure. The LCENet module introduces a bottleneck layer and residual connection mechanism, transforming raw label predictions into correlation-enhanced outputs. The bottleneck design reduces parameter complexity from $$O(L^2)$$ to O(RL), effectively addressing computational feasibility under extreme-scale label spaces. We integrate the LCENet module into various mainstream deep XMTC models-including CNN-, BERT-, and RNN-based architectures-and conduct comprehensive evaluations on three benchmark datasets: EUR-Lex, AmazonCat-13K, and Wiki-500K. Experimental results demonstrate that LCENet significantly improves the performance of baseline models, achieving consistent gains across multiple metrics such as Precision@k and nDCG@k, with a maximum increase of 5.22 percentage points in P@1, while also accelerating model convergence. Ablation studies further verify the effectiveness of key components, including the bottleneck layer, residual connections, and nonlinear activation functions. Training curve analysis shows that LCENet provides stronger learning signals through label correlation constraints, alleviating the early-stage stagnation observed in some baseline models (Fig. 5) and reducing the required convergence steps by nearly half. This study presents a practical and effective enhancement framework for deep multi-label learning, offering excellent scalability and practical applicability. The core idea of LCENet can be extended to other structured prediction tasks such as multimodal learning and sequence labeling.

Named Entity Recognition (NER) is a basic task in Natural Language Processing (NLP). Recently, the sequence-to-sequence (seq2seq) model has been widely used in NLP task. Different from the general NLP task, 60% sentences in the NER task do not contain entities. Traditional seq2seq method cannot address this issue effectively. To solve the aforementioned problem, we propose a novel seq2seq model, named SC-NER, for NER task. We construct a classifier between the encoder and decoder. In particular, the classifier’s input is the last hidden state of the encoder. Moreover, we present the restricted beam search to improve the performance of the proposed SC-NER. To evaluate our proposed model, we construct the patent documents corpus in the communications field, and conduct experiments on it. Experimental results show that our SC-NER model achieves better performance than other baseline methods.

Sequence labeling is a basic task in natural language processing, which is of great help to processing text information. Conventional sequence labeling approaches heavily rely on hand-crafted or language-specific features, which requires a lot of time. Therefore, most of the existing methods are based on the BiLSTM-CRF model, but how to use a neural network to extract useful information for each unit or segment in the input sequence becomes the main factor limiting the efficiency. Several BiLSTM-CRF based models for sequence labeling have been presented, but the major limitation is how to use neural networks for extracting useful representations for each unit or segment in the input sequence. In response to this problem, this paper proposes a sequence labeling algorithm based on hierarchical features and attention mechanism, which uses a hierarchical structure to integrate character-level and word-level information, and applies different attention mechanisms to these two layers of information. According to the structural characteristics of different levels, excavate more potential information. Finally, the previously captured and guided features are used for sequence tag prediction using CRF. Finally, the proposed model is subjected to comparative experiments and the results obtained are analyzed.

Named Entity Recognition (NER) is a challenging task in Natural Language Processing. Recently, machine learning based methods are widely used for the NER task and outperform traditional handcrafted rule based methods. As an alternative way to handle the NER task, stacking, which combines a set of classifiers into one classifier, has not been well explored for the NER task. In this paper, we propose a stacking model for the NER task. We extend the original stacking model from both model and feature aspects. We use Conditional Random Fields as the level-1 classifier, and we also apply meta-features from global aspect and local aspect of the level-0 classifiers and tokens in our model. In the experiments, our model achieves the state-of-the-art performance on the CoNLL 2003 Shared task.

Word Embeddings are low-dimensional distributed representations that encompass a set of language modeling and feature learning techniques from Natural Language Processing (NLP). Words or phrases from the vocabulary are mapped to vectors of real numbers in a low-dimensional space. In previous work, we proposed using an Extreme Learning Machine (ELM) for generating word embeddings. In this research, we apply the ELM-based Word Embeddings to the NLP task of Text Categorization, specifically Sentiment Analysis and Sequence Labeling. The ELM-based Word Embeddings utilizes a count-based approach similar to the Global Vectors (GloVe) model, where the word-context matrix is computed then matrix factorization is applied. A comparative study is done with Word2Vec and GloVe, which are the two popular state-of-the-art models. The results show that ELM-based Word Embeddings slightly outperforms the aforementioned two methods in the Sentiment Analysis and Sequence Labeling tasks.In addition, only one hyperparameter is needed using ELM whereas several are utilized for the other methods. ELM-based Word Embeddings are comparable to the state-of-the-art methods: Word2Vec and GloVe models. In addition, the count-based ELM model have word similarities to both the count-based GloVe and the predict-based Word2Vec models, with subtle differences.