Neural Multi-task Text Normalization and Sanitization with Pointer-Generator

This paper describes a LVCSR system for automatic online subtitling (closed captioning) of TV transmissions of the Czech Parliament meetings The recognition system is based on Hidden Markov Models, lexical trees and bigram language model The acoustic model is trained on 40 hours of parliament speech and the language model on more than 10M tokens of parliament speech trancriptions The first part of the article is focused on text normalization and class-based language model preparation The second part describes the recognition network and its decoding with respect to real-time operation demands using up to 100k vocabulary The third part outlines the application framework allowing generation and displaying of subtitles for any audio/video source Finally, experimental results obtained on parliament speeches with recognition accuracy varying from 80 to 95 % (according to the discussed topic) are reported and discussed.

Social media has become a rich data source for natural language processing tasks with its worldwide use; however, it is hard to process social media data due to its informal nature. Text normalization is the task of transforming the noisy text into its canonical form. It generally serves as a preprocessing task in other NLP tasks that are applied to noisy text. In this study, we apply two approaches for Turkish text normalization: Contextual Normalization approach using distributed representations of words and Sequence-to-Sequence Normalization approach using neural encoder-decoder models. As the approaches applied to Turkish and also other languages are mostly rule-based, additional rules are required to be added to the normalization model in order to detect new error patterns arising from the change of the language use in social media. In contrast to rule-based approaches, the proposed approaches provide the advantage of normalizing different error patterns that change over time by training with a new dataset and updating the normalization model. Therefore, the proposed methods provide a solution to language change dependency in social media by updating the normalization model without defining new rules.

Semi-supervised named entity recognition in multi-level contexts

In domain-specific named entity recognition (NER), the out-of-vocabulary (OOV) problem arises due to linguistic features and rare vocabulary.OOV problem is particularly challenging in agglutinative languages such as Korean.The irregular decomposition of morphemes makes it difficult to represent all of them in language model dictionaries, resulting in poor NER performance.Subword tokenization which segments a word into atomic tokens that are no longer divided can be one of the possible solutions.In the construction industry, existing NER methods do not effective on housing defect complaints which contain many rare words, including jargon, slang, and typos.To address this challenge, we propose subword tokenization algorithms that can mitigate OOV problems based on considering linguistic features and pre-trained language models (PLMs).The primary objective of this study is to identify the optimal NER performance by comparing different subword tokenization methods depending on the language models used.For domain-specific NER, we defined and used 23 defectspecific named entity tags for dataset labelling.We then experimented with a total of three state-of-theart language models: one SentencePiece-based and two WordPiece-based subword tokenization models.The results demonstrate that the SentencePiece-based Korean Bidirectional Encoder Representations from Transformers (KoBERT) outperformed the two WordPiece-based language models (multilingual-BERT and Korean Efficiently Learning an Encoder that Classifies Token Replacements Accurately (KoELECTRA)) with an F1 score of 84.7%.The proposed method is expected to improve not only NER but also other downstream tasks that involve using Korean documents in the construction industry.

One of the most persistent characteristics of written user-generated content (UGC) is the use of non-standard words. This characteristic contributes to an increased difficulty to automatically process and analyze UGC. Text normalization is the task of transforming lexical variants to their canonical forms and is often used as a pre-processing step for conventional NLP tasks in order to overcome the performance drop that NLP systems experience when applied to UGC. In this work, we follow a Neural Machine Translation approach to text normalization. To train such an encoder-decoder model, large parallel training corpora of sentence pairs are required. However, obtaining large data sets with UGC and their normalized version is not trivial, especially for languages other than English. In this paper, we explore how to overcome this data bottleneck for Dutch, a low-resource language. We start off with a small publicly available parallel Dutch data set comprising three UGC genres and compare two different approaches. The first is to manually normalize and add training data, a money and time-consuming task. The second approach is a set of data augmentation techniques which increase data size by converting existing resources into synthesized non-standard forms. Our results reveal that, while the different approaches yield similar results regarding the normalization issues in the test set, they also introduce a large amount of over-normalizations.

Deep language models, like ELMo, BERT and GPT, have achieved impressive results on several natural language tasks. These models are pretrained on large corpora of unlabeled general domain text and later supervisedly trained on downstream tasks. An optional step consists of finetuning the language model on a large intradomain corpus of unlabeled text, before training it on the final task. This aspect is not well explored in the current literature. In this work, we investigate the impact of this step on named entity recognition (NER) for Portuguese legal documents. We explore different scenarios considering two deep language architectures (ELMo and BERT), four unlabeled corpora and three legal NER tasks for the Portuguese language. Experimental findings show a significant improvement on performance due to language model finetuning on intradomain text. We also evaluate the finetuned models on two general-domain NER tasks, in order to understand whether the aforementioned improvements were really due to domain similarity or simply due to more training data. The achieved results also indicate that finetuning on a legal domain corpus hurts performance on the general-domain NER tasks. Additionally, our BERT model, finetuned on a legal corpus, significantly improves on the state-of-the-art performance on the LeNER-Br corpus, a Portuguese language NER corpus for the legal domain.

Named entity (NE) extraction is traditionally applied to written text. Some recent works extend the extraction to broadcast speech but most of these approaches simply cascade a speech-to-text (STT) engine and a named entity tagger without any information sharing between the two. In this paper, we extract named entities from conversational speech and explore approaches to couple the STT and NE extraction beyond a simple cascade. We propose a new approach that adapts the STT language model based on the extracted named entities. This steers the STT to focus on the subset of the vocabulary that is most important to NE extraction. We performed a number of experiments on English conversational speech in the Fisher Corpus with different STT recognition speeds. We show that the language model (LM) adaptation approach increases NE extraction recall and improves NE performance by as much as 15% as measured by F-score, by significantly improving the word error rate (WER) of NE words, with minimal impact on the overall WER.

Grabbing the Long Tail: A data normalization method for diverse and informative dialogue generation

Named Entity Recognition is a subtask in Information Extraction. In Named Entity Recognition (NER) we try to identify each of the words provided into some categories predefined by us. These categories or classes can be organization, name, time, place etc. Named Entity Recognition is a part of Natural Language Processing which is aims at making human and computer interactions more meaningful and efficient. NLP is nowadays being effectively being used in the area of automatic text summarization, cross language information retrieval, speech recognition and query named entity recognition. This paper focuses on Named entity recognition, how it is performed and a how apply NER to web search queries so that correct and more user personalized results are displayed by the web search engines..

TagNorm: Tagalog Text Normalization with Slang Word Detection and Classification of Type of Slang Word using Support Vector Machine was designed to normalize informal word and transform it to their canonical forms. The study aimed to improve text normalization in the Filipino domain in which the informal texts such as Alphanumeric Abbreviation and Phonetic, Stylistic variation, Combined Words, Misspelled words/ typographical errors (except rumbled letters), Omission of Vowels and Slang Words can decrease the performance of Natural Language Processing. Using the Facebook data the accuracy of the system in terms of informal text types the researchers got an accuracy of 91.23% in Alphanumeric Abbreviation and Phonetic, 77.61% in Repetition of Characters, 85.19% in Stylistic variation, 69.64% in Combined Words, 72.41% in Misspelled words/ typographical errors, 90.7S% in Omission of Vowels and 93.89% in Slang Words, while the data from previous study obtained an accuracy of 86.96% in Alphanumeric Abbreviation and Phonetic, 66.67% in Repetition of Characters, 57.14% in Stylistic variation, 80% in Combined Words, 85.29% in Misspelled words/ typographical errors, 93.15 in Omission of Vowels and 75% in Slang Words..

Towards a Novel Weakly Supervised Joint Approach of Named Entity Recognition and Normalization for Noisy Text

Online review platforms, such as restaurant search websites and apps, have become a primary source of information for consumers when choosing restaurants. The large number of reviews available provides insights into service quality, food taste, and previous customer experiences. However, the main challenge is managing and extracting relevant information from the diverse and unstructured reviews, which can make it difficult for users to find accurate and relevant information. This study implements an information retrieval system for Indian restaurant reviews using the Vector Space Model (VSM) algorithm to address this challenge. The dataset from Kaggle, containing 10,000 Indian restaurant reviews, was processed through tokenization, stopword removal, stemming, and text normalization. The TF-IDF method was applied for term weighting, and relevance between the user's query and reviews was calculated using VSM. The evaluation results showed a precision of 70.92%, recall of 84.81%, and F1-score of 77.25%, indicating that the system can provide relevant reviews accurately and efficiently. This system could serve as a reference for developing information retrieval systems in the culinary field and other sectors that require effective customer review analysis.

BackgroundPatient medical information often exists in unstructured text containing abbreviations and acronyms deemed essential to conserve time and space but posing challenges for automated interpretation. Leveraging the efficacy of Transformers in natural language processing, our objective was to use the knowledge acquired by a language model and continue its pre-training to develop an European Portuguese (PT-PT) healthcare-domain language model. MethodsAfter carrying out a filtering process, Albertina PT-PT 900M was selected as our base language model, and we continued its pre-training using more than 2.6 million electronic medical records from Portugal's largest public hospital. MediAlbertina 900M has been created through domain adaptation on this data using masked language modelling. ResultsThe comparison with our baseline was made through the usage of both perplexity, which decreased from about 20 to 1.6 values, and the fine-tuning and evaluation of information extraction models such as Named Entity Recognition and Assertion Status. MediAlbertina PT-PT outperformed Albertina PT-PT in both tasks by 4–6% on recall and f1-score. ConclusionsThis study contributes with the first publicly available medical language model trained with PT-PT data. It underscores the efficacy of domain adaptation and offers a contribution to the scientific community in overcoming obstacles of non-English languages. With MediAlbertina, further steps can be taken to assist physicians, in creating decision support systems or building medical timelines in order to perform profiling, by fine-tuning MediAlbertina for PT- PT medical tasks.

Machine learning interatomic potential (MLIP) is an emerging technique that has helped achieve molecular dynamics simulations with unprecedented balance between efficiency and accuracy. Recently, the body of MLIP literature has been growing rapidly, which propels the need to automatically process relevant information for researchers to understand and utilize. Named entity recognition (NER), a natural language processing technique that identifies and categorizes information from texts, may help summarize key approaches and findings of relevant papers. In this work, we develop an NER model for MLIP literature by fine‐tuning a pre‐trained language model. To streamline text annotation, we build a user‐friendly web application for annotation and proofreading, which is seamlessly integrated into the training procedure. Our model can identify technical entities with an F1 score of 0.8 for new MLIP paper abstracts using only 60 training paper abstracts and up to 0.75 for scientific texts on different topics. Notably, some “errors” in predictions are actually reasonable decisions, showcasing the model's ability beyond what the performance metrics indicate. This work demonstrates the linguistic capabilities of the NER approach in processing textual information of a specific scientific domain and has the potential to accelerate materials research using language models and contribute to a user‐centric workflow.

The language modeling approach centers on the issue of estimating an accurate model by choosing appropriate language models as well as smoothing techniques. In the thesis, we propose a novel context-sensitive semantic smoothing method referred to as a topic signature language model. It extracts explicit topic signatures from a document and then statistically maps them into individual words in the vocabulary. In order to support the new language model, we developed two automated algorithms to extract multiword phrases and ontological concepts, respectively, and an EM-based algorithm to learn semantic mapping knowledge from co-occurrence data. The topic signature language model is applied to three applications: information retrieval, text classification, and text clustering. The evaluations on news collection and biomedical literature prove the effectiveness of the topic signature language model. In the experiment of information retrieval, the topic signature language model consistently outperforms the baseline two-stage language model as well as the context-insensitive semantic smoothing method in all configurations. It also beats the state-of-the-art Okapi models in all configurations. In the experiment of text classification, when the size of training documents is small, the Bayesian classifier with semantic smoothing not only outperforms the classifiers with background smoothing and Laplace smoothing, but it also beats the active learning classifiers and SVM classifiers. On the task of clustering, whether or not the dataset to cluster is small, the model-based k-means with semantic smoothing performs significantly better than both the model-based k-means with background smoothing and Laplace smoothing. It is also superior to the spherical k-means in terms of effectiveness. In addition, we empirically prove that, within the framework of topic signature language models, the semantic knowledge learned from one collection could be effectively applied to other collections. In the thesis, we also compare three types of topic signatures (i.e., words, multiword phrases, and ontological concepts), with respect to their effectiveness and efficiency for semantic smoothing. In general, it is more expensive to extract multiword phrases and ontological concepts than individual words, but semantic mapping based on multiword phrases and ontological concepts are more effective in handling data sparsity than on individual words.