Empirical Laws of Natural Language Processing for Neural Language Generated Text

Word embeddings have become the predominant representation scheme on a token-level for various clinical natural language processing (NLP) tasks. More recently, character-level neural language models, exploiting recurrent neural networks, have again received attention, because they achieved similar performance against various NLP benchmarks. We investigated to what extent character-based language models can be applied to the clinical domain and whether they are able to capture reasonable lexical semantics using this maximally fine-grained representation scheme. We trained a long short-term memory network on an excerpt from a table of de-identified 50-character long problem list entries in German, each of which assigned to an ICD-10 code. We modelled the task as a time series of one-hot encoded single character inputs. After the training phase we accessed the top 10 most similar character-induced word embeddings related to a clinical concept via a nearest neighbour search and evaluated the expected interconnected semantics. Results showed that traceable semantics were captured on a syntactic level above single characters, addressing the idiosyncratic nature of clinical language. The results support recent work on general language modelling that raised the question whether token-based representation schemes are still necessary for specific NLP tasks.

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

The domain of Natural Language Processing covers various tasks, such as classification, text generation, and language model. The data processed using word embeddings, or vectorizers, is then trained using Machine Learning and Deep Learning algorithms. In order to observe the tradeoff between both these types of algorithms, with respect to data available, accuracy obtained and other factors, a binary classification is undertaken to distinguish between insincere and regular questions on Quora. A dataset called Quora Insincere Questions Classification was used to train various machine learning and deep learning models. A Bidirectional-Long Short Term Network (LSTM) was trained, with the text processed using Global Vectors for Word Representation (GloVe). Machine Learning algorithms such as Extreme Gradient Boosting classifier, Gaussian Naive Bayes, and Support Vector Classifier (SVC), by using the TF-IDF vectorizer to process the text. This paper also presents an evaluation of the above algorithms on the basis of precision, recall, f1 score metrics.

Generative large language models (LLMs) are a subset of transformers-based neural network architecture models. LLMs have successfully leveraged a combination of an increased number of parameters, improvements in computational efficiency, and large pre-training datasets to perform a wide spectrum of natural language processing (NLP) tasks. Using a few examples (few-shot) or no examples (zero-shot) for prompt-tuning has enabled LLMs to achieve state-of-the-art performance in a broad range of NLP applications. This article by the American Medical Informatics Association (AMIA) NLP Working Group characterizes the opportunities, challenges, and best practices for our community to leverage and advance the integration of LLMs in downstream NLP applications effectively. This can be accomplished through a variety of approaches, including augmented prompting, instruction prompt tuning, and reinforcement learning from human feedback (RLHF). Our focus is on making LLMs accessible to the broader biomedical informatics community, including clinicians and researchers who may be unfamiliar with NLP. Additionally, NLP practitioners may gain insight from the described best practices. We focus on 3 broad categories of NLP tasks, namely natural language understanding, natural language inferencing, and natural language generation. We review the emerging trends in prompt tuning, instruction fine-tuning, and evaluation metrics used for LLMs while drawing attention to several issues that impact biomedical NLP applications, including falsehoods in generated text (confabulation/hallucinations), toxicity, and dataset contamination leading to overfitting. We also review potential approaches to address some of these current challenges in LLMs, such as chain of thought prompting, and the phenomena of emergent capabilities observed in LLMs that can be leveraged to address complex NLP challenge in biomedical applications.

The integration of Natural Language Processing (NLP) in Human-Robot Interaction (HRI) represents a significant advancement towards achieving more natural and effective communication between humans and robots. This research explores the application of state-of-the-art NLP techniques to enhance HRI, focusing on improving robots' abilities to understand and generate human language. Key components of our approach include advanced speech recognition, natural language understanding (NLU), dialogue management, and natural language generation (NLG). We designed and implemented an HRI system that leverages models such as BERT for language understanding and GPT-3 for generating contextually appropriate responses. Our methodology involves integrating these NLP models with a robotics platform, ensuring real-time interaction capabilities while maintaining a high level of accuracy and context awareness. The system was evaluated through a series of user studies, measuring performance metrics such as accuracy, latency, and user satisfaction. Results indicate that our NLP-enhanced HRI system significantly improves the quality of interactions, demonstrating superior understanding and responsiveness compared to traditional systems. This paper discusses the implementation challenges, including computational constraints and ambiguity resolution, and provides insights into user feedback and system performance. Future work will focus on enhancing context management, exploring multimodal interaction, and addressing ethical considerations in deploying advanced HRI systems. Our findings underscore the potential of NLP to transform human-robot communication, paving the way for more intuitive and effective robotic assistants in various domains. Keywords: Human-Robot Interaction (HRI), Natural Language Processing (NLP), Conversational AI, Speech Recognition, Natural Language Understanding (NLU), Natural Language Generation (NLG), Multimodal Interaction, Dialogue Systems, Context Awareness, Emotion Recognition, Machine Learning in HRI, Personalized Interaction, User Experience (UX) in HRI, Human-Centered Design, Collaborative Robots (Cobots)

Summary Objectives : Analyze papers published in 2019 within the medical natural language processing (NLP) domain in order to select the best works of the field. Methods : We performed an automatic and manual pre-selection of papers to be reviewed and finally selected the best NLP papers of the year. We also propose an analysis of the content of NLP publications in 2019. Results : Three best papers have been selected this year including the generation of synthetic record texts in Chinese, a method to identify contradictions in the literature, and the BioBERT word representation. Conclusions : The year 2019 was very rich and various NLP issues and topics were addressed by research teams. This shows the will and capacity of researchers to move towards robust and reproducible results. Researchers also prove to be creative in addressing original issues with relevant approaches.

Natural Language Processing (NLP) involves the use of algorithms and models and various computational techniques to analyze, process, and generate natural language data, including speech and text. NLP helps computers interact with humans in a more natural way, which has become increasingly important as more humancomputer interactions take place. NLP allows machines to process and analyze voluminous unstructured data, including social media posts, newspaper articles, reviews from customers, emails, and others. It helps organizations extract insights, automate tasks, and improve decision-making by enabling machines to understand and generate human-like language. A linguistic background is essential for understanding NLP. Linguistic theories and models help in developing NLU systems, as NLP specialists need to understand the structure and rules of language. NLU systems are organized into different components, including language modelling, parsing, and semantic analysis. NLU systems may be assessed through the use of metrics that includes measures like precision and recall, as well as indicators that convey meaningful information that include F1 score and others. Semantics and knowledge representation are central to NLU, as they involve understanding the meaning of words and sentences and representing this information in a way that machines can use. Approaches to knowledge representation include semantic networks, ontologies, and vector embeddings. Language modelling is an essential step in NLP that sees usage in applications like speech recognition, text generation, and text completion and also in areas such as machine translation. Ambiguity Resolution remains a major challenge in NLP, as language is often ambiguous and context-dependent. Some common applications of NLP include sentiment analysis, chatbots, virtual assistants, machine translation, speech recognition, text classification, text summarization, and information extraction. In this chapter, we show the applicability of a popular unsupervised learning technique, viz., clustering through K-Means. The efficiency provided by the K-Means algorithm can be improved through the use of an optimization loop. The prospects for NLP are promising, with an increasing demand for AI-powered language technologies in various industries, including healthcare, finance, and e-commerce. There is also a growing need for ethical and responsible AI systems that are transparent and accountable.

414 LANGUAGE, VOLUME 67, NUMBER 2 (1991) Brixhe & Lejeune's 1984 Corpus des inscriptions paléo-phrygiennes. Since Phrygian is both closely related to Greek and preserved in part through Greek testimony, N has chosen to present his account as a history of PhrygianGreek linguistic contact. This procedure may explain his vagueness in discussing shared features of the two languages: although he calls them 'common innovations' (10), he seems elsewhere to regard them as the result of borrowing during a hypothesized period of close contact among Greek, Phrygian, and several other IE languages in the south Balkans (5-6). Some of these shared features are quite striking—e.g. factitives in *-oyô like Gk. orthöö 'straighten'. Phr. kakoioi (11, 13), and laryngeal vocalization followed by umlaut in Gk. ónoma, Phr. onoman = [b] not [bh], this development is trivial and proves only that Phrygian did not share the Greek change to [ph] etc.). Pp. 6-10: Common retentions of Greek and Phrygian (e.g. the augment and the thirdperson present middle ending -toi). Pp. 10-13: Common innovations of Greek and Phrygian (e.g. agent nouns in -tä, Gk. autos 'self = Phr. autos 'particular'). Pp. 13-16: Inherited lexical material of Phrygian which is lost or only marginally preserved in Greek. Pp. 16-18: Sometime after the hypothesized period of south Balkan close contact with Greek, the Phrygians migrated to Anatolia, where they are attested in historical times. Contact with speakers ofAnatolian languages led to borrowings into Phrygian (mainly names, e.g. Midas). Pp. 18-19: Greek expansion in the first millennium led to contact and hence Greek loans into Phrygian. Pp. 19-22: Direct attestation of Phrygian lost from the third century B.C.E. through the first century CE. Pp. 22-23: Neo-Phrygian attested in grave inscriptions, still in part uninterpretable . Although Phrygian is unlikely ever to play a significant role in IE historical linguistics, it does, as N remarks in conclusion (23), repay time spent applying to it our knowledge of IE. N's essay is a valuable contribution to the project of understanding Phrygian. [Andrew Garrett, University of Texas, Austin.] Natural language processing technologies in Artificial Intelligence: The science and industry perspective. By Klaus K. Obermeier. Chichester : Ellis Horwood, 1989. Pp. 263. Natural language processing (NLP) is the area of research concerned with the many aspects of 'natural' language communication with computers . In this book Obermeier provides an introduction to NLP from both commercial and academic perspectives, primarily aimed at businessmen and data-processing professionals. His presentation is nontechnical, though very comprehensive . Linguists will find the book an accessible introduction to the practical issues in NLP, which may help to clarify the direction and emphasis in this area of language research. The book's organization is excellent, making it very easy to read, although the typographic and editorial quality are not the best. The twelve chapters are divided into three sections: 'NLP in perspective' (17-92). which provides an introduction to the structure of NLP systems and the approaches and formalisms used to implement them; 'Applications' (93-185), in which six major applications of NLP are described: and 'Bottom line' (187-232), where O presents his own evaluation of the state of NLP research and current systems. A good list of references (239-59) is also included. Much of the material in the first section will be familiar to linguists, since the design of NLP systems is largely based on formal linguistic analyses in the areas ofphonology, morphology, syntax, and semantics. In addition, methods for knowledge representation and reasoning drawn from Artificial Intelligence are explained in relation to their relevance to NLP development. This nontechnical overview of the techniques used in NLP is up-to-date. It includes a discussion of parsing methods ranging from contextfree parsing, unification-based parsing, and ATNs to such knowledge-representation schemes from AI as production rules, semantic networks, and frames, to the new connectionist models. BOOK NOTICES 415 The second section comprises the largest part of the book, with one chapter devoted to each of the applications addressed: natural-language interfaces, machine translation, text processing/ understanding, text generation, speech processing , and writing aids. For each application there is a brief...

Malay social media text is a text written on social media networks like Twitter. Commonly, this text comprises non-standard words, filled with dialects, foreign languages, word abbreviations, grammatical neglect, spelling errors, and many more. It is well known that this type of text is difficult to process due to its high noise and distinct text structure. Such problems can be resolved using rigorous text normalization, which is critical before any technique can be implemented and evaluated on social media text. In this paper, an improved normalization method towards Malay social media text was proposed by converting non-standard Malay words using a rule-based model. The method normalizes common language words often used by Malaysian users, such as non-standard Malay (like dialect and slangs), Romanized Arabic, and English words. Thus, a Malay text normalizer was proposed using a set of rules that extend across different domains of natural language processing (NLP) and is expected to address the challenges of processing Malay social media text. This study implements the proposed Malay text normalizer in a Part-of-Speech (POS) tagging application to evaluate the normalizer’s performance. The implementation demonstrates a substantial improvement in the POS tagging efficiency over several pre-processing stages, with an improvement of accuracy up to 31.8%. The increase of accuracy in the POS tagging indicates two main points. First, the Malay text normalizer’s rules improve the performance of a Malay text normalizer on social media text. Second, our proposed Malay text normalizer has successfully improved the POS tagging percentage and demonstrates the importance of normalized pre-processing in any NLP application.

Abstract. Large language models (LLMs) have revolutionized the field of natural language processing (NLP), demonstrating remarkable capabilities in understanding, generating, and manipulating human language. This comprehensive review explores the development, applications, optimizations, and challenges of LLMs. This paper begin by tracing the evolution of these models and their foundational architectures, such as the Transformer, GPT, and BERT. We then delve into the applications of LLMs in natural language understanding tasks, including sentiment analysis, named entity recognition, question answering, and text summarization, highlighting real-world use cases. Next, we examine the role of LLMs in natural language generation, covering areas such as content creation, language translation, personalized recommendations, and automated responses. We further discuss LLM applications in other NLP tasks like text style transfer, text correction, and language model pre-training. Subsequently, we explore techniques for optimizing and improving LLMs, including model compression, explainability, robustness, and security. Finally, we address the challenges posed by the significant computational requirements, sample inefficiency, and ethical considerations surrounding LLMs. We conclude by discussing potential future research directions, such as efficient architectures, few-shot learning, bias mitigation, and privacy-preserving techniques, which will shape the ongoing development and responsible deployment of LLMs in NLP.

The advances in deep learning have led to great achievements in many Natural Language Processing (NLP) tasks. With the nature of language, i.e., sequential data, most NLP tasks can be framed into the sequence learning framework, such as text generation. As one of the most important foundations for modern NLP techniques, autoregressive generation models have achieved dominant performance in a great deal of NLP tasks. Therefore, this thesis emphasizes improving the autoregressive generation model for different NLP tasks. While many tasks can naturally fit into the sequence learning framework, some of them, e.g., building discourse parsing tree, require sophisticated designs to fit into neural models. Therefore, this thesis firstly emphasizes a novel unified framework for discourse parsing, which builds a discourse tree in a top-down depth-first manner, and it frames the task as an autoregressive generation task with the goal of each step being the prediction of the node position given a piece of text. The proposed approach is proven effective with extensive empirical experiments. In addition, I extend the above framework by proposing a hierarchical decoder, which leverages the information from parents and siblings of the nodes that are currently processed. The proposed decoder utilizes the nature of the tree structure and further improves the experiment performance on both discourse parsing and dependency parsing tasks. On the other hand, the de facto strategies, i.e., cross entropy loss and teacher forcing, for training the autoregressive generation models have been shown problematic in certain aspects. For example, cross entropy loss, which is one of the widely leveraged training objective functions, often leads to text degeneration in text generation, and teacher forcing suffers from the exposure bias problem, where there exists a mismatch between the training and testing setup. For text degeneration, I introduce a class of diminishing attentions, which enforces the submodularity of the coverage calculated by cross attention in the sequence-to-sequence model. The proposed diminishing attentions achieve notable improvement on several neural text generation tasks, including text summarization, machine translation, and image paragraph generation. Further, I propose a novel training objective, ScaleGrad, to replace cross entropy, which significantly reduces the degeneration problem in different text generation tasks. In fact, ScaleGrad can be extended to problems beyond text degeneration. It provides wide flexibility to inject different inductive biases into the text generation model by directly modifying the gradient information in the output layer. Next, for the exposure bias problem, this thesis introduces a novel type of scheduled sampling based on training accuracy, which requires only minimal hyper-parameter tuning compared to existing scheduled sampling methods. Additionally, a novel imitation loss is proposed to further enforce the model’s generative behavior to match the teacher-forced behavior. Moreover, this thesis demonstrates that reducing exposure bias can improve the robustness of language models against repetition and toxic errors.

What does it mean when we say that computers can ‘write’ and how are recent developments in neural networks and machine learning changing this capacity? This article examines the long-standing literary fear of authorship being replaced by machines while also interrogating the labour and credit implications that sit behind widely used structures of authorship in a technological age. The argument makes reference to one work of computer-generated writing – Johannes Helden & Hakan Jonson’s Evolution [2014] – and to one software paradigm (a character-based recurrent neural networks for language acquisition trained on the corpus of the journal Textual Practice). I here argue that unless we conceive more broadly of the criteria for ‘authorship’ as a labour function, and unless we take seriously the need to see textual production as social production, hybridized (but predominantly) machine identities will come to dominate a literary landscape.

1 IntroductionOrganizations focus on automate their processes in order to improve efficiency, reduce costs, and/or reduce human beings' errors in an easy and rapid manner. Business process management (BPM) methods provide a solution to this issue. In this context, information systems like CRM, ERP, SCM, etc. have known an increasing demand. The main problem consists of the length of the business process specifications. If new regulations appear, these specifications must be adapted. Manually extraction of visual models is time consuming. During time, a series of solutions were proposed. The literature shows a crowd of approaches that extracts data models [1], [2], [3], and process models [4], [5] from Natural Language (NL) text. This paper aims to analyze the Natural Language Processing (NLP) techniques and tools used in order to provide different types of visual representations.On the last few years, approaches based on NLP have been developed in order to automate process conversion from NL text. NLP plays an important role in NL text analysis as NLP tries to understand speech and text as humans beings would do. Colloquialism, abbreviations or typos make this task a challenging one. NLP has the origins in 1950s when Alan Turing proposed the Turing test [6], by introducing the imitation game. Since then, the literature shows a plethora of NLP tools [7], [8], [9], [10] using several machine learning techniques, our focus being on those applied on data models and process models discovery from NL text, starting with the first language parser [11] to the actual ones like NLTk [7], [8], ANTLR1, etc.Linguistic analysis is closely tied to NLP. Liddy [12] highlights 7 levels of linguistic analysis: a) Phonetic or Phonological level: how words are pronounced, b) Morphological level: prefixes, suffixes and roots analysis, c) Lexical level: word level analysis including lexical meaning and Part-Of-Speech (POS) analysis, d) Syntactic level: grammatical analysis of words in a sentence, e) Semantic level: determining the possible meanings of sentences, f) Discourse level: interpreting structure and meaning for texts larger than a sentence, g) Pragmatic level: understanding the purpose of a language.Some of the problems approached by NLP are: POS tagging, parsing, Named Entity Recognition (NER), chunking, Semantic Role Labeling (SRL). Anaphora resolution [13] refers to the interpretation of the link between the anaphor and its antecedents.The remainder of the paper is organized as follows: Section 2 briefly outlines the NLP domain, describing the levels of linguistic analysis and the main NLP approaches of analyzing NL requirements. Section 3 focuses on data and process models extraction from NL text. This section makes an introduction to Object Oriented Analysis and Business Process Modeling and analyses the existing tools that discover data and process models from text. Subsequently, Section 4 summarizes the results of this work and the conclusions are drawn in Section 5.2NLPA detailed review on NLP is given in [14] and in [15]. Jones [14] divides the history of NLP into four phases: the first starts at the beginning of the 1940s and lasts to the late 1960s, the second begins from the end of 60s and lasts to the end of 70s, the third is represented by late 80s, where the fourth phase starts in the late of 90s. Next, we will detail each phase as they were defined in [14] and [15]. First phase treated machine translation issues, while the second focused on artificial intelligence. The third phase can be called grammatico-logical phase, which is followed by the lexical phase. A fifth phase is proposed in [16] where formal theories and statistical data are combined. Since this study was published first in 1994 and then it was re-organized in 2001 we can add the sixth phase: from 2000 until present where NLP techniques are combined in order to contribute to visual models extraction.Software requirements are usually written in NL which is asymmetric and irregular [17]. …

This article presents a novel conversational artificial intelligence (CAI)-enabled active ideation system as a creative idea generation tool to assist novice product designers in mitigating the initial latency and ideation bottlenecks that are commonly observed. It is a dynamic, interactive, and contextually responsive approach, actively involving a large language model (LLM) from the domain of natural language processing (NLP) in artificial intelligence (AI) to produce multiple statements of potential ideas for different design problems. Integrating such AI models with ideation creates what we refer to as an active ideation scenario, which helps foster continuous dialog-based interaction, context-sensitive conversation, and prolific idea generation. An empirical study was conducted with 30 novice product designers to generate multiple ideas for given problems using traditional methods and the new CAI-based interface. The ideas generated by both methods were qualitatively evaluated by a panel of experts. The findings demonstrated the relative superiority of the proposed tool for generating prolific, meaningful, novel, and diverse ideas. The interface was enhanced by incorporating a prompt-engineered structured dialog style for each ideation stage to make it uniform and more convenient for the product designers. A pilot study was conducted and the resulting responses of such a structured CAI interface were found to be more succinct and aligned toward the subsequent design stage. The article thus established the rich potential of using generative AI (Gen-AI) for the early ill-structured phase of the creative product design process.

Speech recognition and text summarization, plagiarism detection, machine translation, chatbots, sentiment analysis (SA), question answering (QA), and dialogue systems are all products of natural language processing (NLP), a branch of AI concerned with modeling natural languages for the purpose of developing relevant applications. NLP draws on several disciplines, not only computer science and linguistics, for its research and development. These include cognitive science, psychology, mathematics, and more. More than 1.5 billion Muslims throughout the world depend on the Arabic language for their daily five-times-prayer practice, and Arabic is one of six official languages used by more than 422 million people in the Arab world, according to UNESCO. Dialectal Arabic is the slang language spoken informally in everyday life and varies from country to country; Classical Arabic is a reflection of the language spoken by the Arabs more than fourteen centuries ago. Modern Standard Arabic is an evolving variety of Arabic that borrows and innovates regularly to meet the changing needs of its speakers. Complexity is added to the Arabic language by the fact that it encompasses not one but three different varieties of spoken language: classical, contemporary, and colloquial. However, Arabic language processing on computers remains difficult for a variety of reasons. Because of the extensive inflection and derivational processes that occur in Arabic, a single lemma may be utilized to produce several words with distinct meanings.