Benchmarking Applied Semantic Inference: The PASCAL Recognising Textual Entailment Challenges

A survey on textual entailment based question answering

Text summarization is referred to the process of rewriting a particular content into its brief version by understanding it. It is a serious need to obtain concise and relevant information from huge information. Automatic text summarization is fast-growing research area in natural language processing from last few years. It has improved from simple heuristics to neural network-based text summarization techniques. The extractive and abstractive text summarization generate the condensed text which saves time in reading the entire document. The abstractive approach is more complex as it needs natural language processing and neural network. Due to the availability of huge unsupervised data, traditional approach fails to provide accuracy in text summarization. The deep learning-based text summarization model gives a good performance as compared to the conventional techniques. In this paper, we have reviewed the recent work on text summarization based on deep learning techniques.

In this work, we present a new unsupervised and language-independent methodology to detect the relations of textual generality. For this, we introduce a particular case of Textual Entailment (TE), namely Textual Entailment by Generality (TEG). TE aims to capture primary semantic inference needs across applications in Natural Language Processing (NLP). Since 2005, in the TE Recognition (RTE) task, systems have been asked to automatically judge whether the meaning of a portion of the text, the Text (T), entails the meaning of another text, the Hypothesis (H). Several novel approaches and improvements in TE technologies demonstrated in RTE Challenges are signaling renewed interest towards a more in-depth and better understanding of the core phenomena involved in TE. In line with this direction, in this work, we focus on a particular case of entailment, entailment by generality, to detect the relations of textual generality. In text, there are different kinds of entailments, yielded from different types of implicative reasoning (lexical, syntactical, common sense based), but here, we focus just on TEG, which can be defined as an entailment from a specific statement towards a relatively more general one. Therefore, we have T→GH whenever the premise T entails the hypothesis H, this also being more general than the premise. We propose an unsupervised and language-independent method to recognize TEGs, from a pair ⟨T,H⟩ having an entailment relation. To this end, we introduce an Informative Asymmetric Measure (IAM) called Simplified Asymmetric InfoSimba (AISs), which we combine with different Asymmetric Association Measures (AAM). In this work, we hypothesize about the existence of a particular mode of TE, namely TEG. Thus, the main contribution of our study is highlighting the importance of this inference mechanism. Consequently, the new annotation data seem to be a valuable resource for the community.

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

Speech recognition and text summarization, plagiarism detection, machine translation, chatbots, sentiment analysis (SA), question answering (QA), and dialogue systems are all products of natural language processing (NLP), a branch of AI concerned with modeling natural languages for the purpose of developing relevant applications. NLP draws on several disciplines, not only computer science and linguistics, for its research and development. These include cognitive science, psychology, mathematics, and more. More than 1.5 billion Muslims throughout the world depend on the Arabic language for their daily five-times-prayer practice, and Arabic is one of six official languages used by more than 422 million people in the Arab world, according to UNESCO. Dialectal Arabic is the slang language spoken informally in everyday life and varies from country to country; Classical Arabic is a reflection of the language spoken by the Arabs more than fourteen centuries ago. Modern Standard Arabic is an evolving variety of Arabic that borrows and innovates regularly to meet the changing needs of its speakers. Complexity is added to the Arabic language by the fact that it encompasses not one but three different varieties of spoken language: classical, contemporary, and colloquial. However, Arabic language processing on computers remains difficult for a variety of reasons. Because of the extensive inflection and derivational processes that occur in Arabic, a single lemma may be utilized to produce several words with distinct meanings.

The concept of polarity is pervasive in natural language. It relates syntax, semantics and pragmatics narrowly (Giannakidou, in: Maienborn, von Heusinger, Portner (eds.), Semantics: an international handbook of natural language meaning, De Gruyter Mouton, Berlin, 2011; Israel in The grammar of polarity: pragmatics, sensitivity, and the logic of scales, Cambridge studies in linguistics, Cambridge University Press, Cambridge, 2014), it refers to items of many syntactic categories such as nouns, verbs and adverbs. Neutral polarity items appear in affirmative and negative sentences, negative polarity items cannot appear in affirmative sentences, and positive polarity items cannot appear in negative sentences. A way of reasoning in Natural Language is through Natural Logic (van Benthem in Essays in logical semantics, vol. 29 of Studies in linguistics and philosophy, Reidel, Dordrecht, 1986; Language in action: categories, lambdas, and dynamic logic, vol. 130 of Studies in logic, Elsevier, Amsterdam, 1991). This logic is based on the concept of polarity in order to make the meaning of a sentence weaker o stronger without changing its truth value. There exist many proposals to compute polarity in the Natural Logic context, the most widely known are the ones by: van Benthem (1986, 1991), Sanchez-Valencia (Studies on natural logic and categorial grammar, Ph.D. thesis, Universiteit van Amsterdam, 1991), Dowty (Proceedings of the 4th conference on semantics and theoretical linguistics, Cornel University, CLC Publications, Rochester, 1994), and van Eijck (in: ten Cate, Zeevat (eds.), 6th international Tbilisi symposium on logic, language, and computation, Batumi, Georgia, Springer, 2007). If Natural Logic is going to be used, as an inferential mechanism between text fragments, in Natural Language Processing applications such as text summarization, question answering, and information extraction, it is a priority to know what the existing relationship among the aforementioned algorithms is; for example, to implement the most general. We show in this paper the equivalence among the analyzed algorithms, filling a gap in Natural Logic research, particularly in computing polarity, and the soundness of their algorithms.

Natural Language Processing (NLP) involves the use of algorithms and models and various computational techniques to analyze, process, and generate natural language data, including speech and text. NLP helps computers interact with humans in a more natural way, which has become increasingly important as more humancomputer interactions take place. NLP allows machines to process and analyze voluminous unstructured data, including social media posts, newspaper articles, reviews from customers, emails, and others. It helps organizations extract insights, automate tasks, and improve decision-making by enabling machines to understand and generate human-like language. A linguistic background is essential for understanding NLP. Linguistic theories and models help in developing NLU systems, as NLP specialists need to understand the structure and rules of language. NLU systems are organized into different components, including language modelling, parsing, and semantic analysis. NLU systems may be assessed through the use of metrics that includes measures like precision and recall, as well as indicators that convey meaningful information that include F1 score and others. Semantics and knowledge representation are central to NLU, as they involve understanding the meaning of words and sentences and representing this information in a way that machines can use. Approaches to knowledge representation include semantic networks, ontologies, and vector embeddings. Language modelling is an essential step in NLP that sees usage in applications like speech recognition, text generation, and text completion and also in areas such as machine translation. Ambiguity Resolution remains a major challenge in NLP, as language is often ambiguous and context-dependent. Some common applications of NLP include sentiment analysis, chatbots, virtual assistants, machine translation, speech recognition, text classification, text summarization, and information extraction. In this chapter, we show the applicability of a popular unsupervised learning technique, viz., clustering through K-Means. The efficiency provided by the K-Means algorithm can be improved through the use of an optimization loop. The prospects for NLP are promising, with an increasing demand for AI-powered language technologies in various industries, including healthcare, finance, and e-commerce. There is also a growing need for ethical and responsible AI systems that are transparent and accountable.

In this paper we gauge the utility of general-purpose, open-domain semantic parsing for textual entailment recognition by combining graph-structured meaning representations with semantic technologies and formal reasoning tools. Our approach achieves high precision, and in two case studies we show that when reasoning over n-best analyses from the parser the performance of our system reaches stateof-the-art for rule-based textual entailment systems. 1 Background and Motivation There is a growing interest in recent years in general-purpose semantic parsing into graphbased meaning representations, which provide greater expressive power than tree-based structures. Recent efforts in this spirit include, for example, Abstract Meaning Representation (Banarescu et al., 2013), and Semantic Dependency Parsing (SDP) (Oepen et al., 2014; Oepen et al., 2015). Simultaneously, in the Semantic Web community, a range of generic semantic technologies for storing and processing graph-structured data has been made available, but these have not been much used for natural language processing tasks. We propose a flexible, generic framework for precision-oriented Textual Entailment (TE) recognition that combines semantic parsing, graph-based representations of sentence meaning, and semantic technologies. During the decade since the TE task was defined, (logical) inference-based approaches have made some important contributions to the field. Systems such as Bos and Markert (2006) and Tatu and Moldovan (2006) employ automated proof search over logical representations of the input sentences. Other systems, such as Bar-Haim et al. (2007), apply transformational rules to linguistic representations of the sentence pairs, and determine entailment through graph subsumption. Because inference-based systems are vulnerable to incomplete knowledge in the rule set and errors in the mapping from natural language sentences to logical forms or linguistics representations, and because the definition of the TE task encourages a more relaxed, non-logical notion of entailment, the majority of TE systems have used more robust approaches, however. Our work supports a notion of logical inference for TE by reasoning with formal rules over graph-structured meaning representations, while achieving results that are comparable with robust approaches. We use a freely available, grammar-driven semantic parser and a well-defined reduction of underspecified logical-form meaning representations into variable-free semantic graphs called Elementary Dependency Structures (EDS) (Oepen and Lonning, 2006). We capitalize on a pre-existing storage and search infrastructure for EDSs using generic semantic technologies. For entailment classification, we create inference rules that enrich the EDS graphs, apply the rules with a generic reasoner, and use graph alignment as a decision tool. To test our generic setup, we perform two case studies where we replicate well-performing TE systems, one from the Parser Evaluation using Textual Entailments (PETE) task (Yuret et al., 2010), and one from SemEval 2014 Task 1 (Marelli et al., 2014). The best published results for the PETE task, Lien (2014), were obtained through heuristic rules that align meaning representations based on structural similarity. Lien and Kouylekov (2014) extend the same basic approach for SemEval 2014 by including lexical relations and negation handling. We recast the handwritten heuristic rules from these systems as formal Semantic Web Rule Language (SWRL) rules, and run them with a generic reasoning tool over EDS

Paraphrase and textual entailment generation can support natural language processing (NLP) tasks that simulate text understanding, e.g., text summariza- tion, plagiarism detection, or question answering. A paraphrase, i.e., a sentence with the same meaning, conveys a certain piece of information with new words and new syntactic structures. Textual entailment, i.e., an inference that humans will judge most likely true, can employ real-world knowledge in order to make some implicit information explicit. Paraphrases can also be seen as mutual entailments. We present a new system that generates paraphrases and textual entailments from a given text in the Czech language. First, the process is rule-based, i.e., the system analyzes the input text, pro- duces its inner representation, transforms it according to transformation rules, and generates new sentences. Second, the generated sentences are ranked according to a statistical model and only the best ones are output. The decision whether a paraphrase or textual entailment is correct or not is left to humans. For this purpose we designed an annotation game based on a conversation between a detective (the human player) and his assis- tant (the system). The result of such annotation is a collection of annotated pairs text-hypothesis. Currently, the system and the game are intended to collect data in the Czech language. However, the idea can be applied for other languages. So far, we have collected 3,321 H-T pairs. From these pairs, 1,563 were judged cor- rect (47.06 %), 1,238 (37.28 %) were judged incorrect entailments, and 520 (15.66 %) were judged non-sense or unknown.

Paraphrase recognition is the task of Natural Language Processing of detecting if an expression restated as another expression contains the same information. Textual Entailment recognition, while being similar to paraphrase recognition, is a task that consists in finding out if a given text can be observed as a consequence of another text fragment, sometimes considering only part of the original meaning, or adding some inferences based on common sense. Tra- ditionally, for solving this problem, several lexical, syntactic and semantic based techniques are used. In this work, we seek to use the less resources as possi- ble, while being effective. For this, we perform a feature analysis for performing Paraphrase Recognition and recognizing Textual Entailment experimenting with the combination of several Natural Language Processing techniques like word overlapping, syntactic analysis, and elimination of stop words. Particularly, we explore using the syntactic n-grams technique combined with some auxiliary ap- proaches such as stemming, synonym detection, similarity measures and linear interpolation. We measure and compare the performance of our system by using the Microsoft Research Paraphrase Corpus, and the RTE-3 test set for Paraphras- ing and Textual Entailment, respectively. Syntactic n-grams produce good results for Paraphrase Recognition. As far as we know, syntactic n-grams had not been used for this task. For Textual Entailment, our best results were obtained by using a simple word overlapping algorithm based on stemming and elimination of stop words.

Summarization of Text is extracting important information from a body of text and present it in the form of a concise summary. The need for summarization has increased in recent times. The importance of having a simple concise summary of information in the news, business and research domains is paramount. Automatic text summarization is a well known task in the NLP (Natural Language Processing) field. Text Summarization techniques can be large divided into two groups: Extractive summarization and Abstractive summarization. Extractive summarization is based on identifying key sentences or phrases from the source text and grouping them to produce a summary without rewriting or paraphrasing the original text. Abstractive summarization is based on utilizing a deeper understanding of the source text and generating new sentences, not present in the original text, which improves the summary by reducing redundancy and focusing on the meaning of the source text. In this study we implement and compare the performance of various automatic summarization methods in order to gain insight into how long the methods take to implement and how accurate and human-like the generated summaries are. We aim to learn the pros and cons of the various techniques used by utilizing summary scoring as well as manual inspection of generated summary.

Leveraging Transformer Summarizer to Extract Sentences for Arabic Text Summarization

Quality Estimation (QE) and error analysis of Machine Translation (MT) output remain active areas in Natural Language Processing (NLP) research. Many recent efforts have focused on Machine Learning (ML) systems to estimate the MT quality, translation errors, post-editing speed or post-editing effort. As the accuracy of such ml tasks relies on the availability of corpora, there is an increasing need for large corpora of machine translations annotated with translation errors and the error annotation guidelines to produce consistent annotations. Drawing on previous work on translation error taxonomies, we present the SCATE (Smart Computer-aided Translation Environment) mt error taxonomy, which is hierarchical in nature and is based upon the familiar notions of accuracy and fluency. In the scate annotation framework, we annotate fluency errors in the target text and accuracy errors in both the source and target text, while linking the source and target annotations. We also propose a novel method for alignment-based Inter-Annotator Agreement (IAA) analysis and show that this method can be used effectively on large annotation sets. Using the scate taxonomy and guidelines, we create the first corpus of MT errors for the English-Dutch language pair, consisting of Statistical Machine Translation (SMT) and Rule-Based Machine Translation (RBMT) errors, which is a valuable resource not only for NLP tasks in this field but also to study the relationship between mt errors and post-editing efforts in the future. Finally, we analyse the error profiles of the smt and the rbmt systems used in this study and compare the quality of these two different mt architectures based on the error types.

Question answering (QA) is a task that deserves more collaboration between natural language processing (NLP) and knowledge representation (KR) communities, not only to introduce reasoning when looking for answers or making use of answer type taxonomies and encyclopaedic knowledge, but also, as discussed here, for answer validation (AV), that is to say, to decide whether the responses of a QA system are correct or not. This was one of the motivations for the first Answer Validation Exercise at CLEF 2006 (AVE 2006). The starting point for the AVE 2006 was the reformulation of the answer validation as a recognizing textual entailment (RTE) problem, under the assumption that a hypothesis can be automatically generated instantiating a hypothesis pattern with a QA system answer. The test collections that we developed in seven different languages at AVE 2006 are specially oriented to the development and evaluation of answer validation systems. We show in this article the methodology followed for developing these collections taking advantage of the human assessments already made in the evaluation of QA systems. We also propose an evaluation framework for AV linked to a QA evaluation track. We quantify and discuss the source of errors introduced by the reformulation of the answer validation problem in terms of textual entailment (around 2%, in the range of inter-annotator disagreement). We also show the evaluation results of the first answer validation exercise at CLEF 2006 where 11 groups have participated with 38 runs in seven different languages. The most extensively used techniques were Machine Learning and overlapping measures, but systems with broader knowledge resources and richer representation formalisms obtained the best results.

Machine-readable semantic knowledge in the form of taxonomies (i.e., a collection of is-a edges) has proved to be beneficial in an array of NLP tasks including inference, textual entailment, question answering and information extraction. Such widespread utility of taxonomies has led to multiple large-scale manual efforts towards taxonomy induction such as WordNet and Cyc. However, manual construction of taxonomies is time-intensive, and usually, requires substantial annotation efforts by domain experts. Furthermore, the resulting taxonomies suffer from low coverage and are unavailable for specific domains or languages. Therefore, in recent years, there has been a growing body of work, which aims to induce taxonomies automatically, either from semi-structured knowledge resources or unstructured text. In this thesis, we focus on the task of automated taxonomy induction under a variety of different settings. We first focus on taxonomy induction from the largest semi-structured knowledge resource, i.e., Wikipedia. More specifically, we introduce a set of novel heuristics aimed towards inducing a large-scale taxonomy from the English Wikipedia categories network. We also propose a novel comprehensive path-based evaluation framework for taxonomies. Taxonomy induced using our approach significantly outperforms the state of the art across both edge-based as well as path-based evaluation metrics. Moreover, our experiments also demonstrate that high accuracy in edge-based evaluation metrics does not always translate to high-accuracy in path-based evaluation metrics. Subsequently, we propose a novel approach, which leverages the interlanguage links of Wikipedia to induce taxonomies in other Wikipedia languages. Compared to the state of the art, our approach is simpler, more principled, and results in taxonomies that are significantly more accurate across both edge-based and path-based evaluation metrics. In the second part of this thesis, we focus on taxonomy induction from unstructured text. We propose a novel approach towards taxonomy induction from an input vocabulary of seed terms. Unlike previous approaches, which extract singular hypernym edges for terms, our approach utilizes a novel probabilistic framework to extract long-range hypernym subsequences. Taxonomy induction from the extracted subsequences is cast as an instance of the minimum-cost flow optimization problem on a carefully-designed flow network. Through experiments, we demonstrate that our approach outperforms the state of the art across four languages. We also show that our approach is robust to the presence of noise in the input vocabulary. Our approach facilitates the relaxation of many simplifying assumptions, which were employed by previous taxonomy induction approaches. As a result, our work serves to automate the process of taxonomy induction from unstructured text in the true sense. Finally, we introduce a task of discovering and generalizing lexicalized templates from the titles of Wikipedia entities. The experimental results on this task demonstrate that taxonomies, which perform better on our proposed path-based metrics, result in a more accurate set of generalizations for a given set of entities. In summary, this thesis proposes new approaches towards automated taxonomy induction. It improves upon the state of the art in a variety of different settings. It also relaxes many simplifying assumptions that limited the applicability of prior approaches.