DISCOS: Bridging the Gap between Discourse Knowledge and Commonsense Knowledge

CoMaSa:Context Multi-aware Self-attention for emotional response generation

Computers are increasingly expected to make smart decisions based on what humans consider commonsense. This would require computers to understand their environment, including properties of objects in the environment (e.g., a wheel is round), relations between objects (e.g., two wheels are part of a bike, or a bike is slower than a car) and interactions of objects (e.g., a driver drives a car on the road).\n\nThe goal of this dissertation is to investigate automated methods for acquisition of large-scale, semantically organized commonsense knowledge. Prior state-of-the-art methods to acquire commonsense are either not automated or based on shallow representations. Thus, they cannot produce large-scale, semantically organized commonsense knowledge.\n\nTo achieve the goal, we divide the problem space into three research directions, constituting our core contributions:\n1. Properties of objects: acquisition of properties like hasSize, hasShape, etc. We develop WebChild, a semi-supervised method to compile semantically organized properties.\n2. Relationships between objects: acquisition of relations like largerThan, partOf, memberOf, etc. We develop CMPKB, a linear-programming based method to compile comparative relations, and, we develop PWKB, a method based on statistical and logical inference to compile part-whole relations.\n3. Interactions between objects: acquisition of activities like drive a car, park a car, etc., with attributes such as temporal or spatial attributes. We develop Knowlywood, a method based on semantic parsing and probabilistic graphical models to compile activity knowledge.\n\nTogether, these methods result in the construction of a large, clean and semantically organized Commonsense Knowledge Base that we call WebChild KB.

Text compression is of considerable theoretical and practical interest. It is, for example, becoming increasingly important for satisfying the requirements of fitting a large database onto a single CD-ROM. Many of the compression techniques discussed in the literature are model based. We here propose the notion of a formal grammar as a flexible model of text generation that encompasses most of the models offered before as well as, in principle, extending the possibility of compression to a much more general class of languages. Assuming a general model of text generation, a derivation is given of the well known Shannon entropy formula, making possible a theory of information based upon text representation rather than on communication. The ideas are shown to apply to a number of commonly used text models. Finally, we focus on a Markov model of text generation, suggest an information theoretic measure of similarity between two probability distributions, and develop a clustering algorithm based on this measure. This algorithm allows us to cluster Markov states, and thereby base our compression algorithm on a smaller number of probability distributions than would otherwise have been required. A number of theoretical consequences of this approach to compression are explored, and a detailed example is given.

Large-scale pre-trained language models have demonstrated strong knowledge representation ability. However, recent studies suggest that even though these giant models contains rich simple commonsense knowledge (e.g., bird can fly and fish can swim.), they often struggle with the complex commonsense knowledge that involves multiple eventualities (verb-centric phrases, e.g., identifying the relationship between ``Jim yells at Bob'' and ``Bob is upset'').To address this problem, in this paper, we propose to help pre-trained language models better incorporate complex commonsense knowledge. Different from existing fine-tuning approaches, we do not focus on a specific task and propose a general language model named CoCoLM. Through the careful training over a large-scale eventuality knowledge graphs ASER, we successfully teach pre-trained language models (i.e., BERT and RoBERTa) rich complex commonsense knowledge among eventualities. Experiments on multiple downstream commonsense tasks that requires the correct understanding of eventualities demonstrate the effectiveness of CoCoLM.

Reasoning over commonsense knowledge bases (CSKBs) whose elements are in the form of free-text is an important yet hard task in NLP.While CSKB completion only fills the missing links within the domain of the CSKB, CSKB population is alternatively proposed with the goal of reasoning unseen assertions from external resources.In this task, CSKBs are grounded to a large-scale eventuality (activity, state, and event) graph to discriminate whether novel triples from the eventuality graph are plausible or not.However, existing evaluations on the population task are either not accurate (automatic evaluation with randomly sampled negative examples) or of small scale (human annotation).In this paper, we benchmark the CSKB population task with a new large-scale dataset by first aligning four popular CSKBs, and then presenting a highquality human-annotated evaluation set to probe neural models' commonsense reasoning ability.We also propose a novel inductive commonsense reasoning model that reasons over graphs.Experimental results show that generalizing commonsense reasoning on unseen assertions is inherently a hard task.Models achieving high accuracy during training perform poorly on the evaluation set, with a large gap between human performance.Codes and

Technologies based on “artificial intelligence” (AI) are transforming every part of our society, including healthcare and medical institutions. An example of this trend is the novel field in oncology and radiology called radiomics, which is the extracting and mining of large-scale quantitative features from medical imaging by machine-learning (ML) algorithms. This paper explores situated work with a radiomics software platform, QuantImage (v2), and interaction around it, in educationally framed hands-on trial sessions where pairs of novice users (physicians and medical radiology technicians) work on a radiomics task consisting of developing a predictive ML model with a co-present tutor. Informed by ethnomethodology and conversation analysis (EM/CA), the results show that learning about radiomics more generally and learning how to use this platform specifically are deeply intertwined. Common-sense knowledge (e.g., about meanings of colors) can interfere with the visual representation standards established in the professional domain. Participants' skills in using the platform and knowledge of radiomics are routinely displayed in the assessment of performance measures of the resulting ML models, in the monitoring of the platform's pace of operation for possible problems, and in the ascribing of independent actions (e.g., related to algorithms) to the platform. The findings are relevant to current discussions about the explainability of AI in medicine as well as issues of machinic agency.

Precisely defining the terminology is the first step in scientific communication. Developing neural text generation models for definition generation can circumvent the labor-intensity curation, further accelerating scientific discovery. Unfortunately, the lack of large-scale terminology definition dataset hinders the process toward definition generation. In this paper, we present a large-scale terminology definition dataset Graphine covering 2,010,648 terminology definition pairs, spanning 227 biomedical subdisciplines. Terminologies in each subdiscipline further form a directed acyclic graph, opening up new avenues for developing graph-aware text generation models. We then proposed a novel graph-aware definition generation model Graphex that integrates transformer with graph neural network. Our model outperforms existing text generation models by exploiting the graph structure of terminologies. We further demonstrated how Graphine can be used to evaluate pretrained language models, compare graph representation learning methods and predict sentence granularity. We envision Graphine to be a unique resource for definition generation and many other NLP tasks in biomedicine.

Machine learning algorithms represent the intelligence that controls many information systems and applications around us. As such, they are targeted by attackers to impact their decisions. Text created by machine learning algorithms has many types of applications, some of which can be considered malicious especially if there is an intention to present machine-generated text as human-generated. In this paper, we surveyed major subjects in adversarial machine learning for text processing applications. Unlike adversarial machine learning in images, text problems and applications are heterogeneous. Thus, each problem can have its own challenges. We focused on some of the evolving research areas such as: malicious versus genuine text generation metrics, defense against adversarial attacks, and text generation models and algorithms. Our study showed that as applications of text generation will continue to grow in the near future, the type and nature of attacks on those applications and their machine learning algorithms will continue to grow as well. Literature survey indicated an increasing trend in using pre-trained models in machine learning. Word/sentence embedding models and transformers are examples of those pre-trained models. Adversarial models may utilize same or similar pre-trained models as well. In another trend related to text generation models, literature showed effort to develop universal text perturbations to be used in both black-and white-box attack settings. Literature showed also using conditional GANs to create latent representation for writing types. This usage will allow for a seamless lexical and grammatical transition between various writing styles. In text generation metrics, research trends showed developing successful automated or semi-automated assessment metrics that may include human judgement. Literature showed also research trends of designing and developing new memory models that increase performance and memory utilization efficiency without validating real-time constraints. Many research efforts evaluate different defense model approaches and algorithms. Researchers evaluated different types of targeted attacks, and methods to distinguish human versus machine generated text.

Introduction This paper introduces MAKEBELIEVE, an interactive story generation agent that uses commonsense knowledge to generate short fictional texts from an initial seed story step supplied by the user. A subset of commonsense describing causality, such as the sentence “a consequence of drinking alcohol is intoxication,” is selected from the ontology of the Open Mind Commonsense Knowledge Base (Singh, 2002). Binary causal relations are extracted from these sentences and stored as crude trans-frames (Minsky, 1988). By performing fuzzy, creativity-driven inference over these frames, creative “causal chains” are produced for use in story generation. The current system has mostly local pair-wise constraints between steps in the story, though global constraints such as narrative structure are being added. Our motivation for this project stems from two questions: Can a large-scale knowledge base of commonsense benefit the artificial intelligence community by supplying new knowledge and methods to tackle difficult AI problems? And given that full commonsense reasoning has yet to mature, can we demonstrate any success with a more “fail-soft” approach? We picked story generation because we feel it is a classic AI problem that can be approached with a creative use of commonsense knowledge. Compared to problem solving or question answering, story generation is a “softer” problem where there is no wrong solution per se, and that solution is evaluated subjectively. This paper is organized as follows. In the first section we frame our approach in the context of previous work in story generation. In the second section, we present the system’s representation of commonsense and techniques for fuzzy, creativity-driven inference. The third section presents an overview of the story generation architecture of MAKEBELIEVE. We conclude with an evaluation of the system followed by some discussion.

Commonsense knowledge (CSK) about concepts and their properties is helpful for AI applications. Prior works, such as ConceptNet, have compiled large CSK collections. However, they are restricted in their expressiveness to subject-predicate-object (SPO) triples with simple concepts for S and strings for P and O. This paper presents a method called ASCENT++ to automatically build a large-scale knowledge base (KB) of CSK assertions, with refined expressiveness and both better precision and recall than prior works. ASCENT++ goes beyond SPO triples by capturing composite concepts with subgroups and aspects, and by refining assertions with semantic facets. The latter is essential to express the temporal and spatial validity of assertions and further qualifiers. Furthermore, ASCENT++ combines open information extraction (OpenIE) with judicious cleaning and ranking by typicality and saliency scores. For high coverage, our method taps into the large-scale crawl C4 with broad web contents. The evaluation with human judgments shows the superior quality of the ASCENT++ KB, and an extrinsic evaluation for QA-support tasks underlines the benefits of ASCENT++. A web interface, data, and code can be accessed at https://ascentpp.mpi-inf.mpg.de/.

This study introduces a comparative framework designed to evaluate the predictive capabilities of two distinct modeling approaches: traditional consideration and choice set generation models, and Machine Learning (ML) models. The framework focuses on a critical and often overlooked stage in decision–making process, which includes the set of alternatives that were considered prior to making a choice. To demonstrate the framework’s applicability, a synthetic route choice scenario was employed as a case study. In this context, routes with randomly assigned attributes were generated alongside synthetic populations of decision-makers, whose socioeconomic profiles were sampled from observed survey data. Each agent was assigned a set generation models to construct their individual consideration sets. ML models were then trained on these consideration sets to predict the considered alternatives of independent decision-maker populations. Two population scenarios were examined: one homogeneous, where all agents used the same set generation model, and one heterogeneous, where models were randomly assigned across agents. In both cases, the accuracy of the prediction was evaluated, along with a comparison between existing set generation and predictive models. The results showed that certain ML models can achieve over 80% accuracy in predicting individual considered items and over 60% accuracy in predicting entire consideration sets, across both population scenarios.

Computer assisted writing system

Recent text generation models are easy to generate relevant and fluent text for the given text, while lack of causal reasoning ability when we change some parts of the given text. Counterfactual story rewriting is a recently proposed task to test the causal reasoning ability for text generation models, which requires a model to predict the corresponding story ending when the condition is modified to a counterfactual one. Previous works have shown that the traditional sequence-to-sequence model cannot well handle this problem, as it often captures some spurious correlations between the original and counterfactual endings, instead of the causal relations between conditions and endings. To address this issue, we propose a sketch-and-customize generation model guided by the causality implicated in the conditions and endings. In the sketch stage, a skeleton is extracted by removing words which are conflict to the counterfactual condition, from the original ending. In the customize stage, a generation model is used to fill proper words in the skeleton under the guidance of the counterfactual condition. In this way, the obtained counterfactual ending is both relevant to the original ending and consistent with the counterfactual condition. Experimental results show that the proposed model generates much better endings, as compared with the traditional sequence-to-sequence model.

Assessment center (AC) exercises such as role‐plays have established themselves as valuable approaches for obtaining insights into interpersonal behavior, but they are often considered the “Rolls Royce” of personnel assessment due to their high costs. The observation and rating process comprises a substantial part of these costs. In an exploratory case study, we capitalize on recent advances in natural language processing (NLP) by developing NLP‐based machine learning (ML) models to investigate the possibility of automatically scoring AC exercises. First, we compared the convergent‐related validity and contamination with word count of ML scores based on models that used different NLP methods to operationalize verbal behavior. Second, for the model that maximized convergence while minimizing contamination with word count (i.e., a model that used both n‐grams and Universal Sentence Encoder embeddings as predictors), we investigated the criterion‐related validity of its scores. Third, we examined how the interrater reliability of the AC role‐play scores affects ML model convergence. To do so, we applied seven NLP methods to 96 assessees' transcriptions and trained 10 sets of ML models across 18 speeded AC role‐plays to automatically score assessee performance. Results suggest that ML scores recovered most of the original variance in the overall assessment ratings, and replacing one or more human assessors with ML scores maintained criterion‐related validity. Additionally, ML models seemed to exhibit higher convergence when assessors consistently detected and utilized observable behaviors to make ratings (i.e., when interrater reliability was higher). Finally, we provide a step‐by‐step guide for practitioners seeking to implement ML scoring in ACs.

Story generation, namely, generating a reasonable story from a leading context, is an important but challenging task. In spite of the success in modeling fluency and local coherence, existing neural language generation models (e.g., GPT-2) still suffer from repetition, logic conflicts, and lack of long-range coherence in generated stories. We conjecture that this is because of the difficulty of associating relevant commonsense knowledge, understanding the causal relationships, and planning entities and events with proper temporal order. In this paper, we devise a knowledge-enhanced pretraining model for commonsense story generation. We propose to utilize commonsense knowledge from external knowledge bases to generate reasonable stories. To further capture the causal and temporal dependencies between the sentences in a reasonable story, we use multi-task learning, which combines a discriminative objective to distinguish true and fake stories during fine-tuning. Automatic and manual evaluation shows that our model can generate more reasonable stories than state-of-the-art baselines, particularly in terms of logic and global coherence.