Realistic Evaluation Principles for Cross-document Coreference Resolution

Cross-document co-reference resolution (CCR) computes equivalence classes over textual mentions denoting the same entity in a document corpus.Named-entity linking (NEL) disambiguates mentions onto entities present in a knowledge base (KB) or maps them to null if not present in the KB.Traditionally, CCR and NEL have been addressed separately.However, such approaches miss out on the mutual synergies if CCR and NEL were performed jointly.This paper proposes C3EL, an unsupervised framework combining CCR and NEL for jointly tackling both problems.C3EL incorporates results from the CCR stage into NEL, and vice versa: additional global context obtained from CCR improves the feature space and performance of NEL, while NEL in turn provides distant KB features for already disambiguated mentions to improve CCR.The CCR and NEL steps are interleaved in an iterative algorithm that focuses on the highest-confidence still unresolved mentions in each iteration.Experimental results on two different corpora, news-centric and web-centric, demonstrate significant gains over state-of-the-art baselines for both CCR and NEL.

Long-form conversations such as email are an important source of information for knowledge capture. For tasks such as knowledge graph construction, conversational search, and entity linking, being able to resolve entities from across documents is important. Building on recent work on within document coreference resolution for email, we study for the first time a cross-document formulation of the problem. Our results show that the current state-of-the-art deep learning models for general cross-document coreference resolution are insufficient for email conversations. Our experiments show that the general task is challenging and, importantly for knowledge intensive tasks, coreference resolution models that only treat entity mentions perform worse. Based on these results, we outline the work needed to address this challenging task.

Towards an integrated weed management decision support system: A simulation model for weed-crop competition and control

Datasets and methods for cross-document coreference resolution (CDCR) focus on events or entities with strict coreference relations. They lack, however, annotating and resolving coreference mentions with more abstract or loose relations that may occur when news articles report about controversial and polarized events. Bridging and loose coreference relations trigger associations that may expose news readers to bias by word choice and labeling. For example, coreferential mentions of “direct talks between U.S. President Donald Trump and Kim” such as “an extraordinary meeting following months of heated rhetoric” or “great chance to solve a world problem” form a more positive perception of this event. A step towards bringing awareness of bias by word choice and labeling is the reliable resolution of coreferences with high lexical diversity. We propose an unsupervised method named XCoref, which is a CDCR method that capably resolves not only previously prevalent entities, such as persons, e.g., “Donald Trump,” but also abstractly defined concepts, such as groups of persons, “caravan of immigrants,” events and actions, e.g., “marching to the U.S. border.” In an extensive evaluation, we compare the proposed XCoref to a state-of-the-art CDCR method and a previous method TCA that resolves such complex coreference relations and find that XCoref outperforms these methods. Outperforming an established CDCR model shows that the new CDCR models need to be evaluated on semantically complex mentions with more loose coreference relations to indicate their applicability of models to resolve mentions in the “wild” of political news articles.KeywordsCross-document coreference resolutionNews analysisMedia bias

Event coreference resolution is a task in which different text fragments that refer to the same real-world event are automatically linked together. This task can be performed not only within a single document but also across different documents and can serve as a basis for many useful Natural Language Processing applications. Resources for this type of research, however, are extremely limited. We compiled the first large-scale dataset for cross-document event coreference resolution in Dutch, comparable in size to the most widely used English event coreference corpora. As data for event coreference is notoriously sparse, we took additional steps to maximize the number of coreference links in our corpus. Due to the complex nature of event coreference resolution, many algorithms consist of pipeline architectures which rely on a series of upstream tasks such as event detection, event argument identification and argument coreference. We tackle the task of event argument coreference to both illustrate the potential of our compiled corpus and to lay the groundwork for a Dutch event coreference resolution system in the future. Results show that existing NLP algorithms can be easily retrofitted to contribute to the subtasks of an event coreference resolution pipeline system.

Cross-document coreference resolution plays an import part in the filed of natural language processing (NLP). It captures the ability of gathering documents for information about a certain entity. Most previous algorithms identify the underlying entity of a given document depending on the original text, which is unreliable if the original text contains multiple parts of different themes. In this paper, we propose a cross-document coreference resolution algorithm based on automatic text summary instead of the original text. In our approach, we extract query-specific and informative-indicative summary from the original text by using Hobbs algorithm and measure the similarity between two summaries. This automatic text summary-based cross-document coreference resolution (ATSCDCR) system is effective in disambiguating different entities of the same mention name and identifying the same entity of different mention names. The results from our experiments show that the macro average of ATSCDCR system is up to 73.16% and the micro average of ATSCDCR system is 67.34 %.

We aim to shed light on the state-of-the-art in NP coreference resolution by teasing apart the differences in the MUC and ACE task definitions, the assumptions made in evaluation methodologies, and inherent differences in text corpora. First, we examine three subproblems that play a role in coreference resolution: named entity recognition, anaphoricity determination, and coreference element detection. We measure the impact of each subproblem on coreference resolution and confirm that certain assumptions regarding these subproblems in the evaluation methodology can dramatically simplify the overall task. Second, we measure the performance of a state-of-the-art coreference resolver on several classes of anaphora and use these results to develop a quantitative measure for estimating coreference resolution performance on new data sets.

Background: Administration of nebulized medications is a very common practice in the ICU. Although several aerosol generating devices can be selected, most ICUs use the small volume jet nebulizers. Although some recommendations on where to place aerosol generators when administering aerosol treatments to patients undergoing invasive and noninvasive mechanical ventilation or high-flow nasal cannula (HFNC) have been published, there are still many inconsistencies in the clinical setting. Placement of these aerosol generators can be adversely affected by humidification devices and lead to significant changes in pulmonary mechanics. The goal of this study was to determine where respiratory therapists (RTs) place nebulizer when administering a treatment in the ICU. Methods: For this prospective, observational study, pediatric and adult subjects who were placed on invasive mechanical ventilation (IMV), and noninvasive mechanical ventilation (NIV) or high flow nasal cannula (HFNC) with an order to receive a nebulized treatment were included. The study was conducted at a large university-affiliated hospital in Texas and was approved by the IRB. The research team communicated with RTs when it was time to administer an aerosol treatment to subjects in the ICU. Once the subject was identified, the electronic medical record was accessed to collect demographic information and placement of the nebulizer was documented in the data collection form as the treatment was administered. Results: A total of 32 subjects were selected for analysis; 18 from the pediatric ICU and 14 from the adult ICU. The majority of the sample was on IMV (50%), followed by HFNC (28.1%), and NIV (21.9%). The nebulizer was placed on the dry side of the heated humidifier (HH) in every patient on IMV. For all patients on NIV, nebulizers were placed on the inspiratory limb of the circuit between the interface and the exhalation valve. When placed on HFNC, the nebulizer was connected to the dry side of the HH in 3 subjects while 6 subjects had their nebulizers placed on the inspiratory limb of the breathing circuit. Conclusions: The results of this study suggest that while there is variation in regard to where RTs place the aerosol generator when using HFNC, they are consistent in their practice of placing the nebulizer when using IMV or NIV as recommended in published guidelines. Further evaluation of practices while delivering aerosol treatments to patients on HFNC may be required.

System-Level Demand Forecasting and Risk-Aware Market Strategy Modeling for Sustainable Infrastructure Growth

This work aims to assess standard evaluation practices used by the research community for evaluating medical imaging classifiers, with a specific focus on the implications of class imbalance. The analysis is performed on chest X-rays as a case study and encompasses a comprehensive model performance definition, considering both discriminative capabilities and model calibration. We conduct a concise literature review to examine prevailing scientific practices used when evaluating X-ray classifiers. Then, we perform a systematic experiment on two major chest X-ray datasets to showcase a didactic example of the behavior of several performance metrics under different class ratios and highlight how widely adopted metrics can conceal performance in the minority class. Our literature study confirms that: (1) even when dealing with highly imbalanced datasets, the community tends to use metrics that are dominated by the majority class; and (2) it is still uncommon to include calibration studies for chest X-ray classifiers, albeit its importance in the context of healthcare. Moreover, our systematic experiments confirm that current evaluation practices may not reflect model performance in real clinical scenarios and suggest complementary metrics to better reflect the performance of the system in such scenarios. Our analysis underscores the need for enhanced evaluation practices, particularly in the context of class-imbalanced chest X-ray classifiers. We recommend the inclusion of complementary metrics such as the area under the precision-recall curve (AUC-PR), adjusted AUC-PR, and balanced Brier score, to offer a more accurate depiction of system performance in real clinical scenarios, considering metrics that reflect both, discrimination and calibration performance. This study underscores the critical need for refined evaluation metrics in medical imaging classifiers, emphasizing that prevalent metrics may mask poor performance in minority classes, potentially impacting clinical diagnoses and healthcare outcomes. Common scientific practices in papers dealing with X-ray computer-assisted diagnosis (CAD) systems may be misleading. We highlight limitations in reporting of evaluation metrics for X-ray CAD systems in highly imbalanced scenarios. We propose adopting alternative metrics based on experimental evaluation on large-scale datasets.

International audience

A number of effective classification algorithms have been developed for spoken language recognition, and it has been a common practice in the NIST Language Recognition Evaluations (LREs) that an information fusion is applied to boost the performance of the recognition system. This paper investigates the fusion of multiple output scores generated using different classifiers that complement to further reduce the classification error rate in spoken language recognition. We introduce a local performance metric to optimize the performance of the classifier fusion. The experiments are conducted on the 2009 NIST LRE corpus. The experimental results show that the proposed fusion effectively improves the performance over individual classifiers.

Current structural design philosophies rely on the inelastic capacity of structures for resisting seismic excitations. To assess such capacity, cyclic loading protocols have been used as a common practice in laboratory and numerical evaluations. The main objective was to obtain quasi-static loading protocols that reflect the increase in the inelastic demands of bridge columns subjected to long duration ground motions caused by subduction megathrust earthquakes. To study the demands imposed on the structural system, results from nonlinear time-history analyses considering numerous subduction ground motions imposed on structures with a wide range of structural periods and target ductilities were used. Because the number of inelastic cycles and the expected damage can be closely related, statistical analyses of the number of inelastic cycles and the cumulative inelastic demands were used to develop the loading protocols. Due to the observed dependence of the demand parameters on the structural period, different protocols were developed for short-, medium-, and long-period responses. The proposed cyclic deformation histories are more representative of the inelastic demands from subduction earthquakes; therefore, their application would improve the seismic assessment of bridge columns exposed to such hazards.

It is common practice in wireless multihop network evaluations to ignore interfering signals below a certain signal strength threshold. This paper investigates the thesis that this produces highly inaccurate evaluations in many cases. We start by defining a bounded version of the physical interference model, in which interference generated by transmitters located beyond a certain distance s from a receiver is ignored. We then derive a lower bound on neglected interference and show that it is approximately two orders of magnitude greater than the noise floor for typical parameter values and a surprisingly small number of nodes. We next evaluate the effect of neglected interference through extensive simulations done with a widely-used packet-level simulator (GTNetS), considering 802.11 MAC with both CBR and TCP traffic in networks of varying size and topology. The results of these simulations show very large evaluation errors when neglecting far-away interference: errors in evaluating aggregate throughput when using the default interference model reached up to 210% with 100 nodes, and errors in individual flow throughput were far greater.

A retrospective review of initial early intervention and early childhood special education (EI/ECSE) evaluation reports was completed to identify practices specific to the evaluation of Latinx children’s communication development. A records abstraction approach employed entailed a random selection of 294 EI/ECSE evaluation reports completed in Oregon with Latinx children with communication concerns. The reports were coded for documentation of background information, evaluation procedures, and eligibility decisions. Quantitative and qualitative analyses were completed to identify common evaluation practices. Adequate evaluation practices documented included the use of a multidisciplinary evaluation team, administration of multiple evaluation measures, and provision of basic information on children’s language background. EI/ECSE providers did not consistently specify children’s language exposure/use, denote the language of the evaluation, prioritize appropriate assessment measures, and indicate attention to cultural and linguistic differences when determining eligibility. Recommendations for decreasing bias in the evaluation of Latinx children and culturally and linguistically diverse (CLD) populations more broadly are offered.