IR meets NLP

Graph theory and the fields of natural language processing and information retrieval are well-studied disciplines. Traditionally, these areas have been perceived as distinct, with different algorithms, different applications and different potential end-users. However, recent research has shown that these disciplines are intimately connected, with a large variety of natural language processing and information retrieval applications finding efficient solutions within graph-theoretical frameworks. This book extensively covers the use of graph-based algorithms for natural language processing and information retrieval. It brings together topics as diverse as lexical semantics, text summarization, text mining, ontology construction, text classification and information retrieval, which are connected by the common underlying theme of the use of graph-theoretical methods for text and information processing tasks. Readers will come away with a firm understanding of the major methods and applications in natural language processing and information retrieval that rely on graph-based representations and algorithms.

Learning a high-dimensional dense representation for vocabulary terms, also known as a word embedding, has recently attracted much attention in natural language processing and information retrieval tasks. The embedding vectors are typically learned based on term proximity in a large corpus. This means that the objective in well-known word embedding algorithms, e.g., word2vec, is to accurately predict adjacent word(s) for a given word or context. However, this objective is not necessarily equivalent to the goal of many information retrieval (IR) tasks. The primary objective in various IR tasks is to capture relevance instead of term proximity, syntactic, or even semantic similarity. This is the motivation for developing unsupervised relevance-based word embedding models that learn word representations based on query-document relevance information. In this paper, we propose two learning models with different objective functions; one learns a relevance distribution over the vocabulary set for each query, and the other classifies each term as belonging to the relevant or non-relevant class for each query. To train our models, we used over six million unique queries and the top ranked documents retrieved in response to each query, which are assumed to be relevant to the query. We extrinsically evaluate our learned word representation models using two IR tasks: query expansion and query classification. Both query expansion experiments on four TREC collections and query classification experiments on the KDD Cup 2005 dataset suggest that the relevance-based word embedding models significantly outperform state-of-the-art proximity-based embedding models, such as word2vec and GloVe.

Keyword extraction is an important task in natural language processing, supporting summarization, indexing, content analysis, and information retrieval. Over time, a wide range of methods have been developed to automate the identification of key concepts in text. This study reviews the main families of keyword extraction techniques, including statistical methods, graph-based algorithms, embedding-based models, and approaches that use neural representations. In addition to summarizing existing research, the study evaluates several widely used tools, such as TF-IDF, YAKE, KeyBERT, KPMiner, TextRank, TopicRank, and MultipartiteRank, and examines how their strengths and limitations appear when applied to real educational materials. Special attention is given to the challenges that arise in multilingual and low-resource settings, particularly for languages with fewer annotated resources. The analysis highlights patterns in algorithm behavior, differences in semantic coverage, and factors that influence performance across syllabus formats and subject areas. The findings aim to help researchers and practitioners choose appropriate keyword extraction methods for educational, multilingual, and domain-specific applications.

Determining the extent to which two text snippets are semantically equivalent is a well-researched topic in the areas of natural language processing, information retrieval and text summarization. The sentence-to-sentence similarity scoring is extensively used in both generic and query-based summarization of documents as a significance or a similarity indicator. Nevertheless, most of these applications utilize the concept of semantic similarity measure only as a tool, without paying importance to the inherent properties of such tools that ultimately restrict the scope and technical soundness of the underlined applications. This paper aims to contribute to fill in this gap. It investigates three popular WordNet hierarchical semantic similarity measures, namely path-length, Wu and Palmer and Leacock and Chodorow, from both algebraical and intuitive properties, highlighting their inherent limitations and theoretical constraints. We have especially examined properties related to range and scope of the semantic similarity score, incremental monotonicity evolution, monotonicity with respect to hyponymy/hypernymy relationship as well as a set of interactive properties. Extension from word semantic similarity to sentence similarity has also been investigated using a pairwise canonical extension. Properties of the underlined sentence-to-sentence similarity are examined and scrutinized. Next, to overcome inherent limitations of WordNet semantic similarity in terms of accounting for various Part-of-Speech word categories, a WordNet “All word-To-Noun conversion” that makes use of Categorial Variation Database (CatVar) is put forward and evaluated using a publicly available dataset with a comparison with some state-of-the-art methods. The finding demonstrates the feasibility of the proposal and opens up new opportunities in information retrieval and natural language processing tasks.

On a wide range of natural language processing and information retrieval tasks, transformer-based models, particularly pre-trained language models like BERT, have demonstrated tremendous effectiveness. Due to the quadratic complexity of the self-attention mechanism, however, such models have difficulties processing long documents. Recent works dealing with this issue include truncating long documents, in which case one loses potential relevant information, segmenting them into several passages, which may lead to miss some information and high computational complexity when the number of passages is large, or modifying the self-attention mechanism to make it sparser as in sparse-attention models, at the risk again of missing some information. We follow here a slightly different approach in which one first selects key blocks of a long document by local query-block pre-ranking, and then few blocks are aggregated to form a short document that can be processed by a model such as BERT. Experiments conducted on standard Information Retrieval datasets demonstrate the effectiveness of the proposed approach.

Metonymy resolution (MR) is a crucial challenge in natural language understanding and information retrieval. Recent large-scale pre-trained language models have shown promising results in various natural language processing (NLP) tasks, including MR. Despite these achievements, current models still struggle in many real-world scenarios. Since these models rely heavily on contextual information and ignore entity information, they are prone to extract irrelevant features and overfit when fine-tuned with less training data. In this article, we propose a mixed information bottleneck framework to address the above issues, which learns optimal data representations based on the principle of minimal sufficiency. Our model can effectively mitigate irrelevant features in context and entity by using different types of information bottlenecks for entity and context information separately while reducing the dimensionality of latent representations. We show that our approach achieves state-of-the-art performance on three benchmark datasets for location MR, outperforming previous Bert-based methods by a large margin. Ablation studies and qualitative analysis show the effectiveness of our models in reducing dimensionality while extracting more relevant features.

Events play an important role in natural language processing and information retrieval due to numerous event-oriented texts and information needs. Many natural language processing and information retrieval applications could benefit from a structured event-oriented document representation. In this paper, we proposeevent graphsas a novel way of structuring event-based information from text. Nodes in event graphs represent the individual mentions of events, whereas edges represent the temporal and coreference relations between mentions. Contrary to previous natural language processing research, which has mainly focused on individual event extraction tasks, we describe a complete end-to-end system for event graph extraction from text. Our system is a three-stage pipeline that performs anchor extraction, argument extraction, and relation extraction (temporal relation extraction and event coreference resolution), each at a performance level comparable with the state of the art. We presentEvExtra, a large newspaper corpus annotated with event mentions and event graphs, on which we train and evaluate our models. To measure the overall quality of the constructed event graphs, we propose two metrics based on the tensor product between automatically and manually constructed graphs. Finally, we evaluate the overall quality of event graphs with the proposed evaluation metrics and perform a headroom analysis of the system.

A major challenge among various applications of computational linguistics, natural language processing and information retrieval is to measure semantic similarity accurately. In this research paper, various ontology-based approaches i.e. compute semantic similarity between words have been studied and listed their benefits and shortcomings on the various identified parameters. Earlier, correlation with human judgment was the single criteria for the judgment of good similarity measures. In this paper, more parameters for semantic similarity measures have been identified and a relative analysis of similarity measures is performed on the identified parameters. These identified parameters can be further utilized for formulating the new semantic similarity-measures in the latest research area of text mining, web mining and information retrieval. We have identified various parameters like features-set, applicability on various ontologies such as single ontology, cross ontology or fuzzy ontology, Ontology type, dataset applied and relationship mapping for the various measures. Through detailed analysis we have found that feature based and hybrid approaches has higher accuracy as compare to edge and content based methods and works in different type of ontologies. Recent research drawing interest to find new feature set in this area like fuzzy distance, graph generation and text snippets etc. Max Accuracy was achieved in single ontology 0.87 and 0.83 over cross ontologies. WordNet and MeSH are maximally utilized Global Ontologies.

The standard method for making the full content of audio and video material searchable is to annotate it with human-generated meta-data that describes the content in a way that the search can understand, as is done in the creation of multimedia CD-ROMs. However, for the huge amounts of data that could usefully be included in digital video and audio libraries, the cost of producing this meta-data is prohibitive. In the Informedia Digital Video Library, the production of the meta-data supporting the library interface is automated using techniques derived from artificial intelligence (AI) research. By applying speech recognition together with natural language processing, information retrieval, and image analysis, an interface has been produced that helps users locate the information they want, and navigate or browse the digital video library more effectively. Specific interface components include automatic titles, filmstrips, video skims, word location marking, and representative frames for shots. Both the user interface and the information retrieval engine within Informedia are designed for use with automatically derived meta-data, much of which depends on speech recognition for its production. Some experimental information retrieval results will be given, supporting a basic premise of the Informedia project: That speech recognition generated transcripts can make multimedia material searchable. The Informedia project emphasizes the integration of speech recognition, image processing, natural language processing, and information retrieval to compensate for deficiencies in these individual technologies. © 1998 John Wiley & Sons, Inc.

A novel unsupervised corpus-based stemming technique using lexicon and corpus statistics

Measuring the similarity between implicit semantic relations is an important task in information retrieval and natural language processing. For example, consider the situation where you know an entity-pair (e.g. Google, YouTube), between which a particular relation holds (e.g. acquisition), and you are interested in retrieving other entity-pairs for which the same relation holds (e.g. Yahoo, Inktomi). Existing keyword-based search engines cannot be directly applied in this case because in keyword-based search, the goal is to retrieve documents that are relevant to the words used in the query -- not necessarily to the relations implied by a pair of words. Accurate measurement of relational similarity is an important step in numerous natural language processing tasks such as identification of word analogies, and classification of noun-modifier pairs. We propose a method that uses Web search engines to efficiently compute the relational similarity between two pairs of words. Our method consists of three components: representing the various semantic relations that exist between a pair of words using automatically extracted lexical patterns, clustering the extracted lexical patterns to identify the different semantic relations implied by them, and measuring the similarity between different semantic relations using an inter-cluster correlation matrix. We propose a pattern extraction algorithm to extract a large number of lexical patterns that express numerous semantic relations. We then present an efficient clustering algorithm to cluster the extracted lexical patterns. Finally, we measure the relational similarity between word-pairs using inter-cluster correlation. We evaluate the proposed method in a relation classification task. Experimental results on a dataset covering multiple relation types show a statistically significant improvement over the current state-of-the-art relational similarity measures.

The advances in deep learning have led to great achievements in many Natural Language Processing (NLP) tasks. With the nature of language, i.e., sequential data, most NLP tasks can be framed into the sequence learning framework, such as text generation. As one of the most important foundations for modern NLP techniques, autoregressive generation models have achieved dominant performance in a great deal of NLP tasks. Therefore, this thesis emphasizes improving the autoregressive generation model for different NLP tasks. While many tasks can naturally fit into the sequence learning framework, some of them, e.g., building discourse parsing tree, require sophisticated designs to fit into neural models. Therefore, this thesis firstly emphasizes a novel unified framework for discourse parsing, which builds a discourse tree in a top-down depth-first manner, and it frames the task as an autoregressive generation task with the goal of each step being the prediction of the node position given a piece of text. The proposed approach is proven effective with extensive empirical experiments. In addition, I extend the above framework by proposing a hierarchical decoder, which leverages the information from parents and siblings of the nodes that are currently processed. The proposed decoder utilizes the nature of the tree structure and further improves the experiment performance on both discourse parsing and dependency parsing tasks. On the other hand, the de facto strategies, i.e., cross entropy loss and teacher forcing, for training the autoregressive generation models have been shown problematic in certain aspects. For example, cross entropy loss, which is one of the widely leveraged training objective functions, often leads to text degeneration in text generation, and teacher forcing suffers from the exposure bias problem, where there exists a mismatch between the training and testing setup. For text degeneration, I introduce a class of diminishing attentions, which enforces the submodularity of the coverage calculated by cross attention in the sequence-to-sequence model. The proposed diminishing attentions achieve notable improvement on several neural text generation tasks, including text summarization, machine translation, and image paragraph generation. Further, I propose a novel training objective, ScaleGrad, to replace cross entropy, which significantly reduces the degeneration problem in different text generation tasks. In fact, ScaleGrad can be extended to problems beyond text degeneration. It provides wide flexibility to inject different inductive biases into the text generation model by directly modifying the gradient information in the output layer. Next, for the exposure bias problem, this thesis introduces a novel type of scheduled sampling based on training accuracy, which requires only minimal hyper-parameter tuning compared to existing scheduled sampling methods. Additionally, a novel imitation loss is proposed to further enforce the model’s generative behavior to match the teacher-forced behavior. Moreover, this thesis demonstrates that reducing exposure bias can improve the robustness of language models against repetition and toxic errors.

IBM Research has over 200 people working on Unstructured Information Management (UIM) technologies with a strong focus on Natural Language Processing (NLP). These researchers are engaged in activities ranging from natural language dialog, information retrieval, topic-tracking, named-entity detection, document classification and machine translation to bioinformatics and open-domain question answering. An analysis of these activities strongly suggested that improving the organization's ability to quickly discover each other's results and rapidly combine different technologies and approaches would accelerate scientific advance. Furthermore, the ability to reuse and combine results through a common architecture and a robust software framework would accelerate the transfer of research results in NLP into IBM's product platforms. Market analyses indicating a growing need to process unstructured information, specifically multilingual, natural language text, coupled with IBM Research's investment in NLP, led to the development of middleware architecture for processing unstructured information dubbed UIMA. At the heart of UIMA are powerful search capabilities and a data-driven framework for the development, composition and distributed deployment of analysis engines. In this paper we give a general introduction to UIMA focusing on the design points of its analysis engine architecture and we discuss how UIMA is helping to accelerate research and technology transfer.

Text clustering is one of the fundamental tasks in natural language processing which involves grouping similar documents together based on their content thus enabling efficient organization and analysis of textual data. In this paper we provide a comprehensive survey of text clustering techniques, its applications, challenges, and future directions. We begin by discussing the fundamentals of text clustering, including key concepts such as similarity measures, text feature representations, and clustering algorithms. We also explore popular text clustering algorithms such as K-means, hierarchical clustering, density- based clustering, spectral clustering, affinity propagation, and Latent Dirichlet Allocation (LDA) popularly used for topic modelling. For every algorithm we discuss its methodology, strengths, limitations, and parameter tuning considerations. We also dive deep into the real-world applications of text clustering across diverse domains, including document organization, information retrieval, text summarization, sentiment analysis, and recommendation systems and highlight their effectiveness with case studies and examples. We also identify several challenges and open research questions in text clustering, such as scalability, handling high-dimensional data, incorporating domain specific knowledge in clustering, evaluation metrics, and integration with other NLP tasks such NER, classification, etc. Finally, we propose potential future directions for research to address these challenges in order to advance the field of text clustering. In conclusion, text clustering continues to be an interesting area of research with immense potential for applications in various domains which helps drive innovation in natural language processing.

Recently, pretrained models have achieved remarkable performance not only in natural language processing but also in information retrieval (IR). Previous studies show that IR-oriented pretraining tasks can achieve better performance than only finetuning pretrained language models in IR datasets. Besides, the massive search log data obtained from mainstream search engines can be used in IR pretraining, for it contains users' implicit judgments of document relevance under a concrete query. However, existing methods mainly use direct query-document click signals to pretrain models. The potential supervision signals from search logs are far from being well explored. In this paper, we propose to comprehensively leverage four query-document relevance relations, including co-interaction and multi-hop relations, to pretrain ranking models in IR. Specifically, we focus on the user's click behavior and construct an Interaction Graph to represent the global relevance relations between queries and documents from all search logs. With the graph, we can consider the co-interaction and multi-hop q-d relationships through their neighbor nodes. Based on the relations extracted from the interaction graph, we propose four strategies to generate contrastive positive and negative q-d pairs and use these data to pretrain ranking models. Experimental results on both industrial and academic datasets demonstrate the effectiveness of our method.