Evaluating Search System Explainability with Psychometrics and Crowdsourcing

Explainable Information Retrieval (XIR) is a growing research area focused on enhancing transparency and trustworthiness of the complex decision-making processes taking place in modern information retrieval systems. While there has been progress in developing XIR systems, empirical evaluation tools to assess the degree of explainability attained by such systems are lacking. To close this gap and gain insights into the true merit of XIR systems, we extend existing insights from a factor analysis of search explainability to introduce SSE (Search System Explainability), an evaluation metric for XIR search systems. Through a crowdsourced user study, we demonstrate SSE's ability to distinguish between explainable and non-explainable systems, showing that systems with higher scores indeed indicate greater interpretability. Additionally, we observe comparable perceived temporal demand and performance levels between non-native and native English speakers. We hope that aside from these concrete contributions to XIR, this line of work will serve as a blueprint for similar explainability evaluation efforts in other domains of machine learning and natural language processing.

This paper presents an iterative and interactive information retrieval (IR) system for Web search using formal concept analysis (FCA). FCA provides a natural way to organise objects according to their properties and it has been used in recent work to organise search engine results. The navigation over the lattice helps the user to explore a structured and synthetic result. Such a lattice contains concepts that are relevant and others that are not relevant regarding a given IR task. In this way, lattices are introduced in an interactive and iterative system. The user expresses his negative or positive agreement with some concept of the lattice in respect of his objective of IR. These user choices are converted into operations over the lattice. The lattice is dynamically updated for a better fit to the request.

Automatic keywords extraction and classification tasks are important research directions in the domains of NLP (natural language processing), information retrieval, and text mining. As the fine granularity abstracted from text data, keywords are also the most important feature of text data, which has great practical and potential value in document classification, topic modeling, information retrieval, and other aspects. The compact representation of documents can be achieved through keywords, which contains massive significant information. Therefore, it may be quite advantageous to realize text classification with high-dimensional feature space. For this reason, this study designed a supervised keyword classification method based on TextRank keyword automatic extraction technology and optimize the model with the genetic algorithm to contribute to modeling the keywords of the topic for text classification.

<span>In the biomedical field due to tremendous medical research, every year the medical papers generated is very huge. In order to process these tremendous amounts of textual data, the information retrieval techniques are combined with machine learning methods like natural language processing (NLP) for finding the useful insights in data for NLP tasks. In this work, we have addressed the challenges involved in existing NLP techniques like Word2Vec, Doc2Vec, and Biosent2Vec and followed by the traditional classification of articles with Concept2Vec and random forest classification. Concept vector techniques are emerging methods in NLP and in information retrieval systems. Pretrained concept encoders are available in Machine learning, whereas no such exists in biomedical domain. In this paper a proposal on Concept2Vec trained from biomedical text is highlighted. Our work accurately emphasizes on the quality of semantics defined between the medical terms, which can help the medical practitioners in taking errorless decisions. In this work 20 thousand biomedical documents are considered for Concept2Vec embeddings and the proposed approach have shown best concept similarity in terms of relevant measures compared with existing approaches such as Word2Vec, global vectors (GloVe), fastText, and BioSentVec, bidirectional encoder representations from transformers (BERT), biomedical-BERT.</span>

Machine Learning (ML) is an Artificial Intelligence (AI) approach that allows systems to adapt to their environment based on past experiences. Machine Learning (ML) and Natural Language Processing (NLP) techniques are commonly used in sentiment analysis and Information Retrieval Techniques (IRT). This study supports the use of ML approaches, such as K-Means, to produce accurate outcomes in clustering and classification approaches. The main objective of this research is to explore the methods for sentiment classification and Information Retrieval Techniques (IRT). So, a combination of different machine learning algorithms is used with a dataset from amazon unlocked mobile reviews and telecom tweets to achieve better accuracy as it is crucial to consider the previous predictions related to sentiment classification and IRT. The datasets consist of user reviews ratings and algorithms utilized consist of K-Means Clustering algorithm, Logistic Regression (LR), Random Forest (RF), and Decision Tree (DT) algorithms. The amalgamation of each algorithm with the K-Means resulted in high levels of accuracy. Specifically, the K-Means combined with Logistic Regression (LR) yielded an accuracy rate of 99.98%. Similarly, the K-Means integrated with Random Forest (RF) resulted in an accuracy of 99.906%. Lastly, when the K-Means was merged with the Decision Tree (DT) Algorithm, the accuracy obtained was 99.83%.We exhibited that we could foresee efficient, effective, and accurate outcomes.

The rapid development in the field of information science and the increase in the usage of information retrieval strategies have empowered users to retrieve more accurate information. The availability of information in a different format and across different has presented colossal difficulties for information retrieval using information retrieval techniques. In this paper, an attempt is made at highlighting the question and answering (QA) systems that provide users the platform to express their question using the Natural Language and also retrieve the response from such systems in Natural Language. The four important modules of any QA system are the Natural Language Question (NLQ) processing, document processing, passage processing, and answer processing. Fundamentally, most QA systems combined several techniques from other fields such as information retrieval, knowledge representation, and natural language processing to process NLQ and present the most insightful response based on the stored document. This paper gives a thorough review of the various survey on the QA systems frameworks, their methodologies, types, approaches, and challenges of QA systems.

Automatic text summarization come under the domains of natural language processing, machine learning and information retrieval. As the abundance of textual information grows so does the need for summarising it. Here we consider a method for single document summarization using natural language processing tools and an extractive approach based on sentence similarity and document context. The technique uses a weighted undirected graph based scoring on paragraphs and a word frequency based scoring system on the entire document to obtain summaries. The proposed method is validated through experiments and the results are promising.

Due to the increasing daily of users documentation in the global communications network, the management, and control of its information in it have been challenged. How to extract useful knowledge in an Internet heterogeneous environment is essential. Information retrieval techniques can retrieve user information needs in a large amount of data. Search engines are the first selection of users to find information. In this search, Web crawler plays a key role in search engines. A web crawler is a script that routinely scans the web. In this paper, a information retrieval method will be presented using a combination of vector space modeling and language statistical model to improve the retrieve of related documents. The proposed approach is based on the structural similarity of the document corpus. Measuring the similarity of the input term is based on the shortest path between each term and the next term. In this method, the number of nodes, edges and links is calculated. Finally, the similarity using the path feature and cosine similarity is measured. Available documents the proposed method by the crawler of the web is compiled from various Wikipedia pages. The results of the conceptual retrieved of documents show that 84% accuracy, 88% recall and average precision of 56% have improved compared to other methods.

The Web has become a large repository of documents (or pages) written in many different languages. In this context, traditional information retrieval (IR) techniques cannot be used whenever the user query and the documents being retrieved are in different languages. To address this problem, new cross‐language information retrieval (CLIR) techniques have been proposed. In this work, we describe a method for cross‐language retrieval of medical information. This method combines query terms and related medical concepts obtained automatically through a categorization procedure. The medical concepts are used to create a linguistic abstraction that allows retrieval of information in a language‐independent way, minimizing linguistic problems such as polysemy. To evaluate our method, we carried out experiments using the OHSUMED test collection, whose documents are written in English, with queries expressed in Portuguese, Spanish, and French. The results indicate that our cross‐language retrieval method is as effective as a standard vector space model algorithm operating on queries and documents in the same language. Further, our results are better than previous results in the literature.

ABSTRACTStemming is the basic operation in Natural language processing (NLP) to remove derivational and inflectional affixes without performing a morphological analysis. This practice is essential to extract the root or stem. In NLP domains, the stemmer is used to improve the process of information retrieval (IR), text classifications (TC), text mining (TM) and related applications. In particular, Urdu stemmers utilize only uni-gram words from the input text by ignoring bigrams, trigrams, and n-gram words. To improve the process and efficiency of stemming, bigrams and trigram words must be included. Despite this fact, there are a few developed methods for Urdu stemmers in the past studies. Therefore, in this paper, we proposed an improved Urdu stemmer, using hybrid approach divided into multi-step operation, to deal with unigram, bigram, and trigram features as well. To evaluate the proposed Urdu stemming method, we have used two corpora; word corpus and text corpus. Moreover, two different evaluation metrics have been applied to measure the performance of the proposed algorithm. The proposed algorithm achieved an accuracy of 92.97% and compression rate of 55%. These experimental results indicate that the proposed system can be used to increase the effectiveness and efficiency of the Urdu stemmer for better information retrieval and text mining applications.

Conversational agents are being used to help in the screening, assessment, diagnosis, and treatment of common mental health disorders. In this paper, we propose a bootstrapping approach for the development of a digital mental health conversational agent (i.e., chatbot). We start from a basic rule-based expert system and iteratively move towards a more sophisticated platform composed of specialized chatbots each aiming to assess and pre-diagnose a specific mental health disorder using machine learning and natural language processing techniques. During each iteration, user feedback from psychiatrists and patients are incorporated into the iterative design process. A survival of the fittest approach is also used to assess the continuation or removal of a specialized mental health chatbot in each generational design. We anticipate that our unique and novel approach can be used for the development of conversational mental health agents with the ultimate goal of designing a smart chatbot that delivers evidence-based care and contributes to scaling up services while decreasing the pressure on mental health care providers.

The Web enables to have access to silo-ed information describing news articles, often offering a multitude of viewpoints that, once combined, can provide a broader picture of the story being reported on the news. In this paper, we propose an approach that automatically extracts representative features of a news item, namely named entities, from textual content attached to a video item (subtitles) and from a set of documents from the Web collected using entity expansion techniques. Approaches relying on entity expansion generally try to collect and process the important facts behinds a particular news item, but they are often too dependent on frequency-based functions and information retrieval techniques thus neglecting the multi-dimensional relationships that are established among the entities. We propose a concentric-based approach that enables to represent the context of a news item, by harmonizing into a single model the representative entities, which can be extracted using information retrieval and natural language processing techniques (Core), and other entities that get prominent according to different dimensions such as informativeness, semantic connectivity, or popularity (Crust). We compare our approach with a baseline by analyzing the compactness of the generated summary on an existing gold standard available on the Web. Results of the experiments show that our approach converges faster to the ideal compact news snapshot with an improvement of 36.9% over the baseline.

As the societal impact of Deep Neural Networks (DNNs) grows, the goals for advancing DNNs become more complex and diverse, ranging from improving a conventional model accuracy metric to infusing advanced human virtues such as fairness, accountability, transparency, and unbiasedness. Recently, techniques in Explainable Artificial Intelligence (XAI) have been attracting considerable attention and have tremendously helped Machine Learning (ML) engineers in understand AI models. However, at the same time, we started to witness the emerging need beyond XAI among AI communities; based on the insights learned from XAI, how can we better empower ML engineers in steering their DNNs so that the model’s reasonableness and performance can be improved as intended? This article provides a timely and extensive literature overview of the field Explanation-Guided Learning (EGL), a domain of techniques that steer the DNNs’ reasoning process by adding regularization, supervision, or intervention on model explanations. In doing so, we first provide a formal definition of EGL and its general learning paradigm. Second, an overview of the key factors for EGL evaluation, as well as summarization and categorization of existing evaluation procedures and metrics for EGL are provided. Finally, the current and potential future application areas and directions of EGL are discussed, and an extensive experimental study is presented aiming at providing comprehensive comparative studies among existing EGL models in various popular application domains, such as Computer Vision and Natural Language Processing domains. Additional resources related to event prediction are included in the article website: https://kugaoyang.github.io/EGL/

The proliferation of the World Wide Web has brought information retrieval (IR) techniques to the forefront of search technology. To the average computer user, “searching” now means using IR-based systems for finding information on the WWW or in other document collections. IR query evaluation methods and workloads differ significantly from those found in database systems. In this paper, we focus on three such differences. First, due to the inherent fuzziness of the natural language used in IR queries and documents, an additional degree of flexibility is permitted in evaluating queries. Second, IR query evaluation algorithms tend to have access patterns that cause problems for traditional buffer replacement policies. Third, IR search is often an iterative process, in which a query is repeatedly refined and resubmitted by the user. Based on these differences, we develop two complementary techniques to improve the efficiency of IR queries: 1) Buffer-aware query evaluation, which alters the query evaluation process based on the current contents of buffers; and 2) Ranking-aware buffer replacement, which incorporates knowledge of the query processing strategy into replacement decisions. In a detailed performance study we show that using either of these techniques yields significant performance benefits and that in many cases, combining them produces even further improvements.

The accelerating complexity of global regulatory frameworks, driven by rapid technological advancements, cross-border transactions, and shifting socio-economic priorities, has placed unprecedented demands on organizations to maintain continuous compliance. Traditional compliance management systems, often rule-based and manually updated, struggle to adapt to the dynamic and fragmented nature of these evolving landscapes. This paper proposes the design of intelligent compliance systems that leverage artificial intelligence (AI), machine learning (ML), and natural language processing (NLP) to automate regulatory monitoring, interpretation, and enforcement. By integrating real-time data streams from multiple jurisdictions, the proposed system employs semantic analysis to extract, classify, and map regulatory requirements to organizational policies, operational processes, and risk controls. A modular architecture is developed to ensure scalability, interoperability, and adaptability, enabling sector-specific customization and rapid incorporation of regulatory changes. The system incorporates predictive analytics to forecast regulatory trends, simulate compliance scenarios, and recommend proactive adjustments, thereby transforming compliance from a reactive obligation into a strategic advantage. Furthermore, explainable AI techniques are embedded to enhance transparency and trust, ensuring that automated decisions align with both legal mandates and ethical standards. Case studies across finance, healthcare, and energy sectors illustrate how intelligent compliance systems reduce operational risk, lower compliance costs, and improve audit readiness. The research underscores the importance of harmonizing technological innovation with robust governance frameworks to mitigate algorithmic bias, protect sensitive data, and meet jurisdiction-specific legal obligations such as GDPR, CCPA, and sectoral regulations. This work concludes that intelligent compliance systems represent a paradigm shift, enabling organizations to navigate the evolving global regulatory landscape with agility, accuracy, and strategic foresight, while fostering regulatory harmonization and operational resilience in an increasingly interconnected world. Keywords: Intelligent Compliance Systems, Artificial Intelligence, Machine Learning, Regulatory Technology, RegTech, Global Regulations, Natural Language Processing, Predictive Analytics, Explainable AI, Compliance Automation, Governance Frameworks, Operational Resilience, Risk Management, Legal Technology, Regulatory Harmonization.