Multi-Dimensional Evaluation of Information Retrieval Results

The evaluation of Information Retrieval (IR) systems typically uses query-document pairs with corresponding human-labelled relevance assessments (qrels). These qrels are used to determine if one system is better than another based on average retrieval performance. Acquiring large volumes of human relevance assessments is expensive. Therefore, more efficient relevance assessment approaches have been proposed, necessitating comparisons between qrels to ascertain their efficacy. Discriminative power, i.e. the ability to correctly identify significant differences between systems, is important for drawing accurate conclusions on the robustness of qrels. Previous work has measured the proportion of pairs of systems that are identified as significantly different and has quantified Type I statistical errors. Type I errors lead to incorrect conclusions due to false positive significance tests. We argue that also identifying Type II errors (false negatives) is important as they lead science in the wrong direction. We quantify Type II errors and propose that balanced classification metrics, such as balanced accuracy, can be used to portray the discriminative power of qrels. We perform experiments using qrels generated using alternative relevance assessment methods to investigate measuring hypothesis testing errors in IR evaluation. We find that additional insights into the discriminative power of qrels can be gained by quantifying Type II errors, and that balanced classification metrics can be used to give an overall summary of discriminative power in one, easily comparable, number.

The study of cross-lingual Information Retrieval Systems (IRSs) and a deep analysis of system performances should provide guidelines, hints, and directions to drive the design and development of the next generation MultiLingual Information Access (MLIA) systems. In addition, effective tools for interpreting and comparing the experimental results should be made easily available to the research community. To this end, we propose a twofold methodology for the evaluation of Cross Language Information Retrieval (CLIR) systems: statistical analyses to provide MLIA researchers with quantitative and more sophisticated analysis techniques; and graphical tools to allow for a more qualitative comparison and an easier presentation of the results. We provide concrete examples about how the proposed methodology can be applied by studying the monolingual and bilingual tasks of the Cross-Language Evaluation Forum (CLEF) 2005 and 2006 campaigns.KeywordsInformation RetrievalMean Average PrecisionGraphical ToolBilingual ComparisonBilingual PerformanceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

PurposeThe purpose of this paper is to demonstrate how to apply traditional information retrieval (IR) evaluation methods based on standards from the Text REtrieval Conference and web search evaluation to all types of modern library information systems (LISs) including online public access catalogues, discovery systems, and digital libraries that provide web search features to gather information from heterogeneous sources.Design/methodology/approachThe authors apply conventional procedures from IR evaluation to the LIS context considering the specific characteristics of modern library materials.FindingsThe authors introduce a framework consisting of five parts: search queries, search results, assessors, testing, and data analysis. The authors show how to deal with comparability problems resulting from diverse document types, e.g., electronic articles vs printed monographs and what issues need to be considered for retrieval tests in the library context.Practical implicationsThe framework can be used as a guideline for conducting retrieval effectiveness studies in the library context.Originality/valueAlthough a considerable amount of research has been done on IR evaluation, and standards for conducting retrieval effectiveness studies do exist, to the authors’ knowledge this is the first attempt to provide a systematic framework for evaluating the retrieval effectiveness of twenty-first-century LISs. The authors demonstrate which issues must be considered and what decisions must be made by researchers prior to a retrieval test.

The development and evaluation of Information Retrieval and Recommender Systems has traditionally focused on the relevance and accuracy of retrieved documents and recommendations, respectively. However, there is an increasing realization that accuracy alone might be a sub-optimal strategy for a successful user experience. Properties such as novelty and diversity have been explored in both fields for assessing and enhancing the usefulness of search results and recommendations. In this doctoral research we study the assessment and enhancement of both properties in the confluence of Information Retrieval and Recommender Systems.

F. Wilfrid Lancaster has earned a reputation for greatness in the evaluation of information storage and retrieval systems. Many of his extensive contributions stem from his early experience with the National Library of Medicine (NLM) MEDLARS system. His evaluation of the MEDLARS Demand Search Service in 1966 and 1967 was an important landmark as one of the earliest evaluations of a computer-based retrieval system and as the first application of re-call and precision measures in a large, operational database setting. In 1971, his evaluation of the MEDLARS AIM-TWX system was an important study of early online systems and their direct use by end users. This paper summarizes Lancaster’s two major evaluations of the MEDLARS system, including the information environment at the time and their impact in the field of information science. Examples of Lancaster’s other evaluation work with information retrieval systems are provided, followed by discussion of the textbooks that grew out of his evaluation experience and expertise. The article closes with comments from current and former NLM staff regarding Lancaster’s time at NLM or his influence on their own career.

This paper deals with evaluation of information retrieval from unsegmented speech. We focus on Mean Generalized Average Precision, the evaluation measure widely used for unsegmented speech retrieval. This measure is designed to allow certain tolerance in matching retrieval results (starting points of relevant segments) against a gold standard relevance assessment. It employs a Penalty Function which evaluates non-exact matches in the retrieval results based on their distance from the beginnings of their nearest true relevant segments. However, the choice of the Penalty Function is usually ad-hoc and does not necessary reflect users' perception of the speech retrieval quality. We perform a lab test to study satisfaction of users of a speech retrieval system to empirically estimate the optimal shape of the Penalty Function.

The traditional evaluation of information retrieval (IR) systems is generally very costly as it requires manual relevance annotation from human experts. Recent advancements in generative artificial intelligence -specifically large language models (LLMs)- can generate relevance annotations at an enormous scale with relatively small computational costs. Potentially, this could alleviate the costs traditionally associated with IR evaluation and make it applicable to numerous low-resource applications. However, generated relevance annotations are not immune to (systematic) errors, and as a result, directly using them for evaluation produces unreliable results.

With the rapid growth of information rate in the web, it is of paramount importance to develop scalable information retrieval approaches. These approaches are based on distributed information retrieval systems. Likewise, the emergence of contextual information retrieval has introduced the research personalisation which attempts to reduce the research ambiguity and to return most appropriate information to the user. Therefore, it seems a compelling issue, to implement techniques related to contextual peer-to-peer information retrieval (P2P-IR) to better exploit the personalisation of information in order to achieve a more efficient access to data. In this respect, evaluating these approaches in contextual P2P-IR is a crucial step. Indeed, to the best of our knowledge, there is no available test-beds dedicated to the evaluation of contextual P2P-IR. In particular, we propose, two methods. The first is based on existing collections that are enriched with external semantic resources. The second offers the possibility to build test-beds from multiple sources data.

Question Answering (QA) Systems are put forward as a real alternative to Information Retrieval systems as they provide the user with a fast and comprehensible answer to his or her information need. It has been 15 years since TREC introduced the first QA track. The principal campaigns in the evaluation of Information Retrieval have been specific tracks focusing on the development and evaluation of this type of system. This study is a brief review of the TREC, CLEF and NTCIR Conferences from the QA perspective. We present a historical overview of 15 years of QA evaluation tracks using the method of systematic review. We have examined identified the different tasks or specific labs created in each QA track, the types of evaluation question used, as well as the evaluation measures used in the different competitions analyzed. Of the conferences, it is CLEF that has applied the greater variety of types of test question (factoid, definition, list, causal, yes/no, amongst others). NTCIR, held on 13 occasions, is the conference which has made use of a greater number of different evaluation measures. Accuracy, precision and recall have been the three most used evaluation measures in the three campaigns.

Humans have used electronic information retrieval (IR) systems for more than 50 years as they evolved from experimental systems to full-scale Web search engines and digital libraries. The fields of library and information science (LIS), cognitive science, human factors and computer science have historically been the leading disciplines in conducting research that seeks to model human interaction with IR systems for all kinds of information related behaviors. As technology problems have been mastered, the theoretical and applied framework for studying human interaction with IR systems has evolved from systems-centered to more user-centered, or cognitive-centered approaches. However, cognitive information retrieval (CIR) research that focuses on user interaction with IR systems is still largely under-funded and is often not included at computing and systems design oriented conferences. But CIR-focused research continues, and there are signs that some IR systems designers in academia and the Web search business are realizing that user behavior research can provide valuable insights into systems design and evaluation. The goal of our book is to provide an overview of new CIR research directions. This book does not provide a history of the research field of CIR. Instead, the book confronts new ways of looking at the human information condition with regard to our increasing need to interact with IR systems. The need has grown due to a number of factors, including the increased importance of information to more people in this information age. Also, IR was once considered document-oriented, but has now evolved to include multimedia, text, and other information objects. As a result, IR systems and their complexity have proliferated as users and user purposes for using them have also proliferated. Human interaction with IR systems can often be frustrating as people often lack an understanding of IR system functionality. New more holistic directions in CIR are emerging that conceptualize human-IR system interaction at the human-computer interaction level (taking into account interface issues), the information behavior level (taking into account the role of IR system interaction in the total range of people’s information behaviors), and the

Retrieval evaluation is always an important aspect in information retrieval web search and metrics are a key factor that needs to be carefully considered. In this paper, we propose a new method of measuring stability and discrimination power of a metric. The problem is initiated by Buckley and Voorhees. The advantage of the proposed method is that we are able to measure both aspects together in a systematic manner. Five metrics are tested in the study. They are average precision over all relevant documents, recall-level precision, normalized discount cumulative gain, precision at 10 documents level, and reciprocal rank. Experimental results show that normalized discount cumulative gain is the best, which is followed by average precision over all relevant documents, recall-level precision, precision at 10 documents level, while reciprocal rank is the worst.

Most books on information-seeking neglect the human side to focus solely on the technical aspects of retrieval systems. This book is the first to provide an information science perspective on information retrieval, with a focus on user behavior and needs. Beginning with the history of information retrieval, coverage goes on to consider behavioral models and interactive techniques as well as the design, implementation and evaluation of information retrieval interfaces. This comprehensive book will be the go-to for LIS students and professionals hoping to better appreciate and utilize information retrieval systems, theories, and practices. Contents Interactive information retrieval : history and background / Colleen Cool and Nicholas J. Belkin Information behavior and seeking / Peiling Wang Task-based information searching and retrieval / Elaine G. Toms Approaches to investigating information interaction and behaviour / Raya Fidel Information representation / Mark D. Smucker Access models / Edie Rasmussen Evaluation / Kalervo Jarvelin Interfaces for information retrieval / Max Wilson Interactive techniques / Ryen W. White Web retrieval, ranking and personalization / Jaime Teevan and Susan Dumais Recommendation, collaboration and social search / David M. Nichols and Michael B. Twidale Multimedia : behaviour, interfaces and interaction / Haiming Liu, Suzanne Little and Stefan Ruger Multimedia : information representation and access / Suzanne Little, Evan Brown and Stefan Ruger

Information Retrieval (IR) systems aim to retrieve data that satisfies certain requirements and constitute an important service in many types of networks, including Delay/Disruption Tolerant Networks (DTNs). In current DTN based IR systems, the data that satisfies a query is assumed to be stored on a single node. Therefore, once a node receives a query in which it has the corresponding data, the query can be resolved completely. However, in scenarios where a query requires data from multiple nodes, these IR systems may fail. Henceforth, in this paper, we propose Distributed Data-Centric Information Retrieval (DDC-IR), a data centric IR system that supports all query types; e.g., continuous and complex. More importantly, it is designed specifically to operate in DTNs. It also incorporates a new packet, aka Query Reply Packet, that includes both a query and one or more replies. We show how this packet facilitates efficient query resolution and enables data centric routing. In addition, it uses caching so that nodes store popular queries that has the effect of speeding up query resolution. We have conducted an extensive simulation study to compare DDC-IR to state of the art IR systems using the popular Random Waypoint model and a trace-file containing student movements on a campus. The results show that DDC-IR is able to resolve 50 % more queries and has an 80 % lower buffer occupancy level than existing IR systems. We also tested DDC-IR in networks with varying sizes. For networks with 100 nodes, DDC-IR is able to resolve queries while current IR systems fail to resolve any queries. In particular, when the number of nodes increases, current IR systems fail to resolve any queries, whilst DDC-IR is able to resolve complex and continuous queries. The influence of the number of sub-queries on query resolution time is also studied. Specifically, when the number of sub-queries in a complex query increases from five to nine, DDC-IR uses 50 % more time to resolve a query. In comparison, prior IR systems fail to resolve any queries.

This report documents the program and the outcome of Dagstuhl Seminar 13441 Evaluation Methodologies in Information Retrieval, which brought together 42 participants from 11 countries. The seminar was motivated by the fact that today's information retrieval (IR) applications can hardly be evaluated based on the classic test collection paradigm, thus there is a need for new evaluation approaches. The event started with five introductory talks on evaluation frameworks, user modeling for evaluation, evaluation criteria, measures, evaluation methodology, and new trends in IR evaluation. The seminar participants then formed working groups addressing specific aspects of IR evaluation, such as reliability and validity, task-based IR, learning as search outcome, searching for fun, IR and social media, graph search, domain-specific IR, interaction measures and models, and searcher-aware information access systems.

Purpose – People often engage in different information-seeking strategies (ISSs) within a single information-seeking episode. A critical concern for the design of information retrieval (IR) systems is how to provide support for these different behaviors in a manner which searchers can easily understand, navigate and use, as they move from one ISS to another. The purpose of this paper is to describe a dialogue structure that was implemented in an experimental IR system, in order to address this concern. Design/methodology/approach – The authors conducted a user-centered experiment to evaluate the IR systems. Participants were asked to search for information on two different task types, with four different topics per task, in both the experimental system and a baseline system emulating state-of-the-art IR systems. The authors report here the results related explicitly to the use of the experimental system's dialogue structure. Findings – For one of the task types, most participants followed the search steps as predicted in the dialogue structures, and those who did so completed the task in fewer moves. For the other task type, predicted order of moves was often not followed, but participants again used fewer moves when following the predicted order. Results demonstrate that the dialogue structures the authors designed indeed support effective human information behavior patterns in a variety of ways, and that searchers can effectively use a system which changes to support different ISSs. Originality/value – This study shows that it is both possible and beneficial, to design an IR system which can support multiple ISSs, and that such a system can be understood and used successfully.