A framework for designing retrieval effectiveness studies of library information systems using human relevance assessments

The evaluation of Information Retrieval (IR) systems typically uses query-document pairs with corresponding human-labelled relevance assessments (qrels). These qrels are used to determine if one system is better than another based on average retrieval performance. Acquiring large volumes of human relevance assessments is expensive. Therefore, more efficient relevance assessment approaches have been proposed, necessitating comparisons between qrels to ascertain their efficacy. Discriminative power, i.e. the ability to correctly identify significant differences between systems, is important for drawing accurate conclusions on the robustness of qrels. Previous work has measured the proportion of pairs of systems that are identified as significantly different and has quantified Type I statistical errors. Type I errors lead to incorrect conclusions due to false positive significance tests. We argue that also identifying Type II errors (false negatives) is important as they lead science in the wrong direction. We quantify Type II errors and propose that balanced classification metrics, such as balanced accuracy, can be used to portray the discriminative power of qrels. We perform experiments using qrels generated using alternative relevance assessment methods to investigate measuring hypothesis testing errors in IR evaluation. We find that additional insights into the discriminative power of qrels can be gained by quantifying Type II errors, and that balanced classification metrics can be used to give an overall summary of discriminative power in one, easily comparable, number.

Evaluation of information retrieval (IR) results is an important and difficult activity. Available evaluation methods are commonly based on the classical recall and precision measures. While these methods provide an evaluation of average system performance, they are not able to identify influence of other factors such as retrieval task and system user. Detailed information is hidden behind the average and no indication can be obtained from the evaluation for further improvement. Therefore, there is a need of new methods for detailed evaluation. In this paper, we propose a multi-dimensional approach to the evaluation of IR results so that effects of both an IR system and the environment where the system performs on retrieval results can be examined. It aids in the identification of problems of the system and the environment with respect to retrieval effectiveness, and assists in making improvement for IR.

The development and evaluation of Information Retrieval and Recommender Systems has traditionally focused on the relevance and accuracy of retrieved documents and recommendations, respectively. However, there is an increasing realization that accuracy alone might be a sub-optimal strategy for a successful user experience. Properties such as novelty and diversity have been explored in both fields for assessing and enhancing the usefulness of search results and recommendations. In this doctoral research we study the assessment and enhancement of both properties in the confluence of Information Retrieval and Recommender Systems.

Retrieval evaluation is always an important aspect in information retrieval web search and metrics are a key factor that needs to be carefully considered. In this paper, we propose a new method of measuring stability and discrimination power of a metric. The problem is initiated by Buckley and Voorhees. The advantage of the proposed method is that we are able to measure both aspects together in a systematic manner. Five metrics are tested in the study. They are average precision over all relevant documents, recall-level precision, normalized discount cumulative gain, precision at 10 documents level, and reciprocal rank. Experimental results show that normalized discount cumulative gain is the best, which is followed by average precision over all relevant documents, recall-level precision, precision at 10 documents level, while reciprocal rank is the worst.

Question Answering (QA) Systems are put forward as a real alternative to Information Retrieval systems as they provide the user with a fast and comprehensible answer to his or her information need. It has been 15 years since TREC introduced the first QA track. The principal campaigns in the evaluation of Information Retrieval have been specific tracks focusing on the development and evaluation of this type of system. This study is a brief review of the TREC, CLEF and NTCIR Conferences from the QA perspective. We present a historical overview of 15 years of QA evaluation tracks using the method of systematic review. We have examined identified the different tasks or specific labs created in each QA track, the types of evaluation question used, as well as the evaluation measures used in the different competitions analyzed. Of the conferences, it is CLEF that has applied the greater variety of types of test question (factoid, definition, list, causal, yes/no, amongst others). NTCIR, held on 13 occasions, is the conference which has made use of a greater number of different evaluation measures. Accuracy, precision and recall have been the three most used evaluation measures in the three campaigns.

The study of cross-lingual Information Retrieval Systems (IRSs) and a deep analysis of system performances should provide guidelines, hints, and directions to drive the design and development of the next generation MultiLingual Information Access (MLIA) systems. In addition, effective tools for interpreting and comparing the experimental results should be made easily available to the research community. To this end, we propose a twofold methodology for the evaluation of Cross Language Information Retrieval (CLIR) systems: statistical analyses to provide MLIA researchers with quantitative and more sophisticated analysis techniques; and graphical tools to allow for a more qualitative comparison and an easier presentation of the results. We provide concrete examples about how the proposed methodology can be applied by studying the monolingual and bilingual tasks of the Cross-Language Evaluation Forum (CLEF) 2005 and 2006 campaigns.KeywordsInformation RetrievalMean Average PrecisionGraphical ToolBilingual ComparisonBilingual PerformanceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

To what extent do MeSH (Medical Subject Headings) terms improve search effectiveness of different kinds of users? We observed four different kinds of searchers using an experimental information retrieval (IR) system: (1) search novices; (2) domain experts; (3) search experts and (4) medical librarians. The information needs were a subset of the relatively difficult topics originally created for the Text REtrieval Conference (TREC). By experimental design, we used 20 search topics in an IR user experiment to alleviate search topic variability. Effectiveness of retrieval was based on the relevance judgments set provided by TREC. Thirty-two participants searched either using a version of the system in which abstracts and MeSH terms were displayed or another version in which they had to formulate their own terms based only on the display of abstracts. We found that MeSH terms were more useful for domain experts than for search experts in terms of the precision measure, even though domain experts did not perceive that MeSH terms were useful. We speculate that because of the technical topics, only the domain experts had the knowledge to understand and therefore make use of the MeSH terms. The primary contributions of this research are: (1) assessment of relative impact of searchers characteristics of domain knowledge and search training on search effectiveness and (2) design and methodology for assessing the usefulness of controlled vocabulary. The effort to create MeSH terms is worthwhile for domain experts' searches on technical topics.

The technology advancement enables the Internet of Things (IoT) and the technology roadmap indicates the demands for Quality of Service (QoS). The IoT is the subset for the QoS in communication channels. In a digital library, the QoS depends on library information system (LIS) and the Radio Frequency Identification (RFID) system. Both systems are vital and interrelated to deliver the information technology services to the digital library users. The LIS and RFID communicate using a standard protocol and language. This protocol differs in terms of application systems. This study is emphasized on the LIS and RFID frameworks, which eventually evaluate the QoS in a digital library. The framework is streamlined to the digital library framework.

I. Introduction Studies reveal that often group members collaborate when searching for information even if they were not explicitly asked to collaborate [1]. The activity that involves a group of people engaging in a common information seeking task is called Collaborative Information Seeking (CIS). Over the past few years, CIS research has focused on providing solutions and frameworks to support the process [2]. However, work in this field to date has always assumed that information seekers engaged in CIS activity are using the same access modality, the visual modality. The attention on this modality has failed to address the needs of users who employ different access modalities such as haptic and/or audio. Visual Impaired (VI) employees in a workplace may often have to collaborate with their sighted team members when searching the web. Given that the VI individual's search behaviour is challenged by poor web design and the shortcomings of current assistive technology [3][4]; collaboratively engaging in w...

With the rapid growth of information rate in the web, it is of paramount importance to develop scalable information retrieval approaches. These approaches are based on distributed information retrieval systems. Likewise, the emergence of contextual information retrieval has introduced the research personalisation which attempts to reduce the research ambiguity and to return most appropriate information to the user. Therefore, it seems a compelling issue, to implement techniques related to contextual peer-to-peer information retrieval (P2P-IR) to better exploit the personalisation of information in order to achieve a more efficient access to data. In this respect, evaluating these approaches in contextual P2P-IR is a crucial step. Indeed, to the best of our knowledge, there is no available test-beds dedicated to the evaluation of contextual P2P-IR. In particular, we propose, two methods. The first is based on existing collections that are enriched with external semantic resources. The second offers the possibility to build test-beds from multiple sources data.

The traditional evaluation of information retrieval (IR) systems is generally very costly as it requires manual relevance annotation from human experts. Recent advancements in generative artificial intelligence -specifically large language models (LLMs)- can generate relevance annotations at an enormous scale with relatively small computational costs. Potentially, this could alleviate the costs traditionally associated with IR evaluation and make it applicable to numerous low-resource applications. However, generated relevance annotations are not immune to (systematic) errors, and as a result, directly using them for evaluation produces unreliable results.

Humans have used electronic information retrieval (IR) systems for more than 50 years as they evolved from experimental systems to full-scale Web search engines and digital libraries. The fields of library and information science (LIS), cognitive science, human factors and computer science have historically been the leading disciplines in conducting research that seeks to model human interaction with IR systems for all kinds of information related behaviors. As technology problems have been mastered, the theoretical and applied framework for studying human interaction with IR systems has evolved from systems-centered to more user-centered, or cognitive-centered approaches. However, cognitive information retrieval (CIR) research that focuses on user interaction with IR systems is still largely under-funded and is often not included at computing and systems design oriented conferences. But CIR-focused research continues, and there are signs that some IR systems designers in academia and the Web search business are realizing that user behavior research can provide valuable insights into systems design and evaluation. The goal of our book is to provide an overview of new CIR research directions. This book does not provide a history of the research field of CIR. Instead, the book confronts new ways of looking at the human information condition with regard to our increasing need to interact with IR systems. The need has grown due to a number of factors, including the increased importance of information to more people in this information age. Also, IR was once considered document-oriented, but has now evolved to include multimedia, text, and other information objects. As a result, IR systems and their complexity have proliferated as users and user purposes for using them have also proliferated. Human interaction with IR systems can often be frustrating as people often lack an understanding of IR system functionality. New more holistic directions in CIR are emerging that conceptualize human-IR system interaction at the human-computer interaction level (taking into account interface issues), the information behavior level (taking into account the role of IR system interaction in the total range of people’s information behaviors), and the

This report documents the program and the outcome of Dagstuhl Seminar 13441 Evaluation Methodologies in Information Retrieval, which brought together 42 participants from 11 countries. The seminar was motivated by the fact that today's information retrieval (IR) applications can hardly be evaluated based on the classic test collection paradigm, thus there is a need for new evaluation approaches. The event started with five introductory talks on evaluation frameworks, user modeling for evaluation, evaluation criteria, measures, evaluation methodology, and new trends in IR evaluation. The seminar participants then formed working groups addressing specific aspects of IR evaluation, such as reliability and validity, task-based IR, learning as search outcome, searching for fun, IR and social media, graph search, domain-specific IR, interaction measures and models, and searcher-aware information access systems.

Most research oriented search queries consist of multiple topics belonging to one or more domains of knowledge. The objective of such search queries is to find the relationship or impact that each topic has on the other(s). Though web search engines provide easy means to retrieve information off the web, search engines are mainly key word oriented and do not consider the different topics and relationships between such topics present in the results. This project presents a software solution, `SearchDroid' that acts as an intermediate user between a search engine and the end user, refining search results based on different topics identified in the search query and their presence and relationships depicted in search results. A discourse parsing approach has been used to build discourse structures that represent the rhetorical relations of text in search results; this is used to re-rank results based on topics identified in the search query. The project combines linguistics research under discourse parsing with web information retrieval techniques. The lack of literature combining discourse parsing techniques with web information retrieval has been compensated for by introducing a fresh algorithmic approach. An abstract information retrieval mechanism has been created with the use of discourse parsing techniques, and can be integrated into any web information retrieval approach. The system was evaluated by Linguistic Experts, Technical Experts and End users. All experts agreed that a discourse parsing approach was suitable for addressing the problem at hand and that the project had high research value in the aspect of linking linguistics research with web information retrieval research. Hands-on testing of the system by End Users produced high user acceptance of the proposed system.

The purpose of this paper is to apply and evaluate the bibliometric method Brad-fordizing for information retrieval (IR) experiments. Bradfordizing is used for generating core document sets for subject-specific questions and to reord result sets from distributed searches. The method will be applied and tested in a controlled scenario of scien-tific literature databases from Social and Political Sciences, Economics, Psy-chology and medical sci-ence (SOLIS, SoLit, USB Köln Opac, CSA Sociologi-cal s, World Affairs Online, Psyndex and Med-line) and 164 standardized topics. An evaluation of the method and its effects is carried out in two labora-tory-based information re-trieval experiments (CLEF and KoMoHe) using a con-trolled document corpus and human relevance as-sessments. The results show that Bradfordizing is a very robust method for re-ranking the main document types (journal articles and monographs) in today’s digital libraries (DL). The IR tests show that relevance distributions after re-ranking improve at a sig-nificant level if articles in the core are compared with articles in the succeeding zones. The items in the core are significantly more often assessed as relevant, than items in zone 2 (z2) or zone 3 (z3). The improvements between the zones are statis-tically significant based on the Wilcoxon signed-rank test and the paired T-Test.