On Obtaining Effort Based Judgements for Information Retrieval

The evaluation of Information Retrieval (IR) systems typically uses query-document pairs with corresponding human-labelled relevance assessments (qrels). These qrels are used to determine if one system is better than another based on average retrieval performance. Acquiring large volumes of human relevance assessments is expensive. Therefore, more efficient relevance assessment approaches have been proposed, necessitating comparisons between qrels to ascertain their efficacy. Discriminative power, i.e. the ability to correctly identify significant differences between systems, is important for drawing accurate conclusions on the robustness of qrels. Previous work has measured the proportion of pairs of systems that are identified as significantly different and has quantified Type I statistical errors. Type I errors lead to incorrect conclusions due to false positive significance tests. We argue that also identifying Type II errors (false negatives) is important as they lead science in the wrong direction. We quantify Type II errors and propose that balanced classification metrics, such as balanced accuracy, can be used to portray the discriminative power of qrels. We perform experiments using qrels generated using alternative relevance assessment methods to investigate measuring hypothesis testing errors in IR evaluation. We find that additional insights into the discriminative power of qrels can be gained by quantifying Type II errors, and that balanced classification metrics can be used to give an overall summary of discriminative power in one, easily comparable, number.

This article presents a knowledge-based approach to the design of document-based retrieval systems. We conducted two empirical studies investigating the users' behavior using an online catalog. The studies revealed a range of knowledge elements which are necessary for performing a successful search. We proposed a semantic network based representation to capture these knowledge elements. The findings we derived from our empirical studies were used to construct a knowledge-based retrieval system. We performed a laboratory experiment to evaluate the search performance of our system. The experiment showed that our system out-performed a conventional retrieval system in recall and user satisfaction. The implications of our study to the design of document-based retrieval systems are also discussed in this article.

With the rapid evolution of smart home environment, the demand for spoken information retrieval (e.g., voice-activated FAQ retrieval) on information appliances is increasing. In spoken information retrieval, users’ spoken queries are converted into text queries using automatic speech recognition (ASR) engines. If top-1 results of the ASR engines are incorrect, the errors are propagated to information retrieval systems. If a document collection is a small set of sentences such as frequently asked questions (FAQs), the errors have additional effect on the performance of information retrieval systems. To improve the performance of such a sentence retrieval system, we propose a post-processing model of an ASR engine. The post-processing model consists of a re-ranking and a query term generation model. The re-ranking model rearranges top-n outputs of the ASR engines using the ranking support vector machine (Ranking SVM). The query term generation model extracts meaningful content words from the re-ranked queries based on term frequencies and query rankings. In the experiments, the re-ranking model improved the top-1 performance results of an underlying ASR engine with 4.4% higher precision and 6.4% higher recall rate. The query term generation model improved the performance results of an underlying information retrieval system with an accuracy 2.4% to 2.6% higher. Based on the experimental result, the proposed model revealed that it could improve the performance of a spoken sentence retrieval system in a restricted domain.

The information explosion across the Internet and elsewhere offers access to an increasing number of document collections. Retrieval system evaluation plays an important role in judging the efficiency and effectiveness of the retrieval process. Validating the performance of information retrieval systems provides assurance that a system as a whole is likely to meet its quantitative performance goals. In this paper, we describe procedure to calculate the response time early in the life cycle and simulate the performance metrics for an information retrieval system in different architectures namely, centralized and distributed. We consider different characterizations for the hardware resources' configurations and simulate the results for two different architectures of Information retrieval systems.

Purpose – People often engage in different information-seeking strategies (ISSs) within a single information-seeking episode. A critical concern for the design of information retrieval (IR) systems is how to provide support for these different behaviors in a manner which searchers can easily understand, navigate and use, as they move from one ISS to another. The purpose of this paper is to describe a dialogue structure that was implemented in an experimental IR system, in order to address this concern. Design/methodology/approach – The authors conducted a user-centered experiment to evaluate the IR systems. Participants were asked to search for information on two different task types, with four different topics per task, in both the experimental system and a baseline system emulating state-of-the-art IR systems. The authors report here the results related explicitly to the use of the experimental system's dialogue structure. Findings – For one of the task types, most participants followed the search steps as predicted in the dialogue structures, and those who did so completed the task in fewer moves. For the other task type, predicted order of moves was often not followed, but participants again used fewer moves when following the predicted order. Results demonstrate that the dialogue structures the authors designed indeed support effective human information behavior patterns in a variety of ways, and that searchers can effectively use a system which changes to support different ISSs. Originality/value – This study shows that it is both possible and beneficial, to design an IR system which can support multiple ISSs, and that such a system can be understood and used successfully.

The study of cross-lingual Information Retrieval Systems (IRSs) and a deep analysis of system performances should provide guidelines, hints, and directions to drive the design and development of the next generation MultiLingual Information Access (MLIA) systems. In addition, effective tools for interpreting and comparing the experimental results should be made easily available to the research community. To this end, we propose a twofold methodology for the evaluation of Cross Language Information Retrieval (CLIR) systems: statistical analyses to provide MLIA researchers with quantitative and more sophisticated analysis techniques; and graphical tools to allow for a more qualitative comparison and an easier presentation of the results. We provide concrete examples about how the proposed methodology can be applied by studying the monolingual and bilingual tasks of the Cross-Language Evaluation Forum (CLEF) 2005 and 2006 campaigns.KeywordsInformation RetrievalMean Average PrecisionGraphical ToolBilingual ComparisonBilingual PerformanceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This article examines the structure and components of information storage and retrieval systems and information filtering systems. Analysis of the tasks performed in such selection systems leads to the identification of 13 components. Eight are necessarily present in all such systems, mechanized or not; the others may, but need not be, present. The authors argue that all selection systems can be represented in terms of combinations of these components. The components are of only two types: representations of data objects and functions that operate on them. Further, the functional components, or rules, reduce to two basic types: 1) transformation, making or modifying the members of a set of representations, and 2) sorting or partitioning. The representational transformations may be in the form of copies, excerpts, descriptions, abstractions, or mere identifying references. By partitioning, we mean dividing a set of objects by using matching, sorting, ranking, selecting, and other logically equivalent operations. The typical multiplicity of knowledge sources and of system vocabularies is noted. Some of the implications for the study, use, and design of information storage and retrieval systems are discussed.

IntroductionIt is usually agreed that the general objectives for the creation of library catalogs as stated by Cutter in 1876 (1889) and Lubetzky in 1961 (1963, 139, 1st objective) are still valid and relevant, despite the fact that these bibliographic tools are constantly evolving and transforming. The Functional Requirements for Bibliographic Records model (FRBR) requires that bibliographic data allow end-users to find, identify, select and obtain documents that correspond to expressed search criteria (IFLA, 1998, 8-9). A library catalog can be described as a type of bibliographic file that has the distinctive feature to gather descriptive records for a given library collection, be it physical or virtual. Thus the principal function of a library catalog is to provide bibliographic control for a collection of documents, i.e., to perform the operations by which the information is organized and structured according to one or several established standards in order to render this information easily identifiable and available. The two fundamental objectives of catalogs, identification and retrieval, are usually achieved by creating standardized descriptions by analyzing the content in order to describe the intellectual content, by classifying the items in order to group them in meaningful categories, by adding a variety of controlled access keys, and indexing, in total or in part, the textual content of these descriptions. The fulfillment of these objectives is undoubtedly guided by the creation and the maintenance of bibliographic description and encoding standards (AACR, LCSH, MARC), but is also influenced by the development of new features of online information retrieval systems. As a result, even if the objectives for the creation of library catalogs remain more or less stable over the years, the means available to fulfill these objectives evolve rapidly. Recently, the improvement of text information retrieval techniques and the development of hypertextual interfaces have facilitated the development of more and more sophisticated retrieval features in library catalogs.Coordinating the development and the application of bibliographic dards, on one hand, and the development of text retrieval systems, on the other, is not a small task. The encounter of these two worlds is fairly recent and greatly suffers from a lack of communication and understanding. The rapid developments in the world of text information retrieval bring new opportunities to information systems designers and allow greater freedom to system managers for the local implementation of these systems. Adding new retrieval features often improves the search capabilities in online catalogs. It also seems that the implementation of these new retrieval functions can sometimes have a negative impact on the quality of retrieval, because these features are not always specifically designed for the type of data contained in library catalogs. Unfortunately, these retrieval features are not standardized and may vary considerably from one system to another. It is not possible to adapt bibliographic standards according to every system's design. Nevertheless, there has been an attempt in past years to develop minimal display standards for bibliographic records in catalogs to provide a standard to guide system designers and managers in their work (IFLA, 2003). The perfect system does not exist and it is always necessary to adapt bibliographic standards locally in order to comply with the constraints and restrictions imposed by the system. The parameters selected and applied during the local implementation of these systems will also have an impact on the manner that data are displayed and used, and this in turn may also require internal modifications to the local metadata creation procedures. We are in an evolutionary environment that focuses increasingly on data exchange and collaboration between organizations, where data and system system interoperability is crucial. …

The maximum entropy principle may be applied to the design of probabilistic retrieval systems. When there are inconsistent expert judgments, the resulting optimization problem cannot be solved. The inconsistency of the expert judgments can be revealed by solving a linear programming formulation. In the case of inconsistent judgment, four plausible schemes are proposed in order to find revised judgments which are consistent with the true data structure but still reflect the original expert judgment. These schemes are the Interactive, Minimum Distance, Minimum Cross-Entropy, and Path methods. © 1991 John Wiley & Sons, Inc.

Most books on information-seeking neglect the human side to focus solely on the technical aspects of retrieval systems. This book is the first to provide an information science perspective on information retrieval, with a focus on user behavior and needs. Beginning with the history of information retrieval, coverage goes on to consider behavioral models and interactive techniques as well as the design, implementation and evaluation of information retrieval interfaces. This comprehensive book will be the go-to for LIS students and professionals hoping to better appreciate and utilize information retrieval systems, theories, and practices. Contents Interactive information retrieval : history and background / Colleen Cool and Nicholas J. Belkin Information behavior and seeking / Peiling Wang Task-based information searching and retrieval / Elaine G. Toms Approaches to investigating information interaction and behaviour / Raya Fidel Information representation / Mark D. Smucker Access models / Edie Rasmussen Evaluation / Kalervo Jarvelin Interfaces for information retrieval / Max Wilson Interactive techniques / Ryen W. White Web retrieval, ranking and personalization / Jaime Teevan and Susan Dumais Recommendation, collaboration and social search / David M. Nichols and Michael B. Twidale Multimedia : behaviour, interfaces and interaction / Haiming Liu, Suzanne Little and Stefan Ruger Multimedia : information representation and access / Suzanne Little, Evan Brown and Stefan Ruger

Despite the centrality of topical relevance in information retrieval system design and evaluation, understanding and implementation of it is usually limited to “direct overall topical matching” between the subject of the query and the subject of the document. The underlying assumption is that only a single type of topical relationship is involved. In related work, a relevance judgment instrument was developed for the Multilingual Access to Large Spoken ArCHives project (MALACH). It incorporates the five topical relevance types (direct relevance, indirect/circumstantial relevance, context relevance, comparison relevance, and pointer relevance) and was applied by four judges to items in the MALACH test collection in Summer 2003. This paper reports on the experiences and perceptions of the judges making more nuanced judgments about topical relevance. The results suggest that more than only one variable/dimension, “whether it is on topic” as usually referred to, contributes to topical relevance, and more than a single topical relationship type, “direct matching” as generally assumed, play an important role in topical relevance.

Searching information retrieval systems is a highly interactive, iterative process that cannot be understood simply by comparing the output of a search session (the “search product”) to a query stated in advance. In this article, we examine evaluation goals and methods for studying information retrieval behavior, drawing examples from our own research and that of others. We limit our review to research that employs online monitoring, also known as transaction log analysis. Online monitoring is one of few methods that can capture detailed data on the search process at a reasonable cost; these data can be used to build quantitative models or to support qualitative interpretations of quantitative results. Monitoring is a data collection technique rather than a research design, and can be employed in experimental or field studies, whether alone or combined with other data collection methods. Based on the research questions of interest, the researcher must determine what variables to collect from each data source, which to treat as independent variables to manipulate, and which to treat as dependent variables to observe effects. Studies of searching behavior often treat search task and searcher characteristics as independent variables and may manipulate other independent variables specific to the research questions addressed. Search outcomes, time, and search paths frequently are treated as dependent variables. We discuss each of these sets of variables, illustrating them with sample results from the literature and from our own research. Our examples are drawn from the Science Library Catalog project, a 7-year study of children's searching behavior on an experimental retrieval system. We close with a brief discussion of the implications of these results for the design of information retrieval systems. © 1996 John Wiley & Sons, Inc.

The difficulty of a user query can affect the performance of Information Retrieval (IR) systems. What makes a query difficult and how one may predict this is an active research area, focusing mainly on factors relating to the retrieval algorithm, to the properties of the retrieval data, or to statistical and linguistic features of the queries that may render them difficult. This work addresses query difficulty from a different angle, namely the users' own perspectives on query difficulty. Two research questions are asked: (1) Are users aware that the query they submit to an IR system may be difficult for the system to address? (2) Are users aware of specific features in their query (e.g., domain-specificity, vagueness) that may render their query difficult for an IR system to address? A study of 420 queries from a Web search engine query log that are pre-categorised as easy, medium, hard by TREC based on system performance, reveals an interesting finding: users do not seem to reliably assess which query might be difficult; however, their assessments of which query features might render queries difficult are notably more accurate. Following this, a formal approach is presented for synthesising the user-assessed causes of query difficulty through opinion fusion into an overall assessment of query difficulty. The resulting assessments of query difficulty are found to agree notably more to the TREC categories than the direct user assessments.

Multimodal image retrieval, which involves retrieving images using various modalities such as text, audio, or other images, has significant research importance due to its wide-ranging applications and potential to enhance user experiences across multiple domains. Traditional image retrieval systems, content-based image retrieval (CBIR) systems rely solely on visual features, which can be limiting. By integrating multiple modalities, multimodal retrieval systems can offer more robust and accurate results. A research problem is considered multimodal when it integrates information from more than one type of data source. In multimodal image retrieval (MMIR) systems, one form of data is used to search for outcomes in the same or different modalities. In contrast, cross-modal systems strictly retrieve information from a different modality. A significant challenge in these systems is the effective comparison of input-output queries from different data types, due to their basic forms and the subjective nature of content similarity. Researchers have proposed various techniques to address this challenge and to bridge the semantic gap in information retrieval across different modalities. This article presents comprehensive analysis of various research works pertained in this multimodal image retrieval. The results and comparative analysis of current research works on benchmark datasets have also been discussed. At the end of the paper, some of open issues are presented for future research directions.

Humans have used electronic information retrieval (IR) systems for more than 50 years as they evolved from experimental systems to full-scale Web search engines and digital libraries. The fields of library and information science (LIS), cognitive science, human factors and computer science have historically been the leading disciplines in conducting research that seeks to model human interaction with IR systems for all kinds of information related behaviors. As technology problems have been mastered, the theoretical and applied framework for studying human interaction with IR systems has evolved from systems-centered to more user-centered, or cognitive-centered approaches. However, cognitive information retrieval (CIR) research that focuses on user interaction with IR systems is still largely under-funded and is often not included at computing and systems design oriented conferences. But CIR-focused research continues, and there are signs that some IR systems designers in academia and the Web search business are realizing that user behavior research can provide valuable insights into systems design and evaluation. The goal of our book is to provide an overview of new CIR research directions. This book does not provide a history of the research field of CIR. Instead, the book confronts new ways of looking at the human information condition with regard to our increasing need to interact with IR systems. The need has grown due to a number of factors, including the increased importance of information to more people in this information age. Also, IR was once considered document-oriented, but has now evolved to include multimedia, text, and other information objects. As a result, IR systems and their complexity have proliferated as users and user purposes for using them have also proliferated. Human interaction with IR systems can often be frustrating as people often lack an understanding of IR system functionality. New more holistic directions in CIR are emerging that conceptualize human-IR system interaction at the human-computer interaction level (taking into account interface issues), the information behavior level (taking into account the role of IR system interaction in the total range of people’s information behaviors), and the