Using Topic Models to Assess Document Relevance in Exploratory Search User Studies

Inferential representation of science documents

Object acquisition and selection are two important functions performed in most of the human computer interaction (HCI) systems. Various techniques are devised by the researchers to perform these operations and the selection of a combination of object acquisition and selection techniques for a particular HCI system has become a research issue, especially, when the user of these systems are differently abled persons. This paper presents a review on object acquisition and selection techniques used in HCI systems. It starts with the introduction to HCI systems, gives an overview on object acquisition & selection techniques, feedback modes, performing mouse analogous actions using eye blinks, the applications of HCI systems, and finally discusses challenges and issues related to these techniques.

Distributed information retrieval methods are growing rapidly because of the rising need to access and search distributed digital documents. However, the content-based information retrieval (CBIR) is concentrated to extract and retrieve the information from massive digital libraries, which require a huge amount of computing and storage resources. The grid computing provides the reliable infrastructure for effective and efficient retrieval on these large collections. In order to build an effective and efficient CBIR technique, varieties of architectures were developed based on grid technologies. The goal of such architecture is to solve interoperability and heterogeneous resource issues, and increase the efficiency and effectiveness of information retrieval (IR) techniques by harnessing the grid computing capabilities. This paper reviews and analyzes latest research carried out in the domain of large-scale dataset IR based on a grid. The evaluation is based on scalability, response time, scope, data type, search technique, middleware, and query type. The contribution is to illustrate the features, capabilities, and shortages of current solutions that can guide the researchers in this evolving area.

Document corpora are key components of information retrieval test collections. However, for certain tasks, such as evaluating the effectiveness of a new retrieval technique or estimating the parameters of a learning to rank model, a corpus alone is not enough. For these tasks, queries and relevance judgments associated with the corpus are also necessary. However, researchers often find themselves in scenarios where they only have access to a corpus, in which case evaluation and learning to rank become challenging. Document corpora are relatively straightforward to gather. On the other hand, obtaining queries and relevance judgments for a given corpus is costly. In production environments, it may be possible to obtain low-cost relevance information using query and click logs. However, in more constrained research environments these options are not available, and relevance judgments are usually provided by humans. To reduce the cost of this potentially expensive process, researchers have developed low-cost evaluation strategies, including minimal test collections [2] and crowdsourcing [1]. Despite the usefulness of these strategies, the resulting relevance judgments cannot easily be “ported” to a new or different corpus. To overcome these issues, we propose a new method to reduce manual annotation costs by transferring relevance judgments across corpora. Assuming that a set of queries and relevance judgments have been manually constructed for a source document corpus Ds, our goal is to automatically construct a test collection for a target document corpus Dt by projecting the existing test collection from Ds onto Dt. The goal of projecting test collections is not to produce manual quality test collections. In fact, it is assumed that projected test collections will contain noisy relevance judgments (i.e., ones which humans are unlikely to agree with). The important question, however, is whether these noisy projected judgments are useful for training ranking models in the target corpus.

Recent studies in information retrieval have shown that gender biases have found their way into representational and algorithmic aspects of computational models. In this paper, we focus specifically on gender biases in information retrieval gold standard datasets, often referred to as relevance judgements. While not explored in the past, we submit that it is important to understand and measure the extent to which gender biases may be presented in information retrieval relevance judgements primarily because relevance judgements are not only the primary source for evaluating IR techniques but are also widely used for training end-to-end neural ranking methods. As such, the presence of bias in relevance judgements would immediately find its way into how retrieval methods operate in practice. Based on a fine-tuned BERT model, we show how queries can be labelled for gender at scale based on which we label MS MARCO queries. We then show how different psychological characteristics are exhibited within documents associated with gendered queries within the relevance judgement datasets. Our observations show that stereotypical biases are prevalent in relevance judgement documents.

From highly relevant to not relevant: examining different regions of relevance

Test collection is used to evaluate the information retrieval systems in laboratory-based evaluation experimentation. In a classic setting, generating relevance judgments involves human assessors and is a costly and time consuming task. Researchers and practitioners are still being challenged in performing reliable and low-cost evaluation of retrieval systems. Crowdsourcing as a novel method of data acquisition is broadly used in many research fields. It has been proven that crowdsourcing is an inexpensive and quick solution as well as a reliable alternative for creating relevance judgments. One of the crowdsourcing applications in IR is to judge relevancy of query document pair. In order to have a successful crowdsourcing experiment, the relevance judgment tasks should be designed precisely to emphasize quality control. This paper is intended to explore different factors that have an influence on the accuracy of relevance judgments accomplished by workers and how to intensify the reliability of judgments in crowdsourcing experiment.

The Web has become a large repository of documents (or pages) written in many different languages. In this context, traditional information retrieval (IR) techniques cannot be used whenever the user query and the documents being retrieved are in different languages. To address this problem, new cross‐language information retrieval (CLIR) techniques have been proposed. In this work, we describe a method for cross‐language retrieval of medical information. This method combines query terms and related medical concepts obtained automatically through a categorization procedure. The medical concepts are used to create a linguistic abstraction that allows retrieval of information in a language‐independent way, minimizing linguistic problems such as polysemy. To evaluate our method, we carried out experiments using the OHSUMED test collection, whose documents are written in English, with queries expressed in Portuguese, Spanish, and French. The results indicate that our cross‐language retrieval method is as effective as a standard vector space model algorithm operating on queries and documents in the same language. Further, our results are better than previous results in the literature.

Relevance judgments play an essential role in the evaluation of information retrieval systems. As many different relevance judgment settings have been proposed in recent years, an evaluation metric to compare relevance judgments in different annotation settings has become a necessity. Traditional metrics, such as ĸ, Krippendorff's α and Φ have mainly focused on the inter-assessor consistency to evaluate the quality of relevance judgments. They encounter "reliable but useless" problem when employed to compare different annotation settings (e.g. binary judgment v.s. 4-grade judgment). Meanwhile, other existing popular metrics such as discriminative power (DP) are not designed to compare relevance judgments across different annotation settings, they therefore suffer from limitations, such as the requirement of result ranking lists from different systems. Therefore, how to design an evaluation metric to compare relevance judgments under different grade settings needs further investigation. In this work, we propose a novel metric named pairwise discriminative power (PDP) to evaluate the quality of relevance judgment collections. By leveraging a small amount of document-level preference tests, PDP estimates the discriminative ability of relevance judgments on separating ranking lists with various qualities. With comprehensive experiments on both synthetic and real-world datasets, we show that PDP maintains a high degree of consistency with annotation quality in various grade settings. Compared with existing metrics (e.g., Krippendorff's α, Φ, DP, etc), it provides reliable evaluation results with affordable additional annotation efforts.

Analyzing and modeling user behavior in the Information Retrieval (IR) process is important for IR research. In this paper, we propose a new research schema in IR research. In this schema, we first investigate humans' cognitive process when completing a specific IR task and build cognitive models for that task. The findings in the investigation and the proposed cognitive models are then utilized to improve machines' performance in the IR task. Through this research schema, we revisit three IR tasks. In the first study, we carefully analyze users' clicking behavior in mobile search and propose a mobile click model to extract relevance feedback from the mobile search logs. In the second study, we conduct an eye-tracking study to investigate how human assessors read a document during relevance judgment task and adopt the findings in building a novel retrieval model that can better approximate humans' relevance judgment. In the last study, we conduct another eye-tracking study to investigate humans' reading behavior when completing the reading comprehension task. We build a prediction model for user attention and leverage the predicted attention signals to improve the machine reading comprehension model. By successfully adopting this research schema to three different IR tasks, we demonstrate its effectiveness and generalizability.

Eye movement data can provide an in‐depth view of human reasoning and the decision‐making process, and modern information retrieval (IR) research can benefit from the analysis of this type of data. The aim of this research was to examine the relationship between relevance criteria use and visual behavior in the context of predictive relevance judgments. To address this objective, a multimethod research design was employed that involved observation of participants’ eye movements, talk‐aloud protocols, and postsearch interviews. Specifically, the results reported in this article came from the analysis of 281 predictive relevance judgments made by 24 participants using the Google search engine. We present a novel stepwise methodological framework for the analysis of relevance judgments and eye movements on the Web and show new patterns of relevance criteria use during predictive relevance judgment. For example, the findings showed an effect of ranking order and surrogate components (Title, Summary, and URL) on the use of relevance criteria. Also, differences were observed in the cognitive effort spent between very relevant and not relevant judgments. We conclude with the implications of this study for IR research.

Human-computer interaction for virtual-real fusion

The Web enables to have access to silo-ed information describing news articles, often offering a multitude of viewpoints that, once combined, can provide a broader picture of the story being reported on the news. In this paper, we propose an approach that automatically extracts representative features of a news item, namely named entities, from textual content attached to a video item (subtitles) and from a set of documents from the Web collected using entity expansion techniques. Approaches relying on entity expansion generally try to collect and process the important facts behinds a particular news item, but they are often too dependent on frequency-based functions and information retrieval techniques thus neglecting the multi-dimensional relationships that are established among the entities. We propose a concentric-based approach that enables to represent the context of a news item, by harmonizing into a single model the representative entities, which can be extracted using information retrieval and natural language processing techniques (Core), and other entities that get prominent according to different dimensions such as informativeness, semantic connectivity, or popularity (Crust). We compare our approach with a baseline by analyzing the compactness of the generated summary on an existing gold standard available on the Web. Results of the experiments show that our approach converges faster to the ideal compact news snapshot with an improvement of 36.9% over the baseline.

The intention of this paper was to determine usability of KOHA OPAC in information retrieval with regard to usability testing by undergraduate users at Technical University of Kenya Library (TU-KL) in Nairobi County. Usability testing can be realized by way of assigning tasks to a representative of the user population in order to determine whether users can be able to achieve the desired goal of retrieving information. To a great extent, success will depend on how users perform the tasks assigned to them. This study conducted usability test on actual users of KOHA OPAC at Technical University of Kenya Library. Qualitative data was collected using the think aloud method where study participants were required to verbalize their thought process as they performed search and retrieval tasks using KOHA OPAC. The number of errors made, time taken to complete search and retrieval activities were coded, recorded and analysed during usability testing. This helped the researcher to determine whether KOHA OPAC is easy to use and whether it saves time for the user. Undergraduate students for the study were selected using a stratified random sampling technique. The population of the study (n = 382) was determined using a simplified formula developed by Yamane. The study initiated a usability test which presented search scenarios where participants were invited to participate in searching and retrieving information resources using KOHA OPAC. The results of the usability test of KOHA OPAC at TU-KL showed that users were slow at accomplishing search tasks, they took long duration to access TU-KL website that hosts KOHA OPAC and there was inefficiency in accessing library materials. This study concluded that usability challenges may have an impact on users' overall satisfaction with KOHA OPAC and information retrieval effectiveness. In order to enhance user experience and boost the effectiveness of information retrieval, the study recommends improvement of KOHA OPAC usability by revamping its interface and enhancing its features.

This article discusses a multiple search session model of end-users' interaction with information retrieval systems based on results from an exploratory study investigating end-users' search sessions over time with online public access catalogs (OPAC) or CD-ROM databases at different stages of their information seeking related to a current research project. Interviews were conducted with 200 academic end-users to investigate the occurrence of multiple search sessions. Results show that at the time of the interview, 57% of end-users had conducted multiple search sessions during their research project and 86% of end-users conducted their first search session at the beginning stage of their information-seeking process. Forty-nine percent of end-users had conducted between 1 and 6 search sessions and 8% more than 6 search sessions. Seventy percent of multiple search session end-users' had modified their search terms since their first search session. The implications of the findings for end-user training, information retrieval systems design, and further research are discussed. © 1996 John Wiley & Sons, Inc.