Toward Cranfield-inspired reusability assessment in interactive information retrieval evaluation

This paper reports results of an evaluation study of MAP (Multi‐faceted Access to PubMed), a metadata induced query suggestion interface for PubMed bibliographic search.A novel evaluation methodology was used to address the challenges involved in evaluating an IIR (Interactive Information Retrieval) system such as the MAP interface. The most significant aspect of this methodology is that, instead of using assigned tasks common in traditional IR evaluation, it asks real users with real search requests to search with real systems in an experimental setting. Several performance measures were created based on which comparisons were made between MAP and PubMed baseline. MAP was shown to perform better in several of these measures, especially when the search requests had not been attempted before.The finding pointed to search characteristics as an important intervening variable in IIR evaluation. The advantages of and potential threats to our methodology were also discussed.

Interest characterizes a cognitive-emotional relationship between people and information and is a key construct in human information interaction. As a motivational variable, interest has been widely studied in psychology and education, but has received less consistent and theory-driven attention in the field of interactive information retrieval (IIR). In this perspective paper we examine the role of interest and review how it has been studied and operationalized in IIR research. We draw upon a survey of 58 research studies that have manipulated, controlled or measured searcher interest in some way. The intent of the paper is to raise the profile of interest as a user-centred variable in IIR and to advocate for more conceptual and methodological consistency in future studies to better evaluate the impact of interest in information search.

We report early results of our investigation of help-seeking behaviors during interactive information retrieval, with specific attention to one aspect of context, affective utterances made by searchers during the process of completing an assigned search task Searchers' utterances were coded as positive and negative assessments of the results of using automatic Help functionalities in two digital libraries. Although preliminary at this time, data presented in this paper indicate that affective utterances can be mapped to specific help-seeking events and therefore can be construed as important aspects of contextual feedback, worthy of further exploration.

When evaluating interactive information retrieval (IIR) interfaces, it is common to collect data using subjective measures such as satisfaction, ease of use, usefulness, and user engagement. However, as these are collected post-task, they serve as surrogate measures for what occurred in the midst of the search activities. Further, such approaches may be subject to recency effects, where the last action in the search process influences the searchers’ opinions about the overall process. With recent improvements in facial emotion classification approaches, we propose that measuring emotional responses may provide a better indication of what is happening throughout search tasks. In this research, we present an approach for collecting real-time emotional responses during a search activity using consumer-grade front-facing cameras and a method of aligning these with search interface feature use. To validate the effectiveness of the approach, we have conducted a controlled laboratory study in which we manipulated the quality of the search results in order to determine if we can detect expected emotional responses, whether search behaviours influencing these emotional responses, and whether recency effects are present in post-task measures. The preliminary results of this study show that our approach is reliable for detecting emotional responses when searchers experience positive and negative emotions throughout the search process, isolate which interactive elements were used when positive and negative emotional responses were experienced, and illustrate how recency effects are present in post-task measures. Our upcoming study will investigate how our approach can be used to evaluate novel search interfaces. We will develop a novel search interface and evaluate it using our approach. Finally, we will create a dashboard to monitor academic literature. Using the same approach, we will demonstrate our approach can extend beyond traditional search interfaces and into more general interface assessment.

PurposeThis study continues a long history of author co-citation analysis of the intellectual structure of information science into the time period of 2011–2020. It also examines changes in this structure from 2006–2010 through 2011–2015 to 2016–2020. Results will contribute to a better understanding of the information science research field.Design/methodology/approachThe well-established procedures and techniques for author co-citation analysis were followed. Full records of research articles in core information science journals published during 2011–2020 were retrieved and downloaded from the Web of Science database. About 150 most highly cited authors in each of the two five-year time periods were selected from this dataset to represent this field, and their co-citation counts were calculated. Each co-citation matrix was input into SPSS for factor analysis, and results were visualized in Pajek. Factors were interpreted as specialties and labeled upon an examination of articles written by authors who load primarily on each factor.FindingsThe two-camp structure of information science continued to be present clearly. Bibliometric indicators for research evaluation dominated the Knowledge Domain Analysis camp during both fivr-year time periods, whereas interactive information retrieval (IR) dominated the IR camp during 2011–2015 but shared dominance with information behavior during 2016–2020. Bridging between the two camps became increasingly weaker and was only provided by the scholarly communication specialty during 2016–2020. The IR systems specialty drifted further away from the IR camp. The information behavior specialty experienced a deep slump during 2011–2020 in its evolution process. Altmetrics grew to dominate the Webometrics specialty and brought it to a sharp increase during 2016–2020.Originality/valueAuthor co-citation analysis (ACA) is effective in revealing intellectual structures of research fields. Most related studies used term-based methods to identify individual research topics but did not examine the interrelationships between these topics or the overall structure of the field. The few studies that did discuss the overall structure paid little attention to the effect of changes to the source journals on the results. The present study does not have these problems and continues the long history of benchmark contributions to a better understanding of the information science field using ACA.

This report describes the program and the results of Dagstuhl Seminar 17092 Search as Learning, which brought together 26 researchers from diverse research backgrounds. The motivation for the seminar stems from the fact that modern Web search engines are largely engineered and optimized to fulfill lookup tasks instead of complex search tasks. The latter though are an essential component of information discovery and learning. The 3-day seminar started with four perspective talks, providing four different views on the topic of search as learning: interactive information retrieval (IR), psychology, education and system-oriented IR. The remainder of the seminar centered around breakout groups leading to new views on the challenges and issues in search as learning, interspersed with research spotlight talks.

Test collections for offline evaluation remain crucial for information retrieval research and industrial practice, yet reusability of test collections is under threat by different factors such as dynamic nature of data collections and new trends in building retrieval systems. Specifically, building reusable test collections that last over years is a very challenging problem as retrieval approaches change considerably per year based on new trends among Information Retrieval researchers. We experiment with a novel temporal reusability test to evaluate reusability of test collections over a year based on leaving mutual topics in experiment, in which we borrow some judged topics from previous years and include them in the new set of topics to be used in the current year. In fact, we experiment whether a new set of retrieval systems can be evaluated and comparatively ranked based on an old test collection. Our experiments is done based on two sets of runs from Text REtrieval Conference (TREC) 2015 and 2016 Contextual Suggestion Track, which is a personalized venue recommendation task. Our experiments show that the TREC 2015 test collection is not temporally reusable. The test collection should be used with extreme care based on early precision metrics and slightly less care based on NDCG, bpref and MAP metrics. Our approach offers a very precise experiment to test temporal reusability of test collections over a year, and it is very effective to be used in tracks running a setup similar to their previous years.

In interactive IR (IIR), users often seek to achieve different goals (e.g. exploring a new topic, finding a specific known item) at different search iterations and thus may evaluate system performances differently. Without state-aware approach, it would be extremely difficult to simulate and achieve real-time adaptive search evaluation and recommendation. To address this gap, our work identifies users' task states from interactive search sessions and meta-evaluates a series of online and offline evaluation metrics under varying states based on a user study dataset consisting of 1548 unique query segments from 450 search sessions. Our results indicate that: 1) users' individual task states can be identified and predicted from search behaviors and implicit feedback; 2) the effectiveness of mainstream evaluation measures (measured based upon their respective correlations with user satisfaction) vary significantly across task states. This study demonstrates the implicit heterogeneity in user-oriented IR evaluation and connects studies on complex search tasks with evaluation techniques. It also informs future research on the design of state-specific, adaptive user models and evaluation metrics.

When we search information on the Web, we usually enter a query that best represents our information need, and then modify it as the search results are shown. Many thus have studied search queries to understand how users express and find their information need on the Web. For example, studies have shown that query length affects search performance (Belkin et al, 2003). Then, researchers have begun to examine image search queries on the Web. They analyzed search logs from general or commercial search engines or studied specific image database systems based on log analysis (Goodrum & Sprink, 2001; Jorgensen & Jorgensen, 2005; Goodrum, Bejune, & Siochi, 2003; Pu, 2005; Cunningham, Bainbridge, & Masoodian, 2004). While these studies provide basic understanding of image query formulation, further characteristics of user's queries in the interactive searching process have not been explored. This study thus builds on previous studies on image search queries by examining the patterns of query formulation and modification in visual information search. Specifically, first, it investigates the effects of tasks and other contextual factors on query length and query changes; and second, it compares the query length and changes with the user's satisfaction of search results. Participants Twenty-nine college students were recruited for data collection during the spring and summer semesters of 2008, mostly from the Department of Media Studies at the Catholic University of America (CUA). They included 22 from Media Studies and 7 from other fields taking courses in the department; 22 were females and 7 males; and an average age of 21 years. Background information collected were years of computer experience, hours per day using the Web, self-rated online searching expertise, and self-rated digital image searching experience. The seven-point Likert scale was used for self-rating variables. Each participant took part in three different search sessions with his or her own search topic at CUA with the researcher's laptop that had an Internet connection. The university's homepage was set as the default home for the Web browser. Each participant filled out a pre-search questionnaire to collect a search task, topic, self-rated topic familiarity using 7-Likert scale. The participants chose their own preferred browser between Internet Explorer and Firefox. The search session was recorded by Camtasia v5.0 as screen capture. After the session, a post-search questionnaire was given to the participants to assess the level of satisfaction and confidence of their searching using 7-Likert scale. There was no time limit to complete a search session. The participants were free to end a search session at any time. Search Tasks The participants brought their own task and topics to find images. There were three types of search tasks identified: academic task, work-related task, and personal interest (Table 1). Search queries were acquired from screen capture log files. A total of 979 queries were obtained as the entire string of terms submitted by the participants. Table 2 shows a distribution of search queries on a general search engine, a search engine's image search, and other specific sites. Findings showed that Google Images Search was the most frequent source for forming a search query. An average word per query was 3.07 during the first session, 2.95 during the second session, and 2.89 for the last session. This finding of short queries is similar to that of previous studies on Web image queries (Goodrum & Spink, 2001; Jorgensen & Jorgensen, 2005). Analysis of data is in progress. A poster will summarize detail findings and provide answers to the following questions: What are the patterns of query modification? Are there any differences of query length and number of query change based on the task and topic familiarities? Is there any relationship between query length and number of query change and users' satisfaction and confidence with search results? Do different stages of a user's particular task affect query length and number of query change? The author would like to thank OCLC/ALISE for their generous funding for this project.

This article describes a new TREC Enterprise Track search test collection -- CERC. The collection is designed to represent some real-world search activity within the enterprise, using as a specific example the Commonwealth Scientific and Industrial Research Organisation (CSIRO). It has a deep crawl of CSIRO's public-facing information, that is very similar to the crawl of a real-world search service provided by CSIRO. The search tasks are based on the activities of CSIRO Science Communicators, who are CSIRO employees that deal with public-facing information. Topics and judgments are tied to the Science Communicators in various ways, for example by involving them in the topic development process. The overall approach is to enhance the validity of the test collection as a model of enterprise search, by tying it to real-world examples.

Creating test collections for modern search tasks is increasingly more challenging due to the growing scale and dynamic nature of content, and need for richer contextualization of the statements of request. To address these issues, the TREC Contextual Suggestion Track explored an open test collection, where participants were allowed to submit any web page as a result for a personalized venue recommendation task. This prompts the question on the reusability of the resulting test collection: How does the open nature affect the pooling process? Can participants reliably evaluate variant runs with the resulting qrels? Can other teams evaluate new runs reliably? In short, does the set of pooled and judged documents effectively produce a post hoc test collection? Our main findings are the following: First, while there is a strongly significant rank correlation, the effect of pooling is notable and results in underestimation of performance, implying the evaluation of non-pooled systems should be done with great care. Second, we extensively analyze impacts of open corpus on the fraction of judged documents, explaining how low recall affects the reusability, and how the personalization and low pooling depth aggravate that problem. Third, we outline a potential solution by deriving a fixed corpus from open web submissions.

Network providers are nowadays facing with a big question in allocating network resources due to the constantly increasing of Internet services. While the network resource is not unlimited, users always desire the best quality of experience (QoE) with the huge information exchange. Therefore, finding a justice of network resource allocation based on the user experience is mandatory. In previous studies, network resources were allocated to all users by using a specific utility function without considering the user characteristics. In fact, the network resource consumptions are very different among individual users and directly depend on users' behavior. For instance, the network demands of relax users are usually lower than those of busy users. Thus, allocating the same amount of resources to all users might not meet their expectations. In this paper, we propose two bandwidth allocation methods by classifying users into different groups based on their characteristics such as relax, short-time busy, and long-time busy users. To allocate the bandwidth in each group, specific utility functions are applied. By using these methods, the obtained results show that users get the different allocated bandwidth while they still experience the same level of QoE. On the other hand, the allocation considers users' behavior to allocate suitable bandwidth based on the real resource consumption of users. As a result from these analyses, our proposal tends toward a fair allocation as well as an efficient management of the network resources.

Accurate estimation of information retrieval evaluation metrics such as average precision require large sets of relevance judgments. Building sets large enough for evaluation of real-world implementations is at best inefficient, at worst infeasible. In this work we link evaluation with test collection construction to gain an understanding of the minimal judging effort that must be done to have high confidence in the outcome of an evaluation. A new way of looking at average precision leads to a natural algorithm for selecting documents to judge and allows us to estimate the degree of confidence by defining a distribution over possible document judgments. A study with annotators shows that this method can be used by a small group of researchers to rank a set of systems in under three hours with 95% confidence. Information retrieval metrics such as average precision require large sets of relevance judgments to be accurately estimated. Building these sets is infeasible and often inefficient for many real-world retrieval implementations. We present a new way of looking at average precision that allows us to estimate the confidence in an evaluation based on the size of the test collection. We use this to build an algorithm for selecting the best documents to judge to have maximum confidence in an evaluation with a minimal number of relevance judgments. A study with annotators shows how the algorithm can be used by a small group of researchers to quickly rank a set of systems with 95% confidence.

We consider a search task as a set of queries that serve the same user information need. Analyzing search tasks from user query streams plays an important role in building a set of modern tools to improve search engine performance. In this paper, we propose a probabilistic method for identifying and labeling search tasks based on the following intuitive observations: queries that are issued temporally close by users in many sequences of queries are likely to belong to the same search task, meanwhile, different users having the same information needs tend to submit topically coherent search queries. To capture the above intuitions, we directly model query temporal patterns using a special class of point processes called Hawkes processes, and combine topic models with Hawkes processes for simultaneously identifying and labeling search tasks. Essentially, Hawkes processes utilize their self-exciting properties to identify search tasks if influence exists among a sequence of queries for individual users, while the topic model exploits query co-occurrence across different users to discover the latent information needed for labeling search tasks. More importantly, there is mutual reinforcement between Hawkes processes and the topic model in the unified model that enhances the performance of both. We evaluate our method based on both synthetic data and real-world query log data. In addition, we also apply our model to query clustering and search task identification. By comparing with state-of-the-art methods, the results demonstrate that the improvement in our proposed approach is consistent and promising.

Medical systematic review query formulation is a highly complex task done by\ntrained information specialists. Complexity comes from the reliance on lengthy\nBoolean queries, which express a detailed research question. To aid query\nformulation, information specialists use a set of exemplar documents, called\n`seed studies', prior to query formulation. Seed studies help verify the\neffectiveness of a query prior to the full assessment of retrieved studies.\nBeyond this use of seeds, specific IR methods can exploit seed studies for\nguiding both automatic query formulation and new retrieval models. One major\nlimitation of work to date is that these methods exploit `pseudo seed studies'\nthrough retrospective use of included studies (i.e., relevance assessments).\nHowever, we show pseudo seed studies are not representative of real seed\nstudies used by information specialists. Hence, we provide a test collection\nwith real world seed studies used to assist with the formulation of queries. To\nsupport our collection, we provide an analysis, previously not possible, on how\nseed studies impact retrieval and perform several experiments using seed-study\nbased methods to compare the effectiveness of using seed studies versus pseudo\nseed studies. We make our test collection and the results of all of our\nexperiments and analysis available at\nhttp://github.com/ielab/sysrev-seed-collection\n