Beyond Comparative Analysis: Making Arguments with Similarity Metrics and Structured Manuscript Data, with a Case Study in Marginal Iconography

Purpose Prior Information Retrieval (IR) research synthesizes progress from individual studies, yet academia-industry collaboration dynamics remain unexplored. This study investigates: (1) productivity patterns and venues, (2) citations-downloads relationships, (3) topic evolution, and (4) collaboration trends. Design/methodology/approach We perform an analysis of 53,471 ACM IR papers (2000–2018) using bibliometrics and DistilBERT topic modeling. Findings We find that industry-involved papers preferred WWW/CIKM venues; collaborations dominated RecSys/CSCW. We see that academia-industry collaborations achieved the highest download-to-citation conversion rates. Academia focused on algorithms; industry on applications; collaborations bridged both with rising human-centered themes. Research implications This is a pioneering large-scale bibliometrics revealing collaboration’s impact on IR knowledge evolution and provides a methodological framework for cross-sector analysis. Practical implications The paper identifies optimal venues (RecSys/CSCW) for partnerships and guides joint initiatives (shared datasets, grants) to bridge academia-industry divides and enhance research translation. Originality/value This is the first large-scale bibliometric analysis of IR academia-industry collaboration. The paper finds many novel insights, including the fact that collaboration boosts citation efficiency, enables complementary specialization, and drives topic convergence.

Humans have used electronic information retrieval (IR) systems for more than 50 years as they evolved from experimental systems to full-scale Web search engines and digital libraries. The fields of library and information science (LIS), cognitive science, human factors and computer science have historically been the leading disciplines in conducting research that seeks to model human interaction with IR systems for all kinds of information related behaviors. As technology problems have been mastered, the theoretical and applied framework for studying human interaction with IR systems has evolved from systems-centered to more user-centered, or cognitive-centered approaches. However, cognitive information retrieval (CIR) research that focuses on user interaction with IR systems is still largely under-funded and is often not included at computing and systems design oriented conferences. But CIR-focused research continues, and there are signs that some IR systems designers in academia and the Web search business are realizing that user behavior research can provide valuable insights into systems design and evaluation. The goal of our book is to provide an overview of new CIR research directions. This book does not provide a history of the research field of CIR. Instead, the book confronts new ways of looking at the human information condition with regard to our increasing need to interact with IR systems. The need has grown due to a number of factors, including the increased importance of information to more people in this information age. Also, IR was once considered document-oriented, but has now evolved to include multimedia, text, and other information objects. As a result, IR systems and their complexity have proliferated as users and user purposes for using them have also proliferated. Human interaction with IR systems can often be frustrating as people often lack an understanding of IR system functionality. New more holistic directions in CIR are emerging that conceptualize human-IR system interaction at the human-computer interaction level (taking into account interface issues), the information behavior level (taking into account the role of IR system interaction in the total range of people’s information behaviors), and the

Welcome to the 40th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017). SIGIR is the premier scientific conference in the broad area of information retrieval. We received a high-quality set of full-paper submissions to consider for inclusion in the conference program. We thank the 66 Senior Program Committee (SPC) members, 216 Program Committee (PC) members, and at least 70 additional reviewers for their contributions to paper selection. This pool of committed SIGIR volunteers was based in 33 countries and over 170 institutions, spanning academia, industry, and beyond. We recognized 8 SPC members and 29 PC members / additional reviewers whose contributions to the review process were outstanding. We reviewed 362 papers for the full paper track, and accepted 78, for an acceptance rate of 22%. In total, there were 1056 authors from 42 countries: 299 from China, 232 from the United States, and 525 from the rest of the world. The countries (with at least 10 submissions) with the highest acceptance rates were the United Kingdom (42%), Italy (42%), Singapore (38%), Australia (27%), and the United States (24%). The most popular call-for-papers topics for accepted papers were ranking algorithms (19%), text representation (17%), behavioral analysis (15%), recommender systems (14%), and user studies (14%). We made several modifications to the paper submission and review process for SIGIR 2017. We transitioned to EasyChair, one of three conference submission applications supported by ACM. We significantly updated the call for papers to broaden the range of topics considered in scope for the conference, and more accurately reflect current research trends. To boost diversity, we increased the number of women SPC members (24% this year, up from 15% last year) and increased the number of SPC members based in Asia (23% this year, up from 8% last year). To help improve review quality, we also required that PC members have at least one full paper at SIGIR and be active at the SIGIR conference in the past five years (some limited exceptions were permitted). The largest change was a modification to the role of the secondary SPC, with the goal of better utilizing their expertise and making better decisions for borderline papers. Each paper had a primary SPC member, who was responsible for writing the metareview and making a recommendation regarding the paper's disposition. If they believed it was necessary, the primary SPC could request an additional review from a secondary SPC member, who did not see the other reviews or the discussion to date. Once they submitted the additional review, the secondary SPC member could see the other reviews and participate in the discussion. The additional SPC review informed discussion and helped the primary SPC make a recommendation. Overall, secondary SPC reviews were provided for 20% of full papers. We ran an in-person SPC meeting at Microsoft in Redmond, WA. To allow more SPC members to take part, we enabled remote participation via video conference. Approximately half of the SPC participated, either in person or remotely. SPC members were asked to submit their full-paper recommendations prior to the meeting. During the meeting, we discussed papers flagged by SPCs as requiring discussion, as well as papers with "accept" and "reject" recommendations. Some of the papers marked as "reject" by SPC members before the meeting had strong support from at least one PC member. A subset of those full papers ended up being included in the conference program after a detailed discussion. The short paper track received a record number of 398 submissions and accepted 121 (30%) of them. The track was chaired by Hui Fang, Mounia Lalmas and Yiqun Liu. The demo paper track received 36 submissions and accepted 17 (47%) of them. The track was chaired by Gareth Jones and Takehiro Yamamoto. The SIGIR Symposium on IR in Practice (SIRIP 2017) had seven invited talks and five accepted regular papers. The track was chaired by Sumio Fujita and Vanessa Murdock. The doctoral consortium track received 14 submissions of which 9 (64%) were accepted. The track was chaired by Makoto P. Kato and Kira Radinsky. The tutorial track received 14 proposals of which 8 (57%) were accepted. The track was chaired by Fernando Diaz and Adam Jatowt. The workshop track received 12 proposals of which 8 (67%) were accepted. The track was chaired by Jaap Kamps and Grace Hui Yang.

INTRODUCTION:Increasing numbers of research papers about information retrieval for Health Technology Assessments (HTA), systematic reviews and other evidence syntheses are being published. It is time-consuming for information specialists to keep up-to-date with the latest developments in the field. To help searchers with this challenge, the Interest Group on Information Retrieval (IRG) of Health Technology Assessment International (HTAi) has compiled the best available research evidence on information retrieval aspects into an open-access web resource: Summarized Research in Information Retrieval for HTA (SuRe Info). The resource can be accessed at http://www.sure-info.orgMETHODS:The Sure Info team run topic-specific search strategies in selected relevant databases to identify information retrieval methods publications that fulfil the SuRe Info inclusion criteria. Eligible publications receive a structured abstract containing a brief critical appraisal. Key messages for search practice based on the appraisals and accepted best practice are summarized into topic-specific chapters.RESULTS:SuRe Info currently offers fourteen chapters, with more in development. SuRe Info chapters fall into two categories: (i) chapters about general search methods that are used across all types of research, such as how to develop search strategies and the availability and use of search filters, and (ii) chapters summarizing the methods to use when searching for specific aspects of HTA (as defined in the European Network for HTA (EUnetHTA) HTA Core Model®), including searching for evidence on clinical effectiveness and safety, and identifying economic evaluations. References at the end of each chapter are linked to appraisals of publications that have been used to develop each chapter. Links to the full-text of the publications are provided when freely available. The SuRe Info chapters are reviewed every six months and updated if new evidence is identified or if resources change.CONCLUSIONS:SuRe Info is a unique resource, identifying and summarizing current best research evidence on information retrieval aspects for HTA. It supports the timely uptake of potential efficiencies arising from new evidence that may be incorporated into the evidence identification processes of HTA organizations.

For many years, vector space models have been used in information retrieval and computational linguistics to represent terms, queries, and documents, using vector addition as a simple operator to model semantic composition. Though surprisingly successful, many aspects of meaning including word order, typed relationships, and nested structures are not captured by this modelling process.

Ongoing research in information retrieval offers new approaches, demonstrating that the traditional methods may have reached their limitations. In fact, actual results to date have shown that advanced systems will demonstrate effectiveness at combining multiple text and user need representations. The paper suggests supplementing the traditional conceptual abstracts with a syntactical description designed to improve retrieval performance. This choice is carefully motivated and examples are given. Without attempting to reason about the document contents, a domain specific knowledge representation is used to form clusters of concepts. The point is to represent the joint structure of the component concepts involved in a composite concept. This approach extends the reach of key concept assignment to: interactive query elicitation; multiword phrase search; finer grained thesaurus relationships. The paper emphasizes that intellectual assignment of features is part of an artificial intelligence approach to information retrieval. The principles of implementation are also described.

Academic research in information retrieval did not make its way into commercial retrieval products until the last 15 years. Early web search engines also made little use of information retrieval research, in part because of significant differences in the retrieval environment on the Web, such as higher transaction volume and much shorter queries. Recently, however, academic research has taken root in search engines. This paper describes recent developments with a probabilistic retrieval model originating prior to the Web, but with features which could lead to effective retrieval on the Web. Just as graph structure algorithms make use of the graph structure of hyper-linking on the Web, which can be considered a form of relevance judgments, the model of this paper suggests how relevance judgments of web searchers, not just web authors, can be taken into account in ranking. This paper also shows how the combination of expert opinion probabilistic information retrieval model can be made computationally efficient through a new derivation of the mean and standard deviation of the model’s main probability distribution.

BackgroundTest collections for information retrieval are scarce. Domain specific test collections even more so, and medical test collections in the Swedish language non-existent prior to the making of the MedEval test collection. Most research in information retrieval has been performed in the English language, thus most test collections contain English documents. However, English is morphologically poor compared to many other European languages and a number of interesting and important aspects have not been investigated. Building a medical test collection in Swedish opens new research opportunities.MethodsThis article describes the making of and potential uses of MedEval, a Swedish medical test collection with assessments, not only for topical relevance, but also for target reader group: Doctors or Patients. A user of the test collection may choose if she wishes to search in the Doctors or the Patients scenario where the topical relevance assessments have been adjusted with consideration to user group, or to search in a scenario which regards only topical relevance.In addition to having three user groups, MedEval, in its present form, has two indexes, one where the terms are lemmatized and one where the terms are lemmatized and the compounds split and the constituents indexed together with the whole compound.ResultsDifferences discovered between the documents written for medical professionals and documents written for laypersons are presented. These differences may be utilized in further studies of retrieval of documents aimed at certain groups of readers. Differences between the groups of documents are, for example, that professional documents have a higher ratio of compounds, have a greater average word length and contain more multi-word expressions.An experiment is described where the user scenarios have been utilized, searching with expert terms and lay terms, separately and in combination in the different scenarios. The tendency discovered is that the medical expert gets best results using expert terms and the lay person best results using lay terms, but also quite good results using expert terms or lay and expert terms in combination.ConclusionsThe many features of MedEval gives a variety of research possibilities, such as comparing the effectiveness of search terms when it comes to retrieving documents aimed at the different user groups or to study the effect of compound decomposition in retrieval of documents. As Swedish, the language of MedEval, is a morphologically more complex language than English, it is possible to study additional aspects of the effect of natural language processing in information retrieval, for example utilizing different inflectional word forms in the retrieval of expert vs lay documents. MedEval is the first Swedish test collection of the medical domain.AvailabilityThe Department of Swedish at the University of Gothenburg is in the process of making the MedEval test collection available to academic researchers.

Accurate ordering or ranking over instances is of paramount importance for several applications (Faria et al. Learning to rank for content-based image retrieval. In: Proceedings of the Multimedia Information Retrieval Conference, pp. 285–294, 2010; Veloso et al. Learning to rank at query-time using association rules. In: Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), pp. 267–274, 2008; Veloso et al. J Inf Data Manag 1(3): 567–582, 2010; Veloso and Meira, Efficient on-demand opinion mining. In: Proceedings of the Brazilian Symposium on Databases (SBBD), pp. 332–346, 2007; Veloso et al. Automatic moderation of comments in a large on-line journalistic environment. In: Proceedings of the International AAAI Conference on Weblogs and Social Media (ICWSM), pp. 234–237, AAAI, 2007).One clear application is Information Retrieval, where documents retrieved by search engines must be ranked according to the corresponding relevance to the query (Trotman, Inf Ret 8(3):359–381, 2005).Many features may affect the relevance of such documents, and, thus, it is difficult to adapt ranking functions manually. Recently, a body of empirical evidence has emerged suggesting that methods that automatically learn ranking functions offer substantial improvements in enough situations to be regarded as a relevant advance for applications that depend on ranking . Hence, learning ranking functions has attracted significant interest from the machine learning community. In the context of Information Retrieval, the conventional approach to this learning task is to assume the availability of examples (i.e., a training data, \({S},\) which typically consists of document features and the corresponding relevance to specific queries), from which a learning function can be learned. When a new query is given, the documents associated with this query are ranked according to the learned function (i.e., this function gives a score to a document indicating its relevance with regard to the query). In this chapter we present ranking algorithms based on demand-driven associative classification .

In this introduction, we present our overview of interactive question answering (IQA). We contextualize IQA in the wider field of question answering, and establish connections to research in Information Retrieval and Dialogue Systems. We highlight the development of QA as a field, and identify challenges in the present research paradigm for which IQA is a potential solution. Finally, we present an overview of papers in this special issue, drawing connections between these and the challenges they address.

Over the last three decades, research in Information Retrieval (IR) shows performance improvement when many sources of evidence are combined to produce a ranking of documents. Most current approaches assess document relevance by computing a single score which aggregates values of some attributes or criteria. They use analytic aggregation operators which either lead to a loss of valuable information, e.g., the min or lexicographic operators, or allow very bad scores on some criteria to be compensated with good ones, e.g., the weighted sum operator. Moreover, all these approaches do not handle imprecision of criterion scores. In this paper, we propose a multiple criteria framework using a new aggregation mechanism based on decision rules identifying positive and negative reasons for judging whether a document should get a better ranking than another. The resulting procedure also handles imprecision in criteria design. Experimental results are reported showing that the suggested method performs better than standard aggregation operators.

Much research in information retrieval (IR) focuses on optimization of the rank of relevant retrieval results for single shot ad hoc IR tasks. Relatively little research has been carried out on user engagement to support more complex search tasks. We seek to improve user engagement for IR tasks by providing richer representation of retrieved information. It is our expectation that this strategy will promote implicit learning within search activities. Specifically, we plan to explore methods of finding semantic concepts within retrieved documents, with the objective of creating improved document surrogates. Further, we would like to study search effectiveness in terms of different facets such as the user's search experience, satisfaction, engagement and learning. We intend to investigate this in an experimental study, where our richer document representations are compared with the traditional document surrogates for the same user queries.

Search engines are based on models to index documents, match queries and documents and rank documents. Research in Information Retrieval (IR) aims at defining these models and their parameters in order to optimize the results. Using benchmark collections, it has been shown that there is not a best system configuration that works for any query, but rather that performance varies from one query to another. It would be interesting if a meta-system could decide which system configuration should process a new query by learning from the context of previousqueries. This paper reports a deep analysis considering more than 80,000 search engine config- urations applied to 100 queries and the corresponding performance. The goal of the analysis is to identify which configuration responds best to a certain type of query. We considered two approaches to define query types: one is post-evaluation, based on query clustering according to the performance measured with Average Precision, while the second approach is pre-evaluation, using query features (including query difficulty predictors) to cluster queries. Globally, we identified two parameters that should be optimized: retrieving model and TrecQueryTags process. One could ex- pect such results as these two parameters are major components of IR process. However our work results in two main conclusions: 1) based on post-evaluation approach, we found that retrieving model is the most influential parameter for easy queries while TrecQueryTags process is for hard queries; 2) for pre-evaluation, current query features do not allow to cluster queries to identify differences in the influential parameters.

Supervised learning and pattern recognition is a crucial area of research in information retrieval, knowledge engineering, image processing, medical imaging, and intrusion detection. Numerous algorithms have been designed to address such complex application domains. Despite an enormous array of supervised classifiers, researchers are yet to recognize a robust classification mechanism that accurately and quickly classifies the target dataset, especially in the field of intrusion detection systems (IDSs). Most of the existing literature considers the accuracy and false-positive rate for assessing the performance of classification algorithms. The absence of other performance measures, such as model build time, misclassification rate, and precision, should be considered the main limitation for classifier performance evaluation. This paper’s main contribution is to analyze the current literature status in the field of network intrusion detection, highlighting the number of classifiers used, dataset size, performance outputs, inferences, and research gaps. Therefore, fifty-four state-of-the-art classifiers of various different groups, i.e., Bayes, functions, lazy, rule-based, and decision tree, have been analyzed and explored in detail, considering the sixteen most popular performance measures. This research work aims to recognize a robust classifier, which is suitable for consideration as the base learner, while designing a host-based or network-based intrusion detection system. The NSLKDD, ISCXIDS2012, and CICIDS2017 datasets have been used for training and testing purposes. Furthermore, a widespread decision-making algorithm, referred to as Techniques for Order Preference by Similarity to the Ideal Solution (TOPSIS), allocated ranks to the classifiers based on observed performance reading on the concern datasets. The J48Consolidated provided the highest accuracy of 99.868%, a misclassification rate of 0.1319%, and a Kappa value of 0.998. Therefore, this classifier has been proposed as the ideal classifier for designing IDSs.

Vocabulary size and its effect on topic representation