Test-bed building process for context-aware peer-to-peer information retrieval evaluation

Due to the dramatically increasing amount of available data, effective and scalable solutions for data organization and search are essential. Distributed solutions naturally provide promising alternatives to standard centralized approaches. With the computational power of thousands or millions of computers in clusters or peer-to-peer systems, the challenges that arise are manifold, ranging from efficient resource discovery to issues in load balancing and distributed query processing. The 2008 edition of the Workshop on Large-Scale Distributed Systems for Information Retrieval (LSDS-IR'08) provided a forum for researchers to discuss these problems and to define new directions for the work on Distributed Information Retrieval. The Workshop program featured research contributions in the areas of similarity search, resource selection, network organization schemes, issues of data quality, result ranking techniques and query routing algorithms.

This paper proposes an efficient and effective "Locality Preserving Mapping" scheme that allows text databases representatives to be mapped onto a global information retrieval system such as Peer-to-Peer Information Retrieval Systems (P2PIR). The proposed approach depends on using Locality Sensitive Hash functions (LSH), and approximate min-wise independent permutations to achieve such task. Experimental evaluation over real data, along with comparison between different proposed schemes (with different parameters) will be presented in order to show the performance advantages of such schemes.

The LSDS-IR'13 workshop aims to bring together both information retrieval practitioners from industry, as well as academic researchers concerned with efficient and distributed IR systems. The workshop also welcomes contributions that propose different ways of leveraging diversity and multiplicity of resources available in distributed systems. The main goal of the workshop is to attract people from industry and academia to present and discuss ideas, problems and results in efficiency of large scale and distributed information retrieval systems, and to foster their participation to the WSDM conference.

PurposeThe purpose of this paper is to demonstrate how to apply traditional information retrieval (IR) evaluation methods based on standards from the Text REtrieval Conference and web search evaluation to all types of modern library information systems (LISs) including online public access catalogues, discovery systems, and digital libraries that provide web search features to gather information from heterogeneous sources.Design/methodology/approachThe authors apply conventional procedures from IR evaluation to the LIS context considering the specific characteristics of modern library materials.FindingsThe authors introduce a framework consisting of five parts: search queries, search results, assessors, testing, and data analysis. The authors show how to deal with comparability problems resulting from diverse document types, e.g., electronic articles vs printed monographs and what issues need to be considered for retrieval tests in the library context.Practical implicationsThe framework can be used as a guideline for conducting retrieval effectiveness studies in the library context.Originality/valueAlthough a considerable amount of research has been done on IR evaluation, and standards for conducting retrieval effectiveness studies do exist, to the authors’ knowledge this is the first attempt to provide a systematic framework for evaluating the retrieval effectiveness of twenty-first-century LISs. The authors demonstrate which issues must be considered and what decisions must be made by researchers prior to a retrieval test.

The study of cross-lingual Information Retrieval Systems (IRSs) and a deep analysis of system performances should provide guidelines, hints, and directions to drive the design and development of the next generation MultiLingual Information Access (MLIA) systems. In addition, effective tools for interpreting and comparing the experimental results should be made easily available to the research community. To this end, we propose a twofold methodology for the evaluation of Cross Language Information Retrieval (CLIR) systems: statistical analyses to provide MLIA researchers with quantitative and more sophisticated analysis techniques; and graphical tools to allow for a more qualitative comparison and an easier presentation of the results. We provide concrete examples about how the proposed methodology can be applied by studying the monolingual and bilingual tasks of the Cross-Language Evaluation Forum (CLEF) 2005 and 2006 campaigns.KeywordsInformation RetrievalMean Average PrecisionGraphical ToolBilingual ComparisonBilingual PerformanceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This report documents the program and the outcome of Dagstuhl Seminar 13441 Evaluation Methodologies in Information Retrieval, which brought together 42 participants from 11 countries. The seminar was motivated by the fact that today's information retrieval (IR) applications can hardly be evaluated based on the classic test collection paradigm, thus there is a need for new evaluation approaches. The event started with five introductory talks on evaluation frameworks, user modeling for evaluation, evaluation criteria, measures, evaluation methodology, and new trends in IR evaluation. The seminar participants then formed working groups addressing specific aspects of IR evaluation, such as reliability and validity, task-based IR, learning as search outcome, searching for fun, IR and social media, graph search, domain-specific IR, interaction measures and models, and searcher-aware information access systems.

Evaluation of information retrieval (IR) results is an important and difficult activity. Available evaluation methods are commonly based on the classical recall and precision measures. While these methods provide an evaluation of average system performance, they are not able to identify influence of other factors such as retrieval task and system user. Detailed information is hidden behind the average and no indication can be obtained from the evaluation for further improvement. Therefore, there is a need of new methods for detailed evaluation. In this paper, we propose a multi-dimensional approach to the evaluation of IR results so that effects of both an IR system and the environment where the system performs on retrieval results can be examined. It aids in the identification of problems of the system and the environment with respect to retrieval effectiveness, and assists in making improvement for IR.

“Update Strategies of Global Statistics in Distributed Information Retrieval System” By Jianjun Lang, Qigang Jiang Journal of Networks Volume 9, Number 10, October 2014, Page(s): 2751-2757 After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, the above paper has been found to be in violation of Academy Publisher’s Publication Principles. The first author (Jianjun Lang) has taken full responsibility and this violation was done without the knowledge of the second author (Qigang Jiang). This paper contains significant portions of original text from the papers cited below. The original text was copied without attribution (including appropriate references to the original authors and/or paper title) and without permission: Due to the nature of this violation, reasonable effort should be made to remove all past references to this paper, and future references should be made to the following article: Lijing Wang, “The Update Strategies of Global Statistics in Distributed Information Retrieval System,” Master Thesis, University of Chinese Academy of Sciences, April 2013, Beijing, China. (In Chinese)

As the number and diversity of distributed information sources on the Internet exponentially increase, various search services are developed to help the users to locate relevant information. But they still exist some drawbacks such as the difficulty of mathematically modeling retrieval process, the lack of adaptivity and the indiscrimination of search. This paper shows how heterogeneous neural networks can be used in the design of an intelligent distributed information retrieval (DIR) system. In particular, three typical neural network models - Kohoren's SOFM Network, Hopfield Network, and Feed Forward Network with Back Propagation algorithm are introduced to overcome the above drawbacks in current research of DIR by using their unique properties. This preliminary investigation suggests that Neural Networks are useful tools for intelligent search for distributed information sources.

Selection of genes that are differentially expressed and critical to a particular biological process has been a major challenge in post-array analysis. Recent development in bioinformatics has made various data sources available such as mRNA and miRNA expression profiles, biological pathway and gene annotation, etc. Efficient and effective integration of multiple data sources helps enrich our knowledge about the involved samples and genes for selecting genes bearing significant biological relevance. In this work, we studied a novel problem of multi-source gene selection: given multiple heterogeneous data sources (or data sets), select genes from expression profiles by integrating information from various data sources. We investigated how to effectively employ information contained in multiple data sources to extract an intrinsic global geometric pattern and use it in covariance analysis for gene selection. We designed and conducted experiments to systematically compare the proposed approach with representative methods in terms of statistical and biological significance, and showed the efficacy and potential of the proposed approach with promising findings.

The development and evaluation of Information Retrieval and Recommender Systems has traditionally focused on the relevance and accuracy of retrieved documents and recommendations, respectively. However, there is an increasing realization that accuracy alone might be a sub-optimal strategy for a successful user experience. Properties such as novelty and diversity have been explored in both fields for assessing and enhancing the usefulness of search results and recommendations. In this doctoral research we study the assessment and enhancement of both properties in the confluence of Information Retrieval and Recommender Systems.

In this chapter we discuss the application of a distributed information retrieval system in an enterprise environment. Focusing on the characteristics of information in enterprises such as the heterogeneity of available information, security policies, and the distributed nature of available data repositories we investigate how state-of-the-art distributed information retrieval approaches can be applied to build a distributed information retrieval system. Introducing a case study, we present an application of these techniques in an office environment that allows employees to find the most relevant documents across different data repositories without neglecting access rights. The application illustrates the advantages of using a multi-agent software infrastructure where individual components of such retrieval engine are realized by specific software agents.

The previous distributed information retrieval research based on the textual information defined new measures to improve the different process in the distributed information system, but neglected the use of the social information. From this point, we propose an approach which exploits the different social entities, to make a new profile adaptation technique, and to personalize the source selection process in a distributed information retrieval system.

Performance analysis of distributed information retrieval architectures using an improved network simulation model

Recent advancements in Generative AI, such as Large Language Models (LLMs), have demonstrated remarkable success across various general tasks. Extensive studies have explored leveraging generative models in finance, but significant challenges persist. This half-day workshop explores potential approaches and research directions to address these challenges by equipping generative models with advanced Information Retrieval (IR) models. Specifically, this workshop seeks to provide a platform for discussing innovative ideas that facilitate the advancement of IR technology to enrich generative models in finance from four key perspectives: (i) financial IR techniques (ii) financial IR benchmarking and evaluation (iii) financial systems and agents/assistants (iv) and trustworthiness, privacy and security when applying financial IR and generative models. This workshop aims to deepen understanding, accelerate progress, and support the advancement of IR technology to enhance generative models to address financial challenges.