Measuring Hypothesis Testing Errors in the Evaluation of Retrieval Systems

Document relevance has been the primary focus in the design, optimization and evaluation of retrieval systems. Traditional testcollections are constructed by asking judges the relevance grade for a document with respect to an input query. Recent work of Yilmaz et al. found an evidence that effort is another important factor in determining document utility, suggesting that more thought should be given into incorporating effort into information retrieval. However, that work did not ask judges to directly assess the level of effort required to consume a document or analyse how effort judgements relate to traditional relevance judgements. In this work, focusing on three aspects associated with effort, we show that it is possible to get judgements of effort from the assessors. We further show that given documents of the same relevance grade, effort needed to find the portion of the document relevant to the query is a significant factor in determining user satisfaction as well as user preference between these documents. Our results suggest that if the end goal is to build retrieval systems that optimize user satisfaction, effort should be included as an additional factor to relevance in building and evaluating retrieval systems. We further show that new retrieval features are needed if the goal is to build retrieval systems that jointly optimize relevance and effort and propose a set of such features. Finally, we focus on the evaluation of retrieval systems and show that incorporating effort into retrieval evaluation could lead to significant differences regarding the performance of retrieval systems.

Despite the centrality of topical relevance in information retrieval system design and evaluation, understanding and implementation of it is usually limited to “direct overall topical matching” between the subject of the query and the subject of the document. The underlying assumption is that only a single type of topical relationship is involved. In related work, a relevance judgment instrument was developed for the Multilingual Access to Large Spoken ArCHives project (MALACH). It incorporates the five topical relevance types (direct relevance, indirect/circumstantial relevance, context relevance, comparison relevance, and pointer relevance) and was applied by four judges to items in the MALACH test collection in Summer 2003. This paper reports on the experiences and perceptions of the judges making more nuanced judgments about topical relevance. The results suggest that more than only one variable/dimension, “whether it is on topic” as usually referred to, contributes to topical relevance, and more than a single topical relationship type, “direct matching” as generally assumed, play an important role in topical relevance.

Background/Objectives: Cefoperazone-sulbactam is frequently prescribed to older inpatients for severe infections but has been associated with coagulation abnormalities, particularly among individuals with malnutrition or hepatic dysfunction. Early identification of at-risk patients remains challenging. To develop and validate a clinically interpretable model for predicting cefoperazone-sulbactam-related coagulation abnormalities in elderly inpatients and to provide practical tools for bedside risk estimation. Methods: We conducted a retrospective, dual-center study of 485 patients aged ≥ 60 years treated at Fuxing Hospital and Xuanwu Hospital of Capital Medical University who received cefoperazone-sulbactam for ≥72 h. Baseline clinical and demographic variables were analyzed using univariate and multivariable logistic regression to identify independent risk factors. Ten supervised machine-learning models were trained and evaluated using area under the ROC curve (AUC), accuracy, sensitivity, specificity, precision, and F1-score. SHapley Additive exPlanations (SHAP) were applied to assess model interpretability. A nomogram was constructed from the final logistic regression model, and a web-based calculator was developed for clinical use. Results: Multivariable analysis identified age ≥ 75 years, hypoproteinemia, total parenteral nutrition, insomnia, and recent oral antibiotic use as independent predictors of coagulation abnormalities. Among machine-learning models, LightGBM achieved the best overall performance by AUC and balanced classification metrics. SHAP analyses provided individualized and global explanations of feature contributions, facilitating clinical interpretation. The nomogram and web calculator enable rapid, patient-specific risk estimation. Conclusions: An interpretable machine-learning approach, complemented by a nomogram and web calculator, accurately stratifies the risk of cefoperazone-sulbactam-induced coagulation abnormalities in elderly inpatients. These tools may support personalized risk evaluation and earlier preventive interventions in routine care. The web-based calculator facilitates rapid bedside risk estimation and may help guide earlier monitoring and preventive interventions in routine care.

PurposeThe purpose of this paper is to demonstrate how to apply traditional information retrieval (IR) evaluation methods based on standards from the Text REtrieval Conference and web search evaluation to all types of modern library information systems (LISs) including online public access catalogues, discovery systems, and digital libraries that provide web search features to gather information from heterogeneous sources.Design/methodology/approachThe authors apply conventional procedures from IR evaluation to the LIS context considering the specific characteristics of modern library materials.FindingsThe authors introduce a framework consisting of five parts: search queries, search results, assessors, testing, and data analysis. The authors show how to deal with comparability problems resulting from diverse document types, e.g., electronic articles vs printed monographs and what issues need to be considered for retrieval tests in the library context.Practical implicationsThe framework can be used as a guideline for conducting retrieval effectiveness studies in the library context.Originality/valueAlthough a considerable amount of research has been done on IR evaluation, and standards for conducting retrieval effectiveness studies do exist, to the authors’ knowledge this is the first attempt to provide a systematic framework for evaluating the retrieval effectiveness of twenty-first-century LISs. The authors demonstrate which issues must be considered and what decisions must be made by researchers prior to a retrieval test.

The development and evaluation of Information Retrieval and Recommender Systems has traditionally focused on the relevance and accuracy of retrieved documents and recommendations, respectively. However, there is an increasing realization that accuracy alone might be a sub-optimal strategy for a successful user experience. Properties such as novelty and diversity have been explored in both fields for assessing and enhancing the usefulness of search results and recommendations. In this doctoral research we study the assessment and enhancement of both properties in the confluence of Information Retrieval and Recommender Systems.

Area under the curve may hide poor generalisation to external datasets

A useful ability for search engines is to be able to rank objects with novelty and diversity : the top k documents retrieved should cover possible interpretations of a query with some distribution, or should contain a diverse set of subtopics related to the user's information need, or contain nuggets of information with little redundancy. Evaluation measures have been introduced to measure the effectiveness of systems at this task, but these measures have worst-case NP-complete computation time. We use simulation to investigate the implications of this for optimization and evaluation of retrieval systems.

Evaluation of information retrieval (IR) results is an important and difficult activity. Available evaluation methods are commonly based on the classical recall and precision measures. While these methods provide an evaluation of average system performance, they are not able to identify influence of other factors such as retrieval task and system user. Detailed information is hidden behind the average and no indication can be obtained from the evaluation for further improvement. Therefore, there is a need of new methods for detailed evaluation. In this paper, we propose a multi-dimensional approach to the evaluation of IR results so that effects of both an IR system and the environment where the system performs on retrieval results can be examined. It aids in the identification of problems of the system and the environment with respect to retrieval effectiveness, and assists in making improvement for IR.

This paper deals with evaluation of information retrieval from unsegmented speech. We focus on Mean Generalized Average Precision, the evaluation measure widely used for unsegmented speech retrieval. This measure is designed to allow certain tolerance in matching retrieval results (starting points of relevant segments) against a gold standard relevance assessment. It employs a Penalty Function which evaluates non-exact matches in the retrieval results based on their distance from the beginnings of their nearest true relevant segments. However, the choice of the Penalty Function is usually ad-hoc and does not necessary reflect users' perception of the speech retrieval quality. We perform a lab test to study satisfaction of users of a speech retrieval system to empirically estimate the optimal shape of the Penalty Function.

The traditional evaluation of information retrieval (IR) systems is generally very costly as it requires manual relevance annotation from human experts. Recent advancements in generative artificial intelligence -specifically large language models (LLMs)- can generate relevance annotations at an enormous scale with relatively small computational costs. Potentially, this could alleviate the costs traditionally associated with IR evaluation and make it applicable to numerous low-resource applications. However, generated relevance annotations are not immune to (systematic) errors, and as a result, directly using them for evaluation produces unreliable results.

Retrieval evaluation is always an important aspect in information retrieval web search and metrics are a key factor that needs to be carefully considered. In this paper, we propose a new method of measuring stability and discrimination power of a metric. The problem is initiated by Buckley and Voorhees. The advantage of the proposed method is that we are able to measure both aspects together in a systematic manner. Five metrics are tested in the study. They are average precision over all relevant documents, recall-level precision, normalized discount cumulative gain, precision at 10 documents level, and reciprocal rank. Experimental results show that normalized discount cumulative gain is the best, which is followed by average precision over all relevant documents, recall-level precision, precision at 10 documents level, while reciprocal rank is the worst.

The study of cross-lingual Information Retrieval Systems (IRSs) and a deep analysis of system performances should provide guidelines, hints, and directions to drive the design and development of the next generation MultiLingual Information Access (MLIA) systems. In addition, effective tools for interpreting and comparing the experimental results should be made easily available to the research community. To this end, we propose a twofold methodology for the evaluation of Cross Language Information Retrieval (CLIR) systems: statistical analyses to provide MLIA researchers with quantitative and more sophisticated analysis techniques; and graphical tools to allow for a more qualitative comparison and an easier presentation of the results. We provide concrete examples about how the proposed methodology can be applied by studying the monolingual and bilingual tasks of the Cross-Language Evaluation Forum (CLEF) 2005 and 2006 campaigns.KeywordsInformation RetrievalMean Average PrecisionGraphical ToolBilingual ComparisonBilingual PerformanceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

INTRODUCTION: Oncotype DX® (ODX) Assay is a valuable prognostic and predictive tool in ER+, Her2- invasive breast cancer (IBC). Initially tested and validated in lymph node (LN) negative patients, the indications of this test have been expanded to include patients with limited LN-positive disease. Several prediction systems have been developed to predict the ODX Recurrence Score (RS) with substantial performance in predicting a high vs low RS score but with low performance when predicting the 3 classes that compose the standard ODX. Additionally, many of these prediction systems have not been developed and/or tested for a population of LN+ patients. OBJECTIVES: The primary objective of this study was to evaluate the performance of several previously published ODX RS predictive systems in a population of LN+ patients. Furthermore, we developed a machine-learning based classification system to accurately predict the ODX 3 category RS for this specific population. METHODS: We conducted a retrospective search of Stanford's pathology database for all patients with LN+ IBC diagnosed between January 2013 and December 2017 with an ODX RS available. A total of 119 patients were identified for inclusion in this cohort. Our multivariate pathologic feature-based discriminatory model aimed to classify each case as belonging to the low, intermediate or high ODX RS category. We performed model validation by the 10-fold cross validation (10F-CV) method. The model's performance was assessed by comparing simple accuracy, balanced accuracy, F1 score (harmonic average of the precision and recall) and several concordance classification metrics. RESULTS: Of the evaluated methods, Magee equations performed well in this population of LN+ patients with the modified Magee equation 2 displaying the best accuracy (70.9%) which was surprisingly better than originally reported (55.8%, in Klein et al. Mod Path. 2013). After an initial screen of methods and tuning of the best performing model, our model achieved an overall accuracy of 78.1% on 10F-CV with a 79.1 % balanced accuracy and no two-step discordances. This corresponded to an increase of weighted Cohen's kappa of 30% versus the best performing Magee equation in this cohort and an increase of 103% versus the modified Magee 1 equation (which uses the same features as our model except for tumor grade). DISCUSSION: Classifiers aimed at providing an alternative to Oncotype DX testing are available and perform consistently across datasets. We are currently validating our approach in a population of 1000 LN-negative patients from the MSKCC and the SEER database. Due to the substantial performance of our machine learning-based classifier based on standard reported pathologic features, our model may be considered an alternative to the ODX standard testing or a screening method for ODX testing, especially for cases with where the cost and availability of the ODX test are a concern. Citation Format: Beca F, Yang S-R, Gruber JG, Barry-Holson K, West R, Wen HY, Allison KH. Development of a machine learning-based classifier for Oncotype DX® category prediction in a population of lymph node positive breast carcinoma patients [abstract]. In: Proceedings of the 2018 San Antonio Breast Cancer Symposium; 2018 Dec 4-8; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2019;79(4 Suppl):Abstract nr P2-07-07.

Question Answering (QA) Systems are put forward as a real alternative to Information Retrieval systems as they provide the user with a fast and comprehensible answer to his or her information need. It has been 15 years since TREC introduced the first QA track. The principal campaigns in the evaluation of Information Retrieval have been specific tracks focusing on the development and evaluation of this type of system. This study is a brief review of the TREC, CLEF and NTCIR Conferences from the QA perspective. We present a historical overview of 15 years of QA evaluation tracks using the method of systematic review. We have examined identified the different tasks or specific labs created in each QA track, the types of evaluation question used, as well as the evaluation measures used in the different competitions analyzed. Of the conferences, it is CLEF that has applied the greater variety of types of test question (factoid, definition, list, causal, yes/no, amongst others). NTCIR, held on 13 occasions, is the conference which has made use of a greater number of different evaluation measures. Accuracy, precision and recall have been the three most used evaluation measures in the three campaigns.

XML is increasingly used to mark up content in present day information repositories. Over the last decade or so, retrieval from XML document collections has emerged as an area of active research. For the Information Retrieval community, XML retrieval poses a two-fold problem: finding effective techniques to retrieve appropriate or the most useful XML elements in response to a user query; and devising an appropriate evaluation methodology to measure the effectivity of such retrieval techniques. . This study examines both these issues. First, we revisited the pivoted length normalization scheme in the Vector Space Model using standard benchmark collections for XML retrieval. We reduced two parameters used in pivoted length normalization to a single combined parameter and experimentally found its optimum value, which works well at both the element and document levels for XML retrieval. We observed that a substantial number of focused queries used in XML retrieval clearly state, besides the information need, what the user does not want. We demonstrated that this negative information, if not handled properly, degrades retrieval performance. We proposed a solution for automatically removing negative information from XML queries. This led to significant improvements in retrieval results. On the evaluation of XML retrieval, we first studied the sensitivity & robustness of various evaluation metrics and reliability and reusability of the assessment pool that has been used at INEX Ad Hoc track since 2007. Specifically we investigated the behaviour of the metrics when assessments are incomplete, or when query sets are small. We observed that early precision metrics are more error-prone and less stable under both these conditions. Average metric, however, performs comparatively better in this respect. System rankings remain largely unaffected even when assessment effort is substantially (but systematically) reduced. We also found that the INEX collections remain usable when evaluating non-participating systems. For a fixed amount of assessment effort, judging shallow pools for many queries is found to be better than judging deep pools for a smaller set of queries. However, when judging only a random sample of a pool, it is better to completely judge fewer topics than to partially judge many topics. We also proposed a simple and pragmatic approach of creating assessment pool for evaluation of retrieval systems. Instead of using an apriori-fixed pool depth for all topics, the pool is incrementally built and judged interactively on a per query basis. When no new relevant document is found for a reasonably long run of pool-depths, pooling was stopped for the topic. Our proposed approach offers a trade-off between the available effort and required level of performance. Moreover, it is flexible to deep pooling for potential topics in order to ensure better estimate of recall. We demonstrated the effectivity of the technique by substantially reducing the assessment effort without seriously compromising on the reliability of evaluation. The approach provided good results in the evaluation of XML retrieval as well as traditional document retrieval. This doctoral work was done at and submitted to the Indian Statistical Institute, Kolkata, under the supervision of Dr. Mandar Mitra. Dr. Ellen Voorhees and Dr. Tetsuya Sakai served as reviewers; Dr. Soumen Chakrabarti was the examiner at the defence of the thesis. Available online at http://www.isical.ac.in/?mandar/sukomal-thesis.pdf.