Intention-based segmentation

No abstract available.

With the emerging needs of creating fairness‐aware solutions for search and recommendation systems, a daunting challenge exists of evaluating such solutions. While many of the traditional information retrieval (IR) metrics can capture the relevance, diversity, and novelty for the utility with respect to users, they are not suitable for inferring whether the presented results are fair from the perspective of responsible information exposure. On the other hand, existing fairness metrics do not account for user utility or do not measure it adequately. To address this problem, we propose a new metric called FAIR. By unifying standard IR metrics and fairness measures into an integrated metric, this metric offers a new perspective for evaluating fairness‐aware ranking results. Based on this metric, we developed an effective ranking algorithm that jointly optimized user utility and fairness. The experimental results showed that our FAIR metric could highlight results with good user utility and fair information exposure. We showed how FAIR related to a set of existing utility and fairness metrics and demonstrated the effectiveness of our FAIR‐based algorithm. We believe our work opens up a new direction of pursuing a metric for evaluating and implementing the FAIR systems.

Keyphrase extraction is a task with many applications in information retrieval, text mining, and natural language processing. In this paper, a keyphrase extraction approach based on the naive Bayesian classifier is proposed. To determine whether a phrase is a keyphrase, the following features of a phrase in a given document are adopted: its term frequency, whether to appear in the title, abstract and headings (subheadings), and its frequency appearing in the paragraphs of the given document. The approach is evaluated by the standard information retrieval metrics of precision and recall. Experiment results show that this approach is very practical: it can achieve high precision and recall; especially the recall it can achieve is over 80 percent.

Accurate estimation of information retrieval evaluation metrics such as average precision require large sets of relevance judgments. Building sets large enough for evaluation of real-world implementations is at best inefficient, at worst infeasible. In this work we link evaluation with test collection construction to gain an understanding of the minimal judging effort that must be done to have high confidence in the outcome of an evaluation. A new way of looking at average precision leads to a natural algorithm for selecting documents to judge and allows us to estimate the degree of confidence by defining a distribution over possible document judgments. A study with annotators shows that this method can be used by a small group of researchers to rank a set of systems in under three hours with 95% confidence. Information retrieval metrics such as average precision require large sets of relevance judgments to be accurately estimated. Building these sets is infeasible and often inefficient for many real-world retrieval implementations. We present a new way of looking at average precision that allows us to estimate the confidence in an evaluation based on the size of the test collection. We use this to build an algorithm for selecting the best documents to judge to have maximum confidence in an evaluation with a minimal number of relevance judgments. A study with annotators shows how the algorithm can be used by a small group of researchers to quickly rank a set of systems with 95% confidence.

Current information systems provide transparent access to multiple, distributed, autonomous and potentially redundant data sources based on a mediation architecture. Their users may not know the sources they questioned, nor their description and content. Consequently, their queries reflect no more a need that must be satisfied but an intention that must be refined based on data sources available at the time of interrogation. The purpose of the personalization is to facilitate the expression of users' needs. It allows them to obtain relevant information by maximizing the exploitation of their preferences grouped in their respective profiles. Our work aims to extend the users queries by extending the research field using ontologies. In a mediation architecture context, founded on the couple mediator-adapter, our process must consider not only the users' profiles but also the semantic description of data sources defined by mediation requests. The mediator solves the problems associated with heterogeneity while adapters describe the available data sources. The users express their requests in terms of a global schema when the system evaluates them over multiple data sources with different structure and content. Each data source is modeled using a local ontology when the global schema is obtained via a global one. The use of an adequate process of ontology alignment will allow us to increase the recall (retrieved information) and the precision or accuracy (relevant information) of our integration system. This article is a comparative study of the existing works that attempt to establish matching and alignment between ontologies. It presents their capabilities in terms of information retrieval metrics namely: precision, recall and F-measure. It highlights then their strength and week points. In addition, it presents the massive role of machine learning techniques to insure the interoperability over large-scale ontologies.

Web search results enjoy an increasing importance in our daily lives. But what can be said about their quality, especially when querying a controversial issue? The traditional information retrieval metrics of precision and recall do not provide much insight in the case of web information retrieval. In this paper we examine new ways of evaluating quality in search results: coverage and independence. We give examples on how these new metrics can be calculated and what their values reveal regarding the two major search engines, Google and Yahoo. We have found evidence of low coverage for commercial and medical controversial queries, and high coverage for a political query that is highly contested. Given the fact that search engines are unwilling to tune their search results manually, except in a few cases that have become the source of bad publicity, low coverage and independence reveal the efforts of dedicated groups to manipulate the search results.

Software-Defined Networking (SDN) switches typically have limited ternary content addressable memory (TCAM) that caches the flow entries on the data plane. The scarcity and strong resource competitiveness of TCAM space put the flow tables at the risk of malicious Distributed Denial-of-Service (DDoS) attacks. In this paper, we propose LtRFT, a Learning-To-Rank (LtR) based scheme for mitigating the low-rate DDoS attacks targeted at flow tables. LtRFT consists of three modules: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">monitor</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ranker</i> , and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mitigator</i> . <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Monitor</i> manages the flow table status and sends alerts to other modules after detecting attacks. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Ranker</i> models the attack mitigation problem as a flow entry ranking task, and ranks malicious flows with a high eviction priority using a pairwise-based LtR algorithm. The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mitigator</i> frees up the flow table space by deleting malicious flow entries according to the flow entry ranking sequence generated by <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ranker</i> . We introduce LtR to network attack detection innovatively and use both classification and information retrieval metrics to describe and evaluate LtRFT. Extensive experiments were conducted to validate the effectiveness and robustness of LtRFT in detecting and mitigating the low-rate data plane DDoS attacks. LtRFT can detect malicious attack flows with an accuracy of over 96%, and can reduce the attack flow duration by 97.7% with an average extra latency of 0.5 seconds, which proves that LtRFT is practicable in SDN deployments.

Semantic heterogeneity in XBRL precludes the full automation of the business reporting pipeline, a key motivation for the SEC’s XBRL mandate. To mitigate this problem, several approaches leveraging Semantic Web technologies have emerged. While some approaches are promising, their mapping accuracy in resolving semantic heterogeneity must be improved to realize the promised benefits of XBRL. Considering this limitation and following the design science research methodology (DSRM), we develop a novel framework, XBRL indexing-based mapping (X-IM), which takes advantage of the representational model of representation theory to map heterogeneous XBRL elements across diverse XBRL filings. The application of representation theory to the design process informs the use of XBRL label linkbases as a repository of regularities constitutive of the relationships between financial item names and the concepts they describe along a set of equivalent financial terms of interest to investors. The instantiated design artifact is thoroughly evaluated using standard information retrieval metrics. Our experiments show that X-IM significantly outperforms existing methods.

Evaluating 5W1H (Who, What, When, Where, Why, and How) information extraction systems remains challenging, as traditional information retrieval metrics like ROUGE and BLEU fail to capture semantic accuracy and narrative coherence. The LLM-as-a-Judge paradigm offers a promising alternative, yet systematic comparisons of judge models for this task are lacking. This study benchmarks multiple large language models, including state-of-the-art models such as GPT, Claude, and Gemini as evaluators of 5W1H extractions from Spanish news articles. We assess judge performance across six quality criteria: Factual Accuracy, Completeness, Relevance and Conciseness, Clarity and Readability, Faithfulness to Source, and Overall Coherence. Our analysis examines inter-judge agreement, score distribution patterns, criterion-level variance, and the relationship between evaluation quality and computational cost. Using two Spanish-language corpora (BASSE and FLARES), we identify which criteria exhibit consistent cross-model agreement and which prove most sensitive to judge selection. The main contribution of this work is providing the first systematic benchmark of LLM-as-a-Judge models for 5W1H extraction evaluation in Spanish, validated against expert journalistic judgment. Results reveal that all evaluated models achieve alignment levels above 90% across all metrics. Specifically, Claude Sonnet 4.5 emerges as the most accurate evaluator with a Global Judgment Acceptance Rate (JAR) of 99.79%. Furthermore, meta-evaluation with human experts demonstrates a substantial inter-annotator agreement of κ=0.6739. Finally, we provide recommendations for judge model selection based on task requirements and resource constraints, contributing practical guidance for researchers implementing LLM-based evaluation pipelines for information extraction tasks.

This paper presents an approach to combine automatic semantic place labeling of robot-generated maps with reasoning on human route descriptions. Enabling robots to understand human route descriptions can simplify HRI situations in household or industrial settings. However, solving this problem requires handling the ambiguity present in route descriptions and the possible unreliability of the semantic perception capabilities of the robot. We address this problem by absorbing these uncertainties in a probability distribution measuring the likelihood of the different interpretations (paths) of a given route description and selecting its MAP solution. The approach is evaluated on a dataset of route descriptions transcribed into a suitable representation using standard information retrieval metrics. These performance measurements indicate that the method can correctly interpret route descriptions even in challenging environments.

Social book marking tools are rapidly emerging on the Web as it can be witnessed by the overwhelming number of participants. In such spaces, users annotate resources by means of any keyword or tag that they find relevant, giving raise to lightweight conceptual structures \emph{aka} folksonomies. In this respect, needless to mention that ontologies can be of benefit for enhancing information retrieval metrics. In this paper, we introduce a novel approach for ontology learning from a \textit{folksonomy}, which provide shared vocabularies and semantic relations between tags. The main thrust of the introduced approach stands in putting the focus on the discovery of \textit{non-taxonomic} relationships. The latter are often neglected, even though they are of paramount importance from a semantic point of view. The discovery process heavily relies on triadic concepts to discover and select related tags and to extract and label non-taxonomically relationships between related tags and external sources for tags filtering and non-taxonomic relationships extraction. In addition, we also discuss a new approach to evaluate obtained relations in an automatic way against WordNet repository and presents promising results for a real world \textit{folksonomy}.

In this paper we evaluate the effect of a document sanitization process on a set of information retrieval metrics, in order to measure information loss and risk of disclosure. As an example document set, we use a subset of the Wikileaks Cables, made up of documents relating to five key news items which were revealed by the cables. In order to sanitize the documents we have developed a semi-automatic anonymization process following the guidelines of Executive Order 13526 (2009) of the US Administration, by (i) identifying and anonymizing specific person names and data, and (ii) concept generalization based on WordNet categories, in order to identify words categorized as classified. Finally, we manually revise the text from a contextual point of view to eliminate complete sentences, paragraphs and sections, where necessary. We show that a significant sanitization can be applied, while maintaining the relevance of the documents to the queries corresponding to the five key news items.

Large, heterogeneous datasets are characterized by missing or even erroneous information. This is more evident when they are the product of community effort or automatic fact extraction methods from external sources, such as text. A special case of the aforementioned phenomenon can be seen in knowledge graphs, where this mostly appears in the form of missing or incorrect edges and nodes.Structured querying on such incomplete graphs will result in incomplete sets of answers, even if the correct entities exist in the graph, since one or more edges needed to match the pattern are missing. To overcome this problem, several algorithms for approximate structured query answering have been proposed. Inspired by modern Information Retrieval metrics, these algorithms produce a ranking of all entities in the graph, and their performance is further evaluated based on how high in this ranking the correct answers appear.In this work we take a critical look at this way of evaluation. We argue that performing a ranking-based evaluation is not sufficient to assess methods for complex query answering. To solve this, we introduce Message Passing Query Boxes (MPQB), which takes binary classification metrics back into use and shows the effect this has on the recently proposed query embedding method MPQE.

This paper addresses the transfer of performance between modern sentence transformer models of semantic search with conventional query expansion based on WordNet. Applying TREC data to compare the performance of original queries to those expanded using WordNet on four pretrained models, SBERT, RoBERTa, all-roberta-large-v1, and Paraphrase-MiniLM. All the models are compared based on standard information retrieval metrics. As we demonstrate, recall may be enhanced using WordNet expansion, but in rare instances, precision may decrease. These results indicate that WordNet may provide a valuable context to the text, but its effect relies on the model applied. This effort contributes to an insight into when conventional lexical resources are useful in current transformer-based retrieval systems.

Standard Information Retrieval (IR) metrics are not well suited for new paradigms like XML or Web IR in which retrievable information units are document elements and/or sets of related documents. Part of the problem stems from the classical hypotheses on the user models: They do not take into account the structural or logical context of document elements or the possibility of navigation between units. This article proposes an explicit and formal user model that encompasses a large variety of user behaviors. Based on this model, we extend the probabilistic precision-recall metric to deal with the new IR paradigms.