New Quality Metrics for Web Search Results

Evaluating 5W1H (Who, What, When, Where, Why, and How) information extraction systems remains challenging, as traditional information retrieval metrics like ROUGE and BLEU fail to capture semantic accuracy and narrative coherence. The LLM-as-a-Judge paradigm offers a promising alternative, yet systematic comparisons of judge models for this task are lacking. This study benchmarks multiple large language models, including state-of-the-art models such as GPT, Claude, and Gemini as evaluators of 5W1H extractions from Spanish news articles. We assess judge performance across six quality criteria: Factual Accuracy, Completeness, Relevance and Conciseness, Clarity and Readability, Faithfulness to Source, and Overall Coherence. Our analysis examines inter-judge agreement, score distribution patterns, criterion-level variance, and the relationship between evaluation quality and computational cost. Using two Spanish-language corpora (BASSE and FLARES), we identify which criteria exhibit consistent cross-model agreement and which prove most sensitive to judge selection. The main contribution of this work is providing the first systematic benchmark of LLM-as-a-Judge models for 5W1H extraction evaluation in Spanish, validated against expert journalistic judgment. Results reveal that all evaluated models achieve alignment levels above 90% across all metrics. Specifically, Claude Sonnet 4.5 emerges as the most accurate evaluator with a Global Judgment Acceptance Rate (JAR) of 99.79%. Furthermore, meta-evaluation with human experts demonstrates a substantial inter-annotator agreement of κ=0.6739. Finally, we provide recommendations for judge model selection based on task requirements and resource constraints, contributing practical guidance for researchers implementing LLM-based evaluation pipelines for information extraction tasks.

Keyphrase extraction is a task with many applications in information retrieval, text mining, and natural language processing. In this paper, a keyphrase extraction approach based on the naive Bayesian classifier is proposed. To determine whether a phrase is a keyphrase, the following features of a phrase in a given document are adopted: its term frequency, whether to appear in the title, abstract and headings (subheadings), and its frequency appearing in the paragraphs of the given document. The approach is evaluated by the standard information retrieval metrics of precision and recall. Experiment results show that this approach is very practical: it can achieve high precision and recall; especially the recall it can achieve is over 80 percent.

Variation in Results of Three Biology‐Focused Search Engines: A Case Study Using North American Tree Species

With the emerging needs of creating fairness‐aware solutions for search and recommendation systems, a daunting challenge exists of evaluating such solutions. While many of the traditional information retrieval (IR) metrics can capture the relevance, diversity, and novelty for the utility with respect to users, they are not suitable for inferring whether the presented results are fair from the perspective of responsible information exposure. On the other hand, existing fairness metrics do not account for user utility or do not measure it adequately. To address this problem, we propose a new metric called FAIR. By unifying standard IR metrics and fairness measures into an integrated metric, this metric offers a new perspective for evaluating fairness‐aware ranking results. Based on this metric, we developed an effective ranking algorithm that jointly optimized user utility and fairness. The experimental results showed that our FAIR metric could highlight results with good user utility and fair information exposure. We showed how FAIR related to a set of existing utility and fairness metrics and demonstrated the effectiveness of our FAIR‐based algorithm. We believe our work opens up a new direction of pursuing a metric for evaluating and implementing the FAIR systems.

No abstract available.

What's the Story

Due to the dynamic nature of web and the complex architectures of modern commercial search engines, top results in major search engines can change dramatically over time. Our experimental data shows that, for all three major search engines (Google, Bing and Yahoo!), approximately 90% of queries have their top 10 results altered within a period of ten days. Although this instability is expected in some situations such as in news-related queries, it is problematic in general because it can dramatically affect retrieval performance measurements and negatively affect users' perception of search quality (for instance, when users cannot re-find a previously found document). In this work we present the first large scale study on the degree and nature of these changes. We introduce several types of query instability, and several metrics to quantify it.We then present a quantitative analysis using 12,600 queries collected from a commercial web search engine over several weeks. Our analysis shows that the results from all major search engines have similar levels of instability, and that many of these changes are temporary. We also identified classes of queries with clearly different instability profiles - for instance, navigational queries are considerably more stable than non-navigational, while longer queries are significantly less stable than shorter ones.

In 2012, Sogou, a major Chinese web search engine released a large-scale query log containing 43.5M user interactions, including submitted queries and clicked web page search results. This query log offers a deep sample of queries over a two day period from 30th December 2011 to 1st January 2012. In August 2013, we identified 1.4M predominantly Chinese language unique search result URLs that were clicked at least three times in this query log. We crawled the HTML content of these URLs to construct the supplementary SOGOU-2012-CRAWL dataset, which we release in this work. A real large-scale query log with accompanying crawl such as this offers several opportunities for reproducible information retrieval (IR) research, including query classification, intent modelling and indexing strategy. In this paper we first detail the query log and crawl dataset construction and characteristics. Following this, to demonstrate potential applications we use the crawl to indicatively analyse various time-based patterns in web content and search behaviour. In particular, we study the distribution of language-independent date expressions in the crawled web content. Based on this, we propose a simple approach for modelling the past/present/future temporal intent of queries based on the date the query was submitted by the user, and the dates appearing in the clicked search results. We observe several prominent temporal patterns which may lead to novel time-aware IR approaches.

In most major search engines, the interface for image search is the same as traditional Web search: a keyword query followed by a paginated, ranked list of results. Although many image search innovations have appeared in both the literature and on the Web, few have seen widespread use in practice. In this work, we explore the differences between image and general Web search to better support users’ needs. First, we describe some unique characteristics of image search derived through informal interviews with researchers, designers, and managers responsible for building and deploying a major Web search engine. Then, we present results from a large scale analysis of image and Web search logs showing the differences in user behaviour. Grounded in these observations, we present design recommendations for an image search engine supportive of the unique experience of image search. We iterate on a number of designs, and describe a functional prototype that we built.

BackgroundAcute pancreatitis is a prevalent condition in China. A plethora of information pertaining to acute pancreatitis is readily available on the Internet, including on major search engines and various short video applications. The objective of this study was to evaluate and compare the quality, content completeness, and accuracy of information related to acute pancreatitis on web pages of the four major search engines and short videos on the two major short video platforms.Materials and methodsA search was conducted on four major search engines (Bing, Baidu, Haosou, and Sougou) and two short video apps (TikTok and BiliBili) using the Chinese keyword “acute pancreatitis.” The sources can be divided into four categories: scientific resources, news/media reports, industrial/commercial profit organizations, and healthcare providers. The quality of the web pages and short videos was evaluated using the DISCERN instrument, the Global Quality Score (GQS), and the Journal of the American Medical Association (JAMA). In accordance with established guidelines and reviews of acute pancreatitis, two lists and scoring systems were devised for the evaluation of content comprehensiveness and accuracy.ResultsA total of 120 unique web pages and 120 unique videos were identified using four search engines and two short video applications, respectively. The most prevalent identity among those producing short videos was that of healthcare providers. With regard to the source of the web pages, the most prevalent category was that of industrial/commercial profit organizations. The median DISCERN total score, median GQS score, and median JAMA score were 26, 3, and 2, respectively. Web pages exhibited significantly higher ratings than short videos (p < 0.001, p = 0.003, and p < 0.001). The median overall content score for the web page was 8 (interquartile range [IQR] 4–13), and the median guideline-related score was 2 (IQR 0–4), indicating that the web page’s content was deficient in terms of completeness and accuracy. In comparison, the performance of the short video was found to be even more deficient.ConclusionIn China, most web pages pertaining to acute pancreatitis were produced by industrial/commercial profit organizations, but the quality of the information provided by these entities was found to be the lowest. The majority of short videos were created by healthcare providers, but the overall quality of these videos was found to be inadequate. In general, the quality of both web pages and short videos is suboptimal. Nevertheless, the quality of web pages was found to be superior to that of short videos. In terms of completeness and accuracy, both the web page and the video exhibit significant deficiencies that are cause for concern.

General purpose Web search engines can be a valuable tool to locate high quality information and can complement the use of specialized systems, such as LexisNexis or WestLaw. The effective use of Web search engines differs somewhat from that of other information retrieval systems and requires an understanding of search engine design and procedures before effective use. We present an overview of search engines and user behavior in Web searching as an introduction to a discussion of the effectiveness of using some of the advanced searching techniques common in other searching environments in Web searches. We empirically test the effectiveness on query operators for Web searching. The use of operators such as AND, OR, and PHRASE was found to have little or no difference on the number of relevant results retrieved for three major search engines, America Online, Google, and Microsoft Search.

Googling for Health Information

Certain spans of utterances in a discourse, referred to here as segments, are widely assumed to form coherent units. Further, the segmental structure of discourse has been claimed to constrain and be constrained by many phenomena. However, there is weak consensus on the nature of segments and the criteria for recognizing or generating them. We present quantitative results of a two part study using a corpus of spontaneous, narrative monologues. The first part evaluates the statistical reliability of human segmentation of our corpus, where speaker intention is the segmentation criterion. We then use the subjects' segmentations to evaluate the correlation of discourse segmentation with three linguistic cues (referential noun phrases, cue words, and pauses), using information retrieval metrics.

Accurate estimation of information retrieval evaluation metrics such as average precision require large sets of relevance judgments. Building sets large enough for evaluation of real-world implementations is at best inefficient, at worst infeasible. In this work we link evaluation with test collection construction to gain an understanding of the minimal judging effort that must be done to have high confidence in the outcome of an evaluation. A new way of looking at average precision leads to a natural algorithm for selecting documents to judge and allows us to estimate the degree of confidence by defining a distribution over possible document judgments. A study with annotators shows that this method can be used by a small group of researchers to rank a set of systems in under three hours with 95% confidence. Information retrieval metrics such as average precision require large sets of relevance judgments to be accurately estimated. Building these sets is infeasible and often inefficient for many real-world retrieval implementations. We present a new way of looking at average precision that allows us to estimate the confidence in an evaluation based on the size of the test collection. We use this to build an algorithm for selecting the best documents to judge to have maximum confidence in an evaluation with a minimal number of relevance judgments. A study with annotators shows how the algorithm can be used by a small group of researchers to quickly rank a set of systems with 95% confidence.

Current information systems provide transparent access to multiple, distributed, autonomous and potentially redundant data sources based on a mediation architecture. Their users may not know the sources they questioned, nor their description and content. Consequently, their queries reflect no more a need that must be satisfied but an intention that must be refined based on data sources available at the time of interrogation. The purpose of the personalization is to facilitate the expression of users' needs. It allows them to obtain relevant information by maximizing the exploitation of their preferences grouped in their respective profiles. Our work aims to extend the users queries by extending the research field using ontologies. In a mediation architecture context, founded on the couple mediator-adapter, our process must consider not only the users' profiles but also the semantic description of data sources defined by mediation requests. The mediator solves the problems associated with heterogeneity while adapters describe the available data sources. The users express their requests in terms of a global schema when the system evaluates them over multiple data sources with different structure and content. Each data source is modeled using a local ontology when the global schema is obtained via a global one. The use of an adequate process of ontology alignment will allow us to increase the recall (retrieved information) and the precision or accuracy (relevant information) of our integration system. This article is a comparative study of the existing works that attempt to establish matching and alignment between ontologies. It presents their capabilities in terms of information retrieval metrics namely: precision, recall and F-measure. It highlights then their strength and week points. In addition, it presents the massive role of machine learning techniques to insure the interoperability over large-scale ontologies.