• Home
  • Search
  • Are We Wasting Time? A Fast, Accurate Performance Evaluation Framework for Knowledge Graph Link Predictors
  • https://doi.org/10.1109/icde65448.2025.00127Copy DOI Icon

Are We Wasting Time? A Fast, Accurate Performance Evaluation Framework for Knowledge Graph Link Predictors

  • May 19, 2025
  • Filip Cornell +3 more
Show More
  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

The standard evaluation protocol for measuring the quality of Knowledge Graph Completion methods - the task of inferring new links to be added to a graph - typically involves a step which ranks every entity of a Knowledge Graph to assess their fit as a head or tail of a candidate link to be added. In Knowledge Graphs on a larger scale, this task rapidly becomes prohibitively heavy. Previous approaches mitigate this problem by using random sampling of entities to assess the quality of links predicted or suggested by a method. However, we show that this approach has serious limitations since the ranking metrics produced do not properly reflect true outcomes. In this paper, we present a thorough analysis of these effects along with the following findings. First, we empirically find and theoretically motivate why sampling uniformly at random vastly overestimates the ranking performance of a method. We show that this can be attributed to the effect of easy versus hard negatives. Second, we propose a framework that uses relational recommenders to guide the selection of candidates for evaluation. We provide both theoretical and empirical justification of our methodology, and find that simple and fast methods work extremely well, matching advanced neural approaches. Even when a large portion of the true candidates for a property are missed, the estimation of the ranking metrics on a downstream model barely deteriorates. With our proposed framework, we can reduce the time and computation needed similar to random sampling strategies while vastly improving the estimation; on ogbl-wikikg2, we show that accurate estimations of the full ranking can be obtained in 20 seconds instead of 30 minutes. We conclude that considerable computational effort can be saved by effective preprocessing and sampling methods and still reliably predict performance accurately of the true performance for the entire ranking procedure. We make our code available to the community<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>Accessible at https://github.com/Filco306/are-we-wasting-time.

Similar Papers
  • Research Article
  • Citations12

Logical Effort Framework for CNFET-Based VLSI Circuits for Delay and Area Optimization

  • Mar 01, 2019
  • IEEE Transactions on Very Large Scale Integration (VLSI) Systems
  • Muhammad Ali +2
  • Research Article
  • Citations2

Measurement insights and error analysis of electronic parameters for ultrasonic transducers

  • Jun 24, 2024
  • Measurement Science and Technology
  • Yuhua Zhou +7
  • Research Article

CO Cryo‐Sorption as a Surface‐Sensitive Spectroscopic Probe of the Active Site Density of Single‐Atom Catalysts

  • Feb 10, 2025
  • Angewandte Chemie
  • Beomgyun Jeong +13
  • Research Article
  • Citations2

Performance Evaluation of UFMC System using Neural Networks for BER Prediction and Efficiency Optimization

  • Nov 20, 2024
  • International Research Journal of Multidisciplinary Technovation
  • Kiran Kumar Padakanti +3
  • Research Article
  • Citations51

A knowledge-based system for overall supply chain performance evaluation: a multi-criteria decision making approach

  • May 07, 2019
  • Supply Chain Management: An International Journal
  • Sharfuddin Ahmed Khan +2
  • Single Report
  • Citations1

Improved in situ performance testing of line-concentrating solar collectors: Comprehensive uncertainty analysis for the selection of measurement instrumentation

  • Oct 01, 2016
  • Annie Zirkel-Hofer +6
  • Conference Article
  • Citations6

Accurate Performance Evaluation, Modelling and Prediction of a Message Passing Simulation Code based on Middleware

  • Jan 01, 1998
  • M Taufer +1
  • Research Article
  • Citations7

A Performance Evaluation Method Based on Interval Evidential Reasoning Approach With Function Monotonicity

  • Jan 01, 2022
  • IEEE Transactions on Instrumentation and Measurement
  • Lei-Yu Chen +4
  • Book Chapter
  • Citations1

Analyzing and Evaluating Current Computer Networks Simulation Models

  • Jan 01, 2012
  • Jafar Ababneh +3
  • Single Report

Combined modeling and experiments to predict corrosion and embrittlement in dual-phase stainless steels within the MARMOT framework

  • Dec 28, 2021
  • Julie Tucker +5
  • Research Article
  • Citations23

Appropriate level of development of in-situ building information modeling for existing building energy modeling implementation

  • Mar 05, 2023
  • Journal of Building Engineering
  • Dong Eun Jung +6
  • Research Article
  • Citations598

A framework for evaluation of deformable image registration spatial accuracy using large landmark point sets

  • Mar 05, 2009
  • Physics in Medicine & Biology
  • Richard Castillo +6
  • Research Article

Estimation of photovoltaic model parameters using the Rao-1 optimization algorithm

  • Dec 07, 2024
  • Caderno Pedagógico
  • Fathi Brioua +3
  • Research Article
  • Citations74

Unveiling the quantification minefield in electrocatalytic urea synthesis

  • Oct 18, 2022
  • Chemical Engineering Journal
  • Yanmei Huang +7
  • Research Article
  • Citations7

A Novel Approach for Evaluating TFT–LCD Manufacturer Operational Performance

  • May 01, 2016
  • Journal of Testing and Evaluation
  • Chia-Chi Sun
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.