Rank aggregation algorithms for fair consensus

ABSTRACTIn many different applications of group decision-making, individual ranking agents or judges are able to rank only a small subset of all available candidates. However, as we argue in this article, the aggregation of these incomplete ordinal rankings into a group consensus has not been adequately addressed. We propose an axiomatic method to aggregate a set of incomplete rankings into a consensus ranking; the method is a generalization of an existing approach to aggregate complete rankings. More specifically, we introduce a set of natural axioms that must be satisfied by a distance between two incomplete rankings; prove the uniqueness and existence of a distance satisfying such axioms; formulate the aggregation of incomplete rankings as an optimization problem; propose and test a specific algorithm to solve a variation of this problem where the consensus ranking does not contain ties; and show that the consensus ranking obtained by our axiomatic approach is more intuitive than the consensus ranking obtained by other approaches.

The problem of aggregating individual rankings to create an overall consensus ranking representative of the group is of longstanding interest in group decision making. The problem arises in situations where a group of k Decision Makers (DMs) are asked to rank order n alternatives. The question is how to combine the DMs' rankings into one consensus ranking. Several different approaches have been suggested to aggregate DM responses into a compromise or consensus ranking, however, none is generally recognised as being the best and the similarity of consensus rankings generated by these algorithms is largely unknown. In this paper, we propose a new Weighted-sum ordinal Consensus ranking Method (WCM) with the weights derived from a Sigmoid function. We run Monte Carlo simulation across a range of k and n to compare the similarity of the consensus rankings generated by our method with the best-known method of Borda-Kendall (BAK; Kendall, M. (1962) Rank correlation methods. New York, NY: Hafner) and two other commonly used techniques proposed by Beck, M.P. and Lin, B.W. (1983) 'Some heuristics for the consensus ranking problem', Computers and Operations Research, Vol. 10,

The problem of interpreting or aggregating multiple rankings is common to many real-world applications. Perhaps the simplest and most common approach is a weighted rank aggregation, wherein a (convex) weight is applied to each input ranking and then ordered. This paper describes a new tool for visualizing and displaying ranking information for the weighted rank aggregation method. Traditionally, the aim of rank aggregation is to summarize the information from the input rankings and provide one final ranking that hopefully represents a more accurate or truthful result than any one input ranking. While such an aggregated ranking is, and clearly has been, useful to many applications, it also obscures information. In this paper, we show the wealth of information that is available for the weighted rank aggregation problem due to its structure. We apply weight set decomposition to the set of convex multipliers, study the properties useful for understanding this decomposition, and visualize the indifference regions. This methodology reveals information–that is otherwise collapsed by the aggregated ranking–into a useful, interpretable, and intuitive decision support tool. Included are multiple illustrative examples, along with heuristic and exact algorithms for computing the weight set decomposition.

We propose a novel method for group decision making. It is applicable in the context of multiple criteria ranking problems, where alternatives need to be ordered from the best to the worst by multiple Decision Makers (DMs). In the first stage, incomplete preference information of each DM is analyzed within the framework of stochastic analysis. Specifically, the Monte Carlo simulation is applied for exploiting the space of preference model parameters compatible with each DM’s preferences. In this way, we estimate the values of stochastic acceptability indices that quantify the support given to the preference, indifference, and incomparability relations for each pair of alternatives. In the second stage, such stochastic rankings are aggregated into a group compromise recommendation that minimizes either an average or a maximal distance from each DM’s input. Apart from accounting for the utilitarian and egalitarian perspectives, the dedicated mathematical programming models deal with the processing and constructing of complete or partial rankings. The proposed method is coupled with the robust variants of PROMETHEE I and II methods, however, it can be combined with any method from the broad family of Stochastic Multicriteria Acceptability Analysis (SMAA) techniques. Its applicability for supporting real-world group decision making is demonstrated in an illustrative case study concerning the ranking of project proposals by a research funding agency.KeywordsMultiple criteria rankingRank aggregationStochastic acceptability indexRobustness analysisBinary linear programmingPROMETHEE

Normally, statistical methods are used to generate rankings for genes in terms of their ability to distinguish between normal and malignant tumors from a gene expression dataset. However, different statistical methods yield different ranks for same gene and there is no universally accepted method for ranking. Therefore rank aggregation is required to find the overall ranking of the set of genes. There are various rank aggregation methods in the existing literature to integrate the rankings produced by various statistical tests. Moreover, the problem of integration of some partial rankings, containing unequal numbers of genes, is more challenging. In this article, a multiobjective genetic algorithm based rank aggregation method is proposed to integrate some partial rankings in an unbiased way. The first objective is to minimize the total distance from the reference ranking to the input rankings. For distance calculation, the Scaled Footrule Distance is used. The second objective is to minimize the standard deviation among those distances in order to avoid bias toward a particular input ranking. The proposed method is applied on some real-life microarray gene expression datasets, and the performance of it is compared with that of several existing rank aggregation techniques with respect to accuracy and the AUC (Area under ROC curve) value. Again, for real-life datasets, accuracy is plotted for visual comparison.

Rank aggregation is a popular problem that combines different ranked lists from various sources (frequently called voters or judges), and generates a single aggregated list with improved ranking of its items. In this context, a portion of the existing methods attempt to address the problem by treating all voters equally. Nevertheless, several related works proved that the careful and effective assignment of different weights to each voter leads to enhanced performance. In this article, we introduce an unsupervised algorithm for learning the weights of the voters for a specific topic or query. The proposed method is based on the fact that if a voter has submitted numerous elements which have been placed in high positions in the aggregated list, then this voter should be treated as an expert, compared to the voters whose suggestions appear in lower places or do not appear at all. The algorithm iteratively computes the distance of each input list with the aggregated list and modifies the weights of the voters until all weights converge. The effectiveness of the proposed method is experimentally demonstrated by aggregating input lists from six TREC conferences.CCS CONCEPTS• Information systems → Rank aggregation; •Theory of computation → Unsupervised learning and clustering.

This paper presents novel approaches for combining re-ranking and rank aggregation methods aiming at improving the effectiveness of Content-Based Image Retrieval (CBIR) systems. Given a query image as input, CBIR systems retrieve the most similar images in a collection by taking into account image visual properties. In this scenario, accurately ranking collection images is of great relevance. Aiming at improving the effectiveness of CBIR systems, re-ranking and rank aggregation algorithms have been proposed. However, different re-ranking and rank aggregation approaches, applied to different image descriptors, may produce different and complementary image rankings. In this paper, we present four novel approaches for combining these rankings aiming at obtaining more effective results. Several experiments were conducted involving shape, color, and texture descriptors. The proposed approaches are also evaluated on multimodal retrieval tasks, considering visual and textual descriptors. Experimental results demonstrate that our approaches can improve significantly the effectiveness of image retrieval systems.

High-centrality nodes have disproportionate influence on the behavior of a network; therefore controlling such nodes can efficiently steer the system to a desired state. Existing multiplex centrality measures typically rank nodes assuming the layers are qualitatively similar. Many real systems, however, are comprised of networks heterogeneous in nature, for example, social networks may have both agnostic and affiliative layers. Here, we use rank aggregation methods to identify intervention targets in multiplex networks when the structure, the dynamics, and our intervention goals are qualitatively different for each layer. Our approach is to rank the nodes separately in each layer considering their different function and desired outcome, and then we use Borda count or Kemeny aggregation to identify a consensus ranking—top nodes in the consensus ranking are expected to effectively balance the competing goals simultaneously among all layers. To demonstrate the effectiveness of consensus ranking, we apply our method to a degree-based node removal procedure such that we aim to destroy the largest component in some layers, while maintaining large-scale connectivity in others. For any multi-objective intervention, optimal targets only exist in the Pareto-sense; we, therefore, use a weighted generalization of consensus ranking to investigate the trade-off between the competing objectives. We use a collection of model and real networks to systematically investigate how this trade-off is affected by multiplex network structure. We use the copula representation of the multiplex centrality distributions to generate model multiplex networks with given rank correlations. This allows us to separately manipulate the marginal centrality distribution of each layer and the interdependence between the layers, and to investigate the role of the two using both analytical and numerical methods.

Rank aggregation is the important task in many areas, and different rank aggregation algorithms are created to find optimal rank. Nevertheless, none of these algorithms is the best for all cases. The main goal of this work is to develop a method, which for each rank list defines, which rank aggregation algorithm is the best for this rank list. Canberra distance is used as a metric for determining the optimal ranking. Three approaches are proposed in this paper and one of them has shown promising result. Also we discuss, how this approach can be applied to learn filtering feature selection algorithm ensemble.

In this work, we study the rank aggregation problem based on election voting model. In addition to classical voting rules: Borda, some elimination voting algorithms, including Kemeny methods, are analyzed in a graph theoretic approach. As the Kemeny ranking problem is NP-hard, a new heuristic elimination voting algorithm is proposed in this work. Some experiments have been carried out on TREC data for evaluating these voting algorithms on rank aggregation. The experiments indicate that these elimination algorithms have comparable performance with Borda algorithms, and sometimes outperform it

In this paper we explore the possibility of deriving consensus rankings by solving consensus optimization problems, characterizing consensus rankings as suitable complete order relations minimizing the average Kemeny-Snell distance to the individual rankings. This optimization problem can be expressed as a binary programming (BP) problem which can typically be solved reasonably efficiently. The underlying theory is discussed in Sect. 1. Applications of the proposed method given in Sect. 2 include a comparison to other mathematical programming (MP) approaches using the data set of Tse [9] and establishing a consensus ranking of marketing journals identified by domain experts from a subset of the Harzing journal quality list [2]. In Sect. 3 we discuss computational details and present the results of a benchmark experiment comparing the performance of the commercial solver CPLEX to three open source mixed integer linear programming (MILP) solvers

Rank aggregation has recently become a common approach for combining individual rankings into a consensus and for quantifying and improving performance in various applications, such as elections, web page rankings, and sports. During the past few years, rankings from many sources have become increasingly high-dimensional and partial. In this study, we develop a rank aggregation method by constructing a directed weighted competition graph. We introduce the concept of “ratio of out- and in-degrees (ROID)” to transform high-dimensional partial rankings into a single consensus. Moreover, we provide a novel effectiveness measure for the aggregate ranking according to its deviations from the ground truth ranking. The proposed method is compared with four typical methods with synthetic rankings. The results indicate that our method outperforms the other four by a significant margin and can be particularly efficient in aggregating high-dimensional rankings. The empirical results validate the effectiveness and feasibility of our method.

Gastrointestinal stromal tumor (GIST) is a relatively rare type of gastrointestinal tract tumors. Thus, it is difficult to perform randomized controlled trials (RCTs) of GIST in a single center, which are often plagued by a small number of participants. Real world study (RWS) is a complement to the evidence derived from traditional RCT. Emerging sources of real world data offer enormous opportunity for deeper understanding of why treatments work (or not) and for whom. Evidence generated from RWD can help clarify best use of treatments for individuals and populations, and care value. Thus, RWS of GIST has attracted much attention. RWS helps us better understand the diagnosis, treatment, prognosis and outcome prediction of GIST in clinical practice. GIST is often misdiagnosed as other tumor. A diagnostic test provides evidence on how well a test correctly identifies or rules out GIST. Therapeutic research aims to evaluate the effectiveness and/or safety of a therapy for GIST. Prognostic research aims to forecast the likely outcome of GIST, explore the factors affecting the outcome, and analyze quality of life. Predictive research aims to quantify the probability of identification or health outcome of GIST based on a set of predictors. Pharmacoeconomic data in real world can evaluate the cost-effectiveness of medicinal products for GISTs and serve as a guidance tool for optimal healthcare resource allocation. Attaching importance to data sourse and data quality, strenthening communicate with a qualified statistician, and the implementation of a standardized process are necessary for performing a high-quality RWS.

Ixazomib-Based Maintenance Therapy Improves Prognosis of Patients with Multiple Myeloma: A Real-World Study

MP71-18 BLUE LIGHT CYSTOSCOPY VERSUS WHITE LIGHT CYSTOSCOPY FOR THE DETECTION OF BLADDER CANCER USING MODERN HD 4K EQUIPMENT: AN ANALYSIS OF PIVOTAL TRIAL AND REAL-WORLD DATA IN CHINA