Unsupervised rank aggregation with distance-based models

Learning the true rank ordering among objects by aggregating a set of expert opinion rank order lists is an important and ubiquitous problem in many applications ranging from social choice theory to recommendation systems and search aggregation. We study the problem of unsupervised rank aggregation where no ground truth ordering information in available, neither about the true preference ordering among any set of objects nor about the quality of individual rank lists. Aggregating the often inconsistent and poor quality rank lists in such an unsupervised manner is a highly challenging problem, and standard consensus-based methods fall short in terms of both quality and scalability. In this manuscript we propose a novel framework to bypass these issues by using object attributes to augment the standard rank aggregation framework. We design algorithms that learn joint models on both rank lists and object features to obtain an aggregated rank ordering that is more accurate and robust, and also helps weed out rank lists of dubious validity. We validate our techniques on synthetic datasets where our algorithm is able to estimate the true rank ordering even when the rank lists are corrupted. Experiments on three real datasets, MQ2008, MQ2007 and OHSUMED, show that using object features can result in significant improvement in performance over existing rank aggregation methods that do not use object information. Furthermore, when at least some of the rank lists are of high quality, our methods are able to effectively exploit such information to output an aggregated rank ordering of high accuracy.

Distance-based tree models for ranking data

Normally, statistical methods are used to generate rankings for genes in terms of their ability to distinguish between normal and malignant tumors from a gene expression dataset. However, different statistical methods yield different ranks for same gene and there is no universally accepted method for ranking. Therefore rank aggregation is required to find the overall ranking of the set of genes. There are various rank aggregation methods in the existing literature to integrate the rankings produced by various statistical tests. Moreover, the problem of integration of some partial rankings, containing unequal numbers of genes, is more challenging. In this article, a multiobjective genetic algorithm based rank aggregation method is proposed to integrate some partial rankings in an unbiased way. The first objective is to minimize the total distance from the reference ranking to the input rankings. For distance calculation, the Scaled Footrule Distance is used. The second objective is to minimize the standard deviation among those distances in order to avoid bias toward a particular input ranking. The proposed method is applied on some real-life microarray gene expression datasets, and the performance of it is compared with that of several existing rank aggregation techniques with respect to accuracy and the AUC (Area under ROC curve) value. Again, for real-life datasets, accuracy is plotted for visual comparison.

BackgroundIn survey studies on health-state valuations, ordinal ranking exercises often are used as precursors to other elicitation methods such as the time trade-off (TTO) or standard gamble, but the ranking data have not been used in deriving cardinal valuations. This study reconsiders the role of ordinal ranks in valuing health and introduces a new approach to estimate interval-scaled valuations based on aggregate ranking data.MethodsAnalyses were undertaken on data from a previously published general population survey study in the United Kingdom that included rankings and TTO values for hypothetical states described using the EQ-5D classification system. The EQ-5D includes five domains (mobility, self-care, usual activities, pain/discomfort and anxiety/depression) with three possible levels on each. Rank data were analysed using a random utility model, operationalized through conditional logit regression. In the statistical model, probabilities of observed rankings were related to the latent utilities of different health states, modeled as a linear function of EQ-5D domain scores, as in previously reported EQ-5D valuation functions. Predicted valuations based on the conditional logit model were compared to observed TTO values for the 42 states in the study and to predictions based on a model estimated directly from the TTO values. Models were evaluated using the intraclass correlation coefficient (ICC) between predictions and mean observations, and the root mean squared error of predictions at the individual level.ResultsAgreement between predicted valuations from the rank model and observed TTO values was very high, with an ICC of 0.97, only marginally lower than for predictions based on the model estimated directly from TTO values (ICC = 0.99). Individual-level errors were also comparable in the two models, with root mean squared errors of 0.503 and 0.496 for the rank-based and TTO-based predictions, respectively.ConclusionsModeling health-state valuations based on ordinal ranks can provide results that are similar to those obtained from more widely analyzed valuation techniques such as the TTO. The information content in aggregate ranking data is not currently exploited to full advantage. The possibility of estimating cardinal valuations from ordinal ranks could also simplify future data collection dramatically and facilitate wider empirical study of health-state valuations in diverse settings and population groups.

Mixtures of weighted distance-based models for ranking data with applications in political studies

Rank aggregation has recently become a common approach for combining individual rankings into a consensus and for quantifying and improving performance in various applications, such as elections, web page rankings, and sports. During the past few years, rankings from many sources have become increasingly high-dimensional and partial. In this study, we develop a rank aggregation method by constructing a directed weighted competition graph. We introduce the concept of “ratio of out- and in-degrees (ROID)” to transform high-dimensional partial rankings into a single consensus. Moreover, we provide a novel effectiveness measure for the aggregate ranking according to its deviations from the ground truth ranking. The proposed method is compared with four typical methods with synthetic rankings. The results indicate that our method outperforms the other four by a significant margin and can be particularly efficient in aggregating high-dimensional rankings. The empirical results validate the effectiveness and feasibility of our method.

Purpose Overtourism has been identified as a significant global problem with numbers of negative externalities. The purpose of this paper is to contribute to the establishment of a standard that objectively measures the extent of tourism to produce a dynamic ranking of selected European settlements, as there is a lack of studies using sophisticated statistical methods to analyse secondary data on overtourism. Design/methodology/approach The 28 selected sites, studied according to their involvement in overtourism, were ranked using multi-criteria decision-Mmaking Methods between 2014 and 2023. Rankings were calculated by VIKOR, TOPSIS and MMOORA, and an aggregate ranking was created by using the cross-entropy optimization. Additionally, the annual changes in the rankings were presented graphically using principal component analysis on a two-dimensional space, referred to as the “sites' space.” Finally, the sites were clustered into three distinct groups based on their level of overtourism: less, medium and more. This was achieved through the use of the K-means algorithm. Findings Following the onset of the pandemic in 2020, there was a notable decline in overtourism. However, following the year 2021, the numbers began to rise once more, and by 2022, they had returned to their pre-pandemic levels. Among the methods, MMOORA demonstrated the most effective performance in comparison to the optimal ranking. Originality/value The applied methods with novel rank aggregation could also shed light on the most relevant indicators to overtourism. These included the number of nights spent in paid accommodation per population, the number of overnight visits per population and the number of air passengers.

We propose the use of probability models for ranked data as a useful alternative to a quantitative data analysis to investigate the outcome of bioassay experiments when the preliminary choice of an appropriate normalization method for the raw numerical responses is difficult or subject to criticism. We review standard distance-based and multistage ranking models and propose an original generalization of the Plackett-Luce model to account for the order of the ranking elicitation process. The usefulness of the novel model is illustrated with its maximum likelihood estimation for a real data set. Specifically, we address the heterogeneous nature of the experimental units via model-based clustering and detail the necessary steps for a successful likelihood maximization through a hybrid version of the expectation-maximization algorithm. The performance of the mixture model using the new distribution as mixture components is then compared with alternative mixture models for random rankings. A discussion on the interpretation of the identified clusters and a comparison with more standard quantitative approaches are finally provided.

Prospects and Limitations of Partial Ranked Data Analysis in Social Science Research – Investigations of Social Networks and Support

ABSTRACTIn many different applications of group decision-making, individual ranking agents or judges are able to rank only a small subset of all available candidates. However, as we argue in this article, the aggregation of these incomplete ordinal rankings into a group consensus has not been adequately addressed. We propose an axiomatic method to aggregate a set of incomplete rankings into a consensus ranking; the method is a generalization of an existing approach to aggregate complete rankings. More specifically, we introduce a set of natural axioms that must be satisfied by a distance between two incomplete rankings; prove the uniqueness and existence of a distance satisfying such axioms; formulate the aggregation of incomplete rankings as an optimization problem; propose and test a specific algorithm to solve a variation of this problem where the consensus ranking does not contain ties; and show that the consensus ranking obtained by our axiomatic approach is more intuitive than the consensus ranking obtained by other approaches.

We propose a novel method for group decision making. It is applicable in the context of multiple criteria ranking problems, where alternatives need to be ordered from the best to the worst by multiple Decision Makers (DMs). In the first stage, incomplete preference information of each DM is analyzed within the framework of stochastic analysis. Specifically, the Monte Carlo simulation is applied for exploiting the space of preference model parameters compatible with each DM’s preferences. In this way, we estimate the values of stochastic acceptability indices that quantify the support given to the preference, indifference, and incomparability relations for each pair of alternatives. In the second stage, such stochastic rankings are aggregated into a group compromise recommendation that minimizes either an average or a maximal distance from each DM’s input. Apart from accounting for the utilitarian and egalitarian perspectives, the dedicated mathematical programming models deal with the processing and constructing of complete or partial rankings. The proposed method is coupled with the robust variants of PROMETHEE I and II methods, however, it can be combined with any method from the broad family of Stochastic Multicriteria Acceptability Analysis (SMAA) techniques. Its applicability for supporting real-world group decision making is demonstrated in an illustrative case study concerning the ranking of project proposals by a research funding agency.KeywordsMultiple criteria rankingRank aggregationStochastic acceptability indexRobustness analysisBinary linear programmingPROMETHEE

The use of probabilistic dose and risk analysis will become widespread in radiological assessments because of the insights it provides and because of the availability of fast personal computers. Although the probabilistic analysis for each site will be unique because of the probabilistic inputs selected, their site-specific distributions, and any interrelationship between the inputs, the insights gained during the participation of the RESRAD-OFFSITE computer code in the Biosphere Modelling and Assessment and Biospheric Model Validation probabilistic studies will be useful guides in performing other probabilistic dose and risk analyses. Correlation and regression analysis can be performed on raw or ranked data, and the appropriate analysis can be chosen on the basis of the coefficient of determination. Partial (rank) correlation coefficients and standardized (rank) regression coefficients can be used to identify probabilistic inputs that have a linear (or monotonic) effect on probabilistic outputs and to quantify these effects. A suite of scatter plots can be used to confirm this identification and to also identify the inputs that have nonlinear, nonmonotonic effects on probabilistic outputs.

For a given query and a set of images ranked lists retrieved from multiple search engines, the metasearch technique aims at combining these lists to build an unified ranking with improved relevance. Rank aggregation is an approach that has been widely used to support this task. This paper investigates the use of rank aggregation methods in the metasearch scenario for diverse image retrieval. Although metasearch systems are usually driven by the relevance of the final result, the impact on diversification has also been analyzed. The experimental findings suggest metasearch based on rank aggregation allows significant improvements, both in terms of relevance and diversity.

Distance-based and ad hoc consensus models in ordinal preference ranking

The elicitation of an ordinal judgment on multiple alternatives is often required in many psychological and behavioral experiments to investigate preference/choice orientation of a specific population. The Plackett-Luce model is one of the most popular and frequently applied parametric distributions to analyze rankings of a finite set of items. The present work introduces a Bayesian finite mixture of Plackett-Luce models to account for unobserved sample heterogeneity of partially ranked data. We describe an efficient way to incorporate the latent group structure in the data augmentation approach and the derivation of existing maximum likelihood procedures as special instances of the proposed Bayesian method. Inference can be conducted with the combination of the Expectation-Maximization algorithm for maximum a posteriori estimation and the Gibbs sampling iterative procedure. We additionally investigate several Bayesian criteria for selecting the optimal mixture configuration and describe diagnostic tools for assessing the fitness of ranking distributions conditionally and unconditionally on the number of ranked items. The utility of the novel Bayesian parametric Plackett-Luce mixture for characterizing sample heterogeneity is illustrated with several applications to simulated and real preference ranked data. We compare our method with the frequentist approach and a Bayesian nonparametric mixture model both assuming the Plackett-Luce model as a mixture component. Our analysis on real datasets reveals the importance of an accurate diagnostic check for an appropriate in-depth understanding of the heterogenous nature of the partial ranking data.