Comparing Rank Aggregation Methods based on Mallows Model

Normally, statistical methods are used to generate rankings for genes in terms of their ability to distinguish between normal and malignant tumors from a gene expression dataset. However, different statistical methods yield different ranks for same gene and there is no universally accepted method for ranking. Therefore rank aggregation is required to find the overall ranking of the set of genes. There are various rank aggregation methods in the existing literature to integrate the rankings produced by various statistical tests. Moreover, the problem of integration of some partial rankings, containing unequal numbers of genes, is more challenging. In this article, a multiobjective genetic algorithm based rank aggregation method is proposed to integrate some partial rankings in an unbiased way. The first objective is to minimize the total distance from the reference ranking to the input rankings. For distance calculation, the Scaled Footrule Distance is used. The second objective is to minimize the standard deviation among those distances in order to avoid bias toward a particular input ranking. The proposed method is applied on some real-life microarray gene expression datasets, and the performance of it is compared with that of several existing rank aggregation techniques with respect to accuracy and the AUC (Area under ROC curve) value. Again, for real-life datasets, accuracy is plotted for visual comparison.

This paper presents novel approaches for combining re-ranking and rank aggregation methods aiming at improving the effectiveness of Content-Based Image Retrieval (CBIR) systems. Given a query image as input, CBIR systems retrieve the most similar images in a collection by taking into account image visual properties. In this scenario, accurately ranking collection images is of great relevance. Aiming at improving the effectiveness of CBIR systems, re-ranking and rank aggregation algorithms have been proposed. However, different re-ranking and rank aggregation approaches, applied to different image descriptors, may produce different and complementary image rankings. In this paper, we present four novel approaches for combining these rankings aiming at obtaining more effective results. Several experiments were conducted involving shape, color, and texture descriptors. The proposed approaches are also evaluated on multimodal retrieval tasks, considering visual and textual descriptors. Experimental results demonstrate that our approaches can improve significantly the effectiveness of image retrieval systems.

Abstract—Recommender systems are often used to provide useful recommendations for users. They use previous history of the users-items interactions, e.g. purchase history and/or users rating on items, to provide a suitable recommendation list for any target user. They may also use contextual information available about items and users. Collaborative filtering algorithm and its variants are the most successful recommendation algorithms that have been applied to many applications. Collaborative filtering method works by first finding the most similar users (or items) for a target user (or items), and then building the recommendation lists. There is no unique evaluation metric to assess the performance of recommendations systems, and one often choose the one most appropriate for the application in hand. This paper compares the performance of a number of well-known collaborative filtering algorithms on movie recommendation. To this end, a number of performance criteria are used to test the algorithms. The algorithms are ranked for each evaluation metric and a rank aggregation method is used to determine the wining algorithm. Our experiments show that the probabilistic matrix factorization has the top performance in this dataset, followed by item-based and user-based collaborative filtering. Non-negative matrix factorization and Slope 1 has the worst performance among the considered algorithms.
 Keywords—Social networks analysis and mining, big data, recommender systems, collaborative filtering.

Recommender system suggests a top-N list from unseen items for its users through a prediction or a ranking order process. From the recommendation perspective, the item’s order in the generated list is more important than its predicted rating. Moreover, finding the top-N list for a multi-criteria recommendation is a challenging problem as we have many criterions for each item. One can find the average over all criteria; however, this requires a score from each criterion and hence a compensation effect will occur. This resembles many prediction-based recommendation systems working in parallel. Alternately, this paper proposes a three-step hybrid ranking order system for finding the top-N list for the multi-criteria recommendation system. The first step decomposes the multi-criteria user-item matrix into many single-rating user-item matrices while the second step finds partial-ranked lists for each item using a learning-to-rank method. This allows us to reflect the interest of the user for each criterion and then pass on this information for the next stage. The last step aggregates the partial-ranked lists into a global-ranked list using a ranking aggregation method. This will reduce the processing time and improve the recommendation quality by representing the user preference for each criterion. Three different sets of experiments are conducted on Yahoo!Movie dataset, and the results show that the proposed multi-criteria-ranking approach outperforms both the traditional no-ranking item-based collaborative recommendation and single-criteria-ranking approach that uses two popular learning-to-rank methods.

Learning the true rank ordering among objects by aggregating a set of expert opinion rank order lists is an important and ubiquitous problem in many applications ranging from social choice theory to recommendation systems and search aggregation. We study the problem of unsupervised rank aggregation where no ground truth ordering information in available, neither about the true preference ordering among any set of objects nor about the quality of individual rank lists. Aggregating the often inconsistent and poor quality rank lists in such an unsupervised manner is a highly challenging problem, and standard consensus-based methods fall short in terms of both quality and scalability. In this manuscript we propose a novel framework to bypass these issues by using object attributes to augment the standard rank aggregation framework. We design algorithms that learn joint models on both rank lists and object features to obtain an aggregated rank ordering that is more accurate and robust, and also helps weed out rank lists of dubious validity. We validate our techniques on synthetic datasets where our algorithm is able to estimate the true rank ordering even when the rank lists are corrupted. Experiments on three real datasets, MQ2008, MQ2007 and OHSUMED, show that using object features can result in significant improvement in performance over existing rank aggregation methods that do not use object information. Furthermore, when at least some of the rank lists are of high quality, our methods are able to effectively exploit such information to output an aggregated rank ordering of high accuracy.

The analytic hierarchy process (AHP) is a method to help people make a complex decision by analyzing and synthesizing multiple criteria for the decision objective in a hierarchy. We adopt the AHP as a new strategy for the semantic concept detection (SCD) in video so that multiple factors involved in SCD, including multiple modalities and relating concepts, can be hierarchically analyzed and synthesized. In this paper, we first explain, by an example, why and how the SCD problem can be analyzed by the AHP. Then, following the idea of the AHP, we develop a new rank aggregation (RA) method, called AHP-RA. Experimental results of RA for SCD in video show the effectiveness of this method.

Metasearch engine is a system that provides unified access to multiple existing search engines. After the results returned from all used component search engines are collected, the metasearch system merges the results into a single ranked list which is expected to be better than the results of the best of the participating search systems. The success of a metasearch engine depends mainly on their rank aggregation method. The system is a better one, if the aggregated list of results displayed before the user satisfies the user with his information need. In this paper, we discuss the development of a metasearch engine that performs user feedback based metasearching using modified rough set based aggregation. Metasearching using the modified rough set based aggregation is performed in two phases namely the ranking rule learning phase and the rank aggregation phase. For each query in the training set, we mine the ranking rules and select the best rules-set by performing cross-validation test. Once the system is trained, we use the best rule set to get the overall ranking for the results returned from different search systems in response to other queries. We also present few snapshots of our system.

We study the problem of combining active learning suggestions to identify informative training examples by empirically comparing methods on benchmark datasets. Many active learning heuristics for classification problems have been proposed to help us pick which instance to annotate next. But what is the optimal heuristic for a particular source of data? Motivated by the success of methods that combine predictors, we combine active learners with bandit algorithms and rank aggregation methods. We demonstrate that a combination of active learners outperforms passive learning in large benchmark datasets and removes the need to pick a particular active learner a priori. We discuss challenges to finding good rewards for bandit approaches and show that rank aggregation performs well.

In the past few years, segmentation and classification techniques have become a cornerstone of many successful remote sensing algorithms aiming at delineating geographic target objects. One common strategy relies on using multiple complex features to guide the delineation process with the objective of gathering complementary information for improving classification results. However, a persistent problem in this approach is how to combine different and noncorrelated feature descriptors automatically. In this regard, one solution is to combine them through multiple classifier systems (MCSs) in which the diversity of simple/noncomplex classifiers is an essential issue in the definition of appropriate strategies for classifier fusion. In this paper, we propose a novel strategy for selecting classifiers (whereby a classifier is taken as a pair of learning method plus image descriptor) to be combined in MCS. In the proposed solution, diversity measures are used to assess the degree of agreement/disagreement between pairs of classifiers and ranked lists are created to sort them according to their diversity score. Thereafter, the classifiers are also sorted according to their performance through different evaluation measures (e.g., kappa and tau indices). In the end, a rank aggregation method is proposed to select the most suitable classifiers based on both the diversity and the effectiveness performance of classifiers. The proposed fusion framework has targeted at coffee crop classification and urban recognition but it is general enough to be used in a variety of other pattern recognition problems. Experimental results demonstrate that the novel strategy yields good results when compared to several baselines while using fewer classifiers and being much more efficient.

Rank aggregation has recently become a common approach for combining individual rankings into a consensus and for quantifying and improving performance in various applications, such as elections, web page rankings, and sports. During the past few years, rankings from many sources have become increasingly high-dimensional and partial. In this study, we develop a rank aggregation method by constructing a directed weighted competition graph. We introduce the concept of “ratio of out- and in-degrees (ROID)” to transform high-dimensional partial rankings into a single consensus. Moreover, we provide a novel effectiveness measure for the aggregate ranking according to its deviations from the ground truth ranking. The proposed method is compared with four typical methods with synthetic rankings. The results indicate that our method outperforms the other four by a significant margin and can be particularly efficient in aggregating high-dimensional rankings. The empirical results validate the effectiveness and feasibility of our method.

The problem of interpreting or aggregating multiple rankings is common to many real-world applications. Perhaps the simplest and most common approach is a weighted rank aggregation, wherein a (convex) weight is applied to each input ranking and then ordered. This paper describes a new tool for visualizing and displaying ranking information for the weighted rank aggregation method. Traditionally, the aim of rank aggregation is to summarize the information from the input rankings and provide one final ranking that hopefully represents a more accurate or truthful result than any one input ranking. While such an aggregated ranking is, and clearly has been, useful to many applications, it also obscures information. In this paper, we show the wealth of information that is available for the weighted rank aggregation problem due to its structure. We apply weight set decomposition to the set of convex multipliers, study the properties useful for understanding this decomposition, and visualize the indifference regions. This methodology reveals information–that is otherwise collapsed by the aggregated ranking–into a useful, interpretable, and intuitive decision support tool. Included are multiple illustrative examples, along with heuristic and exact algorithms for computing the weight set decomposition.

A new hierarchical ranking aggregation method

This article investigates the impact of rating data characteristics on the performance of several popular recommendation algorithms, including user-based and item-based collaborative filtering, as well as matrix factorization. We focus on three groups of data characteristics: rating space, rating frequency distribution, and rating value distribution. A sampling procedure was employed to obtain different rating data subsamples with varying characteristics; recommendation algorithms were used to estimate the predictive accuracy for each sample; and linear regression-based models were used to uncover the relationships between data characteristics and recommendation accuracy. Experimental results on multiple rating datasets show the consistent and significant effects of several data characteristics on recommendation accuracy.

With the rapid advancement of wireless technologies and mobile devices, mobile services offer great convenience and huge opportunities for service creation. However, information overload make service recommendation become a crucial issue in mobile services. Although traditional single-criteria recommendation systems have been successful in a number of personalization applications, obviously individual criterion cannot satisfy consumers’ demands. Relying on multi-criteria ratings, this paper presents a novel recommendation system using the multi-agent technology. In this system, the ratings with respect to the three criteria are aggregated into an overall service ranking list by a rank aggregation algorithm. Furthermore, all of the services are classified into several clusters to reduce information overload further. Finally, Based on multi-criteria rank aggregation, the prototype of a recommendation system is implemented. Successful applications of this recommendation system have demonstrated the efficiency of the proposed approach.

Random walk based rank aggregation to improving web search