What makes data robust

Semi-supervised ranking is a relatively new and important learning problem. Some semi-supervised learning to rank algorithms have been proposed by adapting existing pair-wise ranking algorithms into the semi-supervised setting. In contrast, there is comparatively little work in developing semi-supervised ranking algorithm based on the list-wise approach. In this paper, we proposed a semi-supervised extension of one of the state-of-the art list-wise ranking algorithm--AdaRank, for the task of learning to rank with partially labeled data. The proposed algorithm is a two-staged approach that combines the process of label propagation and a regularized version of AdaRank. The label propagation process is used to propagate the relevance labels from labeled instances to other unlabeled ones, so that more training data are available to learn the ranking function. The regularized version of AdaRank optimizes a novel performance measure, which incorporates both the usual Information Retrieval IR metrics and the smoothness of the ranking function. Experimental results on benchmark datasets show the advantage of our methods over existing pair-wise algorithms for the Information Retrieval measures.

Collecting large-scale data with clean labels for supervised training of neural networks is practically challenging. Although noisy labels are usually cheap to acquire, existing methods suffer a lot from label noise. This paper targets at the challenge of robust training at high label noise regimes. The key insight to achieve this goal is to wisely leverage a small trusted set to estimate exemplar weights and pseudo labels for noisy data in order to reuse them for supervised training. We present a holistic framework to train deep neural networks in a way that is highly invulnerable to label noise. Our method sets the new state of the art on various types of label noise and achieves excellent performance on large-scale datasets with real-world label noise. For instance, on CIFAR100 with a $40\%$ uniform noise ratio and only 10 trusted labeled data per class, our method achieves $80.2{\pm}0.3\%$ classification accuracy, where the error rate is only $1.4\%$ higher than a neural network trained without label noise. Moreover, increasing the noise ratio to $80\%$, our method still maintains a high accuracy of $75.5{\pm}0.2\%$, compared to the previous best accuracy $48.2\%$. Source code available: https://github.com/google-research/google-research/tree/master/ieg

In previous work, Feldmann and Mysliwietz [3] showed that every shuffle-exchange graph contains a Hamilton path. In this paper, we show that there is a linear time ranking algorithm associated with that Hamilton path. The ranking algorithm returns the rank-position of each node of the graph with respect to the order given by the Hamilton path. Ranking algorithms are important for they can be applied to yield efficient implementations of certain network emulations using SIMD-style parallel algorithms for translating node labels.KeywordsCayley GraphHamilton PathRanking AlgorithmInitial NodeGray CodeThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

SummaryAccurate prediction of the solubility of chemical substances in solvents remains a challenge. The sparsity of high-quality solubility data is recognized as the biggest hurdle in the development of robust data-driven methods for practical use. Nonetheless, the effects of the quality and quantity of data on aqueous solubility predictions have not yet been scrutinized. In this study, the roles of the size and the quality of data sets on the performances of the solubility prediction models are unraveled, and the concepts of actual and observed performances are introduced. In an effort to curtail the gap between actual and observed performances, a quality-oriented data selection method, which evaluates the quality of data and extracts the most accurate part of it through statistical validation, is designed. Applying this method on the largest publicly available solubility database and using a consensus machine learning approach, a top-performing solubility prediction model is achieved.

In many applications of classifier learning, training data suffers from label noise. Deep networks are learned using huge training data where the problem of noisy labels is particularly relevant. The current techniques proposed for learning deep networks under label noise focus on modifying the network architecture and on algorithms for estimating true labels from noisy labels. An alternate approach would be to look for loss functions that are inherently noise-tolerant. For binary classification there exist theoretical results on loss functions that are robust to label noise. In this paper, we provide some sufficient conditions on a loss function so that risk minimization under that loss function would be inherently tolerant to label noise for multiclass classification problems. These results generalize the existing results on noise-tolerant loss functions for binary classification. We study some of the widely used loss functions in deep networks and show that the loss function based on mean absolute value of error is inherently robust to label noise. Thus standard back propagation is enough to learn the true classifier even under label noise. Through experiments, we illustrate the robustness of risk minimization with such loss functions for learning neural networks.

This focus issue of Physiological Measurement follows the 38th Annual International Computing in Cardiology (CinC) Conference, hosted in Hangzhou, China in September 2011 by Zhejiang University. Each year, the NIH-sponsored PhysioNet resource (http://physionet.org/) runs an open competition lasting several months, aimed at encouraging the development of solutions to an unsolved or poorly solved problem in biomedicine, in most cases making use of relevant clinical and experimental data provided freely by PhysioNet. Participants in these annual challenges discuss their diverse approaches to the Challenge problems during dedicated scientific sessions at CinC. The topics of these PhysioNet/CinC Challenges range from physiologic signal processing and analysis to forecasting and modelling clinically important events and processes.

BoAu: Malicious traffic detection with noise labels based on boundary augmentation

Cross-Project Defect Prediction (CPDP) as a means to focus quality assurance of software projects was under heavy investigation in recent years. However, within the current state-of-the-art it is unclear which of the many proposals performs best due to a lack of replication of results and diverse experiment setups that utilize different performance metrics and are based on different underlying data. Within this article [2, 3], we provide a benchmark for CPDP. Our benchmark replicates 24 CPDP approaches proposed by researchers between 2008 and 2015. Through our benchmark, we answer the following research questions: • RQ1: Which CPDP approaches perform best in terms of F-measure, G-measure, AUC, and MCC? • RQ2: Does any CPDP approach consistently fulfill the performance criteria for successful predictions postulated by Zimmermann et al. [4], i.e., have at least 0.75 recall, 0.75 precision, and 0.75 accuracy? • RQ3: What is the impact of using only larger products (> 100 instances) with a certain balance (at least 5% defective instances and at least 5% non-defective instances) on the benchmark results? • RQ4: What is the impact of using a relatively small subset of a larger data set on the benchmark results? We identified 5 public data sets, which contain defect data about 86 software products that we used to answer these research question. The advantage of using multiple data sets was that we could increase the number of software products and, thereby, increase the external validity of our results. Moreover, we wanted to use multiple performance criteria for the evaluation of the CPDP approaches. Therefore, RQ1 ranks approaches not just using a single criterion, but using the four performance metrics AUC, F-measure, G-measure, and MCC. Existing approaches for the ranking of statistically different approaches neither account for software products from different data sets, nor multiple performance metrics. Therefore, we defined a new approach for the combination of separate rankings for the performance criteria and data sets, into one common ranking. Figure 1 depicts the results for RQ1. The results show that an approach proposed by Camargo Cruz and Ochimizu [1] performs best and even outperforms cross-validation. Moreover, our results show that only 6 of the 24 approaches outperform one of our baselines, i.e., using all data for training without any transfer learning. Regarding RQ2, we determined that predictions only seldomly achieve a high performance of 0.75 recall, precision, and accuracy. The best CPDP approaches only fulfill the criterion for 4 of the 86 products, i.e., 4.6% of the time. Thus, CPDP still has not reached a point where the performance of the results is sufficient for the application in practice. RQ3 and RQ4 were used to see if results are affected by subsetting data, as is often done for defect prediction experiments. For RQ3, i.e., using a large subset, we determined no difference between using all data and using the subset. For RQ4, i.e., using a small subset of of data, we found that there are statistically signifcant differences in reported performances of up to 5%. Thus, the use of small subsets should be avoided.

Voice-over-Internet-Protocol (VoIP) calls are prone to various speech impairments due to environmental and network conditions resulting in bad user experience. A reliable audio impairment classifier helps to identify the cause for bad audio quality. The user feedback after the call can act as the ground truth labels for training a supervised classifier on a large audio dataset. However, the labels are noisy as most of the users lack the expertise to precisely articulate the impairment in the perceived speech. In this paper, we analyze the effects of massive noise in labels in training dense networks and Convolutional Neural Networks (CNN) using engineered features, spectrograms and raw audio samples as inputs. We demonstrate that CNN can generalize better on the training data with a large number of noisy labels and gives remarkably higher test performance. The classifiers were trained both on randomly generated label noise and the label noise introduced by human errors. We also show that training with noisy labels requires a significant increase in the training dataset size, which is in proportion to the amount of noise in the labels.

Financial institutions have a large amount of data on their borrowers, which can be used to predict the probability of borrowers defaulting their loan or not. Some of the models that have been used to predict individual loan defaults include linear discriminant analysis models and extreme value theory models. These models are parametric in nature since they assume that the response being investigated takes a particular functional form. However, there is a possibility that the functional form used to estimate the response is very different from the actual functional form of the response. The purpose of this research was to analyze individual loan defaults in Kenya using the logistic regression model. The data used in this study was obtained from equity bank of Kenya for the period between 2006 to 2016. A random sample of 1000 loan applicants whose loans had been approved by equity bank of Kenya during this period was obtained. Data obtained was on the credit history, purpose of the loan, loan amount, nature of the saving account, employment status, sex of the applicant, age of the applicant, security used when acquiring the loan and the area of residence of the applicant (rural or urban). This study employed a quantitative research design, it deals with individual loans defaults as group characteristics of a borrower. The data was pre-processed by seeding using R- Software and then split into training dataset and test data set. The train data was used to train the logistic regression model by employing Supervised machine learning approach. The R-statistical software was used for the analysis of the data. The test data set was used to do cross-validation of the developed logistic model which later was used for analysis prediction of individual loan defaults. This study focused on the analysis of individual loan defaults in Kenya using the logistic regression model in Machine learning. The logistic regression model predicted 303 defaults from train data set, 122 non-defaults and misclassified loans were 56 and 69. The model had an accuracy of 0.7727 with the train data and 0.7333 with the test data. The logistic regression model showed a precision of 0.8440 and 0.8244 with the train and test data respectively. The performance of the model with both the train and test data was illustrated using a plot of train errors and test errors against sample size on the same axes. The plot showed that the performance of the model increases with an increase in sample size. The study recommended the use of logistic regression in conjunction with supervised machine learning approach in loan default prediction in financial institutions and also more research should be carried out on ensemble methods of loan defaults prediction in order to increase the prediction accuracy.

Domain adaptation is an important problem in learning to rank due to the lack of training data in a new search task. Recently, an approach based on instance weighting and pairwise ranking algorithms has been proposed to address the problem by learning a ranking model for a target domain only using training data from a source domain. In this paper, we propose a novel framework which extends the previous work using a listwise ranking algorithm for ranking adaptation. Our framework firstly estimates the importance weight of a query in the source domain. Then, the importance weight is incorporated into the state-of-the-art listwise ranking algorithm, known as AdaRank. The framework is evaluated on the Letor3.0 benchmark dataset. The results of experiment demonstrate that it can significantly outperform the baseline model which is directly trained on the source domain, and most of the time not significantly worse than the optimal model which is trained on the target domain.

Noisy label tolerance: A new perspective of Partial Multi-Label Learning

Abstract:Bankruptcy prediction and credit scoring are the two important problems facing financial decision support. The multilayer perceptron (MLP) network has shown its applicability to these problems and its performance is usually superior to those of other traditional statistical models. Support vector machines (SVMs) are the core machine learning techniques and have been used to compare with MLP as the benchmark. However, the performance of SVMs is not fully understood in the literature because an insufficient number of data sets is considered and different kernel functions are used to train the SVMs. In this paper, four public data sets are used. In particular, three different sizes of training and testing data in each of the four data sets are considered (i.e. 3:7, 1:1 and 7:3) in order to examine and fully understand the performance of SVMs. For SVM model construction, the linear, radial basis function and polynomial kernel functions are used to construct the SVMs. Using MLP as the benchmark, the SVM classifier only performs better in one of the four data sets. On the other hand, the prediction results of the MLP and SVM classifiers are not significantly different for the three different sizes of training and testing data.

Classification algorithms are widely applied to predict failures and detect anomalies in various application areas. It is common to assume that the data and labels are correct when training, but this is challenging to guarantee in the real world. If there are erroneous labels in the training data, a model can easily overfit to these, resulting in poor performance. How to handle label noise has been previously researched, however, few works focus on label noise in anomaly detection. In this work, we propose LDAAD, a novel algorithm framework for label de-noising for anomaly detection that combines unsupervised learning and semi-supervised learning methods. Specifically, we apply anomaly detection to partition the training data into low-risk and high-risk sets. We subsequently build upon ideas from cross-validation and train multiple classification models on segments of the low-risk data. The models are used both to relabel the samples in the high-risk set and to filter the low-risk samples. Finally, we merge the two sets to obtain a final sample set with more confident labels. We evaluate LDAAD on multiple real-world datasets and show that LDAAD achieves robust results that outperform the benchmark methods. Specifically, LDAAD achieves a 5% accuracy improvement over the second-best method for symmetric noise while having a minimal detrimental impact when no label noise is present.

A feature synthesis algorithm combined with modified k-NN classifier is described in this paper. The feature information of the training data is extracted firstly. In the classification phase, the feature information of the training data and the testing data are compared to make the initial prediction. If the predicting result is {C} which has only one element, testing data will be labeled class C. If the predicting result is {C1, C2, ...Ci}, the corresponding subset of the training data set will be used in the k-NN algorithm to attain the predicted result. It is shown that this algorithm combined k-NN rule reduces computing time. If the feature information set could well generalize the training data, the error rate of the algorithm is decreased. The effectiveness of this proposed approach is verified on a public remote sensing data set.