Classification of Imbalanced Data in E-Commerce

Error back-propagation algorithm for classification of imbalanced data

Abstract. In European phytosociology, national classifications of corresponding vegetation types show considerable differences even between neighbouring countries. Therefore, the European Vegetation Survey project urgently needs numerical classification methods for large data sets that are able to produce compatible classifications using data sets from different countries. We tested the ability of two methods, TWINSPAN and COCKTAIL, to produce similar classifications of wet meadows (Calthion, incl. Filipendulenion) for Germany (7909 relevés) and the Czech Republic (1287 relevés) in this respect.In TWINSPAN, the indicator ordination option was used for classification of two national data sets, and the extracted assignment criteria (indicator species) were applied crosswise from one to the other national data set. Although the data sets presumably contained similar community types, TWINSPAN revealed almost no correspondence between the groups derived from the proper classification of the national data set and the groups defined by the assignment criteria taken from the other national data set. The reason is probably the difference in structure between the national data sets, which is a typical, but hardly avoidable, feature of any pair of phytosociological data sets. As a result, the first axis of the correspondence analysis, and consequently the first TWINSPAN division, are associated with different environmental gradients; the difference in the first division is transferred and multiplied further down the hierarchy.COCKTAIL is a method which produces relevé groups on the basis of statistically formed species groups. The user determines the starting points for the formation of species groups, and groups already found in one data set can be tested for existence in the other data set. The correspondence between the national classifications produced by COCKTAIL was fairly good. For some relevé groups, the lack of correspondence to groups in the other national data set could be explained by the absence of the corresponding vegetation types in one of the countries, rather than by methodological problems.

Deep phylogenetic incongruence in the angiosperm clade Rosidae

The Parsimony Ratchet1 is presented as a new method for analysis of large data sets. The method can be easily implemented with existing phylogenetic software by generating batch command files. Such an approach has been implemented in the programs DADA (Nixon, 1998) and Winclada (Nixon, 1999). The Parsimony Ratchet has also been implemented in the most recent versions of NONA (Goloboff, 1998). These implementations of the ratchet use the following steps: (1) Generate a starting tree (e.g., a "Wagner" tree followed by some level of branch swapping or not). (2) Randomly select a subset of characters, each of which is given additional weight (e.g., add 1 to the weight of each selected character). (3) Perform branch swapping (e.g., "branch-breaking" or TBR) on the current tree using the reweighted matrix, keeping only one (or few) tree. (4) Set all weights for the characters to the "original" weights (typically, equal weights). (5) Perform branch swapping (e.g., branch-breaking or TBR) on the current tree (from step 3) holding one (or few) tree. (6) Return to step 2. Steps 2-6 are considered to be one iteration, and typically, 50-200 or more iterations are performed. The number of characters to be sampled for reweighting in step 2 is determined by the user; I have found that between 5 and 25% of the characters provide good results in most cases. The performance of the ratchet for large data sets is outstanding, and the results of analyses of the 500 taxon seed plant rbcL data set (Chase et al., 1993) are presented here. A separate analysis of a three-gene data set for 567 taxa will be presented elsewhere (Soltis et al., in preparation) demonstrating the same extraordinary power. With the 500-taxon data set, shortest trees are typically found within 22 h (four runs of 200 iterations) on a 200-MHz Pentium Pro. These analyses indicate efficiency increases of 20×-80× over "traditional methods" such as varying taxon order randomly and holding few trees, followed by more complete analyses of the best trees found, and thousands of times faster than nonstrategic searches with PAUP. Because the ratchet samples many tree islands with fewer trees from each island, it provides much more accurate estimates of the "true" consensus than collecting many trees from few islands. With the ratchet, Goloboff's NONA, and existing computer hardware, data sets that were previously intractable or required months or years of analysis with PAUP* can now be adequately analyzed in a few hours or days.

BackgroundThe goal of class prediction studies is to develop rules to accurately predict the class membership of new samples. The rules are derived using the values of the variables available for each subject: the main characteristic of high-dimensional data is that the number of variables greatly exceeds the number of samples. Frequently the classifiers are developed using class-imbalanced data, i.e., data sets where the number of samples in each class is not equal. Standard classification methods used on class-imbalanced data often produce classifiers that do not accurately predict the minority class; the prediction is biased towards the majority class. In this paper we investigate if the high-dimensionality poses additional challenges when dealing with class-imbalanced prediction. We evaluate the performance of six types of classifiers on class-imbalanced data, using simulated data and a publicly available data set from a breast cancer gene-expression microarray study. We also investigate the effectiveness of some strategies that are available to overcome the effect of class imbalance.ResultsOur results show that the evaluated classifiers are highly sensitive to class imbalance and that variable selection introduces an additional bias towards classification into the majority class. Most new samples are assigned to the majority class from the training set, unless the difference between the classes is very large. As a consequence, the class-specific predictive accuracies differ considerably. When the class imbalance is not too severe, down-sizing and asymmetric bagging embedding variable selection work well, while over-sampling does not. Variable normalization can further worsen the performance of the classifiers.ConclusionsOur results show that matching the prevalence of the classes in training and test set does not guarantee good performance of classifiers and that the problems related to classification with class-imbalanced data are exacerbated when dealing with high-dimensional data. Researchers using class-imbalanced data should be careful in assessing the predictive accuracy of the classifiers and, unless the class imbalance is mild, they should always use an appropriate method for dealing with the class imbalance problem.

Online detection of wave soldering is an important method of inspecting defective products in the workshop. Accurate quality detection can reduce production costs and provide support for quality warnings in the wave soldering process. However, there are still problems in improving the detection accuracy for the defect class. Although class imbalance in data can be addressed by data-level methods such as over-sampling and under-sampling, these methods destroy the integrity of the original data set and may cause information loss and over-fitting problems. In order to solve the above problems, this article focuses on how to design a new loss function that fuses class weights from focal loss (FS) and sample weights from AdaBoost to improve attention to the minority samples without changing data distribution. In this way, an FS-AdaBoost-RegNet model based on transfer learning is constructed to enhance the detection accuracy in industrial environment. Finally, the images of the wave soldering from an electronic assembly workshop are taken to validate the performance of the proposed method. The experiment on 941 testing samples of the imbalance datasets showed that the FS-AdaBoost-RegNet model with new loss function reached the overall accuracy of 98.39%, and the overall recall of 96.19%. The results proved that the proposed method promotes the ability to identify defect class compared with other methods.

Image processing and machine learning communities have long addressed the problems involved in the analysis of large high-dimensional data sets. To deal with high-dimensional data efficiently, learning core properties of given data set is important. The manifold learning methods such as ISOMap try to identify a low-dimensional manifold from a set of unorganized samples. ISOMap method is an extension of the classical multidimensional scaling method for dimension reduction, which find a linear subspace in which dissimilarity between data points is preserved. In order to measure dissimilarity, ISOMap uses the geodesic distances on the manifold instead of Euclidean distance. In this paper, we propose a modification of ISOMap using class information, which is often given in company with input data in many applications such as pattern classification. Since the conventional ISOMap does not use class information in approximating true geodesic distance between each pair of data points, it is difficult to construct a data structure related to class-membership that may give important information for given task such as data visualization and classification. The proposed method utilizes class-membership for measuring distance of data pair so as to find a low-dimensional manifold preserving the distance between classes as well as the distance between data points. Through computational experiments on artificial data sets and real facial data sets, we confirm that the proposed method gives better performance than the conventional ISOMap.

As of this date, approximately 5,000 horizontal Eagle Ford wells have been completed in South Texas. Still, geologists and engineers question whether their companies are using the most appropriate operating practices. Side-by-side case studies may show value or not, given the challenge of small sample size and hidden influences on outcome. Multivariate statistical analysis of larger data sets offers sound interpretation across larger geographic areas, with the caveat that correlations need to be scaled to local conditions. The purpose of this paper is to apply multivariate statistical modeling in conjunction with Geographic Information Systems (GIS) pattern recognition work to the Eagle Ford. The investigation began by acquiring Eagle Ford data using both proprietary and public information. The different data sets were loaded into a common database and put through quality control sanity checks. Production proxies, such as maximum oil rate in the first 12 producing months and normalized 12 month cumulative production, were selected and merged with the other data. Final data sets were then subjected to analysis in both an open-source multivariate statistical analysis and visualization code and a commercial Geographic Information Systems (GIS) application. Similar to other studies in unconventional reservoirs, integration of the two analysis and interpretation methods highlighted the importance of using well location as a proxy for reservoir quality when working with data sets that lack such measurements. The use of multivariate statistical analysis allowed modeling the impact of particular well architecture, completion, and stimulation parameters on the production outcome by integrating out the impact of other variables in the system. This work is a continuation of the prior work designed to address well optimization in unconventional reservoirs. It is significant in that it takes full advantage of GIS map-based methods and multivariate statistical methods to capitalize on the volume of data available through the public domain.

Astronomy datasets are generally terabytes in size and contain hundreds of millions of objects separated into millions of files-factors which makes many analyses impractical to perform on small computers. The key question we answer in this paper is: How can we leverage Grid resources to make the analysis of large astronomy datasets a reality for the astronomy community? To address this question, we have developed a Web Services-based system, AstroPortal, that uses grid computing to federate large computing and storage resources for dynamic analysis of large datasets. Building on the GT4, we have built a prototype and implemented a first analysis, stacking, that sums multiple regions of the sky, a function that can help both identify variable sources and detect faint objects. AstroPortal gives the astronomy community a new tool to advance their research and to open new doors to opportunities never before possible on such a large scale.

Addressing the problem of imbalanced data category distribution in real applications and the problem of traditional classifiers tending to ensure the accuracy of the majority class while ignoring the accuracy of the minority class when processing imbalanced data, this paper proposes a method called RBSP-Boosting for imbalanced data classification. First, RBSP-Boosting introduces the Shapley value and calculates the Shapley value for each sample of the dataset through the truncated Monte Carlo method. Moreover, the proposed method removes the noise data according to the Shapley value and undersamples the samples with Shapley values less than zero in the majority class. Then, it takes the Shapley value as the weight of the sample and oversamples the minority class according to the weight. Finally, the new dataset is trained on the classifier through the AdaBoost classifier. Experiments are conducted on nine groups of UCI and KEEL datasets, and RBSP-Boosting is compared with four sampling algorithms: Random-OverSampler, SMOTE, Borderline-SMOTE and SVM-SMOTE. Experimental results show that the RBSP-Boosting method in the three evaluation metrics of AUC, F-score and G-mean, compared with the best performance of the four comparison algorithms, increases by 4.69%, 10.3% and 7.86%, respectively. The proposed method can significantly improve the effect of imbalanced data classification.

Machine learning for Big Data analytics in plants.

Classification of imbalanced data is an important research topic in machine learning. In unbalanced data, the minority class has significantly fewer training samples than the majority class, resulting in classification results favoring the majority class. Therefore, we propose a new algorithm for the classification of imbalanced data. The new algorithm is divided into the following three main parts. Firstly, by using the bootstrap method and the maximum entropy principle the new equilibrium data is obtained, which is compared with other oversampling algorithms. Secondly, the obtained equilibrium data are filtered by using the ant colony algorithm. The training set conducive to improving the classifier's performance is selected. Finally, by using the filtered training set, the classifier is classified and compared with the previously proposed oversampling algorithm. By selecting eight sets of data from the UCI and KEEL databases for classification experiments, the experimental results show that the new algorithm proposed in this study outperforms existing algorithms in various evaluation metrics such as Auc, F <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> and Acc.

Jenner, R. A. (2004). The scientific status of metazoan cladistics: why current research practice must change. —Zoologica Scripta,33, 293–310.Metazoan phylogenetics is bustling with activity. The use of comprehensive morphological data sets in recent phylogenetic analyses of the Metazoa indicates that morphological evidence continues to play a key role in the reconstruction of metazoan deep history. In this paper I review the scientific status of morphological metazoan cladistics from the perspective of cladistic research cycles. Each research cycle consists of three main steps: (1) the compilation of a data matrix (2) the simultaneous evaluation of all possible cladograms in a character congruence test, and (3) the assessment of the relationship between evidence and hypothesis after finding the optimal tree. I identify a striking discrepancy between the sophistication of the analysis of given data sets (Step 2), and their compilation and the interpretation of the results (Steps 1 and 3). The latter two steps deserve far greater attention than is current practice. Uncritical and nonexplicit character selection, character coding, and character scoring seriously compromise Step 1. Careful comparative morphological study prior to data matrix construction is necessary to remedy this problem in future cladistic analyses. Step 2 is the locus of most recent advances in metazoan cladistics through the increasing availability of computing power, and the development of increasingly efficient phylogenetic software that allows analysis of large data sets. Failure to identify problems and errors generated in Step 1 of the research cycle is testament to the general failure of Step 3. Consequently, recent progress in metazoan cladistics is primarily analytical, while the only empirical anchor of the discipline receives surprisingly little attention. Not surprisingly, the first generation of modern metazoan phylogeneticists used computers principally as a relatively quick and easy means to generate abundant phylogenies from morphological data. The next phase should build on this foundation by critically testing these alternative hypotheses by a thorough qualitative reassessment and elaboration of morphological data matrices, and a more critical approach to data selection. A rigorous research program for metazoan cladistics can only be established when the cladistic research cycle is properly completed, and when subsequent research cycles are effectively linked to previous efforts.

Background Availability of numerous oil deposits with a wide variety of the conditions of occurrence, reservoir properties and physical-chemical properties of fluids makes important the need of grouping of development objects and identification of relatively uniform groups by certain criteria. Grouping allows to define similarities and differences between deposits in determining development objects, and to substantiate systems of development. Analysis of large data sets on objects characterized by the measured or estimated parameters is labor-intensive, requiring a lot of time and effort, so there is a question of dimension reduction, i.e. of reducing the original data set to a smaller number of parameters, and grouping of objects into relatively uniform groups. Here the parameters can be selected from the original ones or obtained by calculation and conversion (with the least loss of information about the objects under study). An effective tool of analysis that can help in this matter is the method of principal component analysis. Aims and Objectives Apply the method of principal component analysis to reduce the number of parameters which define objects of study, so that to find effective ways to develop new oil and gas deposits, and fast and correct solution of practical problems through the use of gained summary experience and assessment of similarities and differences of oil and gas reservoirs by combinations of geological-field attributes. Methods The problem of ranking oil deposits in Western Siberia, is solved by the method of principal component analysis, which is a linear transformation of features in a new set of «independent» random values, which are arranged in descending order of their variances, and the main components are determined by the eigenvectors of the correlation matrix obtained based on input parameters characterizing the objects under investigation. Having selected the components that comprise the bulk of the variances, one can calculate them for different types of objects and classify by clusterable points. Grouping of objects of research was carried out using built-in functions of computer statistics programs (a system of statistical analysis Statistica, analytical applications for Microsoft Excel XLStat). The same calculation was performed on the basis of the theory of mathematical statistics. The calculation results coincided. Results The method of principal component analysis allowed selection from the original data set comprising 50 research objects characterized by 12 parameters of six relatively uniform groups using only 4 factors (principal components) which, however, contain much more information than the individual parameters. As a result of the carried out analysis data were systematized and the dimension of the initial population (transition from multidimensional space to the four-dimensional) was reduced, with no significant loss of information about research objects. Conclusion Based on the proposed algorithm, and relatively uniform groups obtained there is a possibility for inclusion of new deposits into one of these groups and selection of the nearest to it analogous object, that will simplify the design of development system and identify ways and means of improving system’s effectiveness.

Many papers have presented rendering techniques and simplification ideas with the objective of speeding up image generation for irregular grid data sets. For large data sets, however, even the current fastest algorithms are known to require seconds to generate each image, making real time analysis of such data sets very difficult, or even impossible, unless one has access to powerful and expensive computer hardware. In order to synthesize a system for handling very large data set analysis, we have assembled algorithms for rendering, simplification and triangulation, and added to them some optimizations. We have made some improvements on one of the best current algorithms for rendering irregular grids, and added to it some simple approximation methods in both image and object space, resulting in a system that achieves high frame rates, even on slow computers without any specific graphic hardware. The algorithm adapts itself to the time budget it has available for each image generation, using hierarchical representations of the mesh for faster delivery of images when transformations are imposed to the data. When given additional time, the algorithm generates finer images, obtaining the precise final image if given sufficient time. We were able to obtain frame rates of the order of 5 Hz for medium-sized data sets, which is about 20 times faster than previous rendering algorithms. With a trade-off between image accuracy and speed, similar frame rates can be achieved on different computers.