Sparse Stochastic Online AUC Optimization for Imbalanced Streaming Data

High-dimensional sparse numerical data are normally encountered in machine learning, recommender systems, finance and medical imaging. The problem with this type of data is that it has high dimensions (many features) and highly sparse (most values are zero), which is prone to overfitting. The data visualization can be achieved through a neural network architecture called stacked autoencoders. These multilayer autoencoders are designed to reconstruct input data, but overfitting is a major problem. To overcome this problem novel L1 Regularization-dropout technique is introduced to reduce overfitting and boost stacked autoencoder performance. L1 regularization penalizes large weights, simplifying data representations whereas the dropout technique randomly turns off neurons during training and makes the model dependent only on the selected turn-on neurons. The model employs batch normalization to improve the performance of the autoencoder. The approach was implemented on a high-dimensional sparse numerical dataset in the field of cybersecurity to minimize the loss function, measured by Mean Square Error (MSE) and Mean Absolute Error (MAE). The findings were compared to the conventional stacked autoencoder. The study revealed that the suggested method effectively mitigated the issue of overfitting. Stacked autoencoders, when combined with L1 regularisation and the dropout approach, are very successful in handling high-dimensional sparse numerical data in a diverse range of applications.

Fuzzy partition based soft subspace clustering and its applications in high dimensional data

In this paper, we propose an incremental clustering algorithm called INCLUS for high dimensional sparse transactional data using a newly defined similarity measure and a notion of cluster representatives based on locally frequent items of each cluster. INCLUS seeks structures in transactional data with respect to the support and similarity constraints specified by the users. The effectiveness and the order-independence property of INCLUS are empirically studied and compared with two state- of-art algorithms. Though it is a one-pass algorithm without any iterative refinement, INCLUS is not only effective and scalable, but also insensitive to the order of transactions, which is a crucial property for an incremental algorithm. globally frequent items are neglected during clustering, clusters embedded in such transactions cannot be discovered. In this paper, we define a new notion of cluster representative based on frequent items within each cluster that is easy to comprehend by users, and introduce a new similarity measure especially suitable for sparse transactional data. Using these, an incremental clustering algorithm INCLUS is proposed to cope with high cardinality of transactional datasets. INCLUS is a structure seeking algorithm as it produces clusters based on the support and similarity constraints specified by the users. The domains of input parameters (support and similarity thresholds) are well defined and can be easily chosen by users. INCLUS is significantly different from LargeItem, SUMMARY and TrK-Means which also employ frequent items for clustering. LargeItem uses global optimization to minimize both overlapping of frequent items among clusters and the number of infrequent items for a clustering. INCLUS uses frequent items for local optimization such that similarities for a clustering are maximized with respect to the similarity constraints. In SUMMARY, the frequent items used are globally frequent in the whole data set while in INCLUS they are only locally frequent within a cluster. Unlike INCLUS, TrK-Means does not use similarity constraints and it produces k or more clusters based on the user input of k. Squeezer (7) is an incremental clustering algorithm for categorical data. It uses a different similarity function from INCLUS, which is defined as the number of matched attribute values in a pair of tuples.

Imbalanced streaming data is commonly encountered in real-world data mining and machine learning applications, and has attracted much attention in recent years. Both imbalanced data and streaming data in practice are normally encountered together; however, little research work has been studied on the two types of data together. In this paper, we propose a multi-window based ensemble learning method for the classification of imbalanced streaming data. Three types of windows are defined to store the current batch of instances, the latest minority instances, and the ensemble classifier. The ensemble classifier consists of a set of latest sub-classifiers, and the instances employed to train each sub-classifier. All sub-classifiers are weighted prior to predicting the class labels of newly arriving instances, and new sub-classifiers are trained only when the precision is below a predefined threshold. Extensive experiments on synthetic datasets and real-world datasets demonstrate that the new approach can efficiently and effectively classify imbalanced streaming data, and generally outperforms existing approaches.

A challenge for mining large-scale streaming data overlooked by most existing studies on online learning is the skewdistribution of examples over different classes. Many previous works have considered cost-sensitive approaches in an online setting for streaming data, where fixed costs are assigned to different classes, or ad-hoc costs are adapted based on the distribution of data received so far. However, it is not necessary for them to achieve optimal performance in terms of the measures suited for imbalanced data, such as Fmeasure, area under ROC curve (AUROC), area under precision and recall curve (AUPRC). This work proposes a general framework for online learning with imbalanced streaming data, where examples are coming sequentially and models are updated accordingly on-the-fly. By simultaneously learning multiple classifiers with different cost vectors, the proposed method can be adopted for different target measures for imbalanced data, including F-measure, AUROC and AUPRC. Moreover, we present a rigorous theoretical justification of the proposed framework for the F-measure maximization. Our empirical studies demonstrate the competitive if not better performance of the proposed method compared to previous cost-sensitive and resampling based online learning algorithms and those that are designed for optimizing certain measures.

Learning from imbalanced data has conventionally been conducted on stationary data sets. Recently, there have been several methods proposed for mining imbalanced data streams, in which training data is read in consecutive data chunks. Each data chunk is considered as a conventional imbalanced data set, making it easy to apply sampling methods to balance data chunks. However, one drawback of chunk-based learning methods is that the update of classification models is delayed until a full data chunk is received. Therefore, this paper proposes a new method for online learning from imbalanced data streams, which uses naive Bayes as the base learner. To deal with the problem of class imbalance, a new training instance from the minority class is always involved in learning, but one from the majority class is only used with a small probability. In effect, this method corresponds to an under-sampling technique on imbalanced data streams. We show the effectiveness of the proposed online learning method on ten UCI data sets of various domains. Problems in the performance of naive Bayes on imbalanced data sets are also discussed.

Dynamic Classification Ensembles for Handling Imbalanced Multiclass Drifted Data Streams

Co-clustering addresses the problem of simultaneous clustering of both dimensions of a data matrix. When dealing with high dimensional sparse data, co-clustering turns out to be more beneficial than one-sided clustering even if one is interested in clustering along one dimension only. Aside from being high dimensional and sparse, some datasets, such as document-term matrices, exhibit directional characteristics, and the $$L_2$$ normalization of such data, so that it lies on the surface of a unit hypersphere, is useful. Popular co-clustering assumptions such as Gaussian or Multinomial are inadequate for this type of data. In this paper, we extend the scope of co-clustering to directional data. We present Diagonal Block Mixture of Von Mises–Fisher distributions (dbmovMFs), a co-clustering model which is well suited for directional data lying on a unit hypersphere. By setting the estimate of the model parameters under the maximum likelihood (ML) and classification ML approaches, we develop a class of EM algorithms for estimating dbmovMFs from data. Extensive experiments, on several real-world datasets, confirm the advantage of our approach and demonstrate the effectiveness of our algorithms.

High-dimensional sparse matrix data frequently arise in various applications. A notable example is the weighted word–word co-occurrence count data, which summarizes the weighted frequency of word pairs appearing within the same context window. This type of data typically contains highly skewed non-negative values with an abundance of zeros. Another example is the co-occurrence of item–item or user–item pairs in e-commerce, which also generates high-dimensional data. The objective is to utilize these data to predict the relevance between items or users. In this paper, we assume that items or users can be represented by unknown dense vectors. The model treats the co-occurrence counts as arising from zero-inflated Gamma random variables and employs cosine similarity between the unknown vectors to summarize item–item relevance. The unknown values are estimated using the shared parameter alternating zero-inflated Gamma regression models (SA-ZIG). Both canonical link and log link models are considered. Two parameter updating schemes are proposed, along with an algorithm to estimate the unknown parameters. Convergence analysis is presented analytically. Numerical studies demonstrate that the SA-ZIG using Fisher scoring without learning rate adjustment may fail to find the maximum likelihood estimate. However, the SA-ZIG with learning rate adjustment performs satisfactorily in our simulation studies.

Learning from imbalanced and sparse data in multi-mode and high-dimensional tensor formats efficiently is a significant problem in data mining research. On one hand, Coupled Tensor Factorization (CTF) has become one of the most popular methods for joint analysis of heterogeneous sparse data generated from different sources. On the other hand, techniques such as sampling, cost-sensitive learning, etc. have been applied to many supervised learning models to handle imbalanced data. This research focuses on studying the effectiveness of combining advantages of both CTF and imbalanced data learning techniques for missing entry prediction, especially for entries with rare class labels. Importantly, we have also investigated the implication of joint analysis of the main tensor and extra information. One of our major goals is to design a robust weighting strategy for CTF to be able to not only effectively recover missing entries but also perform well when the entries are associated with imbalanced labels. Experiments on both real and synthetic datasets show that our approach outperforms existing CTF algorithms on imbalanced data.

Electricity market reform gives rise to new electricity sales pattern, one product of which is electricity sales package. In view of the scalability and sparse high-dimensional data when we recommend electricity sales package in the big data market, a recommendation pattern is proposed for electricity sales package based on bidirectional clustering and Spark framework. Firstly, an improved K-Means clustering algorithm based on Kruskal algorithm is designed to automatically determine the cluster center, perform bidirectional clustering of electricity sales packages and electricity users, and find the nearest neighbors of package items and similar clusters of target users. Then, an improved clustering-based electricity sales package recommendation pattern is proposed to achieve accurate recommendation of electricity sales packages. Finally, parallelization is implemented on the Spark platform. Experiments show that the proposed recommendation pattern can significantly increase the recommendation accuracy, which has good recommendation efficiency and scalability in a distributed environment, thus enabling users to find interesting electricity sales packages from the massive package information.

Text clustering is a widely studied problem in the text mining domain. The Dirichlet Multinomial Mixture (DMM) model based clustering algorithms have shown good performance to cope with high dimensional sparse text data, obtaining reasonable results in both clustering accuracy and computational efficiency. However, the time complexity of DMM model training is proportional to the average document length and the number of clusters, making it inefficient for scaling up to long text and large corpora, which is common in realworld applications such as documents organization, retrieval and recommendation. In this paper, we leverage a symmetric prior setting for Dirichlet distribution, and build indices to decrease the time complexity of the sampling-based training for DMM from O(K∗L) to O(K∗U), where K is the number of clusters, L the average length of document, and U the average number of unique words in each document. We introduce a Metropolis-Hastings sampling algorithm, which further reduces the sampling time complexity from O(K∗U) to O(U) in the nearly-to-convergence training stages. Moreover, we also parallelize the DMM model training to obtain a further acceleration by using an uncollapsed Gibbs sampler. We combine all these optimizations into a highly efficient implementation, called X-DMM, which enables the DMM model to scale up for long and large-scale text clustering. We evaluate the performance of X-DMM on several real world datasets, and the experimental results show that XDMM achieves substantial speed up compared with existing state-of-the-art algorithms without clustering accuracy degradation.

Factorization machine (FM) is a prevalent approach to modelling pairwise (second-order) feature interactions when dealing with high-dimensional sparse data. However, on the one hand, FMs fail to capture higher-order feature interactions suffering from combinatorial expansion. On the other hand, taking into account interactions between every pair of features may introduce noise and degrade the prediction accuracy. To solve these problems, we propose a novel approach, the graph factorization machine (GraphFM), which naturally represents features in the graph structure. In particular, we design a mechanism to select beneficial feature interactions and formulate them as edges between features. Then the proposed model, which integrates the interaction function of the FM into the feature aggregation strategy of the graph neural network (GNN), can model arbitrary-order feature interactions on graph-structured features by stacking layers. Experimental results on several real-world datasets demonstrate the rationality and effectiveness of our proposed approach. The code and data are available at https://github.com/CRIPAC-DIG/GraphCTR.

The rapid development of Location-based Social Networks (LBSNs) has led to the great demand of personalized Point-of-interests (POIs) recommendation. Although previous researches have presented a variety of methods to recommend POIs by utilizing social relation, geographical mobility data and user content profile, they fail to address user/location’s cold-start problem with high-dimensional sparse data, and overlook the compatibility of social relation, content based methodology and collaborative filtering. To cope with these challenges, we analyze user’s check-in preference and find that it may be influenced in two spaces, namely Social Propagation Influence Space and Individual Attribute Influence Space. To this end, we propose a Social and Content based Collaborative Filtering Model (SCCF), which consists of a Social Relation Preference based Model (SRPB) considering social friends’ preference and a User Location Content-based Model (ULCB) matching the user attributes with location features. Extensive experiments on real-world datasets firmly demonstrate that the proposed SCCF model outperforms the state-of-the-art approaches while addressing cold-start problems in POI recommendation.

Customer churn describes terminating a relationship with a business or reducing customer engagement over a specific period. Two main business marketing strategies play vital roles to increase market share dollar-value: gaining new and preserving existing customers. Customer acquisition cost can be five to six times that for customer retention, hence investing in customers with churn risk is smart. Causal analysis of the churn model can predict whether a customer will churn in the foreseeable future and assist enterprises to identify effects and possible causes for churn and subsequently use that knowledge to apply tailored incentives. This paper proposes a framework using a deep feedforward neural network for classification accompanied by a sequential pattern mining method on high-dimensional sparse data. We also propose a causal Bayesian network to predict cause probabilities that lead to customer churn. Evaluation metrics on test data confirm the XGBoost and our deep learning model outperformed previous techniques. Experimental analysis confirms that some independent causal variables representing the level of super guarantee contribution, account growth, and customer tenure were identified as confounding factors for customer churn with a high degree of belief. This paper provides a real-world customer churn analysis from current status inference to future directions in local superannuation funds.