Resampling Techniques in Cluster Analysis: Is Subsampling Better Than Bootstrapping?

Among the methods proposed for identifying the number of latent traits in multidimensional IRT models, DETECT has attracted the attention of both methodologists and applied researchers as a nonparametric counterpart to other procedures. The current study investigated the overall performance of the DETECT procedure and its outcomes using a real-data sampling design recommended by MacCallum (2003) and compared the results from a purely simulated data set that was generated with a well-specified “perfect” model. The comparison revealed that the sampling behavior of the maximized DETECT value and R-ratio statistics was quite robust to minor factors and other model misspecifications that potentially exist in the real data set, as there were negligible differences between the results of the real and simulated data sets. Item classification accuracy was also nearly identical for the real and simulated data sets. The accuracy of the identified number of dimensions reported by DETECT was the only outcome with an obvious difference between the purely simulated data set and the real data set. While the difference was small for smaller sample sizes, the identified number of dimensions was more accurate for larger sample sizes when the population data set was purely simulated. In many instances, exploratory DETECT analysis outperformed the cross-validated DETECT analysis in terms of overall accuracy.

Extensive Copy-Number Variation of the Human Olfactory Receptor Gene Family

There is a strong need for synthetic yet realistic distribution system test data sets that are as diverse, large, and complex to solve as real systems. Such data sets can facilitate the development of advanced algorithms and the assessment of emerging distributed energy resources while avoiding the need to acquire proprietary critical infrastructure or private data. Such synthetic data sets, however, are useful only if they are realistic enough to look and behave similarly to actual systems. This paper presents a comprehensive framework for validating synthetic distribution data sets using a three-pronged statistical, operational, and expert validation approach. It also presents a set of statistical and operational metric targets for achieving realistic data sets based on detailed characterization of more than 10,000 real U.S. utility feeders. The paper demonstrates the use of the proposed validation approach to validate three large-scale synthetic data sets developed by the authors representing Santa Fe, New Mexico; Greensboro, North Carolina; and the San Francisco Bay Area, California.

With the development of the information age, the awareness of archives informatization of governments, organizations and groups at all levels is constantly increasing. The key content extraction of scanned archive images has been widely concerned by researchers. Based on the deep learning method, we extract text and key elements (ID photo, seal, date of birth) by analyzing the archive images. Based on the real archive images data set and synthetic archive images data set, we use the PP-OCR [1] model to finish the text structured extraction and YOLOv5 [2] to train a target detection model of ID photos, seals, and dates of birth to achieve the extraction of key elements. Since the real data set cannot be made public, we use ordinary archive data for explanation.KeywordsArchive imagesSynthetic archive imagesPP-OCRYOLOv5Text structured extractionKey elements extraction

The matrix factorization (MF) technique has been widely adopted for solving the rating prediction problem in recommender systems. The MF technique utilizes the latent factor model to obtain static user preferences (user latent vectors) and item characteristics (item latent vectors) based on historical rating data. However, in the real world, user preferences are not static but full of dynamics. Though there are several previous works that addressed this time-varying issue of user preferences, it seems (to the best of our knowledge) that none of them are specifically designed for tracking concept drift in individual user preferences . Motivated by this, we develop a temporal MF approach for tracking concept drift in each individual user latent vector. There are two key innovative steps in our approach: 1) we develop a modified stochastic gradient descent method to learn an individual user latent vector at each time step and 2) by Lasso regression, we learn a linear model for the transition of the individual user latent vectors. We test our method on a synthetic data set and several real data sets. In comparison with the original MF, our experimental results show that our temporal method is able to achieve lower root mean square errors (RMSEs) for both the synthetic and real data sets. One interesting finding is that the performance gain in RMSE is mostly from those users who indeed have concept drift in their user latent vectors at the time of prediction. In particular, for the synthetic data set and the Ciao data set, there are quite a few users with that property and the performance gains for these two data sets are roughly 20% and 5%, respectively.

Clustering is a useful exploratory technique for the analysis of gene expression data. Many different heuristic clustering algorithms have been proposed in this context. Clustering algorithms based on probability models offer a principled alternative to heuristic algorithms. In particular, model-based clustering assumes that the data is generated by a finite mixture of underlying probability distributions such as multivariate normal distributions. The issues of selecting a 'good' clustering method and determining the 'correct' number of clusters are reduced to model selection problems in the probability framework. Gaussian mixture models have been shown to be a powerful tool for clustering in many applications. We benchmarked the performance of model-based clustering on several synthetic and real gene expression data sets for which external evaluation criteria were available. The model-based approach has superior performance on our synthetic data sets, consistently selecting the correct model and the number of clusters. On real expression data, the model-based approach produced clusters of quality comparable to a leading heuristic clustering algorithm, but with the key advantage of suggesting the number of clusters and an appropriate model. We also explored the validity of the Gaussian mixture assumption on different transformations of real data. We also assessed the degree to which these real gene expression data sets fit multivariate Gaussian distributions both before and after subjecting them to commonly used data transformations. Suitably chosen transformations seem to result in reasonable fits. MCLUST is available at http://www.stat.washington.edu/fraley/mclust. The software for the diagonal model is under development. kayee@cs.washington.edu. http://www.cs.washington.edu/homes/kayee/model.

This study discusses the scaling properties of the spatial distribution of the December 26, 2004, Sumatra aftershocks. We estimate the spatial correlation dimension D2 of the epicentral distribution of aftershocks recorded by a local network operated by Geological Survey of India. We estimate the value of D2 for five blocks in the source area by using generalized correlation integral approach. We assess its bias due to finite data points, scaling range, effects of location errors, and boundary effects theoretically and apply it to real data sets. The correlation dimension was computed both for real as well as synthetic data sets that include randomly generated point sets obtained using uniform distributions and mimicking the number of events and outlines of the effective areas filled with epicenters. On comparing the results from the real data and random point sets from simulations, we found the lower limit of bias in D2 estimates from limited data sets to be 0.26. Thus, the spatial variation in correlation dimensions among different blocks using local data sets cannot be directly compared unless the influence of bias in the real aftershock data set is taken into account. They cannot also be used to infer the geometry of the faults. We also discuss the results in order to add constraints on the use of synthetic data and of different approaches for uncertainty analysis on spatial variation of D2. A difference in D2 values, rather than their absolute values, among small blocks is of interest to local data sets, which are correlated with their seismic b values. Taking into account the possible errors and biases, the average D2 values vary from 1.05 to 1.57 in the Andaman–Nicobar region. The relative change in D2 values can be interpreted in terms of clustering and diffuse seismic activity associated with the low and high D2 values, respectively. Overall, a relatively high D2 and low b value is consistent with high-magnitude, diffuse activity in space in the source region of the 2004 Sumatra earthquake.

Problem statement: The processing of the XML queries needs an efficient indexing method. The index generation includes the labeling of the nodes. But for the dynamic data, when the changes are made, the re-computation of the labeling is needed. Approach: Based on the structural indexing used the performance of the query processing system will be affected. The dynamic XML document allows inserting, deleting and updating operations. Results: Suppose for the frequently changed documents, the indexes will also be changed. i.e., it requires the re-computing the labels frequently. This leads to an inconvenient system. To avoid this problem, in the scheme New Labeling Scheme for XML (NLSX), only the small, capital letters and digits are used to generate persistent labels for the nodes in the document. In the proposed system New Labeling Scheme for XML using Unicode Characters, characters from Unicode Characters (NLSXU) are used. Thus it can provide more combinations of the characters for persistent labeling the nodes in the document so that it will very much reduce the space needed to store the labels. Using the proposed scheme NLSXU, the index size of the real world data sets will be greatly reduced by 81% of the existing scheme NLSX. The results shows that the proposed scheme NLSXU will reduce the size of the indexes of the synthetic data sets up to 26, 34, 71 and 95% than the NLSX, LSDX, GRP and SP schemes respectively. Also when compared to LSDX scheme, the NLSXU will reduce the time taken for generating the labels by 96 and 80% for the real world datasets and the synthetic data sets respectively. Conclusion: Finally when compared to NLSX scheme, the NLSXU will reduce the time taken for generating the labels by 66 and 15% for the real world datasets and the synthetic data sets respectively. Thus it will improve the performance of the query system.

Reconstruction of gene regulatory networks (GRNs) from expression data is a significant open problem. Common approaches train a machine learning (ML) model to predict a gene's expression using transcription factors' (TFs') expression as features and designate important features/TFs as regulators of the gene. Here, we present an entirely different paradigm, where GRN edges are directly predicted by the ML model. The new approach, named "SPREd," is a simulation-supervised neural network for GRN inference. Its inputs comprise expression relationships (e.g. correlation, mutual information) between the target gene and each TF and between pairs of TFs. The output includes binary labels indicating whether each TF regulates the target gene. We train the neural network model using synthetic expression data generated by a biophysics-inspired simulation model that incorporates linear as well as non-linear TF-gene relationships and diverse GRN configurations. We show SPREd to outperform state-of-the-art GRN reconstruction tools GENIE3, ENNET, PORTIA, and TIGRESS on synthetic datasets with high co-expression among TFs, similar to that seen in real data. A key advantage of the new approach is its robustness to relatively small numbers of conditions (columns) in the expression matrix, which is a common problem faced by existing methods. Finally, we evaluate SPREd on real data sets in yeast that represent gold-standard benchmarks of GRN reconstruction and show it to perform significantly better than or comparably to existing methods. In addition to its high accuracy and speed, SPREd marks a first step toward incorporating biophysics principles of gene regulation into ML-based approaches to GRN reconstruction. Data and code are available from https://github.com/iiiime/SPREd.

Dimensionality reduction is essential in multidimensional data mining since the dimensionality of real time data could easily reach higher dimensions. Most recent efforts on dimensionality reduction, however, are not adequate to multidimensional data due to lack of scalability. In this paper, we improve the performance of the fractional cuckoo search algorithm by utilizing the FCM function as objective function and we utilize the FCM operator which helps to improve the fitness value of worse solution. The fractional cuckoo search (FCS) algorithm is modified to reduce the dimension and select the best dimension. Once the high dimensional data is reduced in to low dimensional data, then the data is supplied to the clustering algorithm to make the partition easily. Finally, the experimentation is made with synthetic and real datasets and has been we have proved that efficiency of the FCM-FCS algorithm is 1.21% better than FPSO algorithm on iris dataset. The proposed algorithm is 0.66% better than FPSO 4.91% better than FCS on wine dataset and 3.4% better than FPSO 0.1% better than FCS on synthetic dataset.

The Exponentiated Gumbel (EG) distribution has been proposed to capture some aspects of the data that the Gumbel distribution fails to specify. it has an increasing hazard rate. The Exponentiated Gumbel distribution has applications in hydrology, meteorology, climatology, insurance, finance and geology, among many others. In this paper Firstly, the mathematical and statistical characteristics of the gumbel and Exponentiated Gumbel distribution are presented, then the applications of this distributions are studied using the real data set. Its first moment about origin and moments about mean have been obtained and expressions for skewness, kurtosis have been given. Estimation of its parameter has been discussed using the method of maximum likelihood. In the end, two applications of the gumbel and exponentiated gumbel distribution have been discussed with two real lifetime data sets. The results also confirmed the suitability of the Exponentiated Gumbel distribution for real data collection.

An important consideration in clustering is the determination of the correct number of clusters and the appropriate partitioning of a given data set. In this paper, a newly developed point symmetry distance is used to propose a new cluster validity index named Sym -index which provides a measure of symmetricity of the different partitionings of a data set. The index is able to address all the above mentioned issues, viz., determining the number of clusters and evolving the proper partitioning as long as the clusters possess the property of symmetry. A Kd-tree-based data structure is used to reduce the complexity of computing the symmetry distance. Results demonstrating the superiority of the Sym-index in appropriately determining the proper partitioning and the number of clusters, as compared to two other recently proposed measures, namely the PS-index and I-index, are provided for three clustering methods viz., two recently developed genetic algorithm based clustering techniques and the average linkage clustering algorithm. Four artificial data sets and two real life data sets are considered for this purpose. The effectiveness of the proposed validity index is then demonstrated for automatically classifying different landcover regions in remote sensing imagery.

A novel clustering algorithm based on the natural reverse nearest neighbor structure

Simultaneous cancer classification and gene selection with Bayesian nearest neighbor method: An integrated approach

Evaluating data mining procedures: techniques for generating artificial data sets