Task-parallel in situ temporal compression of large-scale computational fluid dynamics data

RPCA (Robust Principal Component Analysis) recovers sparse and low rank components from the original observation data, RPCA commonly uses ADM (Alternate Direction Method) for solving, the efficiency of algorithm depends on the nuclear norm optimization solution, that is SVD. And the application of RPCA in computer vision, image and video and so on, large amounts of data, such as images and video, make it difficult for large-scale data SVD. In this paper, we used the random matrix algorithm to improve the SVD, respectively the Count Sketch algorithm, the Prototype Randomized k-SVD algorithm and the Faster Randomized k-SVD algorithm, The main idea is to reduce the size of the original large-scale data matrix and sample randomly. Using the random projection algorithm to obtain an approximation of the original matrix, and operate QR decomposition of the approximate matrix, get the unitary matrix corresponding to the approximate matrix, and do the corresponding SVD, finally we can get the results which is similar to the original matrix calculation. Obtaining approximation of the original data matrix. But the cost time and space have been greatly optimized. Simulation experiments based on single image and video foreground detection show that the proposed methods can greatly improve the efficiency of RPCA iterative optimization.

In this paper, a obfuscation-based technique namely, AROA based BMCG method is developed for secure data transmission in cloud. Initially, the input data with the mixed attributes is provided to the privacy preservation process directly, where the data matrix and bilinear map coefficient generation co-efficient is multiplied through Hilbert space-based tensor product. Here, bilinear map co-efficient is the new co-efficient proposed to multiply with original data matrix and the OB-MECC Encryption is utilized in the privacy preservation phase to maintain the security of the data. The derivation of bilinear map co-efficient is used to handle both the utility and the sensitive information. The new algorithm called, AROA is developed by integrating the ALO with ROA. The performance and the comparative analysis of the proposed AROA based BMCG method is done using the metrics, such as accuracy and information loss. The proposed AROA based BMCG method obtained a maximal accuracy of 94% and minimal information loss of 6% respectively.

With massive high-dimensional data now commonplace in research and industry, there is a strong and growing demand for more scalable computational techniques for data analysis and knowledge discovery. Key to turning these data into knowledge is the ability to learn statistical models with high interpretability. Current methods for learning statistical models either produce models that are not interpretable or have prohibitive computational costs when applied to massive data. In this paper we address this need by presenting a scalable algorithm for partial least squares regression (PLS), which we call compression-based PLS (cPLS), to learn predictive linear models with a high interpretability from massive high-dimensional data. We propose a novel grammar-compressed representation of data matrices that supports fast row and column access while the data matrix is in a compressed form. The original data matrix is grammar-compressed and then the linear model in PLS is learned on the compressed data matrix, which results in a significant reduction in working space, greatly improving scalability. We experimentally test cPLS on its ability to learn linear models for classification, regression and feature extraction with various massive high-dimensional data, and show that cPLS performs superiorly in terms of prediction accuracy, computational efficiency, and interpretability.

The complex formation between a new synthesized Schiff base and the cations Ni2+, Co2+, Cu2+, and Zn2+ in ethanol, dimethylformamide (DMF), and acetonitrile (AN) was studied spectrophotometrically using rank annihilation factor analysis (RAFA). According to molar ratio data and Job's plots, the stoichiometry of complexation between the Schiff base and the cations Co2+, Ni2+, and Zn2+ was 1:2, and that between the Schiff base and Cu2+ in all solvents and Zn2+ in AN was 1:1. Formation constants of these complexes were derived using RAFA on spectrophotometric data. In this process the contribution of the ligand is removed from the absorbance data matrix when the complex stability constant acts as an optimizing object, and simply combined with the pure spectrum of ligand, the rank of the original data matrix can be reduced by one by annihilating the information of the ligand from the original data matrix. Also the effect of surfactants such as sodium dodecyl sulfate (SDS), cetyltrimethylammonium bromide (CTAB), and Triton X-100 on the complex formation constant of Cu2+ with the Schiff base in DMF was investigated.

Whether task execution results of the smart meter are successful is Boolean and task execution has time sequence. By regarding the data of task execution ability of the smart meter as time series and Boolean, a time-domain extended principal component analysis is proposed to evaluate the smart meter task execution ability in this paper. Firstly, the results of smart meter task execution are extended in the time domain to form the original phasor data matrix by converting Boolean quantity to amplitude and time-series information to phasor phase information. Then, principal component analysis is used to process the original phasor data matrix to determine the key indexes affecting the task execution ability of the smart meter and to establish an index system. Finally, analyzation of historical data of remote payment control of the smart meter in Jibei shows that the key indexes proposed in this paper are reasonable and accurate and the tasks of the smart meter can be guaranteed to be executed efficiently and reliably.

Rank annihilation factor analysis for spectrophotometric study of complex formation equilibria

Representation of large data sets became a key question of many scientific disciplines in the last decade. Several approaches for network visualization, data ordering and coarse-graining accomplished this goal. However, there was no underlying theoretical framework linking these problems. Here we show an elegant, information theoretic data representation approach as a unified solution of network visualization, data ordering and coarse-graining. The optimal representation is the hardest to distinguish from the original data matrix, measured by the relative entropy. The representation of network nodes as probability distributions provides an efficient visualization method and, in one dimension, an ordering of network nodes and edges. Coarse-grained representations of the input network enable both efficient data compression and hierarchical visualization to achieve high quality representations of larger data sets. Our unified data representation theory will help the analysis of extensive data sets, by revealing the large-scale structure of complex networks in a comprehensible form.

Any high-dimensional data arising from practical applications usually contains irrelevant features that may impact on the performance of existing subspace clustering methods. This paper proposes a novel subspace clustering method which reconstructs the feature matrix by the means of unsupervised feature selection (UFS) to achieve a better dictionary for subspace clustering (SC). Different from most existing clustering methods, the proposed approach uses the reconstructed feature matrix as the dictionary rather than the original data matrix. As the feature matrix reconstructed by representative features is more discriminative and closer to the ground-truth, it results in improved performance. The corresponding non-convex optimization problem is effectively solved using the half-quadratic and augmented Lagrange multiplier methods. Extensive experiments on four real datasets demonstrate the effectiveness of the proposed method.

For the High Performance Linpack (HPL) benchmark at the coming Exascale and beyond, silent errors like bit flipping in memory are expected to become inevitable. However, since bit flipping errors are difficult to be detected and located, their impact to the numerical correctness of HPL has not been evaluated thoroughly and quantitatively, while the impact at Exascale is especially susceptible. In this paper, an initial quantitative analysis of the impact of bit flipping errors to the numerical correctness of HPL has been investigated. To validate the numerical correctness of computed solution using HPL, there is a residual check after the approximate solution obtained. This paper has shown that in the case of only one bit flipping to any element in the original data matrix, if the flipped position is not the leading position of exponent, the residual check in HPL will almost surely pass at the scale of Exa flops and beyond. Experiments on modified HPL in single precision at small scales have verified the theoretical results in double precision at Exascale. The results obtained in this paper can provide a better understanding to the impact of bit flipping errors to numerical correctness of scientific computing applications.

A method for the construction of virtual reality spaces for visual data mining using multi-objective optimization with genetic algorithms on neural networks is presented. Two neural network layers (output and last hidden) are used for the construction of simultaneous solutions for: a supervised classification of data patterns and the computation of two unsupervised similarity structure preservation measures between the original data matrix and its image in the new space. A set of spaces is constructed from selected solutions along the Pareto front which enables the understanding of the internal properties of the data based on visual inspection of non-dominating spaces with different properties. This strategy represents a conceptual improvement over spaces computed by single-objective optimization. The presented approach is domain independent and is illustrated with an application to the study of hydrochemical properties of ice and water samples from the Arctic.

In this paper we propose a new method to deal with missingness in categorical data. The new proposal is a forward imputation procedure and is presented in the context of the Nonlinear Principal Component Analysis, used to obtain indicators from a large dataset. However, this procedure can be easily adopted in other contexts, and when other multivariate techniques are used. We discuss the statistical features of our imputation technique in connection with other treatment methods which are popular among Nonlinear Principal Component Analysis users. The performance of our method is then compared to the other methods through a simulation study which involves the application to a real dataset extracted from the Euro-barometer survey. Missing data are created in the original data matrix and then the comparison is performed in terms of how close the Nonlinear Principal Component Analysis outcomes from missing data treatment methods are to the ones obtained from the original data. The new procedure is seen to provide better results than the other methods under the different conditions considered.

Any high-dimensional data arising from practical applications usually contains irrelevant features that may impact on the performance of existing subspace clustering methods. This paper proposes a novel subspace clustering method which reconstructs the feature matrix by the means of unsupervised feature selection (UFS) to achieve a better dictionary for subspace clustering (SC). Different from most existing clustering methods, the proposed approach uses the reconstructed feature matrix as the dictionary rather than the original data matrix. As the feature matrix reconstructed by representative features is more discriminative and closer to the ground-truth, it results in improved performance. The corresponding non-convex optimization problem is effectively solved using the half-quadratic and augmented Lagrange multiplier methods. Extensive experiments on four real datasets demonstrate the effectiveness of the proposed method.

Short-wave near-infrared (NIR) spectra of raw milk have been analyzed in the 800–1100 nm region by two-dimensional (2D) correlation spectroscopy. In this study, we have used both the well-known generalized 2D correlation spectroscopy method, which yields correlation coefficients among spectral variances on all the wavelength points (wavelength-wavelength correlation), and a novel sample-sample correlation spectroscopy method, which gives correlation coefficients among the concentration dynamics of the species in the system. The sample-sample correlation spectroscopy develops correlation maps with the samples on the axes. First, a set of 34 spectra was ordered according to increasing fat content, while all other milk components varied freely. Both synchronous and asynchronous wavelength-wavelength correlation maps have shown strong baseline changes despite the use of multiplicative scatter correction as a pretreatment. Bands at 930 and 970 nm due to fat and water, respectively, have been found to be the most significant spectral features. The synchronous sample-sample correlation map was calculated from the original data matrix and was compared with the outer product of the fat concentration vector. The comparison has revealed that the main spectral variances in the raw milk spectra in the short-wave NIR region are due to fat. The same procedure was repeated for a set of 15 samples that contained a constant fat content and were ordered according to increasing protein content. Poor agreement was found between the outer product of the protein concentration vector and the synchronous sample-sample correlation map. This result suggests that the spectral variances in raw milk spectra that have a constant fat content are due not exclusively to proteins but also due to other milk components. Comparisons of partial least-squares (PLS) regression analysis of fat and proteins and both sample-sample and wavelength-wavelength 2D correlation analyses of raw milk spectra have been made.

Multi-objective evolutionary optimization for constructing neural networks for virtual reality visual data mining: Application to geophysical prospecting

Microarray data biclustering is very important for the research on gene regulatory mechanisms. Genes which exhibit similar patterns are often functionally related. In this paper a novel bicluster detection method is proposed. It makes use of one of the existing traditional clustering algorithms such as K-means as an intermediate tool to do data clustering with the submatrices created from the original data matrix. Especially, in order to save the memory storage requirement, reduce the useless clustering processing and accelerate the bicluster detection speed, a clustering and verification combined algorithm is applied. The former helps to find out the row numbers where possible biclusters lie in, while the latter efficiently speed up the detection processing. Based on a characteristic of bicluster, the biclusters are detected one by one. At the end of the paper experiment with the simulated data are presented.KeywordsBicluster detectionK-meansGene expression data