Gaining understanding of multivariate and multidimensional data through visualization

Data with spatio-temporal structure can arise in many contexts, therefore a considerable interest in modelling these data has been generated, but the complexity of spatio-temporal models, together with the size of the dataset, results in a challenging task. The modelization is even more complex in presence of multivariate data. Since some modelling problems are more natural to think through in functional terms, even if only a finite number of observations is available, treating the data as functional can be useful (Berrendero et al. in Comput Stat Data Anal 55:2619–2634, 2011). Although in Ramsay and Silverman (Functional data analysis, 2nd edn. Springer, New York, 2005) the case of multivariate functional data is also contemplated, they do not cope with more than one dimension (only time is considered as covariate) in the domain of the considered functions. In estimating functional data through smoothing methods, a proper framework for incorporating space-time structures can be found in the generalized additive models (GAM), while classical dimension reduction techniques for functional data lead to functional principal component analysis (FPCA). In a previous work Ruggieri et al. (J Appl Stat 40(4):795–807, 2013) extended temporal FPCA, that is FPCA on data modelled as functions of the one-dimensional time, to multivariate (more than one variable, in our case pollutants) context. In this paper the computational aspects of FPCA are extended to more than one dimension: space (long, lat) and/or space-time; moreover, multidimensional (spatial, spatio-temporal) FPCA is extended to multivariate case. In order to provide a generalization of FPCA to multidimensional (spatio-temporal) and simultaneously multivariate data, we link GAM models together with the approach proposed by Ramsay and Silverman (2005). The paper describes all the computational details useful to implement this approach, while its effectiveness will be shown by a multivariate spatio-temporal environmental dataset, whose structure is actually very common in literature.

Methods of qualitative analysis of multidimensional data using visualization of this data consist in using the transformation of a multidimensional space into a two-dimensional one. In this way, multidimensional complicated data can be presented on a two-dimensional computer screen. This allows to conduct the qualitative analysis of this data in a way which is the most natural for people, through the sense of sight. The application of complex algorithms targeted to search for multidimensional data of specific properties can be replaced with such a qualitative analysis. Some qualitative characteristics are simply visible in the two-dimensional image representing this data. The new perspective-based observational tunnels method is an example of the multidimensional data visualization method. This method was used in this paper to present and analyze the real set of seven-dimensional data describing coal samples obtained from two hard coal mines. This paper presents for the first time the application of perspective-based observational tunnels method for the evaluation of possibilities to divide the multidimensional space of coal samples by their susceptibility to fluidal gasification. This was performed in order to verify whether it will be possible to indicate the possibility of such a division by applying this method. Views presenting the analyzed data, enabling to indicate the possibility to separate areas of the multidimensional space occupied by samples with different applicability for the gasification process, were obtained as a result.

Dimensionality reduction is essential in multidimensional data mining since the dimensionality of real time data could easily reach higher dimensions. Most recent efforts on dimensionality reduction, however, are not adequate to multidimensional data due to lack of scalability. In this paper, we improve the performance of the fractional cuckoo search algorithm by utilizing the FCM function as objective function and we utilize the FCM operator which helps to improve the fitness value of worse solution. The fractional cuckoo search (FCS) algorithm is modified to reduce the dimension and select the best dimension. Once the high dimensional data is reduced in to low dimensional data, then the data is supplied to the clustering algorithm to make the partition easily. Finally, the experimentation is made with synthetic and real datasets and has been we have proved that efficiency of the FCM-FCS algorithm is 1.21% better than FPSO algorithm on iris dataset. The proposed algorithm is 0.66% better than FPSO 4.91% better than FCS on wine dataset and 3.4% better than FPSO 0.1% better than FCS on synthetic dataset.

Visualization techniques deal with multidimensional multivariable data sets. We introduce visualization methods for multidimensional data sets, including an effective dimension reduction method for the multivariate genetic algorithm data set.

Enhanced multivariate data fusion and optimized algorithm for comprehensive quality profiling and origin traceability of Chinese jujube.

In this work, we present an effective and scalable system for multivariate volume data visualization and analysis with a novel Transfer Function (TF) interface design that tightly couples parallel coordinates plots (PCP) and MDS-based dimension projection plots. In our system, the PCP visualizes the data distribution of each variate and the MDS plots project features. Together, they are integrated seamlessly to provide flexible feature classification without context switching between different data presentations during the user interaction. The proposed interface enables users to identify relevant correlation clusters and assign optical properties on them. To further support large scale multivariate volume data visualization and analysis, we develop three integrated parallel systems to accelerate the rendering of PCP, the layout of MDS, as well as parallel rendering of multivarite volume data. Our experiments show that the system is effective in multivariate volume data visualization and its performance is scalable for data sets with different sizes and number of variates.

Time series data are significant, and are derived from temporal data, which involve real numbers representing values collected regularly over time. Time series have a great impact on many types of data. However, time series have anomalies. We introduce an anomaly detection paradigm called novel matrix profile (NMP) to solve the all-pairs similarity search problem for time series data in the healthcare. The proposed paradigm inherits the features from two state-of-the-art algorithms: Scalable Time series Anytime Matrix Profile (STAMP) and Scalable Time-series Ordered-search Matrix Profile (STOMP). The proposed NMP caches the output in an easy-to-access fashion for single- and multidimensional data. The proposed NMP can be used on large multivariate data sets and generates approximate solutions of high quality in a reasonable time. It is implemented on a Python platform. To determine its effectiveness, it is compared with the state-of-the-art matrix profile algorithms, i.e., STAMP and STOMP. The results confirm that the proposed NMP provides higher accuracy than the compared algorithms.

Information theory is an outstanding framework for measuring uncertainty, dependence, and relevance in data and systems. It has several desirable properties for real-world applications: naturally deals with multivariate data, can handle heterogeneous data, and the measures can be interpreted. However, it has not been adopted by a wider audience because obtaining information from multidimensional data is a challenging problem due to the curse of dimensionality. We propose an indirect way of estimating information based on a multivariate iterative Gaussianization transform. The proposed method has a multivariate-to-univariate property: it reduces the challenging estimation of multivariate measures to a composition of marginal operations applied in each iteration of the Gaussianization. Therefore, the convergence of the resulting estimates depends on the convergence of well-understood univariate entropy estimates, and the global error linearly depends on the number of times the marginal estimator is invoked. We introduce Gaussianization-based estimates for Total Correlation, Entropy, Mutual Information, and Kullback-Leibler Divergence. Results on artificial data show that our approach is superior to previous estimators, particularly in high-dimensional scenarios. We also illustrate the method's performance in different fields to obtain interesting insights. We make the tools and datasets publicly available to provide a test bed for analyzing future methodologies.

Short-term multivariate time series load data forecasting at low-voltage level using optimised deep-ensemble learning-based models

With the rise of Big Data, the challenge for modern multidimensional data analysis and visualization is how it grows very quickly in size and complexity. In this paper, we first present a classification method called the 5Ws Dimensions which classifies multidimensional data into the 5Ws definitions. The 5Ws Dimensions can be applied to multiple datasets such as text datasets, audio datasets and video datasets. Second, we establish a Pair-Density model to analyze the data patterns to compare the multidimensional data on the 5Ws patterns. Third, we created two additional parallel axes by using pair-density for visualization. The attributes has been shrunk to reduce data over-crowding in pair-density parallel coordinates. This has achieved more than 80i¾?% clutter reduction without the loss of information. The experiment shows that our model can be efficiently used for Big Data analysis and visualization.

Supporting visual analytics of multiple large-scale multidimensional data sets requires a high degree of interactivity and user control beyond the conventional challenges of visualizing such data sets. We present the DataMeadow, a visual canvas providing rich interaction for constructing visual queries using graphical set representations called DataRoses. A DataRose is essentially a starplot of selected columns in a data set displayed as multivariate visualizations with dynamic query sliders integrated into each axis. The purpose of the DataMeadow is to allow users to create advanced visual queries by iteratively selecting and filtering into the multidimensional data. Furthermore, the canvas provides a clear history of the analysis that can be annotated to facilitate dissemination of analytical results to stakeholders. A powerful direct manipulation interface allows for selection, filtering, and creation of sets, subsets, and data dependencies. We have evaluated our system using a qualitative expert review involving two visualization researchers. Results from this review are favorable for the new method.

Visual analytics of multidimensional multivariate data is a challenging task because of the difficulty in understanding metrics in attribute spaces with more than three dimensions. Frequently, the analysis goal is not to look into individual records but to understand the distribution of the records at large and to find clusters of records with similar attribute values. A large number of (typically hierarchical) clustering algorithms have been developed to group individual records to clusters of statistical significance. However, only few visualization techniques exist for further exploring and understanding the clustering results. We propose visualization and interaction methods for analyzing individual clusters as well as cluster distribution within and across levels in the cluster hierarchy. We also provide a clustering method that operates on density rather than individual records. To not restrict our search for clusters, we compute density in the given multidimensional multivariate space. Clusters are formed by areas of high density. We present an approach that automatically computes a hierarchical tree of high density clusters. To visually represent the cluster hierarchy, we present a 2D radial layout that supports an intuitive understanding of the distribution structure of the multidimensional multivariate data set. Individual clusters can be explored interactively using parallel coordinates when being selected in the cluster tree. Furthermore, we integrate circular parallel coordinates into the radial hierarchical cluster tree layout, which allows for the analysis of the overall cluster distribution. This visual representation supports the comprehension of the relations between clusters and the original attributes. The combination of the 2D radial layout and the circular parallel coordinates is used to overcome the overplotting problem of parallel coordinates when looking into data sets with many records. We apply an automatic coloring scheme based on the 2D radial layout of the hierarchical cluster tree encoding hue, saturation, and value of the HSV color space. The colors support linking the 2D radial layout to other views such as the standard parallel coordinates or, in case data is obtained from multidimensional spatial data, the distribution in object space.

When investigating multidimensional data sets with very large numbers of objects and/or a very large number of dimensions, a variety of visualization methods can be employed in order to represent the data effectively and to enable the user to explore the data at different levels of detail. A common strategy for encoding multidimensional data for visual analysis is to use dimensionality reduction techniques that project data from higher dimensions onto a lower-dimensional space. In this paper, we focus on projection techniques that output 2D or 3D scatterplots which can then be used for a range of data analysis tasks. Existing taxonomies for multidimensional data projections focus primarily on tasks in order to evaluate the human perception of class or cluster separation and/or preservation. However, real-world data analysis of complex data sets often includes other tasks besides cluster separation, such as: cluster identification, similarity seeking, cluster ranking, comparisons, counting objects, etc. A contribution of this paper is the identification of subtasks grouped into four main categories of data analysis tasks. We believe that this user-centric task categorization can be used to guide the organization of multidimensional data projection layouts. Moreover, this taxonomy can be used as a guideline for visualization designers when faced with complex data sets requiring dimensionality reduction. Our taxonomy aims to help designers evaluate the effectiveness of a visualization system by providing an expanded range of relevant tasks. These tasks are gathered from an extensive study of visual analytics projects across real-world application domains, all of which involve multidimensional projection. In addition to our survey of tasks and the creation of the task taxonomy, we also explore in more detail specific examples of how to represent data sets effectively for particular tasks. These case studies, while not exhaustive, provide a framework for how specifically to reason about tasks and to decide on visualization methods. That is, we believe that this taxonomy will help visualization designers to determine which visualization methods are appropriate for specific multidimensional data projection tasks.

We introduce a new measure of univariate skewness of a distribution or data based on quantiles and by using the concepts of even and odd functions. Based on this new measure, we then suggest an approach to define the multivariate skewness for the multivariate distributions and multidimensional data and accordingly suggest a measure for it. Using numerous data sets, we illustrate that Mardia’s measure of multivariate skewness appears to be ambiguous in what it actually measures and show that our measure not only has an intuitive appeal, it also unambiguously quantifies what one would view as the multivariate skewness. Approach presented here is data analytic and can be implemented on a computer. Based on the idea of orthogonal transformation of the data, we also suggest another multivariate measure of skewness which may be simpler to compute.

rCOSA is a software package interfaced to the R language. It implements statistical techniques for clustering objects on subsets of attributes in multivariate data. The main output of COSA is a dissimilarity matrix that one can subsequently analyze with a variety of proximity analysis methods. Our package extends the original COSA software (Friedman and Meulman, 2004) by adding functions for hierarchical clustering methods, least squares multidimensional scaling, partitional clustering, and data visualization. In the many publications that cite the COSA paper by Friedman and Meulman (2004), the COSA program is actually used only a small number of times. This can be attributed to the fact that this original implementation is not very easy to install and use. Moreover, the available software is out-of-date. Here, we introduce an up-to-date software package and a clear guidance for this advanced technique. The software package and related links are available for free at: https://github.com/mkampert/rCOSA.