Improving access to multi-dimensional self-describing scientific datasets

We address in this paper data management techniques for efficiently retrieving requested subsets of large datasets stored on mass storage devices. This problem represents a major bottleneck that can negate the benefits of fast networks, because the time to access a subset from a large dataset stored on a mass storage system is much greater that the time to transmit that subset over a network. This paper focuses on very large spatial and temporal datasets generated by simulation programs in the area of climate modeling, but the techniques developed can be applied to other applications that deal with large multidimensional datasets. The main requirement we have addressed is the efficient access of subsets of information contained within much larger datasets, for the purpose of analysis and. interactive visualization. We have developed data partitioning techniques that partition datasets into ``clusters`` based on analysis of data access patterns and storage device characteristics. The goal is to minimize the number of clusters read from mass storage systems when subsets are requested. We emphasize in this paper proposed enhancements to current storage server protocols to permit control over physical placement of data on storage devices. We also discuss in some detail the aspects of the interface between the application programs and the mass storage system, as well as a workbench to help scientists to design the best reorganization of a dataset for anticipated access patterns.

We investigate the use of biased sampling according to the density of the data set to speed up the operation of general data mining tasks, such as clustering and outlier detection in large multidimensional data sets. In density-biased sampling, the probability that a given point will be included in the sample depends on the local density of the data set. We propose a general technique for density-biased sampling that can factor in user requirements to sample for properties of interest and can be tuned for specific data mining tasks. This allows great flexibility and improved accuracy of the results over simple random sampling. We describe our approach in detail, we analytically evaluate it, and show how it can be optimized for approximate clustering and outlier detection. Finally, we present a thorough experimental evaluation of the proposed method, applying density-biased sampling on real and synthetic data sets, and employing clustering and outlier detection algorithms, thus highlighting the utility of our approach.

A common task in data mining is the visualization of multivariate objects on scatterplots, allowing human observers to perceive subtle inter-relations in the dataset such as outliers, groupings or other regularities. Least- squares multidimensional scaling (MDS) is a well known Exploratory Data Analysis family of techniques that produce dissimilarity or distance preserving layouts in a nonlinear way. In this framework, the issue of visualizing large multidimensional datasets through MDS-based methods is addressed. An original scheme providing very accurate layouts of large datasets is introduced. It is a compromise between the computational complexity O(N5/2) and the accuracy of the solution that makes it suitable both for visualization of fairly large datasets and preprocessing in pattern recognition tasks.

Numerical simulations running on very large High Performance Computer clusters still suffer from the I/O bottleneck. The cost of communication can overwhelm the cost of computation, and scales inversely with the number of processors used in the cluster. In previous work we have developed a multiresolution data model to help improve performance for visualizations of very large multi dimensional scientific data sets. In our approach, the data is represented as a multi level hierarchy. Reconstructive error analysis is used to identify regions in the data where the data loss is greatest. We have incorporated this data model into the OpenGGCM solar wind simulation environment. In this paper, we demonstrate that this approach can reduce the I/O and improve the overall performance of a large numerical simulation environment.

Clustering large data sets of high has always been a challenge for clustering algorithms. Many recently developed clustering algorithms have attempted to address either handling data sets with a very large number of records and/or with a very high number of dimensions. We provide a discussion of the advantages and limitations of existing algorithms when they operate on very large multidimensional data sets. To simultaneously overcome both the curse of dimensionality and the scalability problems associated with large amounts of data, we propose a new clustering algorithm called O-Cluster. O-Cluster combines a novel active sampling technique with an axis-parallel partitioning strategy to identify continuous areas of high density in the input space. The method operates on a limited memory buffer and requires at most a single scan through the data. We demonstrate the high quality of the obtained clustering solutions, their robustness to noise, and O-Cluster's excellent scalability.

The skyline query operation (also called the “maximum vector problem”) is used to identify potentially interesting or useful data points in large sets of multi-dimensional data. When the data change over time (through addition and subtraction of points), this is called the “continuous skyline” query. The 2015 MEMOCODE Design Contest problem is to implement a system to efficiently compute the continuous skyline of dynamic data. Contestants were given one month to develop a system to perform the skyline query, aiming to maximize performance or cost-adjusted performance. Teams were encouraged to consider a variety of computational targets including CPUs, FPGAs, and GPGPUs. The two winning teams, which have been invited to contribute papers describing their techniques, combined careful algorithmic and implementation optimizations; both implemented the system on multicore CPUs.

We describe a novel implementation scheme of multidimensional array for handling large scale high dimensional datasets. The scheme implements a dynamic multidimensional extendible array employing a set of two dimensional extendible arrays. The multidimensional arrays provide many advantages but it has some problems as well. The Traditional Multidimensional array is not dynamic extendible. Again, if the length of dimension and number of dimension of a multidimensional array is large then the address space for the array overflows soon. In this paper, we propose a solution against the essential problem of address space overflow for handling large scale multidimensional datasets using our implementation model. We evaluate our proposed scheme by comparing with Traditional Multidimensional Array (TMA) for different operations and find a reasonable delay on address space overflow with no significant performance degradation.

In this paper we propose a new index structure, called the Bkd-tree, for indexing large multi-dimensional point data sets. The Bkd-tree is an I/O-efficient dynamic data structure based on the kd-tree. We present the results of an extensive experimental study showing that unlike previous attempts on making external versions of the kd-tree dynamic, the Bkd-tree maintains its high space utilization and excellent query and update performance regardless of the number of updates performed on it.

Due to the increasing volume of spatio-temporal data generated from remote sensing, sensor networks and computational simulation, there is a need for a generic, domain-independent framework for spatio-temporal data analysis. This research presents a generic set of data processing and manipulation tools for spatio-temporal raster data called multidimensional map algebra (MMA). MMA is an extension of conventional map algebra that operates not only on data that are two-dimensional in space but also on data that are: (1) one-dimensional in time; (2) both two-dimensional in space and one-dimensional in time; (3) three-dimensional in space; and (4) both three-dimensional in space and one-dimensional in time. MMA data types, neighborhoods, lags, and functions are presented, including rules for combining data types of different dimensionality within local, focal, and zonal functions. A prototype implementation in JAVA is provided as a demonstration and syntax specification for the functions. Challenges to continued development of MMA include performance and efficiency issues for processing very large multidimensional data sets.

In this paper, we tackle a novel problem of ranking multivalued objects, where an object has multiple instances in a multidimensional space, and the number of instances per object is not fixed. Given an ad hoc scoring function that assigns a score to a multidimensional instance, we want to rank a set of multivalued objects. Different from the existing models of ranking uncertain and probabilistic data, which model an object as a random variable and the instances of an object are assumed exclusive, we have to capture the coexistence of instances here. To tackle the problem, we advocate the semantics of favoring widely preferred objects instead of majority votes, which is widely used in many elections and competitions. Technically, we borrow the idea from Borda Count (BC), a well-recognized method in consensus-based voting systems. However, Borda Count cannot handle multivalued objects of inconsistent cardinality, and is costly to evaluate top (k) queries on large multidimensional data sets. To address the challenges, we extend and generalize Borda Count to quantile-based Borda Count, and develop efficient computational methods with comprehensive cost analysis. We present case studies on real data sets to demonstrate the effectiveness of the generalized Borda Count ranking, and use synthetic and real data sets to verify the efficiency of our computational method.

In-flight icing is a critical technical issue for aircraft safety and, in particular, the droplet impingement areas on aircraft surfaces must be investigated for anti-/de-icing devices. As a step toward the prediction of droplet impingement on aircraft, an Eulerian-based droplet impingement code that provides collection efficiency for air flows around an airfoil containing water droplets is developed. A computational fluid dynamics (CFD) solver was also developed to solve the clean airflow. Then, a proper orthogonal decomposition (POD) method, a reduced order model (ROM), that optimally captures the energy content from a large multi-dimensional data set is utilized to efficiently predict the collection efficiency and ice accretion shapes on an airfoil following the mean volume diameter, liquid water contents and angle of attacks. As a result it is shown that the collection efficiency and ice shapes were in good agreement with the simulated and predicted results.

The most interesting and challenging gene expression data sets to analyze are large multidimensional data sets that contain expression values for many genes across multiple conditions. In these data sets the use of scientific text can be particularly useful, since there are a myriad of genes examined under vastly different conditions, each of which may induce or repress expression of the same gene for different reasons. There is an enormous complexity to the data that we are examining—each gene is associated with dozens if not hundreds of expression values as well as multiple documents built up from vocabularies consisting of thousands of words. In Section 2.4 we reviewed common gene expression strategies, most of which revolve around defining groups of genes based on common profiles. A limitation of many gene expression analytic approaches is that they do not incorporate comprehensive background knowledge about the genes into the analysis. We present computational methods that leverage the peer-reviewed literature in the automatic analysis of gene expression data sets. Including the literature in gene expression data analysis offers an opportunity to incorporate background functional information about the genes when defining expression clusters. In Chapter 5 we saw how literature- based approaches could help in the analysis of single condition experiments. Here we will apply the strategies introduced in Chapter 6 to assess the coherence of groups of genes to enhance gene expression analysis approaches. The methods proposed here could, in fact, be applied to any multivariate genomics data type. The key concepts discussed in this chapter are listed in the frame box. We begin with a discussion of gene groups and their role in expression analysis; we briefly discuss strategies to assign keywords to groups and strategies to assess their functional coherence. We apply functional coherence measures to gene expression analysis; for examples we focus on a yeast expression data set. We first demonstrate how functional coherence can be used to focus in on the key biologically relevant gene groups derived by clustering methods such as self-organizing maps and k-means clustering.

In this paper, we study how to visualize large amounts of multidimensional data with a radial visualization. For such a visualization, we study a multi-threaded implementation on the CPU and the GPU. We start by reviewing the approaches that have visualized the largest multidimensional datasets and we focus on the approaches that have used CPU or GPU parallelization. We consider the radial visualizations and we describe our approach (called POIViz) that uses points of interest to determine a layout of a large dataset. We detail its parallelization on the CPU and the GPU. We study the efficiency of this approach with different configurations and for large datasets. We show that it can visualize, in less than one second, millions of data with tens of dimensions, and that it can support "real-time" interactions even for large datasets. We conclude on the advantages and limits of the proposed visualization.

ProDA employs wavelets to support exact, approximate, and progressive OLAP queries on large multidimensional datasets, while keeping update costs relatively low. ProDA not only supports online execution of ad hoc analytical queries on massive datasets, but also extends the set of supported analytical queries to include the entire family of polynomial aggregate queries as well as the new class of plot queries.

The objective of this paper is to identify which combinations of constituents of heterogeneous catalysts are related to which final products from a large combinatorially generated dataset. To this end, data on the composition of catalysts are combined with a set of four catalytic activities, and analyzed using a Principle Component Analysis (PCA). From the first attempt of PCA, a dataset of pentanary‐mixed metal oxides, CraMobNicCodMneOx, is visualized on the planes of principal components (PCs) in a way of linking the composition spread with the activities toward each product. It has been possible to identify three key elements (Mo, Co, and Cr) out of five by PCA. Having these key elements identified from PCA, a ternary system of mixed metal oxide catalysts, MoaCobCrcOx, was analyzed again in the PC space for further relationships between composition of catalysts and activities of final products. These two case studies demonstrate how large multidimensional datasets can be analyzed to identify relevant key parameters for catalyst discovery and optimization.