A multiresolution data model for improving simulation I/O performance

In this paper we propose a new index structure, called the Bkd-tree, for indexing large multi-dimensional point data sets. The Bkd-tree is an I/O-efficient dynamic data structure based on the kd-tree. We present the results of an extensive experimental study showing that unlike previous attempts on making external versions of the kd-tree dynamic, the Bkd-tree maintains its high space utilization and excellent query and update performance regardless of the number of updates performed on it.

The task allocation problem with constant communication

Clustering large data sets of high has always been a challenge for clustering algorithms. Many recently developed clustering algorithms have attempted to address either handling data sets with a very large number of records and/or with a very high number of dimensions. We provide a discussion of the advantages and limitations of existing algorithms when they operate on very large multidimensional data sets. To simultaneously overcome both the curse of dimensionality and the scalability problems associated with large amounts of data, we propose a new clustering algorithm called O-Cluster. O-Cluster combines a novel active sampling technique with an axis-parallel partitioning strategy to identify continuous areas of high density in the input space. The method operates on a limited memory buffer and requires at most a single scan through the data. We demonstrate the high quality of the obtained clustering solutions, their robustness to noise, and O-Cluster's excellent scalability.

In this paper, we investigate the problem of enabling block level parallelism, for multi-dimensional data sets, with arbitrary but static causal dependency between blocks that constitute the data set. As the use of video and other multi-dimensional data sets becomes more common place and the algorithms used to process them become more complex, there is greater need for effective parallelization schemes. We describe a method for synchronizing the execution of multiple processors to respect the dependency structure and calculate the total processing time as a function of the number of parallel processors. We also provide an algorithm to calculate the optimal starting times for each processor which enables them to continuously process blocks without the need for synchronizing with other processors, under the assumption that the time to process each block is fixed.

Applications that query into very large multidimensional datasets are becoming more common. Many self-describing scientific data file formats have also emerged, which have structural metadata to help navigate the multi-dimensional arrays that are stored in the files. The files may also contain application-specific semantic metadata. In this paper, we discuss efficient methods for performing searches for subsets of multi-dimensional data objects, using semantic information to build multidimensional indexes, and group data items into properly sized chunks to maximize disk I/O bandwidth. This work is the first step in the design and implementation of a generic indexing library that will work with various high-dimension scientific data file formats containing semantic information about the stored data. To validate the approach, we have implemented indexing structures for NASA remote sensing data stored in the HDF format with a specific schema (HDF-EOS), and show the performance improvements that are gained from indexing the datasets, compared to using the existing HDF library for accessing the data.

Efficient power management in smart grid relies on collecting fine-grained power consumption data from users. However, these data may reveal sensitive information about individuals' habits and lifestyles. Various multidimensional data aggregation schemes leveraging public key encryption (PKE) algorithms have been proposed to address this problem. Never theless, most of these schemes come with significant performance costs. In addition, if the secret key of a smart meter was leaked, the confidentiality of encrypted user power data could be at risk. In this article, we propose a lightweight, multidimensional, and privacy-preserving data aggregation scheme with fault-tolerance and key-leakage resilience for smart grid without relying on a trusted third party (TTP), named FKLM-PDA, in which a novel data packaging method that transforms users' multidimensional data into a one-dimensional format is designed, enabling data center parse aggregated results in each dimension, reducing computation and communication costs. For better efficiency, an effective encryption algorithm is proposed to replace the expensive additive homomorphic PKE, like the Paillier cryptosystem, which combines a random masking with secret-sharing based key separation, ensuring threshold key-leakage resilience under a bounded, non-colluding leakage model. Furthermore, not only does FKLM-PDA enhance the fault tolerance mechanism of data transmission from smart meters to a corresponding fog node, but also it supports dynamic user management for joining and exiting improving scalability. Security analysis confirms that FKLM-PDA is privacy-preserving and secure while guaranteeing key-leakage resilience, fault tolerance, authentication, and data integrity. Through performance evaluations, FKLM-PDA outper forms the existing schemes and is superior in computation and functional in communication.

We investigate the use of biased sampling according to the density of the data set to speed up the operation of general data mining tasks, such as clustering and outlier detection in large multidimensional data sets. In density-biased sampling, the probability that a given point will be included in the sample depends on the local density of the data set. We propose a general technique for density-biased sampling that can factor in user requirements to sample for properties of interest and can be tuned for specific data mining tasks. This allows great flexibility and improved accuracy of the results over simple random sampling. We describe our approach in detail, we analytically evaluate it, and show how it can be optimized for approximate clustering and outlier detection. Finally, we present a thorough experimental evaluation of the proposed method, applying density-biased sampling on real and synthetic data sets, and employing clustering and outlier detection algorithms, thus highlighting the utility of our approach.

Due to the increasing volume of spatio-temporal data generated from remote sensing, sensor networks and computational simulation, there is a need for a generic, domain-independent framework for spatio-temporal data analysis. This research presents a generic set of data processing and manipulation tools for spatio-temporal raster data called multidimensional map algebra (MMA). MMA is an extension of conventional map algebra that operates not only on data that are two-dimensional in space but also on data that are: (1) one-dimensional in time; (2) both two-dimensional in space and one-dimensional in time; (3) three-dimensional in space; and (4) both three-dimensional in space and one-dimensional in time. MMA data types, neighborhoods, lags, and functions are presented, including rules for combining data types of different dimensionality within local, focal, and zonal functions. A prototype implementation in JAVA is provided as a demonstration and syntax specification for the functions. Challenges to continued development of MMA include performance and efficiency issues for processing very large multidimensional data sets.

In this paper, we tackle a novel problem of ranking multivalued objects, where an object has multiple instances in a multidimensional space, and the number of instances per object is not fixed. Given an ad hoc scoring function that assigns a score to a multidimensional instance, we want to rank a set of multivalued objects. Different from the existing models of ranking uncertain and probabilistic data, which model an object as a random variable and the instances of an object are assumed exclusive, we have to capture the coexistence of instances here. To tackle the problem, we advocate the semantics of favoring widely preferred objects instead of majority votes, which is widely used in many elections and competitions. Technically, we borrow the idea from Borda Count (BC), a well-recognized method in consensus-based voting systems. However, Borda Count cannot handle multivalued objects of inconsistent cardinality, and is costly to evaluate top (k) queries on large multidimensional data sets. To address the challenges, we extend and generalize Borda Count to quantile-based Borda Count, and develop efficient computational methods with comprehensive cost analysis. We present case studies on real data sets to demonstrate the effectiveness of the generalized Borda Count ranking, and use synthetic and real data sets to verify the efficiency of our computational method.

In-flight icing is a critical technical issue for aircraft safety and, in particular, the droplet impingement areas on aircraft surfaces must be investigated for anti-/de-icing devices. As a step toward the prediction of droplet impingement on aircraft, an Eulerian-based droplet impingement code that provides collection efficiency for air flows around an airfoil containing water droplets is developed. A computational fluid dynamics (CFD) solver was also developed to solve the clean airflow. Then, a proper orthogonal decomposition (POD) method, a reduced order model (ROM), that optimally captures the energy content from a large multi-dimensional data set is utilized to efficiently predict the collection efficiency and ice accretion shapes on an airfoil following the mean volume diameter, liquid water contents and angle of attacks. As a result it is shown that the collection efficiency and ice shapes were in good agreement with the simulated and predicted results.

The most interesting and challenging gene expression data sets to analyze are large multidimensional data sets that contain expression values for many genes across multiple conditions. In these data sets the use of scientific text can be particularly useful, since there are a myriad of genes examined under vastly different conditions, each of which may induce or repress expression of the same gene for different reasons. There is an enormous complexity to the data that we are examining—each gene is associated with dozens if not hundreds of expression values as well as multiple documents built up from vocabularies consisting of thousands of words. In Section 2.4 we reviewed common gene expression strategies, most of which revolve around defining groups of genes based on common profiles. A limitation of many gene expression analytic approaches is that they do not incorporate comprehensive background knowledge about the genes into the analysis. We present computational methods that leverage the peer-reviewed literature in the automatic analysis of gene expression data sets. Including the literature in gene expression data analysis offers an opportunity to incorporate background functional information about the genes when defining expression clusters. In Chapter 5 we saw how literature- based approaches could help in the analysis of single condition experiments. Here we will apply the strategies introduced in Chapter 6 to assess the coherence of groups of genes to enhance gene expression analysis approaches. The methods proposed here could, in fact, be applied to any multivariate genomics data type. The key concepts discussed in this chapter are listed in the frame box. We begin with a discussion of gene groups and their role in expression analysis; we briefly discuss strategies to assign keywords to groups and strategies to assess their functional coherence. We apply functional coherence measures to gene expression analysis; for examples we focus on a yeast expression data set. We first demonstrate how functional coherence can be used to focus in on the key biologically relevant gene groups derived by clustering methods such as self-organizing maps and k-means clustering.

We present a computational environment for Fast Analysis of multidimensional NMR DAta Sets (FANDAS) that allows assembling multidimensional data sets from a variety of input parameters and facilitates comparing and modifying such "in silico" data sets during the various stages of the NMR data analysis. The input parameters can vary from (partial) NMR assignments directly obtained from experiments to values retrieved from in silico prediction programs. The resulting predicted data sets enable a rapid evaluation of sample labeling in light of spectral resolution and structural content, using standard NMR software such as Sparky. In addition, direct comparison to experimental data sets can be used to validate NMR assignments, distinguish different molecular components, refine structural models or other parameters derived from NMR data. The method is demonstrated in the context of solid-state NMR data obtained for the cyclic nucleotide binding domain of a bacterial cyclic nucleotide-gated channel and on membrane-embedded sensory rhodopsin II. FANDAS is freely available as web portal under WeNMR ( http://www.wenmr.eu/services/FANDAS ).

The skyline query operation (also called the “maximum vector problem”) is used to identify potentially interesting or useful data points in large sets of multi-dimensional data. When the data change over time (through addition and subtraction of points), this is called the “continuous skyline” query. The 2015 MEMOCODE Design Contest problem is to implement a system to efficiently compute the continuous skyline of dynamic data. Contestants were given one month to develop a system to perform the skyline query, aiming to maximize performance or cost-adjusted performance. Teams were encouraged to consider a variety of computational targets including CPUs, FPGAs, and GPGPUs. The two winning teams, which have been invited to contribute papers describing their techniques, combined careful algorithmic and implementation optimizations; both implemented the system on multicore CPUs.

This dissertation focuses on the problem of assigning the modules of a program to the processors in a distributed system with the goal of minimizing the overall cost of running the program. The cost depends on the execution times of the modules on the processors and on the cost of communication between modules. This module allocation problem arises in a variety of situations where one is interested in making optimum use of available computer resources. The general module allocation problem is intractable; however it becomes polynomially-solvable when the communication graph is restricted. In this dissertation, we restrict our attention to k-trees. As the first problem, we study parametric module allocation on partial k-trees. We allow the costs, both execution and communication, to vary linearly as functions of a real parameter t. We show that if the number of processors is fixed, the sequence of optimum assignments that are obtained, as t varies from zero to infinity, can be constructed in polynomial time. As an auxiliary result, we develop a linear-time algorithm to find a separator in a k-tree. We discuss the implications of our results for parametric versions of the weighted vertex cover, independent set, and 0-1 quadratic programming problems on partial k-trees. Next, we consider two variants of the assignment problem. The first problem is to find a minimum-cost assignment when one of the processors has a limited memory. The second is to find an assignment that minimizes the maximum processor load. We present exact dynamic programming algorithms for both problems, which lead to approximation schemes for the case where the communication graph is a partial k-tree. Faster algorithms are presented for trees with uniform costs. In contrast to these results, we show that, for arbitrary graphs, no fully polynomial time approximation schemes exist unless P = NP. Both dynamic programming algorithms have been implemented. The implementation details and our experimental results are presented.

With the highly demanded requirements for manipulating large scientific datasets, scientists are in need of flexible cluster-level software to execute fast scientific data analysis. In this paper, we discuss whether the Apache Spark framework is suitable for scientific data management. We present our system SparkArray, which extends Spark with a multidimensional array data model and a set of common used array operations (e.g., filter, subarray, smooth and join). We present analysis and performance evaluation results on different implementation methods for executing array operations on clusters. We also compared the performance of SparkArray to SciDB, a recent scientific database management system, using the workloads of the Standard Science DBMS Benchmark (SS-DB). The results show that SparkArray is feasible alternative solution for large-scale scientific data management, especially when scientists require fast data loading or one-time data analysis on large scientific datasets.