Spatial online sampling and aggregation

The rise of GPS and broadband-speed wireless devices has led to tremendous excitement about a range of applications broadly characterized as ¿location based services¿. Current database storage systems, however, are inadequate for manipulating the very large and dynamic spatio-temporal data sets required to support such services. Proposals in the literature either present new indices without discussing how to cluster data, potentially resulting in many disk seeks for lookups of densely packed objects, or use static quadtrees or other partitioning structures, which become rapidly suboptimal as the data or queries evolve. As a result of these performance limitations, we built TrajStore, a dynamic storage system optimized for efficiently retrieving all data in a particular spatiotemporal region. TrajStore maintains an optimal index on the data and dynamically co-locates and compresses spatially and temporally adjacent segments on disk. By letting the storage layer evolve with the index, the system adapts to incoming queries and data and is able to answer most queries via a very limited number of I/Os, even when the queries target regions containing hundreds or thousands of different trajectories.

Is power everything? What can we learn from large data sets

Geographic information technologies are evolving from stand-alone systems to a distributed model of independent web services. In parallel, voluminous geographic data are being collected with modern data acquisition techniques such as remote sensing and personal navigation devices. There is an urgent need for effective and efficient methods to integrate and explore relationships between remote sensing and trajectory datasets. When it comes to integration, one would commonly rely on a conventional chain of GIS operations to match trajectory locations to grid values: download grid data, georeference, match each trajectory record to a corresponding image cell, perform overlay, extract cell values for a given location and time and compose values into a resulting table. If one has to deal with large and dynamic spatio-temporal data sets, this approach is clearly unmanageable. We propose an alternative approach: a four-layered system architecture that utilises web services for the integration of trajectory and remote sensing data. We demonstrate how this integration service can be embedded into distributed components for manipulation, analysis and visualisation of geospatial trajectories, using Antarctic iceberg trajectories and wind data as a case study. The prototype can be accessed on the web. Future research will include optimisation and integration of more variables extracted from grid data sets, but the main focus will be on extension of the analytical component by implementing more mechanisms to discover patterns in the integrated data set, and on better visualisation.

The value of protein models obtained with automated protein structure prediction depends primarily on their accuracy. Protein model quality assessment is thus critical to select the model that can best answer biologically relevant questions from an ensemble of predictions. However, despite many advances in the field, different methods capture different types of errors, begging the question of which method to use. We introduce TopScore, a meta Model Quality Assessment Program (meta-MQAP) that uses deep neural networks to combine scores from 15 different primary predictors to predict accurate residue-wise and whole-protein error estimates. The predictions on six large independent data sets are highly correlated to superposition-independent errors in the model, achieving a Pearson's Rall2 of 0.93 and 0.78 for whole-protein and residue-wise error predictions, respectively. This is a significant improvement over any of the investigated primary MQAPs, demonstrating that much can be gained by optimally combining different methods and using different and very large data sets.

Spatial sciences are confronted with increasing amounts of high-dimensional data. These data commonly exhibit spatial and temporal dimensions. To explore, extract, and generalize inherent patterns in large spatiotemporal data sets, clustering algorithms are indispensable. These clustering algorithms must account for the distinct special properties of space and time to outline meaningful clusters in such data sets. Therefore, this research develops a hierarchical method based on self-organizing maps. The hierarchical architecture permits independent modeling of spatial and temporal dependence. To exemplify the utility of the method, this research uses an artificial data set and a socio-economic data set of the Ostregion, Austria, from the years 1961 to 2001. The results for the artificial data set demonstrate that the proposed method produces meaningful clusters that cannot be achieved when disregarding differences in spatial and temporal dependence. The results for the socio-economic data set show that the proposed method is an effective and powerful tool for analyzing spatiotemporal patterns in a regional context.

Passive acoustic monitoring (PAM) is being adopted in a range of contexts. Emerging methods facilitate analysis of large-scale data sets, but ecological interpretation of acoustic indices is not straightforward. In addition, the technical and logistical requirements of using emerging AI methods for big data mean that conservation actors increasingly adopt third-party analysis solutions. We argue that these compounding factors undermine robust ecological inference, clouding insight and decision making. To address this, we present echo-dash, an accessible, interactive dashboard that facilitates rapid, interactive exploration of spatiotemporal soundscape data by conservation actors. Developed through participatory design, echo-dash is built on the simple premise that the potential of PAM can best be realised by keeping human ecological knowledge in the analysis loop. Five key functions to facilitate analysis and interpretation of PAM data were identified and implemented: 1) Calculating soundscape descriptors and probability of species presence; 2) Checking data integrity; 3) Exploring data interactively and in spatiotemporal, environmental contexts; 4) Filtering data by extreme weather, outliers or clusters; 5) Exporting data subsets, plots and code to generate plots. By supporting integration of human, situated ecological knowledge with large scale spatiotemporal data sets, echo-dash bolsters PAM’s potential to transform ecological monitoring for applied and fundamental ecoacoustics.

For many learning algorithms, their learning accuracy will increase as the size of training data increases, forming the well-known learning curve. Usually a learning curve can be fitted by interpolating or extrapolating some points on it with a specified model. The obtained learning curve can then be used to predict the maximum achievable learning accuracy or to estimate the amount of data needed to achieve an expected learning accuracy, both of which will be especially meaningful to data mining on large data sets. Although some models have been proposed to model learning curves, most of them do not test their applicability to large data sets. In this paper, we focus on this issue. We empirically compare six potentially useful models by fitting learning curves of two typical classification algorithms--C4.5 (decision tree) and LOG (logistic discrimination) on eight large UCI benchmark data sets. By using all available data for learning, we fit a full-length learning curve; by using a small portion of the data, we fit a part-length learning curve. The models are then compared in terms of two performances: (1) how well they fit a full-length learning curve, and (2) how well a fitted part-length learning curve can predict learning accuracy at the full length. Experimental results show that the power law (y = a - b * x-c) is the best among the six models in both the performances for the two algorithms and all the data sets. These results support the applicability of learning curves to data mining.

Abstract. With the increasing availability of observational data from different sources at a global level, joint analysis of these data is becoming especially attractive. For such an analysis – oftentimes with little prior knowledge about local and global interactions between the different observational variables at hand – an exploratory, data-driven analysis of the data may be of particular relevance. In the present work we used generalized additive models (GAM) in an exemplary study of spatio-temporal patterns in the tropospheric NO2-distribution derived from GOME satellite observations (1996 to 2001) at global scale. We focused on identifying correlations between NO2 and local wind fields, a quantity which is of particular interest in the analysis of spatio-temporal interactions. Formulating general functional, parametric relationships between the observed NO2 distribution and local wind fields, however, is difficult – if not impossible. So, rather than following a model-based analysis testing the data for predefined hypotheses (assuming, for example, sinusoidal seasonal trends), we used a GAM with non-parametric model terms to learn this functional relationship between NO2 and wind directly from the data. The NO2 observations showed to be affected by wind-dominated processes over large areas. We estimated the extent of areas affected by specific NO2 emission sources, and were able to highlight likely atmospheric transport "pathways". General temporal trends which were also part of our model – weekly, seasonal and linear changes – showed to be in good agreement with previous studies and alternative ways of analysing the time series. Overall, using a non-parametric model provided favorable means for a rapid inspection of this large spatio-temporal NO2 data set, with less bias than parametric approaches, and allowing to visualize dynamical processes of the NO2 distribution at a global scale.

Unlocking massively parallel spectral proper orthogonal decompositions in the PySPOD package

Recent advances in unsteady flow visualization

Parallel coordinate plots (PCPs) are commonly used in information visualization to provide insight into multi-variate data. These plots help to spot correlations between variables. PCPs have been successfully applied to unstructured datasets up to a few millions of points. In this paper, we present techniques to enhance the usability of PCPs for the exploration of large, multi-timepoint volumetric data sets, containing tens of millions of points per timestep. The main difficulties that arise when applying PCPs to large numbers of data points are visual clutter and slow performance, making interactive exploration infeasible. Moreover, the spatial context of the volumetric data is usually lost. We describe techniques for preprocessing using data quantization and compression, and for fast GPU-based rendering of PCPs using joint density distributions for each pair of consecutive variables, resulting in a smooth, continuous visualization. Also, fast brushing techniques are proposed for interactive data selection in multiple linked views, including a 3D spatial volume view. These techniques have been successfully applied to three large data sets: Hurricane Isabel (Vis'04 contest), the ionization front instability data set (Vis'08 design contest), and data from a large-eddy simulation of cumulus clouds. With these data, we show how PCPs can be extended to successfully visualize and interactively explore multi-timepoint volumetric datasets with an order of magnitude more data points.

An approximate algorithm to efficiently solve the k-Closest-Pairs problem on large high-dimensional data sets is presented. The algorithm runs, for a suitable choice of the input parameters, in $\mathcal{O}(d^{2}nk)$ time, where d is the dimensionality and n is the number of points of the input data set, and requires linear space in the input size. It performs at most d+1 iterations. At each iteration a shifted version of the data set is sequentially scanned according to the order induced on it by the Hilbert space filling curve and points whose contribution to the solution has already been analyzed are detected and eliminated. The pruning is lossless, in fact the remaining points along with the approximate solution found can be used for the computation of the exact solution. If the data set is entirely pruned, then the algorithm returns the exact solution. We prove that the pruning ability of the algorithm is related to the nearest neighbor distance distribution of the data set and show that there exists a class of data sets for which the method, augmented with a final step that applies an exact method to the reduced data set, calculates the exact solution with the same time requirements. Although we are able to guarantee a $\mathcal{O}(d^{1+{1}/{t}})$ approximation to the solution, where t∈{1,2,. . .,∞} identifies the Minkowski (Lt) metric of interest, experimental results give the exact k closest pairs for all the large high-dimensional synthetic and real data sets considered and show that the pruning of the search space is effective. We present a thorough scaling analysis of the algorithm for in-memory and disk-resident data sets showing that the algorithm scales well in both cases.

We investigate the use of biased sampling according to the density of the data set to speed up the operation of general data mining tasks, such as clustering and outlier detection in large multidimensional data sets. In density-biased sampling, the probability that a given point will be included in the sample depends on the local density of the data set. We propose a general technique for density-biased sampling that can factor in user requirements to sample for properties of interest and can be tuned for specific data mining tasks. This allows great flexibility and improved accuracy of the results over simple random sampling. We describe our approach in detail, we analytically evaluate it, and show how it can be optimized for approximate clustering and outlier detection. Finally, we present a thorough experimental evaluation of the proposed method, applying density-biased sampling on real and synthetic data sets, and employing clustering and outlier detection algorithms, thus highlighting the utility of our approach.

Uncertain data are inherent in some important applications. Although a considerable amount of research has been dedicated to modeling uncertain data and answering some types of queries on uncertain data, how to conduct advanced analysis on uncertain data remains an open problem at large. In this paper, we tackle the problem of skyline analysis on uncertain data. We propose a novel probabilistic skyline model where an uncertain object may take a probability to be in the skyline, and a p-skyline contains all objects whose skyline probabilities are at least p (0?<?p???1). Computing probabilistic skylines on large uncertain data sets is challenging. We develop a bounding-pruning-refining framework and three algorithms systematically. The bottom-up algorithm computes the skyline probabilities of some selected instances of uncertain objects, and uses those instances to prune other instances and uncertain objects effectively. The top-down algorithm recursively partitions the instances of uncertain objects into subsets, and prunes subsets and objects aggressively. Combining the advantages of the bottom-up algorithm and the top-down algorithm, we develop a hybrid algorithm to further improve the performance. Our experimental results on both the real NBA player data set and the benchmark synthetic data sets show that probabilistic skylines are interesting and useful, and our algorithms are efficient on large data sets.

For Yeast Protein Hubs, More Data Means More Connections