Quantitative Externalization of Visual Data Analysis Results Using Local Regression Models

Digital data have become a torrent engulfing every area of business, science and engineering disciplines. In the age of Big Data, deriving values and insights from large amounts of data using rich analytics becomes an important differentiating capability for competitiveness, success and leadership in every field. Scientists and engineers of many different domains are increasingly clamouring for mechanisms to manage and analyse the massive quantities of information now available in order to obtain new answers and extract from it maximum value. Computational modelling and simulation is the central technology to numerous of these domains. Molecular Dynamics (MD) is a computational simulation technique that describes the physical forces and movements of interacting microscopic elements such atoms and molecules. MD has important applications in the fields of chemistry, biotechnology, pharmaceutical industry, energy, climate or materials science, among others. Advanced MD algorithms include not only Molecular Mechanics (MM), but also Quantum Mechanics (QM) approaches, raising important big data challenges still to be sorted out. MD simulations perform an iterative process generating large amounts of data in streaming. Current software technology is far from being able to manage, analyze and visualize the extremely large and complex data sets generated by important molecular processes. This paper analyzes the current big data limits in the Computational Chemistry field, especially in the MD processes. To overcome these challenging situations, this work provide guidance for future research including advances in scalable algorithms for data analysis, dynamic query technology, data models and storage strategies, parallel executions, I/O optimization, and interactive visual exploration and analysis of MD data.

We present a scalable volume rendering technique that exploits lossy compression and low-cost commodity hardware to permit highly interactive exploration of time-varying scalar volume data. A palette-based decoding technique and an adaptive bit allocation scheme are developed to fully utilize the texturing capability of a commodity 3D graphics card. Using a single PC equipped with a modest amount of memory, a texture-capable graphics card and an inexpensive disk array, we are able to render hundreds of time steps of regularly gridded volume data (up to 42 million voxels each time step) at interactive rates. By clustering multiple PCs together, we demonstrate the data-size scalability of our method. The frame rates achieved make possible the interactive exploration of data in the temporal, spatial and transfer function domains. A comprehensive evaluation of our method based on experimental studies using data sets (up to 134 million voxels per time step) from turbulence flow simulations is also presented.

The curse of dimensionality, i.e., the fact that feature spaces of increasing dimensionality with finite sample sizes tend to be empty, has given incentive to a plethora of research activities in various disciplines and diverse application fields, e.g., statistics or neural networks. Three major application fields are multivariate data classification, data analysis and data visualization. In this contribution, methods for dimensionality reduction from three decades of interdisciplinary research are browsed and their applicability in the above application domains is briefly discussed. Complementing techniques for ensuing interactive data visualization, data navigation and visual exploratory data analysis are presented, which exploit the remarkable human perceptive and associative capabilities for interactive visual exploratory data analysis and systematic recognition system design. The main focus of this paper is on the comparison of feature selection and feature extraction techniques and the potential benefit of their combination. Further, the interesting implications of dimensionality reduction for VLSI design and related area and power consumption are pointed out.

An increasing number of emerging web database applications deal with large georeferenced data sets. However, exploring these large data sets through spatial queries can be very time and resource intensive. The need for interactive spatial queries has arisen in many applications such as Geographic Information Systems (GIS) for efficient decision-support. In this paper, we propose a new interactive spatial query processing technique for GIS. We present a family of the Incremental Refining Spatial Join (IRSJ) algorithms that can be used to report incrementally refined running estimates for aggregate queries while simultaneously displaying the actual query result tuples of the data sets sampled so far. Our goal is to minimize the time until an acceptably accurate estimate of the query result is available (to users) measured by a confidence interval. Our approach enables more interactive data exploration and analysis. While similar work has been done in relational databases, to the best of our knowledge, this is the first work using this approach in GIS. We investigate and evaluate different sampling methodologies through extensive experimental performance comparisons. Experiments on both real and synthetic data show an order of magnitude response time improvement relative to the final answer obtained when using a full R-tree join. We also show the impact of different index structures on the performance of our algorithms using three known sampling methods.

Direct volume rendering allows for interactive exploration of volumetric data and has become an important tool in many visualization domains. But the insight and information that can be obtained are dependent on the transfer function defining the transparency of voxels. Constructing good transfer functions is one of the most time consuming and cumbersome tasks in volume visualization. We present a novel general purpose method for automatically generating an initial set of best transfer function candidates. The generated transfer functions reveal the major structural features within the volume and allow for an efficient initial visual analysis, serving as a basis for further interactive exploration in particular of originally unknown data. The basic idea is to introduce a metric as a measure of the goodness of a transfer function which indicates the information that can be gained from rendered images by interactive visualization. In contrast to prior methods, our approach does not require a user feedback-loop, operates exclusively in image space and takes the characteristics of interactive data exploration into account. We show how our new transfer function generation method can uncover the major structures of an unknown dataset within only a few minutes.

Interactive visual analysis of contrast-enhanced ultrasound data based on small neighborhood statistics

Network inference utilizes experimental high-throughput data for the reconstruction of molecular interaction networks where new relationships between the network entities can be predicted. Despite the increasing amount of experimental data, the parameters of each modeling technique cannot be optimized based on the experimental data alone, but needs to be qualitatively assessed if the components of the resulting network describe the experimental setting. Candidate list prioritization and validation builds upon data integration and data visualization. The application of tools supporting this procedure is limited to the exploration of smaller information networks because the display and interpretation of large amounts of information is challenging regarding the computational effort and the users' experience. The Ondex software framework was extended with customizable context-sensitive menus which allow additional integration and data analysis options for a selected set of candidates during interactive data exploration. We provide new functionalities for on-the-fly data integration using InterProScan, PubMed Central literature search, and sequence-based homology search. We applied the Ondex system to the integration of publicly available data for Aspergillus nidulans and analyzed transcriptome data. We demonstrate the advantages of our approach by proposing new hypotheses for the functional annotation of specific genes of differentially expressed fungal gene clusters. Our extension of the Ondex framework makes it possible to overcome the separation between data integration and interactive analysis. More specifically, computationally demanding calculations can be performed on selected sub-networks without losing any information from the whole network. Furthermore, our extensions allow for direct access to online biological databases which helps to keep the integrated information up-to-date.

BackgroundThe use of RNA-sequencing (RNA-seq) in molecular biology research and clinical settings has increased significantly over the past decade. Despite its widespread adoption, there is a lack of simple and interactive tools to analyze and explore RNA-seq data. Many established tools require programming or Unix/Bash knowledge to analyze and visualize results. This requirement presents a significant barrier for many researchers to efficiently analyze and present RNA-seq data.ResultsHere we present BEAVR, a Browser-based tool for the Exploration And Visualization of RNA-seq data. BEAVR is an easy-to-use tool that facilitates interactive analysis and exploration of RNA-seq data. BEAVR is developed in R and uses DESeq2 as its engine for differential gene expression (DGE) analysis, but assumes users have no prior knowledge of R or DESeq2. BEAVR allows researchers to easily obtain a table of differentially-expressed genes with statistical testing and then visualize the results in a series of graphs, plots and heatmaps. Users are able to customize many parameters for statistical testing, dealing with variance, clustering methods and pathway analysis to generate high quality figures.ConclusionBEAVR simplifies analysis for novice users but also streamlines the RNA-seq analysis process for experts by automating several steps. BEAVR and its documentation can be found on GitHub at https://github.com/developerpiru/BEAVR. BEAVR is available as a Docker container at https://hub.docker.com/r/pirunthan/beavr.

As video cameras proliferate, the ability to scalably capture and search their data becomes important. Scalability is improved by performing video analytics on cloudlets at the edge of the Internet, and only shipping extracted index information and meta-data to the cloud. In this setting, we describe interactive data exploration (IDE), which refers to human-in-the-loop content-based retrospective search using predicates that may not have been part of any prior indexing. We also describe a new technique called just-in-time indexing (JITI) that improves response times in IDE.

Smuggling of nuclear and radiological material is a complex problem that will need to be analyzed from a variety of angles in order to effectively combat something so rare, yet so hazardous. One crucial tool in the effort to detect and deter smuggling of rad/nuc materials is the radiation portal monitor (RPM)—machines that scan passengers at airports, vehicles and trains at border crossings, and shipping containers at ports throughout the world. While data from RPMs are regularly analyzed at the individual lane level to ensure monitor health and good operation, aggregate analysis of the data has not gone far beyond summary statistics such as number of occupancies and alarm rates. In particular, there are opportunities to carry out interactive exploratory data analysis with the help of tools like the R package Shiny and methods that lend themselves to the visual display of spatiotemporal correlations such as spatial PCA and detrended cross-correlation analysis. By doing so, we discover interesting spatiotemporal patterns in the data that point to specific events or anomalous moments in time that warrant further analysis.

Interactive data exploration plays a fundamental role in analyzing three dimensional scientific data. Occlusion management and context preservation are among the key factors to ensure effective identification and extraction of three-dimensional features. In this paper, we present an interactive data exploration system that utilizes a physically based deformation method to investigate hidden structures of data in three dimensional data sets. While non-physically based methods are popular for visual analytic applications due to their lower computational cost, physically based deformation methods can often better preserve features and their context. Our physically based deformation method preserves data features by setting the mesh properties according to interesting data attributes. We design effective and intuitive interfaces by using a metaphor of virtual retractor, which reflects the cutting and splitting of data that our system is simulating. We demonstrate case studies on multiple particle datasets and volume datasets, and present feedback from a domain user.

This article presents a platform for interactive graph mining and relational machine learning called GraphVis. The platform combines interactive visual representations with state-of-the-art graph mining and relational machine learning techniques to aid in revealing important insights quickly as well as learning an appropriate and highly predictive model for a particular task (e.g., classification, link prediction, discovering the roles of nodes, and finding influential nodes). Visual representations and interaction techniques and tools are developed for simple, fast, and intuitive real-time interactive exploration, mining, and modeling of graph data. In particular, we propose techniques for interactive relational learning (e.g., node/link classification), interactive link prediction and weighting, role discovery and community detection, higher-order network analysis (via graphlets, network motifs), among others. GraphVis also allows for the refinement and tuning of graph mining and relational learning methods for specific application domains and constraints via an end-to-end interactive visual analytic pipeline that learns, infers, and provides rapid interactive visualization with immediate feedback at each change/prediction in real-time. Other key aspects include interactive filtering, querying, ranking, manipulating, exporting, as well as tools for dynamic network analysis and visualization, interactive graph generators (including new block model approaches), and a variety of multi-level network analysis techniques.

Three-dimensional (3D) digital brain atlases and high-throughput brain-wide imaging techniques generate large multidimensional datasets that can be registered to a common reference frame. Generating insights from such datasets depends critically on visualization and interactive data exploration, but this a challenging task. Currently available software is dedicated to single atlases, model species or data types, and generating 3D renderings that merge anatomically registered data from diverse sources requires extensive development and programming skills. Here, we present brainrender: an open-source Python package for interactive visualization of multidimensional datasets registered to brain atlases. Brainrender facilitates the creation of complex renderings with different data types in the same visualization and enables seamless use of different atlas sources. High-quality visualizations can be used interactively and exported as high-resolution figures and animated videos. By facilitating the visualization of anatomically registered data, brainrender should accelerate the analysis, interpretation, and dissemination of brain-wide multidimensional data.

Interactive data exploration at scale remains constrained by 1) weak adaptability to shifting query workloads, 2) limited and post hoc error guarantees, 3) poor scalability under dynamic, high-dimensional data, 4) sparse user guidance during query formulation, and 5) non-trivial system overheads from learned or probabilistic components. We propose an end-to-end, privacy-aware framework that dynamically forms SQL queries for multi-dimensional data using randomized signals derived from personal web usage. The method integrates: 1) on-device user modeling that converts browsing interactions into preference embeddings under local differential privacy; 2) a constrained-randomization layer that enforces coverage and diversity to avoid filter bubbles while remaining responsive to user intent; 3) a contextual bandit policy (with optional deep reinforcement learning extension) that selects or completes query templates using signals from user profiles, session context, and data synopses; and 4) an error-aware AQP executor combining stratified/pilot sampling, synopsis reuse, and confidence-interval gating with automatic sample escalation. This design directly addresses the above limitations: the bandit adapts online to workload shifts; the AQP layer provides pre-execution feasibility checks and per-query error control; synopsis reuse and AB-tree–style random sampling maintain low latency under updates; and a guidance module (predictive autocompletion with information-gain scoring) reduces user effort while preserving exploration diversity. To evaluate effectiveness, we introduce a privacy-preserving training regimen (federated updates over DP-noised profiles) and a novel benchmark protocol measuring time-to-insight, error compliance under differential privacy, session diversity, and latency against strong baselines. The result is an ML-driven exploration loop that achieves error-bounded interactivity, robust personalization, and scalable performance on evolving, high-dimensional datasets, while providing evaluation metrics that capture both user experience and privacy-preserving guarantees.

Many Geographic Information Systems (GIS) handle large geospatial datasets stored in raster representation. Spatial joins over raster data are important queries in GIS for data analysis and decision support. However, evaluating spatial joins can be very time intensive due to the size of these datasets. In this paper we propose a new interactive framework that allows users to get approximate answers in near instantaneous time, thus allowing for truly interactive data exploration. Our method utilizes two proposed statistical approaches: probabilistic join and sampling based join. Our probabilistic join method provides speedup of two orders of magnitude with no correctness guarantee, while our sampling based method provides an order of magnitude improvement over the full quad-tree join and also provides running confidence intervals. We propose a framework that combines the two approaches to allow end users to tradeoff speed versus bounded accuracy. The two approaches are evaluated empirically with real and synthetic datasets.