Visual Analysis of a Large and Noisy Dataset

The aim of this article is to present a dynamic record linkage solution using high level machine learning to solve the problems associated with noisy, inconsistent and dynamic data sets. Classic deterministic and probabilistic models fail to work in such environments because they are static and based on assumptions. Incorporating supervised learning algorithm, active learning, and ensemble techniques such as random forests and boosting allows the proposed structure to change in response to different data types and thus become more accurate and scaleable. Some of its highlights are feature selection to ensure match accuracy, clustering to support noisy inputs, and active learning to minimize reliance on large labeled datasets. Simulations with real data, such as government or healthcare data, show that compared with the traditional approach, linkage is significantly more accurate and efficient. The flexible model led to 15% higher F1-scores on noisy datasets and was scalable across large data sets. This work demonstrates how adaptive machine learning can revolutionize the modern record linkage tasks and provides a powerful and effective solution to ever-changing data conditions.

The study of resistomes using whole metagenomic sequencing enables high-throughput identification of resistance genes in complex microbial communities, such as the human microbiome. Over recent years, sophisticated and diverse pipelines have been established to facilitate raw data processing and annotation. Despite the progress, there are no easy-to-use tools for comprehensive visual, statistical and functional analysis of resistome data. Thus, exploration of the resulting large complex datasets remains a key bottleneck requiring robust computational resources and technical expertise, which creates a significant hurdle for advancements in the field. Here, we introduce ResistoXplorer, a user-friendly tool that integrates recent advancements in statistics and visualization, coupled with extensive functional annotations and phenotype collection, to enable high-throughput analysis of common outputs generated from metagenomic resistome studies. ResistoXplorer contains three modules—the ‘Antimicrobial Resistance Gene Table’ module offers various options for composition profiling, functional profiling and comparative analysis of resistome data; the ‘Integration’ module supports integrative exploratory analysis of resistome and microbiome abundance profiles derived from metagenomic samples; finally, the ‘Antimicrobial Resistance Gene List’ module enables users to intuitively explore the associations between antimicrobial resistance genes and the microbial hosts using network visual analytics to gain biological insights. ResistoXplorer is publicly available at http://www.resistoxplorer.no.

Business analytics involves using various technologies to analyze data. Data mining focuses on the automated search for knowledge, patterns, or regularities in data. As a business analyst, it is essential to recognize the type of analytical technique appropriate for solving a specific problem. Exploratory Data Analysis (EDA) describes data using statistical and visualization techniques to highlight important aspects of that data for further analysis. This involves examining a data set from many angles, describing it, and summarizing it without making assumptions about its content. Exploratory data analysis is an essential step before diving into statistical modeling or machine learning to ensure that the data is really what it claims to be and that there are no apparent errors. This type of analysis should be part of data science projects in every organization. Visual analytics is sometimes confused with data visualization. Visual analysis is not simply a matter of graphically representing data. Modern, interactive visual analytics makes combining data from multiple sources easy and performs in-depth data analysis directly in the visualization. Additionally, artificial intelligence and machine learning algorithms can offer recommendations for exploration. Ultimately, visual analytics helps transform massive data sets into business insights that can positively impact an organization. Considering the previous comment about visual data analysis, it should be added that the system has extensive capabilities to create graphical dashboards containing reports and charts. It is essential that in the case of desktops, in addition to the visualizations included in the system itself, it is possible to embed reports from third-party tools.

Optical illusions from visual data analysis: Example of the New Zealand asthma mortality epidemic

The widespread use of computational simulation in science and engineering provides challenging research opportunities. Multiple independent variables are considered and large and complex data are computed, especially in the case of multi-run simulation. Classical visualization techniques deal well with 2D or 3D data and also with time-dependent data. Additional independent dimensions, however, provide interesting new challenges. We present an advanced visual analysis approach that enables a thorough investigation of families of data surfaces, i.e., datasets, with respect to pairs of independent dimensions. While it is almost trivial to visualize one such data surface, the visual exploration and analysis of many such data surfaces is a grand challenge, stressing the users' perception and cognition. We propose an approach that integrates projections and aggregations of the data surfaces at different levels (one scalar aggregate per surface, a 1D profile per surface, or the surface as such). We demonstrate the necessity for a flexible visual analysis system that integrates many different (linked) views for making sense of this highly complex data. To demonstrate its usefulness, we exemplify our approach in the context of a meteorological multi-run simulation data case and in the context of the engineering domain, where our collaborators are working with the simulation of elastohydrodynamic (EHD) lubrication bearing in the automotive industry.

The proliferation of next-generation sequencing (NGS) technologies and analysis tools present new challenges to genome browsers. These challenges include supporting very large datasets, integrating analysis tools with data visualization to help reason about and improve analyses, and sharing or publishing fully interactive visualizations. The Galaxy Track Browser (GTB) is a Web-based genome browser integrated into the Galaxy platform that addresses these challenges. GTB is the first Web-based genome browser to provide a full multi-resolution data model; this model supports efficient data retrieval from very large datasets. GTB leverages the Galaxy platform to combine data visualization and data analysis; users can specify parameter values and run tools to produce new data, all within GTB. GTB also provides interactive filters that dynamically show and hide data and can be used to identify data for further investigation. GTB is available on every Galaxy server, and visualizations can be created for both standard and custom genome builds. Fully interactive GTB visualizations can be shared with colleagues and published on the Web using a simple graphical user interface.

Interactive visual analysis of contrast-enhanced ultrasound data based on small neighborhood statistics

Both interactive visualization and computational analysis methods are useful for data studies and an integration of both approaches is promising to successfully combine the benefits of both methodologies. In interactive data exploration and analysis workflows, we need successful means to quantitatively externalize results from data studies, amounting to a particular challenge for the usually qualitative visual data analysis. In this paper, we propose a hybrid approach in order to quantitatively externalize valuable findings from interactive visual data exploration and analysis, based on local linear regression models. The models are built on user-selected subsets of the data, and we provide a way of keeping track of these models and comparing them. As an additional benefit, we also provide the user with the numeric model coefficients. Once the models are available, they can be used in subsequent steps of the workflow. A model-based optimization can then be performed, for example, or more complex models can be reconstructed using an inversion of the local models. We study two datasets to exemplify the proposed approach, a meteorological data set for illustration purposes and a simulation ensemble from the automotive industry as an actual case study.

Author(s): Bethel, E. Wes; Childs, Hank; Mascarenhas, Ajith; Pascucci, Valerio; Prabhat | Abstract: Visualization is a highly data intensive science: visualization algorithms take as input vast amounts of data produced by simulations or experiments, and then transform that data into imagery. It turns out, as we shall explore in this chapter, that visualization reveals a somewhat different view of scientific data management challenges than are examined elsewhere in this book. For example, a data ordering and storage layout that works well for saving data from memory to disk may not be the best thing for subsequent visual data analysis algorithms. This chapter will present four broad topic areas under this general rubric: (1) a view of SDM-related issues from the perspective of implementing a production-quality, parallel capable visual data analysis infrastructure; (2) novel data storage formats for multi-resolution, streaming data movement, access and use by post-processing tools; (3) data models, formats and APIs for performing efficient I/O for both simulations and post-processing tools, discussion of issues and previous work in this space; (4) how combining state-of- the-art techniques from scientific data management and visualization enables visual data analysis of truly massive datasets.

With the rapid development of Internet and big data, the method of directing the film and television through big data analysis has attracted more and more attention. The visual analysis of data is an important method of big data analysis. The demand for big data in film industry is becoming larger and larger. The development trend of the industry relies on visualization for film and television data analysis. The basic concept of big data, the theory of big data visualization analysis and the film evaluation research base on the film expert evaluation system are presented in this paper.

One of the cardinal impacts of Cyber Security (CS) is the combating of financial e-fraud and other related crimes. Formal Concept Analysis (FCA) could serve as a great useful weapon in the detection and retrieving of various crime activities perpetrated on the cyberspace. This paper proposes a CS-based investigation process through visualization and data analysis of mobile communication devices using the formal Concept Analysis which is a data analysis technique based on lattice theory and propositional Calculus. The method to be employed visualizes the lattice that maybe conceived as a set of common and distinct attributes of data in such a way that classifications are done based on related data with respect to time and events of the crime performance within a some Internet geographical space. The lattices were then built using Galicia 3.2 Lattice building software using the data connected of criminal activities that were collected from the mobile phones, Laptops etc.. The results obtained were used in building a more defined and conceptualized system for analysis of e-fraud data in the cyberspace that could be easily visualized and intelligently analysed by the cyberspace computational systems processes.

The curse of dimensionality, i.e., the fact that feature spaces of increasing dimensionality with finite sample sizes tend to be empty, has given incentive to a plethora of research activities in various disciplines and diverse application fields, e.g., statistics or neural networks. Three major application fields are multivariate data classification, data analysis and data visualization. In this contribution, methods for dimensionality reduction from three decades of interdisciplinary research are browsed and their applicability in the above application domains is briefly discussed. Complementing techniques for ensuing interactive data visualization, data navigation and visual exploratory data analysis are presented, which exploit the remarkable human perceptive and associative capabilities for interactive visual exploratory data analysis and systematic recognition system design. The main focus of this paper is on the comparison of feature selection and feature extraction techniques and the potential benefit of their combination. Further, the interesting implications of dimensionality reduction for VLSI design and related area and power consumption are pointed out.

We present an approach to effectively use millions of images with noisy annotations in conjunction with a small subset of cleanly-annotated images to learn powerful image representations. One common approach to combine clean and noisy data is to first pre-train a network using the large noisy dataset and then fine-tune with the clean dataset. We show this approach does not fully leverage the information contained in the clean set. Thus, we demonstrate how to use the clean annotations to reduce the noise in the large dataset before fine-tuning the network using both the clean set and the full set with reduced noise. The approach comprises a multi-task network that jointly learns to clean noisy annotations and to accurately classify images. We evaluate our approach on the recently released Open Images dataset, containing ~9 million images, multiple annotations per image and over 6000 unique classes. For the small clean set of annotations we use a quarter of the validation set with ~40k images. Our results demonstrate that the proposed approach clearly outperforms direct fine-tuning across all major categories of classes in the Open Image dataset. Further, our approach is particularly effective for a large number of classes with wide range of noise in annotations (20-80% false positive annotations).

Digital data have become a torrent engulfing every area of business, science and engineering disciplines. In the age of Big Data, deriving values and insights from large amounts of data using rich analytics becomes an important differentiating capability for competitiveness, success and leadership in every field. Scientists and engineers of many different domains are increasingly clamouring for mechanisms to manage and analyse the massive quantities of information now available in order to obtain new answers and extract from it maximum value. Computational modelling and simulation is the central technology to numerous of these domains. Molecular Dynamics (MD) is a computational simulation technique that describes the physical forces and movements of interacting microscopic elements such atoms and molecules. MD has important applications in the fields of chemistry, biotechnology, pharmaceutical industry, energy, climate or materials science, among others. Advanced MD algorithms include not only Molecular Mechanics (MM), but also Quantum Mechanics (QM) approaches, raising important big data challenges still to be sorted out. MD simulations perform an iterative process generating large amounts of data in streaming. Current software technology is far from being able to manage, analyze and visualize the extremely large and complex data sets generated by important molecular processes. This paper analyzes the current big data limits in the Computational Chemistry field, especially in the MD processes. To overcome these challenging situations, this work provide guidance for future research including advances in scalable algorithms for data analysis, dynamic query technology, data models and storage strategies, parallel executions, I/O optimization, and interactive visual exploration and analysis of MD data.

Concepts are fundamental to nursing theory, practice, and research, serving as abstract constructs that represent key aspects of human experience. However, the reporting of concept analysis in nursing lacks standardised guidelines, leading to inconsistencies in methodological transparency and rigour. Existing frameworks for qualitative and mixed-methods research do not adequately address the unique aspects of concept analysis. This study aims to develop a comprehensive reporting guideline for concept analysis in nursing research, guided by the EQUATOR Network toolkit. The first stage of guideline development involves a systematic review of MEDLINE, CINAHL, and PsycINFO to identify key components of concept analysis reporting in nursing. The review will focus on studies utilising concept analysis within the nursing metaparadigms: person, environment, health, and nursing, while adhering to PRISMA guidelines to ensure methodological rigour. The systematic review synthesised existing literature to establish a foundational framework of essential elements for reporting concept analysis in nursing. Findings from this review will inform a subsequent Delphi study and structured guideline development process. The resulting standardised reporting framework will enhance methodological consistency, transparency, and reproducibility in concept analysis, thereby contributing to the advancement of nursing theory, research, and practice.