Using relational graphs for exploratory analysis of network traffic data

With the rapid increasing of network scale, the size of traffic data also expands a lot. In traditional traffic data analysis, there are some problems, such as high computation complexity, low analysis efficiency, long learning period, and difficulty of development. To address these problems, we design and implement a GPU-accelerated parallel analysis scheme for network traffic - EasyAnalyze. In EasyAnalyze, we introduce GPU parallel computing, Map/Reduce architecture into network traffic analysis, which greatly improves the efficiency but does not increase the difficulty in programming. In the experiments, EasyAnalyze shows very promising results: 1 the speed is 6-17 times faster than conventional serial analysis in network traffic data analysis; and 2 the size of code is only 2% of the mainstream GPU Map/Reduce.

With the growth of the internet of things (IoT) smart objects, managing these objects becomes a very important challenge, to know the total number of interconnected objects on a heterogeneous network, and if they are functioning correctly; the use of IoT objects can have advantages in terms of comfort, efficiency, and cost. In this context, the identification of IoT objects is the first step to help owners manage them and ensure the security of their IoT environments such as smart homes, smart buildings, or smart cities. In this paper, to meet the need for IoT object identification, we have deployed an intelligent environment to collect all network traffic traces based on a diverse list of IoT in real-time conditions. In the exploratory phase of this traffic, we have developed learning models capable of identifying and classifying connected IoT objects in our environment. We have applied the six supervised machine learning algorithms: support vector machine, decision tree (DT), random forest (RF), k-nearest neighbors, naive Bayes, and stochastic gradient descent classifier. Finally, the experimental results indicate that the DT and RF models proved to be the most effective and demonstrate an accuracy of 97.72% on the analysis of network traffic data and more particularly information contained in network protocols. Most IoT objects are identified and classified with an accuracy of 99.21%.

The main data-driven techniques for detecting cybersecurity attacks are based on the analysis of network traffic data and/or of application/system logs (stored in a host or in some other kind of device). A wide range of machine-learning techniques (and possible alternative configurations of them) have been proposed in the literature so far, for this purpose, but none of them has been proven to consistently overcome the others across different datasets. In order to ensure better accuracy and stability, the ensemble paradigm can be exploited as an effective solution for combining such techniques. However, as attack detection problems are hard to cope with and, usually, entail the analysis of large and fast streams of data, different types of ensemble (and of base algorithms composing the ensemble) should have experimented, exploiting distributed architecture to suitably reduce the high-execution times necessary to run them. In order to handle all these issues, a p2p environment to validate ensemble-based approaches in the cybersecurity domain is proposed in this paper. Two case studies are analyzed by using this framework, which concern the detection of intrusions in network-traffic data and of deviant process instances. Preliminary scalability results demonstrate that the framework is a viable solution for these challenging kind of problems.

The desire to conceptualize network traffic in a prevailing communication network is a facet for many types of network research studies. In this research, real traffic traces collected over trans-Pacific backbone links (the MAWI repository, providing publicly available anonymized traces) are analyzed to study the underlying traffic patterns. All data analysis and visualization is carried out using Matlab (Matlab is a trademark of The Mathworks, Inc.). At packet level, we first measure parameters such as distribution of packet lengths, distribution of protocol types, and then fit following analytical models. Next, the concept of flow is introduced and flow based analysis is studied. We consider flow related parameters such as top ports seen, duration of the flow, distribution of flow lengths, and number of flows with different timeout values and provide analytical models to fit the flow lengths. Further, we study the amount of data flowing between source-destination pairs. Finally, we focus on TCP-specific aspects of captured traces such as retransmissions and packet round-trip times. From the results obtained, we infer the Zipf-type nature of distribution for number of flows, heavy-tailness of flow sizes and the contribution of well-known ports at packet and flow level. Our study helps a network analyst to farther the knowledge and helps optimize the network resources, while performing efficient traffic engineering.

Underground flow paths (UFP) often play an important role in the illustration of geological data by geologists, especially in illustrating geological data and revealing stratigraphic structures, which can help domain experts in their exploration of petroleum information. In this paper, we present a new immersive visualization tool to help domain experts better illustrate stratigraphic data. We use a visualization method based on bit-array-based 3-D texture to represent stratigraphic data. Our visualization tool has three major advantages: it allows for flexible interaction at the immersive device, it enables domain experts to obtain their desired UFP structure through the execution of quadratic surface queries, and supports different stratigraphic display modes, as well as switching and integration geological information flexibly. Feedback from domain experts has shown that our tool can contribute more for domain experts in the scientific exploration of stratigraphic data, compared to the existing UFP visualization tools in the field. Thus, experts in geology can have a more comprehensive understanding and more effective illustration of the structure and distribution of UFPs.

ABSTRACTOverviews of spatiotemporal data are acknowledged to play an important role in visualization in initiating and supporting geovisualization and exploratory data analysis (EDA). However, relatively little research has focused on the visual overviews themselves, and their potential impacts on EDA outcomes. In a user study, we evaluated the influence of different levels of domain knowledge on the usefulness of four distinct types of static visual overview of spatiotemporal data. Beyond simply orienting users, our results indicate that visual overviews can be important in gaining insights into a data set, for example, in learning about metadata. Although subjects without domain knowledge struggled to judge the quality of their findings, they were as successful at identifying interesting patterns in the data as those with domain expertise. Our results suggest that detailed background knowledge of a data set can actively hinder EDA. Being already familiar with their own data sets, our results highlight the tendency of data experts to disregard findings that do not match their pre-existing domain knowledge. Based on these findings, our conclusions identify a range of potential avenues for future work, including the use of visual overviews that deliberately do not, from first view, reveal the context of the data they show. This later approach could help in cases where domain experts need to see their data with ‘fresh eyes’, and detect interesting patterns in spatiotemporal data before relating the findings to specific knowledge about the data sets and the domain.

To remain competitive in dynamic environment, enterprises need to make effective and efficient decisions in response to changes. Good modeling tools and analysis support is a necessity in this regard. Such modeling and analysis tools should to be able to visually model and programmatically analyze several descriptive and prescriptive modeling languages in concert. We recount our experience with visual modeling editor and ontological representation for both descriptive and prescriptive models for enterprise decision making. Starting with purposive modeling tools, we shifted to integrated modeling environment where all relevant models of enterprise coexist and are analyzed together. Our ongoing research suggests that apart from integrated modeling environment, scalable modeling facilities for better interaction between modelers and domain experts are also necessary to make modeling and analysis of enterprise models more streamlined.

Monitoring conditions in the periglacial areas of Antarctica helps geographers and geologists to understand physical processes associated with mesoscale land systems. Analyzing these unique temporal datasets poses a significant challenge for domain experts, due to the complex and often incomplete data, for which corresponding exploratory tools are not available. In this paper, we present a novel visual analysis tool for extraction and interactive exploration of temporal measurements captured at the polar station at the James Ross Island in Antarctica. The tool allows domain experts to quickly extract information about the snow level, originating from a series of photos acquired by trail cameras. Using linked views, the domain experts can interactively explore and combine this information with other spatial and non-spatial measures, such as temperature or wind speed, to reveal the interplay of periglacial and aeolian processes. An abstracted interactive map of the area indicates the position of measurement spots to facilitate navigation. The design of the tool was made in tight collaboration with geographers, which resulted in an early prototype, tested in the pilot study. The following version of the tool and its usability has been evaluated in the user study with five domain experts and their feedback was incorporated into the final version, presented in this paper. This version was again discussed with two experts in an informal interview. Within these evaluations, they confirmed the significant benefit of the tool for their research tasks.

Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts - who often have no expertise in ML or data science - are asked to use ML predictions to make high-stakes decisions. Multiple ML usability challenges can appear as result, such as lack of user trust in the model, inability to reconcile human-ML disagreement, and ethical concerns about oversimplification of complex problems to a single algorithm output. In this paper, we investigate the ML usability challenges that present in the domain of child welfare screening through a series of collaborations with child welfare screeners. Following the iterative design process between the ML scientists, visualization researchers, and domain experts (child screeners), we first identified four key ML challenges and honed in on one promising explainable ML technique to address them (local factor contributions). Then we implemented and evaluated our visual analytics tool, Sibyl, to increase the interpretability and interactivity of local factor contributions. The effectiveness of our tool is demonstrated by two formal user studies with 12 non-expert participants and 13 expert participants respectively. Valuable feedback was collected, from which we composed a list of design implications as a useful guideline for researchers who aim to develop an interpretable and interactive visualization tool for ML prediction models deployed for child welfare screeners and other similar domain experts.

Morse-Smale (MS) complexes have been gaining popularity as a tool for feature-driven data analysis and visualization. However, the quality of their geometric embedding and the sole dependence on the input scalar field data can limit their applicability when expressing application-dependent features. In this paper we introduce a new combinatorial technique to compute an MS complex that conforms to both an input scalar field and an additional, prior segmentation of the domain. The segmentation constrains the MS complex computation guaranteeing that boundaries in the segmentation are captured as separatrices of the MS complex. We demonstrate the utility and versatility of our approach with two applications. First, we use streamline integration to determine numerically computed basins/mountains and use the resulting segmentation as an input to our algorithm. This strategy enables the incorporation of prior flow path knowledge, effectively resulting in an MS complex that is as geometrically accurate as the employed numerical integration. Our second use case is motivated by the observation that often the data itself does not explicitly contain features known to be present by a domain expert. We introduce edit operations for MS complexes so that a user can directly modify their features while maintaining all the advantages of a robust topology-based representation.

Biological networks describe complex relationships in biological systems, which represent biological entities as vertices and their underlying connectivity as edges. Ideally, for a complete analysis of such systems, domain experts need to visually integrate multiple sources of heterogeneous data, and visually, as well as numerically, probe said data in order to explore or validate (mechanistic) hypotheses. Such visual analyses require the coming together of biological domain experts, bioinformaticians, as well as network scientists to create useful visualization tools. Owing to the underlying graph data becoming ever larger and more complex, the visual representation of such biological networks has become challenging in its own right. This introduction and survey aims to describe the current state of biological network visualization in order to identify scientific gaps for visualization experts, network scientists, bioinformaticians, and domain experts, such as biologists, or biochemists, alike. Specifically, we revisit the classic visualization pipeline, upon which we base this paper’s taxonomy and structure, which in turn forms the basis of our literature classification. This pipeline describes the process of visualizing data, starting with the raw data itself, through the construction of data tables, to the actual creation of visual structures and views, as a function of task-driven user interaction. Literature was systematically surveyed using API-driven querying where possible, and the collected papers were manually read and categorized based on the identified sub-components of this visualization pipeline’s individual steps. From this survey, we highlight a number of exemplary visualization tools from multiple biological sub-domains in order to explore how they adapt these discussed techniques and why. Additionally, this taxonomic classification of the collected set of papers allows us to identify existing gaps in biological network visualization practices. We finally conclude this report with a list of open challenges and potential research directions. Examples of such gaps include (i) the overabundance of visualization tools using schematic or straight-line node-link diagrams, despite the availability of powerful alternatives, or (ii) the lack of visualization tools that also integrate more advanced network analysis techniques beyond basic graph descriptive statistics.

BackgroundHistology images comprise one of the important sources of knowledge for phenotyping studies in systems biology. However, the annotation and analyses of histological data have remained a manual, subjective and relatively low-throughput process.ResultsWe introduce Graph based Histology Image Explorer (GRAPHIE)-a visual analytics tool to explore, annotate and discover potential relationships in histology image collections within a biologically relevant context. The design of GRAPHIE is guided by domain experts' requirements and well-known InfoVis mantras. By representing each image with informative features and then subsequently visualizing the image collection with a graph, GRAPHIE allows users to effectively explore the image collection. The features were designed to capture localized morphological properties in the given tissue specimen. More importantly, users can perform feature selection in an interactive way to improve the visualization of the image collection and the overall annotation process. Finally, the annotation allows for a better prospective examination of datasets as demonstrated in the users study. Thus, our design of GRAPHIE allows for the users to navigate and explore large collections of histology image datasets.ConclusionsWe demonstrated the usefulness of our visual analytics approach through two case studies. Both of the cases showed efficient annotation and analysis of histology image collection.

Exploratory and spatial data analysis (EDA–SDA) for determining regional background levels and anomalies of potentially toxic elements in soils from Catorce–Matehuala, Mexico

The development of custom interactive visualization tools for specific domains and applications has been made much simpler recently by a surge of visualization tools, libraries and frameworks. Most of these tools are developed for classical data science applications, where a user is supported in analyzing measured or simulated data. But recently, there has also been an increasing interest in visual support for understanding machine learning algorithms and frameworks, especially for deep learning. Many, if not most, of the visualization support for (deep) learning addresses the developer of the learning system and not the end user (data scientist). Here we show on a specific example, namely the development of a matrix calculus algorithm, that supporting visualizations can also greatly benefit the development of algorithms in classical domains like in our case computer algebra. The idea is similar to visually supporting the understanding of learning algorithms, namely provide the developer with an interactive, visual tool that provides insights into the workings and, importantly, also into the failures of the algorithm under development. Developing visualization support for matrix calculus development went similar as the development of more traditional visual support systems for data analysts. First, we had to acquaint ourselves with the problem, its language and challenges by talking to the core developer of the matrix calculus algorithm. Once we understood the challenge, it was fairly easy to develop visual support that streamlined the development of the matrix calculus algorithm significantly.

Exploratory data analysis refers to the existing data to explore under the assumption of less as far as possible, through drawing, tabulation, calculation methods to explore characteristics of data structure and regularity of a kind of analysis method. However, exploratory data by calculation method is a very general method to find the key of data. In this paper, we introduce a correlation measure for exploratory analysis based on maximal information coefficient. First, we briefly introduce the traditional data analysis methods and features, expound the necessity of exploring analysis and content. Then, correlation measurement which used commonly are expounded, summarized their characteristics and models. Therefore, we propose a weighted measure based on Maximal Information Coefficient to improve effectiveness of exploratory analysis. Then we get eigenvalues of the Maximal Information Coefficient and Pearson correlation coefficient in linear and nonlinear function of plus noise. Finally, explore analysis display visualization of the results by test dataset, emphasis direction of further research.