Developing guidelines for assessing visual analytics environments

Evaluation of software can take many forms ranging from algorithm correctness and performance to evaluations that focus on the value to the end user. This article presents a discussion of the development of an evaluation methodology for visual analytics environments. The Visual Analytics Science and Technology Challenge was created as a community evaluation resource. This resource is available to researchers and developers of visual analytics environments and allows them to test out their designs and visualization and compare the results with the solution and the entries prepared by others. Sharing results allows the community to learn from each other and to hopefully advance more quickly. In this article, we discuss the original challenge and its evolution during the 7 years since its inception. While the Visual Analytics Science and Technology Challenge is the focus of this article, there are lessons for many involved in setting up a community evaluation program, including the need to understand the purpose of the evaluation, decide upon the right metrics to use, and the appropriate implementation of those metrics including datasets and evaluators. For ongoing evaluations, it is also necessary to track the evolution and to ensure that the evaluation methodologies are keeping pace with the science being evaluated. The discussions on the Visual Analytics Science and Technology Challenge on these topics should be pertinent to many interested in community evaluations.

The 21st century will be heavily impacted by the capability to create value from recorded data of all kinds. In this regard, industrial manufacturing intuitions increasingly rely on new data-driven technologies, such as the Internet of Things or Machine Learning. In terms of data collection, manufacturing processes increasingly include sophisticated sensor equipment, which results in interconnected networks of manufacturing all assembling parts and producing data. However, manufacturing institutions currently face two challenges. First, large amounts of parts and hence data are produced during fully automated manufacturing processes. Second, due to the overwhelming amount of recorded data, it is particularly challenging to efficiently analyze manufacturing data. Hence, it is important to efficiently store and share gained knowledge from performed data analyses. Information visualization and Visual Analytics are two prolific branches of data analysis exploiting sophisticated visualization techniques to support the execution of analytical tasks and to store gained knowledge. This thesis looks at how data visualization approaches can help industrial manufacturing organizations to create value from large amounts of manufacturing data, as well as how to efficiently store and share knowledge from manufacturing data analyses. The goal is to understand how Visual Analytics can improve manufacturing processes in the context of knowledge management. Five Visual Analytics systems were designed, developed, and evaluated to tackle different domain problems that emerged from manufacturing setups. Findings from each system were used to carry out additional studies to enhance established theories of knowledge management. As a result, five success stories are provided of how Visual Analytics can significantly improve manufacturing processes and how knowledge can be efficiently created, formalized, and shared in an organization with Visual Analytics.

The practice of data science is ad hoc and agile, where needs and requirements evolve continuously and are resolved through collaboration among stakeholders. To support such practice visual analytics systems need to evolve as the needs and requirements in terms of data, users, tasks, medium, visualization, and interaction capabilities change continuously. In this paper we present a case study and illustrate several dimensions of the requirements in visual analytics. We put forward a vision of a dynamic agile visual analytics process and system model in support of data science, in which the user and system can cooperate to facilitate discovery while requirements change on demand. We argue that such a system needs an underlying language and algebra that defines not only operands and operators for performing visual analytics but also specifies guidelines that take them into account and produce useful visual analytics transformations leading to a specific insight. Our intent is not to present a fully developed system but rather a vision, illustrated through a use case. While the algebra presented here is sufficiently defined to illustrate our viewpoints, further refinement and completion is necessary to facilitate application in a visual analytics system.

Image and video data analysis has become an increasingly important research area with applications in different domains such as security surveillance, healthcare, augmented and virtual reality, video and image editing, activity analysis and recognition, synthetic content generation, distance education, telepresence, remote sensing, sports analytics, art, non-photorealistic rendering, search engines, and social media. Recent advances in Artificial Intelligence (AI) and particularly deep learning have sparked new research challenges and led to significant advancements, especially in image and video analysis. These advancements have also resulted in significant research and development in other areas such as visualization and visual analytics, and have created new opportunities for future lines of research. In this survey article, we present the current state of the art at the intersection of visualization and visual analytics, and image and video data analysis. We categorize the visualization articles included in our survey based on different taxonomies used in visualization and visual analytics research. We review these articles in terms of task requirements, tools, datasets, and application areas. We also discuss insights based on our survey results, trends and patterns, the current focus of visualization research, and opportunities for future research.

This special issue is devoted to the new research addressing challenges in the areas of visualization and visual analytics. Visualization and visual analytics are closely related research areas, both concentrating on developing visual techniques to reveal meaningful information out of various data in real-life applications. Visualization as a field has its roots in Computer Graphics and has become a popular research area over the years. The field of visual analytics is relatively young with a concentration on analytical reasoning facilitated by interactive visual interfaces. In general, visualization and visual analytics research is tightly connected with certain types of data or applications and researchers in both fields strive to discover known or unknown data patterns for domain users.

The disparity between data collected in rural and urban counties is often detrimental in the appropriate analysis of cancer care statistics. Low counts drastically affect the incidence and mortality rates of the data, leading to skewed statistics. In order to more accurately report the data, various levels of aggregation have been used (grouping counties by population, age percentages, etc.); however, such data aggregation methods have often been ad hoc and/or time consuming. Such groupings are performed on a user defined basis; however, grouping based purely on population demographics does not take into account the spatial relationships between data. Furthermore, researchers want to search for spatiotemporal correlations within their data domain. In this work, we introduce a visual analytics system for exploring cancer care statistics in a series of linked views and interactive user interface queries. We also apply the AMOEBA algorithm [1] for clustering counties based on population demographics in a visual analytics environment. Users select the population demographics field on which they wish to cluster, and these county clusters then form the basis for the data aggregation. Such a system allows the user to group their data by fields (age, gender, income) while maintaining spatial structure and provides and interactive mapping system in which to compare and explore such groupings. By utilizing such geographical groupings, we hope to better enhance the underlying structure of the data and help alleviate reporting problems associated with small area statistics.

This book discusses semantic interaction, a user interaction methodology for visual analytic applications that more closely couples the visual reasoning processes of people with the computation. This methodology affords user interaction on visual data representations that are native to the domain of the data. User interaction in visual analytics systems is critical to enabling visual data exploration. Interaction transforms people from mere viewers to active participants in the process of analyzing and understanding data. This discourse between people and data enables people to understand aspects of their data, such as structure, patterns, trends, outliers, and other properties that ultimately result in insight. Through interacting with visualizations, users engage in sensemaking, a process of developing and understanding relationships within datasets through foraging and synthesis. The book provides a description of the principles of semantic interaction, providing design guidelines for the integration of semantic interaction into visual analytics, examples of existing technologies that leverage semantic interaction, and a discussion of how to evaluate these technologies. Semantic interaction has the potential to increase the effectiveness of visual analytic technologies and opens possibilities for a fundamentally new design space for user interaction in visual analytics systems.

Visual Analytics provides with a combination of automated techniques and interactive visualizations huge analysis possibilities in technology and innovation management. Thereby not only the use of machine learning data mining methods plays an important role. Due to the high interaction capabilities, it provides a more user-centered approach, where users are able to manipulate the entire analysis process and get the most valuable information. Existing Visual Analytics systems for Trend Analytics and technology and innovation management do not really make use of this unique feature and almost neglect the human in the analysis process. Outcomes from research in information search, information visualization and technology management can lead to more sophisticated Visual Analytics systems that involved the human in the entire analysis process. We propose in this paper a new interaction approach for Visual Analytics in technology and innovation management with a special focus on technological trend analytics.

Bibliographic data such as collections of scientific articles and citation networks have been studied extensively in information visualization and visual analytics research. Powerful systems have been built to support various types of bibliographic analysis, but they require some training and cannot be used to disseminate the insights gained. In contrast, we focused on developing a more accessible visual analytics system, called SurVis, that is ready to disseminate a carefully surveyed literature collection. The authors of a survey may use our Web-based system to structure and analyze their literature database. Later, readers of the survey can obtain an overview, quickly retrieve specific publications, and reproduce or extend the original bibliographic analysis. Our system employs a set of selectors that enable users to filter and browse the literature collection as well as to control interactive visualizations. The versatile selector concept includes selectors for textual search, filtering by keywords and meta-information, selection and clustering of similar publications, and following citation links. Agreement to the selector is represented by word-sized sparkline visualizations seamlessly integrated into the user interface. Based on an analysis of the analytical reasoning process, we derived requirements for the system. We developed the system in a formative way involving other researchers writing literature surveys. A questionnaire study with 14 visual analytics experts confirms that SurVis meets the initially formulated requirements.

Collaborative visual analytics (CVA) involves sensemaking activities within teams of analysts based on coordination of work across team members, awareness of team activity, and communication of hypotheses, observations, and insights. We introduce a new type of CVA tools based on the notion of “team-first” visual analytics, where supporting the analytical process and needs of the entire team is the primary focus of the graphical user interface before that of the individual analysts. To this end, we present the design space and guidelines for team-first tools in terms of conveying analyst presence, focus, and activity within the interface. We then introduce InsightsDrive, a CVA tool for multidimensional data, that contains team-first features into the interface through group activity visualizations. This includes (1) in-situ representations that show the focus regions of all users integrated in the data visualizations themselves using color-coded selection shadows, as well as (2) ex-situ representations showing the data coverage of each analyst using multidimensional visual representations. We conducted two user studies, one with individual analysts to identify the affordances of different visual representations to inform data coverage, and the other to evaluate the performance of our team-first design with exsitu and in-situ awareness for visual analytic tasks. Our results give an understanding of the performance of our team-first features and unravel their advantages for team coordination.

Visual analytics (VA) is typically applied to complex data, thus requiring complex tools. While visual analytics empowers analysts in data analysis, analysts may get lost in the complexity occasionally. This highlights the need for intelligent assistance mechanisms. However, even the latest LLM-assisted VA systems only provide help when explicitly requested by the user, making them insufficiently intelligent to offer suggestions when analysts need them the most. We propose a ProactiveVA framework in which LLM-powered UI agent monitors user interactions and delivers context-aware assistance proactively. To design effective proactive assistance, we first conducted a formative study analyzing help-seeking behaviors in user interaction logs, identifying when users need proactive help, what assistance they require, and how the agent should intervene. Based on this analysis, we distilled key design requirements in terms of intent recognition, solution generation, interpretability and controllability. Guided by these requirements, we develop a three-stage UI agent pipeline including perception, reasoning, and acting. The agent autonomously perceives users' needs from VA interaction logs, providing tailored suggestions and intuitive guidance through interactive exploration of the system. We implemented the framework in two representative types of VA systems, demonstrating its generalizability, and evaluated the effectiveness through an algorithm evaluation, case and expert study and a user study. We also discuss current design trade-offs of proactive VA and areas for further exploration.

We present research focused on designing visual analytics (VA) systems for workers in organizational environments. We focus on business analysts and asset managers, who work collaboratively to analyze information and make decisions. Through extensive investigations in two organizational environments, we found that these users struggle with managing and analyzing information from multiple perspectives. Their current tools lack support for aggregating, organizing, and sharing such information. To address their needs, we characterized their analytic workflows, extracted specific key knowledge actions for each task commonly found in these workflows, and designed and evaluated two visual analytics systems that support and encapsulate these knowledge actions. We provide design guidelines that should be used when designing visual analytics systems, and illustrate their effectiveness with two systems built by following them.

Several visual analytics (VA) systems are used for analyzing eye-tracking data because they synergize human-in-the-loop exploration with speed and accuracy of the computer. In the VA systems, the choices of visualization techniques could afford discovering certain types of insights while hindering others. Understanding these affordances and hindrances is essential to design effective VA systems. In this paper, we focus on two approaches for visualizing AOI transitions: the transition-based approach (exemplified by the radial transition graph, RTG) and the sequence-based approach (exemplified by the Alpscarf). We captured the insights generated by two analysts who individually use each visualization technique on the same dataset. Based on the results, we identify four phases of analytic activities and discuss opportunities that the two visualization approaches can complement each other. We point out design implications for VA systems that combine these visualization approaches.

With the growing need for visualization to aid users in understanding large, complex datasets, the ability for users to interact and explore these datasets is critical. As visual analytic systems have advanced to leverage powerful computational models and data analytics capabilities, the modes by which users engage and interact with the information are limited. Often, users are taxed with directly manipulating parameters of these models through traditional GUIs (e.g., using sliders to directly manipulate the value of a parameter). However, the purpose of user interaction in visual analytic systems is to enable visual data exploration -- where users can focus on their task, as opposed to the tool or system. As a result, users can engage freely in data exploration and decision-making, for the purpose of gaining insight. In this position paper, we discuss how evaluating visual analytic systems can be approached through user interaction analysis, where the goal is to minimize the cognitive translation between the visual metaphor and the mode of interaction (i.e., reducing the "interaction junk"). We motivate this concept through a discussion of traditional GUIs used in visual analytics for direct manipulation of model parameters, and the importance of designing interactions the support visual data exploration.

Time is an important data dimension with distinct characteristics that is common across many application domains. This demands specialized methods in order to support proper analysis and visualization to explore trends, patterns, and relationships in different kinds of time-oriented data. The human perceptual system is highly sophisticated and specifically suited to spot visual patterns. For this reason, visualization is successfully applied in aiding these tasks. But facing the huge volumes of data to be analyzed today, applying purely visual techniques is often not sufficient. Visual analytics systems aim to bridge this gap by combining both, interactive visualization and computational analysis. In this paper, we introduce a concept for designing visual analytics frameworks and tailored visual analytics systems for time and time-oriented data. We present a number of relevant design choices and illustrate our concept by example.