MIVA: Multimodal Interactions for Facilitating Visual Analysis with Multiple Coordinated Views

Although graphics and direct manipulation are effective interface technologies for some classes of problems, they are limited in many ways. In particular, they provide little support for identifying objects not on the screen, for specifying temporal relations, for identifying and operating on large sets and subsets of entities, and for using the context of interaction. On the other hand, these are precisely strengths of natural language. This paper presents an interface that blends natural language processing and direct manipulation technologies, using each for their characteristic advantages. Specifically, the paper shows how to use natural language to describe objects and temporal relations, and how to use direct manipulation for overcoming hard natural language problems involving the establishment and use of context and pronominal reference. This work has been implemented in SRI's Shoptalk system, a prototype information and decision-support system for manufacturing.

The widespread use of computational simulation in science and engineering provides challenging research opportunities. Multiple independent variables are considered and large and complex data are computed, especially in the case of multi-run simulation. Classical visualization techniques deal well with 2D or 3D data and also with time-dependent data. Additional independent dimensions, however, provide interesting new challenges. We present an advanced visual analysis approach that enables a thorough investigation of families of data surfaces, i.e., datasets, with respect to pairs of independent dimensions. While it is almost trivial to visualize one such data surface, the visual exploration and analysis of many such data surfaces is a grand challenge, stressing the users' perception and cognition. We propose an approach that integrates projections and aggregations of the data surfaces at different levels (one scalar aggregate per surface, a 1D profile per surface, or the surface as such). We demonstrate the necessity for a flexible visual analysis system that integrates many different (linked) views for making sense of this highly complex data. To demonstrate its usefulness, we exemplify our approach in the context of a meteorological multi-run simulation data case and in the context of the engineering domain, where our collaborators are working with the simulation of elastohydrodynamic (EHD) lubrication bearing in the automotive industry.

Complex combinations of coordinated multiple views are increasingly used to design tools for highly interactive visual exploration and analysis of multidimensional data. While complex coordination patterns provide substantial utility through expressive querying, they also exhibit usability problems for users when learning required interaction sequences, recalling past queries, and interpreting visual states. As visual analysis tools grow more sophisticated, there is a growing need to make them more understandable as well. Our long-term goal is to exploit natural language familiarity and literacy to directly facilitate individual and collaborative use of visual analysis tools. In this poster, we present work in progress on an automatically generated query-to-question user interface to translate interactive states during visual analysis into an accompanying visual log of formatted text. Our effort currently focuses on a symmetric and thus relatively simple coordination pattern: cross-filtered views. We describe our current thinking about query-to-question translation in a typical cross-filtered visualization of movies, people, and genres in the Internet Movie Database.

A multimodal user interface allows users to communicate with computers using multiple modalities, such as a mouse, a keyboard or voice, in various combined ways. This paper discusses a multimodal drawing tool, whereby the user can use a mouse, a keyboard and voice effectively. Also, it describes an interpretation method, by which the system integrates voice inputs and pointing inputs using context.

In this paper, we present our design study on developing an interactive visual firewall log analysis system in collaboration with an IT service provider. We describe the human-centered design process, in which we additionally considered hedonic qualities by including the usage of personas, psychological need cards and interaction vocabulary. For the problem characterization we especially focus on the demands of the two main clusters of requirements: high-level overview and low-level analysis, represented by the two defined personas, namely information security officer and network analyst. This resulted in the prototype of a visual analysis system consisting of two interlinked parts. One part addresses the needs for rather strategical tasks while also fulfilling the need for an appealing appearance and interaction. The other part rather addresses the requirements for operational tasks and aims to provide a high level of flexibility. We describe our design journey, the derived domain tasks and task abstractions as well as our visual design decisions, and present our final prototypes based on a usage scenario. We also report on our capstone event, where we conducted an observed experiment and collected feedback from the information security officer. Finally, as a reflection, we propose the extension of a widely used design study process with a track for an additional focus on hedonic qualities.

This paper shows how the integration of natural language with direct manipulation produces a multimodal interface that overcomes limitations of these techniques when used separately. Natural language helps direct manipulation in being able to specify objects and actions by description, while direct manipulation enables users to learn which objects and actions are available in the system. Furthermore, graphical rendering and manipulation of context provides a partial solution to difficult problems of natural language anaphora.

Business analytics involves using various technologies to analyze data. Data mining focuses on the automated search for knowledge, patterns, or regularities in data. As a business analyst, it is essential to recognize the type of analytical technique appropriate for solving a specific problem. Exploratory Data Analysis (EDA) describes data using statistical and visualization techniques to highlight important aspects of that data for further analysis. This involves examining a data set from many angles, describing it, and summarizing it without making assumptions about its content. Exploratory data analysis is an essential step before diving into statistical modeling or machine learning to ensure that the data is really what it claims to be and that there are no apparent errors. This type of analysis should be part of data science projects in every organization. Visual analytics is sometimes confused with data visualization. Visual analysis is not simply a matter of graphically representing data. Modern, interactive visual analytics makes combining data from multiple sources easy and performs in-depth data analysis directly in the visualization. Additionally, artificial intelligence and machine learning algorithms can offer recommendations for exploration. Ultimately, visual analytics helps transform massive data sets into business insights that can positively impact an organization. Considering the previous comment about visual data analysis, it should be added that the system has extensive capabilities to create graphical dashboards containing reports and charts. It is essential that in the case of desktops, in addition to the visualizations included in the system itself, it is possible to embed reports from third-party tools.

With the rapid development of global maritime trade, the demand for visual analysis of ship activities is growing. Without very professional knowledge, users are difficult to quickly select appropriate visual methods to analyze ship activities and face the problem of “cold start”. Based on multisource heterogeneous ship activity data, this paper constructs a ship activity knowledge graph including four types of ship activity elements of “time, place, ship and activity”. Combining with the needs of ship activity analysis, four analysis tasks are designed, including ship object feature analysis, ship activity event analysis, ship activity correlation analysis and ship activity space-time analysis; Besides, based on the knowledge graph of ship activities, the visual analysis method of ship activities based on activity elements is designed, and the visual analysis model of ship activity knowledge graph is designed combined with the interactive modes of filtering, association, and navigation. In this paper, the AIS data and the “boiling ring trade” maritime event data are used as the ship activity data. Based on the above model, a prototype system for visual analysis of the ship activity knowledge graph is developed. The results show that the system can recommend relatively appropriate visual methods for users, provide more perspectives for the analysis of ship activities, and also verify the conclusions such as the changes in the laws of ship activities caused by special circumstances such as the closed fishing periods.

With the growing need for visualization to aid users in understanding large, complex datasets, the ability for users to interact and explore these datasets is critical. As visual analytic systems have advanced to leverage powerful computational models and data analytics capabilities, the modes by which users engage and interact with the information are limited. Often, users are taxed with directly manipulating parameters of these models through traditional GUIs (e.g., using sliders to directly manipulate the value of a parameter). However, the purpose of user interaction in visual analytic systems is to enable visual data exploration -- where users can focus on their task, as opposed to the tool or system. As a result, users can engage freely in data exploration and decision-making, for the purpose of gaining insight. In this position paper, we discuss how evaluating visual analytic systems can be approached through user interaction analysis, where the goal is to minimize the cognitive translation between the visual metaphor and the mode of interaction (i.e., reducing the "interaction junk"). We motivate this concept through a discussion of traditional GUIs used in visual analytics for direct manipulation of model parameters, and the importance of designing interactions the support visual data exploration.

Comparison is a core task in visual analysis. Although there are numerous guidelines to help users design effective visualizations to aid known comparison tasks, there are few techniques available when users want to make ad hoc comparisons between marks, trends, or charts during data exploration and visual analysis. For instance, to compare voting count maps from different years, two stock trends in a line chart, or a scatterplot of country GDPs with a textual summary of the average GDP. Ideally, users can directly select the comparison targets and compare them, however what elements of a visualization should be candidate targets, which combinations of targets are safe to compare, and what comparison operations make sense? This article proposes a conceptual model that lets users compose combinations of values, marks, legend elements, and charts using a set of composition operators that summarize, compute differences, merge, and model their operands. We further define a View Composition Algebra (VCA) that is compatible with datacube-based visualizations, derive an interaction design based on this algebra that supports ad hoc visual comparisons, and illustrate its utility through several use cases.

Article Free Access Share on A generic platform for addressing the multimodal challenge Authors: Laurence Nigay Laboratoire de Génie Informatique (LGI-IMAG), BP 53, 38041 Grenoble Cedex 9, France Laboratoire de Génie Informatique (LGI-IMAG), BP 53, 38041 Grenoble Cedex 9, FranceView Profile , Joëlle Coutaz Laboratoire de Génie Informatique (LGI-IMAG), BP 53, 38041 Grenoble Cedex 9, France Laboratoire de Génie Informatique (LGI-IMAG), BP 53, 38041 Grenoble Cedex 9, FranceView Profile Authors Info & Claims CHI '95: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsMay 1995 Pages 98–105https://doi.org/10.1145/223904.223917Published:01 May 1995Publication History 115citation867DownloadsMetricsTotal Citations115Total Downloads867Last 12 Months53Last 6 weeks13 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteView all FormatsPDF

In this paper, we present the multimodal interface and analysis system which is based on biometric signals and applicable to contents. The multimodal interface includes a biometric analysis module that analyzes and recognizes human biometric signal patterns. The biometric multimodal interface can recognize a user's emotion and concentration status by analyzing ECG(electrocardiogram) and EEG(electroencephalogram) patterns. The electroencephalogram analysis system utilizes 5 basic signal values to predict the concentration status of the user: MID_BETA, THETA, ALPHA, DELTA, and GAMMA signal. To recognize the user's electrocardiogram signal patterns, K-means-based EM algorithm was applied. In emotion recognition, the neural emotion showed the highest accuracy, and three emotions were in a range of 55.8 to 75.1% accuracy. Stress recognition showed a high performance result of 83.2% accuracy.

With the maturity technology of Internet platform and multi-user social network, the reference value of group user consumption experience is expanding. In the massive comment data, negative comments are particularly important for enterprises and consumers. Therefore, effective visual analysis of negative comments is necessary. According to the multidimensional and multivariate characteristics of comment data, the paper takes negative comments as the starting point and defines their scopes. Furthermore, the paper proposes an interactive visual analysis system, which provides enterprises and consumers with a new perspective of comment analysis. Firstly, the paper uses the emotion analysis and opinion mining methods to process user data, and proposes a quantitative strategy for the difference of the consumer individual influences. Secondly, the paper designs a series of interactive visual representation methods, such as the thematic sentiment ripple graph and the comment comparison view. Then, a multi-dimensional correlation view is constructed to explore the factors that affect the generation and causes of negative comments, and their individual differences with dynamic interactive methods. Finally, three case studies are used to verify the effectiveness and practicality of the system, which can also be extended to comment data of other visual analysis fields.

The curse of dimensionality, i.e., the fact that feature spaces of increasing dimensionality with finite sample sizes tend to be empty, has given incentive to a plethora of research activities in various disciplines and diverse application fields, e.g., statistics or neural networks. Three major application fields are multivariate data classification, data analysis and data visualization. In this contribution, methods for dimensionality reduction from three decades of interdisciplinary research are browsed and their applicability in the above application domains is briefly discussed. Complementing techniques for ensuing interactive data visualization, data navigation and visual exploratory data analysis are presented, which exploit the remarkable human perceptive and associative capabilities for interactive visual exploratory data analysis and systematic recognition system design. The main focus of this paper is on the comparison of feature selection and feature extraction techniques and the potential benefit of their combination. Further, the interesting implications of dimensionality reduction for VLSI design and related area and power consumption are pointed out.

Visual impact analysis: a case study of a computer-based system