A study on eye-tracking-based Interface for VR/AR education platform

In this paper we propose vrLib, a toolkit which aims to make 3D graphical user interface and interaction very easy to design in virtual environments. This project provides well known and new interaction techniques and ready to use widgets, which are totally hardware independent. The main contribution allows to elaborate complex 3D user interfaces in same way it is done in 2D. This toolkit provides adaptive interaction, which means interaction techniques can automatically adapt their behavior to 2D and 3D widgets. This short paper describes the vrLib architecture for construction of 3D interactive applications.

The thesis presents the research of inventing a novel and natural spatial human-system interface – MagicPad that is to be used in a fully immersive virtual environment. With the advance in Virtual Reality (VR) technology, modern computer systems are capable of immersing users into a synthetic environment. Instead of sitting in front of the monitor, virtual objects are surrounding the users in an immersive VR system and therefore the traditional human-computer interfaces such as mouse and keyboard are no longer appropriate / fit for such purpose. Since then, researchers have been actively seeking new means of human-computer interaction in the three-dimensional space. The objective of this research is to innovate, design and build a novel spatial user interface for immersive VR systems by investigating and identifying the core elements of an intuitive and effective spatial user interface. The research methodology adopted is based on the quantitative research methods where the perception and interaction of human subjects with a synthetic virtual environment such user interfaces are collected to investigate the performance and effectiveness of the spatial user interface. The key elements of an intuitive and effective spatial user interface are first identified and the design concept of the MagicPad user interface prototype is then established. In order to realize the overall design and study the concept of the spatial human-system interface, an immersive Virtual Reality system imseCAVE, which serves as the test bed of the MagicPad user interface is designed and built from scratch. In this research, the MagicPad interface is implemented and quantitative user evaluations with the imseCAVE are carried out to draw conclusion and future research directions in the field of Virtual Reality and 3D User Interfaces.

Recently, a lot of researches have been conducted on a platform that provides a Visual Programming development environment capable of 3D editing and interactive editing in an In-VR (Virtual Reality) environment so that VR/AR (Augmented Reality) contents can be quickly worked on. In this paper, a functional language-based VR/AR specialized language(VAL) and Visual Language were developed, and VR/AR prototypes were easily implemented using these languages on the platform. Then, prototypes based on the 5G network can be shared and collaborated with others, and content creation can be continued in Unity and Unreal without interruption. In addition, this paper shows the recognition process for users, the PS/CS check process and the node processing process for transmitting content by limiting the process for transmitting VAL-based mass data in 5G environment. The proposed platform can easily create a prototype of their intended VR/AR App (or content) even for users (beginners) who do not have a certain level of knowledge and experience in 3D graphics and software coding, and share it with others. It has characteristics that can be done. Consequently, we propose a method to use VAL more effectively in a 5G environment.

PurposeThe purpose of this paper is to present a prototype pluggable service‐oriented virtual learning environment, enabling teachers to create an integrated teaching environment using tools that have been chosen to best meet their academic requirements.Design/methodology/approachThis is an implementation of a WAFFLE Bus. A microkernel software design pattern is used to enable tools to be added and removed from the system. An enterprise service bus is used to provide workflow and message transformation functionality. Tools are managed through web service interfaces and Shibboleth is used to effect interoperability at the web application user interface. The initial services for the prototype were chosen to implement a simple web service teaching workflow.FindingsFirst, Shibboleth is shown to provide a solution to the virtual learning environment tools' interoperability problem. Second, the service‐oriented virtual learning environment naturally leads to the ability to operate with many different types of information channels in and out of the system. This leads to a multiplicity of possible types of context‐dependent user interface. Third, immersive 3D, possibly the most interesting interface, will provide a context amenable to even the smallest development teams for the introduction of artificial intelligence into teaching. Finally, web service workflow is shown to provide a viable option for the implementation of learning designs with advantages and disadvantages compared to existing approaches.Research limitations/implicationsDifferent types of information channels are associated with different security problems. It will be important to determine what the best ways are of establishing secure channels to student personal learning environments. The present web service workflow design tools are of the highest quality and usability, but the design process is still a job for a specialist. It might be possible, however, to modify these open source tools to bring the design process within the grasp of non‐specialists.Originality/valueThe software system presented herein represents one possible path leading away from VLE monolithy using a service‐oriented approach. A new solution to the tools' interoperability problem is presented along with a multi‐faceted approach to the user interface. The enterprise service bus creates a flexible platform for the delivery of web service teaching and learning workflows. It is posited that the use of an immersive 3D user interface will create a context that facilitates the introduction of an artificial intelligence layer into the virtual learning environment that can serve robot teaching avatars.

Virtual environments and other applications of computer-generated scenes are emerging. This increases the demand for tools for efficient creation of the virtual worlds and interaction with the components in the virtual world. Introduction of immersiveness in the modeling process seems promising, but a number of issues related to immersive modeling still have to be investigated. The authors present a framework for an immersive modeling system with a hybrid 2D/3D user interface. The designer is immersed in the design space, which improves his awareness of the design objects under consideration. 2D and 3D human-computer interaction techniques are combined in the user interface to improve the efficiency of the modeler. While 3D interaction is natural for a number of design tasks, 2D interaction is particularly helpful for interaction with menus and dialog boxes, for precise manipulations and for editing operations that are subject to constraints.

Desktop environments represent a powerful user interface and have been used as the de facto standard human-computer interaction paradigm for over 20 years. But the rising demand of 3D applications dealing with complex datasets exceeds the capabilities of traditional interaction devices and two-dimensional displays. Such applications need more immersive and intuitive interfaces. In order to be accepted by the users, technology-driven solutions that require inconvenient instrumentation, e.g., stereo glasses or tracked gloves, should be avoided. Autostereoscopic display environments equipped with tracking systems enable humans to experience virtual 3D environments more naturally, for instance via gestures, without having to use annoying devices. However, currently these approaches are used only for specially designed or adapted applications. In this paper we introduce new 3D user interface concepts for such setups which require minimal instrumentation of the user and can be integrated easily in everyday working environments. We propose an interaction framework which supports simultaneous display of and simultaneous interaction with both monoscopic as well as stereoscopic contents. We identify the challenges for combined mouse-, keyboard- and gesture-based input paradigms in such an environment and introduce novel interaction strategies.KeywordsHCIautostereoscopic display environments3D user interfaces

This paper presents a distributed, multi-user virtual reality testbed for a mobile robot. The main components of the testbed include user interface, networking, simulation and robot control. The user interface is implemented as a combination of a virtual environment and a standard graphical user interface (GUI). The virtual environment is developed using the Virtual Reality Modeling Language (VRML) and it corresponds to the laboratory environment where the robot is placed and tested. The standard GUI is implemented using Java Programming language. The World Wide Web (WWW) browser is used to integrate the virtual environment (embedded VRML) and the standard GUI (Java applet) in a single user interface. Users can access a dedicated WWW server and download the user interface. Networking component (communication among users and robots) is implemented using the Distributed Interactive Simulation (DIS) protocol. The existing robot development system has been utilized to provide simulation capabilities. A separate stand-alone Java application has been developed to communicate and control the robot using a wireless Ethernet link. In addition to user's direct control, control algorithms (e.g. fuzzy logic based) can be used to replace the user. A Nomad200 mobile robot has been used for initial measurements and evaluations of the testbed.

Eye tracking is a process that identifies a specific point in both space and time that is being looked at by the observer. This information can also be used in real-time to control applications using the eyes. Recent innovations in the video game industry include alternative input modalities to provide an enhanced, more immersive user experience. In particular, eye gaze control has recently been explored as an input modality in video games. This book is an introduction for those interested in using eye tracking to control or analyze video games and virtual environments. Key concepts are illustrated through three case studies in which gaze control and voice recognition have been used in combination to control virtual characters and applications. The lessons learned in the case studies are presented and issues relating to incorporating eye tracking in interactive applications are discussed. The reader will be given an introduction to human visual attention, eye movements and eye tracking technologies. Previous work in the field of studying fixation behavior in games and using eye tracking for video game interaction will also be presented. The final chapter discusses ideas for how this field can be developed further to create richer interaction for characters and crowds in virtual environments. Alternative means of interaction in video games are especially important for disabled users for whom traditional techniques, such as mouse and keyboard, may be far from ideal. This book is also relevant for those wishing to use gaze control in applications other than games.

We present the Responsive Multimedia System, an interactive edutainment system. The system consists of an AR-based tabletop tangible user interface (ARTable) and a virtual environment manager (VEManager) integrated by enhanced Personalized User Interface Framework. In this system, a user interface and virtual environments are integrated seamlessly by sharing user-centric context. To share user-centric context efficiently, our system optimizes the number of network connection between objects in real and virtual environments. The synchronization problem between ARTable and a virtual environment is solved by calculating coordinates with additional information such as collision with terrain and physics law in a virtual environment as well as user-centric context. Moreover, it reorganizes responses of an object according to a user-centric context and condition-action rules. To demonstrate the effectiveness of the proposed system we developed an application on the subject 'Dream of Mee-luck'. According to the experimental results, we found that the proposed system could be a promising technology that enables users to experience an interactive story. In the future, the proposed system can play an important role in developing interactive applications for edutainment.

User interfaces are important for streamlining the interactions between humans and computers. However, there are few effective approaches for collecting users’ preferences implicitly and objectively for the purpose of user interface (UI) design optimization. This paper presents an effective approach to interactive genetic algorithm (IGA) optimization-based UI design via eye tracking, including eye-movement data based users’ preference inferring, gene coding for real UI components and design features, and the visualization and interaction mechanisms. Then we design and build a prototype system that applies IGA to generate and optimize music player UI solutions automatically. An evaluation of our prototype system suggests that it can generate and identify personalized UI designs reliably with minimal user intervention and high efficiency and user satisfaction.

Information and Communication Technology (ICT) has revolutionized science and commerce and has increased the innovation and the spread of a variety of virtual environments applications. These innovations are the result of the both technological development and cognitive studies. The chapter aims to underline the relationships between Human-Computer Interaction (HCI) and consumer behaviour, focusing the attention on the 3-D virtual environments dedicated to electronic and Internet e-commerce (e-retail) services. We introduce how the 3-D interfaces can contribute to the successful impact of online retail. The importance of the relationship between customer and system concerns the effective potentiality of the user interface. If a user interface is ineffective, the system’s functionalities and usefulness will be limited and the users will be confused, frustrated, and annoyed, and therefore less likely to use the system again. Finally, we aim to outline the cognitive and technological aspects involved in the communication process between user and virtual e-retail system interface and directions for possible future research.

At the time of this paper, the advances in light field technology already offer 3D displays that immerse the users without the need for additional viewing devices. Despite the numerous advantages and attractive capabilities of such glasses-free 3D displays, their user interface methods are quite complicated and they are currently underwhelming when compared to conventional 2D displays, due to the fact that visual feedback can only be rendered sharply on the emission surface of light field displays. The sharp rendering of user interfaces is a necessity, as blur may hinder their fundamental functions. When it comes to 2D displays, many user interaction techniques and interfaces have been devised. Rendering a user interface on a 2D display could be done in various ways, such as rendering overlays on top of the rendered scene, or by using billboards. These are extensively used in modern video games. User interaction methods have proven their importance and added efficiency to virtual environments throughout the years. Due to their overall value and usefulness, interaction techniques develop immediately as new types of displays arise. With the recent advancements in visualization technologies, user interfaces have been redesigned for use in AR, VR and MR visualization. This includes on-screen augmentation, which enables interaction with visual content on the screen. Although light field displays contain immense potentials, only basic user interfaces have been devised thus far, including FOX (Focus Sliding surface), which grants users the option to scale and to rotate 3D objects. In this paper, we visualize the theater model on real light field displays and we test the different interactions by means of a monitor room. The theatre model is analogous to real-life theatres, where viewers may observe the theatrical presentation on the stage from various angles. The motivation to choose the theater model was the fact that light field visualization similarly allows multiple simultaneous viewers within its field of view, in which the content can be observed in an angle- dependent manner. Moreover, from the users’ perspective, the theater model is thus familiar and it provides high-quality visual feedback. Furthermore, theater stages encompass a lot of interactions, including rigging and flying systems, pulleys, rotating stages, lights, curtains etc. In order to test the different interaction methods on light field displays, a theater model depicting the virtual environment was implemented. Methods for rendering the monitor room and the results of the interactions are discussed in the paper, illustrated by images of the actual visualization on light field displays. It is shown that producing plausible results with no noticeable visual artifacts is challenging, yet possible. The scientific contributions of the paper also highlight the various novel user interfaces for future light field systems and services.

The paper examines the necessity and possibility of interface interaction of the information modeling program Revit Autodesk with the common office program MS Excel. Such interaction makes it easier to prepare project documentation in accordance with the requirements of Russian standards, GOST. An analysis of the capabilities of the built-in Revit tools for creating specifications and their design according to GOST requirements showed an insufficient level of automation of these processes. Ready-made foreign and Russian solutions in the field of Revit plugins that implement interaction with Excel have a number of disadvantages. The purpose of this article is to demonstrate the capabilities of the application programming interface (API) for creating applications using the example of a developed software solution for automating the unloading of information from a BIM model into an Excel electronic document, in order to bring data presentation standards in line with GOST requirements. The application is developed using a tool environment: the C# programming language, the MS Visual Studio development environment, the WPF user interface creation tool, the Revit-side application programming interface .Net API, an interface for interacting with Excel ClosedXML files. As a result, the interface of the developed plugin is presented. The program allows you to export documents-statements and specifications - to Excel, designed according to the requirements of GOST. The user-friendly tab-based user interface, the compliance of the program design with the Revit system and the implemented functionality give the program advantages over its analogues. The application is available for free download to any Revit user, which is of practical importance.

An integrated data flow visual language and software development environment

An appropriate user interface to collect human demonstration data for deformable object manipulation has been mostly overlooked in the literature. We present an inter-action design for demonstrating cloth folding to robots. Users choose pick and place points on the cloth and can preview a visualization of a simulated cloth before real-robot execution. Two interfaces are proposed: A 2D display-and-mouse interface where points are placed by clicking on an image of the cloth, and a 3D Augmented Reality interface where the chosen points are placed by hand gestures. We conduct a user study with 18 participants, in which each user completed two sequential folds to achieve a cloth goal shape. Results show that while both interfaces were acceptable, the 3D interface was more suitable for understanding the task, and the 2D interface was suitable for repetition. Results also found that fold previews improve three key metrics: task efficiency, the ability to predict the final shape of the cloth, and overall user satisfaction.