Visuo-Tactile Interaction with Virtual Objects that Yields Kinetic Effects on Real Objects

This paper proposes a Blended Reality Agent called "BReA" that can exist in both the real and virtual world and has communicative advantages that is more than simply the sum of those of a robotic agent and an on-screen agent. BReA can seamlessly transfer between the real world and virtual world and users can communicate with BReA without interruption during the transference. We conducted a field test and showed that users can recognize when it points from the virtual world to a real world object after it moves from the real world to virtual world. This result supports a hypothesis that communicating with BReA helps users recognize the link between real world and virtual world objects and virtual world information.

Hand-tracking technologies allow us to use our own hands to interact with real and virtual objects in Augmented Reality (AR) environments. This enables us to explore the interplay between hand-visualizations and hand-object interactions. We present a user study that examines the effect of different hand visualizations (invisible, transparent, opaque) on manipulation performance when interacting with real and virtual objects. For this, we implemented video-see-through (VST) AR-based virtual building blocks and hot wire tasks with real one-to-one counterparts that require participants to use gross and fine motor hand movements. To evaluate manipulation performance, we considered three measures: task completion time, number of collisions (hot wire task), and percentage of object displacement (building block task). Additionally, we explored the sense of agency and subjective impressions (preference, ease of interaction, successful and awkwardness) evoked by the different hand-visualizations. The results show that (1) manipulation performance is significantly higher when interacting with real objects compared to virtual ones, (2) invisible hands lead to fewer errors, higher agency, higher perceived success and ease of interaction during fine manipulation tasks with real objects, and (3) having some visualization of the virtual hands (transparent or opaque) overlayed on the real hands is preferred when manipulating virtual objects even when there are no significant performance improvements. Our empirical findings about the differences when interacting with real and virtual objects can aid hand visualization choices for manipulation tasks in AR.

An extensible infrastructure and a representation scheme for distributed smart virtual proxies of real world objects as presented in this Master's Thesis could be a foundation to make the environment smart by enhancing simple objects with smart virtual counterparts. Design, implementation and deployment issues of concepts such as virtual counterparts (Virtual Objects), proximity (Virtual Location), meta objects (Virtual Meta Objects and Locations), lifecycle of virtual counterparts (Virtual Object Manager) and a queriable persistent Artifact Memory are discussed.

Recent advancements in the Internet of Things (IoT) and the Web of Things (WoT) accompany a smart life where real world objects, including sensing devices, are interconnected with each other. The Web representation of smart objects empowers innovative applications and services for various domains. To accelerate this approach, Web of Objects (WoO) focuses on the implementation aspects of bringing the assorted real world objects to the Web applications. In this paper; we propose an emergency fire management system in the WoO infrastructure. Consequently, we integrate the formation and management of Virtual Objects (ViO) which are derived from real world physical objects and are virtually connected with each other into the semantic ontology model. The charm of using the semantic ontology is that it allows information reusability, extensibility and interoperability, which enable ViOs to uphold orchestration, federation, collaboration and harmonization. Our system is context aware, as it receives contextual environmental information from distributed sensors and detects emergency situations. To handle a fire emergency, we present a decision support tool for the emergency fire management team. The previous fire incident log is the basis of the decision support system. A log repository collects all the emergency fire incident logs from ViOs and stores them in a repository.

This paper describes an approach to dealing with mutual occlusions between virtual and real objects on a table-top display. Display tables use stereoscopy to make virtual content appear to exist in 3 dimensions on or above a table top. The actual image, however, lies on the physical plane of the display table. Any real physical object introduced above this plane therefore obstructs our view of the display surface and disrupts the illusion of the virtual scene. The occlusions result between real objects and the display surface, not between real objects and virtual objects. For the same reason virtual objects cannot occlude real ones. Our approach uses an additional projector located near the user's head to project those parts of virtual objects that should occlude real ones directly onto the real objects. We describe possible applications and limitations of the approach and its current implementation. Despite its limitations, we believe that the proposed approach can significantly improve interaction quality and performance for mixed reality scenarios.

Mixed reality is different from the virtual reality in that users can feel immersed in a space which is composed of not only virtual but also real objects. Thus, it is essential to realize seamless integration and interaction of the virtual and real worlds. We need depth information of the real scene to synthesize the real and virtual objects. We propose a two-stage algorithm to find smooth and precise disparity vector fields with sharp object boundaries in a stereo image pair for depth estimation. Hierarchical region-dividing disparity estimation increases the efficiency and the reliability of the estimation process, and a shape-adaptive window provides high reliability of the fields around the object boundary region. At the second stage, the vector fields are regularized with a energy model which produces smooth fields while preserving their discontinuities resulting from the object boundaries. The vector fields are used to reconstruct 3D surface of the real scene. Simulation results show that the proposed algorithm provides accurate and spatially correlated disparity vector fields in various kinds of images, and synthesized 3D models produce natural space where the virtual objects interact with the real world as if they are in the same world.

Augmented reality (AR) is anticipated for use in medical practice. However, virtual objects cannot be overlaid exactly on real objects using theoretical data. Therefore, this study is intended to determine the correction method to overlay a virtual object with real object accurately. Experimental results confirmed that differences of perceived depth position between real and virtual objects can be made negligible (within 1 mm on average) by correcting sizes of virtual objects such that they can be seen identically to the real object.

We introduce the problem of computing a human-perceived softness measure for virtual 3D objects. As the virtual objects do not exist in the real world, we do not directly consider their physical properties but instead compute the human-perceived softness of the geometric shapes. In an initial experiment, we find that humans are highly consistent in their responses when given a pair of vertices on a 3D model and asked to select the vertex that they perceive to be more soft. This motivates us to take a crowdsourcing and machine learning framework. We collect crowdsourced data for such pairs of vertices. We then combine a learning-to-rank approach and a multi-layer neural network to learn a non-linear softness measure mapping any vertex to a softness value. For a new 3D shape, we can use the learned measure to compute the relative softness of every vertex on its surface. We demonstrate the robustness of our framework with a variety of 3D shapes and compare our non-linear learning approach with a linear method from previous work. Finally, we demonstrate the accuracy of our learned measure with user studies comparing our measure with the human-perceived softness of both virtual and real objects, and we show the usefulness of our measure with some applications.

Sophisticated multimodal interfaces are the key to success in future computer development and many ideas have been proposed so far. Augmented reality (AR) is such an idea. Computer-generated digital objects are mixed with objects in the real world, and they are perceptually assimilated. People don't need to be aware of the computer. What to do for them is just behave as usual in the real world. However, interaction with the objects is limited in existing AR systems. In fact, many of the systems allow the user just to change his/her viewpoint to real world objects. There are some other systems which accept manipulation of virtual objects, but they assume users wear or hold special devices. In this paper we propose a handheld AR interface supporting two interaction techniques based on the understanding that both real and virtual objects are compatible in manipulation. One of the techniques is for the user to directly manipulate virtual objects by his/her finger. One more technique concerns indirect manipulation of virtual objects. Motion of a handheld device serves as a trigger to make a certain change to target virtual objects.

One of the main problems of monoscopic video see-through augmented reality (AR) is the lack of reliable depth information. This makes it difficult to correctly represent complex spatial interactions between real and virtual objects, e.g., when rendering shadows. The most obvious graphical artifact is the incorrect display of the occlusion of virtual models by real objects. Since the graphical models are rendered opaquely over the camera image, they always appear to occlude all objects in the real environment, regardless of the actual spatial relationship. In this paper, we propose to utilize a new type of hardware in order to solve some of the basic challenges of AR rendering. We introduce a depth-of-flight range sensor into AR, which produces a 2D map of the distances to real objects in the environment. The distance map is registered with high resolution color images delivered by a digital video camera. When displaying the virtual models in AR, the distance map is used in order to decide whether the camera image or the virtual object is visible at any position. This way, the occlusion of virtual models by real objects can be correctly represented. Preliminary results obtained with our approach show that a useful occlusion handling based on time-of-flight range data is possible.

Inherent to the field of computer-aided surgery (CAS) is the necessity to handle sophisticated equipment in the operating room; an undesired side-effect of this development is the fact that the surgeon’s attention is drawn from the operating field since surgical progress is partially monitored on the computer’s screen. The overlay of computer-generated graphics over a real-world scene, also referred to as augmented reality (AR), provides a possibility to solve this problem. The considerable technical problems associated with this approach such as viewing of the scenery within a common focal range on the head-mounted display (HMD) or latency in display on the HMD have, however, kept AR from widespread usage in CAS. In this paper, the concept of the Varioscope AR, a lightweight head-mounted operating microscope used as a HMD, is introduced. The registration of the patient to the preoperative image data as well as preoperative planning take place on VISIT, a surgical navigation system developed at our hospital. Tracking of the HMD and stereoscopic visualisation take place on a separate POSIX.4 compliant realtime operating system running on PC hardware. While being in a very early stage of laboratory testing, we were able to overcome the technical problems described above; our work resulted in an AR visualisation system with an update rate of 6 Hz and a latency below 130 ms. First tests with 2D/3D registration have shown a match between 3D world coordinates and 2D HMD display coordinates in the range of 1.7 pixels. It integrates seamlessly into a surgical navigation system and provides a common focus for both virtual and real world objects. On the basis of our current results, we conclude that the Varioscope AR with the realtime visualisation unit is a major step towards the introduction of AR into clinical routine.

Augmented Reality (AR) is the combination of digital information and user's target in real world. The handheld devices like Smartphone have all the components that are necessary to use an augmented reality application, where the virtual three dimensional objects imported on real world environment. In AR based mobile applications utilizing an image target trackers are growing. In these continue growing trends can lead the applications to require more interaction technique interfaces, which will attract the user attentions (e.g. hand interaction). In this proposes work a new approach used to dynamic nature hand interaction on Virtual Reality (VR) objects through the mobile based AR application. Virtual link and buttons are placed on the image target surface and interact by the human 3-D hand. Hand detection algorithm used for fast detecting and track the hand on virtual objects. By this proposed system the user can directly change the behavior and position of the virtual object by human hand. Virtual links and buttons are allowing the users to view other virtual objects on same virtual space. Filter data technique by reducing redundancies and creating layout for remaining information on virtual objects for change the behavior or functions of the virtual interface objects.

Augmented Reality and Mixed Reality technologies make us feel the real world and the virtual world are mixed with each other through visual matching. However, existing mixed reality contents using these technologies have a limitation that interact only with virtual objects and cannot interact with real objects. If a dynamic real object need to be influenced by a virtual object, or vice versa, illusion of Mixed Reality can be broken by the incongruity of the interaction. If these limitations are improved, real objects, which were always static, can operate dynamically and interact with virtual objects, so becomes difficult to distinguish between real objects and virtual objects. In this paper, we propose extended interactions in a mixed reality environment that can give illusion which is difficult to distinguish between real objects and virtual objects. And to confirm this, we implement authoring environment based Hololens.

A hologram is described as a 'screenless display' that provides a realism and three-dimensional effect as if seeing a real object, and refers to a technology that implements a three-dimensional stereoscopic image using laser light. These holograms are currently being used in various cultural industries such as performances, exhibitions, and advertisements. In particular, pseudo-holograms such as floating techniques are mainly used in the cultural heritage exhibition field. However, pseudo-holograms can implement the same effect as holograms that show virtual 3D images in the real world, but there are still practical and technical limitations in satisfying audiences who want perfect 3D images like those seen in science fiction movies. The hologram, which is attracting attention as a future high-value-added imaging technology, can naturally reproduce real or virtual objects in a three-dimensional space as if they were real. In addition, holography technology is expected to lead the growth of the immersive content industry in the future by improving the technical limitations of the current augmented reality(AR) or virtual reality(VR). Therefore, if the current holography technology is gradually improved and supplemented with hologram screen, hologram printing, and plenoptic visualization technology, it is expected that the field of digital heritage contents and exhibitions using hologram will further develop in the future.

Deep learning has become the main way of the object detection task for autonomous vehicles. Meanwhile, this method typically requires vast amounts of training data to reach their full potential. However, collecting the data from real world and labeling manually is an expensive, time-consuming and error-prone process. Synthetic image has the potential to replace real image for training neural networks, because image creation and labeling annotations are free in this way. For the network trained by synthetic images, the reality gap between real and synthetic images is the main obstacle to use it in the real world. And most previous works are only devoted to generate synthetic images with a good performance on model training, but lack of analysis of the domain gap that affects the performance. This work designs a method of generating the real and synthetic images to analyze and reduce the reality gap between synthetic and real images for car detection. Firstly, this work put one single car with no-background in a random background image to generate real single car and synthetic single car images. In order to further reduce the domain gap in content level, this method keeps the car distribution in synthetic images is similar with the distribution of car in real world. For the purpose of reducing the domain gap in appearance level, the parameters of camera model are same as the camera parameters of image collecting cars and the image is rendered by using the PBRT(Physically Based Ray Tracing) when we generated the synthetic images. Secondly, by training the neural network of instance segmentation with different datasets, the across validation result proves that the reality gap between synthetic and real images is no more than the domain gap between real images. Thirdly, the training results of datasets with different samples diversity show that the diversity of the samples yields better generalization between different datasets for car detection which can effectively reduce the domain gap.