Interaction with an agent in Blended Reality

Mixed reality is different from the virtual reality in that users can feel immersed in a space which is composed of not only virtual but also real objects. Thus, it is essential to realize seamless integration and interaction of the virtual and real worlds. We need depth information of the real scene to synthesize the real and virtual objects. We propose a two-stage algorithm to find smooth and precise disparity vector fields with sharp object boundaries in a stereo image pair for depth estimation. Hierarchical region-dividing disparity estimation increases the efficiency and the reliability of the estimation process, and a shape-adaptive window provides high reliability of the fields around the object boundary region. At the second stage, the vector fields are regularized with a energy model which produces smooth fields while preserving their discontinuities resulting from the object boundaries. The vector fields are used to reconstruct 3D surface of the real scene. Simulation results show that the proposed algorithm provides accurate and spatially correlated disparity vector fields in various kinds of images, and synthesized 3D models produce natural space where the virtual objects interact with the real world as if they are in the same world.

Inherent to the field of computer-aided surgery (CAS) is the necessity to handle sophisticated equipment in the operating room; an undesired side-effect of this development is the fact that the surgeon’s attention is drawn from the operating field since surgical progress is partially monitored on the computer’s screen. The overlay of computer-generated graphics over a real-world scene, also referred to as augmented reality (AR), provides a possibility to solve this problem. The considerable technical problems associated with this approach such as viewing of the scenery within a common focal range on the head-mounted display (HMD) or latency in display on the HMD have, however, kept AR from widespread usage in CAS. In this paper, the concept of the Varioscope AR, a lightweight head-mounted operating microscope used as a HMD, is introduced. The registration of the patient to the preoperative image data as well as preoperative planning take place on VISIT, a surgical navigation system developed at our hospital. Tracking of the HMD and stereoscopic visualisation take place on a separate POSIX.4 compliant realtime operating system running on PC hardware. While being in a very early stage of laboratory testing, we were able to overcome the technical problems described above; our work resulted in an AR visualisation system with an update rate of 6 Hz and a latency below 130 ms. First tests with 2D/3D registration have shown a match between 3D world coordinates and 2D HMD display coordinates in the range of 1.7 pixels. It integrates seamlessly into a surgical navigation system and provides a common focus for both virtual and real world objects. On the basis of our current results, we conclude that the Varioscope AR with the realtime visualisation unit is a major step towards the introduction of AR into clinical routine.

최근, 현실세계와 가상세계를 결합하여 디지털 컨텐츠 형태로 보여주는 증강 현실 기술이 등장하고 있다. 증강현실 기술의 효과를 극대화하기 위해서, 사용자가 현실세계의 사물을 조작하는 것과 유사한 방법으로 디지털 콘텐츠와 상호작용하는 것을 가능하게 해 주는 실감형 인터페이스가 적용된다. 특히, 교육 분야에서는 이러한 기술들이 학습자의 흥미와 몰입도를 높이고, 학습 효과를 극대화할 수 있는 새로운 학습 콘텐츠의 제작을 가능하게 할 것으로 기대되고 있다. 본 논문에서는 멀티 터치스크린과 실감형 인터페이스를 적용한 과학 실험 학습 시스템을 제안한다. 이 시스템은 종래의 실험대 위에서 이루어지는 실험도구의 조작을 대체하기 위하여 대형 멀티 터치스크린을 장착한 실험 테이블과 간단한 사용자의 제스쳐를 인식할 수 있는 실감형 학습 디바이스를 사용한다. 실제 과학 실험에서는 높은 비용이나 긴 시간, 또는 위험 물질들이 요구되기도 하는데, 본 시스템은 이러한 요인들을 극복하면서 학습자들에게 다양한 실험을 현실감 있게 제공할 수 있다. Recently, Augmented Reality(AR) technologies have been emerged, which shows the types of digital contents integrating real and virtual worlds. To maximize the effect of AR technology, tangible user interface, which enables users to interact with the contents in the same way in which they manipulate objects in real world, is applied. In particular, we expect that the technologies are able to enhance learners' interests and degree of immersion, and produce new learning contents in order to maximize the effect of learning. In this paper, we propose a learning system for scientific experiments with multi-touch screen and tangible user interface. The system consists of an experiment table equipped with a large multi-touch screen and a realistic learning device that can detect the user's simple gestures. In real world, some scientific experiments involve high cost, long time or dangerous objects, but this system will overcome such hindrance and provide learners with a variety of experiment experience in realistic ways.

Extended-Reality Technologies: An Overview of Emerging Applications in Medical Education and Clinical Care.

Connecting users to virtual worlds within MPEG-V standardization

For many of us, obesity is the outcome of an energy imbalance: more energy input than expenditure. However, our waistlines are growing in spite of the huge amount of diets and fat-free/low-calorie products available to cope with this issue. Even when we are able to reduce our waistlines, maintaining the new size is very difficult: in the year after the end of a nutritional and/or behavioral treatment obese persons typically regain from 30% to 50% of their initial losses. A possible strategy for improving the treatment of obesity is the use of advanced information technologies. In the past, different technologies (internet, virtual reality, mobile phones) have shown promising effects in producing a healthy lifestyle in obese patients. Here we suggest that a new technological paradigm - Interreality – that integrates assessment and treatment within a hybrid experiential environment - including both virtual and real worlds - has the potential to improve the clinical outcome of obesity treatments. The potential advantages offered by this approach are: (a) an extended sense of presence: Interreality uses advanced simulations (virtual experiences) to transform health guidelines and provisions in experiences; (b) an extended sense of community: Interreality uses virtual communities to provide users with targeted – but also anonymous, if required - social support in both real and virtual worlds; (c) real-time feedback between physical and virtual worlds: Interreality uses bio and activity sensors and devices (smartphones) both to track in real time the behavior/health status of the user, and to provide targeted suggestions and guidelines. This paper describes in detail the different technologies involved in the Interreality vision. In order to illustrate the concept of Interreality in practice, a clinical scenario is also presented and discussed: Daniela, a 35-year-old fast-food worker with obesity problems.

Virtual reality (VR) technology has become a growing force in entertainment, education, science, and manufacturing due to the capability of providing users with immersive experiences and natural interaction. Although common input devices such as controllers, gamepads, and trackpads have been integrated into mainstream VR systems for user-content interaction, they cannot provide users with realistic haptic feedback. Some prior work tracks and maps the physical objects into the virtual space to allow users to interact with these objects directly, which improves users’ sense of reality in the virtual environment. However, most of them use additional hardware sensors, which inevitably increases the cost. In this research, a lightweight approach is proposed to synchronize the positions and motions between physical and digital objects without any extra costs. We use the real-time captured video data from in-built cameras on a VR headset and employ feature points based algorithms to generate projections of the physical objects in the virtual world. Our approach does not rely on additional sensors but just uses components available in a VR headset. Our approach allows users to interact with target objects with their hands directly without the need for specially designed trackers, markers, and other hardware devices as used in previous work. With our approach, users can get more realistic operational feedback when interacting with corresponding virtual objects.

In today’s world, people increasingly rely on mobile apps to do their day-today activities, like checking their Instagram feed and online shopping from websites like Amazon and Flipkart. People are depending on WhatsApp and Instagram stories to communicate with local businesses and to leverage the said platforms for online advertising. Using Google Maps to find their way when they travel and finding out the immediate road and traffic conditions with digital banners around the road, has obviously led to a boom in advertising and marketing. Recently, internet users have increasingly desired to immerse themselves in a Metaverse-like platform where they can interact and socialize. Meta’s Metaverse is a tightly connected network of 3D digital spaces that allow users to escape into a virtual world. It is designed to change the way you socialize, work, shop, and connect with the real and virtual world around you. These platforms are not fully submerged in the real world; they are inclined toward virtual spaces only, making it obvious to fill this gap. Thus, the proposed framework in this chapter would be a new kind of system that may develop a socio-meta platform, powered by augmented reality and other technologies like photogrammetry and LiDAR. Augmented reality provides an interactive way of experiencing the real world, where the objects of the natural world are enhanced by computer-generated perceptual vision. One of the significant problems with augmented reality is the process of building virtual 3D objects that can be augmented into real spaces, which could be solved with photogrammetry and LiDAR. Photogrammetry is the technique of producing 3D objects using 2D images of a physical object taken from different angles and orientations. LiDAR, on the other hand, is another 3D reconstruction technology used widely by Apple’s eco-system. The functioning of LIDAR is very similar to sonar and radar, and the detection and ranging part are where it stands out from the others. The idea behind this platform is to open tons of virtual dimensions in the real world using the principles of mixed reality and geographic mapping tools such as Google Maps, MapBox, and GeoJSON; it would consequently transition the way people spend their time on social media by opening a portal for generating 3D objects that can be augmented to the real-world location using photogrammetry and cloud anchors by just a few 2D digital photographs taken from their camera.

Both augmented-reality and virtual world applications must model semantic knowledge about real- or virtual world objects. The current generation of virtual world platforms provides limited facilities for representing this kind of knowledge, but a next generation will provide the means to tie semantic information to general or application-specific ontology services. This article motivates the need for ontology services, outlines several approaches for associating ontology concepts with objects and locations, and discusses how to populate common-sense ontologies using data harvested from real and virtual worlds.

We present a novel paradigm for human to human asymmetric collaboration. There is a need for people at geographically separate locations to seamlessly collaborate in real time as if they are physically co-located. In our system one user (novice) works in the real world and the other user (expert) works in a parallel virtual world. They are assisted in this task by an Intelligent Agent (IA) with considerable knowledge about the environment. Current tele-collaboration systems deal primarily with collaboration purely in the real or virtual worlds. The use of a combination of virtual and real worlds allows us to leverage the advantages from both the worlds.

This study is about the new expressive techniques for promotional videos for culture contents and aims to propose a new design model for making effective promotional videos using augmented reality technology and E-museum concepts. Augmented reality is one of the up-rising intelligent information technology, but has been barely used for promotional videos. However the technology enables the interaction between physical objects in the real world and virtual image though digital devices. Augmented reality opens the door for connecting real and virtual world, and invites humans to the virtual environment. There are growing numbers attempt to move off-line culture contents into on-line space. Considering the connection between culture contents and digital environment, more studies about the convergence is necessary in the cultural-art industry, especially for marketing business. Therefore, this study aims to take a brief look at augmented reality technology, and proposes the new experimental model and expressive techniques for cultural-art organizations’ promotional video, which set the experiential learning environment, and enables consumers to take contents actively.

Recent advancements in the Internet of Things (IoT) and the Web of Things (WoT) accompany a smart life where real world objects, including sensing devices, are interconnected with each other. The Web representation of smart objects empowers innovative applications and services for various domains. To accelerate this approach, Web of Objects (WoO) focuses on the implementation aspects of bringing the assorted real world objects to the Web applications. In this paper; we propose an emergency fire management system in the WoO infrastructure. Consequently, we integrate the formation and management of Virtual Objects (ViO) which are derived from real world physical objects and are virtually connected with each other into the semantic ontology model. The charm of using the semantic ontology is that it allows information reusability, extensibility and interoperability, which enable ViOs to uphold orchestration, federation, collaboration and harmonization. Our system is context aware, as it receives contextual environmental information from distributed sensors and detects emergency situations. To handle a fire emergency, we present a decision support tool for the emergency fire management team. The previous fire incident log is the basis of the decision support system. A log repository collects all the emergency fire incident logs from ViOs and stores them in a repository.

Augmented Reality Systems (ARS) attempt to enhance humans’ perception of their indoors and outdoors working and living environments and understanding of tasks that they need to carry out. The enhancement is effected by complementing the human senses with virtual input. For example, when the human visual sense is enhanced, an ARS allows virtual objects to be superimposed on a real world by projecting the virtual objects onto real objects. This provides the human user of the ARS with additional information that he/she could not perceive with his/her senses. In order to receive the virtual input and sense the world around them augmented with real time computer-generated features, users of an ARS need to wear special equipment, such as head-mounted devices or wearable computing gears. Tracking technologies are very important in an ARS and, in fact, constitute one challenging research and development topic. Tracking technologies involve both hardware and software issues, but in this chapter we focus on tracking computation. Tracking computation refers to the problem of estimating the position and orientation of the ARS user’s viewpoint, assuming the user to carry a wearable camera. Tracking computation is crucial in order to display the composed images properly and maintain correct registration of real and virtual worlds. This tracking problem has recently become a highly active area of research in ARS. Indeed, in recent years, several approaches to vision-based tracking using a wearable camera have been proposed, that can be classified into two main categories, namely “marker-based tracking” and “marker-less tracking.” In this chapter, we provide a concise introduction to vision-based tracking for mobile ARS and present an overview of the most popular approaches recently developed in this research area. We also present several practical examples illustrating how to conceive and to evaluate such systems.

In this presentation, we describe an HMD-based immersive VR environment which allow us to touch virtual objects and to apply actual kinetic effects on corresponding objects in real world. We utilize foci of airborne ultrasound for generating tactile feedback to users’ hands and applying kinetic force to the real objects. Our system offers a new tele-operation system, where bare-handed users can manipulate real objects located away from them with tactile feedback.

In this paper we analyze the similarities and differences between traditional (physical, verbal, relational, sexual) and electronic forms of violent behavior among peers. Violent behavior, which is a characteristic of the real world (insults, disinformation, slander) has been extended to the virtual world, but modern technology has also allowed: a) new forms (flaming, electronic harassment and intimidation, false imprisonment, assault videotaping and sexting) and b) more intense and brutal youth participation in violent activities, most likely because of the anonymity, indirect contact with the victim and deflection of responsibility. Violence perpetrators in the real and virtual worlds both have similar characteristics (lack of empathy and shame, neglect of moral norms, a tendency to minimize personal guilt and responsibility, ignoring the consequences and victim suffering, and similar family problems), and the victims experience almost identical consequences: psychosomatic, emotional (especially anxiety, depression, suicidal thoughts), educational, as well as difficulties in establishing and maintaining social contacts. Because of the large exposure and serious risks to psychological health, safety and welfare of children, there is a need to educate children, parents, teachers and doctors about the possible threats in the real and virtual worlds, and scientific research should provide tools for effective prevention and targeted interventions to reduce peer violence. Conclusion - With regard to peer-to-peer violence in the real world, modern technology has made new modalities and the more intensive participation possible of children in violent activities, but the perpetrators of these forms of violence have similar characteristics, and the victims identical consequences.