A Remote Collaboration System with Force Feedback

Remote collaboration systems allow users at different sites to perform joint tasks, which are required by many real-life applications. For example, environmental pollution is a complex problem requiring many kinds of expertise to fully understand, as pollutants disperse not only locally but also regionally or even globally. This paper presents a remote collaborative visualization system through providing co-presence, information sharing, and collaborative analysis functions based on mixed reality techniques. We start with developing an immersive visualization approach for analyzing multi-attribute and geo-spatial data with intuitive multi-model interactions, simulating co-located collaboration effects. We then go beyond by designing a set of information sharing and collaborative analysis functions to support different users to share and analyze their sensemaking processes collaboratively. We provide example results and usage scenario to demonstrate that our system enables users to perform a variety of immersive and collaborative analytics tasks effectively. Through two small user studies focusing on evaluating our design of information sharing and system usability, the evaluation results confirm the effectiveness of comprehensive sharing among user, data, physical, and interaction spaces for improving remote collaborative analysis experience.

We present a novel Mixed Reality (MR) remote collaboration system that integrates a telepresence drone to overcome viewpoint limitations in previous approaches. Current remote collaboration systems often restrict remote users to the local user's perspective or a fixed viewpoint, which may reduce spatial awareness and collaboration effectiveness. Our proposed system enables a remote user wearing a VR headset to independently navigate the local environment through a drone while guiding a local AR headset user. The drone represents the remote user physically and virtually with an AR avatar. The system provides the remote user with a 3D reconstruction of the local environment, the local user's avatar, and real-time drone camera video to facilitate drone operation and collaboration. A user study comparing our drone-enhanced MR system with 2D video and 360-degree video MR systems during inspection tasks revealed that the telepresence drone significantly improved collaboration efficiency and enhanced users' social presence and spatial presence. However, the remote users perceived a higher workload. Our findings demonstrate the benefits of independent spatial navigation through drones in remote collaboration and offer insights for future drone-enhanced MR remote collaboration systems.

Supporting natural communication cues is critical for people to work together remotely and face-to-face. In this paper we present a Mixed Reality (MR) remote collaboration system that enables a local worker to share a live 3D panorama of his/her surroundings with a remote expert. The remote expert can also share task instructions back to the local worker using visual cues in addition to verbal communication. We conducted a user study to investigate how sharing augmented gaze and gesture cues from the remote expert to the local worker could affect the overall collaboration performance and user experience. We found that by combing gaze and gesture cues, our remote collaboration system could provide a significantly stronger sense of co-presence for both the local and remote users than using the gaze cue alone. The combined cues were also rated significantly higher than the gaze in terms of ease of conveying spatial actions.

We present ConnectBoard, a new system for remote collaboration where users experience natural interaction with one another, seemingly separated only by a vertical, transparent sheet of glass. It overcomes two key shortcomings of conventional video communication systems: the inability to seamlessly capture natural user interactions, like using hands to point and gesture at parts of shared documents, and the inability of users to look into the camera lens without taking their eyes off the display. We solve these problems by placing the camera behind the screen, where the remote user is virtually located. The camera sees through the display to capture images of the user. As a result, our setup captures natural, frontal views of users as they point and gesture at shared media displayed on the screen between them. Users also never have to take their eyes off their screens to look into the camera lens. Our novel optical solution based on wavelength multiplexing can be easily built with off-the-shelf components and does not require custom electronics for projector-camera synchronization.

A tension exists between designers' comfort with physical artifacts and the need for effective remote collaboration: physical objects live in one place. Previous research and technologies to support remote collaboration have focused on shared electronic media. Current technologies force distributed teams to choose between the physical tools they prefer and the electronic communication mechanisms available. We present Distributed Designers' Outpost, a remote collaboration system based on The Designers' Outpost, a collaborative web site design tool that employs physical Post-it notes as interaction primitives. We extended the system for synchronous remote collaboration and introduced two awareness mechanisms: transient ink input for gestures and a blue shadow of the remote collaborator for presence. We informally evaluated this system with six professional designers. Designers were excited by the prospect of physical remote collaboration but found some coordination challenges in the interaction with shared artifacts.

In this research, we explore how different types of augmented gesture communication cues can be used under different view sharing techniques in a remote collaboration system. In a pilot study, we compared four conditions: (1) Pointers on Still Image, (2) Pointers on Live Video, (3) Annotation on Still Image, and (4) Annotation on Live Video. Through this study, we found three results. First, users collaborate more efficiently using annotation cues than pointer cues for communicating object position and orientation information. Second, live video becomes more important when quick feedback is needed. Third, the type of gesture cue has more influence on performance and user preference than the type of view sharing method.

Typical view sharing system has same camera alignment that camera take images from back of remote instructor. We change this alignment to camera take images from front of remote instructor for preventing occlusions caused by a body of remote instructor self. Also as visual feedbacks, a mirror image of remote instructor is indicated in display of remote instructor side. Eventually remote instructor can confirm own instruction in the display. Therefore due to displaying the mirror image of remote instructor and changing camera alignment, we proposed and implement a novel remote collaboration system which prevents occlusion problems caused by instructor body self when he/she sends clear instructions by whole body gesture and allows instructor to use direct manipulation.

Automatic Speech Recognition (ASR) technologies can be used to address people with auditory disabilities by integrating them in an interpersonal communication via textual visualization of speech. Especially in avatar-based Mixed Reality (MR) remote collaboration systems, speech is an important additional modality and allows natural human interaction. Therefore, we propose an easy to integrate ASR and textual visualization extension for an avatar-based MR remote collaboration system that visualizes speech via spatial floating speech bubbles. In a small pilot study, we achieved word accuracy of our extension of 97% by measuring the widely used word error rate.

Collaboration in scientific visualization

In this paper, we investigate the influence of the network delay on the fairness between two game players in a remote control system with free-viewpoint video and haptic media by Quality of Experience (QoE) assessment. In the assessment, we clarify the influence by changing the difference in the network delay between the two players. Assessment results show that the difference larger than about 50 ms leads to unfairness. We also demonstrate that the allowable range of the difference in a real space is wider than that in a 3D virtual space.

Recently, VNC (virtual network computing) has become one of the popular remote access techniques for Internet applications such as remote distant learning and collaborative application systems. However, there is no any multimedia functions have been implemented to enhance the performance. In this paper, we presents a framework based on remote frame buffer (RFB) and real-time protocol (RTP) to provide real-time video and audio transmissions for multimedia applications. The framework can be applied to build distance collaborative environments with multimedia and RTP (real-time transport) features efficiently.

Based on Web Service and Multi-agents technology, this paper proposes to build a remote distributed fault diagnosis system for equipments of an incineration plant which requires real-time diagnosis in a widely distributed environment. The system is modeled and implemented by multi-agent technology and is composed of a large number of agents, which are deployed at different web sites. To extend system capabilities, a user can provide additional diagnosis procedures, diagnosis services or data services. Based on diagnosis goals, relevant agents are assembled together to form a remote collaborative diagnosis system. The system can be used in any incineration plants, which greatly simplifies the deployment and maintenance efforts.

In this paper, we propose position follow-up control for dealing with hand delivery of an object between moving robot arms of remote robot systems with force feedback, and we investigate the effect of the control by experiment. We also examine influences of the network delay on the hand delivery of the object under the control. In each system, a user can operate the moving robot arm having a force sensor by using a haptic interface device while watching video. An electric hand which can hold the object is attached to the tip of the robot arm. In the position follow-up control, the position of one robot arm is automatically moved close to the other moving robot arm, and then smooth hand delivery is realized in combination with manual operation. In the experiment, we make a comparison between the case in which the position follow-up control is carried out and the case in which the control is not performed. Experimental results show that the average operation time is greatly decreased under the control, and the average operation time increases as the network delay becomes larger.

In this paper, we examine the effect of QoS (Quality of Service) control for cooperation between two remote robot systems with force feedback. Previously, we applied the adaptive Δ-causality control as QoS control and the stabilization control with filters to cooperative work of carrying an object together by using the systems and demonstrated the effectiveness of the two types of control. However, when changing the movement direction, somewhat large force was applied to the object. We may be able to solve the problem by performing the robot position control using force information, which we previously proposed as QoS control. This paper applies the control to the work, and then the effect of the control is investigated by experiment.

Network delay in haptic-based CVEs (collaborative virtual environments) severely deteriorates the haptic interaction quality (e.g., larger force feedback than real). In order to compensate this delay effect, existing studies dynamically change spring and damper coefficients according to the network delay. However, it is difficult to choose proper coefficients to offset the delay effect by precisely reflecting virtual object characteristics. In this paper, a new delay-compensation scheme based on the force feedback prediction is proposed to improve the force feedback experience. By predicting the virtual object movements and force feedback, the proposed scheme in client side provides timely force feedback to a user. Then, it gradually converges to real (but delayed) information from the server in order to maintain the consistency of virtual environment. According to the experiment results, the proposed scheme can improve the haptic interaction quality by providing more realistic force feedback similar to that of no network delay.