ITEM: Immersive Telepresence for Entertainment and Meetings—A Practical Approach

Tele-immersive systems enable 3D interactive experience in a virtual space by bringing together objects born in physical and virtual reality environments that are geographically distributed. The immersive and interactive experiences are achieved by fusing real-time color plus depth video of physical scenes from multiple stereo cameras, displaying 3D reconstructions of physical and virtual objects, and tracking intersections of objects in a virtual space to facilitate interactions between objects. While tele-immersive (TI) systems have been attracting a lot of attention these days as a possible advancement of the current 2D video-based communication systems and of the virtual reality spaces, the challenges lie in limited resources available to satisfy the many requirements on a usable TI system. One of the TI challenges remains the quality of real-time 3D reconstruction of physical scenes. Our work is motivated by the lack of understanding of the importance of various TI system parameters on the quality which is critical for allocating limited resources to make TI systems useful. This paper investigates the impact of different quality parameters of TI systems on accuracy and speed of task executions in TI environments. We have designed a task of moving a virtual ball from an initial center position to a basketball basket close to top in a closed virtual cuboid space. Successful completion of the task involves localization, orientation and motion coordination for a human in TI environment. We report quantitative and qualitative measurements collected during task executions under simulated variations of 3D reconstruction quality parameters. Based on our experimental results with 29 subjects under 13 quality variations, we concluded that the most important parameter is the frame rate followed by the presence of of a human rendering rather than just an avatar representation of hands.

Several applications in robotics require complex sensor arrangements that must be carefully calibrated, both intrinsically and extrinsically, to allow information fusion and enable the system to function as a whole. These arrangements can combine different sensing modalities--such as color cameras, laser-rangefinders, and depth cameras--in an attempt to obtain richer descriptions of the environment. Finding the location of multi-modal sensors in a common world reference frame is a difficult problem that is largely unsolved whenever sensors observe distinct, disjoint parts of the scene. This article builds on recent results in object pose estimation using mirror reflections to provide an accurate and practical solution for the extrinsic calibration of mixtures of color cameras, LRFs, and depth cameras with non-overlapping field-of-view. The method is able to calibrate any possible sensor combination as far as the setup includes at least one color camera. The technique is tested in challenging situations not covered by the current state-of-the-art, proving to be practical and effective. The calibration software is made available to be freely used by the research community.

동영상 부호화에서 움직임 추정은 구현 복잡도에 가장 크게 영향을 미친다. 본 논문에서는 깊이 카메라와 색상 카메라를 동시에 이용하여 움직임 추정의 복잡도를 줄이는 방법을 제안한다. 깊이 카메라로부터 얻어진 거리 정보로 동영상내 객체 정보를 얻고, 비슷한 거리에 있는 화소들을 그룹화하여 동일한 객체로 라벨링을 수행한다. 라벨링된 현재 및 참조 화면내에서 움직임 추정 블록을 세부분(배경, 객체내부, 경계)의 탐색영역으로 구분하여 적응적으로 판단한다. 현재 블록이 객체내부영역이면 참조 화면에서 객체내부영역에만 움직임 추정을 탐색하고, 배경영역이면 블록은 참조 화면에서 배경영역에만 탐색한다. 모의실험을 바탕으로 전역탐색방법에 비하여 제안된 방법은 움직임 추정 차신호가 동일하게 유지되면서 탐색 복잡도가 크게 줄어듬을 확인할 수 있다. Motion estimation in video coding greatly affects implementation complexity. In this paper, a reducing method of the complexity in motion estimation is proposed by using both the depth and color cameras. We obtain object information with video sequence from distance information calculated by depth camera, then perform labeling for grouping pixels within similar distances as the same object. Three search regions (background, inside-object, boundary) are determined adaptively for each of motion estimation blocks within current and reference pictures. If a current block is the inside-object region, then motion is searched within the inside-object region of reference picture. Also if a current block is the background region, then motion is searched within the background region of reference picture. From simulation results, we can see that the proposed method compared to the full search method remains the almost same as the motion estimated difference signal and significantly reduces the searching complexity.

In recent years interactive visualization of 3D data has become increasingly important and widespread due to the requirements of several application areas. However, current user interfaces often lack adequate support for 3D interactions: 2D desktop systems are often limited in cases where natural interaction with 3D content is required, and sophisticated 3D user interfaces consisting of stereoscopic projections and tracked input devices are rarely adopted by ordinary users. Touch interaction has received considerable attention for 2D interfaces, and more recently for 3D interfaces. Many touch devices now support multiple degrees of freedom input by capturing multiple 2D contact positions on the surface as well as varying levels of pressure and even depth. There is, therefore, great potential for multi-touch interfaces to provide the traditionally difficult to achieve combination of natural 3D interaction without any instrumentation. When combined with a stereoscopic display of 3D data as well as 3D depth cameras, we believe that multi-touch technology can form the basis for a next generation of intuitive and expressive 3D user interfaces. Several research groups have begun to explore the potential, limitations, and challenges of this and other 3D touch environments, and first commercial systems are already available. The goal of the SIG Touching the 3rd Dimension (T3D) is to address the research and industrial challenges involved in exploring the space where the flat digital world of surface computing meets the physical, spatially complex, 3D space in which we live. The meeting will provide a common forum to attract groups of conference attendees who share their visions of the future and recent results in the area of improving 3D interaction and visualization by taking advantage of the strengths of advanced multi-touch computing.

As electronic gadgets become more user friendly, we find natural interaction with the gadgets becoming increasingly popular. Haptic devices have become immensely popular and touch and gesture based interfaces are the logical extension to natural user interaction. We propose to use facial expressions for natural interaction, especially for gadgets like smart phones and tablets where it is more appropriate to capture human faces for user interaction. We present a use-case scenario of an eBook Reader application wherein the user performs certain facial expressions naturally to control the device while using this application. We detect, recognize and interpret these facial expressions for natural interaction with the application. More broadly we want to fit such facial expression based natural interaction to the entire device, not limiting it to a particular application. As we move forward, devices will have to be personalized to comprehend subtle and implicit cues from the user instead of waiting for an explicit input.

A view synthesis problem is to generate a virtual view based on the given one or multiple views and their associated depth maps. We adopt the depth image based rendering (DIBR) approach in this paper for synthesizing the new views. No explicit 3D modeling is involved. Another component of this study is the popular commodity RGB-D (color plus depth) cameras. The color and depth images captured by a pair of RGB-D cameras (Microsoft Kinect for Windows v2) are our inputs to synthesize intermediate virtual views between these two cameras. Several methods include depth to color warping, disocclusion filling, and color to color warping are adopted and designed to achieve this target. One of our major contributions is a new disocclusion detection algorithm proposed to improve the disocclusion filling result. Furthermore, an improved camera calibration method is proposed to make use of the additional depth information. Good quality synthesized views are shown at the end.

The practical approach and multi-agent platform development for adaptive scheduling systems for real time resource management are considered. The approach is based on concept of demand and resource networks (DRN) where agents of demands and resources operate on virtual market and continuously trying to improve their individual values of satisfaction functions that reflects given multi-criteria objectives. To achieve the best possible results agents use the virtual money account that regulates their behavior and can increase by getting bonuses or decrease by penalties depending of their individual cost functions. The key rule of designed virtual market is that any agent that is searching for new better position in schedule must compensate losses for those conflicting agents who are able and agree to change their allocations to other resources after the initial agent request, with required amount of compensation determined in the process of re-allocations. This approach allows to balance many criteria for getting consensus between agents and adaptation of the schedules “on the fly” by events without any stop and restart of the system. The developed platform includes key classes of DRN agents and protocols of their negotiations and other components that help to develop the solution manage data and visualize results of scheduling. The platform provides rapid prototyping of multi-agent systems for real time resource management and helps to reduce man-efforts and time of development. The platform was applied for developing of multi-agent scheduling systems for managing resources in aircraft jet production, load balancing in computer grid networks and energy production in power-, gas- and heating networks.

In applications such as immersive telepresence we want to extract high quality 3D models of collaborators in real time from multiview image sequences. One way to improve the quality of stereo or visual hull based models is to estimate the kinematic pose of the user first and then constrain 3D reconstruction accordingly. To serve as a preprocessing step such pose extraction must be very fast, precluding the usual generate and test techniques. We examine a method based on psychophysical evidence that known relative hand position can be used to directly compute the pose of the arm. First we explore a number of possible models for this relationship using motion capture data. We then examine how reconstruction of face and hand position as well as a patch on the torso, allow us to exploit these simple direct calculations to estimate the pose of a user in a desktop collaboration environment.

Designing of a novel depth camera is presented, which targets close-range (20-60cm) natural human-computer interaction especially for mobile terminals. In order to achieve high precision through the working range, a two-stepping method is employed to match the near infrared intensity image to absolute depth in real-time. First, we use structured light achieved by an 808nm laser diode and a Dammann grating to coarsely quantize the output space of depth values into discrete bins. Then use a learning-based classification forest algorithm to predict the depth distribution over these bins for each pixel in the image. The quantitative experimental results show that this depth camera has 1% precision over range of 20-60cm, which show that the camera suit resource-limited and low-cost application.

Respiratory pattern is a representation of human breathing activity, which can reflect people’s physical and psychological condition. Capturing the unexpected abnormal respiratory pattern unobtrusively of the patient or the potential patient has great significance. In the current work, we attempt to capitalize on depth camera and deep learning architecture to achieve the accurate and unobtrusive measurement of abnormal respiratory patterns, and the whole system can classify multiple people’s respiratory patterns in a real-time manner. The challenges in this task are threefold: 1) the real-time online system means that the Region of Interest (ROI) needs to be located and tracked automatically; 2) the amount of real-world data is not enough for training to obtain the robust deep neural network; and 3) the intraclass variation is large and the outer class variation is small. Consequently, human joints tracking is applied to determine the location of subjects shoulder and chest. Based on the characteristics of actual respiratory signals, a novel and efficient respiratory simulation model (RSM) is proposed to generate abundant and high-quality training data. Finally, we apply a gated recurrent unit (GRU) neural network with bidirectional and attentional mechanisms (BI-AT-GRU) to classify six clinically significant respiratory patterns (Eupnea, Tachypnea, Bradypnea, Biots, Cheyne–Stokes, and Central-Apnea). The performance of the obtained BI-AT-GRU is tested by the data that is actually measured by the depth camera. The experimental results demonstrate that the proposed model can classify six different respiratory patterns with the accuracy, precision, recall, and F1 of 94.5%, 94.4%, 95.1%, and 94.8%, respectively. In comparative experiments, the obtained BI-AT-GRU specific to respiratory pattern classification outperforms the existing state-of-the-art, viz ., BI-AT-LSTM, GRU, long short-term memory (LSTM), and BI-AT-GRU. Moreover, other experimental results indicate that the proposed online measuring system, deep neural network, and the modeling ideas have the potential to be extended to the large-scale applications, such as public places, sleep scenario, and office environment. The demo videos of the proposed system are available at: https://doi.org/10.6084/m9.figshare.11493666.v1 .

This paper proposes a new method for simultaneous 3D reconstruction and semantic segmentation for indoor scenes. Unlike existing methods that require recording a video using a color camera and/or a depth camera, our method only needs a small number of (e.g., 3~5) color images from uncalibrated sparse views, which significantly simplifies data acquisition and broadens applicable scenarios. To achieve promising 3D reconstruction from sparse views with limited overlap, our method first recovers the depth map and semantic information for each view, and then fuses the depth maps into a 3D scene. To this end, we design an iterative deep architecture, named IterNet, to estimate the depth map and semantic segmentation alternately. To obtain accurate alignment between views with limited overlap, we further propose a joint global and local registration method to reconstruct a 3D scene with semantic information. We also make available a new indoor synthetic dataset, containing photorealistic high-resolution RGB images, accurate depth maps and pixel-level semantic labels for thousands of complex layouts. Experimental results on public datasets and our dataset demonstrate that our method achieves more accurate depth estimation, smaller semantic segmentation errors, and better 3D reconstruction results over state-of-the-art methods.

Proceedings of the 2011 joint ACM workshop on Human gesture and behavior understanding

In traditional cloud computing research, it is often considered that the network resources between edge devices and cloud platform are sufficient, and the symmetry between the upward link from edge devices to the cloud platform and the downward link from cloud platform to edge devices is definite. However, in many application scenarios, the network resources between edge devices and cloud platform might be limited, and the link symmetry may not be guaranteed. To solve this problem, network relay nodes are introduced to realize the data transmission between edge devices and the cloud platform. The environment in which network relay nodes that can cooperate with the cloud platform is called cloud network collaborative environment (CNCE). In CNCE, how to optimize data transmission from edge devices to cloud platform through relay nodes has become one of the most important research topics. In this paper, we focus on the following two influencing factors that previous studies ignored: (1) the multi-link and multi-constraint transmission process; and (2) the timely resource state of the relay node. Taking these factors into consideration, we design a novel data transmission scheduling algorithm, called ant colony based transmission scheduling approach (ACTS). First, we propose a multi-link optimization mechanism to optimize the constraint limits. This mechanism divides the transmission into two links called the downlink relay link and uplink relay link. For the downlink relay link, we use the store-and-forward method for the optimization. For the uplink relay link, we use the min–min method for the optimization. We use the ant colony algorithm for the overall optimization of the two links. Finally, we improve the pheromone update rule of the ant colony algorithm to avoid the algorithm from falling into a local optimum. Extensive experiments demonstrate that our proposed approach has better results in transmission efficiency than other advanced algorithms.

Due to technology advances, reconstructing three-dimensional representations of physical environments in real time using cheap and non-professional cameras and PCs becomes possible. These advances will make 3-D tele-immersive environments available for everyone in the near future. However, the amount of data captured, by a Tele-Immersion (TI) system can be very large. Compression is needed to ensure real-time transmission of the data. Due to this real-time requirement, compressing TI data can be a challenging task and no current techniques can effectively address this issue. In this paper, we propose a skeleton-based compression. The main idea of this approach is to take advantage of prior knowledge of objects, e.g., human figures, in the physical environments and to represent their motions using just a few parameters. The proposed compression method provides tunable and high compression ratios (from 50:1 to 5000:1) with reasonable reconstruction quality, Moreover, the proposed method can estimate motions from the noisy data captured by our TI system in real time.

In order to improve the efficiency of locomotive organization and the quality of locomotive operation, this paper analyzes and discusses the big data platform and some key technologies suitable for the big data application of the railway locomotive system. Firstly, the definition of big data of the railway locomotive system is proposed, and the current data characteristics of the railway locomotive system are summarized, then the status quo and demands of big data application of the railway locomotive system are analyzed. Secondly, the overall architecture of the big data platform for the railway locomotive system is proposed. Furthermore, seven application scenarios available for the big data platform are analyzed, including locomotive running organization, high-speed railway, repair, maintenance and other fields. Finally, some key technologies, which consist of data collection system of front-line operations, locomotive equipment portrait analysis, staff portrait analysis, transmission and analysis of locomotive video, intelligent auxiliary driving system, are provided to increase efficiency of the locomotive organization and capability of safety management. The obtained results can play a positive role in the construction and application of big data of the railway locomotive system.