Foreground Based Borderline Adjusting for Real Time Multi-camera Video Stitching

This paper proposes a novel approach to stitch videos fast and with high quality. In general, scene depth is frequently varying for dynamic video content. For example, moving foreground objects will move closely or far from a camera. The scene depth will be changing according to changed scene content. Therefore, accurate projection transform estimation corresponding to current scene depth is required in video stitching algorithms. However, projection transform estimation is a time-consuming task. To tackle this problem, we propose an algorithm to estimate projection transforms fast. The algorithm calculates initial projection transforms in a camera position calibration stage first. The projection transforms for successive frames are then obtained approximately based on a depth compensation method. In addition, a pixel indexing scheme is designed to speed up pixel projection between source video frames and panorama video frames. Using above optimization strategies, our video stitching algorithm can work fast and with high visual quality. Several real video stitching experiments are carried out. The experimental results show that our algorithm can improve stitching visual quality obviously with acceptable computing cost.

This article has been retracted. A retraction notice can be found at https://doi.org/10.3233/JIFS-219433.

Agnostic attribute segmentation of dynamic scenes with limited spatio-temporal resolution

A consensus-based method for tracking: Modelling background scenario and foreground appearance

When multiple viewers attend to the same static naturalistic scene there is a high degree of agreement in terms of the regions viewers attend to but no synchronization of when they attend there. By comparison, real-world scenes are rich with temporally-defined visual events such as motion which have the potential to involuntarily capture attention. Initial studies of attention during dynamic scenes have reported a high degree of spatiotemporal agreement, henceforth referred to as Attentional Synchrony. However, these dynamic scenes are composed with viewer attention in mind e.g. film and TV footage depicting a single subject tracked by a moving camera. Therefore, it is not currently known if attentional synchrony is caused by the presence of motion, scene content, or compositional factors.

Panoramic video has existed for a relatively long time. Different from panorama pictures, panoramic video requires front-end interaction. The view of dynamic scenes can be controlled via both interactive devices and smart phones. In this paper, we describe the initiative of a convenient browser-to-browser application for streaming video using WebRTC tested on a peer-to-peer module. With optional adaption to various kinds of panoramic cameras, this web-based system permits the user to experience interactively web-based panoramic video streams.

The problem of multiple people detection in monocular video streams is addressed. The proposed method involves a human model based on skin color and foreground information. Robustness to local motion of background and global color changes is achieved by modeling images as fields of color distributions, and robustly estimating temporal background global variations. The estimation of the human model parameters is done via Monte Carlo simulations to deal with the multimodal nature of the posterior distribution, introduced by the presence of multiple people and cluttered scene. Promising results are presented for transportation vehicles sequences

Background subtraction in real applications: Challenges, current models and future directions

We present a novel hierarchical MRFs optimization method for dense and deformable motion extraction in dynamic scenes. In particular, this hierarchical MRFs structure consists of two layers, the segmentation and the correspondence layer. Firstly, dynamic RGB-D foreground data is segmented through a pixel-level MRF in the segmentation layer. Subsequently, the extracted foreground data is transformed into a 3D point-level MRF in the correspondence layer. A new surface descriptor named deformable color and shape histogram is proposed. It is combined with photometric and geometric features to represent a deformable surface. Finally, the dynamic scene motion is retrieved from correspondences established in the image sequence. Discrete optimization schemes are used for the binary classification and multi-labeling problems. We provide an RGB-D dataset of dynamic scenes, which involves different motion patterns and surface properties of foreground objects. The effectiveness and efficiency of our proposed approach for high accurate foreground segmentation and motion extraction is validated in experiments.

Simultaneous localization and map building (SLAM) is crucial in autonomous robot navigation. However, existing SLAM systems generally assume a static environment, which makes it difficult to cope with the interference caused by moving objects in dynamic scenes, affecting the system’s localization accuracy and robustness. To address this challenge, this paper proposes YKD-SLAM, a visual SLAM system for indoor dynamic environments, which is based on the ORB-SLAM2 framework and incorporates YOLOv8 target detection, RCF-KMeans (Region-ConstrainedFastK-Means), and epipolar geometric constraints to realize the accurate rejection of dynamic feature points and improve the localization performance in dynamic environments. YKD-SLAM first uses YOLOv8 to detect dynamic objects in the scene, generates a detection frame, optimizes the depth map through open operations, and performs multi-region segmentation of the region within the detection frame by combining RCF-KMeans. Subsequently, through the dynamic feature point rejection strategy based on epipolar geometric constraints, different regions in the detection frame are discriminated into dynamic and static regions, and the feature points in the dynamic region are rejected to improve the localization accuracy and robustness of the system in dynamic environments. The experimental results show that YKD-SLAM performs well in several dynamic scenes in the TUMRGB-D dataset. Compared with ORB-SLAM2, its ATE is reduced by 98.37%; compared with DynaSLAM, the system operation efficiency is improved by 95.35%. In addition, practical experiments conducted in indoor dynamic scenes further validate its potential in real applications.

A scene can be defined as one of the subdivisions of a play in which the setting is fixed, or when it presents continuous action in one place. We propose a novel two-pass algorithm for scene boundary detection, which utilizes the motion content, shot length and color properties of shots as the features. In our approach, shots are first clustered by computing Backward Shot Coherence (BSC) - a shot color similarity measure that detects Potential Scene Boundaries (PSBs) in the videos. In the second pass we compute Scene Dynamics (SD), a function of shot length and the motion content in the potential scenes. In this pass, a scene merging criteria has been developed to remove weak PSBs in order to reduce over segmentation. We also propose a method to describe the content of each scene by selecting one representative image. The segmentation of video data into number of scenes facilitates an improved browsing of videos in electronic form, such as video on demand, digital libraries, Internet. The proposed algorithm has been tested on a variety of videos that include five Hollywood movies, one sitcom, and one interview program and promising results have been obtained.

This paper presents a hybrid pixel-based background (HPB) model, which is constructed by single stable record and multi-layer astable records after initial learning. The image foreground object detection must face the problems of moving background, illumination changes, chaotic, etc. in real word applications. In our approach, the HPB model can be used for background subtraction to extract objects precisely in various complex scenes. Using the multi-layer astable records, we also propose the homogeneous background subtraction that can detect the foreground object with less record memory. Based on the benchmark videos, the experimental results show that single stable and 3-layer multi-layer astable records can be enough for background model construction and then updated quickly to overcome the background variation. The proposed approach can improve the averages Error Rate of foreground object detection up to 86% when comparing with the latest works.

Chapter 2 - Multi-View Calibration, Synchronization, and Dynamic Scene Reconstruction

With the limited field of view of human vision, our perception of most scenes is built over time while our eyes are scanning the scene. In the case of static scenes, this process can be modeled by panoramic mosaicing: stitching together images into a panoramic view. Can a dynamic scene, scanned by a video camera, be represented with a dynamic panoramic video even though different regions were visible at different times? In this paper, we explore time flow manipulation in video, such as the creation of new videos in which events that occurred at different times are displayed simultaneously. More general changes in the time flow are also possible, which enable re-scheduling the order of dynamic events in the video, for example. We generate dynamic mosaics by sweeping the aligned space-time volume of the input video by a time front surface and generating a sequence of time slices in the process. Various sweeping strategies and different time front evolutions manipulate the time flow in the video, enabling many unexplored and powerful effects, such as panoramic movies.

A method is presented for extracting object information from an image sequence taken by a static monocular camera. The method was developed towards a low computational complexity in order to be used in real-time surveillance applications. Our approach makes use of both intensity and edge information of each frame and works efficiently in an indoor environment. It consists of two major parts: background processing and foreground extraction. The background estimation and updating makes the object detection robust to environment changes like illumination changes and camera jitter. The fusion of intensity and edge information allows a more precise estimation of the position of the different foreground objects in a video sequence. The result obtained are quite reliable, under a variety of environmental conditions.