Improved Video Object Segmentation Method Based on Background Registration

This paper proposes an effective method for video object segmentation without background construction. In the proposed method, the coarse foreground extraction and fine foreground extraction are obtained using the motion information, edge information, and gradient-variation information which are first evaluated by two successive frames. Next, the video object is extracted using the horizontal/vertical filling scheme based on the coarse foreground extraction and fine foreground extraction. Finally, video object refining is used to obtain the more accurate video object. Experimental results show that the proposed method has good performance in sensitivity, specificity, and spatial accuracy.

Detection of video objects under bad weather and poor illumination condition is a challenging task. We address this issue using the notion of background modeling. LBP-based background modeling and model learning has been used to detect the video objects but performance degrades in the above complex background. We propose the adjacency- and reinforced adjacency-based variants of LBP for complex real-world background modeling and model learning for object detection. In this regard, we have proposed the four variants; (1) enhanced adjacent local binary pattern, (2) enhanced reinforced adjacent local binary pattern, (3) enhanced adjacent local ternary pattern, and (4) enhanced reinforced adjacent local ternary pattern. Besides, we have embedded the Gabor and LBP features to obtain an embedded feature, which is subsequently used with the notions of adjacency. Unlike the background learning approach where the model learns only the background, our model learning algorithm learns the background together with the foreground objects and hence named as mixed-mode learning strategy. These models together with the new learning strategy are tested with CD2014 (snowfall and blizzard) and PETS (2014 and 2016) data sets, and the performance of the proposed models has been compared with LBP-based methods and other state-of-the-art methods.

A novel automatic Video Object (VO) segmentation method is presented in this paper, which is based on intersection of frame differences. Horizontal scan is used to acquire coarse VO mask and edge detection is performed on VO boundaries to remove uncovered background contained in the intersection. And morphological operator <i>open</i> is applied to smooth VO contours after extraction. Experimental results show that it is accurate and especially efficient, and can wonderfully meet the real-time requirements of applications such as stationary camera video surveillance.

Due to its lack of performance and flexibility, the traditional frame difference method does not handle precise moving object detection by motion of the object region in each frame of the various video sequences. So, the object cannot be seen in the foreground accurately. It remains a serious concern. The time computation for object detection using the three frame difference and five frame difference approaches is longer, and frame information is lost. To address these flaws, a new method known as the inter frame difference method is suggested (MIFD). It detects moving objects in video under various environmental conditions with little data loss in a short period of time. MIFD involves constructing a reference frame, computing inter frame difference, a motion frame and detecting moving object(s) in a frame by drawing a rectangle blobs using connected components in the video sequence. The performance of the proposed algorithm is compared with the previous results of the code book model (CB), self-organizing background subtraction method (SOBS), local binary pattern histogram (LBPH), robust background subtraction for network surveillance in H.264, GMM, VIBE, frame difference, three frame difference, improved three frame difference, and combined three frame difference & background subtraction model. The experimental results demonstrate that the proposed methodology performance is better than the other methods in accurately detecting moving object(s) in video under challenging environmental conditions.KeywordsVideo surveillanceModified inter frame difference methodMotion frameObject detectionConnected components

Video object segmentation and tracking are two essential building blocks of smart surveillance systems. However, there are several issues that need to be resolved. Threshold decision is a difficult problem for video object segmentation with a multi-background model. In addition, some conditions make robust video object tracking difficult. These conditions include nonrigid object motion, target appearance variations due to changes in illumination, and background clutter. In this paper, a video object segmentation and tracking framework is proposed for smart cameras in visual surveillance networks with two major contributions. First, we propose a robust threshold decision algorithm for video object segmentation with a multi-background model. Second, we propose a video object tracking framework based on a particle filter with the likelihood function composed of diffusion distance for measuring color histogram similarity and motion clue from video object segmentation. The proposed framework can track nonrigid moving objects under drastic changes in illumination and background clutter. Experimental results show that the presented algorithms perform well for several challenging sequences, and our proposed methods are effective for the aforementioned issues.

Many background subtraction techniques have been proposed to detect changes such as moving objects in videos. In this paper, we classify them into two approaches according to forms of the regions detected as moving object. The first one is using frame difference and the second one is exploiting background model. We show characteristics of the two approaches and the relational properties among their products. Then we also show these properties can be used for estimating another result from one approach and identifying the region should be updated in adaptive background model.

In the new MPEG-4 video coding standard, automatic video object segmentation plays a key role in supporting object-oriented coding and enabling content-based functionalities. Background subtraction is one of the basic automatic video object segmentation methods. But various environmental illumination conditions often make it hard to work. A robust background subtraction method is presented in this paper. A statistical background model is first setup in this algorithm. Then the hypothesis testing is applied to the following frames to segment the video objects. The HSV color model is used and its color components are efficiently analyzed and treated separately so that the proposed algorithm can adapt to different environmental illumination conditions. Shadows are detected and a new background update algorithm is also presented based on the observation that the illumination changes are temporal and will not influence all the following frames. All of them contribute to the robustness of the method. The experimental results show that the proposed background subtraction method can automatically segment video objects robustly and accurately in various illuminating environments.

High-quality video object segmentation is a challenging visual computing task. Interactive segmentation can improve segmentation results. This paper proposes a multi-round interactive dynamic propagation instance-level video object segmentation network based on click interaction. The network consists of two parts: a user interaction segmentation module and a bidirectional dynamic propagation module. A prior segmentation network was designed in the user interaction segmentation module to better segment objects of different scales that users click on. The dynamic propagation network achieves high-precision video object segmentation through the bidirectional propagation and fusion of segmentation masks obtained from multiple rounds of interaction. Experiments on interactive segmentation datasets and video object segmentation datasets show that our method achieves state-of-the-art segmentation results with fewer click interactions.

We present a deep learning method for the interactive video object segmentation. Our method is built upon two core operations, interaction and propagation, and each operation is conducted by Convolutional Neural Networks. The two networks are connected both internally and externally so that the networks are trained jointly and interact with each other to solve the complex video object segmentation problem. We propose a new multi-round training scheme for the interactive video object segmentation so that the networks can learn how to understand the user's intention and update incorrect estimations during the training. At the testing time, our method produces high-quality results and also runs fast enough to work with users interactively. We evaluated the proposed method quantitatively on the interactive track benchmark at the DAVIS Challenge 2018. We outperformed other competing methods by a significant margin in both the speed and the accuracy. We also demonstrated that our method works well with real user interactions.

In the new MPEG-4 video coding standard, the semi-automatic video segmentation plays a key role insupporting object-oriented coding and enabling content-based functionalities. A novel hierarchy optical flow basedsemi-automatic video segmentation method is presented in this paper. The proposed segmentation method containsspatial and temporal segmentation. For the spatial segmentation, a point-based graphic user interface (PBGUI) ispresented, with which the user can input easily, and then active contour model and tracking bug algorithm are applied toprecisely define the video object of interest to be segmented. With the result of spatial segmentation, the temporalsegmentation involves non-rigid object contour tracking and rigid object whole-tracking by hierarchy optical flowalgorithm based on the Lucas-Kanade algorithm. And the tracking point selection algorithm is proposed to greatlyimprove the tracking performance in the rigid object whole-tracking. The experimental results show that the proposedalgorithm can precisely segment video objects from video streams.Keywords: MPEG-4, optical flow, video segmentation, object-oriented coding

Emerging multimedia applications demand content-based video processing. Content based video retrieval or concept detection systems require video to be segmented in to objects. A large number of video object segmentation algorithms have been developed such as semiautomatic and automatic. Semiautomatic methods requires the human intervention and also not suitable for some applications. Many applications require automatic segmentation but still there is lot of scope for the improvement. The proposed work aims at contributing to identify the lacunas that are present in the existing segmentation system and also to give the possible solutions to overcome those lacunas so that the precise and effective video segmentation system can be developed. The proposed system deals with the issue of automatic video segmentation system such as uncovered background, Temporary poses and Global motion of background.

Video Object Segmentation (VOS) is essential for the automated quantitative analysis of complex mouse behaviors. However, multi-mouse segmentation remains challenging, as highly similar visual features, rapid motion patterns, and frequent inter-object occlusions often cause tracking drift and identity (ID) switching in existing VOS methods, particularly over long video sequences. To address these challenges, we propose Refinement-based Multi-Mouse Realtime Video Segmentation (RMMRVS), a closed-loop framework that improves temporal stability and spatial accuracy in long-term multi-object tracking. RMMRVS extends the CUTIE architecture by introducing a feature-aware confidence prediction module that continuously assesses tracking reliability and detects potential drift online. When confidence degradation is detected, an adaptive refinement mechanism is activated, employing Mask2Former to perform high-precision spatial resegmentation. The refined masks are then reinjected into the VOS memory, restoring a stable and reliable reference for subsequent tracking. This “pre-dict-evaluate-correct” strategy effectively suppresses error accumulation and ID switching over extended sequences. Extensive experiments on challenging multi-mouse datasets demonstrate that RMMRVS consistently outperforms baselines in segmentation accuracy and temporal consistency, highlighting its effectiveness and scalability for reliable quantitative behavioral analysis.

As a critical technology for computer vision and video processing, video object segmentation has far-going pragmatism significance and application importance. In this paper, a video object segmentation algorithm based on background reconstruction is proposed to extract moving objects from video sequences, which were taken by stationary cameras. Firstly, the change detection is used to achieve the mask representing moving regions with an estimation noise parameter, then the methods of maximum in eight-neighbor regions is present to fill the interior holes. Secondly, the background image is available by mapping the mask to the correspondence frame of sequences, then the comparison of frame difference mask is adopted to rebuild the background image, in this way, video objects which have long stayed in the background will be removed from the moving regions after they turn to be stationary. Finally, the initial video object is derived in each frame by subtracting the background from this image, after that, mathematic morphology post-processing is used to get an accurate video object. Experiments on typical sequences have successfully demonstrated the validity of the proposed algorithm.

This paper proposes an unsupervized offline video object segmentation method that introduces a number of improvements to existing work in the area. It consists of the following steps. The initial segmentation utilizes object color and motion variance to more accurately classify image pixels in the first frame. Histogram-based merging is then employed to reduce oversegmentation of the first frame. During object tracking, segmentation quality measures based on object color and motion contrast are taken. These measures are then used to enhance video objects through selective pixel reclassification. After object enhancement, cumulative histogram-based merging, occlusion handling, and island detection are used to help group regions into meaningful objects. Compared to two reference methods, greater success and improved accuracy in segmenting video objects are first demonstrated by subjectively examining selected frames from a set of standard video sequences. Objective results are obtained through the use of a set of measures that aim at evaluating the accuracy of object boundaries and temporal stability through the use of color, motion, and histograms.

The field of computer vision has undergone rapid development. Starting from recognition tasks such as classification, detection, and segmentation, the focus of visual analysis has gradually shifted towards learning spatiotemporal information. This thesis presents research on spatiotemporal learning, with a particular emphasis on motion estimation and visual recognition. First, we address the problem of video object tracking. Previous methods have primarily re-lied on learning improved appearance representations, while the spatiotemporal relationships of individual objects have been underexplored. We propose leveraging optical flow features to achieve higher generalization in semi-supervised video object segmentation, directly incorporating these features into both the target representation and the decoder network. Our experiments and analysis show that enriching feature representations with spatiotemporal information improves segmentation quality and generalization capability. Next, we investigate spatiotemporal learning in 3D for motion estimation, specifically scene flow estimation. Scene flow estimation as an important research topic in 3D computer vision is crucial for applications such as robotics, autonomous driving, embodied navigation, and tracking. We investigate the problem in different perspectives: 1. What is the best formulation for solving the problem and how to learn a better spatiotemporal feature representation? 2. Can we introduce uncertainty estimation to the task, which is of crucial importance for safety-critical downstream tasks? 3. How to scale the estimation to large-scale data, e.g., autonomous scenes, and leverage the temporal information without introducing much computation overheads? To answer these questions, we explore the use of transformers for improved feature representation, diffusion models for uncertainty estimation, and efficient feature learning methods for multi-frame, large-scale autonomous driving scenarios. Finally, we extend our research to joint visual segmentation, tracking, and open-vocabulary recognition in LiDAR sequences, particularly for autonomous scenes. In such environments, precise segmentation, tracking, and recognition of objects are essential for downstream analysis and control. Current human-annotated open-source datasets allow for reasonable tracking of traffic participants such as cars and pedestrians. However, we aim to advance beyond this towards segmenting and tracking any object in LiDAR data. To this end, we propose a pseudo-labeling engine that leverages the 2D visual foundation model SAM v2 and the vision-language model CLIP to automatically label LiDAR streams. We further introduce the SAL-4D model, capable of segmenting, tracking, and recognizing any object in a zero-shot manner. In summary, we explore the learning of spatiotemporal information in both 2D image and 3D point cloud domains. In the image domain, we demonstrate that spatiotemporal information improves video object segmentation quality and generalization. In the 3D point cloud domain, we show that spatiotemporal learning enables more accurate motion estimation and facilitates the first method for zero-shot segmentation, tracking, and open-vocabulary recognition of arbitrary objects.