Video segmentation based on sequential change detection

In content-based video analysis, shot boundary detection is a common first step to segment video data into fundamental units of shots, each composing consecutive frames filmed with a single camera act. Many methods have been proposed in the literature for detection of shot boundaries. In this paper, we propose a new and effective post-refinement method on the detected shot boundaries by performing sequential detection of abrupt change in two underlying distributions. Experimental results show that the proposed method can eliminate most false detections and also recover many missed detections from the original detected shot boundaries, attaining better detection performance.

We present a new non-parametric statistic, called the weighed <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> divergence, based on empirical distributions for sequential change detection. We start by constructing the weighed <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> divergence as a fundamental building block for two-sample tests and change detection. The proposed statistic is proved to attain the optimal sample complexity in the offline setting. We then study the sequential change detection using the weighed <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> divergence and characterize the fundamental performance metrics, including the average run length (ARL) and the expected detection delay (EDD). We also present practical algorithms to find the optimal projection to handle high-dimensional data and the optimal weights, which is critical to quick detection since, in such settings, there are not many post-change samples. Simulation results and real data examples are provided to validate the good performance of the proposed method.

We propose a method of shot boundary detection based on the co-occurrence of global motion in video stream. In addition to the conventional features based on appearance and local motion, we apply ST (Space-Time) patch analysis for detecting global motion in video stream. And then we perform shot boundary detection by constructing AdaBoost classifiers which represent the co-occurrence of global motion and the conventional features. Experimental results show that our method had 3.8% higher F-measure value than that of the conventional method for gradual shot boundary detection.

Shot boundary detection is essentially to detect the position of frames where the shot changes. It has been actively studied in video analysis and management for convenience, which becomes a key technique with the rapid proliferation of rich and diverse videos. With respect to the complex characteristics of different shots in varying length and content variation property, in this paper we present a two stage method for shot boundary detection (TSSBD) which distinguishes abrupt shot by fusing color histogram and deep features, and locate gradual shot changes with C3D-based deep analysis. Abrupt shot changes are detected firstly as it occurs between two frames, which divides the complete video into segments containing gradual transitions; Over these video segments, gradual shot change detection is implemented using 3D-convolutional neural network, which classifies clips into specific gradual shot change types; Finally, an effective merging strategy is proposed to locate positions of gradual shot transitions. The experimental analysis illustrates that the proposed progressive method is capable of detecting both abrupt shot transitions and gradual shot transitions accurately.

Sequential change detection is a fundamental problem in statistics and signal processing, with the CUSUM procedure widely used to achieve minimax detection delay under a prescribed false alarm rate when pre- and post-change distributions are fully known. However, in many practical settings, raw observations cannot be shared with a trusted central curator, and privacy must be enforced at the data source, which prevents the computation of exact CUSUM statistics. We therefore introduce a local differentially private (DP) variant called LDP-CUSUM, which first applies a local DP mechanism to transform the raw data into privatized observations and then applies a CUSUM procedure to detect the change. We derive closed-form bounds on the average run length to false alarm and on the worst-case average detection delay, explicitly characterizing the tradeoff among privacy level, false alarm rate, and detection efficiency. Numerical simulations and a real-data case study were conducted to demonstrate the detection efficiency of our proposed LDP-CUSUM across various scenarios.

Nous proposons une methode « operationnelle » pour le probleme de tests d’hypotheses composites convexes lorsque les distributions sous-jacentes possedent des queues lourdes. Elle s’appuie sur la separation euclidienne des ensembles convexes et peut etre vue comme une extension de la methode developpee dans (Electron. J. Stat. 9 (2015) 1645–1712 ; Electron. J. Stat. 10 (2016) 2204–2242) pour l’etude des tests d’hypotheses reposant sur des techniques d’optimisation convexe. En particulier, nous montrons comment construire des tests quasi-optimaux pour des familles de distributions qui sont majorees, dans un sens precis, par une distribution symetrique quasi-spherique et etudions la relation entre les tests bases sur la separation euclidienne et les tests utilisant des potentiels. Nous appliquons la methodologie proposee au probleme de la detection sequentielle et decrivons sa mise en oeuvre pour la detection sequentielle de ruptures dans l’entree d’un systeme dynamique.

Many recent cases of illegal activities have been reported regarding the video and all the unlawful activities designed to destroy the originality of the video data. A slight modification can alter the meaning and intent of the video. Nowadays, many devices have gained the power of tapping or capturing digital video, and all these videos can be approved over the internet and many other unsafe channels. Therefore, the detection o of video authentication proving is a widespread and critical problem in various areas such as criminal, forensic investigations, and court cases. The preceding study proposed numerous techniques to ensure the authentication of video data and structure. This study emphasized retrieving a video structure to detect an alteration made in the video using shot boundary detection to prove authentication. The detection of shot boundary is a complicated task and is usually utilized for structured video because these techniques can identify modifications made with a fundamental shot change frequency. This study focuses on retrieving a video structure to detect an alteration made in the video by detecting shot boundary to prove authentication. The detection of shot boundary is a complicated task and is usually utilized for structured video because these techniques can identify modifications made with a regular shot change frequency. Furthermore, this study aims to develop a new model for video authentication detection using the shot boundary detection method, which is considered the practical tool to track down the originality based on their fundamental structure.

Detecting change points sequentially in a streaming setting, especially when both the mean and the variance of the signal can change, is often a challenging task. A key difficulty in this context often involves setting an appropriate detection threshold, which for many standard change statistics may need to be tuned depending on the prechange and postchange distributions. This presents a challenge in a sequential change detection setting when a signal switches between multiple distributions. Unfortunately, change point detection schemes that use the log-likelihood ratio, such as cumulative sum (CUSUM) and the generalized log-likelihood ratio (GLR), are quick to react to changes but are not symmetric when both the mean and the variance of the signal change. This makes it difficult to set a single threshold to detect multiple change points sequentially in a streaming setting. We propose a modified version of CUSUM that we call data-adaptive symmetric CUSUM (DAS-CUSUM). The DAS-CUSUM procedure is symmetric for changes between distributions, making it suitable to set a single threshold to detect multiple change points sequentially in a streaming setting. We provide results that relate the expected detection delay and average run length for our proposed procedure when both prechange and postchange distributions are normally distributed. Experiments on simulated and real-world data show the utility of DAS-CUSUM.

Video shot boundary detection (SBD) is the first step of video analysis, summarization, indexing, and retrieval. In SBD process, videos are segmented into basic units called shots. In this paper, a new SBD method is proposed using color, edge, texture, and motion strength as vector of features (feature vector). Features are extracted by projecting the frames on selected basis vectors of Walsh-Hadamard transform (WHT) kernel and WHT matrix. After extracting the features, based on the significance of the features, weights are calculated. The weighted features are combined to form a single continuity signal, used as input for Procedure Based shot transition Identification process (PBI). Using the procedure, shot transitions are classified into abrupt and gradual transitions. Experimental results are examined using large-scale test sets provided by the TRECVID 2007, which has evaluated hard cut and gradual transition detection. To evaluate the robustness of the proposed method, the system evaluation is performed. The proposed method yields F1-Score of 97.4% for cut, 78% for gradual, and 96.1% for overall transitions. We have also evaluated the proposed feature vector with support vector machine classifier. The results show that WHT-based features can perform well than the other existing methods. In addition to this, few more video sequences are taken from the Openvideo project and the performance of the proposed method is compared with the recent existing SBD method.

Sequential change detection in high-dimensional dynamic networks has attracted growing attention in modern applications. A major challenge is that as network scale increases, limited sensing resources make it difficult to fully observe links at each time point, resulting in partial observability and complicating detection. To tackle this, we propose a Latent Bi-Transit Network (LBTN) that learns unobserved edge formation, latent node states, and the evolution mechanisms of dynamic networks. Based on LBTN, we design a variational Bayesian method to infer sparse node-level changes and employ a likelihood ratio test as the detection statistic. By further formulating the statistic as the reward function of a combinatorial multi-armed bandit (CMAB) problem, we develop a Thompson sampling strategy to adaptively select edges for observation, balancing exploration and exploitation. Comprehensive simulations and real-world experiments show that our method consistently outperforms existing baselines and its variants across diverse scenarios.

The problem of sequential detection of spontaneous changes in equations for unobserved state variables and observations of linear and nonlinear multivariate state-space models is considered. We consider the case of new additive terms and the case of changing coefficients of these equations. The proposed nonparametric method uses the idea of the “moving window” of observations. In Theorem 4.1 the exponential rate of convergence to zero for type 1 error probability (a wrong decision about a change) is proved. In Theorem 4.2 we consider type 2 error probability and the normalized (by the volume N of the moving window) delay time in change-point detection. Asymptotic optimality of the proposed method is proved in Theorem 5.1. Experimental study includes Monte Carlo examples of sequentially detected changes in coefficients of linear and nonlinear state-space models.

Recent advances in computing, communication, and sensor technology are pushing the development of many new applications. This trend is especially evident in pervasive computing. The content-based video analysis is one example of this innovation. Shot boundary detection servers as an important step to structure the contents of videos, many research efforts have been devoted to this field. A novel approach is proposed to detect cuts and gradual shots. An effective feature selection mechanism is introduced using shot boundary coefficient (SBC) combined with HSV divisional histogram. SBC is derived from a model called average sliding-window difference. It is a novel way to enlarge frame difference. SBC can remove the factors affected by camera/object motion, illumination and other artifacts on the curve of frame-wise histogram difference. Each area of the frame has different weight according to its respective importance. In order to improve the speed, at the beginning of the program, we adopt n rank Newton interpolation formula to deflate the frame size. We use Self-Organizing Map (SOM) Network to detect the abrupt shot. It can avoid the threshold completely. Gauss Model is selected to detect gradual shots. Based on the set of abrupt shots, we qualify the candidate set of the gradual shot furthermore. The experimental results have shown that a good compromise between speed and accuracy is achieved. It also has good noise tolerance characteristics.

Video summarization aims at providing compact representation containing enough information for users to understand the entire content or important events, which serves as the fundamental process in content-based video analysis. This paper presents a novel sport video summarization algorithm by mining consistent field-of-views applied visual and temporal information in a totally unsurprised manner. After videos are broken into shots, a content-based similarity measure is proposed in the shot level to structurally analysis the visually matching cost of original videos. Then modified agglomerative hierarchical clustering is performed with an energy-based function to match the statistical distribution of various views in game videos and a refined distance metric is proposed as similarity measure of two shots. Extended temporal prior is introduced to meet the fact that temporally neighbored shots with similar duration are more likely to be in the same clusters. Experiments on a database of 6 sport genres with over 10251 minutes of videos from different sources achieved an average accuracy of 91.5% and quantitative results are presented to justify each choice made in the design of our algorithm. Our proposed algorithm is applied for the non-linear browsing service of Orangesports by France Telecomm and an android based app has been implemented for smart mobile devices.

Video shot partition is the first step towards content-based video analysis. The objective of this paper is to present a novel solution to this problem in detail. This solution is provided in the form of a combination of both spatial and temporal information that are interrelated to each other in video sequences. We model the required measure functions using a robust metric for visual and temporal content similarity and employ a seeded region growing based method to the visual and temporal features of video sequences. The experimental results are demonstrated regarding various transitions and the improvements are shown.

Sequential Detection of Abrupt Changes in ARMA Models