Video Data Management Using a Seeded Region Growing Technique

In the context of automotive driving assistance, this paper describes a generic (i.e. applicable to both vehicle interior and exterior scenes) vision based approach for scene content analysis. It makes use of temporal and spatial information from a stereoscopic sequence of images to localize objects and estimate their position and motion. The proposed method is divided into three steps. First, image features are selected, tracked and reconstructed in the 3D world space. Second, a clustering step is processed in the 5D space made of the positions and 2D motions parameters. The last step is devoted to clusters interpretation: it is out of the scope of the paper, however orientations are given to illustrate the capabilities of the proposed approach. The paper is organized as follows: first, the use of temporal and spatial information from a stereoscopic sequence is investigated. A state of the art of existing methods is presented. Then, a generic approach for object segmentation is proposed. Lastly, experimental results are presented.

Video summarization aims at providing compact representation containing enough information for users to understand the entire content or important events, which serves as the fundamental process in content-based video analysis. This paper presents a novel sport video summarization algorithm by mining consistent field-of-views applied visual and temporal information in a totally unsurprised manner. After videos are broken into shots, a content-based similarity measure is proposed in the shot level to structurally analysis the visually matching cost of original videos. Then modified agglomerative hierarchical clustering is performed with an energy-based function to match the statistical distribution of various views in game videos and a refined distance metric is proposed as similarity measure of two shots. Extended temporal prior is introduced to meet the fact that temporally neighbored shots with similar duration are more likely to be in the same clusters. Experiments on a database of 6 sport genres with over 10251 minutes of videos from different sources achieved an average accuracy of 91.5% and quantitative results are presented to justify each choice made in the design of our algorithm. Our proposed algorithm is applied for the non-linear browsing service of Orangesports by France Telecomm and an android based app has been implemented for smart mobile devices.

Video-based facial expression recognition is a long-standing problem owing to a gap between visual features and emotions, difficulties in tracking the subtle movement of muscles and limited datasets. The key to solving this problem is to exploit effective features characterizing facial expression to perform facial expression recognition. We propose an effective framework to solve these problems. In our work, both spatial information and temporal information are utilized through the aggregation layer of a framework that fuses two state-of-the-art stream networks. We investigate different strategies for pooling across spatial information and temporal information. We find that it is effective to pool jointly across spatial information and temporal information for video-based facial expression recognition. Our framework is end-to-end trainable for whole-video recognition. In addressing the problem of facial recognition, the main contribution of this project is the design of a novel, trainable deep neural network framework that fuses spatial information and temporal information of video according to CNNs and LSTMs for pattern recognition. The experimental results on two public datasets, i.e., the RML and eNTERFACE05 databases, show that our framework outperforms previous state-of-the-art frameworks.

There is context-related temporal and spatial information in video. How to make better use of these information is the key to improving the accuracy and speed of video object detection. The method of processing spatial and temporal information through 3D convolution is better than that of processing spatial information first and then processing time dimension information. There are two reasons for this assertion. One is that the 3D convolution can process spatial information and temporal information at the same time which will not divide them into two parts in feature extraction. The other is that the human visual derivation mechanism needs to memorize multi frames and process a video as a whole. After the training test on NFS dataset, it is verified that 3D convolution has stronger ability to extract temporal and spatial information. In addition, this paper also proposes a downward sampling method by connecting the low-resolution layer and the high-resolution layer, which can effectively improve the FPN model and enrich its high-level feature representation. After training and testing with different number of downward sample modules on MOT16 dataset, the improvement of model performance by down-sampling module is verified. Finally, this paper combines the downward sample module with the 3D network, and obtains the best test results on the NFS dataset.

We propose in this paper some new methods for analyzing and modeling sports based on their low-level visual are first automatically identified by using the color features derived from its video are first automatically identified by using the color features derived from its video shots. To improve the performance on the identification of dominant scenes and reduce the dependency on a proper threshold, a comparison on different forms of shot color features, including shot color histograms, their principal components, and subspace linear discriminant representations, is performed. Second, the content compactness and motion attributes of clustered video shots are analyzed to differentiate dominant scene types. A scene transition diagram is then constructed to form a structural descriptor for sports video contents. Third, the video shots belonging to each dominant scene are processed, using customized schemes and domain specific knowledge, to identify interesting play events in the sports video. Experimental results on identification of dominant scenes, structural desprictors and high-level ball videos, are presented to demonstrate the possible applications of the proposed methods.

Visual information is taking a predominant place in our society. With the advent of digital technologies, visual information will find new applications in domains ranging from communication, commerce to entertainment. New functionalities will be required that permit an extended interaction with the visual information. In particular, this is the case for digital television and the related problem of video sequence compression. The extent of possible interactions depends on the manner in which the visual information is represented. Up to now, the canonical representation is used. Also referred to as the waveform representation, it is a technical artifact of the image capture procedure. This representation severely restricts the functionalities available to the end-user. The latter is unable to freely manipulate or customize the received visual information. In this dissertation, we propose to describe the visual information through a semantically meaningful representation. This representation directly derives from the scene content, which is decomposed in terms of its constituting objects. Consequently, the viewing process is totally disconnected from the image capture procedure. This permits full interactivity with the visual information, leading to enhanced functionalities for the end-user. The essence of this dissertation is to automatically define the objects forming the scene arid to automatically track them through the video sequence. Two main issues are identified: the initial segmentation of the objects and their tracking. The first issue deals with segmenting the scene into its constituent objects. This has to be performed on the basis of the information available in two consecutive frames. In order to solve the problem, a split-and-merge approach is used. First, the image is segmented into small, spatio-temporally homogeneous regions. This is achieved through a top-down approach where spatial, temporal and change information is combined. These regions are used as a starting point to define the objects forming the scene. A bottom-up approach is used which iteratively merges the regions. The propensity of the regions to form an object is assessed in terms of both spatial and temporal information. The second issue deals with segmenting and tracking the objects in the successive frames. The coherence between the successive segmentations is ensured by using past and current information. Also, the different objects composing the scene are identified throughout the video sequence. This identification relies on temporal, spatial and spatio-temporal features of the objects. The proposed representation of the visual information finds a natural application in video coding. A second generation video coding scheme is presented which combines compression efficiency with extended functionalities. In particular, scalable coding is achieved in terms of both scene content and object coding quality.

The analysis of a multitemporal sequence of images of a given site makes it possible to exploit temporal information in addition to spectral and spatial information, and therefore represents a way to improve the accuracy with respect to the non-contextual single-time classification. The proposed contextual multitemporal classification scheme consists of two stages of multilayer perceptron (MLP) neural networks for each single-time image of the multitemporal sequence. The first stage is a one-hidden layer MLP whose ro/spl circ/le is to estimate, for each pixel, the single-time posterior probability of each class, given the feature vector. These probability estimates represent spectral information; in addition, they are utilized to generate a non-contextual classification map. The neighboring class labels of a given pixel in the non-contextual classification map are exploited to extract spatial information, while temporal information is deduced from the non-contextual maps produced by the remaining single-time images in the multitemporal sequence. Spatial and temporal contextual information together with spectral information serve as inputs for the second stage network of the classification scheme where the fusion takes place. As the network configuration can influence the classification performances, three MLP-based configurations are investigated. Experimental results on a multitemporal data set consisting of two multisensor (Landsat TM and ERS-1 SAR) images are presented. The performances of the proposed methods are discussed and compared with those obtained by a reference classifier based on the Markov random fields fusion approach in terms of classification accuracy. The results show that the proposed fusion approach based on neural networks may represent an interesting solution to the problem of multitemporal contextual fusion.

Recent advances in computing, communication, and sensor technology are pushing the development of many new applications. This trend is especially evident in pervasive computing. The content-based video analysis is one example of this innovation. Shot boundary detection servers as an important step to structure the contents of videos, many research efforts have been devoted to this field. A novel approach is proposed to detect cuts and gradual shots. An effective feature selection mechanism is introduced using shot boundary coefficient (SBC) combined with HSV divisional histogram. SBC is derived from a model called average sliding-window difference. It is a novel way to enlarge frame difference. SBC can remove the factors affected by camera/object motion, illumination and other artifacts on the curve of frame-wise histogram difference. Each area of the frame has different weight according to its respective importance. In order to improve the speed, at the beginning of the program, we adopt n rank Newton interpolation formula to deflate the frame size. We use Self-Organizing Map (SOM) Network to detect the abrupt shot. It can avoid the threshold completely. Gauss Model is selected to detect gradual shots. Based on the set of abrupt shots, we qualify the candidate set of the gradual shot furthermore. The experimental results have shown that a good compromise between speed and accuracy is achieved. It also has good noise tolerance characteristics.

Modern industrial plants generally consist of multiple manufacturing units, and the local correlation within each unit can be used to effectively alleviate the effect of spurious correlation and meticulously reflect the operation status of the process system. Therefore, the local correlation, which is called spatial information here, should also be taken into consideration when developing the monitoring model. In this study, a cascaded monitoring network (MoniNet) method is proposed to develop the monitoring model with concurrent analytics of temporal and spatial information. By implementing convolutional operation to each variable, the temporal information that reveals dynamic correlation of process data and spatial information that reflects local characteristics within individual operation unit can be extracted simultaneously. For each convolutional feature, a submodel is developed and then all the submodels are integrated to generate a final monitoring model. Based on the developed model, the operation status of the newly collected sample can be identified by comparing the calculated statistics with their corresponding control limits. Similar to the convolutional neural network (CNN), the MoniNet can also expand its receptive field and capture deeper information by adding more convolutional layers. Besides, the filter selection and submodel development in MoniNet can be replaced to generalize the proposed network to many existing monitoring strategies. The performance of the proposed method is validated using two real industrial processes. The illustration results show that the proposed method can effectively detect process anomalies by concurrent analytics of temporal and spatial information.

Home Network-Attached Storage (NAS) provides an easy way for data sharing and backup among multiple consumer electronic devices in home networks. Because of large capacity and cost effectiveness, disks are widely adopted in home NAS devices. In addition, data writes are common for many home NAS devices since these devices are usually used for data storage and backup. A writeback policy selects which dirty buffers are to be flushed to the disk, which is critical to the system performance under write-intensive workloads. In this paper, an intelligent writeback policy for home NAS called TESA is proposed. In contrast to most existing writeback policies, TESA considers both temporal and spatial information of the dirty buffers to improve the writeback performance. Considering the temporal information helps to reduce the frequency of writing back recently used dirty buffers, and hence leads to reduced write traffic. Considering the spatial information causes a reduction in the overall seek time and rotation delay of the dirty buffer writeback. The TESA writeback policy was implemented on a NAS evaluation board running Linux kernel. The performance results shows that TESA yields a significant performance improvement (i.e., up to 33.1%) over the original writeback policy of Linux. Moreover, TESA outperforms the policy that considers only the temporal information of the dirty buffers by up to 21.0%, and it also has up to 10.9% performance improvement over a previous policy that considers both the temporal and spatial information.

To locate a video clip in large collections is very important for retrieval applications, especially for digital rights management. In this paper, we present a novel technique for automatic identification of digital video. This new algorithm is based on dynamic programming that fully uses the temporal dimension to measure the similarity between two video sequences. A normalized chromaticity histogram is used as a feature which is illumination-invariant. Dynamic programming is applied on shot-level to find the optimal nonlinear mapping between video sequences. Two new normalized distance measures are presented for video sequence matching. One measure is based on the normalization of the optimal path found by dynamic programming. The other measure combines both the visual features and the temporal information. Experimental results show that the shot-level approach is robust to frame rate conversion, color correction, and compressions. The proposed distance measures are suitable for variable-length comparisons.

Most of the existing hyperspectral image change detection methods based on deep learning utilize convolutional neural networks (CNN), recurrent neural networks (RNN), and long-short-term memory (LSTM). However, CNNs primarily focus on extracting spatial information from hyperspectral images, thereby failing to fully utilize channel and temporal information. In this paper, we propose a network called STSTNet that can jointly extract spatial, spectral, and temporal feature information. To achieve a richer feature representation space, we employ a spatial attention module to extract spatial feature information from feature maps, and a spectral transformer encoder structure to extract abundant spectral sequence information. Specifically, we design two temporal information extraction structures after each spatial information extraction structure or spectral information extraction structure to capture correlations and interactions within the hyperspectral images. Extensive experiments have been conducted on three hyperspectral datasets, and the experimental results demonstrate the network’s excellent performance.

Episodic memory involves the processing of spatial and temporal aspects of personal experiences. The lateral entorhinal cortex (LEC) plays an essential role in subserving memory. However, the mechanisms by which LEC integrates spatial and temporal information remain elusive. Here, we recorded LEC neurons while male rats performed one-dimensional tasks. Many LEC cells displayed spatial firing fields and demonstrated selectivity for traveling directions. Furthermore, some LEC neurons changed the firing rates of their spatial rate maps during a session (rate remapping). Importantly, this temporal modulation was consistent across sessions, even when the spatial environment was altered. Notably, the strength of temporal modulation was greater in LEC compared to other brain regions, such as the medial entorhinal cortex, CA1, and CA3. Thus, we demonstrate spatial rate mapping in LEC neurons, which may serve as a coding mechanism for temporal context, and allow for flexible multiplexing of spatial and temporal information.

MPEG Dynamic Adaptive Streaming over Hypertext Transfer Protocol (MPEG DASH) is adaptive streaming standard developed to unify previously developed adaptive streaming technologies. Over the years various MPEG DASH based adaptive streaming algorithms have been developed, mainly hybrid algorithms that take into account the buffer occupancy, available bandwidth and other parameters like video segment size. Adaptive streaming algorithms are used to determine the next video representation in order to ensure the best possible quality of experience in varying network conditions. They must balance well between the initial video delay and the number and depth of representation switching, as well as avoid video stalling events. This paper compares three different algorithms: Rate Adaptation Algorithm (LIU), Segment-Aware Rate Adaptation (SARA) and Content-Aware Rate Adaptation (CARA) in five network test cases with three video sequences. Obtained results show that on average LIU has the most representation switching events regardless of the spatial (SI) and temporal information (TI) of video sequence and it has the lowest achieved bitrate values. Although CARA and SARA achieve similar results, due to the fact that SARA has four phases of deciding on the representation of the next segment, it has the least number of switching events and a larger value of switching depth. CARA achieves highest average values of achieved bitrate in cases when video sequences Elephants Dream and Big Buck Bunny with lower SI and TI are used.

In content-based video analysis, shot boundary detection is a common first step to segment video data into fundamental units of shots, each composing consecutive frames filmed with a single camera act. Many methods have been proposed in the literature for detection of shot boundaries. In this paper, we propose a new and effective post-refinement method on the detected shot boundaries by performing sequential detection of abrupt change in two underlying distributions. Experimental results show that the proposed method can eliminate most false detections and also recover many missed detections from the original detected shot boundaries, attaining better detection performance.