A semantic-based video scene segmentation using a deep neural network

Currently, video scene segmentation is an important part of realizing content-based video retrieval (CBVR). Aiming at the problem that low efficiency of video scene segmentation in CBVR, this paper proposed a multi-modal video scene segmentation optimization algorithm based on feature extraction of convolutional neural network (CNN). According to the large amount of information contained in the multi-modal data of video, the VGG19 network has been improved in a targeted manner and the underlying features and semantic features of various modes are extracted from each video shots. By forming these features into vectors and using the method such as triplet loss learning and shot similarity calculation, scene segmentation task is converted to a binary classification problem for shot boundary. Then the scoring mechanism is established to optimize the results, finally the scene segmentation task is completed. Experimental results show that the algorithm can be effective in video scene segmentation, and the overall recall and precision can reach 85.77% and 87.01%, respectively. Compared with the shot similarity graph method, two indicators have increased by 10% and 9% respectively. Compared with the DeepSSS method that also uses the deep learning network model, the comprehensive metric F-messure has increased by 8%.

In this paper we propose temporal and optimized video scene segmentation techniques. Temporal video scene segmentation methods aims to partition the video to elementary image sequences termed scenes. The main purpose of video scene segmentation is to extract objects from series of consecutive video frames. One important goal in video analysis is to group the shots, such that all shots in a single scene are related to the same subject. In this paper we make an analysis on four methods of video scene segmentation. The methods proposed for segmentation is Backward shot coherence(BSC), Best first model merging(BFMM) and Markov chain Monte Carlo (MCMC) and Hierarchical chain partitioning(HCP). Video scene segmentation is performed by considering both content coherence and temporally contextual dissimilarity. This task is formulated as a ratio function of content coherence within a scene and contextual dissimilarity of adjacent scenes. Finally we propose an optimized video scene segmentation which is a novel approach that is used to segment the video scene and thereby optimize them to produce efficiency. An effective and efficient hierarchical chain partitioning(HCP) approach is used to find the optimal scene segmentation. HCP gets competitive performance at low tolerance and better performance at high tolerance The HCP method approaches 100 % performance at tolerance = 1. The output segments can be used for further analysis and processing of the videos that is for indexing, storage, searching and retrieval of particular portion of video. It can be used in several applications in the fields of Image processing, video content analysis, Computer vision, Movies, Medical, Sports, and News. We have tested the proposed methods in Feature Films, News videos, TV Shows and high degree of accuracy has been obtained.

With the development of theories and technologies in medical imaging, most of the tumors can be detected in the early stage. However, the nature of ovarian cysts lacks accurate judgement, leading to that many patients with benign nodules still need Fine Needle Aspiration (FNA) biopsies or surgeries, increasing the physical pain and mental pressure of patients as well as unnecessary medical health care costs. Therefore, we present an image diagnosis system for classifying the ovarian cysts in color ultrasound images, which novelly applies the image features fused by both high-level features from deep learning network and low-level features from texture descriptor. Firstly, the ultrasound images are enhanced to improve the quality of training data set and the rotation invariant uniform local binary pattern (ULBP) features are extracted from each of the images as the low-level texture features. Then the high-level deep features extracted by the fine-tuned GoogLeNet neural network and the low-level ULBP features are normalized and cascaded as one fusion feature that can represent both the semantic context and the texture patterns distributed in the image. Finally, the fusion features are input to the Cost-sensitive Random Forest classifier to classify the images into "malignant" and "benign". The high-level features extracted by the deep neural network from the medical ultrasound image can reflect the visual features of the lesion region, while the low-level texture features can describe the edges, direction and distribution of intensities. Experimental results indicate that the combination of the two types of features can describe the differences between the lesion regions and other regions, and the differences between lesions regions of malignant and benign ovarian cysts.

Video scene segmentation is a process for dividing video into semantically meaningful blocks. This can help e.g. search engines to divide video into better manageable parts and enable more relevant search in video. Unfortunately, scene segmentation is based on the semantic and therefore it is a difficult task for computers. This work is preliminary study involved into supervised video scene segmentation, which is driven by the way how human segments scenes in a movie. Since these video segments represent semantic parts in video, it can be used for better video annotation and also for searching in videos. As a training set, only high quality movies were used and from these movies 100 training samples have been extracted and used for evaluation. Resulting model is a method based on general color layout, Tamura similarity measure and k-nearest neighbors achieving 97.00% accuracy.

ABSTRACTThe proposed content-based video retrieval method in this study uses feature extraction, shot boundary detection, shot noise filter, shot cluster, the key feature set of shot extraction and shot classification. Shot boundary detection is conducted using a weight selection genetic algorithm. In order to verify the video retrieval method in this paper, three video databases were used in the experiment. These video databases are used to verify the shot boundary detection, the shot classification and video retrieval of this study. A series of comparisons and analyses were then conducted using the above video databases. The experiment results showed that the shot boundary detection method exhibits better performance than the other compared methods. Moreover, the video retrieval experiment results showed that the proposed method is able to precisely catch a shot in the search video, and do so with reduced video retrieval time.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

In this work, we present a system for video shot boundary detection and key-frame extraction from video sequences based on colour histograms. The growth of archived video material has made automated indexing and browsing tools necessary. The aim of this paper is to provide a framework for such a video indexing and browsing tool. The proposed system segments video sequences based on colour similarity, and then extracts key-frames by clustering the images. Computations of colour histogram differences and self-similarity modelling have been performed in segmentation process. We have extracted three-, four- and five-frames video summaries with unsupervised k-means clustering. We have also dynamically determined the number of key-frames representing the video content without any prior information based on dominant set clustering. Our results show that our approach is successful in both segmentation and key-frame extraction processes.

Content-based video retrieval system aims at assisting a user to retrieve targeted video sequence in a large database. Most of the search engines use textual annotations to retrieve videos. These types of engines offer a low-level abstraction while the user seeks high-level semantics. Bridging this type of semantic gap in video retrieval remains an important challenge. In this paper, colour, texture and shapes are considered to be low-level features and motion is a high-level feature. Colour histograms convert the RGB colour space into YcbCr and extract hue and saturation values from frames. After colour extraction, filter mask is applied and gradient value is computed. Gradient and threshold values are compared to draw the edge map. Edges are smoothed for sharpening to remove the unnecessary connected components. These diverse shapes are then extracted and stored in shape feature vectors. Finally, an SVM classifier is used for classification of low-level features. For high-level features, depth images are extracted for motion feature identification and classification is done via echo state neural networks (ESN). ESN are a supervised learning technique and follow the principle of recurrent neural networks. ESN are well known for time series classification and also proved their effective performance in gesture detection. By combining the existing algorithms, a high-performance multimedia event detection system is constructed. The effectiveness and efficiency of proposed event detection mechanism is validated using MSR 3D action pair dataset. Experimental results show that the detection accuracy of proposed combination is better than those of other algorithms

Image retrieval is one of the most interesting and fastest growing research areas in all fields. It is an effective and efficient tool for managing large image databases. In most Content-Based Image Retrieval (CBIR) systems, an image is represented by a set of low-level visual features; hence a direct correlation with high-level semantic information will be absent. Therefore, a gap exists between high-level information and low-level features, which is the main reason that hinders the improvement of the image retrieval accuracy. In this work, main focus is on the semantic based image retrieval system using Gray Level Co-occurrence Matrix (GLCM) for texture feature extraction. Based on the texture features, semantic interpretations are given to the extracted textures. The images are retrieved according to user satisfaction and thereby reduce the semantic gap between low level features and high level features.

Exploiting high level feature for dynamic textures recognition

Current deep learning based image enhancement algorithms attempt to learn the mapping relationship between degraded images and clear images directly. These algorithms often ignore the fidelity constraint of the observational model. In order to improve the image enhancement performance, an improved deep residual neural network based image enhancement algorithm (DRNN-IE) for low dose CT images is proposed in this paper. DRNN-IE embeds the image enhancement task into a deep neural network, and achieves data consistency using multiple enhancement modules and back-projection modules. The enhancement modules in DRNN-IE produce new features through fusing low-level and high-level features. In order to improve the algorithm’s generalization ability, a dual-parameter loss function is adopted to train and optimize the neural network. Experiments on real CT images show that the proposed algorithm has excellent enhancement performance and retains detailed information of low-dose CT images.

The growing popularity of robotic minimally invasive surgeries has made deep learning–based surgical training a key area of research. A thorough understanding of the surgical scene components is crucial, which semantic segmentation models can help achieve. However, most existing work focuses on surgical tools and overlooks anatomical objects. Additionally, current state-of-the-art (SOTA) models struggle to balance capturing high-level contextual features and low-level edge features. We propose a Feature-Adaptive Spatial Localization model (FASL-Seg), designed to capture features at multiple levels of detail through two distinct processing streams, namely a Low-Level Feature Projection (LLFP) and a High-Level Feature Projection (HLFP) stream, for varying feature resolutions - enabling precise segmentation of anatomy and surgical instruments. We evaluated FASL-Seg on surgical segmentation benchmark datasets EndoVis18 and EndoVis17 on three use cases. The FASL-Seg model achieves a mean Intersection over Union (mIoU) of 72.71% on parts and anatomy segmentation in EndoVis18, improving on SOTA by 5%. It further achieves a mIoU of 85.61% and 72.78% in EndoVis18 and EndoVis17 tool type segmentation, respectively, outperforming SOTA overall performance, with comparable per-class SOTA results in both datasets and consistent performance in various classes for anatomy and instruments, demonstrating the effectiveness of distinct processing streams for varying feature resolutions.

In order to search channel of Higgs bosons in high energy physics (HEP), in this paper we propose Deep Extreme Feature Extraction (DEFE), a new ensemble multi-variate Analysis method (MVA). DEFE is a deep ensemble learning scheme that trains a strongly diverse set of neural feature learners without explicitly encouraging diversity and penalizing correlations, which is achieved by adopting an implicit neural controller that directly controls and distributes gradient flows from higher level deep prediction network. Based on the construction and approximation of the extreme selection region presented in this paper, the DEFE model is able to be trained efficiently, and extract discriminative features from multiple angles and dimensions, hence the improvement of the selection region of searching new particles in HEP can be achieved. With the application in this model, the selection regions full of signal process can be obtained through the training of a miniature collision events set. In comparison of the classic deep neural network (DNN), DEFE shows a state-of-the-art performance: the error rate has decreased by about 37%, the accuracy has broken through 90% for the first time along with the discovery significance which has reached a standard deviation of 6.0. Experiment shows that DEFE is able to train an ensemble of discriminative feature learners that boosts the over-performance of final prediction. Furthermore, among high-level features, there are still some important patterns that are unidentified by DNN and are independent from low-level features, while DEFE can identify these significant patterns more effectively and efficiently.

With the development of multimedia data types and available bandwidth there is huge demand of video retrieval systems as users shift from text based retrieval systems to content based retrieval systems (CBVRS). Selection of extracted features play an important role in content based video retrieval, regardless of video attributes under consideration. Good features selection also allows the time, space and cost saving in the retrieval process. This leads to the interest of selecting only those features that can be extracted from video data and used for indexing and retrieval along with similarity measurement methods. There are certain issues like video store, search, index and retrieve as a trend in area of content based video retrieval systems. In this paper, we have proposed a method. In this paper fractional discrete cosine transform coefficients are used to extract key frames which eventually save space and time with better completeness in the retrieval process.

Point cloud semantic segmentation in urban scenes plays a vital role in intelligent city modeling, autonomous driving, and urban planning. Point cloud semantic segmentation based on deep learning methods has achieved significant improvement. However, it is also challenging for accurate semantic segmentation in large scenes due to complex elements, variety of scene classes, occlusions, and noise. Besides, most methods need to split the original point cloud into multiple blocks before processing and cannot directly deal with the point clouds on a large scale. We propose a novel context-aware network (CAN) that can directly deal with large-scale point clouds. In the proposed network, a Local Feature Aggregation Module (LFAM) is designed to preserve rich geometric details in the raw point cloud and reduce the information loss during feature extraction. Then, in combination with a Global Context Aggregation Module (GCAM), capture long-range dependencies to enhance the network feature representation and suppress the noise. Finally, a Context-Aware Upsampling Module (CAUM) is embedded into the proposed network to capture the global perception from a broad perspective. The ensemble of low-level and high-level features facilitates the effectiveness and efficiency of 3D point cloud feature refinement. Comprehensive experiments were carried out on three large-scale point cloud datasets in both outdoor and indoor environments to evaluate the performance of the proposed network. The results show that the proposed method outperformed the state-of-the-art representative semantic segmentation networks, and the overall accuracy (OA) of Tongji-3D, Semantic3D, and S3DIS is 96.01%, 95.0%, and 88.55%, respectively.