Scene Change Detection VIA Deep Convolution Canonical Correlation Analysis Neural Network

With the continuous development of high-spatial-resolution ground observation technology, it is now becoming possible to obtain more and more high-resolution images, which provide us with the possibility to understand remote sensing images at the semantic level. Compared with traditional pixel- and object-oriented methods of change detection, scene-change detection can provide us with land use change information at the semantic level, and can thus provide reliable information for urban land use change detection, urban planning, and government management. Most of the current scene-change detection methods are based on the visual-words expression of the bag-of-visual-words model and the single-feature-based latent Dirichlet allocation model. In this article, a scene-change detection method for high-spatial-resolution imagery is proposed based on a multi-feature-fusion latent Dirich- let allocation model. This method combines the spectral, textural, and spatial features of the high-spatial-resolution images, and the final scene expression is realized through the topic features extracted from the more abstract latent Dirichlet allocation model. Post-classification comparison is then used to detect changes in the scene images at different times. A series of experiments demonstrates that, compared with the traditional bag-of-words and topic models, the proposed method can obtain superior scene-change detection results.

Scene classification is a fundamental task for numeral remote sensing applications, which aims to assign semantic labels to image patches. Although deep neural networks (DNN) demonstrated unique strength in scene classification, their performances are still limited due to the lack of training samples in the remote sensing field. Recent studies show that the performance of scene classification can be improved by taking advantage of the knowledge transferred from models pre-trained on RGB images. However, the modalities differences between input images hinder the knowledge transfer across models, especially when the input of the models has distinct spectral bands. To tackle the challenges, we propose a cross-modal knowledge distillation framework to improve the performance of multispectral scene classification by transferring the prior knowledge from teacher models pre-trained on RGB images to the student network with limited samples. Moreover, a teacher assistant (TA) network is introduced to further improve the classification performance by bridging the gap between the teacher and student networks. The proposed strategy is evaluated on models with multimodality inputs with distinct spectral bands and demonstrates superior performance as compared to the state-of-the-art methods.

내용기반 검색을 위한 비디오 데이터 장면전환 검출에서 점진적인 전환을 검출하는 것은 갑작스런 전환을 검출하는 것에 비해 일반적으로 어려운 문제로 알려져 있다. 본 논문에서는 가변형 동적 임계값과 가장 최근에 검출된 컷 프레임과 현재 프레임간의 특징값 차이인 컷 프레임 차를 이용하여 갑작스런 전환과 점진적인 전환을 찾아내는 기법을 제안한다. 이를 위하여 본 논문에서는 점진적인 전환이 갖는 특성과 수학적 모델을 제시하고 컷 프레임 차를 이용하여 점진적인 전환을 검출할 수 있음을 보인다. 그리고 이를 바탕으로 갑작스런 전환과 점진적인 전환을 함께 검출할 수 있는 방법을 제시한다. 실세계 동영상 데이터를 대상으로 한 실험을 통해 제안하는 기법이 점진적인 전환 효과의 종류에 종속적이지 않으며 적은 연산 비용으로 쉽게 점진적인 전환 유무를 검출 할 수 있음을 보인다. Gradual scene change detection is known as more difficult problem then abrupt scene change detection on video data analysis for contents based retrieval. In this paper, we present a new method for scene change detection far both abrupt and gradual change using the variable dynamic threshold arid cut frame difference (CFD). For this, We present the characteristics arid mathematical models of gradual transitions anti then, how can be detected by the CFD. And also we present new scene change detection algorithm based on cut frame difference. By the experimental result using real world video data indicate that the proposed method detect various scene changes both abrupt and gradual change efficiently without time-consuming computation and any dependency on a kind of gradual change effects.

대용량 컨텐츠 수요와 공급의 증가에 따라 UHD 비디오의 분석, 색인, 편집 등을 위한 장면전환을 검출하는 방법이 요구되고 있다. 이전까지 많은 연구자들이 다양한 장면전환 검출 방법을 연구해왔지만 카메라의 다양한 움직임과 장면의 변화를 정확하게 검출하기는 어려웠다. 또한, 4K (3820x2160) 해상도 이상의 UHD 비디오들은 데이터 량을 더욱더 증가시키기 때문에 이전의 장면전환 검출 방법은 UHD 비디오 컨텐츠에 적용하기에는 너무 많은 시간이 걸리는 문제점이 발생한다. 따라서, 압축률이 높은 차세대 고효율 코덱 HEVC를 이용하여 장면전환을 검출하는 방법이 요구되고 있다. 본 논문에서는 차세대 고효율 코덱 HEVC의 부호화 부가정보를 이용한 4가지 장면전환 검출 방법을 제안하며, 대용량 비디오의 장면전환 검출을 위한 픽셀 기반의 새로운 장면전환 검출 시스템 구조를 제안한다. 연산량을 줄이기 위해 프레임 특징에 따른 방법을 각각 제시하며, 실험 결과를 통해 HEVC로 부호화 된 UHD 컨텐츠들의 장면전환 검출 가능성을 확인한다. With the increase of mass contents data, a method of a scene change detection is required for analysis, indexing and editing. Although many researchers are studying a variety of scene change detection method, it is too difficult to accurately detect various movements of the cameras and scene changes. Also, earlier scene change detection methods take too much time to apply to UHD video contents. That is because the UHD video contents with 4K (3820x2160) resolution or higher have greater amount of data. Therefore a method for detecting a scene change by using the next-generation codec, HEVC, is required. In this paper, we propose four scene change detection methods using the coding additional information of HEVC, and a new pixel-based scene change detection system. Furthermore, through the experimental results, we check the possibility of detecting the scene changes of UHD videos encoded in HEVC format.

An efficient video de-interlacing technique with scene change detection is proposed and its performances are examined. Scene changes happen quite often in film broadcasting and they tend to destabilize the quality of performance such as jagged effect, blurred effect, and artifacts effect, while de-interlacing technique is utilized. Therefore, the issue of scene change needs to be addressed with de-interlacing process. In the proposed method, de-interlacing begins with scene change detection, which is to ensure that the interfield information is used correctly. To improve the quality of de-interlacing, the factors of scene change are taken into account when de-interlacing techniques are applied. The simulation results show that the proposed algorithm exhibits better performances than other interpolation algorithms

Video is an important and challenging medium and requires sophisticated indexing schemes for efficient retrieval from visual databases. An important step in video indexing is scene change detection. Recently, several scene change detection algorithms in the pixel and compressed (MPEG-2) domains have been reported in the literature. These algorithms are computationally complex and are not very robust in detecting gradual scene changes. The authors propose an efficient technique for detecting scene changes in the MPEG-2 compressed domain. The proposed algorithm has the advantage of fast scene change detection. In addition, this algorithm has the potential to detect gradual scene changes.

An algorithm is proposed to detect abrupt and gradual scene changes in H.264 compressed videos. In H.264, the inter prediction in the encoding process generates the DC component of the prediction residual. In this paper, we use the DC component of inter prediction residuals as well as motion vectors to detect scene changes. Using the proposed algorithm, we can detect scene changes for H.264 compressed videos more effectively, as demonstrated by simulations.

Semantic-level land-use scene classification is a challenging problem, in which deep learning methods, e.g., convolutional neural networks (CNNs), have shown remarkable capacity. However, a lack of sufficient labeled images has proved a hindrance to increasing the land-use scene classification accuracy of CNNs. Aiming at this problem, this paper proposes a CNN pre-training method under the guidance of a human visual attention mechanism. Specifically, a computational visual attention model is used to automatically extract salient regions in unlabeled images. Then, sparse filters are adopted to learn features from these salient regions, with the learnt parameters used to initialize the convolutional layers of the CNN. Finally, the CNN is further fine-tuned on labeled images. Experiments are performed on the UCMerced and AID datasets, which show that when combined with a demonstrative CNN, our method can achieve 2.24% higher accuracy than a plain CNN and can obtain an overall accuracy of 92.43% when combined with AlexNet. The results indicate that the proposed method can effectively improve CNN performance using easy-to-access unlabeled images and thus will enhance the performance of land-use scene classification especially when a large-scale labeled dataset is unavailable.

In order to fulfill the requirement of limited channel bandwidth and of growing video demand like streaming media delivery on internet, and digital library, video compression is necessary. In video compression, temporal redundancy between adjacent frames is removed with block based motion estimation algorithms. Video represents a sequence of frames captured from camera. Scene is a series of consecutive frames captured from narrative point of view. In this paper we present an effective scene change detection method for an uncompressed video. We have divided frames in to blocks and applied a canny edge detector in consecutive frames. Count no of pixels (ones) in each block and compare it with consecutive frames. If scene change happens then number of pixels per block will change, based on that change we can detect scene change in consecutive frame. Here we have presented a hybrid approach, in which we have used scene change detection along with block based motion estimation algorithms (BME) to compress video.

While convolutional neural networks (CNN) have been excellent for object recognition, the greater spatial variability in scene images typically meant that the standard full-image CNN features are suboptimal for scene classification. In this paper, we investigate a framework allowing greater spatial flexibility, in which the Fisher vector (FV) encoded distribution of local CNN features, obtained from a multitude of region proposals per image, is considered instead. The CNN features are computed from an augmented pixel-wise representation comprising multiple modalities of RGB, HHA and surface normals, as extracted from RGB-D data. More significantly, we make two postulates: (1) component sparsity - that only a small variety of region proposals and their corresponding FV GMM components contribute to scene discriminability, and (2) modal non-sparsity - within these discriminative components, all modalities have important contribution. In our framework, these are implemented through regularization terms applying group lasso to GMM components and exclusive group lasso across modalities. By learning and combining regressors for both proposal-based FV features and global CNN features, we were able to achieve state-of-the-art scene classification performance on the SUNRGBD Dataset and NYU Depth Dataset V2.

Scene classification of high-resolution remote sensing images (HRRSI) is one of the most important means of land-cover classification. Deep learning techniques, especially the convolutional neural network (CNN) have been widely applied to the scene classification of HRRSI due to the advancement of graphic processing units (GPU). However, they tend to extract features from the whole images rather than discriminative regions. The visual attention mechanism can force the CNN to focus on discriminative regions, but it may suffer from the influence of intra-class diversity and repeated texture. Motivated by these problems, we propose an attention-based deep feature fusion (ADFF) framework that constitutes three parts, namely attention maps generated by Gradient-weighted Class Activation Mapping (Grad-CAM), a multiplicative fusion of deep features and the center-based cross-entropy loss function. First of all, we propose to make attention maps generated by Grad-CAM as an explicit input in order to force the network to concentrate on discriminative regions. Then, deep features derived from original images and attention maps are proposed to be fused by multiplicative fusion in order to consider both improved abilities to distinguish scenes of repeated texture and the salient regions. Finally, the center-based cross-entropy loss function that utilizes both the cross-entropy loss and center loss function is proposed to backpropagate fused features so as to reduce the effect of intra-class diversity on feature representations. The proposed ADFF architecture is tested on three benchmark datasets to show its performance in scene classification. The experiments confirm that the proposed method outperforms most competitive scene classification methods with an average overall accuracy of 94% under different training ratios.

Deep learning now plays an important role in solving complex problems in computer vision fields. The highly challenging high-resolution remote sensing image scene classification problem can also be solved using deep learning methods. The most commonly used method of deep learning is the convolutional neural network model. In this letter, based on deep learning, a combined model named Inception-long short-term memory (LSTM) is proposed. First, we combine the deep learning feature extracted from the pretrained Inception-V3 model with a hand-crafted feature: the GIST feature. The different features are then combined and input into the batch normalization (BN) layer. Second, the BN layer plays the role of the bridge to combine the InceptionV3 model with the LSTM model, which features a softmax classifier. The LSTM model is used to analyze the features and classify the different high-resolution remote sensing scene images. The proposed model, as a whole, can be uniformly trained. Three different datasets-the NWPU-RESISC45 dataset, the UC Merced dataset, and the SIRI-WHU dataset-were used to verify the effectiveness of the proposed model. The results show that the proposed Inception-LSTM model shows an outstanding performance in the scene classification task.

Acoustic scene classification is the task of identifying the scene from which the audio signal is recorded. Convolutional neural network (CNN) models are widely adopted with proven successes in acoustic scene classification. However, there is little insight on how an audio scene is perceived in CNN, as what have been demonstrated in image recognition research. In the present study, the Class Activation Mapping (CAM) is utilized to analyze how the log-magnitude Mel-scale filter-bank (log-Mel) features of different acoustic scenes are learned in a CNN classifier. It is noted that distinct high-energy time-frequency components of audio signals generally do not correspond to strong activation on CAM, while the background sound texture are well learned in CNN. In order to make the sound texture more salient, we propose to apply the Difference of Gaussian (DoG) and Sobel operator to process the log-Mel features and enhance edge information of the time-frequency image. Experimental results on the DCASE 2017 ASC challenge show that using edge enhanced log-Mel images as input feature of CNN significantly improves the performance of audio scene classification.

Remote sensing image (RSI) scene classification is the foundation and important technology of ground object detection, land use management and geographic analysis. During recent years, convolutional neural networks (CNNs) have achieved significant success and are widely applied in RSI scene classification. However, crafted images that serve as adversarial examples can potentially fool CNNs with high confidence and are hard for human eyes to interpret. For the increasing security and robust requirements of RSI scene classification, the adversarial example problem poses a serious problem for the classification results derived from systems using CNN models, which has not been fully recognized by previous research. In this study, to explore the properties of adversarial examples of RSI scene classification, we create different scenarios by testing two major attack algorithms (i.e., the fast gradient sign method (FGSM) and basic iterative method (BIM)) trained on different RSI benchmark datasets to fool CNNs (i.e., InceptionV1, ResNet and a simple CNN). In the experiment, our results show that CNNs of RSI scene classification are also vulnerable to adversarial examples, and some of them have a fooling rate of over 80%. These adversarial examples are affected by the architecture of CNNs and the type of RSI dataset. InceptionV1 has a fooling rate of less than 5%, which is lower than the others. Adversarial examples generated on the UCM dataset are easier than other datasets. Importantly, we also find that the classes of adversarial examples have an attack selectivity property. Misclassifications of adversarial examples of RSIs are related to the similarity of the original classes in the CNN feature space. Attack selectivity reveals potential classes of adversarial examples and provides insights into the design of defensive algorithms in future research.

This study proposes a framework for describing a scene change using natural language text based on indoor scene observations conducted before and after a scene change. The recognition of scene changes plays an essential role in a variety of real-world applications, such as scene anomaly detection. Most scene understanding research has focused on static scenes. Most existing scene change captioning methods detect scene changes from single-view RGB images, neglecting the underlying three-dimensional structures. Previous three-dimensional scene change captioning methods use simulated scenes consisting of geometry primitives, making it unsuitable for real-world applications. To solve these problems, we automatically generated large-scale indoor scene change caption datasets. We propose an end-to-end framework for describing scene changes from various input modalities, namely, RGB images, depth images, and point cloud data, which are available in most robot applications. We conducted experiments with various input modalities and models and evaluated model performance using datasets with various levels of complexity. Experimental results show that the models that combine RGB images and point cloud data as input achieve high performance in sentence generation and caption correctness and are robust for change type understanding for datasets with high complexity. The developed datasets and models contribute to the study of indoor scene change understanding.