Indoor Scene Change Captioning Based on Multimodality Data

대용량 컨텐츠 수요와 공급의 증가에 따라 UHD 비디오의 분석, 색인, 편집 등을 위한 장면전환을 검출하는 방법이 요구되고 있다. 이전까지 많은 연구자들이 다양한 장면전환 검출 방법을 연구해왔지만 카메라의 다양한 움직임과 장면의 변화를 정확하게 검출하기는 어려웠다. 또한, 4K (3820x2160) 해상도 이상의 UHD 비디오들은 데이터 량을 더욱더 증가시키기 때문에 이전의 장면전환 검출 방법은 UHD 비디오 컨텐츠에 적용하기에는 너무 많은 시간이 걸리는 문제점이 발생한다. 따라서, 압축률이 높은 차세대 고효율 코덱 HEVC를 이용하여 장면전환을 검출하는 방법이 요구되고 있다. 본 논문에서는 차세대 고효율 코덱 HEVC의 부호화 부가정보를 이용한 4가지 장면전환 검출 방법을 제안하며, 대용량 비디오의 장면전환 검출을 위한 픽셀 기반의 새로운 장면전환 검출 시스템 구조를 제안한다. 연산량을 줄이기 위해 프레임 특징에 따른 방법을 각각 제시하며, 실험 결과를 통해 HEVC로 부호화 된 UHD 컨텐츠들의 장면전환 검출 가능성을 확인한다. With the increase of mass contents data, a method of a scene change detection is required for analysis, indexing and editing. Although many researchers are studying a variety of scene change detection method, it is too difficult to accurately detect various movements of the cameras and scene changes. Also, earlier scene change detection methods take too much time to apply to UHD video contents. That is because the UHD video contents with 4K (3820x2160) resolution or higher have greater amount of data. Therefore a method for detecting a scene change by using the next-generation codec, HEVC, is required. In this paper, we propose four scene change detection methods using the coding additional information of HEVC, and a new pixel-based scene change detection system. Furthermore, through the experimental results, we check the possibility of detecting the scene changes of UHD videos encoded in HEVC format.

An efficient video de-interlacing technique with scene change detection is proposed and its performances are examined. Scene changes happen quite often in film broadcasting and they tend to destabilize the quality of performance such as jagged effect, blurred effect, and artifacts effect, while de-interlacing technique is utilized. Therefore, the issue of scene change needs to be addressed with de-interlacing process. In the proposed method, de-interlacing begins with scene change detection, which is to ensure that the interfield information is used correctly. To improve the quality of de-interlacing, the factors of scene change are taken into account when de-interlacing techniques are applied. The simulation results show that the proposed algorithm exhibits better performances than other interpolation algorithms

Video is an important and challenging medium and requires sophisticated indexing schemes for efficient retrieval from visual databases. An important step in video indexing is scene change detection. Recently, several scene change detection algorithms in the pixel and compressed (MPEG-2) domains have been reported in the literature. These algorithms are computationally complex and are not very robust in detecting gradual scene changes. The authors propose an efficient technique for detecting scene changes in the MPEG-2 compressed domain. The proposed algorithm has the advantage of fast scene change detection. In addition, this algorithm has the potential to detect gradual scene changes.

An algorithm is proposed to detect abrupt and gradual scene changes in H.264 compressed videos. In H.264, the inter prediction in the encoding process generates the DC component of the prediction residual. In this paper, we use the DC component of inter prediction residuals as well as motion vectors to detect scene changes. Using the proposed algorithm, we can detect scene changes for H.264 compressed videos more effectively, as demonstrated by simulations.

In modern greenhouses, complicated tasks and unstructured environments generate the imperious demand for advanced semantic information about each object at work scenes. A significant problem that mainstream methods intend to resolve is that the refinement and understanding of environmental information cannot efficiently cover the entire task in real time. Therefore, this paper proposes a panoptic semantic mapping method to identify each object that is supposed to be concerned in greenhouses. This method builds grid maps with advanced semantic information based on RGB and depth images. For the agricultural task with tomato as the working object, the categories of various objects in the grid map are divided into four groups: fruits, pedicels, stems and obstacles. This method consists of three steps: semantic segmentation from RGB images with K-Net, reconstruction of point cloud data based on depth images and semantic masks and transformation of the point cloud data into OctoMap. Experimental results show the semantic segmentation algorithm reaches a mean precision of semantic segmentation of 93.83%, a mean IoU of 88.39% and an average accuracy of 98.28%. Meanwhile, the refresh frequency of publishing point cloud data with advanced semantic information holds steady at 2 Hz with the resolution of 8 mm.

Unmanned aerial vehicles (UAVs), also known as drones, are a cost-effective alternative to traditional surveying methods, and they can be used to collect geospatial data over inaccessible or hard-to-reach locations. UAV-integrated miniaturized remote sensing sensors such as hyperspectral and LiDAR sensors, which formerly operated on airborne and spaceborne platforms, have recently been developed. Their accuracies can still be guaranteed when incorporating pieces of equipment such as ground control points (GCPs) and field spectrometers. This study conducted three experiments for geometric and radiometric accuracy assessments of simultaneously acquired RGB, hyperspectral, and LiDAR data from a single mission. Our RGB and hyperspectral data generated orthorectified images based on direct georeferencing without any GCPs. Because of this, a base station is required for the post-processed Global Navigation Satellite System/Inertial Measurement Unit (GNSS/IMU) data. First, we compared the geometric accuracy of orthorectified RGB and hyperspectral images relative to the distance of the base station to determine which base station should be used. Second, point clouds could be generated from overlapped RGB images and a LiDAR sensor. We quantitatively and qualitatively compared RGB and LiDAR point clouds in this experiment. Lastly, we evaluated the radiometric quality of hyperspectral images, which is the most critical factor of the hyperspectral sensor, using reference spectra that was simultaneously measured by a field spectrometer. Consequently, the distance of the base station for post-processing the GNSS/IMU data was found to have no significant impact on the geometric accuracy, indicating that a dedicated base station is not always necessary. Our experimental results demonstrated geometric errors of less than two hyperspectral pixels without using GCPs, achieving a level of accuracy that is comparable to survey-level standards. Regarding the comparison of RGB- and LiDAR-based point clouds, RGB point clouds exhibited noise and lacked details; however, through the cleaning process, their vertical accuracy was found to be comparable with LiDAR’s accuracy. Although photogrammetry generated denser point clouds compared with LiDAR, the overall quality for extracting the elevation data greatly relies on factors such as the original image quality, including the image’s occlusions, shadows, and tie-points, for matching. Furthermore, the image spectra derived from hyperspectral data consistently demonstrated high radiometric quality without the need for in situ field spectrum information. This finding indicates that in situ field spectra are not always required to guarantee the radiometric quality of hyperspectral data, as long as well-calibrated targets are utilized.

Scene change detection is the process of identifying the differences between the multi-temporal image scenes at the semantic level, which has significant potential in the application of urban development and land management. In this paper, we propose a novel deep convolution canonical correlation analysis neural network (DCCANet) architecture, which could consider the spectral-spatial-temporal correlation feature for scene change detection in remote sensing images. For this purpose, we put together the convolutional neural networks (CNNs) and the deep canonical correlation analysis (DCCA) into the end-to-end network. The CNN could get the spectral-spatial feature information for scene representation, while the later could enhance the temporal correlation by nonlinear high- dimensional transformations between the multi-temporal image scenes for scene change detection. Experiments with high-resolution remote sensing image scene datasets demonstrated that our proposed approach can get a better performance in scene classification and change detection.

3D reconstruction from images is a core problem in computer vision. With recent advances in deep learning, it has become possible to recover plausible 3D shapes even from single RGB images for the first time. However, obtaining detailed geometry and texture for objects with arbitrary topology remains challenging. In this paper, we propose a novel approach for reconstructing point clouds from RGB images. Unlike other methods, we can recover dense point clouds with hundreds of thousands of points, and we also include RGB textures. In addition, we train our model on multiple categories which leads to superior generalization to unseen categories compared to previous techniques. We achieve this using a two-stage approach, where we first infer an object coordinate map from the input RGB image, and then obtain the final point cloud using a reprojection and completion step. We show results on standard benchmarks that demonstrate the advantages of our technique. Code is available at https://github.com/TaoHuUMD/3D-Reconstruction.

In order to fulfill the requirement of limited channel bandwidth and of growing video demand like streaming media delivery on internet, and digital library, video compression is necessary. In video compression, temporal redundancy between adjacent frames is removed with block based motion estimation algorithms. Video represents a sequence of frames captured from camera. Scene is a series of consecutive frames captured from narrative point of view. In this paper we present an effective scene change detection method for an uncompressed video. We have divided frames in to blocks and applied a canny edge detector in consecutive frames. Count no of pixels (ones) in each block and compare it with consecutive frames. If scene change happens then number of pixels per block will change, based on that change we can detect scene change in consecutive frame. Here we have presented a hybrid approach, in which we have used scene change detection along with block based motion estimation algorithms (BME) to compress video.

In this paper, we leverage multimodal data to classify minerals using a multi-stream neural network. In a previous study on the Tinto dataset, which consisted of a 3D hyperspectral point cloud from the open-pit mine Corta Atalaya in Spain, we successfully identified mineral classes by employing various deep learning models. However, this prior work solely relied on hyperspectral data as input for the deep learning models. In this study, we aim to enhance accuracy by incorporating multimodal data, which includes hyperspectral images, RGB images, and a 3D point cloud. To achieve this, we have adopted a graph-based neural network, known for its efficiency in aggregating local information, based on our past observations where it consistently performed well across different hyperspectral sensors. Subsequently, we constructed a multi-stream neural network tailored to handle multimodality. Additionally, we employed a channel attention module on the hyperspectral stream to fully exploit the spectral information within the hyperspectral data. Through the integration of multimodal data and a multi-stream neural network, we achieved a notable improvement in mineral classification accuracy: 19.2%, 4.4%, and 5.6% on the LWIR, SWIR, and VNIR datasets, respectively.

In this paper, we propose a new scene change detection method based on multi-histogram for frame rate up-conversion. The proposed method manages multiple per-block histograms to extract the locality of image change. Thus, it can detect local scene change as well as global scene change between frames. Experiments show that the proposed method improves by 14.05dB the image quality of the interpolated frame with the local scene change.

Video scene change detection has great importance of managing and analyzing large amount of videos. Traditionally this technique used for indexing, segmenting and categorizing different types of videos. Very few works addressed to classify obscene using scene change detection method. In this research we proposed a simple approach for sensing objectionable videos by observing scene changes into different video genres. Video scenes are grouped into set of key frames. After analyzing duration of each scene and counting the number of key frames of designated scene, it has been shown that obscene videos have infrequent scene changing nature. While in sports, dramas, music and action films have large number of scene changes. We used six types of video genres and the decision has been made by setting a threshold based on extracted key frames. Experimental result showed that the accuracy is 83.33% and false positive rate is 16.67%.

Rate control plays a key role in a video encoder, and it has been a challenging research topic. In this paper, an adaptive variable length group of picture (GOP) level rate control algorithm for H.264 based on scene change detection is proposed. In the proposed algorithm, the length of current GOP changes adaptively with the occurrence of scene change, so that the entire sequence contains a fixed number of I frame to avoid the actual bit rate increasing. Furthermore, it adjusts the QP of the scene change frame to ease the pressure of the buffer. Simulation results show that the proposed algorithm improves the video quality significantly on the basis of meeting bit rate constraints accurately, which shows better adaptability for sequences with scene change occurrence frequently. When compared with existed schemes, the average peak signal-to-noise ratio (PSNR) improvement is more than 1.4 dB.

A common framework for rapid scene analysis for detecting scene changes in compressed motion JPEG and MPEG videos is proposed. We develop algorithms to detect both abrupt and gradual scene changes. The algorithms operate directly on the DC sequence which can be easily extracted from motion JPEG and MPEG compressed video without decompression. The DC images capture most of the essential "global" information, but is of a small fraction of the original data size. Operating on these images offers significant computation savings. Experimental results show that the proposed algorithms are fast and effective in detecting abrupt scene changes and gradual transitions.

Length Measurement of Potato Leaf using Depth Camera