DEEP LEARNING BASED OPTICAL FLOW ESTIMATION FOR CHANGE DETECTION: A CASE STUDY IN INDONESIA EARTHQUAKE

Various scene-based nonuniformity correction (SBNUC) methods have been proposed to diminish the residual nonuniformity (RNU) of the infrared focal plane array (IRFPA) sensors. Most existing SBNUC techniques require a relatively large number of image frames to reduce the RNU. In some applications, however, there is not enough time for capturing a large number of image frames prior to the camera operation, or only several image frames are available to users. A new scene-based approach that can correct the RNU using only several image frames is proposed. The proposed method formulates the SBNUC process as an energy minimization problem. In the proposed energy function, we introduce regularization terms for the parameter regarding the responsivity of the IRFPA as well as for the true scene irradiance. Correction results are obtained by minimizing the energy function using a numerical technique. Experimental results demonstrate the effectiveness of the proposed method.

The explosive growth of online short videos has brought great challenges to the efficient management of video content classification, retrieval, and recommendation. Video features for video management can be extracted from video image frames by various algorithms, and they have been proven to be effective in the video classification of sensor systems. However, frame-by-frame processing of video image frames not only requires huge computing power, but also classification algorithms based on a single modality of video features cannot meet the accuracy requirements in specific scenarios. In response to these concerns, we introduce a short video categorization architecture centered around cross-modal fusion in visual sensor systems which jointly utilizes video features and text features to classify short videos, avoiding processing a large number of image frames during classification. Firstly, the image space is extended to three-dimensional space–time by a self-attention mechanism, and a series of patches are extracted from a single image frame. Each patch is linearly mapped into the embedding layer of the Timesformer network and augmented with positional information to extract video features. Second, the text features of subtitles are extracted through the bidirectional encoder representation from the Transformers (BERT) pre-training model. Finally, cross-modal fusion is performed based on the extracted video and text features, resulting in improved accuracy for short video classification tasks. The outcomes of our experiments showcase a substantial superiority of our introduced classification framework compared to alternative baseline video classification methodologies. This framework can be applied in sensor systems for potential video classification.

The seminal work of Horn and Schunck is the first variational method for optical flow estimation. It introduced a novel framework where the optical flow is computed as the solution of a minimization problem. From the assumption that pixel intensities do not change over time, the optical flow constraint equation is derived. This equation relates the optical flow with the derivatives of the image. There are infinitely many vector fields that satisfy the optical flow constraint, thus the problem is ill-posed. To overcome this problem, Horn and Schunck introduced an additional regularity condition that restricts the possible solutions. Their method minimizes both the optical flow constraint and the magnitude of the variations of the flow field, producing smooth vector fields. One of the limitations of this method is that, typically, it can only estimate small motions. In the presence of large displacements, this method fails when the gradient of the image is not smooth enough. In this work, we describe an implementation of the original Horn and Schunck method and also introduce a multi-scale strategy in order to deal with larger displacements. For this multi-scale strategy, we create a pyramidal structure of downsampled images and change the optical flow constraint equation with a nonlinear formulation. In order to tackle this nonlinear formula, we linearize it and solve the method iteratively in each scale. In this sense, there are two common approaches: one approach that computes the motion increment in the iterations; or the one we follow, that computes the full flow during the iterations. The solutions are incrementally refined over the scales. This pyramidal structure is a standard tool in many optical flow methods.

<p indent=0mm>Combining the deep learning model to compute end-to-end optical flow is a hot topic in current computer vision field. The optical flow estimation methods based on deep learning are summarized and reviewed. Firstly, the origin and concept of optical flow is introduced. Secondly, the optical flow datasets and evaluation metrics are summarized. Most importantly, classical methods are introduced and the deep optical flow estimation methods are reviewed in three aspects, including supervised deep optical flow estimation methods, unsupervised deep optical flow estimation methods and the performance of these methods. The analysis shows designing compact and generalized deep optical flow model is the future research direction. On this basis, the joint learning of optical flow estimation and specific video analysis tasks are introduced. It is pointed out that the design of task-driven deep optical flow network is a valuable research direction in practice application. Finally, the problems and challenges of deep optical flow estimation are summarized and the future work of this field is prospected.

This study addresses the challenges of selecting a suitable visual tracking method for real-time mobile robot applications, particularly in scenarios where the target is moving on the ground. The primary research problem addressed is the need for a flexible, computationally efficient tracking method that does not rely on pre-existing labelled datasets, as is often required by deep learning approaches. Unsupervised methods can overcome this problem by utilizing object motion information in each image frame without prior training. With many unsupervised tracking methods available, choosing an appropriate algorithm that can perform efficiently under dynamic conditions becomes a critical problem. The study compares the performance of three unsupervised visual tracking methods: particle filter, optical flow, and channel and spatial reliability tracker (CSRT) under various tracking conditions. The dataset used includes challenges such as moving target variations, changes in object scale, viewpoint changes, suboptimal lighting, image blurring, partial occlusions, and abrupt movements. Evaluation criteria include tracking accuracy, resistance to occlusion, and computational efficiency. The particle filter with ORB and a constant velocity model achieves a root mean square error (RMSE) of 36.47 pixels at 13 frames per second (fps). Optical flow performs best with an RMSE of 10.79 pixels at 30 fps, while CSRT shows an RMSE of 252.35 pixels at 4 fps. These findings highlight the effectiveness of optical flow for real-time applications, making it a promising solution for mobile robot visual tracking in challenging situations.

The gradient-based methods for optical flow estimation are based on a constraint equation which is defined for each image pixel. The structure of the constraint equation makes the problem ill-posed so in the past some solutions have been proposed based on regularization. Under the assumption that in the immediate neighbourhood of a pixel the optical flow field is smooth, the constraint equations in that neighbourhood should have a common solution, in this case the problem is not ill-posed. Following this reasoning, an algorithm for evaluating the optical flow, which is suitable for parallel implementation is proposed in this paper. Moreover, parallel implementations of selected algorithms from the literature, for optical flow estimation, are presented in this paper with the intention of comparing their complexity and performance with respect to the proposed approach on a Connection Machine-2.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

There is no consensus regarding the benefit of implantable hearing aids in congenital unilateral conductive hearing loss (UCHL). This study aimed to measure sound source localization performance in patients with congenital UCHL and contralateral normal hearing who received a new bone conduction implant. Evaluation of within-subject performance differences for sound source localization in a horizontal plane. Tertiary referral center. Five patients with atresia of the external auditory canal and contralateral normal hearing implanted with transcutaneous bone conduction implant at the Medical University of Vienna were tested. Activated/deactivated implant. Sound source localization test; localization performance quantified using the root mean square (RMS) error. Sound source localization ability was highly variable among individual subjects, with RMS errors ranging from 21 to 40 degrees. Horizontal plane localization performance in aided conditions showed statistically significant improvement compared with the unaided conditions, with RMS errors ranging from 17 to 27 degrees. The mean RMS error decreased by a factor of 0.71 (p < 0.001). Analysis revealed improved sound localization performance in a horizontal plane with the activated transcutaneous bone conduction implant. Some patients with congenital UCHL might be capable of developing improved horizontal plane localization abilities with the binaural cues provided by this device.

The change point detection in periodic time series is much desirable in many practical usages. We present a novel algorithm for this task, which includes two phases: 1) anomaly measure- on the basis of a typical regression model, we propose a new computation method to measure anomalies in time series which does not require any reference data from other measurement(s); 2) change detection- we introduce a new martingale test for detection which can be operated in an unsupervised and nonparametric way. We have conducted extensive experiments to systematically test our algorithm. The results make us believe that our algorithm can be directly applicable in many real-world change-point-detection applications.

This paper presents an evolutionary algorithm for modeling the arrival dates of document streams, which is any time-stamped collection of documents, such as newscasts, e-mails, scientific journals archives and weblog postings. The goal is to find a frequency curve that fits the data circumventing the unavoidable noise. Classical dynamic programming algorithms are limited by memory and efficiency requirements, which can be a problem when dealing with long streams. This suggests to explore alternative search methods which although do not guarantee optimality, are far more efficient. Experiments have shown that the designed evolutionary algorithm is able to reach high quality solutions in a short time. We have also explored different approaches to infer whether new arrivals increase or decrease interest in the topic the document stream is about. In particular, we present a variant of the evolutionary algorithm, which is able to very quickly fit a stream extended with new data, by taking advantage of the fit obtained for the original substream. These mechanisms can be used for real time detection of changes in the trend of interest in a topic, an important application of this kind of models.

: Work on improving gradient-based methods for optical flow estimation has been completed. An understanding of how errors arise makes it possible to define the inherent limitations of the gradient-based technique, obtain estimates of the accuracy of computed values, enhance the performance of the technique, and demonstrate the informative value of some types of errors. Significant results have been achieved on the problems associated with motion-based segmentation. An approach based on understanding the three-dimensional scene structure leading to an edge in optical flow has been developed. As a result, it is possible to simultaneously detect edges and determine important three-dimensional properties of the associated scene surfaces. The methods which have been developed make it possible to distinguish between occluding and occluded surfaces at a boundary. This technique may make it possible to link image regions corresponding to a partially occuled object and to produce descriptions of object boundaries that are less affected by occlusion. In addition, being able to distinguish between occluding and occluded boundaries is a crucial step towards determining the three-dimension position of surfaces. Keywords: image understanding, visual motion.

Robust optical flow estimation via edge preserving filtering

The keyframe image character of 3D animation refers to the image state of the character or scene and other objects presented at specific keyframes in the animation timeline. The character texture will be blurred, distorted and discontinuous in the animation sequence due to motion, perspective change or lighting difference. Due to the lack of optical flow information to guide the current 3D animation keyframe image enhancement process, it is difficult to effectively distinguish the foreground character from the background elements, which affects the algorithm’s ability of recognizing the animation character’s movement patterns. To this end, a character texture enhancement method based on the SIFT algorithm and LK (Lucas-Kanade) optical flow is investigated for 3D animation keyframe images. The SIFT algorithm is used to extract the feature points of the character in each frame of 3D animation image as the initial tracking template, the LK optical flow method is used to calculate the optical flow vector of the feature points in the new frame, and the weighted directional histogram is used to estimate the direction of the character’s movement, and then the feature points in the tracking template are optimized through feature matching and weight updating to complete the tracking of the character’s target in each frame of the image. According to the target tracking results, a fractional-order differential mask is constructed in the character target region, and an adaptive function is constructed by combining the modes of the local gradient, information entropy and variance parameters of the image, so as to realize the enhancement of the character texture of the key frame images of the 3D animation by dynamically adjusting the order of the fractional-order differential. The experimental results show that the method can realize the dynamic tracking of 3D animation image character, and the extracted 3D animation image character feature points show a balanced distribution, and the distribution ratio is reasonable. It can realize the texture enhancement of 3D animation keyframe images, and the enhanced animation character clothing, facial details and other information are clearly recognizable, so as to improve the overall quality and visual impact of 3D animation.

In this study, the accuracy of ocean analysis data, which are produced from the Korea Meteorological Administration (KMA) Nucleus for European Modelling of the Ocean/Variational Data Assimilation (NEMO/NEMOVAR, hereafter NEMO) system and the HYbrid Coordinate Ocean Model/Navy Coupled Ocean Data Assimilation (HYCOM/NCODA, hereafter HYCOM) system, was evaluated using various oceanic observation data from March 2015 to February 2016. The evaluation was made for oceanic thermal environments in the tropical Pacific, the western North Pacific, and the Korean peninsula. NEMO generally outperformed HYCOM in the three regions. Particularly, in the tropical Pacific, the RMSEs (Root Mean Square Errors) of NEMO for both the sea surface temperature and vertical water temperature profile were about 50% smaller than those of HYCOM. In the western North Pacific, in which the observational data were not used for data assimilation, the RMSE of NEMO profiles up to 1000 m (0.49℃) was much lower than that of HYCOM (0.73℃). Around the Korean peninsula, the difference in RMSE between the two models was small (NEMO, 0.61oC; HYCOM, 0.72oC), in which their errors show relatively big in the winter and small in the summer. The differences reported here in the accuracy between NEMO and HYCOM for the thermal environments may be attributed to horizontal and vertical resolutions of the models, vertical coordinate and mixing scheme, data quality control system, data used for data assimilation, and atmosphere forcing. The present results can be used as a basic data to evaluate the accuracy of NEMO, before it becomes the operational model of the KMA providing real-time ocean analysis and prediction data.

Change detection using deep learning approach with object-based image analysis

Intracellular pH and redox states are critical for multiple processes and partly determine cell behavior. Here, we developed a genetically encoded dual-function probe, named p H and redox-sensitive fluorescent protein (pHaROS), for simultaneous real-time detection of changes in redox potential and pH in living cells. pHaROS consists of the Arabidopsis flavin mononucleotide-binding fluorescent protein iLOV and an mKATE variant, mBeRFP. Using pHaROS in Saccharomyces cerevisiae cells, we confirmed that H2O2 raises the overall redox potential of the cell and found that this increase is accompanied by a decrease in cytosolic pH. Furthermore, we observed spatiotemporal pH and redox homeostasis within the nucleus at various stages of the cell cycle in budding yeast (Saccharomyces cerevisiae) during cellular development and responses to oxidative stress. Importantly, we could tailor pHaROS to specific applications, including measurements in different organelles and cell types and the GSH/GSSG ratio, highlighting pHaROS's high flexibility and versatility. In summary, we have developed pHaROS as a dual-function probe that can be used for simultaneously measuring cellular pH and redox potential, representing a very promising tool for determining the cross-talk between intracellular redox- and pH-signaling processes in yeast and mammalian U87 cell.