A self-organization based optical flow estimator with GPU implementation

Temporal segmentation of a video sequence into different shots is fundamental to a number of video retrieval and analysis applications. Motion estimation has been wildly used in many applications of video processing, since it provides the most essential information for an image sequence. In this paper, we explore the possibility to exploit motion and illumination estimation in a video sequence to detect various types of shot changes. Optical flow is the motion vector computed at each pixel in an image sequence from intensity variation. Traditionally, optical flow computational algorithms were derived from the brightness constancy assumption. In this paper, we employ a generalized optical flow constraint that includes an illumination parameter to model local illumination changes. An iterative optical flow and illumination estimation algorithm is developed in this paper to refine the generalized optical flow constraints step by step, thus leading to a very accurate estimation of the optical flow and illumination parameters. Two robust measures are defined from the mean and standard deviation of the estimated intensity compensation values for all the blocks in the same image. Either of these two measures corresponds significantly to various types of shot changes. We show the usefulness of these two measures through experiments.

Using self-organizing maps to accelerate similarity search

Superresolution localization microscopy strongly relies on robust identification algorithms for accurate reconstruction of the biological systems it is used to measure. The fields of machine learning and computer vision have provided promising solutions for automated object identification, but usually rely on well represented training sets to learn object features. However, using a static training set can result in the learned identification algorithm making mistakes on data that is not well represented by the training set. Here, we present a method for training an artificial neural network without providing a training set in advance. This method uses the data to be analyzed, and the fitting algorithm to train an artificial neural network tailored to that data set. We show that the same artificial neural network can learn to identify at least two types of molecular emissions: the regular point spread functions (PSFs), and the astigmatism PSF. Simulations indicate that this method can be extremely reliable in extracting molecular emission signatures. Additionally, we implemented the artificial neural network calculation to be performed on a graphics processing unit (GPU) for massively parallelized calculation which drastically reduces the time required for the identification process. By implementing the neural identification on a GPU, we allow this method of identification to be used in a real time analysis algorithm. RESEARCH HIGHLIGHTS: Here, we present a machine learning algorithm for identifying point-spread functions without the need for an a priori training set. We show that this method can detect over 90% of molecules with less than 1% false positive identification in simulations. We further show that because this algorithm does not make assumptions of about the shape of molecular emission, it is compatible with models beyond the symmetric 2D Gaussian.

Motion and structure estimation are elementary problems of computer vision. These are active areas of research, even though the first methods were proposed several decades ago. We develop new approaches for motion and structure estimation for autonomous driving. An autonomous vehicle requires an accurate model of its environment, wrong decisions made by an autonomous car can have severe consequences. We assume the monocular setup, where only a single camera is mounted on the car. Outdoor traffic sequences are challenging for optical flow estimation. The high speed of the car causes large displacements in the optical flow field, the lighting conditions are unstable and there can be strong distortions due to reflections and difficult weather conditions. We propose new discrete methods, which determine optical flow as optimal configuration of probabilistic graphical models. The first approach selects sparse locations in the reference frame, and matches them across the second image. The best correspondences, which match constraints from a multiple view configuration, are considered motion vectors in a graphical model. In a second approach, we solve for dense optical flow by approximating the original infeasible graphical model with a sequence of reduced models. The monocular configuration poses challenges to the estimation of scene structure, camera positions and scene parameters need to be estimated jointly. The geometry of multiple views creates blind spots in the images, and adds a scale ambiguity, which both to not exist in a setup with multiple cameras. We propose two methods for structure estimation. The first approach determines the energy optimal camera track, given optical flow and depth observations. A further approach estimates camera motion and a piecewise planar scene description jointly from a single optical flow field. The scene description contains depth and plane normal information. We evaluate our approaches for motion and structure estimation on different real world and rendered datasets. In addition to evaluation on publicly available evaluation data, we evaluate on a new rendered dataset with ground truth plane normals.

The recent video compression standard, HEVC (high efficiency video coding), will most likely be used in various applications in the near future. However, the encoding process is far too slow for real-time applications. At the same time, computing capabilities of GPUs (graphics processing units) have become more powerful in these days. In this paper, we have proposed a GPU-based parallel motion estimation (ME) algorithm to enhance the performance of an HEVC encoder. A frame is partitioned into two subframes for pipelined execution to improve GPU utilization. The flow chart is redetermined to solve data hazards in the pipelined execution. Two new methods are introduced in the proposed ME: decision of a representative search center position (RSCP) and warp-based concurrent parallel reduction (WCPR). A RSCP employs motion vectors of a co-located CTU in a previously encoded frame to solve a dependency problem in parallel computation with negligible coding loss. WCPR concurrently executes several parallel reduction operations, which increases the thread utilization from 20 to 89 % without any thread synchronization. The proposed encoder can make the portion of ME in the encoder negligible with 2.2 % bitrate increase against the HEVC test model (HM) encoder. In terms of ME, the proposed ME is 130.7 times faster than that of the HM encoder.

For specular regions (SRs), the assumption of brightness (or other) constancy between images corresponding to multiple views of a scene breaks down. As a consequence, optical-flow (OF) based motion-estimation (ME) algorithms that rely on constancy assumptions fail for specular regions. At the same time estimation of SRs in an image is also prone to errors, particularly to false positives from bright regions in the scene. In this paper, motivated by the fact that specular regions are typically encountered in image regions corresponding to portions of relatively smooth 3D surfaces, we propose an algorithm for improving ME and SRs localization via joint processing. Initial estimates of OF and of the SRs are obtained by conventional methods. The estimate of the SRs is updated using inconsistency of the OF with respect to the neighboring region to reinforce true positives and to reject false positives. The OF is then re-computed with a modified energy functional that, in effect, emphasizes regularization in a spatially adaptive neighborhood of the SRs to improve the estimated OF. Experimental results on synthetic and real image pairs demonstrate that the proposed algorithm offers a significant improvement in both SRs localization and ME over recently proposed methods for tackling these problems.

Sensor devices are regularly used on unmanned aerial vehicles (UAVs) as reconnaissance and intelligence gathering systems and as support for front line troops on operations. This platform provides a wealth of sensor data and has limited computational power available for processing. The objective of this work is to detect and identify objects in real-time, with a low power footprint so that it can operate on a UAV. An appraisal of current computer vision methods is presented, with reference to their performance and applicability to the objectives. Experimentation with real-time methods of background subtraction and motion estimation was carried out and limitations of each method described. A new, assumption free, data driven method for object detection and identification was developed. The core ideas of the development were based on models that propose that the human vision system analyses edges of objects to detect and separate them and perceives motion separately, a function which has been modelled here by optical flow. The initial development in the temporal domain combined object and motion detection in the analysis process. This approach was found to have limitations. The second iteration used a detection component in the spatial domain that extracts texture patches based on edge contours, their profile, and internal texture structure. Motion perception was performed separately on the texture patches using optical flow. The motion and spatial location of texture patches was used to define physical objects. A clustering method is used on the rich feature set extracted by the detection method to characterise the objects. The results show that the method carries out detection and identification of both moving and static objects, in real-time, irrespective of camera motion.

In Motion estimation (ME), the block matching algorithms have a great potential of parallelism. This process of the best match is performed by computing the similarity for each block position inside the search area, using a similarity metric, such as Sum of Absolute Differences (SAD). It is used in the various steps of motion estimation algorithms. Moreover, it can be parallelized using Graphics Processing Unit (GPU) since the computation algorithm of each block pixels is similar, thus offering better results. In this work a fixed OpenCL code was performed firstly on several architectures as CPU and GPU, secondly a parallel GPU-implementation was proposed with CUDA and OpenCL for the SAD process using block of sizes from 4x4 to 64x64. A comparative study established between execution time on GPU on the same video sequence. The experimental results indicated that GPU OpenCL execution time was better than that of CUDA times with performance ratio that reached the double.

Tissue motion estimation is an essential step for ultrasound elastography. Our previous study has shown that the affine-model-based optical flow (OF) method outperforms the normalized cross-correlation-based block matching (BM) method in motion estimation. However, the quality of lateral estimation using OF is still low due to inherent limitation of ultrasound imaging. BM-based spatial angular compounding (SAC) has been developed to obtain better motion estimation. In this paper, OF-based SAC (OF-SAC) is proposed to further improve the performance of lateral (and axial) estimation, and it is compared with BM-based SAC (BM-SAC). Plane wave as well as focused wave is transmitted in both simulations and phantom experiments on a linear array. In order to compare the performance quantitatively, the root-mean-square error (RMSE) of axial/lateral displacement and strain, and signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) of axial/lateral strain are used as the evaluation criteria in the simulations. In the phantom experiments, the SNR and CNR are used to assess the quality of axial/lateral strain. The results show that for both OF and BM, SAC improves the performance of motion estimation, regardless of using plane or focused wave transmission. More importantly, OF-SAC is shown to outperform BM-SAC with lower RMSE, higher SNR, and higher CNR. In addition, preliminary in vivo experiments on the carotid artery of a healthy human subject also prove the superiority of OF-SAC. These results suggest that OF-SAC is preferred for both axial and lateral motion estimation to BM-SAC.

We have developed a system to detect the presence of seizures in the multichannel scalp EEG. At the heart of the system is the Self-Organising Feature Map (SOFM) that has been trained on normal and epileptiform EEG segments of 12 patients (64 seizures). Following preliminary spatial analysis, autoregressive (AR) parameters are extracted from variable width segments which have been delineated through the use of a non-linear energy operator. The AR parameters are used as feature vectors for the SOFM training process. Following initial training, probability values are automatically assigned to the ‘prototype’ seizure segments based on the consensus of 3 EEGers. The use of a self-organising network retains objectivity in calculating the prototype seizure segments.Preliminary results (using the training set only) are given here. With a detection threshold of d th =0.49 the Sensitivity and Selectivity were both measured at 75% with a corresponding false detection rate of 0.5 / hour. These preliminary results indicate that the system shows promise for use as a generic seizure detection system - i.e., a non-patient specific seizure detection system.KeywordsCurrent DipoleSeizure DetectionFalse Detection RateFixed Model OrderAutomate Seizure DetectionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Database mining applied to central nervous system (CNS) activity

In this paper a new fast block based motion estimation algorithm is presented. The new algorithm is based on well known diamond search with incorporated recursive and multi stage techniques. Its main benefit against other existing fast block based motion estimation algorithms is the ability to overcome the ambiguity in motion estimation (aperture problem, indeterminate flow vector in regions with constant brightness) that yield from optical flow equation. These problems are eliminated due to our new search strategy which employs fact that the image pixels corresponding to the same object projection all account similar movement. The algorithm with combined recursive and multi stage techniques is capable of estimating vector motion fields with eliminated chaotic motion fields and approaching true motion field.

Objective: This study aimed to fit Autoregressive Integrated Moving Average (ARIMA) models to the prevalence of overweight in Türkiye's overall, female, and male populations and to forecast future trends using the best-performing ARIMA models. Methods: The dataset comprised annual overweight prevalence values for Türkiye's overall, female, and male populations from 1974 to 2022, obtained from the World Health Organization and World Bank Group databases. The dataset was divided into training and test sets in a chronological sequence with the ratio 80:20, respectively. Training sets were used to fit ARIMA models, while test sets were used to evaluate the predictive performance of the models. Best ARIMA models were chosen based on various evaluation metrics. Results: The best models were identified as ARIMA (1,3,1) for the overall population, ARIMA (1,3,1) for females, and ARIMA (3,3,1) for males, yielding the lowest error metrics. These models effectively captured the increasing trend in overweight prevalence. Short-term forecasts indicated that the upward trend is likely to continue in the near future. Conclusion: This study contributes to a foundational understanding of the trajectory of overweight prevalence in Türkiye, providing a basis for evidence-based interventions and long-term health planning.

DCM Research Resources LLC, Germantown, Maryland 20874Abstract. In our previous work, we proposed a joint optical flow and principal componentanalysis (PCA) approach to improve the performance of optical flow based detection, wherePCA is applied on the calculated two-dimensional optical flow image, and motion detectionis accomplished by a metric derived from the two eigenvalues. To reduce the computationaltime when processing airborne videos, parallel computing using graphic processing unit(GPU) is implemented on NVIDIA GeForce GTX480. Experimental results demonstrate thatour approach can efficiently improve detection performance even with dynamic background,and processing time can be greatly reduced with parallel computing on GPU.

Addressing on monocular visual odometry problem, this paper presents a novel end-to-end network for estimation of camera ego-motion. The network learns the latent space of optical flow (OF) and models sequential dynamics so that the motion estimation is constrained by the relations between sequential images. We compute the OF field of consecutive images and extract the latent OF representation in a self-encoding manner. A Recurrent Neural Network is then followed to examine the OF changes, i.e., to conduct sequential learning. The extracted sequential OF latent space is used to compute the regression of the 6-dimensional pose vector. Particularly, we separately train the encoder in an unsupervised manner. By this means, we avoid non-convergence during the training of the whole network and allow more generalized and effective feature representation. Substantial experiments have been conducted on KITTI and Malaga datasets, and the results demonstrate that our model outperforms most learning-based VO approaches.