ROVINS: Robust Omnidirectional Visual Inertial Navigation System

Dynamic backgrounds and sudden illumination changes are two of the major problems associated with background subtraction techniques. In this paper, we present a novel approach to background subtraction that addresses both of these challenges. In particular, we present an improved codebook background modelling and subtraction technique. We utilise image segmentation on the background image and model the background with a codebook for each pixel along with a pseudo background layer. We perceive background motion as an occlusion of one background layer by a nearby background layer. In other words, sliding of one background layer over a neighbouring layer causes background motion and will hence result in false segmentation. We present our approach of codeword spreading across layer boundaries to handle background motion. Furthermore, we present a two-step update of the background codebook to handle both sudden and gradual illumination changes.

The schooling behavior of both light- and dark-adapted Astyanax mexicanus (Fillipi) was observed under sudden changes in illumination in the range 10−7 to 100 ft-c at different times in the diel cycle. In darkness the fish were dispersed, swam slowly, and showed no signs of mutual recognition of one another. There was no simple relationship between the schooling response and the illumination, but in general the frequency, size, and durability of schools were proportional to the illumination. Schooling was maximal above 10−1 ft-c and approached zero at 10−6 ft-c. The schooling at any level of illumination varied with the testing procedures. The lowest responses were obtained with a gradual reduction of illumination in the evening. At illumination below 10−2 ft-c maximal schooling was obtained from dark-adapted fish. From 10−2 ft-c upward, it was obtained from light-adapted fish. The schooling rate (number of fish in a school) reached 100% for a few seconds at 10−3 ft-c and for longer periods at higher illumination, but was never sustained at the 100% level throughout any 4-min period. Schooling was generally enhanced by any sudden change in illumination but was never sustained at a high rate under monotonous conditions. The decline with monotony was equated with reduction of the fright situation. The decline with reduced illumination was attributed to the reduction of visual acuity and visibility, and occurred at a greater rate among faster-swimming fish. Notemigonus crysoleucas (Mitchill), swimming very slowly, exhibited high schooling rates at 10−5 ft-c. Earlier work showed that the retina of Astyanax in continuous darkness undergoes a rhythmic diel shift. The behavior showed that this retinal shift is not the classical transformation from scotopic to photopic vision. Dark-adapted during the day, the fish demonstrated rapid visual perception under sudden illumination at any value from 10−6 to 10−0.69 ft-c. Dark-adapted fish at night exhibited increasingly longer light-shock reactions with sudden illumination above 10−2 ft-c. According to schooling responses the cone threshold was about 10−3 ft-c, and the rod threshold about 10−6 ft-c. The mesopic range extended from about 10−3 to 10−2 ft-c.

본 논문은 조명의 변화가 심한 영상에서 손 형태를 안정적으로 인지하는 기법에 관한 것이다. 제안한 방법은 HSI 색상공간에서 색상(Hue) 및 색상 기울기(Hue-Gradient)를 기반으로 정의된 배경모델을 구축하고, 실시간으로 입력되는 영상과의 배경차분(background subtraction)기법을 이용하여 배경과 손을 구분한다. 추출된 손의 영역으로부터 18가지의 특징요소를 추출하고 이를 기반으로 다중클래스 SVM(Support Vector Machine) 학습 기법을 사용하여 손의 형태를 인지한다. 제안 기법은 색상 기울기를 배경 차분에 적용함으로써, 조명 환경이 배경 모델의 조명과 다르게 급격한 변화가 이루어졌을 때도 안정적으로 손의 윤곽정보를 추출할 수 있도록 하였다. 또한, 실시간 처리를 저해하는 복잡한 손의 특성정보 대신, OBB의 크기에 대하여 정규화된 두 개의 고유값과 객체 기반 바운딩 박스(OBB)를 구성하는 16개 세부 영역에서의 손 윤곽픽셀의 개수를 손의 특성정보로 사용하였다. 본 논문에서는 급격한 조명 변화 상황에서 기존 RGB 색상요소를 기반으로 하는 배경차분법과 색상을 기반으로 하는 배경차분법, 본 논문에서 제안하는 색상 기울기 기반 배경 차분법의 결과를 비교함으로써 제안 기법의 안정성을 입증하였다. 6명의 실험대상자의 1부터 9까지의 수지화 2700개의 영상으로부터 손 특성 정보를 추출하고 이에 대하여 훈련을 통한 학습 모델을 생성하였다. 학습모델을 기반으로 실험자 6인의 손 형태 1620개의 데이터에 대하여 인지 실험을 실시하여 92.6%에 이르는 손 형태 인식 성공률을 얻었다. In this paper, we present a robust hand recognition approach to sudden illumination changes. The proposed approach constructs a background model with respect to hue and hue gradient in HSI color space and extracts a foreground hand region from an input image using the background subtraction method. Eighteen features are defined for a hand pose and multi-class SVM(Support Vector Machine) approach is applied to learn and classify hand poses based on eighteen features. The proposed approach robustly extracts the contour of a hand with variations in illumination by applying the hue gradient into the background subtraction. A hand pose is defined by two Eigen values which are normalized by the size of OBB(Object-Oriented Bounding Box), and sixteen feature values which represent the number of hand contour points included in each subrange of OBB. We compared the RGB-based background subtraction, hue-based background subtraction and the proposed approach with sudden illumination changes and proved the robustness of the proposed approach. In the experiment, we built a hand pose training model from 2,700 sample hand images of six subjects which represent nine numerical numbers from one to nine. Our implementation result shows 92.6% of successful recognition rate for 1,620 hand images with various lighting condition using the training model.

In order to improve the trajectory control effect of multi‐degree‐of‐freedom industrial robots, this paper combines visual image technology to conduct research on trajectory control of multi‐degree‐of‐freedom industrial robots. Aiming at the problem of video segmentation under sudden illumination changes, this paper uses a Gaussian mixture model based on the global illumination function to adopt a variety of illumination invariant features, and proposes a scene segmentation algorithm suitable for sudden illumination changes. Moreover, this paper compares and verifies the algorithm from the subjective and objective perspectives through experiments, which shows that the algorithm in this paper can segment the scene more accurately even in the environment of sudden changes in illumination. In addition, the results of the accuracy test and the trajectory control test show that the research method of the multi‐degree‐of‐freedom industrial robot trajectory control based on the visual image proposed in this paper can effectively improve the trajectory control effect of the robot.

Abstract. Localization is the process of determining the path, position and orientation of a robot in an environment. The data from different sensors such as LiDAR, inertial measurement units (IMU) or cameras are used in the localization process. This task is of fundamental importance to enable robots to navigate autonomously and perform tasks effectively. The robot localization can be performed utilizing different techniques either using hardware or software designs. Visual localization algorithms can be shown as one of the localization techniques that enable robots to determine their position and orientation. In this aspect, visual odometry is one of the mostly used methods. It is a technique that enables robots to determine their positions and movements by analysing sequential images. It tracks features in consecutive images to determine the movement of the camera between those frames which enables the determination of the robot’s position in the environment. The process commonly involves detecting and matching key points or features in the images, such as corners or edges, and then using algorithms to calculate the camera's motion. Visual odometry is useful in environments where Global Navigation Satellite Systems (GNSS) or other external positioning systems cannot operate. In this study, the use of visual odometry is assessed in comparison with the iPad Pro LiDAR and steel tape results to determine the distances between each image-taking point. The iPad and steel tape results were taken as the ground truth and the root mean square values were determined by comparing the algorithm and their results.

To solve the problem of indoor surveillance, this paper is concerned about the framework that allows the detection of moving objects based on IP camera. The current background subtraction algorithms cannot correctly classify foreground and background if there are sudden illumination changes. To eliminate the influence of illumination changes, intensity flicker correction technique is employed before background subtraction in this paper. Meanwhile to reduce the false alarms further, the union of two separate detection results is employed. In the experimental results, both global illumination changes and local ones are included. The results show that the improved subtractor proposed in this paper performs better than ViBe both in the aspect of correct detection rate and false alarms.

In this study, we built a monocular visual inertial odometry (VIO) dataset with a proposed self-calibrating platform for an inertial measurement unit (IMU). For a learning-based method, lack of database is the biggest limitation. One goal of this study is to generate a high-quality VIO database. Meanwhile, a properly calibrated inertial measurement unit (IMU) is critical for improving VIO system accuracy and precision. However, a professional IMU calibration tool is usually expensive and large. The proposed self-calibrating platform shows its benefits in eliminating the manual work in calibration and reducing the cost of the expensive platform, which brings more opportunities to students and researchers. This self-calibrating system was useful for collecting monocular visual inertial odometry datasets while driving. . The calibrated IMU generates 14 numbers at 100Hz: time, 3-axis accelerometer measurements, 3-axis gyroscope measurements, 3 Euler angles, and 4 quaternions. This VIO system is composed of a monocular camera, GoPro Hero 3, an IMU, BNO055, and a GPS, ANT-555. The sensors are mounted on a John Deere Gator™ Full-Size Crossover Utility Vehicle and are operated on an Intel NUC, a mini-size computer. The final dataset contains monocular images, IMU measurements, and GPS outputs.

The traditional visual inertial simultaneous localisation and mapping system does not fully consider the dynamic objects in the scene, which can reduce the quality of visual feature point matching. In addition, dynamic objects in the scene can cause illumination changes which reduce the performance of the visual front end and loop closure detection of the system. To address this problem, this study combines 3D light detection and ranging (LiDAR), camera, and inertial measurement units in a tightly coupled manner to estimate the pose of mobile robots, thereby proposing a robust LiDAR visual inertial odometry that can effectively filter out dynamic feature points. In addition, a dynamic feature point detection algorithm with attention mechanism is introduced for target detection and optical flow tracking. In experimental analyses on public datasets and real indoor scenes, the proposed method improved the accuracy and robustness of pose estimation in scenes with dynamic objects and varying illumination compared with traditional methods.

Robust visual tracking is the important stage in the computer vision applications such as robotics, man-free control systems, and the visual surveillance. Accurate motion states estimation and the target representation in visual tracking system are based on the appearances of the target. The factor affects the learning of target representation is the accumulated error due to the pose, illumination changes, and the uneven background. The presence of dynamic background and the shadowing effects causes the visual drift and destructive information. Besides, the misclassification of target region induces the false detection of moving objects. The K-means and Fuzzy-C-means clustering algorithms are available to segment the foreground/background and suppress the shadow region on the basis of the non-changing background of the surveillance area. This paper proposes the novel background normalization technique with textural pattern analysis to suppress the shadow region. The Neighborhood Chain Prediction (NCP) algorithm is used to cluster the uneven background and the Differential Boundary Pattern (DBP) extracts the texture of the video frame to suppress the shadow pixels present in the frame. The lower intensity estimation and the prediction of the area around the lower intensity in proposed work enhance the pixels for shadow removal. The shadow-free frame split up into several grids and the histograms of features are extracted from the grid formatted frame. Finally, the Machine Level Classification (MLC) finds the matching grid corresponds to the tracking region and provides the binary labeling to separate the background and foreground. The proposed DBP-based visual tracking system is high robustness over the sudden illumination changes and the dynamic background due to the texture pattern analysis. The comparison of proposed NCP-DBP combination with the existing segmentation techniques regarding the accuracy, precision, recall, F-measure, success and error rate assured the effectiveness in visual tracking applications.

Foreground segmentation in a video is a key process in many applications such as object detection, an object tracking, and a behavior analysis. Since the extracted foreground objects are often used in the analytical process, the quality of the foreground is a significant factor to the success of these applications. However, there are many key challenges in the foreground segmentation, including dynamic backgrounds, gradual illumination changes, sudden illumination changes, shadows, and long-term scene changes. This paper proposes a novel dynamic codebook method to address such challenges. The dynamic codebook aims to significantly improve the conventional well-known codebook technique by introducing a technique to make a dynamic boundary of each codeword. In this technique, the lab color space is used in order to make the model more resilient to the illumination change. The experimental results and comprehensive comparisons demonstrate that the proposed method can achieve very promising performance.

In the real-time segmentation of the moving objects in a image sequence, methods based on background subtraction are used widely. Bi-polar Radial Reach Correlation (BP-RRC) has realized robust background subtraction by evaluating a local texture per pixel, suppressing the influence of brightness change. However, this method has a fault that is affected by the local illumination change of a background. This paper proposes a robust and stabilized background subtraction algorithm which can cope with texture change of various illumination changes by improving this texture background model. We verified the proposed method using the image sequence including a loose lighting change and a rapid lighting change, and adaptability higher than the conventional method on the sudden illumination changes was attained as a result of verification.

The analysis of moving objects in video sequences has been a paramount issue in applications related to intelligent surveillance systems, robotics, and medicine. Although several works aimed to analyze objects in video sequences have been reported, many of them need manual parameter adjustments and they are not tolerant to illumination changes and dynamic backgrounds. Therefore, a novel scheme termed Dynamic Retinotopic SOM based on an adaptive artificial neural network, to detect moving objects is proposed in this work. The neural network is a model based on the mechanisms of the visual cortex that we called Retinotopic SOM (RESOM) and it is also proposed in this paper. Furthermore, RESOM is a real-time neural network that can adapt its learning parameters based on the scene behavior and it mimics perception abilities. A quantitative comparison with other segmentation methods reported in the literature using real video scenes showed that the proposed DR-SOM segmentation method automatically adjusts its parameters and outperforms the reported methods in condition of dynamic backgrounds, and gradual and sudden illumination changes.

Developmental changes in behavioral and retinomotor responses of Pacific bluefin tuna on exposure to sudden changes in illumination

The YTU dataset and recurrent neural network based visual-inertial odometry

For their complete realization, autonomous vehicles (AVs) fundamentally rely on the Global Navigation Satellite System (GNSS) to provide positioning and navigation information. However, in area such as urban cores, parking lots, and under dense foliage, which are all commonly frequented by AVs, GNSS signals suffer from blockage, interference, and multipath. These effects cause high levels of errors and long durations of service discontinuity that mar the performance of current systems. The prevalence of vision and low-cost inertial sensors provides an attractive opportunity to further increase the positioning and navigation accuracy in such GNSS-challenged environments. This paper presents enhancements to existing multisensor integration systems utilizing the inertial navigation system (INS) to aid in Visual Odometry (VO) outlier feature rejection. A scheme called Aided Visual Odometry (AVO) is developed and integrated with a high performance mechanization architecture utilizing vehicle motion and orientation sensors. The resulting solution exhibits improved state covariance convergence and navigation accuracy, while reducing computational complexity. Experimental verification of the proposed solution is illustrated through three real road trajectories, over two different land vehicles, and using two low-cost inertial measurement units (IMUs).