DTFS-eHarris: A High Accuracy Asynchronous Corner Detector for Event Cameras in Complex Scenes

Event-based cameras, also known as biologically inspired visual sensors, are capable of capturing real-time scene changes efficiently. Unlike traditional frame-based cameras, event cameras solely report triggered pixel-level brightness changes which are referred to as events. Event-based cameras show many advantages such as high temporal resolution, low latency, and high dynamic range, making them very attractive in robotics and computer vision, especially in challenging scenarios that are too demanding for traditional cameras. In this paper, we provide a comprehensive overview of the emerging field of event-based vision, focusing on the operation principle, sampling mechanisms, and algorithms that take advantage of their superior features. We also delve into the various tasks for which event cameras are utilized, such as object tracking, optical flow estimation, 3D reconstruction, SLAM, image reconstruction, and recognition. Additionally, we highlight the challenges and future opportunities for event cameras, seeking a more efficient way for machines to perceive and interact with the world.

Event cameras are bioinspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in the order of microseconds. However, due to the fundamentally different structure of the sensor's output, new algorithms that exploit the high temporal resolution and the asynchronous nature of the sensor are required. Recent work has shown that a continuous-time representation of the event camera pose can deal with the high temporal resolution and asynchronous nature of this sensor in a principled way. In this paper, we leverage such a continuous-time representation to perform visual-inertial odometry with an event camera. This representation allows direct integration of the asynchronous events with microsecond accuracy and the inertial measurements at high frequency. The event camera trajectory is approximated by a smooth curve in the space of rigid-body motions using cubic splines. This formulation significantly reduces the number of variables in trajectory estimation problems. We evaluate our method on real data from several scenes and compare the results against ground truth from a motion-capture system. We show that our method provides improved accuracy over the result of a state-of-the-art visual odometry method for event cameras. We also show that both the map orientation and scale can be recovered accurately by fusing events and inertial data. To the best of our knowledge, this is the first work on visual-inertial fusion with event cameras using a continuous-time framework.

Cameras are appealing sensors for mobile robots because they are small, passive, inexpensive and provide rich information about the environment. While cameras have been used successfully on a plenitude of robots, such as autonomous cars or drones, serious challenges remain: power consumption, latency, dynamic range, and frame rate, among others. The sequences of images acquired by a camera are highly redundant (both in space and time), and both acquiring and processing such an amount of data consumes significant power. This limits the operation time of mobile robots and, moreover, defines a fundamental power-latency tradeoff. Specialized cameras designed for high-speed or high-dynamic-range scenarios are expensive, heavy, and require additional power, which prevents their use in agile mobile robots. In this thesis, we investigate event cameras as a biologically-inspired alternative to overcome the limitations of standard cameras. These neuromorphic vision sensors work in a completely different way: instead of providing a sequence of images (i.e., frames) at a constant rate, event cameras transmit only information from those pixels that undergo a significant brightness change. These pixel-level brightness changes, called events, are timestamped with micro-second resolution and transmitted asynchronously at the time they occur. Hence, event cameras are power efficient because they convey only non-redundant information, and are able to capture very high-speed motions, thus they directly address the power-latency tradeoff. Additionally, event cameras achieve a dynamic range of more than 140dB, compared to about 60dB of standard cameras, because each pixel is autonomous and operates at its own set-point. However, since the output of an event camera is fundamentally different from that of standard cameras for which computer-vision algorithms have been developed during the past fifty years, new algorithms that can deal with the asynchronous nature of the sensor and exploit its high temporal resolution are required to unlock its potential. This thesis presents algorithms for using event cameras in the context of robotics. Since event cameras are novel sensors that are being intensively prototyped and have been commercially available only recently (ca. 2008), the literature on event-based algorithms is scarce. This poses some operational challenges as well as uncountable opportunities to explore in research. This thesis focuses on exploring the possibilities that event cameras bring to some fundamental problems in robotics and computer vision, such as localization and actuation. Amongst others, this thesis provides contributions to solving the localization problem, i.e., for a robot equipped with an event camera to be able to infer its location with respect to a given map of the environment. Classical approaches for robot localization build upon lower-level vision algorithms, and so, this thesis also presents contributions in the topics of detection, extraction, and tracking of salient visual features with an event camera, whose applicability expands far beyond the localization problem. This thesis also presents contributions in the use of event cameras for actuation and closed-loop control, i.e., in endowing the robot with the capabilities to interact with the environment to fulfill a given task. Additionally, this thesis also presents the infrastructure developed to work with event cameras in a de-facto standard robotics platform. The following is a list of contributions: * Software infrastructure, consisting of publicly available drivers, calibration tools, sensor delay characterization, and the first event camera dataset and simulator tailored for 6-DOF (degrees of freedom) camera pose estimation and SLAM (Simultaneous localization and mapping). * We introduce the concept of event and provide an algorithm to compute it. The lifetime endows the events with a finite temporal extent for a proper continuous representation of events in time. * The first method to extract FAST-like visual features (i.e., interest points or corners) from the output of an event camera. The detector operates an order of magnitude faster than previous corner detectors. * The first method to extract and track features from the output of a DAVIS camera (an event camera that also outputs standard frames from the same pixel array). Using these feature tracks, we developed the first sparse, feature-based visual-odometry pipeline. * The first two methods to track the 6-DOF pose of an event camera in a known map.While the first method minimizes the reprojection error of the events and only works on black-and-white scenes consisting of line segments, the second method uses a probabilistic filtering framework that allows tracking at high speeds on natural scenes. * The first application of a continuous-time framework to estimate the trajectory of an event camera, possibly incorporating inertial measurements, showing superior performance over pose-tracking-only methods. * An application of event cameras to collision avoidance of a quadrotor, showing how event cameras can be used to control a robot with very low latency. * An application of the use of an event camera for human-vs-machine slot-car racing, showing that event-driven algorithms are power efficient and can outperform human control.

Recently, the event camera has become a popular and promising vision sensor in the research of simultaneous localization and mapping and computer vision owing to its advantages: low latency, high dynamic range, and high temporal resolution. As a basic part of the feature-based SLAM system, the feature tracking method using event cameras is still an open question. In this article, we present a novel asynchronous event feature generation and tracking algorithm operating directly on event-streams to fully utilize the natural asynchronism of event cameras. The proposed algorithm consists of an event-corner detection unit, a descriptor construction unit, and an event feature tracking unit. The event-corner detection unit addresses a fast and asynchronous corner detector to extract event-corners from event-streams. For the descriptor construction unit, we propose a novel asynchronous gradient descriptor inspired by the scale-invariant feature transform descriptor, which helps to achieve quantitative measurement of similarity between event feature pairs. The construction of the gradient descriptor can be decomposed into three stages: speed-invariant time surface maintenance and extraction, principal orientation calculation, and descriptor generation. The event feature tracking unit combines the constructed gradient descriptor and an event feature matching method to achieve asynchronous feature tracking. We implement the proposed algorithm in C++ and evaluate it on a public event dataset. The experimental results show that our proposed method achieves improvement in terms of tracking accuracy and real-time performance when compared with the state-of-the-art asynchronous event-corner tracker and with no compromise on the feature tracking lifetime.

Event cameras as bioinspired vision sensors have shown great advantages in high dynamic range and high temporal resolution in vision tasks. Asynchronous spikes from event cameras can be depicted using the marked spatiotemporal point processes (MSTPPs). However, how to measure the distance between asynchronous spikes in the MSTPPs still remains an open issue. To address this problem, we propose a general asynchronous spatiotemporal spike metric considering both spatiotemporal structural properties and polarity attributes for event cameras. Technically, the conditional probability density function is first introduced to describe the spatiotemporal distribution and polarity prior in the MSTPPs. Besides, a spatiotemporal Gaussian kernel is defined to capture the spatiotemporal structure, which transforms discrete spikes into the continuous function in a reproducing kernel Hilbert space (RKHS). Finally, the distance between asynchronous spikes can be quantified by the inner product in the RKHS. The experimental results demonstrate that the proposed approach outperforms the state-of-the-art methods and achieves significant improvement in computational efficiency. Especially, it is able to better depict the changes involving spatiotemporal structural properties and polarity attributes.

Event cameras with high dynamic range and temporal resolution, which are bio-inspired vision sensors, have shown great potential in event-based tracking tasks, particularly in scenarios involving rapid motion and low levels of illumination. Nonetheless, the efficient extraction of sparse information from event camera remains a persistent challenge. Meanwhile, the event camera works asynchronously, generating a continuous stream of events, rendering it highly compatible with Spiking Neural Networks (SNNs) due to their event-driven nature and low power consumption. Motivated by the issues mentioned above, we propose an Efficient Hybrid Cascade Tracker ( EHCT ) with SNN for object tracking in the event domain. We combine the transformer, convolutional network, and SNN structure skillfully to form the basic Hybrid CNN-SNN-Transformer (HCST) block structure, which is also the central component part of our EHCT network. The HCST block is primarily utilized to process the incoming event data and extract information from both local and global contexts. After several cascade HCST blocks, these two types of information will be efficiently integrated with the preprocessed raw data, which are then fed into the Classifier and Regressor head to produce the bounding box (bbox) of the tracked target. Extensive experiments on various event and RGB frame-based datasets demonstrated that our proposed EHCT algorithm outperforms most of the existing state-of-the-art trackers by a significant margin and also achieves a great advantage in terms of energy consumption. Our source code will be available at https://github.com/masac11/EHCT .

Early detection of lung cancer is crucial in reducing mortality. Magnetic resonance imaging (MRI) may be a viable imaging technique for lung cancer detection. Numerous lung nodule detection methods have been studied for computed tomography (CT) images. However, to the best of our knowledge, no detection methods have been carried out for the MR images. In this paper, a lung nodule detection method based on deep learning is proposed for thoracic MR images. With parameter optimizing, spatial three-channel input construction, and transfer learning, a faster R-convolution neural network (CNN) is designed to locate the lung nodule region. Then, a false positive (FP) reduction scheme based on anatomical characteristics is designed to reduce FPs and preserve the true nodule. The proposed method is tested on 142 T2-weighted MR scans from the First Affiliated Hospital of Guangzhou Medical University. The sensitivity of the proposed method is 85.2% with 3.47 FPs per scan. The experimental results demonstrate that the designed faster R-CNN network and the FP reduction scheme are effective in the lung nodule detection and the FP reduction for MR images.

We are developing a computer-aided detection system to assist radiologists in the detection of lung nodules on thoracic computed tomography (CT) images. The purpose of this study was to improve the false-positive (FP) reduction stage of our algorithm by developing features that extract three-dimensional (3D) shape information from volumes of interest identified in the prescreening stage. We formulated 3D gradient field descriptors, and derived 19 gradient field features from their statistics. Six ellipsoid features were obtained by computing the lengths and the length ratios of the principal axes of an ellipsoid fitted to a segmented object. Both the gradient field features and the ellipsoid features were designed to distinguish spherical objects such as lung nodules from elongated objects such as vessels. The FP reduction performance in this new 25-dimensional feature space was compared to the performance in a 19-dimensional space that consisted of features extracted using previously developed methods. The performance in the 44-dimensional combined feature space was also evaluated. Linear discriminant analysis with stepwise feature selection was used for classification. The parameters used for feature selection were optimized using the simplex algorithm. Training and testing were performed using a leave-one-patient-out scheme. The FP reduction performances in different feature spaces were evaluated by using the area Az under the receiver operating characteristic curve and the number of FPs per CT section at a given sensitivity as accuracy measures. Our data set consisted of 82 CT scans (3551 axial sections) from 56 patients with section thickness ranging from 1.0 to 2.5 mm. Our prescreening algorithm detected 111 of the 116 solid nodules (nodule size: 3.0-30.6 mm) marked by experienced thoracic radiologists. The test Az values were 0.95 +/- 0.01, 0.88 +/- 0.02, and 0.94 +/- 0.01 in the new, previous, and combined feature spaces, respectively. The number of FPs per section at 80% sensitivity in these three feature spaces were 0.37, 1.61, and 0.34, respectively. The improvement in the test Az with the 25 new features was statistically significant (p<0.0001) compared to that with the previous 19 features alone.

This study presents an advanced data augmentation framework to enhance fault diagnostics in industrial centrifugal pumps using vibration data. The proposed framework addresses the challenge of insufficient defect data in industrial settings by integrating traditional augmentation techniques, such as Gaussian noise (GN) and signal stretching (SS), with advanced models, including Long Short-Term Memory (LSTM) networks, Autoencoders (AE), and Generative Adversarial Networks (GANs). Our approach significantly improves the robustness and accuracy of machine learning (ML) models for fault detection and classification. Key findings demonstrate a marked reduction in false positives and a substantial increase in fault detection rates, particularly in complex operational scenarios where traditional statistical methods may fall short. The experimental results underscore the effectiveness of combining these augmentation techniques, achieving up to a 30% improvement in fault detection accuracy and a 25% reduction in false positives compared to baseline models. These improvements highlight the practical value of the proposed framework in ensuring reliable operation and the predictive maintenance of centrifugal pumps in diverse industrial environments.

We are developing a computer-aided detection system to aid radiologists in diagnosing lung cancer in thoracic computed tomographic (CT) images. The purpose of this study was to improve the false-positive (FP) reduction stage of our algorithm by developing and incorporating a gradient field technique. This technique extracts 3D shape information from the gray-scale values within a volume of interest. The gradient field feature values are higher for spherical objects, and lower for elongated and irregularly-shaped objects. A data set of 55 thin CT scans from 40 patients was used to evaluate the usefulness of the gradient field technique. After initial nodule candidate detection and rule-based first stage FP reduction, there were 3487 FP and 65 true positive (TP) objects in our data set. Linear discriminant classifiers with and without the gradient field feature were designed for the second stage FP reduction. The accuracy of these classifiers was evaluated using the area Az under the receiver operating characteristic (ROC) curve. The Az values were 0.93 and 0.91 with and without the gradient field feature, respectively. The improvement with the gradient field feature was statistically significant (p=0.01).

The rapid growth of the Internet of Things (IoT) has revolutionized various industries by enabling interconnected devices to exchange data seamlessly. However, IoT systems face significant security challenges due to decentralized architectures, resource-constrained devices, and dynamic network environments. These challenges include denial-of-service (DoS) attacks, anomalous network behaviors, and data manipulation, which threaten the security and reliability of IoT ecosystems. New methods based on machine learning have been reported in the literature, addressing topics such as intrusion detection and prevention. This paper proposes an advanced anomaly detection framework for IoT networks expressed in several phases. In the first phase, data preprocessing is conducted using techniques like the Median-KS Test to remove noise, handle missing values, and balance datasets, ensuring a clean and structured input for subsequent phases. The second phase focuses on optimal feature selection using a Genetic Algorithm enhanced with eagle-inspired search strategies. This approach identifies the most significant features, reduces dimensionality, and enhances computational efficiency without sacrificing accuracy. In the final phase, an ensemble classifier combines the strengths of the Decision Tree, Random Forest, and XGBoost algorithms to achieve the accurate and robust detection of anomalous behaviors. This multi-step methodology ensures adaptability and scalability in handling diverse IoT scenarios. The evaluation results demonstrate the superiority of the proposed framework over existing methods. It achieves a 12.5% improvement in accuracy (98%), a 14% increase in detection rate (95%), a 9.3% reduction in false positive rate (10%), and a 10.8% decrease in false negative rate (5%). These results underscore the framework’s effectiveness, reliability, and scalability for securing real-world IoT networks against evolving cyber threats.

Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in the order of microseconds. We propose a novel, accurate tightly-coupled visual-inertial odom- etry pipeline for such cameras that leverages their outstanding properties to estimate the camera ego-motion in challenging conditions, such as high-speed motion or high dynamic range scenes. The method tracks a set of features (extracted on the image plane) through time. To achieve that, we consider events in overlapping spatio-temporal windows and align them using the current camera motion and scene structure, yielding motion-compensated event frames. We then combine these feature tracks in a keyframe- based, visual-inertial odometry algorithm based on nonlinear optimization to estimate the camera’s 6-DOF pose, velocity, and IMU biases. The proposed method is evaluated quantitatively on the public Event Camera Dataset [19] and significantly outperforms the state-of-the-art [28], while being computationally much more efficient: our pipeline can run much faster than real-time on a laptop and even on a smartphone processor. Fur- thermore, we demonstrate qualitatively the accuracy and robustness of our pipeline on a large-scale dataset, and an extremely high-speed dataset recorded by spinning an event camera on a leash at 850 deg/s.

Event cameras are bio-inspired vision sensors with a high dynamic range (140 dB for event cameras <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">vs.</i> 60 dB for traditional cameras) and can be used to tackle the image degradation problem under extremely low-illumination scenarios, which is still not well-explored yet. In this article, we propose a joint framework to compose the underexposed frames and event streams captured by the event camera to reconstruct clear images with detailed textures under almost dark conditions. A residual fusion module is proposed to reduce the domain gap between event streams and frames by using the residuals of both modalities. A multi-level reconstruction loss based on the variability of the contrast distribution is proposed to reduce the perceptual errors of the output image. In addition, we construct the first real-world low-illumination image enhancement dataset (mainly under 2 lux illumination scenes), named LIE, containing event streams and frames collected under indoor and outdoor low-light scenarios together with the ground truth clear images. Experimental results on our LIE dataset demonstrate that our proposed method could achieve significant improvements compared with existing methods.

Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. These cameras do not suffer from motion blur and have a very high dynamic range, which enables them to provide reliable visual information during high-speed motions or in scenes characterized by high dynamic range. These features, along with a very low power consumption, make event cameras an ideal complement to standard cameras for VR/AR and video game applications. With these applications in mind, this paper tackles the problem of accurate, low-latency tracking of an event camera from an existing photometric depth map (i.e., intensity plus depth information) built via classic dense reconstruction pipelines. Our approach tracks the 6-DOF pose of the event camera upon the arrival of each event, thus virtually eliminating latency. We successfully evaluate the method in both indoor and outdoor scenes and show that-because of the technological advantages of the event camera-our pipeline works in scenes characterized by high-speed motion, which are still inaccessible to standard cameras.

Recent advances in event-based cameras have led to significant developments in robotics, particularly in visual simultaneous localization and mapping (VSLAM) applications. This technique enables real-time camera motion estimation and simultaneous environment mapping using visual sensors on mobile platforms. Event cameras offer several distinct advantages over frame-based cameras, including a high dynamic range, high temporal resolution, low power consumption, and low latency. These attributes make event cameras highly suitable for addressing performance issues in challenging scenarios such as high-speed motion and environments with high-range illumination. This review paper delves into event-based VSLAM (EVSLAM) algorithms, leveraging the advantages inherent in event streams for localization and mapping endeavors. The exposition commences by explaining the operational principles of event cameras, providing insights into the diverse event representations applied in event data preprocessing. A crucial facet of this survey is the systematic categorization of EVSLAM research into three key parts: event preprocessing, event tracking, and sensor fusion algorithms in EVSLAM. Each category undergoes meticulous examination, offering practical insights and guidance for comprehending each approach. Moreover, we thoroughly assess state-of-the-art (SOTA) methods, emphasizing conducting the evaluation on a specific dataset for enhanced comparability. This evaluation sheds light on current challenges and outlines promising avenues for future research, emphasizing the persisting obstacles and potential advancements in this dynamically evolving domain.