Multi-Modal Feature Constraint Based Tightly Coupled Monocular Visual-LiDAR Odometry and Mapping

Motion estimation is one of the key perception problems for Micro Aerial Vehicles (MAVs). Recent developments based on visual-inertial sensors exhibit promising performance. In this paper, we propose a fast stereo-inertial odometry approach leveraging the advantages of both visual feature tracking and visual inertial fusion. Instead of tracking visual feature on every frame, our method estimates short-term motion with inertial-assisted flow tracking. By that, the frame-rate of visual feature detection and tracking is reduced to save computational power. Also, we design an Extended Kalman Filter (EKF) to achieve full state estimation, including position, velocity and attitude. Our method is able to estimate camera motion with less computational effort, while maintaining comparative state estimation performance. Experiments on the Euroc dataset which is captured onboard a MAV show the good performance of our method.

The task of multi-label image classification involves recognizing multiple objects within a single image. Considering both valuable semantic information contained in the labels and essential visual features presented in the image, tight visual-linguistic interactions play a vital role in improving classification performance. Moreover, given the potential variance in object size and appearance within a single image, attention to features of different scales can help to discover possible objects in the image. Recently, Transformer-based methods have achieved great success in multi-label image classification by leveraging the advantage of modeling long-range dependencies, but they have several limitations. Firstly, existing methods treat visual feature extraction and cross-modal fusion as separate steps, resulting in insufficient visual-linguistic alignment in the joint semantic space. Additionally, they only extract visual features and perform cross-modal fusion at a single scale, neglecting objects with different characteristics. To address these issues, we propose a Hierarchical Scale-Aware Vision-Language Transformer (HSVLT) with two appealing designs: (1)A hierarchical multi-scale architecture that involves a Cross-Scale Aggregation module, which leverages joint multi-modal features extracted from multiple scales to recognize objects of varying sizes and appearances in images. (2)Interactive Visual-Linguistic Attention, a novel attention mechanism module that tightly integrates cross-modal interaction, enabling the joint updating of visual, linguistic and multi-modal features. We have evaluated our method on three benchmark datasets. The experimental results demonstrate that HSVLT surpasses state-of-the-art methods with lower computational cost.

Fake news that combines text and images has a better story-telling ability than text-only fake news, making it more deceptive and easier to spread maliciously. Therefore, multi-modal fake news detection has become a new hot topic. There are two main challenges in this task. First, the traditional pre-training model BERT has more parameters and a relatively slow training speed, which limits the level of extracted text features. Second, in the multi-modal form, the fusion process is only a simple splicing of visual and textual features of news, and the obtained multi-modal features are insufficient to express the complementarity between multi-modal data and may have redundant information, potentially leading to biased detection results.In order to solve the above issues, we propose ALB-MCF, using ALBERT and Multi-modal Circulant Fusion(MCF) for fake news detection. ALB-MCF consists of four main modules: a multi-modal feature extractor, a multi-modal feature fusion, a fake news detection and a domain classifier. Specifically, the multi-modal feature extractor innovatively uses a pre-trained ALBERT model to extract text features and a pre-trained VGG19 model to extract visual features. Then, the text features and the visual features are fused into a multi-modal feature representation by MCF, which improves the fusion capability while avoiding an increase in parameters and computational cost. Finally, the multi-modal features are fed into the detector to detect fake news. The role of the domain classifier is mainly to remove event-specific features while retaining shared features between events, thus providing effective detection of emerging events. We have conducted extensive experiments on two real-world datasets. The results demonstrated that our model can handle multi-modal data more effectively, thus improving the accuracy of fake news detection.

<p>Visual grounding locates target objects or areas in the image based on natural language expression. Most current methods extract visual features and text embeddings independently, and then carry out complex fusion reasoning to locate target objects mentioned in the query text. However, such independently extracted visual features often contain many features that are irrelevant to the query text or misleading, thus affecting the subsequent multimodal fusion module, and deteriorating target localization. This study introduces a combined network model based on the transformer architecture, which realizes more accurate visual grounding by using query text to guide visual feature generation and multi-stage fusion reasoning. Specifically, the visual feature generation module reduces the interferences of irrelevant features and generates visual features related to query text through the guidance of query text features. The multi-stage fused reasoning module uses the relevant visual features obtained by the visual feature generation module and the query text embeddings for multi-stage interactive reasoning, further infers the correlation between the target image and the query text, so as to achieve the accurate localization of the object described by the query text. The effectiveness of the proposed model is experimentally verified on five public datasets and the model outperforms state-of-the-art methods. It achieves an improvement of 1.04%, 2.23%, 1.00% and +2.51% over the previous state-of-the-art methods in terms of the top-1 accuracy on TestA and TestB of the RefCOCO and RefCOCO+ datasets, respectively.</p> <p> </p>

Much progress has been achieved during the past two decades in audio-visual automatic speech recognition (AVASR). However, challenges persist that hinder AVASR deployment in practical situations, most notably, robust and fast extraction of visual speech features. We review our efforts in overcoming this problem, based on an appearance-based visual feature representation of the speaker's mouth region. We cover three topics in particular. Firstly, we discuss AVASR in realistic, visually challenging domains, where lighting, background, and head-pose vary significantly. To enhance visual-front-end robustness in such environments, we employ an improved statistical-based face detection algorithm that significantly outperforms our baseline scheme. However, visual-only recognition remains inferior to visually clean (studio-like) data, thus demonstrating the importance of accurate mouth region extraction. We then consider a wearable audio-visual sensor to capture the mouth region directly, thus eliminating face detection. Its use improves visual-only recognition, even over full-face videos recorded in the studio-like environment. Finally, we address the speed issue in visual feature extraction, by discussing our real-time AVASR prototype implementation. The reported progress demonstrates the feasibility of practical AVASR.

Continuous sign language recognition (CSLR) is to recognize the glosses in a sign language video. Enhancing the generalization ability of CSLR's visual feature extractor is a worthy area of investigation. In this paper, we model glosses as priors that help to learn more generalizable visual features. Specifically, the signer-invariant gloss feature is extracted by a pre-trained gloss BERT model. Then we design a gloss prior guidance network (GPGN). It contains a novel parallel densely-connected temporal feature extraction (PDC-TFE) module for multi-resolution visual feature extraction. The PDC-TFE captures the complex temporal patterns of the glosses. The pre-trained gloss feature guides the visual feature learning through a cross-modality matching loss. We propose to formulate the cross-modality feature matching into a regularized optimal transport problem, it can be efficiently solved by a variant of the Sinkhorn algorithm. The GPGN parameters are learned by optimizing a weighted sum of the cross-modality matching loss and CTC loss. The experiment results on German and Chinese sign language benchmarks demonstrate that the proposed GPGN achieves competitive performance. The ablation study verifies the effectiveness of several critical components of the GPGN. Furthermore, the proposed pre-trained gloss BERT model and cross-modality matching can be seamlessly integrated into other RGB-cue-based CSLR methods as plug-and-play formulations to enhance the generalization ability of the visual feature extractor.

Localization is one of the most difficult and costly problems in mobile robotics. To avoid this problem, this paper presents a new controller for the trajectory tracking of nonholonomic mobile robots using visual feedback without direct position measurement. This controller works on the basis of a novel adaptive algorithm for estimating the global position of the mobile robot online using natural visual features measured by a vision system and its orientation and velocity measured by odometry and Attitude and Heading Reference System (IMU&Compass) sensors. The nonholonomic motion constraint of mobile robots is fully taken into account, compared with most of the existing visual servo controllers for mobile robots. The Lyapunov theory is used to prove that the proposed adaptive visual servo controller gives rise to asymptotic tracking of a desired trajectory and convergence of the position estimation to the actual position. A graphical processing unit is adopted to implement the proposed adaptive controller in parallel to achieve real-time detection and tracking of visual features. Experiments on a mobile robot are conducted to validate the effectiveness and robust performance of the proposed controller.

Cameras are appealing sensors for mobile robots because they are small, passive, inexpensive and provide rich information about the environment. While cameras have been used successfully on a plenitude of robots, such as autonomous cars or drones, serious challenges remain: power consumption, latency, dynamic range, and frame rate, among others. The sequences of images acquired by a camera are highly redundant (both in space and time), and both acquiring and processing such an amount of data consumes significant power. This limits the operation time of mobile robots and, moreover, defines a fundamental power-latency tradeoff. Specialized cameras designed for high-speed or high-dynamic-range scenarios are expensive, heavy, and require additional power, which prevents their use in agile mobile robots. In this thesis, we investigate event cameras as a biologically-inspired alternative to overcome the limitations of standard cameras. These neuromorphic vision sensors work in a completely different way: instead of providing a sequence of images (i.e., frames) at a constant rate, event cameras transmit only information from those pixels that undergo a significant brightness change. These pixel-level brightness changes, called events, are timestamped with micro-second resolution and transmitted asynchronously at the time they occur. Hence, event cameras are power efficient because they convey only non-redundant information, and are able to capture very high-speed motions, thus they directly address the power-latency tradeoff. Additionally, event cameras achieve a dynamic range of more than 140dB, compared to about 60dB of standard cameras, because each pixel is autonomous and operates at its own set-point. However, since the output of an event camera is fundamentally different from that of standard cameras for which computer-vision algorithms have been developed during the past fifty years, new algorithms that can deal with the asynchronous nature of the sensor and exploit its high temporal resolution are required to unlock its potential. This thesis presents algorithms for using event cameras in the context of robotics. Since event cameras are novel sensors that are being intensively prototyped and have been commercially available only recently (ca. 2008), the literature on event-based algorithms is scarce. This poses some operational challenges as well as uncountable opportunities to explore in research. This thesis focuses on exploring the possibilities that event cameras bring to some fundamental problems in robotics and computer vision, such as localization and actuation. Amongst others, this thesis provides contributions to solving the localization problem, i.e., for a robot equipped with an event camera to be able to infer its location with respect to a given map of the environment. Classical approaches for robot localization build upon lower-level vision algorithms, and so, this thesis also presents contributions in the topics of detection, extraction, and tracking of salient visual features with an event camera, whose applicability expands far beyond the localization problem. This thesis also presents contributions in the use of event cameras for actuation and closed-loop control, i.e., in endowing the robot with the capabilities to interact with the environment to fulfill a given task. Additionally, this thesis also presents the infrastructure developed to work with event cameras in a de-facto standard robotics platform. The following is a list of contributions: * Software infrastructure, consisting of publicly available drivers, calibration tools, sensor delay characterization, and the first event camera dataset and simulator tailored for 6-DOF (degrees of freedom) camera pose estimation and SLAM (Simultaneous localization and mapping). * We introduce the concept of event and provide an algorithm to compute it. The lifetime endows the events with a finite temporal extent for a proper continuous representation of events in time. * The first method to extract FAST-like visual features (i.e., interest points or corners) from the output of an event camera. The detector operates an order of magnitude faster than previous corner detectors. * The first method to extract and track features from the output of a DAVIS camera (an event camera that also outputs standard frames from the same pixel array). Using these feature tracks, we developed the first sparse, feature-based visual-odometry pipeline. * The first two methods to track the 6-DOF pose of an event camera in a known map.While the first method minimizes the reprojection error of the events and only works on black-and-white scenes consisting of line segments, the second method uses a probabilistic filtering framework that allows tracking at high speeds on natural scenes. * The first application of a continuous-time framework to estimate the trajectory of an event camera, possibly incorporating inertial measurements, showing superior performance over pose-tracking-only methods. * An application of event cameras to collision avoidance of a quadrotor, showing how event cameras can be used to control a robot with very low latency. * An application of the use of an event camera for human-vs-machine slot-car racing, showing that event-driven algorithms are power efficient and can outperform human control.

In recent years, content-based image retrieval method has attracted the attention of many researchers. With the emergence of large-scale computer vision data set, large-scale image classification is going to become an important challenge in the field of computer vision. The existing image classification algorithms mainly have the following defects: (1) lack of sufficient theoretical basis; (2) lower classification accuracy and larger error; (3) algorithm running time is too long because of large amount of data. In order to solve these problems, we introduce an algorithm based on visual saliency and feature extraction theory. Visual saliency means that intelligent algorithm can mark the significant areas in the picture through the simulation of human visual characteristics, and we can preliminarily screen the large-scale date set in terms of saliency extraction, in order to constitute an initial categories set. Image classification mainly includes three processes, that are feature extraction, computation of image pattern vector, classification. Feature extraction is the key factor that affects the performance of image classification. Comparing to global features of image with blur background, partial occlusion, and illumination changes, local features are mode adaptive. Traditional image classification algorithms only use single feature, thus these methods bring limitations. In this paper, we have a thorough research of image classification methods based on feature combination, and we also propose an algorithm based on multiple feature. At the same time, in order to solve the problem that traditional methods dealing with the whole image in a non-hierarchical way, we introduce the theory of visual saliency detection. The results obviously shows that efficient feature extraction combined with the theory of visual saliency are significant to image classification.

The National Highway Traffic Safety Administration (NHTSA) estimates that in the USA alone approximately 100,000 crashes each year are caused primarily by driver drowsiness or fatigue. The major cause for inattentiveness has been found to be a deficit in what we call in this paper an extended view of ergonomics, i.e. the ”extended ergonomics status” of the driving process. This deficit is multidimensional as it includes aspects such as drowsiness (sleepiness), fatigue (lack of energy) and emotions/stress (for example sadness, anger, joy, pleasure, despair and irritation). Different approaches have been proposed for monitoring driver states, especially drowsiness and fatigue, using visual features of the driver such as head movement patterns eyelid movements, facial expressions or all of these together. The effectiveness of the approach depends on the quality of the extracted features, efficiency and the responsiveness of the classification algorithm. In this work, we propose the usage of acoustic information along with visual features to increase the robustness of the emotion/stress measurement system. In terms of the acoustic signals, this work will enlist the appropriate features for the driving situation and correlate them to parameters/dimensions of the ”extended ergonomics status” vector. Prosodic features as well as the phonetic features of the acoustic signal are taken into account for the emotion recognition here. In this paper, a linear discriminant analysis based on a classification method using the Hausdorff distance measure is proposed for classifying the different emotional states. Experimental evaluation based on the Berlin voice database shows that the proposed method results in 85% recognition accuracy in speaker-independent emotion recognition experiments.

In content based image retrieval (CBIR) process, every image has been represented in a compact set of local visual features i.e. color, texture, and/or shape of images. This set of local visual features is known as feature vector. In the CBIR process, feature vectors of images have been used to represent or to identify similar images in adequate way. As a result, feature vector construction has always been considered as an important issue since it must reflect proper image semantics using minimal amount of data. The proposed CBIR scheme is based on the combination of color and texture features. In this work initially, we have converted the given RGB image into HSV color image. Subsequently, we have considered H (hue), S (saturation), and V (intensity) components for extraction of visual image features. The texture features have been extracted from the V component of the image using 2D dual-tree complex wavelet transform (2D DT-CWT) where it analyzes the textural patterns in six different directions i.e. ±15\(^{\circ }\), ±45\(^{\circ }\), and ±75\(^{\circ }\). At the same time, we have computed the probability histograms of H and S components of the image respectively and subsequently those are divided into non-uniform bins based on cumulative probability for extraction of color based features. So, in this work both the color and texture features have been extracted simultaneously. Finally, the obtained features have been concatenated to attain the final feature vector and same is considered in image retrieval process. We have tested the novelty and performance of the proposed work in two Corel, two objects, and, a texture image datasets. The experimental results reveal the acceptable retrieval performances for different types of datasets.

Investigation of acoustic and visual features for pig cough classification

Front-end feature tracking based on vision is the process in which a robot captures images of its surrounding environment using a camera while in motion. Each frame of the image is then analyzed to extract feature points, which are subsequently matched between pairwise frames to estimate the robot’s pose changes by solving for the variations in these points. While feature matching methods that rely on descriptor-based approaches perform well in cases of significant lighting and texture variations, the addition of descriptors increases computational cost and introduces instability. Therefore, in this paper, a novel approach is proposed that combines sparse optical flow tracking with Shi-Tomasi corner detection, replacing the use of descriptors. This new method offers improved stability in situations of challenging lighting and texture variations while maintaining lower computational cost. Experimental results, validated using the OpenCV library on the Ubuntu operating system, demonstrate the algorithm's effectiveness and efficiency.

Image captioning generates text that describes scenes from input images. It has been developed for high-quality images taken in clear weather. However, in bad weather conditions, such as heavy rain, snow, and dense fog, poor visibility as a result of rain streaks, rain accumulation, and snowflakes causes a serious degradation of image quality. This hinders the extraction of useful visual features and results in deteriorated image captioning performance. To address practical issues, this study introduces a new encoder for captioning heavy rain images. The central idea is to transform output features extracted from heavy rain input images into semantic visual features associated with words and sentence context. To achieve this, a target encoder is initially trained in an encoder-decoder framework to associate visual features with semantic words. Subsequently, the objects in a heavy rain image are rendered visible by using an initial reconstruction subnetwork (IRS) based on a heavy rain model. The IRS is then combined with another semantic visual feature matching subnetwork (SVFMS) to match the output features of the IRS with the semantic visual features of the pretrained target encoder. The proposed encoder is based on the joint learning of the IRS and SVFMS. It is trained in an end-to-end manner, and then connected to the pretrained decoder for image captioning. It is experimentally demonstrated that the proposed encoder can generate semantic visual features associated with words even from heavy rain images, thereby increasing the accuracy of the generated captions.

High dimensional visual features and multimodal based search systems are popular due to their visual retrieval accuracy. But high dimensionality and multimodality negatively affect efficiency and computational complexity. This paper brings the performance advantages of low dimensional visual feature called Compacted Dither Pattern Code (CDPC). CDPC is accompanied with texel properties, spatial colour arrangements, high compactness, and robust feature representation. CDPC feature uses a low computational cost in visual feature extraction which is illustrated using four abstract stages. This feature has proven its performances in our experimental study. In our experiment, CDPC was used alone as a syntactic visual feature for semantic visual depiction and retrieval. Ten irregular shapes based visual concepts were used in visual description. The average precision results generated from ranked retrieval set were used as a metric in evaluation. The retrieval result based on Trecvid 2005 dataset is presented compared to well known low dimensional and high dimensional visual features. Results show the balanced made by CDPC with Bhattacharyya classification in between retrieval accuracy and visual feature space reduction. Competitive advantages of this method over state of art systems are also technically described.