Monitoring and Mining Animal Sounds in Visual Space

We investigated whether a deep neural network (DNN)-based source enhancement function can be self-optimized by reinforcement learning (RL). The use of a DNN is a powerful approach to describing the relationship between two sets of variables and can be useful for source enhancement function design. By training the DNN using a huge amount of training data, sound quality of output signals are improved. However, collecting a huge amount of training data is often difficult in practice. To use limited training data efficiently, we focus on the "self-optimization" of DNN-based source enhancement function in which RL is commonly utilized in the development of game playing computers. As a reward for RL, quantitative metrics that reflect a human's perceptual score (perceptual score), e.g., perceptual evaluation methods for audio source separation (PEASS), are utilized. To investigate whether the sound quality is improved by RL-based source enhancement, subjective tests were conducted. It was confirmed that the output sound quality of the RL-based source enhancement function improved as the number of iterations was increased and finally outperformed the conventional method.

To interact naturally and achieve mutual sympathy between humans and machines, emotion recognition is one of the most important function to realize advanced human-computer interaction devices. Due to the high correlation between emotion and involuntary physiological changes, physiological signals are a prime candidate for emotion analysis. However, due to the need of a huge amount of training data for a high-quality machine learning model, computational complexity becomes a major bottleneck. To overcome this issue, brain-inspired hyperdimensional (HD) computing, an energy-efficient and fast learning computational paradigm, has a high potential to achieve a balance between accuracy and the amount of necessary training data. We propose an HD Computing-based Multimodality Emotion Recognition (HDC-MER). HDCMER maps real-valued features to binary HD vectors using a random nonlinear function, and further encodes them over time, and fuses across different modalities including GSR, ECG, and EEG. The experimental results show that, compared to the best method using the full training data, HDC-MER achieves higher classification accuracy for both valence (83.2% vs. 80.1%) and arousal (70.1% vs. 68.4%) using only 1/4 training data. HDC-MER also achieves at least 5% higher averaged accuracy compared to all the other methods in any point along the learning curve.

Use of autonomous vehicles aims to eventually reduce the number of motor vehicle fatalities caused by humans. Deep learning plays an important role in making this possible because it can leverage the huge amount of training data that comes from autonomous car sensors. Automatic recognition of traffic light and vehicle signal is a perception module critical to autonomous vehicles because a deadly car accident could happen if a vehicle fails to follow traffic lights or vehicle signals. A practical Traffic Light Recognition (TLR) or Vehicle Signal Recognition (VSR) faces some challenges, including varying illumination conditions, false positives and long computation time. In this chapter, we propose a novel approach to recognize Traffic Light (TL) and Vehicle Signal (VS) with high dynamic range imaging and deep learning in real-time. Different from existing approaches which use only bright images, we use both high exposure/bright and low exposure/dark images provided by a high dynamic range camera. TL candidates can be detected robustly from low exposure/dark frames because they have a clean dark background. The TL candidates on the consecutive high exposure/bright frames are then classified accurately using a convolutional neural network. The dual-channel mechanism can achieve promising results because it uses undistorted color and shape information of low exposure/dark frames as well as rich texture of high exposure/bright frames. Furthermore, the TLR performance is boosted by incorporating a temporal trajectory tracking method. To speed up the process, a region of interest is generated to reduce the search regions for the TL candidates. The experimental results on a large dual-channel database have shown that our dual-channel approach outperforms the state of the art which uses only bright images. Encouraged by the promising performance of the TLR, we extend the dual-channel approach to vehicle signal recognition. The algorithm reported in this chapter has been integrated into our autonomous vehicle via Data Distribute Service (DDS) and works robustly in real roads.

We present a supervised deep learning method to automatically segment 3D volumes of biomedical image data. The presented method takes advantage of a neural network with the main layers consisting of multi-dimensional gated recurrent units. We apply an on-the-fly data augmentation technique which allows for accurate estimations without the need for either a huge amount of training data or advanced data pre- or postprocessing. We show that our method performs amongst the leading techniques on a popular brain segmentation challenge dataset in terms of speed, accuracy and memory efficiency. We describe in detail advantages over a similar method which uses the well-established long short-term memory.

This paper introduces an approach to employ deep features for person re-identification. In contrast to existing works, we focus on using pre-trained deep models and their concept-based output to enhance attribute presentations of person images. There are two main contributions. First, we investigate recent state-of-the-art deep learning models for the task and provide a comprehensive evaluation. Second, we present an approach to improve identification accuracy of a standard attribute-based person re-identification approach. By using pre-trained models, we avoid re-training new deep models which always require a huge amount of training data and high computational cost. The idea is to utilize the correlation between generic concepts learned by the deep models and specific pre-defined attributes commonly used for person re-identification. We employ the deeply learned features from generic concepts to represent person images. These images and their manually annotated attributes are then used to train attribute classifiers. Given the classifiers, attributes of the probe and gallery images can be automatically extracted to compose attribute-based feature vectors. Re-identification is done by matching the vectors. Experiments conducted on several benchmark datasets demonstrated the effectiveness of the proposed approach.

Airplane detection from satellite imagery is a challenging task due to the complex backgrounds in the images and differences in data acquisition conditions caused by the sensor geometry and atmospheric effects. Deep learning methods provide reliable and accurate solutions for automatic detection of airplanes; however, huge amount of training data is required to obtain promising results. In this study, we create a novel airplane detection dataset called High Resolution Planes (HRPlanes) by using images from Google Earth (GE) and labeling the bounding box of each plane on the images. HRPlanes include GE images of several different airports across the world to represent a variety of landscape, seasonal and satellite geometry conditions obtained from different satellites. We evaluated our dataset with two widely used object detection methods namely YOLOv4 and Faster R-CNN. Our preliminary results show that the proposed dataset can be a valuable data source and benchmark data set for future applications. Moreover, proposed architectures and results of this study could be used for transfer learning of different datasets and models for airplane detection.

In this paper, we propose a novel method of online learning of sparse pseudo-data, representative of the whole training data, for Gaussian Process (GP) regressions. We call the proposed method Incremental Sparse Pseudo-input Gaussian Process (ISPGP) regression. The proposed ISPGP algorithm allows for training from either a huge amount of training data by scanning through it only once or an online incremental training dataset. Thanks to the nature of the incremental learning algorithm, the proposed ISPGP algorithm can theoretically work with infinite data to which the conventional GP or SPGP algorithm is not applicable. From our experimental results on the KIN40K dataset, we can see that the proposed ISPGP algorithm is comparable to the conventional GP algorithm using the same number of training data. Although the proposed ISPGP algorithm performs slightly worse than Snelson and Ghahramani's SPGP algorithm, the level of performance degradation is acceptable.

Many classification/segmentation tasks in medical imaging are particularly challenging for machine learning algorithms because of the huge amount of training data required to cover biological variability. Learning methods scaling badly in the number of training data points may not be applicable. This may exclude powerful classifiers with good generalization performance such as standard non-linear support vector machines (SVMs). Further, many medical imaging problems have highly imbalanced class populations, because the object to be segmented has only few pixels/voxels compared to the background. This article presents a two-stage classifier for large-scale medical imaging problems. In the first stage, a classifier that is easily trainable on large data sets is employed. The class imbalance is exploited and the classifier is adjusted to correctly detect background with a very high accuracy. Only the comparatively few data points not identified as background are passed to the second stage. Here a powerful classifier with high training time complexity can be employed for making the final decision whether a data point belongs to the object or not. We applied our method to the problem of automatically segmenting tibial articular cartilage from knee MRI scans. We show that by using nearest neighbor (kNN) in the first stage we can reduce the amount of data for training a non-linear SVM in the second stage. The cascaded system achieves better results than the state-of-the-art method relying on a single kNN classifier.

Rotating machinery is typical complex electromechanical equipment under non-stationary working conditions and hazardous environments, which is thus vulnerable to an unexpected fault. The currently wide-applied deep learning-based fault diagnosis methods for rotating machines require a huge amount of training data and can only extract features from independent samples. In this study, a Semi-Supervised Self-Correcting Graph Neural Network (SSGNN) is proposed for fault diagnosis, which effectively extracts features from vibrational signals and generates a graph-structured representation of fault knowledge. First, a preprocessing layer is introduced to mine the propinquity of vibration samples and construct a graph, where circle loss is applied to enhance convergence. Second, the state transform algorithm is improved by proposing a graph convolutional layer for efficiently implementing state propagation. Finally, an alternate learning method is proposed based on the expectation maximization (EM) algorithm, in which the feature extraction network and graph structure are optimized alternately to reduce the training complexity. The proposed method is validated through experiments conducted on a publicly available dataset and a dynamic condition dataset obtained from a drivetrain dynamic simulator. The results show that the proposed method has higher accuracy and faster convergence speed compared to the state-of-the-art methods.

Histopathological image analysis remains at the forefront of computational pathology presenting numerous challenges and demanding tasks, primarily due to the complex nature of tissue structures and the extensive scale of whole slide images (WSIs). Deep learning models have been widely used in histopathology image analysis, especially convolutional neural network (CNN)-based models for classification. However, CNNs have certain limitations due to their small receptive field. Recent works employed adaptations of the classical transformer architecture to visual data [1] [2]. Models such as Vision Transformer (ViT) and Swin Transformer leverage the powerful multi-head self-attention mechanism and have demonstrated comparable or superior performance to state-of-the-art CNN-based classification models. Despite their successes, these models require huge amounts of training data to effectively learn representations as they lack the inherent inductive biases of CNNs. This work compares Vision Transformers with baseline CNN models using a breast cancer histopathological dataset. Further, we employ a novel knowledge-distillation approach to enhance the learning efficiency of vits, When trained with a limited amount of data, Unlike previous works, we aimed to minimize convolution operations when generating patch embeddings to preserve spatial information before reaching the transformer attention layers, we achieved an accuracy of 87.7% for the ViT-base trained as a student of ResNet50, which represents a 1.2% improvement in accuracy over the standalone ViT-base. [3]

Deep Neural Networks (DNNs) with sparse input features have been widely used in recommender systems in industry. These models have large memory requirements and need a huge amount of training data. The large model size usually entails a cost, in the range of millions of dollars, for storage and communication with the inference services. In this paper, we propose a hybrid hashing method to combine frequency hashing and double hashing techniques for model size reduction, without compromising performance. We evaluate the proposed models on two product surfaces. In both cases, experiment results demonstrated that we can reduce the model size by around 90 while keeping the performance on par with the original baselines.

Existing deep learning based visual servoing approaches regress the relative camera pose between a pair of images. Therefore, they require a huge amount of training data and sometimes fine-tuning for adaptation to a novel scene. Furthermore, current approaches do not consider underlying geometry of the scene and rely on direct estimation of camera pose. Thus, inaccuracies in prediction of the camera pose, especially for distant goals, lead to a degradation in the servoing performance. In this paper, we propose a two-fold solution: (i) We consider optical flow as our visual features, which are predicted using a deep neural network. (ii) These flow features are then systematically integrated with depth estimates provided by another neural network using interaction matrix. We further present an extensive benchmark in a photo-realistic 3D simulation across diverse scenes to study the convergence and generalisation of visual servoing approaches. We show convergence for over 3m and 40 degrees while maintaining precise positioning of under 2cm and 1 degree on our challenging benchmark where the existing approaches that are unable to converge for majority of scenarios for over 1.5m and 20 degrees. Furthermore, we also evaluate our approach for a real scenario on an aerial robot. Our approach generalizes to novel scenarios producing precise and robust servoing performance for 6 degrees of freedom positioning tasks with even large camera transformations without any retraining or fine-tuning.

Though the classical robotics is highly proficient in accomplishing a lot of complex tasks, still it is far from exhibiting the human-like natural intelligence in terms of flexibility and reliability to work in dynamic scenarios. In order to render these qualities in the robots, reinforcement learning could prove to be quite effective. By employing learning based training provided by reinforcement learning methods, a robot can be made to learn to work in previously unforeseen situations. Still this learning task can be quite cumbersome due to its requirement of the huge amount of training data which makes the training quite inefficient in the real world scenarios. The paper proposes a model based path planning method using the ε greedy based Q-learning. The scenario was modeled using a grid-world based simulator which is being used in the initial training of the agent. The trained policy is then improved to learn the real world dynamics by using the real world samples. This study proves the efficiency and reliability of the simulator-based training methodology.

Personalization related research has been conducted for decades with the objective of investigating personal-level effectiveness in irregular or abnormal cases among many types of domains. However, the activity recognition community generally investigates one-fits-all model, i.e., a single recognition model for all people and suffers from poor performance generalization or a huge amount of training data for high generalization. In this paper, we propose Compatibility-based classifier personalization (CbCP) as a subject-dependent activity recognition method that uses the classifier with the highest compatibility (similarity) with a particular user. Furthermore, we explore the effectiveness of classifier personalization through a particular group of activities, which results in a hierarchical recognition model. We present a comparative evaluation of 1) the traditional one-fits-all classifier vs. CbCP models and 2) compatibility evaluation through all activities (non-hierarchical classification) vs. a group of activities (hierarchical classification). The results of the two public datasets imply the effectiveness of compatibility-based approach for a hierarchical classifier formation.

Machine learning algorithms (MLAs) are increasingly being used as effective techniques for processing vibration signals obtained from complex industrial machineries. Previous applications of automatic fault detection algorithms in the diagnosis of rotating machines were mainly based on historical operating data sets, limiting the diagnostic reliability to devices with an extended operating history. Moreover, physically collected data are often restricted by the conditions of acquisition and the specific elements for which they were recorded. Digital twin (DT) provides a powerful tool able to generate a huge amount of training data for MLAs. However, the DT model must be accurate enough to substitute the experiments. This work aims to escape the experience requirement by using a simulation-driven MLA based on the multifactorial analysis of fault indicators associated with a DT. To achieve this approach, a numerical model of a rotor-ball bearing system is developed. The latter is updated according to a parameter update scheme based on a comparison between the relevant features of the experimentally measured signals and the signals simulated by the model. These features are chosen as the selected input parameters of the MLA classifier. The results show that after updating, the developed DT has provided a reliable diagnostic with an adaptive degradation analysis, which makes the simulated data suitable for the construction of a machine learning predictive model. Two common MLAs, (multi-kernel support vector machine) and (k nearest neighbor’s algorithm), were trained using the simulated data and validated later against experimental datasets.