Cross-Modality Feature Learning via Convolutional Autoencoder

Efficient training of supervised deep learning models for semantic segmentation requires a massive volume of annotated data. In this paper, we propose an unsupervised semantic segmentation method through the application of a radial transform method in the polar coordinate system to unannotated images. This method generates radial transformed images up to the number of pixels in the input image. Each generated image corresponds to a pixel in the original image, which is a spatial representation of the selected pixel with respect to other pixels, in the polar coordinate system. A dimension reduction method, such as a convolutional autoencoder (CAE), can extract features of these representations for clustering and later, labeling the original image by mapping the labels back to the original image. The advantage of the proposed radial transform technique is that it generates a massive number of training images by sampling pixels in the polar coordinate system from a very limited number of original images in the Cartesian coordinate system. The proposed approach achieved 88.20% accuracy in pixel-level segmentation of left kidney, right kidney, and non-kidney pixels in contrast-enhanced computed tomography (CT) images.

MRI pulse sequences and imaging parameters substantially influence the variation of MRI radiomics features, thus impose a critical challenge on MRI radiomics reproducibility and reliability. This study aims to prospectively investigate the impact of various imaging parameters on MRI radiomics features in a 3D T2-weighted (T2W) turbo-spin-echo (TSE) pulse sequence for MR-guided-radiotherapy (MRgRT). An anthropomorphic phantom was scanned using a 3D-T2W-TSE MRgRT sequence at 1.5T under a variety of acquisition imaging parameter changes. T1 and T2 relaxation times of the phantom were also measured. 93 first-order and texture radiomics features in the original and 14 transformed images, yielding 1,395 features in total, were extracted from 10 volumes-of-interest (VOIs). The percentage deviation (d%) of radiomics feature values from the baseline values and intra-class correlation coefficient (ICC) with the baseline were calculated. Robust radiomics features were identified based on the excellent agreement of radiomics feature values with the baseline, i.e., the averaged d% <5% and ICC >0.90 in all VOIs for all imaging parameter variations. The radiomics feature values changed considerably but to different degrees with different imaging parameter adjustments, in the ten VOIs. The deviation d% ranged from 0.02% to 321.3%, with a mean of 12.5% averaged for all original features in all ten VOIs. First-order and GLCM features were generally more robust to imaging parameters than other features in the original images. There were also significantly different radiomics feature values (ANOVA, P<0.001) between the original and the transformed images, exhibiting quite different robustness to imaging parameters. 330 out of 1395 features (23.7%) robust to imaging parameters were identified. GLCM and GLSZM features had the most (42.5%, 153/360) and least (3.8%, 9/240) robust features in the original and transformed images, respectively. This study helps better understand the quantitative dependence of radiomics feature values on imaging parameters in a 3D-T2W-TSE sequence for MRgRT. Imaging parameter heterogeneity should be considered as a significant source of radiomics variability and uncertainty, which must be well harmonized for reliable clinical use. The identified robust features to imaging parameters are helpful for the pre-selection of radiomics features for reliable radiomics modeling.

There are many applications, for example image copyright protection, where transformed images of a given test image need to be identified. The solution to the identification problem consists of two main stages. In stage one, certain representative features are detected for all images. In stage two, the representative features of the test image and the stored images are compared to identify the transformed images for the test image. We have reported the technique to extract robust representative features – corners – in our previous work [1]. This paper will focus on our stage-two work on effective corner matching technique for transformed image identification. Experimental results show that the proposed corner matching technique is very much effective in identifying the transformed images for a given test image.

Activity and action detection, tracking and recognition are very demanding research area in computer vision and human computer interaction. In this paper, a video-based novel approach for human activity recognition is presented using robust hybrid features and embedded Hidden Markov Models. In the proposed HAR framework, depth maps are analyzed by temporal motion identification method to segment human silhouettes from noisy background and compute depth silhouette area for each activity to track human movements in a scene. Several representative features, including invariant, depth sequential silhouettes and spatiotemporal body joints features were fused together to explore gradient orientation change, intensity differentiation, temporal variation and local motion of specific body parts. Then, these features are processed by the dynamics of their respective class and learned, trained and recognized with specific embedded HMM having active feature values. Our experiments on two depth datasets demonstrate that the proposed features are efficient and robust over the state of the arts features for human activity recognition especially when there are similar postures of different activities.

Nowadays cross-media retrieval is an useful technology that helps people find expected information from the huge amount of multimodal data more efficiently. A common cross-media retrieval framework is first to map features of different modalities into an isomorphic semantic space so that the similarity between heterogeneous data can be measured. For most of semantic space based methods, the mapping mechanism from original to semantic space of each modality is optimized independently, yet the more discriminative characteristic of a certain modality is not taken into account. In this paper, we propose a deep framework which introduces a latent embedding layer to learn joint parameters to obtain semantically meaningful representations of images and texts. Specifically, the discriminative characteristic embedded in the textual modality can be transferred to images through the latent embedding layer and joint parameters to enhance the consistency between semantic representations. Extensive experiments on the three popular publicly available datasets well demonstrate the superiority of the proposed method, which achieves the new state-of-the-arts.

Multimedia information have strong temporal correlations that shape the way modalities co-occur over time. In this paper we study the dynamic nature of multimedia and social-media information, where the temporal dimension emerges as a strong source of evidence for learning the temporal correlations across visual and textual modalities. So far, cross-media retrieval models, explored the correlations between different modalities (e.g. text and image) to learn a common subspace, in which semantically similar instances lie in the same neighbourhood. Building on such knowledge, we propose a novel temporal cross-media neural architecture, that departs from standard cross-media methods, by explicitly accounting for the temporal dimension through temporal subspace learning. The model is softly-constrained with temporal and inter-modality constraints that guide the new subspace learning task by favouring temporal correlations between semantically similar and temporally close instances. Experiments on three distinct datasets show that accounting for time turns out to be important for cross-media retrieval. Namely, the proposed method outperforms a set of baselines on the task of temporal cross-media retrieval, demonstrating its effectiveness for performing temporal subspace learning.

A comprehensive comparison of handcrafted features and convolutional autoencoders for epileptic seizures detection in EEG signals

In this paper, a novel black box adversarial computer vision attack is proposed. The introduced attack is based on removing from images some components described by their Tchebichef discrete orthogonal moments, rather than to perturb them. The contribution of this work is focused on the addition of one more clue, supporting the critical hypothesis that computer vision systems fail because they support their decisions not only in robust features but also in others non-robust ones. In this, context non-robust image features described in terms of Tchebichef moments are excluded from the original images and the approximated reconstructed versions of them are used as adversarial examples in order to attack some popular deep learning models. The experiments justify the effectiveness of the proposed adversarial attack in terms of imperceptibility and recognition error rate of the deep learning classifiers. It is worth noting that the top-1 accuracy of the attacked models was degraded by a factor between 9.48%-70.89% for adversarial images of 65dB to 57dB PSNR values. The corresponding degradation of the top-5 models’ accuracy was between 6.9% and 55.14% for the same quality images. Moreover, the proposed attack seems to have more strength than the Fast Gradient Sign Method (FGSM) attacking method traditionally applying in most cases. These results reveal that the proposed attack is able to exploit the vulnerability of the deep learning models’ towards degrading their generalization abilities. In this paper, a novel black box adversarial computer vision attack is proposed. The introduced attack is based on removing from images some components described by their Tchebichef discrete orthogonal moments, rather than to perturb them. The contribution of this work is focused on the addition of one more clue, supporting the critical hypothesis that computer vision systems fail because they support their decisions not only in robust features but also in others non-robust ones. In this, context non-robust image features described in terms of Tchebichef moments are excluded from the original images and the approximated reconstructed versions of them are used as adversarial examples in order to attack some popular deep learning models. The experiments justify the effectiveness of the proposed adversarial attack in terms of imperceptibility and recognition error rate of the deep learning classifiers. It is worth noting that the top-1 accuracy of the attacked models was degraded by a factor between 9.48%-70.89% for adversarial images of 65dB to 57dB PSNR values. The corresponding degradation of the top-5 models’ accuracy was between 6.9% and 55.14% for the same quality images. Moreover, the proposed attack seems to have more strength than the Fast Gradient Sign Method (FGSM) attacking method traditionally applying in most cases. These results reveal that the proposed attack is able to exploit the vulnerability of the deep learning models’ towards degrading their generalization abilities.

The focus of this paper is to illustrate how computational image processing and machine learning can help address two of the challenges of histological image analysis, namely, the cellular heterogeneity, and the imprecise labeling. We propose an unsupervised method of generating representative image signatures based on an autoencoder architecture which reduces the dependency on labels that tend to be imprecise and tedious to get. We have modified and enhanced the architecture to simultaneously produce representative image features as well as perform dictionary learning on these features to enable robust characterization of the cellular phenotypes. We integrate the extracted features in a disease grading framework, test it in prostate tissues immunostained for different protein visualization and show significant improvement in terms of grading accuracy compared to alternative supervised feature-extraction methods.

Multi-task learning (MTL) and attention mechanism have been proven to effectively extract robust acoustic features for various speech-related tasks in noisy environments. In this study, we propose an attention-based MTL (ATM) approach that integrates MTL and the attention-weighting mechanism to simultaneously realize a multi-model learning structure that performs speech enhancement (SE) and speaker identification (SI). The proposed ATM system consists of three parts: SE, SI, and attention-Net (AttNet). The SE part is composed of a long-short-term memory (LSTM) model, and a deep neural network (DNN) model is used to develop the SI and AttNet parts. The overall ATM system first extracts the representative features and then enhances the speech signals in LSTM-SE and specifies speaker identity in DNN-SI. The AttNet computes weights based on DNN-SI to prepare better representative features for LSTM-SE. We tested the proposed ATM system on Taiwan Mandarin hearing in noise test sentences. The evaluation results confirmed that the proposed system can effectively enhance speech quality and intelligibility of a given noisy input. Moreover, the accuracy of the SI can also be notably improved by using the proposed ATM system.

In order to make the robot obtain the optimal action directly from the original visual perception without any hand-crafted features and features matching, a novel end-to-end path planning method-mobile robot path planning using deep reinforcement learning is proposed. Firstly, a deep Q-network (DQN) is designed and trained to approximate the mobile robot state-action value function. Then, the Q value corresponding to each possible mobile robot action (i.e., turn left, turn right, forward) is determined by the well trained DQN, here, the input of the DQN is the original RGB image (image pixels) captured from the environment without any hand-crafted features and features matching; Finally, the current optimal mobile robot action is selected by the action selection strategy. Mobile robot reach to the goal point while avoiding obstacles ultimately. 30 times path planning experiments are conducted in the seekavoid_arena_01 environment on DeepMind Lab platform. The experimental results show that our deep reinforcement learning based robot path planning method is an effective end-to-end mobile robot path planning method.

During complex interventions, patient tracking is needed for optimal motion compensation, in order to guide the physician in a minimally invasive way. Nowadays, optical tracking systems are used for tracking of markers. Despite the unobtrusiveness of this technology, the approach is cumbersome because it requires manual placement of markers, which can alter during the operations due to the presence of liquids. To improve the clinical workflow, a new feature tracking algorithm is designed, involving feature detection and tracking without optical markers. The new markers are created with the multispectral imaging. Maximally stable extremal regions (MSER) and Speeded Up Robust Feature (SURF) methods are applied to design features and track natural landmarks, e.g. moles and veins. Both methods are tested and compared in accuracy with the mean shift tracking method. SURF reaches the highest accuracy of 0.257 pixels for images at 430 nm and 0.562 pixels for images at 970 nm. This study shows that incorporating multispectral imaging in the surgical scenario leads to an attractive benefit minimizing the risk of marker obstruction and displacement.

Biometrics identification using multiple modalities has attracted the attention of many researchers as it produces more robust and trustworthy results than single modality biometrics. In this paper, we present a novel multimodal recognition system that trains a deep learning network to automatically learn features after extracting multiple biometric modalities from a single data source, i.e., facial video clips. Utilizing different modalities, i.e., left ear, left profile face, frontal face, right profile face, and right ear, present in the facial video clips, we train supervised denoising auto-encoders to automatically extract robust and non-redundant features. The automatically learned features are then used to train modality specific sparse classifiers to perform the multimodal recognition. Moreover, the proposed technique has proven robust when some of the above modalities were missing during the testing. The proposed system has three main components that are responsible for detection, which consists of modality specific detectors to automatically detect images of different modalities present in facial video clips; feature selection, which uses supervised denoising sparse auto-encoders network to capture discriminative representations that are robust to the illumination and pose variations; and classification, which consists of a set of modality specific sparse representation classifiers for unimodal recognition, followed by score level fusion of the recognition results of the available modalities. Experiments conducted on the constrained facial video dataset (WVU) and the unconstrained facial video dataset (HONDA/UCSD), resulted in a 99.17% and 97.14% Rank-1 recognition rates, respectively. The multimodal recognition accuracy demonstrates the superiority and robustness of the proposed approach irrespective of the illumination, nonplanar movement, and pose variations present in the video clips even in the situation of missing modalities.

Steganography is the method of providing Computer security in which hiding the required information is done by inserting messages within other messages, which is a string of characters containing the useful information, in a carrier image. Using this technique, the required information from the secret image is embedded into individual rows as well as columns present in the pixels of carrier image. In this paper, a novel fuzzy logic based technique is proposed to hide the secret message in individual rows and in individual columns of pixels of the carrier image and to extract the hidden message in the same carrier image. The fuzzification process transforms the image in to various bitplanes. Pixel number and Correlation Value is computed in the original image for hiding the secret information in to the original image. The pixel number and Correlation value is also used as the key for retrieving the embedded image from the receiver side. Pixel merging is done in the sender side by assigning a steganographic value of white and black pixels in original image based on the fuzzy rules by comparing the pixels present in the original and secret images. The information which is hided can be retrieved by using the same fuzzy rules. Experimental results show that the proposed method can hide and retrieve the secret and important messages in an image more effectively and accurately.

Extracting discriminative and robust features from video sequences is the first and most critical step in human action recognition. In this paper, instead of using handcrafted features, we automatically learn spatio-temporal motion features for action recognition. This is achieved via an evolutionary method, i.e., genetic programming (GP), which evolves the motion feature descriptor on a population of primitive 3D operators (e.g., 3D-Gabor and wavelet). In this way, the scale and shift invariant features can be effectively extracted from both color and optical flow sequences. We intend to learn data adaptive descriptors for different datasets with multiple layers, which makes fully use of the knowledge to mimic the physical structure of the human visual cortex for action recognition and simultaneously reduce the GP searching space to effectively accelerate the convergence of optimal solutions. In our evolutionary architecture, the average cross-validation classification error, which is calculated by an support-vector-machine classifier on the training set, is adopted as the evaluation criterion for the GP fitness function. After the entire evolution procedure finishes, the best-so-far solution selected by GP is regarded as the (near-)optimal action descriptor obtained. The GP-evolving feature extraction method is evaluated on four popular action datasets, namely KTH, HMDB51, UCF YouTube, and Hollywood2. Experimental results show that our method significantly outperforms other types of features, either hand-designed or machine-learned.