Simultaneous Image Reconstruction and Feature Learning with 3D-CNNs for Image Set–Based Classification

The temporal and spatial information of electroencephalogram (EEG) are essential for the emotion recognition model to learn the discriminative features. Hence, we propose a novel hybrid spatial-temporal feature fusion neural network (STFFNN) to extract the discriminative features and integrate complementary information. The generated power topographic maps, which capture dependencies among the electrodes, are fed to convolutional neural network (CNN) for spatial feature learning. Furthermore, instance normalizations (INs) and batch normalizations (BNs) within the CNN are appropriately combined to alleviate the individual difference and preserve the domain-invariant information. Meanwhile, a feedforward network is adopted for temporal feature learning. Due to the high dimensionality of EEG features, we propose a grid-search-based configurational optimization method to robustly reduce the dimensionality. Finally, inspired by the multimodal fusion strategies that leverage the complementarity of data to obtain more robust predictions, we utilize a bidirectional long short-term memory (Bi-LSTM) network for temporal and spatial feature fusion. To validate the effectiveness of the proposed method, the tenfold cross-validation experiments and subject-dependent experiments are both conducted on the DEAP database. The experimental results demonstrate that the proposed method achieves outstanding performance in emotion recognition with arousal and valence level.

When using convolutional neural networks (CNNs) for segmentation of organs and lesions in medical images, the conventional approach is to work with inputs and outputs either as single slice [two-dimensional (2D)] or whole volumes [three-dimensional (3D)]. One common alternative, in this study denoted as pseudo-3D, is to use a stack of adjacent slices as input and produce a prediction for at least the central slice. This approach gives the network the possibility to capture 3D spatial information, with only a minor additional computational cost. In this study, we systematically evaluate the segmentation performance and computational costs of this pseudo-3D approach as a function of the number of input slices, and compare the results to conventional end-to-end 2D and 3D CNNs, and to triplanar orthogonal 2D CNNs. The standard pseudo-3D method regards the neighboring slices as multiple input image channels. We additionally design and evaluate a novel, simple approach where the input stack is a volumetric input that is repeatably convolved in 3D to obtain a 2D feature map. This 2D map is in turn fed into a standard 2D network. We conducted experiments using two different CNN backbone architectures and on eight diverse data sets covering different anatomical regions, imaging modalities, and segmentation tasks. We found that while both pseudo-3D methods can process a large number of slices at once and still be computationally much more efficient than fully 3D CNNs, a significant improvement over a regular 2D CNN was only observed with two of the eight data sets. triplanar networks had the poorest performance of all the evaluated models. An analysis of the structural properties of the segmentation masks revealed no relations to the segmentation performance with respect to the number of input slices. A post hoc rank sum test which combined all metrics and data sets yielded that only our newly proposed pseudo-3D method with an input size of 13 slices outperformed almost all methods. In the general case, multislice inputs appear not to improve segmentation results over using 2D or 3D CNNs. For the particular case of 13 input slices, the proposed novel pseudo-3D method does appear to have a slight advantage across all data sets compared to all other methods evaluated in this work.

A semantic RBM-based model for image set classification

Action recognition is an active research field that aims to recognize human actions and intentions from a series of observations of human behavior and the environment. Unlike image-based action recognition mainly using a two-dimensional (2D) convolutional neural network (CNN), one of the difficulties in video-based action recognition is that video action behavior should be able to characterize both short-term small movements and long-term temporal appearance information. Previous methods aim at analyzing video action behavior only using a basic framework of 3D CNN. However, these approaches have a limitation on analyzing fast action movements or abruptly appearing objects because of the limited coverage of convolutional filter. In this paper, we propose the aggregation of squeeze-and-excitation (SE) and self-attention (SA) modules with 3D CNN to analyze both short and long-term temporal action behavior efficiently. We successfully implemented SE and SA modules to present a novel approach to video action recognition that builds upon the current state-of-the-art methods and demonstrates better performance with UCF-101 and HMDB51 datasets. For example, we get accuracies of 92.5% (16f-clip) and 95.6% (64f-clip) with the UCF-101 dataset, and 68.1% (16f-clip) and 74.1% (64f-clip) with HMDB51 for the ResNext-101 architecture in a 3D CNN.

Over large areas, land cover classification has conventionally been undertaken using satellite time series. Typically temporal metric percentiles derived from single pixel location time series have been used to take advantage of spectral differences among land cover classes over time and to minimize the impact of missing observations. Deep convolutional neural networks (CNNs) have demonstrated potential for land cover classification of single date images. However, over large areas and using time series their application is complicated because they are sensitive to missing observations and they may misclassify small and spatially fragmented surface features due to their spatial patch-based implementation. This study demonstrates, for the first time, a one-dimensional (1D) CNN single pixel time series land classification approach that uses temporal percentile metrics and that does not have these issues. This is demonstrated for all the Conterminous United States (CONUS) considering two different 1D CNN structures with 5 and 8 layers, respectively. CONUS 30 m land cover classifications were derived using all the available Landsat-5 and -7 imagery over a seven-month growing season in 2011 with 3.3 million 30 m land cover class labelled samples extracted from the contemporaneous CONUS National Land Cover Database (NLCD) 16 class land cover product. The 1D CNNs and, a conventional random forest model, were trained using 10%, 50% and 90% samples, and the classification accuracies were evaluated with an independent 10% proportion. Temporal metrics were classified using 5, 7 and 9 percentiles for each of five Landsat reflective wavelength bands and their eight band ratios. The CONUS and detailed 150 × 150 km classification results demonstrate that the approach is effective at scale and locally. The 1D CNN classification land cover class boundaries were preserved for small axis dimension features, such as roads and rivers, with no stripes or anomalous spatial patterns. The 8-layer 1D CNN provided the highest overall classification accuracies and both the 5-layer and 8-layer 1D CNN architectures provided higher accuracies than the random forest by 1.9% - 2.8% which as all the accuracies were > 83% is a meaningful increase. The CONUS overall classification accuracies increased marginally with the number of percentiles (86.21%, 86.40%, and 86.43% for 5, 7 and 9 percentiles, respectively) using the 8-layer 1D-CNN. Class specific producer and user accuracies were quantified, with lower accuracies for the developed land, crop and pasture/hay classes, but no systematic pattern among classes with respect to the number of temporal percentiles used. Application of the trained model to a different year of CONUS Landsat ARD showed moderately decreased accuracy (80.79% for 7 percentiles) that we illustrate is likely due to different intra-annual surface variations between years. These encouraging results are discussed with recommended research for deep learning using temporal metric percentiles.

Recently, three dimensional (3D) convolutional neural networks (CNNs) have emerged as dominant methods to capture spatiotemporal representations in videos, by adding to pre-existing 2D CNNs a third, temporal dimension. Such 3D CNNs, however, are anti-causal (i.e., they exploit information from both the past and the future frames to produce feature representations, thus preventing their use in online settings), constrain the temporal reasoning horizon to the size of the temporal convolution kernel, and are not temporal resolution-preserving for video sequence-to-sequence modelling, as, for instance, in action detection. To address these serious limitations, here we present a new 3D CNN architecture for the causal/online processing of videos. Namely, we propose a novel Recurrent Convolutional Network (RCN), which relies on recurrence to capture the temporal context across frames at each network level. Our network decomposes 3D convolutions into (1) a 2D spatial convolution component, and (2) an additional hidden state 1 × 1 convolution, applied across time. The hidden state at any time t is assumed to depend on the hidden state at t - 1 and on the current output of the spatial convolution component. As a result, the proposed network: (i) produces causal outputs, (ii) provides flexible temporal reasoning, (iii) preserves temporal resolution. Our experiments on the largescale large Kinetics and MultiThumos datasets show that the proposed method performs comparably to anti-causal 3D CNNs, while being causal and using fewer parameters.

In this paper, we propose a simultaneous feature and dictionary learning (SFDL) method for image set based face recognition, where each training and testing example contains a face image set captured from different poses, illuminations, expressions and resolutions. While several feature learning and dictionary learning methods have been proposed for image set based face recognition in recent years, most of them learn the features and dictionaries separately, which may not be powerful enough because some discriminative information for dictionary learning may be compromised in the feature learning stage if they are applied sequentially, and vice versa. To address this, we propose a SFDL method to learn discriminative features and dictionaries simultaneously from raw face images so that discriminative information can be jointly exploited. Extensive experimental results on four widely used face datasets show that our method achieves better performance than state-of-the-art image set based face recognition methods.KeywordsFace recognitionimage setfeature learningdictionary learningsimultaneous learning

Crop yield prediction from multi-spectral, multi-temporal remotely sensed imagery using recurrent 3D convolutional neural networks

The past several years have seen both an explosion in the use of Convolutional Neural Networks (CNNs) and accelerators to make CNN inference practical. In the architecture community, the lion share of effort has targeted CNN inference for image recognition. The closely related problem of video recognition has received far less attention as an accelerator target. This is surprising, as video recognition is more computationally intensive than image recognition, and video traffic is predicted to be the majority of internet traffic in the coming years. This paper fills the gap between algorithmic and hardware advances for video recognition by providing a design space exploration and flexible architecture for accelerating 3D Convolutional Neural Networks (3D CNNs)—the core kernel in modern video understanding. When compared to (2D) CNNs used for image recognition, efficiently accelerating 3D CNNs poses a significant engineering challenge due to their large (and variable over time) memory footprint and higher dimensionality. To address these challenges, we design a novel accelerator called Morph, that can adaptively support different spatial and temporal tiling strategies depending on the needs of each layer of each target 3D CNN. We codesign a software infrastructure alongside the Morph hardware to find good-fit parameters to control the hardware. Evaluated on state-of-the-art 3D CNNs, Morph achieves up to 2.7× (1.9× average) reduction in energy consumption and improves performance/watt up to 4.4× (3× average) compared to a baseline 3D CNN accelerator, with an area overhead of 2%. Morph further achieves a 11.6× average energy reduction on 3D CNNs when compared to Eyeriss, a popular 2D CNN accelerator, while reducing efficiency compared to Eyeriss on a 2D CNN by 71%.

Recently, several convolutional neural networks have been proposed not only for 2D images, but also for 3D and 4D volume segmentation. Nevertheless, due to the large data size of the latter, acquiring a sufficient amount of training annotations is much more strenuous than in 2D images. For 4D time-series tomograms, this is usually handled by segmenting the constituent tomograms independently through time with 3D convolutional neural networks. Inter-volume information is therefore not utilized, potentially leading to temporal incoherence. In this paper, we attempt to resolve this by proposing two hidden Markov model variants that refine 4D segmentation labels made by 3D convolutional neural networks working on each time point. Our models utilize not only inter-volume information, but also the prediction confidence generated by the 3D segmentation convolutional neural networks themselves. To the best of our knowledge, this is the first attempt to refine 4D segmentations made by 3D convolutional neural networks using hidden Markov models. During experiments we test our models, qualitatively, quantitatively and behaviourally, using prespecified segmentations. We demonstrate in the domain of time series tomograms which are typically undersampled to allow more frequent capture; a particularly challenging problem. Finally, our dataset and code is publicly available.

Modern industrial plants generally consist of multiple manufacturing units, and the local correlation within each unit can be used to effectively alleviate the effect of spurious correlation and meticulously reflect the operation status of the process system. Therefore, the local correlation, which is called spatial information here, should also be taken into consideration when developing the monitoring model. In this study, a cascaded monitoring network (MoniNet) method is proposed to develop the monitoring model with concurrent analytics of temporal and spatial information. By implementing convolutional operation to each variable, the temporal information that reveals dynamic correlation of process data and spatial information that reflects local characteristics within individual operation unit can be extracted simultaneously. For each convolutional feature, a submodel is developed and then all the submodels are integrated to generate a final monitoring model. Based on the developed model, the operation status of the newly collected sample can be identified by comparing the calculated statistics with their corresponding control limits. Similar to the convolutional neural network (CNN), the MoniNet can also expand its receptive field and capture deeper information by adding more convolutional layers. Besides, the filter selection and submodel development in MoniNet can be replaced to generalize the proposed network to many existing monitoring strategies. The performance of the proposed method is validated using two real industrial processes. The illustration results show that the proposed method can effectively detect process anomalies by concurrent analytics of temporal and spatial information.

Radar technology for at home health-care has many advantages such as safety, reliability, privacy-preserving, and contact-less sensing nature. Detecting falls using radar has recently gained attention in smart health care. In this paper, CapsFall, a new method for fall detection using an ultra-wideband radar that leverages the recent deep learning advances is proposed. To this end, a radar time series is derived from the radar back-scattered matrix and its time-frequency representation is obtained and used as input to the capsule network for automatic feature learning. In contrast to other existing methods, the proposed CapsFall method relies on multi-level feature learning from radar time-frequency representations. In particular, the proposed method utilizes a capsule network for automating feature learning and enhancing model discriminability. The experiments are conducted using a set of radar signals collected from ten subjects when performing various activities in a room environment. The performance of the proposed CapsFall method is evaluated in terms of classification metrics and compared with those of the other existing methods based on convolutional neural network, multi-layer perceptron, decision tree, and support vector machine. The results show that the proposed CapsFall method outperforms the other methods in terms of accuracy, precision, sensitivity, and specificity values.

Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network-time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31-50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

Unsupervised feature learning refers to the problem of learning useful feature extraction functions from unlabeled data. Despite the great success of deep learning networks in this task in recent years, both for static and for sequential data, these systems can in general still not compete with the high performance of our brain at learning to extract useful representations from its sensory input. We propose the Neocortex-Inspired Locally Recurrent Neural Network: a new neural network for unsupervised feature learning in sequential data that brings ideas from the structure and function of the neocortex to the well-established fields of machine learning and neural networks. By mimicking connection patterns in the feedforward circuits of the neocortex, our system tries to generalize some of the ideas behind the success of convolutional neural networks to types of data other than images. To evaluate the performance of our system at extracting useful features, we have trained different classifiers using those and other learnt features as input and we have compared the obtained accuracies. Our system has shown to outperform other shallow feature learning systems in this task, both in terms of the accuracies achieved and in terms of how fast the classification task is learnt. The results obtained confirm our system as a state-of-the-art shallow feature learning system for sequential data, and suggest that extending it to or integrating it into deep architectures may lead to new successful networks that are competent at dealing with complex sequential tasks.

Most of the existing hyperspectral image change detection methods based on deep learning utilize convolutional neural networks (CNN), recurrent neural networks (RNN), and long-short-term memory (LSTM). However, CNNs primarily focus on extracting spatial information from hyperspectral images, thereby failing to fully utilize channel and temporal information. In this paper, we propose a network called STSTNet that can jointly extract spatial, spectral, and temporal feature information. To achieve a richer feature representation space, we employ a spatial attention module to extract spatial feature information from feature maps, and a spectral transformer encoder structure to extract abundant spectral sequence information. Specifically, we design two temporal information extraction structures after each spatial information extraction structure or spectral information extraction structure to capture correlations and interactions within the hyperspectral images. Extensive experiments have been conducted on three hyperspectral datasets, and the experimental results demonstrate the network’s excellent performance.