Three-dimensional convolutional neural networks for target classification with volumetric sonar data

In underwater synthetic aperture sonar (SAS) imagery, there is a need for accurate target recognition algorithms. Automated detection of underwater objects has many applications, not the least of which being the safe extraction of dangerous explosives. In this paper, we discuss experiments on a deep learning approach to binary classification of target and non-target SAS image tiles. Using a fused anomaly detector, the pixels in each SAS image have been narrowed down into regions of interest (ROIs), from which small target-sized tiles are extracted. This tile data set is created prior to the work done in this paper. Our objective is to carry out extensive tests on the classification accuracy of deep convolutional neural networks (CNNs) using location-based cross validation. Here we discuss the results of varying network architectures, hyperparameters, loss, and activation functions; in conjunction with an analysis of training and testing set configuration. It is also in our interest to analyze these unique network setups extensively, rather than comparing merely classification accuracy. The approach is tested on a collection of SAS imagery.

Characteristics of underwater targets displayed in synthetic aperture sonar (SAS) imagery vary depending on their environmental context. Discriminative features in sea grass may differ from the features that are discriminative in sand ripple, for example. Environmentally-adaptive target detection and classification systems that take into account environmental context, therefore, have the potential for improved results. This paper presents an end-to-end environmentally-adaptive target detection system for SAS imagery that performs target recognition while accounting for environmental context. First, locations of interest are identified in the imagery using the Reed-Xiaoli (RX) detector and a Non-Gaussian detector based on the multivariate Laplace distribution. Then, the Multiple Instance Learning via Embedded Instance Selection (MILES) approach is used to identify the environmental context of the targets. Finally, target features are extracted and a set of environmentally-specific k-Nearest Neighbors (k-NN) classifiers are applied. Experiments were conducted on a collection of both high and low frequency SAS imagery with a variety of environmental contexts and results show improved classification accuracy between target classes when compared with classification results with no environmental consideration.

Sidescan imaging sonar performance is often described using metrics such as resolution, maximum range, and area coverage rate. These attributes focus on describing how finely and over what area a sensor can make imagery of the seafloor and targets. While these metrics are important, they are inadequate for fully characterizing the quality of sonar imagery. Shadow regions often carry as much, and sometimes even more, information than the direct return from a target. The shadow contrast within an image is therefore a critical property, and it is the result of a complex interaction between the sonar hardware, the environment, and the signal processing. This paper builds on key results from the synthetic aperture radar and sonar literature to develop a comprehensive, quantitative model for predicting the shadow contrast ratio. A model for the contrast ratio is constructed using an approach that is similar to the development of the sonar equation. This ratio describes the average relative level between the seafloor and a shadow. The model includes the effects of the transducer and array design, ambient noise, quantization noise, unwanted volume and surface reverberation, and the signal processing used to construct the synthetic aperture sonar (SAS) imagery. The shadow contrast predicted by the model is compared to the contrast measured from SAS imagery, where close agreement is observed. Examples are presented where the model is applied to a hypothetical high-frequency imaging SAS sensor in a typical operating environment. The shadow contrast ratio is estimated as a function of the sensor range and key parameters of interest, such as receiver channel spacing, sediment backscatter coefficient, and the time-bandwidth product of the transmitted signal. At long ranges, the contrast ratio is limited by additive noise interference, while at near ranges the effect of the receiver channel spacing dominates the contrast ratio through along-track ambiguities. The use of this tool for analysis and prediction of image quality has significant implications for system design, mission planning, and data processing.

The problem of detecting buried unexploded ordnance (UXO) is addressed with a sensor deployed from a shallow-draft surface vessel. This sonar system produces three dimensional synthetic aperture sonar (SAS) imagery of both surficial and buried UXO across a range of environments. The sensor's hardware design was based in part upon data created using a hybrid modeling approach that combined results from separate environmental scattering and target scattering models. This hybrid model produced synthetic sensor data where the sensor/environment/target space could be modified to explore the expected operating conditions. The simulated data were also used to adapt a set of existing signal processing algorithms for formation of three-dimensional acoustic imagery. Recently, the sonar system has been integrated to a test platform, and experiments have been conducted at a trial site in the Foster Joseph Sayers Reservoir near Howard, PA. This test site has been prepared with several buried man-made objects. Initial results show that fully buried targets can be detected.

Seizure detection in epilepsy monitoring units (EMUs) is essential for the clinical assessment of drug-resistant epilepsy. Automated video analysis using machine learning provides a promising aid for seizure detection, with resultant reduction in the resources required for diagnostic monitoring. We employ a three-dimensional (3D) convolutional neural network with fully fine-tuned backbone layers to identify seizures from EMU videos. A two-stream inflated 3D-ConvNet architecture (I3D) classified video clips as a seizure or not a seizure. A pretrained action classifier was fine-tuned on 11 h of video containing 49 tonic-clonic seizures from 25 patients monitored at a large academic hospital (site A) using leave-one-patient-out cross-validation. Performance was evaluated by comparing model predictions to ground-truth annotations obtained from video-electroencephalographic review by an epileptologist on videos from site A and a separate dataset from a second large academic hospital (site B). The model achieved a leave-one-patient-out cross-validation F1-score of .960 ± .007 (mean ± SD) and area under the receiver operating curve score of .988 ± .004 at site A. Evaluation on full videos detected all seizures (95% binomial exact confidence interval = 94.1%-100%), with median detection latency of 0.0 s (interquartile range = 0.0-3.0) from seizure onset. The site A model had an average false alarm rate of 1.81 alarms per hour, although 36 of the 49 videos (73.5%) had no false alarms. Evaluation at site B demonstrated generalizability of the architecture and training strategy, although cross-site evaluation (site A model tested on site B data and vice versa) resulted in diminished performance. Our model demonstrates high performance in the detection of epileptic seizures from video data using a fine-tuned I3D model and outperforms similar models identified in the literature. This study provides a foundation for future work in real-time EMU seizure monitoring and possibly for reliable, cost-effective at-home detection of tonic-clonic seizures.

Deep convolutional neural networks are used to perform underwater target classification in synthetic aperture sonar (SAS) imagery. The deep networks are learned using a massive database of real, measured sonar data collected at sea during different expeditions in various geographical locations. A novel training procedure is developed specially for the data from this new sensor modality in order to augment the amount of training data available for learning and to avoid overfitting. The deep networks learned are employed for several binary classification tasks in which different classes of objects in real sonar data are to be discriminated. The proposed deep approach consistently achieves superior performance to a traditional feature-based classifier that we had relied on previously.

The sequence of echoes detected by an active Synthetic Aperture Sonar (SAS) is coherently added in an appropriate way to produce an image with greatly enhanced resolution in the azimuth, or along-track direction when compared with an image obtained from a standard Side Looking Sonar (SLS). The SAS processing originates from the Synthetic Aperture Radar (SAR) concept. A complete introduction to SAR technique can be found in (Sherwin et al., 1962), (Walker, 1980), (Wiley, 1985) and (Curlander & McDonough, 1991). Raytheon was issued in 1969 a patent for a high-resolution seafloor imaging SAS (Walsh, 1969) and 1971 analyzed a potential system in terms of its resolution and signal-to-noise ratio. Cutrona was the first well-known radar specialist to point out how the various aspects of SAR could be translated to an underwater SAS (Cutrona, 1975). Hughes (1977) compared the performance of a standard SLS to an SAS and showed that the potential mapping rate for SAS was significantly higher than for side-looking sonar. At the time there was an assumption that the instability of the oceanic environment would prevent the formation of SAS imagery. Experimental work, which was performed by Williams (1976) and Christoff et al. (1982), refuted the instability worry. The verification of this assertion performed at higher frequencies by Gough & Hawkins (1989). Later, other concerns regarding the stability of the towed platform were also raised and some railor wire-guided trails where set up to avoid this extra complication. Nowadays there are a multiple of systems as hull mounted SAS systems, towed SAS systems and Autonomous Underwater Vehicles (AUV) systems. For further reading one can find an extended historical background of SAS in (Gough & Hawkins, 1997). Time and experience were needed to adapt SAR algorithms to SAS systems; the SAS systems use smaller radiating elements in proportion to the wavelength, which leads to higher radiation pattern of SAS with respect to SAR. The range migration effect on synthetic aperture processing is significant and pronounced in SAS imagery. An additional difference between SAR and SAS systems is the synthetic aperture time being greater in one order of magnitude in SAS, which leads to a phase corruption due to the medium fluctuations and platform instabilities. Typical synthetic aperture times for SAR are of the order of seconds with a medium coherence of some days, whereas for SAS the typical synthetic aperture time is of the order of several minutes with a similar medium coherence time (Marx et al. 2000). O pe n A cc es s D at ab as e w w w .in te ch w eb .o rg

Convolutional neural networks (CNN) have given rise to a new generation of video super-resolution (SR) technique. However, most existing CNN-based video SR algorithms treat the consecutive frames as a series of feature maps, just as the procedure performed in single image SR algorithms. We propose an end-to-end three-dimensional (3-D) CNN video SR framework. The input frames are considered as a cube in our framework. 3-D convolution is performed on it to extract features along spatial and temporal dimension. Image prior knowledge, such as optical flows, is introduced in reconstruction. A combination of mean square error loss and multiscale structure similarity index (MS-SSIM) loss is used to optimize the model. Experimental results show that the proposed method reconstructs high-resolution frames with more accurate and visually pleasant structures compared with state-of-the-art video SR algorithms. We also achieve comparable PSNR/SSIM results with less computation time.

Identification of crops is an important topic in the agricultural domain. Hyperspectral remote sensing data are very useful for crop feature extraction and classification. Remote sensing data is an unstructured data and Convolutional Neural Network (CNN) can work on unstructured data efficiently. This paper presents an evaluation of CNN for crop classification using the Indian Pines standard dataset obtained from the AVIRIS sensor and the study area dataset obtained from the EO-1hyperion sensor. Optimized CNN has been tuned by training the model on different parameters. It has been compared with two classification algorithms: Deep Neural Network (DNN) and Convolutional Autoencoder. According to the test results, the proposed optimized CNN model provided better results as compared to the other two methods. CNN has given 97 ± 0.58% overall accuracy for the Indian Pines standard dataset and 78 ± 2.43% for our study area dataset.

The report presents an algorithm for constructing a convolutional hierarchical neural network classifier, which is a modification of the algorithm for constructing hierarchical neural network classifiers suggested before. The original algorithm was designed to exploit intrinsic class hierarchy to build a class tree with a neural network in each node classifying groups of initial classes (in a non-terminal node) or a subset of original classes (in a terminal node). The convolutional modification utilizes convolutional neural networks instead of regular fully connected networks in order to apply the model to image classification tasks. Use of class hierarchy for image classification should reduce the number of adjusted neural network parameters compared to deep convolutional neural networks, and therefore it should reduce training and inference time. In this context the algorithm may be compared with some pruning techniques. The convolutional hierarchical neural network classifier inherits some hyperparameters of a conventional hierarchical neural network classifier, like the activation threshold and the threshold by the share of voting patterns. The goal of this study was to explore different strategies of choosing these hyperparameters. To test these strategies, we used the CIFAR-10 dataset. Also, for demonstration purposes we apply the convolutional hierarchical neural network classifier to the CIFAR-100 dataset.

White blood cells (WBCs) are the most diverse types of cells observed in the healing process of injured skeletal muscles. In the recovery process, WBCs exhibit a dynamic cellular response and undergo multiple changes of the protein expression. The progress of healing can be analyzed by the number of WBCs or by the number of specific proteins observed in light microscopy images obtained at different time points after injury. We propose a deep learning quantification and analysis system called DeepQuantify to analyze WBCs in light microscopy images of uninjured and injured muscles of female mice. The DeepQuantify system features in segmentation using the localized iterative Otsu's thresholding method, masking postprocessing, and classification of WBCs with a convolutional neural network (CNN) classifier to achieve a high accuracy and a low manpower cost. The proposed two-layer CNN classifier designed based on the optimization hypothesis is evaluated and compared with other CNN classifiers. The DeepQuantify system adopting these CNN classifiers is evaluated for quantifying CD68-positive macrophages and 7/4-positive neutrophils and compared with the state-of-the-art deep learning segmentation architectures. DeepQuantify achieves an accuracy of 90.64% and 89.31% for CD68-positive macrophages and 7/4-positive neutrophils, respectively. The DeepQuantify system employing the proposed two-layer CNN architecture achieves better performance than those deep segmentation architectures. The quantitative analysis of two protein dynamics during muscle recovery is also presented.

Phase gradient autofocus (PGA) is widely used in spotlight SAR autofocus with strong robustness. However, being the different nature of synthetic aperture sonar (SAS) imagery, traditional PGA algorithm usually fails to yield satisfactory result on SAS image without modification. Three measures are taken to ensure that PGA can be applied in stripmap SAS autofocus. A SAS image is divided into sub-images which ensures that the phase error is invariant in each sub-image; The image shift in azimuth can be avoided by removing the mean value of the phase gradient and by using strong point targets in the image; The estimation accuracy can be improved by selecting range bins according to energy criteria combined with quality criteria. The proposed method is successfully applied on SAS image and excellent results are attained.

A new, two-stage approach for performing automatic target recognition (ATR) in multi-band synthetic aperture sonar (SAS) imagery is proposed. The approach consists of a simple detection algorithm, called MondrianB, followed by a more sophisticated classification stage based on convolutional neural networks (CNNs). The approach is extensible in that the same method can be used, without any modifications, for sensor data in the form of an arbitrary number of co-registered image bands (including one). The detection stage does not require any labeled training data. The classification stage relies on ensembles of small, efficient CNNs that collectively provide robust alarm predictions by leveraging unique network architectures and multiple input-data representations. The performance of the proposed ATR approach is verified on a sizable set of data collected at sea by a three-band SAS system. Specifically, the experiments conducted quantity the utility of each imaging band on both the detection and classification stages of a man-made object-recognition task. It is shown that leveraging information from the additional bands improves both detection and classification performance.

Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network-time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31-50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

Instead of using current deep-learning segmentation models (like the UNet and variants), we approach the segmentation problem using trained Convolutional Neural Network (CNN) classifiers, which automatically extract important features from images for classification. Those extracted features can be visualized and formed into heatmaps using Gradient-weighted Class Activation Mapping (Grad-CAM). This study tested whether the heatmaps could be used to segment the classified targets. We also proposed an evaluation method for the heatmaps; that is, to re-train the CNN classifier using images filtered by heatmaps and examine its performance. We used the mean-Dice coefficient to evaluate segmentation results. Results from our experiments show that heatmaps can locate and segment partial tumor areas. But use of only the heatmaps from CNN classifiers may not be an optimal approach for segmentation. We have verified that the predictions of CNN classifiers mainly depend on tumor areas, and dark regions in Grad-CAM's heatmaps also contribute to classification.