Scene Classification Using Deep Networks Combined with Visual Attention

Cardiovascular MRI image analysis by using the bio inspired (sand piper optimized) fully deep convolutional network (Bio-FDCN) architecture for an automated detection of cardiac disorders

In recent years, the widespread adaptation of Unmanned Aerial Vehicles (UAVs), commonly known as drones, among the public has led to significant security concerns, prompting intense research into drones’ classification methodologies. The swift and accurate classification of drones poses a considerable challenge due to their diminutive size and rapid movements. To address this challenge, this paper introduces (i) a novel drone classification approach utilizing deep convolution and deep transfer learning techniques. The model incorporates bypass connections and Leaky ReLU activation functions to mitigate the ‘vanishing gradient problem’ and the ‘dying ReLU problem’, respectively, associated with deep networks and is trained on a diverse dataset. This study employs (ii) a custom dataset comprising both audio and visual data of drones as well as analogous objects like an airplane, birds, a helicopter, etc., to enhance classification accuracy. The integration of audio–visual information facilitates more precise drone classification. Furthermore, (iii) a new Finite Impulse Response (FIR) low-pass filter is proposed to convert audio signals into spectrogram images, reducing susceptibility to noise and interference. The proposed model signifies a transformative advancement in convolutional neural networks’ design, illustrating the compatibility of efficacy and efficiency without compromising on complexity and learnable properties. A notable performance was demonstrated by the proposed model, with an accuracy of 100% achieved on the test images using only four million learnable parameters. In contrast, the Resnet50 and Inception-V3 models exhibit 90% accuracy each on the same test set, despite the employment of 23.50 million and 21.80 million learnable parameters, respectively.

Images are powerful tools with which to convey human emotions, with different images stimulating diverse emotions. Numerous factors affect the emotions stimulated by the image, and many researchers have previously focused on low-level features such as color, texture and so on. Inspired by the successful use of deep convolutional neural networks (CNN) in the visual recognition field, we used a data augmentation method for small data sets to gain the sufficient number of the training dataset. In this paper, we use low-level features (color and texture features) of the image to assist the extraction of advanced features (image object category features and deep emotion features of images), which are automatically learned by deep networks, to obtain more effective image sentiment features. Then, we use the stack sparse auto-encoding network to recognize the emotions evoked by the image. Finally, high-level semantic descriptive phrases including image emotions and objects are output. Our experiments are carried out on the IAPS and GAPED data sets of the dimension space and the artphoto data set of the discrete space. Compared with the traditional manual extraction methods and other existing models, our method is superior to in terms of test performance.

Seeing through disguise: Getting to know you with a deep convolutional neural network

In recent years, there has been a significant increase in the production of high-resolution aerial and satellite imagery. Analyzing and extracting urban features from these images, especially building components, is a significant challenge in photogrammetry and remote sensing. Meanwhile, deep convolution neural networks have been used as a powerful model in the semantic segmentation of building features. However, due to the structure of these types of deep convolution networks, accurate retrieval of building boundary information during training will be difficult, and this will lead to ambiguous boundary areas. On the other hand, the use of distributed-based loss functions, such as the binary cross-entropy loss alone cannot improve the segmentation accuracy in the boundary areas of building features. Accordingly, in this paper, a derivative boundary loss function is introduced to optimize and enhance the extraction of buildings in border areas. To calculate the proposed boundary loss function, distance transform image (DTI), is obtained from ground-truth images and predicted images, that derived from the segmentation network. In this paper, a derivative method for calculating DTI is presented. Another advantage of the proposed loss function is its ability to be used in a wide range of deep convolution segmentation networks. The proposed method was investigated and tested using the ISPRS Potsdam and Vaihingen data sets. The results of this implementation show the superiority in the criteria of evaluation and improvement of buildings extraction if the proposed boundary loss function is used along with the distributed-based loss function.

In the transient electromagnetic method (TEM) exploration, due to the complex acquisition environment, the original data has serious interference, and the entire data segment and the entire point are missing. During data processing, the missing and cut-off data due to interference are difficult to complete based on reliable geophysical models. In response to this problem, this paper establishes a deep convolutional generative countermeasure network model (DCGAN) for TEM data completion based on the generative countermeasure network (GAN). The model uses GAN network as the framework, embedding deep convolutional neural network modules, that is, using deep deconvolution and convolution layers to replace the fully connected layer in the GAN model to fully extract the spatial features of TEM data. In order to test the performance of the model, the test set is divided into two categories: random missing and overall missing. On this basis, increase the number of deleted data, control the missing rate in the test data set to 10%, 20%, 30%, 40%, and 50%, and gradually carry out the test. Finally, the test results are compared with the current mainstream data completion method GAN. The results show that the overall effect of the DCGAN model in TEM data completion is higher than that of GAN. At the same time, when the randomly missing data is less than or equal to 40% and the continuous missing data is less than or equal to 30%, the R2 of the DCGAN model is above 0.85, which can maintain good performance. Through random missing, continuous missing data completion and step-by-step experiments with different missing rates, the model can ensure the accuracy of the completion while clarifying the scope of application, which has certain practical application value.

Hyperspectral remote sensing image classification based on residual generative Adversarial Neural Networks

Purpose: The aim of the article is to develop an algorithm for classifying cracks in the analyzed images using modern methods of deep machine learning and transfer learning based on pretrained convolutional neural network - Inception-ResNet-v2. Design/Methodology/Approach: Transfer learning based on the pretrained convolutional neural network was used to categorize the images. The fully conected layer of the Inception-ResNet-v2 network has been modified. The last layer was trained using a two-class (binary) linear SVM (Support Vector Machine). In total, 20,000 training cases (images) were used to train the fully connected layer within transfer learning process. The research analyzed the possibility of using the deep neural networks for quick and fully automatic identification of cracks / defects on the surface of analyzed parts. Findings: The results indicate that pretrained convolutional neural network using SVM to train a fully connected layer is a very effective solution for visual crack / fault detection. In the analyzed model, a positive classification was obtained at the level of 99.89%. Practical Implications: The model presented in the article can be used in quality control carried out by process monitoring. An effective model for identifying defective parts can be used in both logistics and production processes. Originality/Value: A novelty is the use of a freely available, deep neural network trained to classify 1000 categories of various images for binary categorization of faults (cracks). The algorithm was adjusted by replacing the primary, 1000-output fully connected layer in the Inception-ResNet-v2 network with a binary layer (2 categories). The fully connected layer has been trained using the classification version of the popular SVM learner, but thanks to the combination of this layer with the sophisticated fearure extraction ability of the pre-trained Inception-ResNet-v2 deep network, the resulting predictive model enables the classification of defects with a very high level of accuracy.

We study the eigenvalue distribution of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the analog of the population covariance matrix is now a function of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [J. Pennington, S. Schoenholz and S. Ganguli, The emergence of spectral universality in deep networks, Proc. Mach. Learn. Res. 84 (2018) 1924–1932, arXiv:1802.09979] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability in this case has to be justified. The justification has been given in [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] for Gaussian data matrices with independent entries, a standard analytical model of free probability, by using a version of the techniques of random matrix theory. In this paper, we use another version of the techniques to extend the results of [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] to the case where the entries of the data matrices are just independent identically distributed random variables with zero mean and finite fourth moment. This, in particular, justifies the mean field approximation in the infinite width limit for the deep untrained neural networks and the property of the macroscopic universality of random matrix theory in this case.

Understanding the context of the scene is one of the most important aspects for new generation of autonomous vehicles. It is a very trivial task for a human to recognize the scene context by only a single look at the picture, however, for a computer, it is still a challenging task. This problem can be solved by automatic data labeling using deep-learning models for scene classification. Relying on scene type labels we can select relevant scenes to prepare a balanced dataset to train more advanced instance detection models using data from a specific road condition. This study presents a novel framework based on a deep convolutional neural network (CNN) for the automatic road scene classification of street-level automotive images. For the evaluation of our approach, we use a well-known autonomous benchmark dataset, from which we extract geo-position data and combine them with predictions from the scene classification model to get ground truth labels to train and evaluate a ResNet-50 model for scene classification. The results and comparison with state-of-the-art methods are presented.

Real-time smoke detection is of great significance for early warning of fire, which can avoid the serious loss caused by fire. Detecting smoke in actual scenes is still a challenging task due to large variance of smoke color, texture, and shapes. Moreover, the smoke detection in the actual scene is faced with the difficulties in data collection and insufficient smoke datasets, and the smoke morphology is susceptible to environmental influences. To improve the performance of smoke detection and solve the problem of too few datasets in real scenes, this paper proposes a model that combines a deep convolutional generative adversarial network and a convolutional neural network (DCG-CNN) to extract smoke features and detection. The vibe algorithm was used to collect smoke and nonsmoke images in the dynamic scene and deep convolutional generative adversarial network (DCGAN) used these images to generate images that are as realistic as possible. Besides, we designed an improved convolutional neural network (CNN) model for extracting smoke features and smoke detection. The experimental results show that the method has a good detection performance on the smoke generated in the actual scenes and effectively reduces the false alarm rate.

Text recognition in natural scene images is a challenging problem in computer vision. Different than the optical character recognition (OCR), text recognition in natural scene images is more complex due to variations in text size, colors, fonts, orientations, complex backgrounds, occlusion, illuminations and uneven lighting conditions. In this paper, we propose a segmentation-free method based on a deep convolutional recurrent neural network to solve the problem of cursive text recognition, particularly focusing on Urdu text in natural scenes. Compared to the non-cursive scripts, Urdu text recognition is more complex due to variations in the writing styles, several shapes of the same character, connected text, ligature overlapping, stretched, diagonal and condensed text. The proposed model gets a whole word image as an input without pre-segmenting into individual characters, and then transforms into the sequence of the relevant features. Our model is based on three components: a deep convolutional neural network (CNN) with shortcut connections to extract and encode the features, a recurrent neural network (RNN) to decode the convolutional features, and a connectionist temporal classification (CTC) to map the predicted sequences into the target labels. To increase the text recognition accuracy further, we explore deeper CNN architectures like VGG-16, VGG-19, ResNet-18 and ResNet-34 to extract more appropriate Urdu text features, and compare the recognition results. To conduct the experiments, a new large-scale benchmark dataset of cropped Urdu word images in natural scenes is developed. The experimental results show that the proposed deep CRNN network with shortcut connections outperform than other network architectures. The dataset is publicly available and can be downloaded from <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://data.mendeley.com/datasets/k5fz57zd9z/1</uri> .

In the human brain, the incoming light to the retina is transformed into meaningful representations that allow us to interact with the world. In a similar vein, the RGB pixel values are transformed by a deep neural network (DNN) into meaningful representations relevant to solving a computer vision task it was trained for. Therefore, in my research, I aim to reveal insights into the visual representations in the human visual cortex and DNNs solving vision tasks. In the previous decade, DNNs have emerged as the state-of-the-art models for predicting neural responses in the human and monkey visual cortex. Research has shown that training on a task related to a brain region’s function leads to better predictivity than a randomly initialized network. Based on this observation, we proposed that we can use DNNs trained on different computer vision tasks to identify functional mapping of the human visual cortex. To validate our proposed idea, we first investigate a brain region occipital place area (OPA) using DNNs trained on scene parsing task and scene classification task. From the previous investigations about OPA’s functions, we knew that it encodes navigational affordances that require spatial information about the scene. Therefore, we hypothesized that OPA’s representation should be closer to a scene parsing model than a scene classification model as the scene parsing task explicitly requires spatial information about the scene. Our results showed that scene parsing models had representation closer to OPA than scene classification models thus validating our approach. We then selected multiple DNNs performing a wide range of computer vision tasks ranging from low-level tasks such as edge detection, 3D tasks such as surface normals, and semantic tasks such as semantic segmentation. We compared the representations of these DNNs with all the regions in the visual cortex, thus revealing the functional representations of different regions of the visual cortex. Our results highly converged with previous investigations of these brain regions validating the feasibility of the proposed approach in finding functional representations of the human brain. Our results also provided new insights into underinvestigated brain regions that can serve as starting hypotheses and promote further investigation into those brain regions. We applied the same approach to find representational insights about the DNNs. A DNN usually consists of multiple layers with each layer performing a computation leading to the final layer that performs prediction for a given task. Training on different tasks could lead to very different representations. Therefore, we first investigate at which stage does the representation in DNNs trained on different tasks starts to differ. We further investigate if the DNNs trained on similar tasks lead to similar representations and on dissimilar tasks lead to more dissimilar representations. We selected the same set of DNNs used in the previous work that were trained on the Taskonomy dataset on a diverse range of 2D, 3D and semantic tasks. Then, given a DNN trained on a particular task, we compared the representation of multiple layers to corresponding layers in other DNNs. From this analysis, we aimed to reveal where in the network architecture task-specific representation is prominent. We found that task specificity increases as we go deeper into the DNN architecture and similar tasks start to cluster in groups. We found that the grouping we found using representational similarity was highly correlated with grouping based on transfer learning thus creating an interesting application of the approach to model selection in transfer learning. During previous works, several new measures were introduced to compare DNN representations. So, we identified the commonalities in different measures and unified different measures into a single framework referred to as duality diagram similarity. This work opens up new possibilities for similarity measures to understand DNN representations. While demonstrating a much higher correlation with transfer learning than previous state-of-the-art measures we extend it to understanding layer-wise representations of models trained on the Imagenet and Places dataset using different tasks and demonstrate its applicability to layer selection for transfer learning. In all the previous works, we used the task-specific DNN representations to understand the representations in the human visual cortex and other DNNs. We were able to interpret our findings in terms of computer vision tasks such as edge detection, semantic segmentation, depth estimation, etc. however we were not able to map the representations to human interpretable concepts. Therefore in our most recent work, we developed a new method that associates individual artificial neurons with human interpretable concepts. Overall, the works in this thesis revealed new insights into the representation of the visual cortex and DNNs...

In large-scale image retrieval, deep features extracted by Convolutional Neural Network (CNN) can effectively express more image information than those extracted by traditional manual methods. However, the deep feature dimensions obtained by Deep Convolutional Neural Network (DCNN) are too high and redundant, which leads to low retrieval efficiency. We propose a novel image retrieval method, which combines deep features selection with improved DCNN and hash transform based on high-dimension features reduction to gain lowdimension deep features and realizes efficient image retrieval. Firstly, the improved network is based on the existing deep model to build a more profound and broader network by adding multiple groups of different branches. Therefore, it is named DFS-Net (Deep Feature Selection Network). The adaptive learning deep features of the Network can effectively alleviate the influence of over-fitting and improve the feature expression of image content. Secondly, the information gain rate method is used to filter the extracted deep features to reduce the feature dimension and ensure the information loss is small. The last step of the method, hash Transform, sparsifies and binarizes this representation to reduce the computation and storage pressure while maintaining the retrieval accuracy. Finally, the scheme is based on the distinguished ResNet50, InceptionV3, and MobileNetV2 models, and studied and evaluated deeply on the CIFAR10 and Caltech256 datasets. The experimental results show that the novel method can train the deep features with stronger recognition ability on limited training samples, and improve the accuracy and efficiency of image retrieval effectively.

Recently, the fusion of hyperspectral and light detection and ranging (LiDAR) data has obtained a great attention in the remote sensing community. In this paper, we propose a new feature fusion framework using deep neural network (DNN). The proposed framework employs a novel 3D convolutional neural network (CNN) to extract the spectral-spatial features of hyperspectral data, a deep 2D CNN to extract the elevation features of LiDAR data, and then a fully connected deep neural network to fuse the extracted features in the previous CNNs. Through the aforementioned three deep networks, one can extract the discriminant and invariant features of hyperspectral and LiDAR data. At last, logistic regression is used to produce the final classification results. The experimental results reveal that the proposed deep fusion model provides competitive results. Furthermore, the proposed deep fusion idea opens a new window for future research.