Dual-Domain Prior-Driven Deep Network for Infrared Small-Target Detection

We study the eigenvalue distribution of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the analog of the population covariance matrix is now a function of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [J. Pennington, S. Schoenholz and S. Ganguli, The emergence of spectral universality in deep networks, Proc. Mach. Learn. Res. 84 (2018) 1924–1932, arXiv:1802.09979] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability in this case has to be justified. The justification has been given in [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] for Gaussian data matrices with independent entries, a standard analytical model of free probability, by using a version of the techniques of random matrix theory. In this paper, we use another version of the techniques to extend the results of [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] to the case where the entries of the data matrices are just independent identically distributed random variables with zero mean and finite fourth moment. This, in particular, justifies the mean field approximation in the infinite width limit for the deep untrained neural networks and the property of the macroscopic universality of random matrix theory in this case.

Purpose: The aim of the article is to develop an algorithm for classifying cracks in the analyzed images using modern methods of deep machine learning and transfer learning based on pretrained convolutional neural network - Inception-ResNet-v2. Design/Methodology/Approach: Transfer learning based on the pretrained convolutional neural network was used to categorize the images. The fully conected layer of the Inception-ResNet-v2 network has been modified. The last layer was trained using a two-class (binary) linear SVM (Support Vector Machine). In total, 20,000 training cases (images) were used to train the fully connected layer within transfer learning process. The research analyzed the possibility of using the deep neural networks for quick and fully automatic identification of cracks / defects on the surface of analyzed parts. Findings: The results indicate that pretrained convolutional neural network using SVM to train a fully connected layer is a very effective solution for visual crack / fault detection. In the analyzed model, a positive classification was obtained at the level of 99.89%. Practical Implications: The model presented in the article can be used in quality control carried out by process monitoring. An effective model for identifying defective parts can be used in both logistics and production processes. Originality/Value: A novelty is the use of a freely available, deep neural network trained to classify 1000 categories of various images for binary categorization of faults (cracks). The algorithm was adjusted by replacing the primary, 1000-output fully connected layer in the Inception-ResNet-v2 network with a binary layer (2 categories). The fully connected layer has been trained using the classification version of the popular SVM learner, but thanks to the combination of this layer with the sophisticated fearure extraction ability of the pre-trained Inception-ResNet-v2 deep network, the resulting predictive model enables the classification of defects with a very high level of accuracy.

In recent years, deep neural networks (DNNs) have achieved great successes in many areas, such as cognitive computation, pattern recognition, and computer vision. Although many hand-crafted deep networks have been proposed in the literature, designing a well-behaved neural network for a specific application requires high-level expertise yet. Hence, the automatic architecture design of DNNs has become a challenging and important problem. In this paper, we propose a new reinforcement learning method, whose action policy is to select neural blocks and construct deep networks. We define the action search space with three types of neural blocks, i.e., dense block, residual block, and inception-like block. Additionally, we have also designed several variants for the residual and inception-like blocks. The optimal network is automatically learned by a Q-learning agent, which is iteratively trained to generate well-performed deep networks. To evaluate the proposed method, we have conducted experiments on three datasets, MNIST, SVHN, and CIFAR-10, for image classification applications. Compared with existing hand-crafted and auto-generated neural networks, our auto-designed neural network delivers promising results. Moreover, the proposed reinforcement learning algorithm for deep networks design only runs on one GPU, demonstrating much higher efficiency than most of the previous deep network search approaches.

Extracting and inserting knowledge into stacked denoising auto-encoders

Domain-invariant representation learning using an unsupervised domain adversarial adaptation deep neural network

Deep neural network is an important and effective approach to discover patterns inside data. Most existing neural networks, like deep neural network and convolutional neural network, have a fixed network connection among neurons. This will cause overfitting problems and long training time. This article discusses an algorithm to combat these problems, through creating and removing connections during training, leading to faster training and fewer symmetric problems. This enables the network to adapt itself to the proper number of neurons according to the dataset. It greatly speeds up training process, so it is useful for large datasets and deep networks. This method is simple to implement and requires less memory and computing power than existing machine learning algorithm. It also has good results in machine learning problems.

The raising number of elderly people urges the research of systems able to monitor and support people inside their domestic environment. An automatic system capturing data about the position of a person in the house, through accelerometers and RGBd cameras can monitor the person activities and produce outputs associating the movements to a given tasks or predicting the set of activities that will be executes. We considered, for the task the classification of the activities a Deep Convolutional Neural Network. We compared two different deep network and analyzed their outputs.

Single image super-resolution (SISR) with deep convolutional neural networks has recently attracted increasing attention due to its potentials to generate rich details. To obtain better fidelity and visual quality, most of existing methods are of heavy design with the depth of network. However, the lack of high-frequency information in the deep network and the limit of non-local context relations of pixels remain the core issue. In addition, there are many arbitrary-scale image super-resolution tasks, so it is a focus that provides an effective and efficient model for super resolution with arbitrary-scale. To this end, we propose a Feature Preserving and Enhancing Network (FPEN) based on implicit representation, which is aims to preserving high-frequency information in deep network and enhancing non-local contextual features of pixels to output more realistic images at arbitrary scales. In particular, our proposed High Frequency Preserving Block (HFPB) can divide the features into high-frequency components and low-frequency components, and allocate more operations to high-frequency components to ensure that high-frequency components can be preserved in the network. Since high-frequency information contains the details and texture of the image, it can restore the finer details of the image. Moreover, Pixel Continuity Attention (PiCA) module our proposed utilizes visual cues observed from pixels to adaptively recalibrate the pixel range that needs attention in the image to better achieve feature enhancement and generate smoother images. Extensive experiments conducted on benchmark SISR models and datasets show that Feature Preserving and Enhancing Network can be employed for various SISR tasks with arbitrary-scale to obtain the better visual quality than other state-of-the-art SR algorithms.

Layer-by-layer training is an alternative end-to-end backpropagation approach for training deep convolutional neural networks. Layer-by-layer training on specific architectures can yield very competitive results. In the ImageNet database (www.imagenet. org), layer-by-layer trained networks can perform comparable to many modern end-to-end trained networks. This article compares the performance gap between the two training procedures over a wide range of network architectures and further analyzes the potential limitations of layer-by-layer training. The results show that layer-by-layer learning quickly saturates after a certain critical level due to overfitting of early levels in neural networks. Several approaches that have been used to solve this problem are discussed, and a methodology for improving layer-by-layer learning in various neural network architectures is discussed. Fundamentally, this research highlights the need to open up the black box that represents modern deep neural networks and explore the layer-by-layer interactions between intermediate hidden layers within deep networks through the lens of layer-by-layer learning.

By the development of social media, sentiment analysis has changed to one of the most remarkable research topics in the field of natural language processing which tries to dig information from textual data containing users’ opinions or attitudes toward a particular topic. In this regard, deep neural networks have emerged as promising techniques that have been extensively used for this aim in recent years and obtained significant results. Considering the fact that deep neural networks can automatically extract features from data, it can be claimed that intermediate representations extracted from these networks can be also used as appropriate features. While different deep neural networks are able to extract various types of features due to their distinct structures, we decided to combine features extracted from heterogeneous neural networks using multi-view classifiers to enhance the overall performance of document-level sentiment analysis by considering the correlation between them. The proposed multi-view deep network makes use of intermediate features extracted from convolutional and recursive neural networks to perform classification. Based on the results of the experiments, the proposed multi-view deep network not only outperforms single-view deep neural networks but also has superior efficiency and generalization performance.

Loan approval prediction is a critical task in the financial sector, enabling institutions to assess credit risk and make informed lending decisions. This research presents a comprehensive comparative study of various machine learning approaches for loan approval prediction, evaluating both traditional algorithms and deep learning architectures. We implemented and compared five traditional machine learning models (Logistic Regression, Decision Tree, Random Forest, Gradient Boosting, and SVM) against three neural network architectures (Simple Feed-Forward Neural Network, Deep Neural Network with Batch Normalization, and Wide & Deep Network). Using a synthetic dataset of 1,000 loan applications with features including income, credit score, and employment status, we achieved the highest accuracy of 93.5% with Gradient Boosting, closely followed by the Wide & Deep Neural Network at 93.5%. Feature importance analysis revealed credit score (45%) as the most influential factor, followed by income (40%) and employment status (15%). Additionally, we developed a professional Tkinter-based graphical user interface that enables real-time predictions, batch processing, and interactive visualizations. This system demonstrates the practical application of machine learning in financial decision-making and provides a framework for deploying predictive models in production environments. Keywords — Loan Approval, Machine Learning, Deep Learning, Neural Networks, Gradient Boosting, Tkinter GUI, Credit Risk Assessment

Training very deep neural networks is very difficult because of gradient degradation. However, the incomparable expressiveness of the many deep layers is highly desirable at testing time and usually leads to better performance. Recently, training techniques such as residual networks that enable us to train very deep networks have proved to be a great success. In this paper, we studied the application of the recently proposed deep networks with stochastic depth (DNSD) to train deeper acoustic models for speech recognition. By randomly dropping a subset of layers during training, the studied stochastic depth training method helps reduce the training time substantially, yet the networks trained are much deeper since all the layers are kept during testing. We investigated this approach on the TIMIT data set. Our preliminary experimental results show that when training data are limited, deep networks with stochastic depth helps very little. However, when more training data are available, DNSD significantly improves the recognition accuracy, compared with a conventional deep neural networks.

Advances in both parallel processing capabilities because of graphical processing units (GPUs) and computer vision algorithms have led to the development of deep neural networks (DNN) and their utilization in real-world applications. Starting from the LeNet-5 architecture of the 1990s, modern deep neural networks may have tens to hundreds of layers to solve complex problems such as license plate detection or recognition tasks. In this manuscript, we present a review of the state-of-the-art methods related to automatic license plate recognition. Since deep networks have demonstrated a remarkable ability to outperform other machine learning techniques, we focus only on neural network based license plate recognition methods. We highlight the particular types of networks, i.e., convolutional, residual recurrent, or long-short-term-memory, used for the specific tasks of license plate detection, extraction, or recognition in different existing works. The presented summary also highlights some of the most widely used data sets for comparison and shares the results reported in the reviewed papers. We also give an overview of the effects of fog, motion, or the use of synthetic data on license plate recognition. Finally, promising directions for future research in this domain are presented.

Accurate segmentation of the retinal microvasculature is a critical step in the quantitative analysis of the retinal circulation, which can be an important marker in evaluating the severity of retinal diseases. As manual segmentation remains the gold standard for segmentation of optical coherence tomography angiography (OCT-A) images, we present a method for automating the segmentation of OCT-A images using deep neural networks (DNNs). Eighty OCT-A images of the foveal region in 12 eyes from 6 healthy volunteers were acquired using a prototype OCT-A system and subsequently manually segmented. The automated segmentation of the blood vessels in the OCT-A images was then performed by classifying each pixel into vessel or nonvessel class using deep convolutional neural networks. When the automated results were compared against the manual segmentation results, a maximum mean accuracy of 0.83 was obtained. When the automated results were compared with inter and intrarater accuracies, the automated results were shown to be comparable to the human raters suggesting that segmentation using DNNs is comparable to a second manual rater. As manually segmenting the retinal microvasculature is a tedious task, having a reliable automated output such as automated segmentation by DNNs, is an important step in creating an automated output.

Short-term power load prediction is not only a critical part of power system dispatching but also an essential task for power marketing, grid planning, and other management departments; deep learning is an artificial intelligence method that has gained extensive attention in recent years; this paper selected three models with the most typical signs of deep learning recurrent neural networks and studied their model performance for short-term power load predictions, including power load data preprocessing, feature selection of prediction models, determination of model parameters, and comparison of prediction characteristics between shallow and deep networks, to explore and prove the applicability of deep learning neural networks in short-term power load predictions, ultimately improving the accuracy of short-term load predictions.KeywordsPower loadDeep learningPrediction