Generalization in layered classification neural networks

Unsupervised anomaly detection in pressurized water reactor digital twins using autoencoder neural networks

Deep-time neural networks: An efficient approach for solving high-dimensional PDEs

This study discusses the negative impact of the derivative of the activation functions in the output layer of artificial neural networks, in particular in continual learning. We propose Hebbian descent as a theoretical framework to overcome this limitation, which is implemented through an alternative loss function for gradient descent we refer to as Hebbian descent loss. This loss is effectively the generalized log-likelihood loss and corresponds to an alternative weight update rule for the output layer wherein the derivative of the activation function is disregarded. We show how this update avoids vanishing error signals during backpropagation in saturated regions of the activation functions, which is particularly helpful in training shallow neural networks and deep neural networks where saturating activation functions are only used in the output layer. In combination with centering, Hebbian descent leads to better continual learning capabilities. It provides a unifying perspective on Hebbian learning, gradient descent, and generalized linear models, for all of which we discuss the advantages and disadvantages. Given activation functions with strictly positive derivative (as often the case in practice), Hebbian descent inherits the convergence properties of regular gradient descent. While established pairings of loss and output layer activation function (e.g., mean squared error with linear or cross-entropy with sigmoid/softmax) are subsumed by Hebbian descent, we provide general insights for designing arbitrary loss activation function combinations that benefit from Hebbian descent. For shallow networks, we show that Hebbian descent outperforms Hebbian learning, has a performance similar to regular gradient descent, and has a much better performance than all other tested update rules in continual learning. In combination with centering, Hebbian descent implements a forgetting mechanism that prevents catastrophic interference notably better than the other tested update rules. When training deep neural networks, our experimental results suggest that Hebbian descent has better or similar performance as gradient descent.

Deep Learning (DL) is an area of artificial intelligence (AI), wherein machine learning (ML) using convolutional neural networks to detect patterns in data and forecast its worth. It excels in tough settings like climate change and can handle sophisticated jobs such as image and speech recognition, natural language processing, and others. Deep learning models have found applications in a variety of fields due to their ability to reflect complicated patterns in data. Deep learning applications have the capacity to analyze and automatically classify large volumes of Internet traffic. Deep learning-based solutions that automate the detection of attacks and address complicated cybersecurity issues are gaining popularity. This study extensively describes the promising uses of deep learning, which is based on many layers of artificial neural networks, in a wide range of security challenges. Before critically and comparatively reviewing state-of-the-art solutions from the literature, we will highlight the main properties of typical deep learning architectures used in cybersecurity applications. Deep learning highlights developing concepts and provides an overview of required resources, such as a general framework and appropriate datasets. The limits of the evaluated works are highlighted, as well as a vision of the present challenges in the field, providing useful insights and best practices for researchers and developers working on related problems.

Neural machine learning methods, such as deep neural networks (DNN), have achieved remarkable success in a number of complex data processing tasks. These methods have arguably had their strongest impact on tasks such as image and audio processing - data processing domains in which humans have long held clear advantages over conventional algorithms. In contrast to biological neural systems, which are capable of learning continuously, deep artificial networks have a limited ability for incorporating new information in an already trained network. As a result, methods for continuous learning are potentially highly impactful in enabling the application of deep networks to dynamic data sets. Here, inspired by the process of adult neurogenesis in the hippocampus, we explore the potential for adding new neurons to deep layers of artificial neural networks in order to facilitate their acquisition of novel information while preserving previously trained data representations. Our results on the MNIST handwritten digit dataset and the NIST SD 19 dataset, which includes lower and upper case letters and digits, demonstrate that neurogenesis is well suited for addressing the stability-plasticity dilemma that has long challenged adaptive machine learning algorithms.

The article suggests an algorithm of graphical image segmentation. The suggested algorithm uses a neural network to identify one particular pixel as belonged to the certain segment of an image. As a segmentation method, is used a method of growing areas based on comparing the nearest neighbors of one particular pixel. To make a decision about the similarity of two pixels, a three-layer perceptron is used. Three RGB color components are compared during processing. Thus, there are 6 neurons in the input layer of the neural network, namely 3 for the RGB component of the first pixel and 3 for the RGB component of the second one. In a specific implementation there are 50 neurons in the middle layer of the neural network. In the output layer of the neural network there are 2 neurons that represent similarity or difference of the comparing pixels. A training set of the neural network is formed using a specially generated impulse noise. There is a linear congruent generator of pseudo-random number used for noise generation. This generator is used to generate both the color and the coordinates of the noisy pixel. To form a training set, the certain number of noisy pixels is generated. In the article, this number is 10% of all pixels in the image. Then, for each damaged pixel, the training sets are formed so that all the nearest neighbors are considered to be in different clusters with a damaged pixel. A computer experiment was carried out both in automatic mode and in interactive one. The results of the experiment have shown that the algorithm provides fairly good training neural network for image segmentation without involving additional images.

The article focuses on studying the effectiveness of two different Hybrid Neural Networks (HNNs) architectures for solving real-world image classification problems. The first approach investigated in the research is a hybridization technique that allows creation of HNN based on a classical neural network by replacing a number of hidden layers of the neural network with a variational quantum circuit, which allows to reduce the complexity of the classical part of the neural network and move part of computations to a quantum device. The second approach is a hybridization technique based on utilizing quanvolutional operations for image processing as the first quantum convolutional layer of the hybrid neural network, thus building a Quanvolutional Neural Network (QNN). QNN leverages quantum phenomena to facilitate feature extraction, enabling the model to achieve higher accuracy metrics than its classical counterpart. The effectiveness of both architectures was tested on several image classification problems. The first one is a classical image classification problem of CIFAR10 images classification, widely used as a benchmark for various imagery-related tasks. Another problem used for the effectiveness study is the problem of geospatial data analysis. The second problem represents a real-world use case where quantum computing utilization can be very fruitful in the future. For studying the effectiveness, several models were assembled: HNN with a quantum device that replaces one of the hidden layers of the neural network, QNN based on quanvolutional operation and utilizes VGG-16 architecture as a classical part of the model, and also an unmodified VGG-16 was used as a reference model. Experiments were conducted to measure the models' key efficiency metrics: maximal accuracy, complexity of a quantum part of the model and complexity of a classical part of the model. The results of the research indicated the feasibility of both approaches for solving both proposed image classification problems. Results were analyzed to outline the advantages and disadvantages of every approach in terms of selected key metrics. Experiments showed that QNN architectures proved to be a feasible and effective solution for critical practical tasks requiring higher levels of model prediction accuracy and, simultaneously, can tolerate higher processing time and significantly increased costs due to a high number of quantum operations required. Also, the results of the experiments indicated that HNN architectures proved to be a feasible solution for time-critical practical tasks that require higher processing speed and can tolerate slightly decreased accuracy of model predictions.

The storage capacity of the complex phasor neural network

We study the absolute storage capacity of the q-state Potts-glass neural network which determines how many memory patterns can be really retrieved. By using theoretical analysis combined with a characteristic of the distribution of local field, a general formula for estimating the storage capacity is proposed, and dynamical simulations for q = 2 and q = 3 are presented for comparison. Compared with the previous theory, it is found that in the case of q = 2 our estimate is in good agreement with the simulation result, while for the case of q = 3 it provides a lower boundary of the storage capacity instead. The result may provide useful information for possible applications of neural networks.

A survey on deep learning for cybersecurity: Progress, challenges, and opportunities

A Statistical Approach to Learning and Generalization in Layered Neural Networks

In this study, it compares two different types of neural networks. First is a single layer neural network and other is multiple hidden layer neural network. For just comparisons it is ensured that both uses the same activation and output functions and have the same number of nodes and parameters. The networks are trained by the gradient descent algorithm to approximate linear and quadratic functions and examine their convergence properties. Here it is predicted as to how much a customer is willing to spend at a store using linear regression and different layers: single and multiple layers of neural network. Then comparison of linear regression with a single layer of neural network and single layer with multiple layers of network has done. Keywords : Comparisons, gradient descent algorithm, hidden layers, linear regression, neural networks Cite this Article Amin Deep Prakashbhai. Comparative Analysis of Neural Network and Linear Regression Applied to Black Friday Data. Current Trends in Signal Processing . 2019; 9(3): 1–4p.

This paper proposes a new universal linear transformation mechanism, inspired by the von Neumann-Morgenstern utility theory, known as the Von Neumann–Morgenstern Mechanism (VNM). This mechanism structurally reconstructs the linear layer in the neural network by introducing the "utility transformation" method to the traditional linear weights. Experiments on the image classification task CIFAR-10 have shown that the model using the VNM mechanism has obvious performance improvements over traditional methods in multiple evaluation indicators, showing stronger stability and generalization ability. This paper emphasizes that the mechanism has wide portability and is suitable for linear transformation modules in various neural network structures, providing a new idea for designing more effective deep learning models.

The digitized text of handwriting would conduce to automate the business processes of many companies, simplifying the work of human being. For example, our state postal service does not have an automated mail processing system that recognizes handwritten addresses on an envelope. Each incoming correspondence is registered in the system by the operator. Automation of this business process on registering post mailing will significantly reduce expenses of postal service on mail delivery.There are two main approaches to handwriting recognition, namely hidden Markov models (HMM) and artificial neural networks (ANN). The methods proposed in this article are based on ANN. The first model is based on deep convolutional neural networks (DCNN) [1] for feature extraction and a fully connected multilayer perceptron (fully connected MLP) for word classification. The next model under consideration, called SimpleHTR, proposed by Harald Scheidl [2], has layers of a convolutional neural network (CNN) and layers of a recurrent neural network (RNN) for disseminating information through an image. Finally, the Connectionist Temporal Classification (CTC) decoding algorithm is executed, which adduces the text to the final version.Models were learned on the dataset of handwritten city names from Cyrillic words. 21,000 images were collected (42 classes of 500 handwriting samples). To increase the data set for training, 207,438 images from available samples were generated.As a result, two approaches for handwriting recognition were analyzed and the SimpleHTR model showed the best results over all.

A training procedure that adapts the weights of a trained layered perceptron type artificial neural network to training data originating from a slowly varying nonstationary process is proposed. The procedure for adaptive dating assures proper response to previous training data by seeking to minimize a weight sensitivity cost function while, at the same time, minimizing the mean square error normally ascribed to the layered perceptron. The process is illustrated through application to an interpolation problem and by its use on an electric load forecasting problem with data collected from the power industry. >