Seq2Seq-Attention based text topic representation

We study the natural function space for infinitely wide two-layer neural networks with ReLU activation (Barron space) and establish different representation formulae. In two cases, we describe the space explicitly up to isomorphism. Using a convenient representation, we study the pointwise properties of two-layer networks and show that functions whose singular set is fractal or curved (for example distance functions from smooth submanifolds) cannot be represented by infinitely wide two-layer networks with finite path-norm. We use this structure theorem to show that the only $$C^1$$ -diffeomorphisms which preserve Barron space are affine. Furthermore, we show that every Barron function can be decomposed as the sum of a bounded and a positively one-homogeneous function and that there exist Barron functions which decay rapidly at infinity and are globally Lebesgue-integrable. This result suggests that two-layer neural networks may be able to approximate a greater variety of functions than commonly believed.

Accurate forecasting of high-risk clinical symptoms, like epileptic seizures, has the potential to transform clinical epilepsy care and to create new therapeutic strategies for individuals in clinical decision support systems. With the development of pervasive sensor technologies, physiological signals can be captured continuously to prevent the serious outcomes caused by epilepsy. However, the progress on seizure prediction has been hindered by the lack of automatic early warning system. The existing research is classifying electroencephalograph (EEG) clips and is distinguishing the clips of onset epileptic seizures. Deep learning is a promising method to analyze the large-scale unlabeled data and to widely spread the clinical treatment and risk prediction. In this paper, we outline a patient-specific method for extracting the frequency domain and time-series data features based on the two-layer convolutional neural networks (CNNs). A data preprocessing method based on the discrete Fourier transform is proposed to convert the time-domain signal of the EEG data to the frequency-domain signal. Long short-term memory networks are introduced in seizure prediction using pre-seizure clips of the EEG dataset, expanding the use of deep learning algorithms with recurrent neural networks (RNNs). Furthermore, the proposed CNN and RNN are compared with the traditional machine learning algorithms, such as linear discriminant analysis and logistic regression, and the evaluation criteria are on the area under the curve. The extensive experimental results demonstrate that our method can effectively extract the latent features with meaningful interpretation and exhibits excellent performance for predicting epileptic preictal state changes, and hence is an effective method in detecting the epileptic seizure.

A neural network controller which is used for controlling unknown discrete-time DARMA systems is described. In a first stage, a two-layered neural network is used to estimate the unknown plant dynamics. The Widrow-Hoff delta rule is used as the learning algorithm for this network so as to minimize the difference between the plant actual response and that predicted by the neural network. In a second stage, the control law is generated online using a second two-layered neural network so that the plant output is brought to a desired reference signal. Simulation examples are presented to evaluate the design.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Efficient land management and farming practices are critical to maintaining agricultural production, especially in Europe with limited arable land. It is very time consuming to rely on a manual field inspection of cultivated land to archive farm crops. But with the help of satellite monitoring data on the earth’s surface, it is a new vision to classify farmland based on deep learning. This article has studied the Sentinel 2 (S2) data, which are top-of-atmosphere (TOA) reflectance values at the processing level-1C (L1C) observed from some areas of Germany and France. Aiming at the problem that the interference of atmosphere and cloud coverage weakens the recognition accuracy of subsequent algorithms, a method of combining feature expansion and feature importance analysis is proposed to optimize the raw S2 data. Specifically, the new 13 spectral features are expanded based on the linear and nonlinear combination of the raw 13 spectral bands of S2. The random forest (RF) algorithm is used to score the importance of features, and the important features of each time series are selected to form a new dataset. Then, an end-to-end deep learning model has been used for training. The structure of the model is a two-layer unidirectional recurrent neural network with long short-term memory (LSTM) as the backbone. And two linear layers as the output, which form two decision-making heads, respectively, representing output classification probability and the stop decision. The results show that adding features and selecting features is beneficial for the model to improve classification accuracy and predict the classification without all of the input data. This end-to-end classification pattern with early prediction would support intelligent monitoring of farm crops with a great advantage to the implementation of various agricultural policies.

Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher–student setup, where one network, the student, is trained on data generated by another network, called the teacher. We show how the dynamics of stochastic gradient descent (SGD) is captured by a set of differential equations and prove that this description is asymptotically exact in the limit of large inputs. Using this framework, we calculate the final generalisation error of student networks that have more parameters than their teachers. We find that the final generalisation error of the student increases with network size when training only the first layer, but stays constant or even decreases with size when training both layers. We show that these different behaviours have their root in the different solutions SGD finds for different activation functions. Our results indicate that achieving good generalisation in neural networks goes beyond the properties of SGD alone and depends on the interplay of at least the algorithm, the model architecture, and the data set.

The lower and upper bounds for the information capacity of two-layer feedforward neural networks with binary interconnections, integer thresholds for the hidden units, and zero threshold for the output unit is obtained through two steps. First, through a constructive approach based on statistical analysis, it is shown that a specifically constructed (N-2L-1) network with N input units, 2L hidden units, and one output unit is capable of implementing, with almost probability one, any dichotomy of O(W/ln W) random samples drawn from some continuous distributions, where W is the total number of weights of the network. This quantity is then used as a lower bound for the information capacity C of all (N-2L-1) networks with binary weights. Second, an upper bound is obtained and shown to be O(W) by a simple counting argument. Therefore, we have /spl Omega/(W/ln W)/spl les/C/spl les/O(W).

The momentum method has become the workhorse in the deep learning community. To theoretical understand its success, researchers put efforts in demystifying its convergence properties when optimizing neural networks. For the convex problem, it is well-known that the triple momentum (TM) method owns the fastest theoretical convergence rate among all first-order methods. However, there exists no theoretical convergence results about the TM method in solving the non-convex neural networks training problem, let alone its acceleration guarantee. In this paper, we focus on the training process of the TM method for a two-layer ReLU neural network. Inspired by the accurate characterization of the high-resolution dynamical system, we consider the high-resolution ordinary differential equation (ODE) of the TM method. Under the over-parameterized assumption, we derive that the original non-convex optimization problem can be transformed to a strongly convex task. By applying an appropriate Lyapunov function, we prove that the TM method can linearly converge to a global minimum. Compared to the heavy ball method and Nesterov’s accelerated gradient method, our result provides the first guarantee for the acceleration of the TM method in training neural networks. Through empirical experiments, the accelerated convergence of the TM method and the effect of over-parameterization are validated.

The key of fast identification algorithm of time-varying modal parameter based on subspace tracking is to find efficient and fast subspace-tracking algorithm.This paper presents a new version of NIC(Novel Information Criterion) using two-layer linear neural network learning for subspace tracking.Comparing with the original algorithm, there is no need to set a key control parameter in advance.Simulation experiments show that new algorithm has a faster convergence in the initial period.

A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated. General initialization schemes as well as general regimes for the network width and training data size are considered. In the over-parametrized regime, it is shown that gradient descent dynamics can achieve zero training loss exponentially fast regardless of the quality of the labels. In addition, it is proved that throughout the training process the functions represented by the neural network model are uniformly close to that of a kernel method. For general values of the network width and training data size, sharp estimates of the generalization error is established for target functions in the appropriate reproducing kernel Hilbert space.

Phishing remains a continual security threat, causing global losses exceeding 3.5 billion USD in 2019, according to the FBI’s Internet Crime Complaint Center. The Anti-Phishing Working Group (APWG) reported as many as 2,172 unique phishing websites detected per day in 2019. Most of the methods to solve the phishing websites’ detection problem proposed by the scientific community are based on classical classification algorithms on phishing datasets with hand-extracted features. Although these methods demonstrate high accuracies, unfortunately, they are sensitive to changing environment: phishers can learn the most relevant URL features and adapt their attacks to overcome the security check. Therefore, in search of less sensitive methods, deep neural networks were started to employ, as they do not require manual feature extraction and can directly learn a representation from the URL’s sequence of characters. The purpose of this research is to propose a new method for phishing websites’ URL detection based on ensembles of Recurrent neural networks and other types of deep neural networks. The results of our approach are presented in this paper and compared with the performance of other Recurrent neural networks. These results are additionally compared with the performance of classical classification algorithms on the same dataset with 48 features extracted. Our method with no manually extracted feature gives a significant increase in classification accuracy, compared with single Recurrent neural networks, and matches the accuracy of classical classification ensembles with manually extracted features.

The study presents the results of the development and testing of deep learning neural network architectures, which demonstrate high accuracy rates in classifying neurophysiological data, in particular hemodynamic brain activation patterns obtained by functional near-infrared spectroscopy, during solving mathematical problems on spatial-numerical associations. The analyzed signal represents a multidimensional time series of oxyhemoglobin and deoxyhemoglobin dynamics. Taking the specificity of the fNIRS signal into account, a comparative analysis of 2 types of neural network architectures was carried out: (1) architectures based on recurrent neural networks: recurrent neural network with long short-term memory, recurrent neural network with long short-term memory with fully connected layers, bidirectional recurrent neural network with long short-term memory, convolutional recurrent neural network with long short-term memory; (2) architectures based on convolutional neural networks with 1D convolutions: convolutional neural network, fully convolutional neural network, residual neural network. Trained long short-term memory recurrent neural network architectures showed worse results in accuracy in comparison with 1D convolutional neural network architectures. Residual neural network (model_Resnet) demonstrated the highest accuracy rates in three experimental conditions more than 88% in detecting age-related differences in brain activation during spatial-numerical association tasks considering the individual characteristics of the respondents’ signal.

Faults that are often invisible to existing inspection methods can significantly impact the reliability and efficiency of Photovoltaic (PV) systems. This study introduces an innovative framework for identifying PV panel faults using advanced machine learning algorithms and thermal images captured by unmanned aerial vehicles. The proposed Deep Recurrent Convolutional Neural Network (DRCNN) combines the strengths of Recurrent Neural Networks (RNNs) for learning temporal sequences and Convolutional Neural Networks (CNNs) for feature extraction. The DRCNN is designed to analyze thermal images of PV panels by capturing spatial and temporal patterns associated with fault-induced anomalies, such as hot spots, cracked cells or broken electrical connections. By leveraging CNN layers to extract critical features from thermal images and RNN layers to model the temporal dependencies and variations in thermal processes over time, the proposed approach integrates thermal image data collected under diverse environmental conditions.This dual strategy not only identifies faults based on spatial features but forecasts potential fault progression or the End-of-Life (EoL) for the panels by analyzing the evolution of thermal anomalies. The research aims to develop a highly accurate and efficient system for preventive maintenance and early problem detection, thereby improving the efficiency and durability of PV systems.The results demonstrate the DRCNN model's effectiveness in detecting various types of defects and delivering highly accurate real-time fault detection. By incorporating both spatial and temporal information, the proposed model significantly outperforms existing methods providing more robust and reliable defect identification.

To improve the accuracy of lane-change intention prediction and analyze the influence of driving styles on prediction outcomes, the T-Encoder-Sequence model is proposed in this paper. It integrates the Transformer’s encoder module with various recurrent neural network (RNN) models and introduces a multimodal fusion input structure. Building on this, a risk indicator model, capable of reflecting driver stress, is established to calculate the model’s input parameters. Consequently, the model can simultaneously capture global information and consider the impact of vehicle classes on drivers. Furthermore, the k-means++ algorithm is employed to categorize vehicle trajectories into conservative, conventional, and aggressive types for further analysis. The results demonstrate that training the model with risk indicator parameters markedly enhances prediction performance. Under identical input parameters, the T-Encoder-Sequence model exhibits notably superior prediction efficacy compared to the original model. The T-Encoder-Sequence model, trained with risk indicator parameters, demonstrates substantial advantages compared to other studies.

The de novo drug design based on SMILES format is a typical sequence-processing problem. Previous methods based on recurrent neural network (RNN) exhibit limitation in capturing long-range dependency, resulting in a high invalid percentage in generated molecules. Recent studies have shown the potential of Transformer architecture to increase the capacity of handling sequence data. In this work, the encoder module in the Transformer is used to build a generative model. First, we train a Transformer-encoder-based generative model to learn the grammatical rules of known drug molecules and a predictive model to predict the activity of the molecules. Subsequently, transfer learning and reinforcement learning were used to fine-tune and optimize the generative model, respectively, to design new molecules with desirable activity. Compared with previous RNN-based methods, our method has improved the percentage of generating chemically valid molecules (from 95.6 to 98.2%), the structural diversity of the generated molecules, and the feasibility of molecular synthesis. The pipeline is validated by designing inhibitors against the human BRAF protein. Molecular docking and binding mode analysis showed that our method can generate small molecules with higher activity than those carrying ligands in the crystal structure and have similar interaction sites with these ligands, which can provide new ideas and suggestions for pharmaceutical chemists.

In this paper, a Stator Current Model Reference Adaptive System (SC_MRAS) based estimator using neural network for sensorless controlled Six Phase Induction Motor (SPIM) drive is presented. For this scheme the measured stator current components are used as the reference model of the MRAS estimator to avoid the use of a pure integrator and reduce influence of motor parameter variation. The adaptive model of proposal estimator use a two-layer Neural Network (NN) to estimate the stator current, which requires the rotor flux information that can be obtained from the voltage model. A back-propagation learning algorithm is used to minimise the error in current estimation and hence in generating the estimated speed. Speed estimation performance of the proposal MRAS scheme is verified and compared with the classical rotor flux MRAS when applied to sensorless vector control SPIM drive. Simulation results have demonstrated that the proposed MRAS estimator for sensorless control of SPIM drive has good dynamic response, high precision in both the transient and steady modes, the performance of the SC_MRAS using NN is significantly improved especially at low and near zero speed range.