Exploiting Space Folding by Neural Networks

Extreme weather modeling is a challenge for modeling experts in Indonesia and the world. Extreme weather prediction is a complex problem because the chances of it happening are very small, so the developed models often have a low level of accuracy. The purpose of this research is to combine the classic model, Autoregressive Integrated Moving Average (ARIMA), recurrent neural network (RNN) model using Adam and SGD estimation (RNN-Adam and RNN-SGD) with the reLU, tanh, sigmoid and gaussian activation functions. In addition, the ARIMA-RNN mix model was also demonstrated in this study. These models are applied to monthly period extreme weather data obtained from the Meteorology, Climatology and Geophysics Agency (BMKG) of West Sulawesi Province which are converted into training data and test data. The RMSE value is used to see the level of prediction accuracy in both training data and test data. Based on the research results, the best model obtained for modeling Indonesia’s extreme weather is the ARIMA-RNN-Adam mix model with the reLU activation function based on the RMSE value on the training and test data. At n = 50, the smallest RMSE and MSE values of the third model are the ARIMA-RNN-Adam model which is 0.23212 using the reLU activation function, then the ARIMA-RNN-SGD model which is 0.25432 with the same activation function, while the ARIMA value is 0.3270. At n=100 it can be seen that the smallest RMSE and MSE values of the three models are the ARIMA-RNN-Adam model which is equal to 0.25149 using the reLU activation function, then the ARIMA-RNN-SGD model which is equal to 0.25256 with the same activation function, while the ARIMA value is 0.2644.

Measuring the similarity or difference of two strings is a fundamental task in computer science and engineering. Among the various measures, Hamming distance and Levenshtein distance have been widely utilized. In this paper, we present neural networks for computing these distances, employing the Rectified Linear Unit function as the activation function. We show that the number of units in the networks is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O(n)</i> for the Hamming distance between two strings of length <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i>, and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O(mn)</i> for the Levenshtein distance between two strings of lengths <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i>. These complexities align with those of traditional algorithms. While the depth of the latter network is large (O(m+n)), we show that the depth can be significantly reduced at the expense of employing a larger number of units. Furthermore, we show that the networks can be modified for learning a string from a set of strings along with their respective distances to the target string. We performed computational experiments to evaluate the potential usufulness of our proposed networks. The results suggest that these networks can efficiently compute distances in a time proportional to the network size, and can proficiently learn a string when utilizing the Hamming distance.

A k-nearest neighbor (k-NN) query retrieves k objects from a database that are considered to be the closest to a given query point. Numerous techniques have been proposed in the past for supporting efficient k-NN searches in continuous data spaces. No such work has been reported in the literature for k-NN searches in a non-ordered discrete data space (NDDS). Performing k-NN searches in an NDDS raises new challenges. The Hamming distance is usually used to measure the distance between two vectors (objects) in an NDDS. Due to the coarse granularity of the Hamming distance, a k-NN query in an NDDS may lead to a large set of candidate solutions, creating a high degree of non-determinism for the query result. We propose a new distance measure, called granularity-enhanced Hamming (GEH) distance, that effectively reduces the number of candidate solutions for a query. We have also considered using multidimensional database indexing for implementing k-NN searches in NDDSs. Our experiments on synthetic and genomic data sets demonstrate that our index-based k-NN algorithm is effective and efficient in finding k-NNs in NDDSs.

This study discusses the negative impact of the derivative of the activation functions in the output layer of artificial neural networks, in particular in continual learning. We propose Hebbian descent as a theoretical framework to overcome this limitation, which is implemented through an alternative loss function for gradient descent we refer to as Hebbian descent loss. This loss is effectively the generalized log-likelihood loss and corresponds to an alternative weight update rule for the output layer wherein the derivative of the activation function is disregarded. We show how this update avoids vanishing error signals during backpropagation in saturated regions of the activation functions, which is particularly helpful in training shallow neural networks and deep neural networks where saturating activation functions are only used in the output layer. In combination with centering, Hebbian descent leads to better continual learning capabilities. It provides a unifying perspective on Hebbian learning, gradient descent, and generalized linear models, for all of which we discuss the advantages and disadvantages. Given activation functions with strictly positive derivative (as often the case in practice), Hebbian descent inherits the convergence properties of regular gradient descent. While established pairings of loss and output layer activation function (e.g., mean squared error with linear or cross-entropy with sigmoid/softmax) are subsumed by Hebbian descent, we provide general insights for designing arbitrary loss activation function combinations that benefit from Hebbian descent. For shallow networks, we show that Hebbian descent outperforms Hebbian learning, has a performance similar to regular gradient descent, and has a much better performance than all other tested update rules in continual learning. In combination with centering, Hebbian descent implements a forgetting mechanism that prevents catastrophic interference notably better than the other tested update rules. When training deep neural networks, our experimental results suggest that Hebbian descent has better or similar performance as gradient descent.

Protein structures can be encoded into binary sequences (Gabarro-Arpa et al., Comput Chem 2000;24:693-698) these are used to define a Hamming distance in conformational space: the distance between two different molecular conformations is the number of different bits in their sequences. Each bit in the sequence arises from a partition of conformational space in two halves. Thus, the information encoded in the binary sequences is also used to characterize the regions of conformational space visited by the system. We apply this distance and their associated geometric structures to the clustering and analysis of conformations sampled during a 4-ns molecular dynamics simulation of the HIV-1 integrase catalytic core. The cluster analysis of the simulation shows a division of the trajectory into two segments of 2.6 and 1.4 ns length, which are qualitatively different: the data points to the fact that equilibration is only reached at the end of the first segment. The Hamming distance is compared also to the r.m.s. deviation measure. The analysis of the cases studied so far shows that under the same conditions the two measures behave quite differently, and that the Hamming distance appears to be more robust than the r.m.s. deviation.

Convolution neural network takes in input image, assign importance (learnable weights and biases) to various aspects/objects in the image and be able to differentiate one from the other. At each layer, a linear transformation is carried out on the received data. Aiming for solving non-linear problems, a mandatory unit named activation function is used by neural networks. The activation function greatly influences the success of training deep neural networks. In this paper, a new rectified non linear function unit that we call Rectified non-linear unit (ReNU) is proposed. The presented activation function returns x − log(x + 1) for positive values, and zero for negative ones. The ReNU multiplies the received gradient by values between 0 and 1 depending on the importance of neurons ( 1 for the biggest neurons and 0 for the smaller ones) unlike the ReLU that returns the same received gradient for all positive values in its back propagation. Using the proposed activation function the CNN performance remarkably raises. The ReNU has been tested on MNIST and CIFAR-10 data sets and compared to the ReLU and ELU activation functions. The experimental results are satisfactory when compared to known activation functions in terms of convergence speed and CNN accuracy.

The fuzzy neural network technology is one of the hot topics of Data Mining. According to the Max Similarity Rule, this paper sets forth the cross entropy theory with formulae deduction in detail and a new activation function. Compare with the BP algorithm (error back propagation), which based on the error square sum rule and Sigmoid or Hyperbolical function, the classify algorithm based on the cross entropy theory and the new activation function can speed up the learning process and at same time without worry about to put it into the non-convergence state or lose in local small point. The new activation function not only has the value range from 0 to 1, but can also tune up the learning speed by adjusting its slope. So it can improve the algorithm's dynamic performance and make the processing of the FNN get into convergence as soon as possible, which can improve the algorithm's efficiency. In order to identify the user in the Web mining, an idea, which uses the biometric recognition technology, is put forward. At the same time, a method based on the Hidden Markov Model to build an iris identification system is proposed either. The robust of iris matching can be achieved by only depending on the orientation field of the iris and less sensitive to the noise and the distortions of the iris image than the conventional approaches in which need many iris details. This method is more efficient by sampling the pretreatment processes.

Artificial neural networks (ANN) have been used in many applications such like handwriting recognition and speech recognition. It is well-known that learning rate is a crucial value in the training procedure for artificial neural networks. It is shown that the initial value of learning rate can confoundedly affect the final result and this value is always set manually in practice. A new parameter called beta stabilizer has been introduced to reduce the sensitivity of the initial learning rate. But this method has only been proposed for deep neural network (DNN) with sigmoid activation function. In this paper we extended beta stabilizer to long short-term memory (LSTM) and investigated the effects of beta stabilizer parameters on different models, including LSTM and DNN with relu activation function. It is concluded that beta stabilizer parameters can reduce the sensitivity of learning rate with almost the same performance on DNN with relu activation function and LSTM. However, it is shown that the effects of beta stabilizer on DNN with relu activation function and LSTM are fewer than the effects on DNN with sigmoid activation function.

The estimation of state of charge (SOC) is a critical issue in the energy management of electric vehicle (EV) power batteries. However, the current accuracy of SOC estimation methods does not meet the requirements of practical applications. Therefore, this study proposes an improved lithium-ion battery SOC estimation method that combines deep residual shrinkage network (DRSN) and bidirectional gated recurrent unit (BiGRU) to enhance the SOC estimation accuracy. First, we insert the bidirectional gated recurrent unit neural network between the global average pooling layer and the output fully connected layer of the deep residual shrinkage network. This improvement enhances the model’s expressiveness, robustness, and data learning effect. Second, we develop a new activation function called “∂_swish” to replace the original ReLU activation function in the deep residual shrinkage network. The ∂_swish activation function improves the accuracy of the deep network model and reduces the risk of overfitting by utilizing its regularization effect. Finally, we conduct experimental tests at three different temperatures using the FUDS driving cycle dataset and the DST-US06-FUDS continuous driving cycle dataset. The algorithm model’s convergence speed is verified by comparing it with other models. The results show that compared to other models, the proposed method significantly improves SOC estimation accuracy at three different temperatures. In addition, the method demonstrates a high convergence speed.

This paper has made a survey on motor imagery EEG signals and different classifiers to analyze them. Resolution for medical images like CT, MRI can be improved using deep sense CNN and improved resolution technology. Drowsiness of a student can be analyzed using deep CNN and it helps in teaching, assessment of the student. The authors have proposed 1D-CNN with 2 layers and 3 layers architecture to classify EEG signal for eyes open and eyes closed conditions. Various activation functions and combinations are tried for 2-layer 1D-CNN. Similarly, various loss models are applied in compile model to check the CNN performance. Simulation is carried out using Python 2.7 and 1D-CNN with 3 layers show better performance as it increases number of training parameters by increasing number of layers in the architecture. Accuracy and kappa coefficient increase whereas hamming loss and logloss decreases by increasing number of layers in CNN architecture.

This paper presents a complex-valued neuron (CVN) model for real-valued classification problems incorporating a new activation function. The activation function maps complex-valued net-inputs (sum of weighted inputs) of a neuron into bounded real-values, and its role is to divide the net-input space into different regions for different classes. A gradient-descent learning rule has been derived to train the CVN. Such a CVN is able to solve all possible two-input Boolean functions. For further investigation, single layered complex-valued neural networks (Le. without hidden units) are applied on the real-world multi-class classification problems. The results are comparable to the conventional multilayer real-valued neural networks. It is also shown that the performance can be improved further by using their ensembles. Negative correlation learning (NCL) algorithm has been used to create the ensembles. Since NCL is a gradient-descent based algorithm, the proposed activation function is well suited for it.

Road surface condition classification using deep learning

The desire to provide robust guarantees on neural networks has never been more important, as their prevalence in society is increasing. One popular method that has seen a large amount of success is to use bounds on the activation functions within these networks to provide such guarantees. However, due to the large number of possible ways to bound the activation functions, there is a trade-off between conservativeness and complexity. We approach the problem from a different perspective, using polynomial optimisation and real algebraic geometry (the Positivstellensatz) to assert the emptiness of a semi-algebraic set. We show that by using the Positivstellen- satz, bounds on the robustness guarantees can be tightened significantly over other popular methods, at the expense of computational resource. We demonstrate the effectiveness of this approach on networks that use the ReLU, sigmoid and tanh activation functions. This method can be extended to more activation functions, and combined with recent sparsity- exploiting methods can result in a computationally acceptable method for verifying neural networks.

SummaryThe present existing weather prediction systems deployed by weather forecasting agencies predict the anomalous events using numerical simulation models developed based on geophysics and oceanographic theories. The major disadvantage with existing models is that are explicit simulation models. So, we try to build a real time prediction system that will automatically predict the intensity of any marine natural disaster events and alert the authorities to take necessary actions using a satellite and observational real time data set including moored met ocean data. We propose a novel hybrid model which is a combination of Convolutional Neural Network (CNN) and Long Short‐Term Memory (LSTM). The paper analyzes various other models such as Gated Recurrent Unit (GRU), Bidirectional LSTM (Bi‐LSTM) Recurrent Neural Network (RNN), and Extreme Learning Machine. In addition the other standard CNN architectures are also analyzed. Since the residual CNN network is the most effective architecture, the paper proposes a modification of the residual CNN network by combining the residual CNN network with LSTM/GRU RNN network along with a proposed new activation function which was experimentally proved to be more effective than the deep learning community standard activation function ReLU. The proposed new activation function overcomes the dead neuron propagation problem in ReLU activation function and hence the model's accuracy is increased by more than 1%. The proposed CNN + LSTM/GRU model is benchmarked with UCR time series benchmark dataset and it is experimentally seen that the model performs better than all existing algorithms.

In order to solve the deep self-coding neural network training process, the Sigmoid function back-propagation gradient is easy to disappear, a method based on ReLU activation function is proposed for training the self coding neural network. This paper analyzes the performance of different activation functions and comparing ReLU with traditional Tanh and Sigmoid activation function and in Reuters-21578 standard for experiments on the test set. The experimental results show that using ReLU as the activation function, not only can improve the network convergence speed, and can also improve the accuracy.