SAU: Smooth Activation Function Using Convolution with Approximate Identities

Recently, Deep learning has made a great deal of success in processing images, audios, and natural languages and so on. The activation function is one of the key factors in Deep learning. In this paper, according to characteristics of biological neurons, an improved Leaky Single-Peaked Triangle Linear Unit (LSPTLU) activation function is presented for the right-hand response unbounded of Rectified Linear Unit (ReLU) and Leaky ReLU (LReLU). LSPTLU is more in line with the biological neuron essence and achieves the excellent performance of equivalent or beyond ReLU and LReLU on different datsets, e.g., MNIST, Fashion-MNIST, SVHN, IMAGENET, CALTECH101 and CIFAR10 datasets.

PurposeThe purpose of this study is to explore the potential of trainable activation functions to enhance the performance of deep neural networks, specifically ResNet architectures, in the task of image classification. By introducing activation functions that adapt during training, the authors aim to determine whether such flexibility can lead to improved learning outcomes and generalization capabilities compared to static activation functions like ReLU. This research seeks to provide insights into how dynamic nonlinearities might influence deep learning models' efficiency and accuracy in handling complex image data sets.Design/methodology/approachThis research integrates three novel trainable activation functions – CosLU, DELU and ReLUN – into various ResNet-n architectures, where “n” denotes the number of convolutional layers. Using CIFAR-10 and CIFAR-100 data sets, the authors conducted a comparative study to assess the impact of these functions on image classification accuracy. The approach included modifying the traditional ResNet models by replacing their static activation functions with the trainable variants, allowing for dynamic adaptation during training. The performance was evaluated based on accuracy metrics and loss profiles across different network depths.FindingsThe findings indicate that trainable activation functions, particularly CosLU, can significantly enhance the performance of deep learning models, outperforming the traditional ReLU in deeper network configurations on the CIFAR-10 data set. CosLU showed the highest improvement in accuracy, whereas DELU and ReLUN offered varying levels of performance enhancements. These functions also demonstrated potential in reducing overfitting and improving model generalization across more complex data sets like CIFAR-100, suggesting that the adaptability of activation functions plays a crucial role in the training dynamics of deep neural networks.Originality/valueThis study contributes to the field of deep learning by introducing and evaluating the impact of three novel trainable activation functions within widely used ResNet architectures. Unlike previous works that primarily focused on static activation functions, this research demonstrates that incorporating trainable nonlinearities can lead to significant improvements in model performance and adaptability. The introduction of CosLU, DELU and ReLUN provides a new pathway for enhancing the flexibility and efficiency of neural networks, potentially setting a new standard for future deep learning applications in image classification and beyond.

Price movement predictions of financial instruments using traditional time-series models with a predefined mathematical structure are common, but this method restricts their ability to learn latent patterns in the data. In recent times, artificial neural networks (ANN) have been able to learn complex hidden patterns from financial datasets using a highly nonlinear architecture. However, most experiments with neural networks require a lot of time to search a suitable network and subsequently train the network. The authors have developed a deep multilayer perceptron (MLP) classifier with a zero-centered piecewise linear unit activation that yielded better classification performance according to their accuracy metric and required consistently less training time compared to a similar MLP network with leaky rectified linear unit and tanh activation functions. The authors illustrate their technique with a large high-frequency dataset on selected bank shares from the Indian stock market. The authors also discuss the theoretical properties and advantages of their proposal. <b>TOPICS:</b>Statistical methods, simulations, big data/machine learning <b>Key Findings</b> • Time-series models have been used extensively to understand stock price movements. They are constrained by a predefined structure. Artificial neural network (ANNs), due to their universal approximation properties, offer more flexibility; however, they suffer from long training times. • In this article, the authors introduce a zero-centered piecewise linear unit (PLU), a hybrid activation function derived from tanh and leaky rectified linear unit (ReLU) functions that exhibits the zero-centered property from tanh and the nonzero gradient property from leaky ReLU. • The authors developed a multilayer perceptron (MLP) classifier with PLU activation, which yielded better classification performance according to the accuracy metric and required consistently less training time compared to a similar MLP network with leaky ReLU and tanh activation.

In this article, we propose a generalization of the batch normalization (BN) algorithm, diminishing BN (DBN), where we update the BN parameters in a diminishing moving average way. BN is very effective in accelerating the convergence of a neural network training phase that it has become a common practice. Our proposed DBN algorithm retains the overall structure of the original BN algorithm while introducing a weighted averaging update to some trainable parameters. We provide an analysis of the convergence of the DBN algorithm that converges to a stationary point with respect to the trainable parameters. Our analysis can be easily generalized to the original BN algorithm by setting some parameters to constant. To the best of our knowledge, this analysis is the first of its kind for convergence with BN. We analyze a two-layer model with arbitrary activation functions. Common activation functions, such as ReLU and any smooth activation functions, meet our assumptions. In the numerical experiments, we test the proposed algorithm on complex modern CNN models with stochastic gradients (SGs) and ReLU activation on regression, classification, and image reconstruction tasks. We observe that DBN outperforms the original BN algorithm and benchmark layer normalization (LN) on the MNIST, NI, CIFAR-10, CIFAR-100, and Caltech-UCSD Birds-200-2011 datasets with modern complex CNN models such as Resnet-18 and typical FNN models.

The distinguishing characteristics belong to a person are personality traits, which is predicted from person’s behavioral pattern. As most of the people provide lots of information knowingly or unknowingly into their writings, it is possible to extract personality traits from those texts. Individual personality traits detection from texts, yields enormous possibilities toward various applications named as forensic department, mental health diagnosis, etc. Meanwhile, deep learning algorithm performs fairly well in text based personality detection; however, its performance may vary with activation functions. Hence, this paper proposed an empirical approach to find the best personality detection performance by comparing several activation functions named as sigmoid, tanh, and leaky ReLU. Here, text documents were pre-processed and vectorized for input in convolutional neural network. The input size was multiple to length of word, sentence, documents, and feature vectors. Five personality traits named as EXT, NEU, AGR, CON, and OPN have been used for experimental analysis. The result showed that tanh and leaky ReLU performs over sigmoid in all datasets. The average F1-score of sigmoid, tanh and leaky ReLU showed 33.11%, 47.25%, and 49.07% respectively. However, Fl-score of leaky ReLU was high only for CON, tanh showed better result for others datasets. The overall performance showed by tanh is better than sigmoid and leaky ReLU for personality detection from text.

Different parameters of a fully convolutional network (FCN) are experimented to evaluate which combination predicts sound velocity models from a single configuration of seismic modeling. The evaluation is made considering some fixed parameters of the deep learning model, such as number of epochs, batch size and loss function, but with variations of the optimizer and activation function. The considered optimizers were RMSprop, Adam and Adamax, whilst the activations functions were the Rectified Linear Unit (ReLU), Leaky ReLU, Exponential Linear Unit (ELU) and Parametric ReLU (PReLU). Five metrics were used to evaluate the model during the testing stage: R2, Pearson's r, factor of two, mean absolute error and mean squared error. To the extent of these experiments, it was found that the optimizers have much more influence than the activation functions when determining the resolution of the output model. The best combination was the one using the PReLU activation function with the Adamax optimizer.

Alzheimer's disease (AD) is a progressive brain disease. The goal of this study is to provide a new computer-vision based technique to detect it in an efficient way. The brain-imaging data of 98AD patients and 98 healthy controls was collected using data augmentation method. Then, convolutional neural network (CNN) was used, CNN is the most successful tool in deep learning. An 8-layer CNN was created with optimal structure obtained by experiences. Three activation functions (AFs): sigmoid, rectified linear unit (ReLU), and leaky ReLU. The three pooling-functions were also tested: average pooling, max pooling, and stochastic pooling. The numerical experiments demonstrated that leaky ReLU and max pooling gave the greatest result in terms of performance. It achieved a sensitivity of 97.96%, a specificity of 97.35%, and an accuracy of 97.65%, respectively. In addition, the proposed approach was compared with eight state-of-the-art approaches. The method increased the classification accuracy by approximately 5% compared to state-of-the-art methods.

This paper considers the interconnection of a continuous time linear time-invariant system and a multivariable magnitude nonlinearity. A number of different quadratic constraints are established for the magnitude nonlinearity and then used to derive stability criteria based on quadratic and Lurie-type Lyapunov functions. The new stability criteria are cast as matrix inequalities and in some cases solved using semi-definite programming. Connections are made between the magnitude nonlinearity and neural network activation functions, such as the rectified linear unit (ReLU) and leaky ReLU, effectively allowing the stability criteria derived here to be used to analyse interconnections of dynamical systems and neural networks. Using the positive homogeneity property, shared by the (leaky) ReLU and magnitude functions, mild conditions are also established to show that the existence of a unique equilibrium point is sufficient for local and global stability to be equivalent. Finally, the new global stability criteria are tested on several numerical examples, including a Hopfield network with 100 states and neurons, and compare favourably with competing criteria from the literature.

With the rapid development of artificial intelligence, deep learning is developing rapidly. As the main framework of in-depth learning, convolution neural network achieved much in semantic segmentation and image classification. Based on the low speed engineering vehicle tracking system project, this paper studies the activation function, which is an important component of convolution neural network. Firstly, it summarizes the advantages and disadvantages of four traditional activation functions such as Sigmoid, and then introduces three new activation functions: Swish, FTS, and Relu-Softplus. Finally, using the TensorFlow framework, a single new activation function and a hybrid new activation function are applied to the convolution layer of the four neural networks constructed, and the image recognition comparison is performed in the data set CIFAR-10. It is concluded that the recognition precision of mixing the new activation functions is higher than that of using a new activation function alone, and the average time is shorter and the convergence speed is faster.

Two different approaches are employed to investigate the global attractivity of delayed Hopfield neural network models. Without assuming the monotonicity and differentiability of the activation functions, Liapunov functionals and functions (combined with the Razumikhin technique) are constructed and employed to establish sufficient conditions for global asymptotic stability independent of the delays. In the case of monotone and smooth activation functions, the theory of monotone dynamical systems is applied to obtain criteria for global attractivity of the delayed model. Such criteria depend on the magnitude of delays and show that self-inhibitory connections can contribute to the global convergence.

Convolution neural network takes in input image, assign importance (learnable weights and biases) to various aspects/objects in the image and be able to differentiate one from the other. At each layer, a linear transformation is carried out on the received data. Aiming for solving non-linear problems, a mandatory unit named activation function is used by neural networks. The activation function greatly influences the success of training deep neural networks. In this paper, a new rectified non linear function unit that we call Rectified non-linear unit (ReNU) is proposed. The presented activation function returns x − log(x + 1) for positive values, and zero for negative ones. The ReNU multiplies the received gradient by values between 0 and 1 depending on the importance of neurons ( 1 for the biggest neurons and 0 for the smaller ones) unlike the ReLU that returns the same received gradient for all positive values in its back propagation. Using the proposed activation function the CNN performance remarkably raises. The ReNU has been tested on MNIST and CIFAR-10 data sets and compared to the ReLU and ELU activation functions. The experimental results are satisfactory when compared to known activation functions in terms of convergence speed and CNN accuracy.

The choice of activation function can have a large effect on the performance of a neural network. While there have been some attempts to hand-engineer novel activation functions, the Rectified Linear Unit (ReLU) remains the most commonly-used in practice. This paper shows that evolutionary algorithms can discover novel activation functions that outperform ReLU. A tree-based search space of candidate activation functions is defined and explored with mutation, crossover, and exhaustive search. Experiments on training wide residual networks on the CIFAR-10 and CIFAR-100 image datasets show that this approach is effective. Replacing ReLU with evolved activation functions results in statistically significant increases in network accuracy. Optimal performance is achieved when evolution is allowed to customize activation functions to a particular task; however, these novel activation functions are shown to generalize, achieving high performance across tasks. Evolutionary optimization of activation functions is therefore a promising new dimension of metalearning in neural networks.

Currently, we are dealing with a very limited set of activation functions such as Sigmoid, ReLu, Leaky ReLu among others. These activation functions used in the existing digital brain neural network systems are chosen using assumption with the help of “trial and error” approach. However, they do not ethically and appropriately establish any relationship with the referenced AI datasets. Jamilu (2019) proposed that a digital brain should have at least 2000 to 100 billion distinct activation functions implying distinct artificial neurons satisfies Jameel’s criterion(s) for it to normally mimic the human brain. The objectives of this paper are to propose a theorem called “Digital Brain Completeness Theorem”, “superintelligent digital brain neural network systems” and why it is tremendously important to have an extremely huge distinct activation functions implying distinct artificial neurons in a digital brain just like in the case of its counterpart for it to function rationally.

The convolutional neural networks (CNN), including AlexNet, GoogleNet, VGGNet, etc. extract features for many computer vision problems which are very discriminative. The trained CNN model over one dataset performs reasonably well whereas on another dataset of similar type the hand-designed feature descriptor outperforms the same trained CNN model. The Rectified Linear Unit (ReLU) layer discards some values in order to introduce the non-linearity. In this paper, it is proposed that the discriminative ability of deep image representation using trained model can be improved by Average Biased ReLU (AB-ReLU) at the last few layers. Basically, AB-ReLU improves the discriminative ability in two ways: 1) it exploits some of the discriminative and discarded negative information of ReLU and 2) it also neglects the irrelevant and positive information used in ReLU. The VGGFace model trained in MatConvNet over the VGG-Face dataset is used as the feature descriptor for face retrieval over other face datasets. The proposed approach is tested over six challenging, unconstrained and robust face datasets (PubFig, LFW, PaSC, AR, FERET and ExtYale) and also on a large scale face dataset (PolyUNIR) in retrieval framework. It is observed that the AB-ReLU outperforms the ReLU when used with a pre-trained VGGFace model over the face datasets. The validation error by training the network after replacing all ReLUs with AB-ReLUs is also observed to be favorable over each dataset. The AB-ReLU even outperforms the state-of-the-art activation functions, such as Sigmoid, ReLU, Leaky ReLU and Flexible ReLU over all seven face datasets.

On robustness of neural ODEs image classifiers