Multivariate activation functions

We propose a new perceptron, called functional perceptron, which consists of a multi-dimensional activation function capable of learning a specific function. The functional perceptron does not use the traditional activation functions such as Sigmoid and ReLU. Instead, the proposed perceptron trains a function in a multi-dimensional space to accomplish a specific functionality and uses it as the learning task specific activation function. To realize this perceptron, we teach a comparison functionality to a multi-dimensional function by training two comparable inputs and producing a value of similarity as output. In order to show the efficacy of the functional perceptron, we apply the proposed perceptron to the XOR problem, the IRIS classification problem and an indoor positioning problem based on multi-signal fingerprints. Extensive experiments show that the proposed perceptron achieves about 96% accuracy in the IRIS classification and shows 1.737m accuracy in indoor positioning problem.

This paper shows that the performance of the Hopfield network for solving optimization problems can be improved by using a new activation (output) function. The effects of the activation function on the performance of the Hopfield network are analyzed. It is shown that the sigmoid activation function in the Hopfield network is sensitive to noise of neurons. The reason is that the sigmoid function is most sensitive in the range where noise is most predominant. A new activation function that is more robust against noise is proposed. The new activation function has the capability of amplifying the signals between neurons while suppressing noise. The performance of the new activation function is evaluated through simulation. Compared with the sigmoid function, the new activation function reduces the error rate of tour length by 30.6% and increases the percentage of valid tours by 38.6% during simulation on 200 randomly generated city distributions of the 10-city traveling salesman problem.

With the rapid development of artificial intelligence, deep learning is developing rapidly. As the main framework of in-depth learning, convolution neural network achieved much in semantic segmentation and image classification. Based on the low speed engineering vehicle tracking system project, this paper studies the activation function, which is an important component of convolution neural network. Firstly, it summarizes the advantages and disadvantages of four traditional activation functions such as Sigmoid, and then introduces three new activation functions: Swish, FTS, and Relu-Softplus. Finally, using the TensorFlow framework, a single new activation function and a hybrid new activation function are applied to the convolution layer of the four neural networks constructed, and the image recognition comparison is performed in the data set CIFAR-10. It is concluded that the recognition precision of mixing the new activation functions is higher than that of using a new activation function alone, and the average time is shorter and the convergence speed is faster.

Neural networks have been widely applied in system dynamics modeling. However, traditional neural networks still encounter limitations in capturing long-term dynamics, nonlinear modeling, and interpretability. To address these challenges, this study proposes a novel neural network architecture, Deep Stacked State-observer based Neural Network (DSSO-NN). Firstly, the state-space representation is introduced, integrating discretized state-space equations into the neural network design to leverage both system state information and deep learning capabilities. Subsequently, two optimization measures are employed to enhance the network's nonlinear modeling ability with activation functions and the state observer, respectively. Finally, DSSO-NN is validated using the Case Western Reserve University bearing dataset. Experimental results demonstrate that activation functions provide minimal improvement to model performance. In contrast, the incorporation of the state observer significantly enhances the DSSO-NN's ability to capture system dynamics behaviors and improves modeling accuracy. DSSO-NN exhibits higher precision and greater stability, offering a novel perspective on using the state observer as an alternative to traditional activation functions.

Multilevel image segmentation with adaptive image context based thresholding

The zeroing neural network (ZNN) model, a powerful approach for addressing time-varying problems, has been extensively applied in the calculation and optimization fields. In this article, a new pattern activation function, the power piecewise activation function (PPAF), is proposed to establish a predefined fixed-time convergent ZNN (PFTZNN) for finding solutions to the time-varying quadratic programming problem. In comparison with the traditional activation functions, multisegmentation is a remarkable feature of the PPAF; consequently, the advantage of PPAF is that its parameters can be flexibly adjusted according to actual needs. Specifically, because of the multisegment characteristics of the PPAF, the convergence speed of the PPAF-activated PFTZNN model can be flexibly adjusted based on distinct requirements. The fixed-time convergence property of the PPAF-activated PFTZNN model is validated by detailed mathematical theoretical analysis, and its upper bound convergence time is directly calculated. Then, the comparative simulation results of the PPAF-activated PFTZNN model with other existing ZNN models for time-varying quadratic programming are provided for the further verification of its superior convergence speed and robustness. In addition, the proposed PFTZNN model is applied for dual-arm manipulator cooperative trajectory tracking, and its practical application ability is demonstrated by united simulation experiments of MATLAB and Robot studio. Finally, the PFTZNN model is also applied to control a real dual-arm manipulator to complete the trajectory tracking task, which further validates its superior performance together and widespread applicability.

Approximate computing along with quantized low-precision computing has gained significant interest in today’s neural network (NN) implementation. This paper proposes a library of VLSI implementations of different activation functions, aimed towards designing hardware-efficient NN accelerators. Cartesian genetic programming (CGP), an evolutionary algorithm was employed to generate gate-level designs of approximate and exact representations of activation functions. We open-source the hardware library of 9444 circuits containing a majority of the activation functions employed in NN architectures, including Sigmoid, Hyperbolic-Tangent, Gaussian, ReLU, GeLU, Softplus, and Binary-Step. The library also presents the error characteristics and hardware metrics of the designs which will aid in the usage of the library in future research. Additionally a hardware comparison of the proposed circuits against existing implementations including piecewise-linear (PWL), memory-based, hls4ml, DNNweaver implementations to realize activation functions on FPGA and ASIC flow is presented. The CGP evolved hardware library shows minimal silicon space requirement, least power consumption when investigated for ASIC flow, and the least LUT utilization’s in FPGA flow. Besides, SOMALib designs are purely combinatorial, allowing various synthesis stage optimizations towards the target Power-Performance-Area budget, which is not possible in standard memory block implementations.

In recurrent neural networks such as the long short-term memory (LSTM), the sigmoid and hyperbolic tangent functions are commonly used as activation functions in the network units. Other activation functions developed for the neural networks are not thoroughly analyzed in LSTMs. While many researchers have adopted LSTM networks for classification tasks, no comprehensive study is available on the choice of activation functions for the gates in these networks. In this paper, we compare 23 different kinds of activation functions in a basic LSTM network with a single hidden layer. Performance of different activation functions and different number of LSTM blocks in the hidden layer are analyzed for classification of records in the IMDB, Movie Review, and MNIST data sets. The quantitative results on all data sets demonstrate that the least average error is achieved with the Elliott activation function and its modifications. Specifically, this family of functions exhibits better results than the sigmoid activation function which is popular in LSTM networks.

In this chapter, we will discuss the basic architecture of neural networks including activation functions, forward propagation, and backpropagation. We will also create a simple neural network model from scratch using the sigmoid activation function. In particular, this chapter will discuss: Neural network architecture. Activation functions used in neural networks. Forward propagation. Loss function of neural networks. Backpropagation.

Gaussian-type activation function with learnable parameters in complex-valued convolutional neural network and its application for PolSAR classification

Quantification of habitability is a complex task. Previous attempts at measuring habitability are well documented. Classification of exoplanets, on the other hand, is a different approach and depends on quality of training data available in habitable exoplanet catalogs. Classification is the task of predicting labels of newly discovered planets based on available class labels in the catalog. We present analytical exploration of novel activation functions as consequence of integration of several ideas leading to implementation and subsequent use in habitability classification of exoplanets. Neural networks, although a powerful engine in supervised methods, often require expensive tuning efforts for optimized performance. Habitability classes are hard to discriminate, especially when attributes used as hard markers of separation are removed from the data set. The solution is approached from the point of investigating analytical properties of the proposed activation functions. The theory of ordinary differential equations and fixed point are exploited to justify the “lack of tuning efforts” to achieve optimal performance compared to traditional activation functions. Additionally, the relationship between the proposed activation functions and the more popular ones is established through extensive analytical and empirical evidence. Finally, the activation functions have been implemented in plain vanilla feed-forward neural network to classify exoplanets. The mathematical exercise supplements the grand idea of classifying exoplanets, computing habitability scores/indices and automatic grouping of the exoplanets converging at some level.

The design of neural network models involves numerous complexities, including the determination of input vectors, choosing the number of hidden layers and their computational units, and specifying activation functions for the latter. The combinatoric possibilities are daunting, yet experience has yielded informal guidelines that can be useful. Alternatively, current research on genetic algorithms (GA) suggests that they might be of practical use as a formal method of determining ‘good’ architectures for neural networks. In this paper, we use a genetic algorithm to find effective architectures for backpropagation neural networks (BP). We compare the performance of heuristically designed BP networks with that of GA-designed BP networks. Our test domains are sets of problems having compensatory, conjunctive, and mixed-decision structures. The results of our experiment suggest that heuristic methods produce architectures that are simpler and yet perform comparatively well. © 1998 John Wiley & Sons, Ltd.

In order to solve the deep self-coding neural network training process, the Sigmoid function back-propagation gradient is easy to disappear, a method based on ReLU activation function is proposed for training the self coding neural network. This paper analyzes the performance of different activation functions and comparing ReLU with traditional Tanh and Sigmoid activation function and in Reuters-21578 standard for experiments on the test set. The experimental results show that using ReLU as the activation function, not only can improve the network convergence speed, and can also improve the accuracy.

Medical imaging is a technique to get images of the human body for medical science or clinical purposes. Segmentation of a medical image is a challenging task to isolate the suspicious region from the complex medical images. Genetic algorithms (GAs) are an effective tool to handle the problem of medical image segmentation. In this chapter, an application of color magnetic resonance (MR) brain image segmentation is presented by the parallel optimized multilevel sigmoidal (ParaOptiMUSIG) activation function with the parallel self-organizing neural network (PSONN) architecture. Not only confined within this approach, color MR brain image is also segmented by the NSGA-II-based ParaOptiMUSIG activation function to incorporate the multiple objective function-based scenario. These methods are compared with the process of color MR brain image segmentation by the MUSIG activation function with the PSONN architecture. All the methods are applied on a real-life color MR brain image and the quality of the segmented images are accessed by four standard objective functions. The comparison shows that the ParaOptiMUSIG activation function-based method and the NSGA-II-based ParaOptiMUSIG activation function-based method perform better than the MUSIG activation function-based method to segment the color MR brain image.

A novel architecture combined with optimal parameters for back propagation neural networks applied to anomaly network intrusion detection