SOMALib: Library of Exact and Approximate Activation Functions for Hardware-efficient Neural Network Accelerators

Multilevel image segmentation with adaptive image context based thresholding

This letter presents a high frame rate and low area cost partial histogram scheme for d-ToF depth imaging sensor based on SPAD. This scheme realizes high frame rate through a novel peak seeking algorithm based on gray code and pipeline operation that firstly adopted. Area cost is reduced by partial histogram and overlap is used to prevent data loss. Finally, a 24 bins partial-histogram design is implemented by ASIC flow with a total area of 11724 μm2 under 130 nm. FPGA flow is also used to verify feasibility. Compared with other known partial histogram schemes, proposed method has no data and frame rate loss.

Neural networks have been widely applied in system dynamics modeling. However, traditional neural networks still encounter limitations in capturing long-term dynamics, nonlinear modeling, and interpretability. To address these challenges, this study proposes a novel neural network architecture, Deep Stacked State-observer based Neural Network (DSSO-NN). Firstly, the state-space representation is introduced, integrating discretized state-space equations into the neural network design to leverage both system state information and deep learning capabilities. Subsequently, two optimization measures are employed to enhance the network's nonlinear modeling ability with activation functions and the state observer, respectively. Finally, DSSO-NN is validated using the Case Western Reserve University bearing dataset. Experimental results demonstrate that activation functions provide minimal improvement to model performance. In contrast, the incorporation of the state observer significantly enhances the DSSO-NN's ability to capture system dynamics behaviors and improves modeling accuracy. DSSO-NN exhibits higher precision and greater stability, offering a novel perspective on using the state observer as an alternative to traditional activation functions.

This paper presents a FPGA based approach for a modular architecture of Fuzzy Neural Networks (FNN) to embed with easily different topologies set up. The project is based on a Takagi - Hayashi (T-H) method for the construction and tuning of fuzzy rules, this is commonly referred as neural network driven fuzzy reasoning. The proposed architecture approach consists of two main configurable modules: a Multilayer Perceptron - MLP with sigmoidal activation function that composes the first module to determine a Fuzzy membership function; the second employs an MLP with pure linear activation function to define the consequents. The DSPBuilder® software along the Simulink® is used to connect, set and synthesize the Fuzzy Neural Network desired. Other hardware components employed in the architecture proposed cooperate to the system modularity. The system was tested and validated through a control problem and an interpolation problem. Several papers proposed different hardware architecture to implement hybrid systems by using Fuzzy logic and Neural Network. However, there is no approach with this specific neural network driven fuzzy reasoning by T-H method and the aim to be embedded. The Self-Organizing Map (SOM) and Levenberg-Marquardt backpropagation were used to train the FNN proposed off-line.

The choice of activation function can have a large effect on the performance of a neural network. While there have been some attempts to hand-engineer novel activation functions, the Rectified Linear Unit (ReLU) remains the most commonly-used in practice. This paper shows that evolutionary algorithms can discover novel activation functions that outperform ReLU. A tree-based search space of candidate activation functions is defined and explored with mutation, crossover, and exhaustive search. Experiments on training wide residual networks on the CIFAR-10 and CIFAR-100 image datasets show that this approach is effective. Replacing ReLU with evolved activation functions results in statistically significant increases in network accuracy. Optimal performance is achieved when evolution is allowed to customize activation functions to a particular task; however, these novel activation functions are shown to generalize, achieving high performance across tasks. Evolutionary optimization of activation functions is therefore a promising new dimension of metalearning in neural networks.

This research mainly based on multilayer perceptron (MLP) neural networks technique of data mining to forecast the wheat crop yield at the district level. There are many statistical and simulation models available, but the proposed algorithm with new activation function provides promising results in a shorter time with more accuracy. Sigmoid and hyperbolic tangent activation functions are widely used in the neural network. The activation functions play an important role in the neural network learning algorithm. The main objective of the proposed work is to develop an amended MLP neural network with new activation function and revised random weights and bias values for crop yield estimation by using the different weather parameter datasets. MLP model has been tested by existing activation functions and newly created activation functions with different cases including weights and bias values. In this research study, we evaluate the result of different activation functions and recommend some new simple activation functions, named DharaSig, DharaSigm and SHBSig, to improve the performance of neural networks and accurate results. Also, three new activation functions created with little variations in the DharaSig function named DharaSig1, DharaSig2 and DharaSig3. In this research study, variable numbers of hidden layers are tested with the variable number of neurons per hidden layer for the agriculture dataset. Variable values of momentum, seed and learning rate are also used in this study. Experiments show that newly created activation functions provide better results compared to ‘sigmoid’ default neural network activation function for agriculture datasets.

The fact that security is not the main functionality of an IoT device means that even lesser portion of its computing power is available for security. Security measures implemented in traditional computers such as cryptography, present a challenge from this context when applied in IoT devices. Different algorithms have been presented in the literature that meet IoT requirements and yet maintain the needed security level. Most of the studies have focused on popular algorithms, especially AES finalists. However, more IoT oriented algorithms have been presented to provide better performance in terms of power and capacity, those are known as lightweight cryptographic algorithms. PRESENT as one of the lightweight cryptographic algorithms is the main focus of this study. The algorithm has been implemented using MATLAB, ASIC and FPGA flows. Lowest achievable power consumption is 120.9 µW at throughput of 20.64 Mbps for the ASIC. For the FPGA, best achievable results was for 16 s-box design utilizing a power of 225.95 mW and 352 LUTs.

In this chapter, we will discuss the basic architecture of neural networks including activation functions, forward propagation, and backpropagation. We will also create a simple neural network model from scratch using the sigmoid activation function. In particular, this chapter will discuss: Neural network architecture. Activation functions used in neural networks. Forward propagation. Loss function of neural networks. Backpropagation.

Neural networks are a state-of-the-art approach that performs well for many tasks. The activation function (AF) is an important hyperparameter that creates an output against the coming inputs to the neural network model. AF significantly affects the training and performance of the neural network model. Therefore, selecting the most optimal AF for processing input data in neural networks is important. Determining the optimal AF is often a difficult task. To overcome this difficulty, studies on trainable AFs have been carried out in the literature in recent years. This study presents a different approach apart from fixed or trainable AF approaches. For this purpose, the activation function cyclically switchable convolutional neural network (AFCS-CNN) model structure is proposed. The AFCS-CNN model structure does not use a fixed AF value during training. It is designed in a self-regulating model structure by switching the AF during model training. The proposed model structure is based on the logic of starting training with the most optimal AF selection among many AFs and cyclically selecting the next most optimal AF depending on the performance decrease during neural network training. Any convolutional neural network (CNN) model can be easily used in the proposed model structure. In this way, a simple but effective perspective has been presented. In this study, first, ablation studies have been carried out using the Cifar-10 dataset to determine the CNN models to be used in the AFCS-CNN model structure and the specific hyperparameters of the proposed model structure. After the models and hyperparameters were determined, expansion experiments were carried out using different datasets with the proposed model structure. The results showed that the AFCS-CNN model structure achieved state-of-the-art success in many CNN models and different datasets.

Activation functions are critical components in neural network architectures, significantly influencing model performance and learning efficiency. While traditional univariate activation functions have been extensively studied and optimized, the exploration of multidimensional (multivariate) activation functions remains relatively nascent. In this paper, a comprehensive study of various multidimensional activation functions is presented, focusing on both Gaussian Mixture Models (GMM) and interpolation-based methods with learnable parameters. Through extensive experiments, the performance of these novel activation functions is evaluated in diverse neural network architectures and tasks. Results demonstrate that while the proposed multidimensional activation functions achieve performance on par with known activation functions like Swiglu, they do not consistently outperform it. This investigation provides valuable insights into the potential and limitations of multidimensional activation functions, paving the way for future research in this promising area of deep learning.

Artificial neural networks (ANNs, or simply NNs) are inspired by biological nervous systems and consist of simple processing units (artificial neurons) that are interconnected by weighted connections. Neural networks can be trained to solve problems that are difficult to solve by conventional computer algorithms. The usage of the FPGA (Field Programmable Gate Array) for neural Network implementation provides flexibility in programmable systems. For the neural network based instrument prototype in real time application, conventional specific VLSI neural chip design suffers the limitation in time and cost. With low precision artificial neural network design, FPGAs have higher speed and smaller size for real time application than the VLSI design. Several work show that FPGA are real opportunity for flexible hardware implementation of neural network and yet representation of standard neural network face some problem. The difficulty such as limit of size and architecture of neural network that can mapped on to FPGA. This paper discuss the usage of neural network implementations. Both assets and obstacles are described and various solution are outlined.

In this paper, we propose a new architecture of Pulse Mode Neural Network (PMNN) with very simple activation function. Pulse mode is gaining support in the field of hardware Neural Networks thanks to its higher density of integration. However, the complexity of the activation functions presents a drawback for hardware implementation of Neural Networks and limits its area of application. In this context, the main idea is to apply a new kind of activation function, simply generated by the product of two sigmoidal functions, which are very simple and already implemented in previous work. Details of important aspects concerning the hardware implementation are given. To verify the performance and capacity of the proposed design, we apply it for approximation of image denoising function. The filtered results are verified in terms of the Peak Signal to Noise Ratio (PSNR). Experimental results reveal that the proposed PMNN filter has a greater ability to recover the informative pixel intensities from the infected image with a recovery of 7.5 dB for Gaussian noise and 5.3 dB for Speckle noise. Besides, such results demonstrate the performance and efficiency of our Neural filter when compared to other conventional filtering techniques. The designed network is implemented on a field-programmable gate array (FPGA) platform and synthesis results are presented and discussed.

The design of neural network models involves numerous complexities, including the determination of input vectors, choosing the number of hidden layers and their computational units, and specifying activation functions for the latter. The combinatoric possibilities are daunting, yet experience has yielded informal guidelines that can be useful. Alternatively, current research on genetic algorithms (GA) suggests that they might be of practical use as a formal method of determining ‘good’ architectures for neural networks. In this paper, we use a genetic algorithm to find effective architectures for backpropagation neural networks (BP). We compare the performance of heuristically designed BP networks with that of GA-designed BP networks. Our test domains are sets of problems having compensatory, conjunctive, and mixed-decision structures. The results of our experiment suggest that heuristic methods produce architectures that are simpler and yet perform comparatively well. © 1998 John Wiley & Sons, Ltd.

Evolutionary design for energy-efficient approximate digital circuits

Activation functions facilitate deep neural networks by introducing non-linearity to the learning process. The non-linearity feature gives the neural network the ability to learn complex patterns. Recently, the most widely used activation function is the Rectified Linear Unit (ReLU). Though, other various existing activation including hand-designed alternatives to ReLU have been proposed. However, none has succeeded in replacing ReLU due to their existing inconsistencies. In this work, activation function called ReLUMemristor-like Activation Function (RMAF) is proposed to leverage benefits of negative values in neural networks. RMAF introduces a constant parameter (α) and a threshold parameter (p) making the function smooth, non-monotonous, and introduces non-linearity in the network. Our experiments show that, the RMAF works better than ReLU and other activation functions on deeper models and across number of challenging datasets. Firstly, experiments are performed by training and classifying on multi-layer perceptron (MLP) over benchmark data such as the Wisconsin breast cancer, MNIST, Iris and Car evaluation. RMAF achieves high performance of 98.74%, 99.67%, 98.81% and 99.42% respectively, compared to Sigmoid, Tanh and ReLU. Secondly, experiments were performed on convolution neural network (ResNet) over MNIST, CIFAR-10 and CIFAR-100 data and observed the proposed activation function achieves higher performance accuracy of 99.73%, 98.77% and 79.82% respectively than Tanh, ReLU and Swish. Additionally, we experimented our work on deep networks i.e. squeeze network (SqueezeNet), Dense connected neural network (DenseNet121) and ImageNet dataset, which RMAF produced the best performance. We note that, the RMAF converges faster than the other functions and can replace ReLU in any neural network due to the efficiency, scalability and its similarity to both ReLU and Swish.