Functional Perceptron using Multi-dimensional Activation Functions

Activation functions are critical components in neural network architectures, significantly influencing model performance and learning efficiency. While traditional univariate activation functions have been extensively studied and optimized, the exploration of multidimensional (multivariate) activation functions remains relatively nascent. In this paper, a comprehensive study of various multidimensional activation functions is presented, focusing on both Gaussian Mixture Models (GMM) and interpolation-based methods with learnable parameters. Through extensive experiments, the performance of these novel activation functions is evaluated in diverse neural network architectures and tasks. Results demonstrate that while the proposed multidimensional activation functions achieve performance on par with known activation functions like Swiglu, they do not consistently outperform it. This investigation provides valuable insights into the potential and limitations of multidimensional activation functions, paving the way for future research in this promising area of deep learning.

With the rapid development of artificial intelligence, deep learning is developing rapidly. As the main framework of in-depth learning, convolution neural network achieved much in semantic segmentation and image classification. Based on the low speed engineering vehicle tracking system project, this paper studies the activation function, which is an important component of convolution neural network. Firstly, it summarizes the advantages and disadvantages of four traditional activation functions such as Sigmoid, and then introduces three new activation functions: Swish, FTS, and Relu-Softplus. Finally, using the TensorFlow framework, a single new activation function and a hybrid new activation function are applied to the convolution layer of the four neural networks constructed, and the image recognition comparison is performed in the data set CIFAR-10. It is concluded that the recognition precision of mixing the new activation functions is higher than that of using a new activation function alone, and the average time is shorter and the convergence speed is faster.

The zeroing neural network (ZNN) model, a powerful approach for addressing time-varying problems, has been extensively applied in the calculation and optimization fields. In this article, a new pattern activation function, the power piecewise activation function (PPAF), is proposed to establish a predefined fixed-time convergent ZNN (PFTZNN) for finding solutions to the time-varying quadratic programming problem. In comparison with the traditional activation functions, multisegmentation is a remarkable feature of the PPAF; consequently, the advantage of PPAF is that its parameters can be flexibly adjusted according to actual needs. Specifically, because of the multisegment characteristics of the PPAF, the convergence speed of the PPAF-activated PFTZNN model can be flexibly adjusted based on distinct requirements. The fixed-time convergence property of the PPAF-activated PFTZNN model is validated by detailed mathematical theoretical analysis, and its upper bound convergence time is directly calculated. Then, the comparative simulation results of the PPAF-activated PFTZNN model with other existing ZNN models for time-varying quadratic programming are provided for the further verification of its superior convergence speed and robustness. In addition, the proposed PFTZNN model is applied for dual-arm manipulator cooperative trajectory tracking, and its practical application ability is demonstrated by united simulation experiments of MATLAB and Robot studio. Finally, the PFTZNN model is also applied to control a real dual-arm manipulator to complete the trajectory tracking task, which further validates its superior performance together and widespread applicability.

Solving Large Scale Classification Problems with Stochastic Based Optimization

Benefiting from deep learning, the accuracy of face expression recognition tasks based on convolutional neural networks has been greatly improved. However, the traditional SoftMax activation function lacks the ability to discriminate between classes. To solve this problem, the industry has proposed several activation functions based on softmax, such as A-softmax, LMCL, etc. We investigate the geometric significance of the weights from a fully connected layer and consider the weights as the class centers. By extracting the feature vector of several samples and extending the corresponding means to the weights, the model can develop the ability to recognize custom classes without training, while maintaining the accuracy of the original classification. On the expression task, the original seven-category classification is validated to obtain 97.10% accuracy on the CK+ dataset and 88% accuracy on the custom dataset.

Convolutional neural networks (CNNs) have become a useful tool for a wide range of applications such as text classification. However, CNNs are not always sufficiently accurate to be useful in certain applications. The selection of activation functions within CNN architecture can affect the efficacy of the CNN. However, there is limited research regarding which activation functions are best for CNN text classification. This study tested sixteen activation functions across three text classification datasets and six CNN structures, to determine the effects of activation function on accuracy, iterations to convergence, and Positive Confidence Difference (PCD). PCD is a novel metric introduced to compare how activation functions affected a network’s classification confidence. Tables were presented to compare the performance of the activation functions across the different CNN architectures and datasets. Top performing activation functions across the different tests included the symmetrical multi-state activation function, sigmoid, penalised hyperbolic tangent, and generalised swish. An activation function’s PCD was the most consistent evaluation metric during activation function assessment, implying a close relationship between activation functions and network confidence that has yet to be explored.

Combining DEKF algorithm and trace rule for fast on-line invariance extraction and recognition

Univ São Paulo, Inst Biomed Sci, Dept Immunol, BR-05508000 São Paulo, Brazil

This paper provides an analysis of the combining effect of novel activation function and loss function based on M-estimation in application to extreme learning machine (ELM), a feed-forward neural network. Due to the computational efficiency and classification/prediction accuracy of ELM and its variants, they have been widely exploited in the development of new technologies and applications. However, in real applications, the performance of classical ELMs deteriorates in the presence of outliers, thus, negatively impacting the precision and accuracy of the system. To further enhance the performance of ELM and its variants, we proposed novel activation functions based on the psi function of M and redescend the M-estimation method along with the smooth l 2-norm weight-loss functions to reduce the negative impact of the outliers. The proposed psi functions of several M and redescending M-estimation methods are more flexible to make more distinct features space. For the first time, the idea of the psi function as an activation function in the neural network is introduced in the literature to ensure accurate prediction. In addition, new robust l 2 norm-loss functions based on M and redescending M-estimation are proposed to deal with outliers efficiently in ELM. To evaluate the performance of the proposed methodology against other state-of-the-art techniques, experiments have been performed in diverse environments, which show promising improvements in application to regression and classification problems.

Within the framework of the German national project multiSat multifunctional composite structures for satellite applications are developed. The objective is the integration of passive and active functions into the load-bearing spacecraft structure by using suitable materials, components and mechanisms. The passive functions include heat transfer, radiation shielding and protection against space debris impacts, whereas the active functions comprise electric energy and data transfer and vibration reduction. Due to their multi-layer build-up composite materials are suitable for functional integration since each layer can be defined and designed to provide one or more specific functions. The concept of a multifunctional structure allows for the reduction of the overall satellite mass and of installation space required for subsystems. It also opens up new opportunities for highly integrative and standardized production processes and lower total costs and time for manufacturing, qualification and launch. This paper describes the development and design of a concept for a multifunctional sandwich panel and the results of the analyses, numerical simulations and experiments conducted at coupon level.

In many real time applications, we often have to deal with classification, regression or clustering problems that involve multiple tasks. The conventional machine learning approaches solve these tasks independently by ignoring the task relatedness. In multi-task learning (MTL), these related tasks are learned simultaneously by extracting and utilizing the shared information across tasks. This approach of learning related tasks together increases the sample size for each task and improves the generalization performance. Thus MTL is especially beneficial when the training size is small for each task. This paper describes multi-task learning using kernel online learning approach. As many real world applications are online in nature, development of efficient online learning techniques is very much needed. Since online learning processes only one data at a time, these techniques could be effectively applied on large data sets. The MTL model we developed involves a global function and a task specific function corresponding to each task. The cost function used for finding the task specific function makes use of the global model for incorporating the necessary information from other tasks. Such modeling strategies improve the generalization capacity of the model. The problem of finding global and task specific functions is formulated as two separate problems and at each step of arrival of new data, the global vector is solved at the first instant and its information is used to update the task specific vector. The updation rule for the task specific function involves the approximation of global components using task specific components by means of projection. We applied the developed frame work on real world problems and the results were found to be promising.

Complex valued neural network (CVNN) has been developed to process complex valued data directly. In CVNN, one of the most important factors is selecting the node’s activation function. Choosing the right activation function for each layer is also crucial and may have a significant impact on metric scores and the training speed of the model. This paper introduces three new activation functions for CVNNs which is closely related to the activation function complex swish. These new activation functions are complex modified swish, complex E-swish and complex Flatten-T swish. In order to verify the validity and practicability of the proposed three new activation functions are tested and compared with complex swish activation function on complex valued four bit XOR problem, three inputs symmetry detection and the fading equalization problems. We show that complex E-swish ( β=1.4) has the best overall performance when compared to other networks using complex swish, complex modified swish and complex Flatten-T swish activation functions on the considered tasks.

Most fuzzy learning algorithms can generate simple fuzzy rules from a set of examples having the same type of attributes. However, the case for fuzzy rules in which their knowledge representation power has been enhanced by the inclusion of several parameters such as weight and certainty factor has not been addressed. It is important but difficult to automatically acquire fuzzy rules with parameters from a set of examples with mixed attributes. The paper presents an approach to handle mixed attributes and introduces the concept of degree of importance of each attribute-value contributing to the consequent of a given rule. Based on this new concept, a new heuristic for generating fuzzy decision trees with parameters is given and a set of weighted fuzzy rules with local weight and certainty factor are extracted from the trees. The advantages of the learning is initially verified by the Iris classification problem.

In the family of recurrent neural networks the long short-term model network provides promising solutions for many complex applications such as speech and voice recognition, machine translation and time series analysis. When building these networks, many tunable hyper-parameters need to be set early. Among these hyperparameters, the activation function greatly influences the learning behavior of the neural networks. The present work proposes a differential evolution algorithm (DEA)-based hierarchical combined activation to surrogate the default activation functions of the LSTM cell. A DEA-based neuroevolution method is proposed to discover an optimal combination of function for the LSTM network. To investigate the performance of the proposed neuroevolution method, several experiments were done on three datasets for human activity recognition using two LSTM networks. The results show that the newly evolved activation functions using the DEA on each dataset outperform the traditional activation functions. The classification accuracy of the proposed LSTM with the DEA-based hierarchical activations is higher than that of other state-of-the-art models in the literature.

Quantification of habitability is a complex task. Previous attempts at measuring habitability are well documented. Classification of exoplanets, on the other hand, is a different approach and depends on quality of training data available in habitable exoplanet catalogs. Classification is the task of predicting labels of newly discovered planets based on available class labels in the catalog. We present analytical exploration of novel activation functions as consequence of integration of several ideas leading to implementation and subsequent use in habitability classification of exoplanets. Neural networks, although a powerful engine in supervised methods, often require expensive tuning efforts for optimized performance. Habitability classes are hard to discriminate, especially when attributes used as hard markers of separation are removed from the data set. The solution is approached from the point of investigating analytical properties of the proposed activation functions. The theory of ordinary differential equations and fixed point are exploited to justify the “lack of tuning efforts” to achieve optimal performance compared to traditional activation functions. Additionally, the relationship between the proposed activation functions and the more popular ones is established through extensive analytical and empirical evidence. Finally, the activation functions have been implemented in plain vanilla feed-forward neural network to classify exoplanets. The mathematical exercise supplements the grand idea of classifying exoplanets, computing habitability scores/indices and automatic grouping of the exoplanets converging at some level.