Trainable ReLU Activation Function Improves the Performance of MLP-Mixer

Abstract

Translate Article

The selection of appropriate activation functions has become an important research domain around neural networks. The aim of the present article is to investigate the properties of trainable rectified linear unit (ReLU) transfer function from performance of neural network point of view. We focus on the role of trainable parameters for ReLU activation function to achieve high neural network performance. We also give a comparative analysis for MLP-Mixer using trainable ReLU activation function and GELU. Simulation results are presented that show the improved accuracy capability.