Physics Informed Piecewise Linear Neural Networks for Process Optimization

Neural networks (NNs) are increasingly used in design to construct the objective functions and constraints, which leads to the needs of optimization of NN models with respect to design variables. A Neural Optimization Machine (NOM) is proposed for constrained single/multi-objective optimization by appropriately designing the NN architecture, activation function and loss function. The NN's built-in backpropagation algorithm conducts the optimization and is seamlessly integrated with the additive manufacturing (AM) process-property model. The NOM is tested using several numerical optimization problems. It is shown that the increase in the dimension of design variables does not increase the computational cost significantly. Next, a brief review of the physics-guided machine learning model for fatigue performance prediction of AM components is given. Finally, the NOM is applied to design processing parameters in AM to optimize the mechanical fatigue properties through the physics-guided NN under uncertainties. One novel contribution of the proposed methodology is that the constrained process optimization is integrated with physics/knowledge and the data-driven AM process-property model. Thus, a physics-compatible process design can be achieved. Another significant benefit is that the training and optimization are achieved in a unified NN model, and no separate process optimization is needed. This article is part of the theme issue 'Physics-informed machine learning and its structural integrity applications (Part 1)'.

Hybrid surrogate model for online temperature and pressure predictions in data centers

Activation functions facilitate deep neural networks by introducing non-linearity to the learning process. The non-linearity feature gives the neural network the ability to learn complex patterns. Recently, the most widely used activation function is the Rectified Linear Unit (ReLU). Though, other various existing activation including hand-designed alternatives to ReLU have been proposed. However, none has succeeded in replacing ReLU due to their existing inconsistencies. In this work, activation function called ReLUMemristor-like Activation Function (RMAF) is proposed to leverage benefits of negative values in neural networks. RMAF introduces a constant parameter (α) and a threshold parameter (p) making the function smooth, non-monotonous, and introduces non-linearity in the network. Our experiments show that, the RMAF works better than ReLU and other activation functions on deeper models and across number of challenging datasets. Firstly, experiments are performed by training and classifying on multi-layer perceptron (MLP) over benchmark data such as the Wisconsin breast cancer, MNIST, Iris and Car evaluation. RMAF achieves high performance of 98.74%, 99.67%, 98.81% and 99.42% respectively, compared to Sigmoid, Tanh and ReLU. Secondly, experiments were performed on convolution neural network (ResNet) over MNIST, CIFAR-10 and CIFAR-100 data and observed the proposed activation function achieves higher performance accuracy of 99.73%, 98.77% and 79.82% respectively than Tanh, ReLU and Swish. Additionally, we experimented our work on deep networks i.e. squeeze network (SqueezeNet), Dense connected neural network (DenseNet121) and ImageNet dataset, which RMAF produced the best performance. We note that, the RMAF converges faster than the other functions and can replace ReLU in any neural network due to the efficiency, scalability and its similarity to both ReLU and Swish.

Machine learning (ML) constitutes a division of artificial intelligence (AI) that aims to train computers how to perform specific tasks without explicit programming. Traditional ML tools are widely used for classification and identification of animals. However, these methods have some drawbacks because of the extensive manual reliance and the delay in data interpretation. To overcome this, Applied Deep Learning algorithms are used with Artificial Neural Networks (ANN) and Convolution Neural Network (CNN) models introduced to address species classification, characteristics detection, and pattern recognition tasks helping in accurate identification and classification of animals. In this paper, we have tried to compile and deliver a recent comprehensive information on latest available investigations in the field of life sciences particularly used for animal identification. We have also accentuated the diverse applications of machine learning models including other parameters like, features, accuracy gained, database used and their limitations. The red flour beetle, Tribolium castaneum (Coleoptera; Tenebrionidae) is a prevailing and detrimental secondary insect pest of stored grains along with derived products causing 7% to 35% annual loss. Despite of that, nowdays it is also extensively considered as a model organism for genetic disease investigation. While using it in scientific research, exact sex identification of these insects becomes a crucial preliminary step. Generally, pupal stage is used to sort these insects according to their sex and needs expert humans. It is crucial to employ image processing and ML algorithms to quickly identify gender of this insect which is not done yet. We have used a CNN-based smart technique to recognize and categorize gender differences in T. castaneum using microscopic images in order to build an intelligent system for applied research. For this study, a dataset is created by taking 116 microscopic images of both the dorsal and ventral sides of pupae of two different sexes. In this algorithm, a 2D matrix of feature map is selected sequentially and the maximum value in the matrix is selected to generate a pooled feature map. The Rectified Linear Unit (ReLU) activation function is used for the CNN. The classification model has an accuracy between 97 and 98% with an F-score of 0.67. These results demonstrate the robustness of the classification model, which does not rely heavily on manual intervention compared to traditional machine learning (ML) tools and automates the processes of feature extraction and gender classification regardless of the position of the pupae in the images.

This paper proposes an automatic sleep apnea monitoring device for adults employing of single ECG patch and a pulse oximeter. The device is designed to automatically detect sleep apneic (SA) events with the inference of feedforward neural network (FNN) model embedded in digital hardware. The three-layer (8-6-4) FNN model was trained over several epochs with a 5-fold cross validation technique where the training set had a mini-batch size of 10. Open-source Apnea ECG dataset collected from the PhysioNET bank was used in training, validating, and testing the model. Rectified Linear Unit (ReLU) activation function was used in the input and hidden layers of the network and sigmoid function was used as the output classifier. ADAM optimizer was used for optimization of the model while mean-squared-error (MSE) was used for calculating model loss. The final trained and validated model was implemented onto re-programmable digital hardware called Field-Programmable Gate Array (FPGA). The hardware implementation of the model yielded an accuracy of over 87 percent with a power consumption rate of around 52 W, which is 5x times lower than that of commercially available machine learning hardware accelerators. The proposed system design will be realized in integrated circuits on CMOS platform for developing energy-efficient, smart, wearable, and automated sleep apnea detection and screening device for adults.

Recently, Deep learning has made a great deal of success in processing images, audios, and natural languages and so on. The activation function is one of the key factors in Deep learning. In this paper, according to characteristics of biological neurons, an improved Leaky Single-Peaked Triangle Linear Unit (LSPTLU) activation function is presented for the right-hand response unbounded of Rectified Linear Unit (ReLU) and Leaky ReLU (LReLU). LSPTLU is more in line with the biological neuron essence and achieves the excellent performance of equivalent or beyond ReLU and LReLU on different datsets, e.g., MNIST, Fashion-MNIST, SVHN, IMAGENET, CALTECH101 and CIFAR10 datasets.

This study evaluates the Soviet-era ground motion prediction equation (named as A&K-1979) and introduces an Artificial Neural Network (ANN)-based GMM specifically designed for Azerbaijan to improve prediction accuracy. Ground motion models (GMMs) are essential for predicting earthquake-induced ground motions, aiding seismic hazard assessments and engineering designs. While traditional linear empirical models have been widely used, they often struggle to capture complex nonlinear ground motion behaviors. The performance of A&K-1979 was assessed using a strong-motion dataset comprising 500 records collected between 2022 and 2024. Two variants of A&K-1979 were tested: A&K-1979-1 for PGA ≥160 cm/s2 and A&K-1979-2 for PGA <160 cm/s2. An ANN-based GMM was developed using earthquake magnitude and hypocentral distance as inputs, followed by three hidden layers (32-32-16 neurons) with the Rectified Linear Unit (ReLU) activation function. The model was validated with a separate dataset of 268 records, evaluated using metrics such as bias, standard deviation of residuals (σ), mean absolute error (MAE), root mean squared error (RMSE), and R2. The A&K-1979 model exhibited notable prediction biases: A&K-1979-1 overestimated PGA values, while A&K-1979-2 underestimated them. The ANN-based GMM achieved improved performance metrics, with a bias of -0.0076, σ of 0.5971, MAE of 0.4416, RMSE of 0.5972, and an R2 of 0.4601. The improved accuracy of the ANN-based GMM highlights its potential as a valuable tool for seismic hazard assessments in Azerbaijan. By providing enhanced prediction capabilities, the ANN model demonstrates greater reliability and practical value than A&K-1979, reinforcing the need for updated predictive models in the region and supporting its use in preliminary hazard analysis.

The convolutional neural networks (CNN), including AlexNet, GoogleNet, VGGNet, etc. extract features for many computer vision problems which are very discriminative. The trained CNN model over one dataset performs reasonably well whereas on another dataset of similar type the hand-designed feature descriptor outperforms the same trained CNN model. The Rectified Linear Unit (ReLU) layer discards some values in order to introduce the non-linearity. In this paper, it is proposed that the discriminative ability of deep image representation using trained model can be improved by Average Biased ReLU (AB-ReLU) at the last few layers. Basically, AB-ReLU improves the discriminative ability in two ways: 1) it exploits some of the discriminative and discarded negative information of ReLU and 2) it also neglects the irrelevant and positive information used in ReLU. The VGGFace model trained in MatConvNet over the VGG-Face dataset is used as the feature descriptor for face retrieval over other face datasets. The proposed approach is tested over six challenging, unconstrained and robust face datasets (PubFig, LFW, PaSC, AR, FERET and ExtYale) and also on a large scale face dataset (PolyUNIR) in retrieval framework. It is observed that the AB-ReLU outperforms the ReLU when used with a pre-trained VGGFace model over the face datasets. The validation error by training the network after replacing all ReLUs with AB-ReLUs is also observed to be favorable over each dataset. The AB-ReLU even outperforms the state-of-the-art activation functions, such as Sigmoid, ReLU, Leaky ReLU and Flexible ReLU over all seven face datasets.

Focusing on the problems of complex structure and low feature extraction efficiency that exist in some traditional neural network algorithms, an improved convolutional neural network, combined with a convolutional attention mechanism, is proposed to im- prove the network feature extraction efficiency and reduce the network complexity. First, the convolutional block attention module (CBAM) was introduced, which can strengthen the extraction of useful features from the data, thereby improving the network feature extraction efficiency. Second, the residual structure was optimized, and the original multi-layer convolution was simplified to a single-layer convolution, which reduces the complexity of the model and improves the diagnostic efficiency. Then, the Rectified Linear Unit (ReLU) activation function in the model was replaced with the better-performing Exponential Linear Unit (ELU) activation function. Finally, the overall structure of the ResNet model was optimized. The CBAM-ResNet network model was based on the traditional ResNet18 model and consists of a CBAM module and the optimized ResNet18 model. Finally, experiments were performed using sensor data collected from rolling bearing test benches. The experimental results show that the feature extraction capability and efficiency of the new model are better than those of several other classic deep network models and the original network model, and this reduces the time complexity and model size while also maintaining a high accuracy rate.

Neural networks with ReLU powers need less depth

In deep learning, the choice of activation function plays a vital role in enhancing model performance. We propose AHerfReLU, a novel activation function that combines the rectified linear unit (ReLU) function with the error function (erf), complemented by a regularization term 1/(1 + x 2 ), ensuring smooth gradients even for negative inputs. The function is zero centered, bounded below, and nonmonotonic, offering significant advantages over traditional activation functions like ReLU. We compare AHerfReLU with 10 adaptive activation functions and state‐of‐the‐art activation functions, including ReLU, Swish, and Mish. Experimental results show that replacing ReLU with AHerfReLU leads to 3.18% improvement in Top‐1 accuracy on the LeNet network for the CIFAR100 dataset, 0.63% improvement on CIFAR10%, and 1.3% improvement in mean average precision (mAP) on the SSD300 model in the Pascal VOC dataset. Our results demonstrate that AHerfReLU enhances model performance, offering improved accuracy, loss reduction, and convergence stability. The function outperforms existing activation functions, providing a promising alternative for deep learning tasks.

Price movement predictions of financial instruments using traditional time-series models with a predefined mathematical structure are common, but this method restricts their ability to learn latent patterns in the data. In recent times, artificial neural networks (ANN) have been able to learn complex hidden patterns from financial datasets using a highly nonlinear architecture. However, most experiments with neural networks require a lot of time to search a suitable network and subsequently train the network. The authors have developed a deep multilayer perceptron (MLP) classifier with a zero-centered piecewise linear unit activation that yielded better classification performance according to their accuracy metric and required consistently less training time compared to a similar MLP network with leaky rectified linear unit and tanh activation functions. The authors illustrate their technique with a large high-frequency dataset on selected bank shares from the Indian stock market. The authors also discuss the theoretical properties and advantages of their proposal. <b>TOPICS:</b>Statistical methods, simulations, big data/machine learning <b>Key Findings</b> • Time-series models have been used extensively to understand stock price movements. They are constrained by a predefined structure. Artificial neural network (ANNs), due to their universal approximation properties, offer more flexibility; however, they suffer from long training times. • In this article, the authors introduce a zero-centered piecewise linear unit (PLU), a hybrid activation function derived from tanh and leaky rectified linear unit (ReLU) functions that exhibits the zero-centered property from tanh and the nonzero gradient property from leaky ReLU. • The authors developed a multilayer perceptron (MLP) classifier with PLU activation, which yielded better classification performance according to the accuracy metric and required consistently less training time compared to a similar MLP network with leaky ReLU and tanh activation.

Power Quality Disturbances (PQD) has result in numerous failures and damage to electrical equipment. This paper utilized MATLAB Application to propose ways in detecting and classifying Voltage Sag, Swell and Transient. The proposal was divided into three parts which are detection, classification, and performance evaluation. The detection stage was done using Discrete Wavelet Transform in Wavelet Analyzer to obtain signal decomposition in different energy levels to be used in Energy Distribution Deviation (EDD) method. The classification stage was done in Classification Learner to check how good Multilayer Perceptron Neural Network able to trains, validates, and predicts as a classification model. The performance evaluation stage was done in Neural Net Fitting using Levenberg-Marquardt (LM) as training algorithm to see how well the model perform in term of Mean Square Error (MSE) and regression. This paper also discusses the effect of input ratio, activation function (Sigmoid, Tangent Hyperbolic, Rectified Linear Unit) and training algorithm (Levenberg-Marquardt, Bayesian Regularization, Scale Conjugate Gradient) towards accuracy in a Neural Network model. This study found that EDD was able to detect the difference in energy distribution of PQD properly. The Multilayer Perceptron model was observed to performed better and had higher accuracy when fed with more sample data, bigger layer size and activated using Tangent Hyperbolic (Tanh) activation function. Increasing layer size also resulted in slower prediction speed and longer training time. The model performance was evaluated with the lowest MSE and highest regression when Levenberg-Marquardt (LM) was implemented compared to Bayesian Regularization (BR) and Scale Conjugate Gradient (SCG).

The scale of deer breeding has gradually increased in recent years and better information management is necessary, which requires the identification of individual deer. In this paper, a deer face dataset is produced using face images obtained from different angles, and an improved residual neural network (ResNet)-based recognition model is proposed to extract the features of deer faces, which have high similarity. The model is based on ResNet-50, which reduces the depth of the model, and the network depth is only 29 layers; the model connects Squeeze-and-Excitation (SE) modules at each of the four layers where the channel changes to improve the quality of features by compressing the feature information extracted through the entire layer. A maximum pooling layer is used in the ResBlock shortcut connection to reduce the information loss caused by messages passing through the ResBlock. The Rectified Linear Unit (ReLU) activation function in the network is replaced by the Exponential Linear Unit (ELU) activation function to reduce information loss during forward propagation of the network. The preprocessed 6864 sika deer face dataset was used to train the recognition model based on SE-Resnet, which is demonstrated to identify individuals accurately. By setting up comparative experiments under different structures, the model reduces the amount of parameters, ensures the accuracy of the model, and improves the calculation speed of the model. Using the improved method in this paper to compare with the classical model and facial recognition models of different animals, the results show that the recognition effect of this research method is the best, with an average recognition accuracy of 97.48%. The sika deer face recognition model proposed in this study is effective. The results contribute to the practical application of animal facial recognition technology in the breeding of sika deer and other animals with few distinct facial features.

Enhancing deterministic prediction in unidirectional ocean waves using an Artificial Neural Network with exponential linear unit