Diminishing Batch Normalization.

The convolutional neural network (CNN) has been widely used in image recognition field due to its good performance. This paper proposes a facial expression recognition method based on the CNN model. Regarding the complexity of the hierarchic structure of the CNN model, the activation function is its core, because the nonlinear ability of the activation function really makes the deep neural network have authentic artificial intelligence. Among common activation functions, the ReLu function is one of the best of them, but it also has some shortcomings. Since the derivative of the ReLu function is always zero when the input value is negative, it is likely to appear as the phenomenon of neuronal necrosis. In order to solve the above problem, the influence of the activation function in the CNN model is studied in this paper. According to the design principle of the activation function in CNN model, a new piecewise activation function is proposed. Five common activation functions (i.e., sigmoid, tanh, ReLu, leaky ReLus and softplus–ReLu, plus the new activation function) have been analysed and compared in facial expression recognition tasks based on the Keras framework. The Experimental results on two public facial expression databases (i.e., JAFFE and FER2013) show that the convolutional neural network based on the improved activation function has a better performance than most-of-the-art activation functions.

Small unmanned aerial vehicles (UAVs) rapidly develop and are implemented in various industries to make people’s lives easier. However, there are potential risks in their use, such as unauthorized surveillance of critical infrastructure objects and the delivery of explosive devices, which poses a significant threat to public and national security. The acoustic method promises direction for solving this issue by analyzing the sound characteristics and Doppler shift signatures of UAVs, using microphone arrays and machine learning techniques. The aim of this article is to develop an algorithm for effective detection and classification of drone audio signals using a deep learning convolutional neural network (CNN), constructing its architecture, and evaluating its performance. Before submitting the drone audio dataset into the neural network, the quality of the audio recordings is improved through normalization, Wiener filtering, and segmentation. The audio is segmented into frames with a duration of 25 ms and a 50% overlap, applying Hamming windowing for better accuracy in the time domain, as temporal precision is crucial in audio signal processing. The obtained data is divided into three sets in a 60/20/20 ratio: for training, validation, and testing purposes. Next, the data is represented by a simplified set of features, extracting mel-spectrograms from each frame of the processed audio signals to capture their temporal and spectral characteristics. The frequency range of analysis corresponds to the working frequency limits of the microphone model (20 Hz - 20 kHz), with a frequency resolution of 50 Hz and 30 working mel frequency bands. Using the training data and the extracted audio features, a neural network architecture is developed to investigate the performance of the drone detection and classification algorithm. It consists of 10 pairs of convolutional layers, ReLU activation, batch normalization, and max-pooling layers. The number of these layers is determined by the size of the pooling window along the time dimension. This follows by flattening, dropout, fully connected, and Softmax layers. A classification layer is applied to normalize the output data and obtain final probabilities. The Adam optimizer is chosen for model training. Based on the dataset set, the initial learning rate is set to 0.001, gradually decreasing by a factor of 10 after 75% of the epochs to enhance convergence. The accuracy of the input data recognition reaches 99%, and the F1 score of the trained model is 0.93, indicating a high level of overall architecture performance. The maximum distance of effective detection of drones by the algorithm is 200 m.

In recent years, AI(Artificial Intelligence) has achieved great development in modern society. More and more modern technologies are applied in surveillance and monitoring. Healthcare monitoring is growing ubiquitous in modern wearable devices, such as a smart watch, electrocardiogram (ECG) necklace, smart band. Many sensors are attached to these smart devices to record and monitor physiological signals caused by activities, and then propagated those recorded electrical data to be further processed to give health diagnosis, disease prevention or making a distress call automatically. Obstructive sleep apnea (OSA) is a sleep disorder with a high occurrence in adult people and observed as an autonomous risk factor for circulatory problems such as ischemic heart attacks and stroke. Numerous traditional neural network based methods have been developed to detect OSA, where these methods however could not provide the intended result because they rely on shallow network. In this paper, we propose an effective OSA detection based on Convolutional neural network. Our method first extracts features from Apnea-Electrocardiogram (ECG) recordings using RR-intervals (time interval from one R-wave to the next R-wave in an ECG signal) and then CNN model having three convolution layers and three fully connected layers is trained with extracted features and applied for OSA detection. The first two convolution layers are followed by batch normalization and pooling layer, and softmax is connected to the last fully connected layer to give final decision. Experimental results on extracted feature of Apnea-ECG signal reveal that our model have better results in terms of performance measure sensitivity, specificity and accuracy. It is expected that the related technology can be applied into smart sensors, especially wearable devices.

Generalizable person re-identification aims to learn a model with only several labeled source domains that can perform well on unseen domains. Without access to the unseen domain, the feature statistics of the batch normalization (BN) layer learned from a limited number of source domains is doubtlessly biased for unseen domain. This would mislead the feature representation learning for unseen domain and deteriorate the generalizaiton ability of the model. In this paper, we propose a novel Debiased Batch Normalization via Gaussian Process approach (GDNorm) for generalizable person re-identification, which models the feature statistic estimation from BN layers as a dynamically self-refining Gaussian process to alleviate the bias to unseen domain for improving the generalization. Specifically, we establish a lightweight model with multiple set of domain-specific BN layers to capture the discriminability of individual source domain, and learn the corresponding parameters of the domain-specific BN layers. These parameters of different source domains are employed to deduce a Gaussian process. We randomly sample several paths from this Gaussian process served as the BN estimations of potential new domains outside of existing source domains, which can further optimize these learned parameters from source domains, and estimate more accurate Gaussian process by them in return, tending to real data distribution. Even without a large number of source domains, GDNorm can still provide debiased BN estimation by using the mean path of the Gaussian process, while maintaining low computational cost during testing. Extensive experiments demonstrate that our GDNorm effectively improves the generalization ability of the model on unseen domain.

Laser-induced breakdown spectroscopy (LIBS) has played a critical role in Mars exploration missions, substantially contributing to the geochemical analysis of Martian surface substances. However, the complex nonlinearity of LIBS processes can considerably limit the quantification accuracy of conventional LIBS chemometric methods. Hence chemometrics based on artificial neural network (ANN) algorithms have become increasingly popular in LIBS analysis due to their extraordinary ability in nonlinear feature modeling. The hidden layer activation functions are key to ANN model performance, yet common activation functions usually suffer from problems such as gradient vanishing (e.g., Sigmoid and Tanh) and dying neurons (e.g., ReLU). In this study, we propose a novel LIBS quantification method, named the Bayesian optimization-based tunable Softplus backpropagation neural network (BOTS-BPNN). Based on a dataset comprising 1800 LIBS spectra collected by a laboratory duplicate of the MarSCoDe instrument onboard the Zhurong Mars rover, we have revealed that a BPNN model adopting a tunable Softplus activation function can achieve higher prediction accuracy than BPNN models adopting other common activation functions if the tunable Softplus parameter β is properly selected. Moreover, the way to find the proper β value has also been investigated. We demonstrate that the Bayesian optimization method surpasses the traditional grid search method regarding both performance and efficiency. The BOTS-BPNN model also shows superior performance over other common machine learning models like random forest (RF). This work indicates the potential of BOTS-BPNN as an effective chemometric method for analyzing Mars in situ LIBS data and sheds light on the use of chemometrics for data analysis in future planetary explorations.

Communication is not always direct; it often involves nuanced elements like humor, irony, and sarcasm. This study introduces a novel two-level approach for sarcasm detection, leveraging Convolutional Neural Networks (CNNs). Convolutional neural networks (CNNs) are crucial for many deep learning applications, yet their deployment on IoT devices is challenged by resource constraints and the need for low latency, particularly in on-device training. Traditional methods of deploying large CNN models on these devices often lead to suboptimal performance and increased energy consumption. To address this, our paper proposes an energy- efficient CNN design by optimising batch normalisation operations. Batch normalisation is vital for deep learning, aiding in faster convergence and stabilising gradient flow, but there has been limited research on creating energy-efficient and lightweight CNNs with optimised batch normalisation. This study proposes a 3R (reduce, reuse, recycle) optimisation technique for batch normalization. This technique introduces an energy-efficient CNN architecture. We investigate the use of batch normalization optimization to streamline memory usage and computational complexity, aiming to uphold or improve model performance on CPU- based systems. Additionally, we evaluate its effectiveness across diverse datasets, focusing on energy efficiency and adaptability in different settings. Furthermore, we analyze how batch normalization influences the performance and effectiveness of activation functions and pooling layers in neural network designs. Our results highlight batch normalization’s ability to enhance computational efficiency, particularly on devices with limited resources.

Deep convolutional neural network (DCNN) is a powerful model for learning significant data at multiple levels of abstraction form an input image. However, training DCNN is often complicated because of parameter initialization, overfitting and convergence problems. Hence this work has been targeted to overcome the challenges of training DCNN with an optimized model. This chapter describes a deep learning framework for image classification with cifar-10 dataset. The model contains a set of convolutional layers with rectified linear unit activation function, max-pooling layers, and a fully-connected layer with softmax activation function. This model learns the features automatically and classifies the image without using the hand-crafted image based features. In this investigation, various optimizers have been applied in gradient descent technique for minimizing the loss function. Model with Adam optimizer constantly minimizes the objective function compared with other standard optimizers such as momentum, Rmsprop, and Adadelta. Dropout and batch normalization techniques are adapted to improve the model performance further by avoiding overfitting. Dropout function deactivates the insignificant node form the model after every epoch. The initialization of a large number of parameters in DCNN is regularized by batch normalization. Results obtained from the proposed model shows that batch normalization with dropout significantly improves the accuracy of the model with the tradeoff of computational complexity.

Normalization layers play an important role in deep network training. As one of the most popular normalization techniques, batch normalization (BN) has shown its effectiveness in accelerating the model training speed and improving model generalization capability. The success of BN has been explained from different views, such as reducing internal covariate shift, allowing the use of large learning rate, smoothing optimization landscape, etc. To make a deeper understanding of BN, in this work we prove that BN actually introduces a certain level of noise into the sample mean and variance during the training process, while the noise level depends only on the batch size. Such a noise generation mechanism of BN regularizes the training process, and we present an explicit regularizer formulation of BN. Since the regularization strength of BN is determined by the batch size, a small batch size may cause the under-fitting problem, resulting in a less effective model. To reduce the dependency of BN on batch size, we propose a momentum BN (MBN) scheme by averaging the mean and variance of current mini-batch with the historical means and variances. With a dynamic momentum parameter, we can automatically control the noise level in the training process. As a result, MBN works very well even when the batch size is very small (e.g., 2), which is hard to achieve by traditional BN.

Long-short term memory (LSTM) is a state-of-art and widely used model to forecast financial time series. However, primitive LSTM networks do not perform well due to over-fitting problems of the deep learning model and nonlinear and non-stationary characteristics of financial time series data. In addition, complete ensemble empirical mode decomposition with adaptive noise (CEEMDAN) is an outstanding data frequency decomposition technique that can decompose original time series into several intrinsic mode functions and a residue. Thus, this paper proposed a novel hybrid network CEEMDAN-LSTM-BN based on LSTM. Specifically, to avoid over-fitting, the modified LSTM-BN network consists of two LSTM layers, two Batch Normalization (BN) layers following each LSTM layer, and a dropout layer. Each of the intrinsic mode functions and the residue would be processed by CEEMDAN-LSTM-BN and the final prediction results are obtained by reconstructing each predictive series. The advantages of the proposed CEEMDANLSTM-BN networks are verified by comparing them to primitive LSTM, other hybrid models, and some famous machine learning models. Moreover, the robustness of the networks is assessed by numerical experiments on different stock indices datasets.

This pioneering work explores the immense potential of young coconut waste, a continuously marginalized residue of the food and beverage industry, to serve as an indispensable feedstock in the production of biochar. Through an examination of the key carbonization factors that include time, temperature, and concentrations of the activating agent, KOH, the outcomes offer relevant insights that could be leveraged to maximize biochar production for tailored applications. This study stands out for its innovative use of Artificial Neural Network (ANN) approaches for predictive modeling. Fifty datasets, supplemented with secondary data obtained from the literature and experiments, were utilized for the purposes of training, testing, and validating the neural network model. Here, the datasets were processed utilizing the Deep Neural Network (DNN) framework, which was designed and implemented with the minimal loss function framework feasible. The architectural configuration comprises the following; an input layer, four hidden layers (128-neuron dense layer, batch normalization, and 64-neuron dense layer, batch normalization), a dropout layer, and an output layer. With an R2 of 0.8238 for biochar yield and 0.7324 for iodine number, the trained DNN model showed a relatively high degree of accuracy in making predictions.

In the training process of convolutional neural networks (CNN), a batch normalization (BN) layer is often inserted after a convolution layer to accelerate the convergence of CNN training. In this work, we propose a BN processor that supports both the training and inference processes. To speed up the training of CNN, the proposed work develops an efficient dataflow integrating a novel BN processor design and the processing elements for convolution acceleration. We exploited the similarities in the calculations required for the BN forward and backward passes by sharing hardware elements between both passes, therefore reducing the area overhead. In addition to functional verification of the BN processor, we also completed Automatic Placement & Routing (APR) and conducted post-APR simulation on neural network training. Finally, the proposed solution not only significantly speeds up the CNN training process, but also achieves hardware saving.

Test Time Adaptation (TTA) has garnered significant attention in recent years, with the research focus on addressing distribution shifts during test time. As one fundamental component of many TTA methods, the Batch Normalization (BN) layer plays a crucial role in enabling the model adaptability. However, existing BN strategies can prove detrimental when the batch size is (extremely) small. In numerous real-world scenarios, limited hardware resources or just-in-time demand often necessitates adjusting models with very small batch sizes, making existing methods less practical. In this paper, we first showcase and thoroughly analyze the negative transfer phenomenon in previous TTA methods encountering extremely small batch sizes. Subsequently, we propose a novel batch size-agnostic method called NanoAdapt to effectively mitigate the negative transfer even with batch size 1. NanoAdapt is composed of three key components: a dynamic BN calibration strategy that leverages historical information and the Taylor series to refine the statistics estimations, an entropy-weighted gradient accumulation strategy that uses the entropy of each sample's label prediction to weigh and accumulate the loss for backpropagation, and a novel proxy computation graph to capture the sample interactions. Extensive experiments are conducted to validate the superiority of NanoAdapt, showing its consistent efficacy in improving existing TTA methods.

Well-known activation functions like ReLU or Leaky ReLU are non-differentiable at the origin. Over the years, many smooth approximations of ReLU have been proposed using various smoothing techniques. We propose new smooth approximations of a non-differentiable activation function by convolving it with approximate identities. In particular, we present smooth approximations of Leaky ReLU and show that they outperform several well-known activation functions in various datasets and models. We call this function Smooth Activation Unit (SAU). Replacing ReLU by SAU, we get 5.63%, 2.95%, and 2.50% improvement with ShuffleNet V2 (2.0x), PreActResNet 50 and ResNet 50 models respectively on the CIFAR100 dataset and 2.31% improvement with ShuffleNet V2 (1.0x) model on ImageNet-1k dataset.KeywordsSmooth Activation FunctionNeural Network

Rice, a staple crop that is extensively cultivated, is susceptible to the diseases of leaves that adversely affect yield and quality, namely bacterial blight, brown spot, and leaf smut. Common machine learning and simple deep learning models usually lack the accuracy, generalization, and computational competency needed for the confidence-boosting classification of diseases, particularly in real-world field settings. The outlined limitations, therefore, are to be overcome by an improved model of 19 layers of Convolutional Neural Network (CNN), which includes adaptive convolutional and pooling layers, ReLU activation, batch normalization, and dropout mechanisms to avoid overfitting. The system uses hyperparameter tuning to guarantee the best convergence and stable performance. The methodology involves state-of-the-art preprocessing techniques of images (noise reduction, contrast enhancement, data augmentation) for better input quality and increased dataset diversity. The model is trained on a labeled dataset of healthy and sick images of rice leaves. It applies categorical cross-entropy loss and the Adam optimizer to multi-class classification. The proposed model achieves a superlative classification accuracy of 98%, outdoing traditional machine learning and standard CNN baselines in terms of computational efficiency and precision. Findings affirm that the system is robust and scalable and, thus, can be deployed on edge devices in smart farming settings to provide prompt and fast detection of rice leaf diseases for timeous agricultural responses.

Algebraic Riccati Equations (AREs) are fundamental to optimal control design, particularly Linear Quadratic Regulator (LQR), Linear Quadratic Gaussian (LQG), and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$H_{\infty}$</tex> control. Traditional numerical methods for solving AREs, such as Schur decomposition, eigenvector methods, and Newton's method, while reliable, require computationally intensive iterative algorithms. For large systems or applications requiring real-time control decisions, these computational demands can become impractical. At times, finding a stabilizing solution can be challenging. This paper explores the use of neural networks as a fast, data-driven alternative for approximating ARE solutions. In this work, we trained a neural network to predict the solution matrix <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$P$</tex> directly from system parameters, bypassing traditional iterative solvers. The network architecture consists of four hidden layers with ReLU activation, dropout regularization, and batch normalization. As input, a 30-dimensional vector was used which consisted of the flattened system matrix and five additional stability-related features. Using a motivating aircraft control dynamical system from the literature as our benchmark, we generated a training dataset of 50,000 system-solution pairs through controlled perturbations of the motivating system, maintaining the original sparsity structure while varying numerical values. The trained network achieved a relative error of 6.730% when predicting the ARE solution for the original aircraft system. Closedloop simulations demonstrate that this approximation error does not significantly degrade control performance, with system trajectories closely matching those obtained using traditional ARE solvers. Once trained, the neural network provides near-instantaneous predictions, offering substantial computational advantages over iterative methods. This preliminary study explores the feasibility of neural network-based ARE solvers and provides a foundation for future work extending the approach to higher-dimensional systems and infinite-dimensional systems governed by partial differential equations.