Error Entropy in Classification Problems: A Univariate Data Analysis

Minimum total complex error entropy for adaptive filter

Methods for decoding movements from neural spike counts using adaptive filters often rely on minimizing the mean-squared error. However, for non-Gaussian distribution of errors, this approach is not optimal for performance. Therefore, rather than using probabilistic modeling, we propose an alternate non-parametric approach. In order to extract more structure from the input signal (neuronal spike counts) we propose using minimum error entropy (MEE), an information-theoretic approach that minimizes the error entropy as part of an iterative cost function. However, the disadvantage of using MEE as the cost function for adaptive filters is the increase in computational complexity. In this paper we present a comparison between the decoding performance of the analytic Wiener filter and a linear filter trained with MEE, which is then mapped to a parallel architecture in reconfigurable hardware tailored to the computational needs of the MEE filter. We observe considerable speedup from the hardware design. The adaptation of filter weights for the multiple-input, multiple-output linear filters, necessary in motor decoding, is a highly parallelizable algorithm. It can be decomposed into many independent computational blocks with a parallel architecture readily mapped to a field-programmable gate array (FPGA) and scales to large numbers of neurons. By pipelining and parallelizing independent computations in the algorithm, the proposed parallel architecture has sublinear increases in execution time with respect to both window size and filter order.

Global Navigation Satellite Systems (GNSSs) observables, such as those of the Global Positioning System (GPS), are frequently affected by multipath effects that cause unpredictable signal interference at the receiver, posing significant challenges for accurate state estimation in complex environments with non-Gaussian noise or outliers. The traditional extended Kalman filter (EKF), based on the minimum mean square error (MMSE) criterion, assumes Gaussian noise distributions and exhibits degraded performance under non-Gaussian conditions. To overcome this limitation, the minimum error entropy (MEE) criterion was proposed to reduce random uncertainty in estimation error distributions; however, due to its translation invariance property, MEE may inadvertently increase bias when errors contain systematic offsets, leading to poor convergence. In contrast, the maximum correntropy criterion (MCC) concentrates the error probability density function (PDF) around zero, enabling effective entropy adjustment even in the presence of bias and achieving superior error convergence. This paper presents the centered error entropy (CEE) extended Kalman filter (CEE-EKF) that integrates the complementary merits of both MEE and MCC approaches to overcome their individual limitations. Experimental validation in complex nonlinear GPS environments with non-Gaussian noise demonstrates that the CEE-EKF significantly outperforms individual algorithms in noise suppression, particularly exhibiting enhanced robustness and accuracy when handling outliers. These results offer an effective approach to enhancing the reliability of GPS navigation in challenging real-world environments, and the algorithm can be readily extended to other GNSS applications.

Error entropy is a potent tool for quantifying the similarity between two random vectors, occupying a significant position in state estimation. However, the Gaussian kernel function lacks flexibility in adjusting the shape of error entropy, thereby restricting its capacity to effectively handle non-Gaussian noise with unknown distribution. To address this issue, by introducing the degree of freedom (dof), this article constructs a student's t minimum error entropy (SMEE) criterion and derives a more robust Kalman filter termed SMEEKF based on this criterion, along with its corresponding Kalman smoother named SMEEKS. Furthermore, we prove the sufficient conditions for fixed-point iteration convergence and compute the floating-point complexity of proposed algorithms. Moreover, we provide algorithm's mean error behavior and mean-square error behavior in detail. In addition, we analyze the sensitivity of dof and kernel bandwidth to the proposed algorithms and validate the effectiveness of the proposed algorithms with complex noise in different scenarios.

In this paper, an adaptive method based on error entropy criterion is presented in order to eliminate noise from Electroencephalogram (EEG) signals. Conventionally, the Mean-Squared Error (MSE) criterion is the dominant criterion deployed in the adaptive filters for this purpose. By deploying MSE, only second-order moment of the error distribution is optimized, which is not adequate for the noisy EEG signal in which the contaminating noises are typically non-Gaussian. By minimizing error entropy, all moments of the error distribution are minimized; hence, using the Minimum Error Entropy (MEE) algorithm instead of MSE-based adaptive algorithms will improve the performance of noise elimination. Simulation results indicate that the proposed method has a better performance compared to conventional MSE-based algorithm in terms of signal to noise ratio and steady state error.

Shannon Entropy and Mean Square Errors for speeding the convergence of Multilayer Neural Networks: A comparative approach

The past decade has seen the rapid development of information theoretic learning and its applications in signal processing and machine learning. Specifically, minimum error entropy (MEE) and maximum correntropy criterion (MCC) have been widely studied in the literature. Although MEE and MCC are applied in many branches of knowledge and could outperform statistical criteria (such as mean square error), they have not been compared with each other from theoretical point of view. In some cases, MEE and MCC perform similarly to each other; however, under some conditions (e.g., in non-Gaussian environments), they act differently. This letter derives a new information theoretic relation between MEE and MCC, leading to better understanding of the theoretical differences, and illustrates the findings in a common example.

Noise and interference are the two most common and basic problems in wireless communication systems. The noise in wireless communication channels has the characteristics of randomness and impulsivity, so the performance of adaptive filtering algorithms based on geometric algebra (GA) and second-order statistics is greatly reduced in the wireless communication systems. In order to improve the performance of adaptive filtering algorithms in wireless communication systems, this paper proposes two novel GA-based adaptive filtering algorithms, which are deduced from the robust algorithms based on the minimum error entropy (MEE) criterion and the joint criterion (MSEMEE) of the MEE and the mean square error (MSE) with the help of GA theory. The noise interference in wireless communication is modeled by alpha-stable distribution which is in good agreement with the actual data in this paper. Simulation results show that for the mean square deviation (MSD) learning curve, the GA-based MEE (GA-MEE) algorithm has faster convergence rate and better steady-state accuracy compared to the GA-based maximum correntropy criterion algorithm (GA-MCC) under the same generalized signal-to-noise ratio (GSNR). The GA-MEE algorithm reduces the convergence rate, but improves the steady-state accuracy by 10–15 dB compared to the adaptive filtering algorithms based on GA and second-order statistics. For GA-based MSEMEE (GA-MSEMEE) algorithm, when GA-MSEMEE and the adaptive filtering algorithms based on GA and second-order statistics keep the same convergence rate, its steady-state accuracy is improved by 10–15 dB, and when GA-MSEMEE and GA-MEE maintain approximately steady-state accuracy, its convergence rate is improved by nearly 100 iterations. In addition, when the algorithms are applied to noise cancellation, the average recovery error of the two proposed algorithms is 7 points lower than that of other GA-based adaptive filtering algorithms. The results validate the effectiveness and superiority of the GA-MEE and GA-MSEMEE algorithms in the alpha-stable noise environment, providing new methods to deal with multi-channel interference in wireless networks.

In this paper, we propose the Fixed-Point Minimum Error Entropy (Fixed-Point MEE) as an alternative to the Minimum Error Entropy (MEE) algorithm for training adaptive systems. The fixed-point algorithms are different from the gradient methods like MEE, and are proven to be faster, more stable and step-size free. This characteristic is due to the second order update similar to Recursive Least-Squares (RLS) that tracks the Wiener solution with every update. We study the effect of design parameters, namely the forgetting factor, the window length, and the kernel size, on the convergence properties of the newly introduced recursive Fixed-Point MEE. Also, we test the performance of both the algorithms for two classic problems of system identification. Finally, we conclude that the Fixed-Point MEE performs better than MEE.

The minimum error entropy (MEE) criterion closely relies on the quadratic information potential (QIP) estimates of Renyi’s entropy. Nevertheless, the conventional obtained QIP estimates are numerically unstable, especially when the error samples contain outliers, resulting in the optimal solution of the MEE criterion deviating from the target vector to a certain extent. To address this problem, we introduce the M-estimate method from robust statistics into the QIP estimation process; thus, a robust estimation method called instantaneous M-estimate QIP (IM-QIP) is proposed, and several important properties of IM-QIP are presented. The proposed IM-QIP estimates could be implemented simply while ensuring great robustness. Naturally, we further propose the corresponding instantaneous M-estimate MEE (IM-MEE) criterion and apply it to adaptive filtering. A robust recursive adaptive filtering algorithm, called the recursive IM-MEE (RIM-MEE) algorithm, is proposed and analyzed in this article, along with its stability and theoretical steady-state performance. Additionally, we demonstrate that the RIM-MEE algorithm achieves a smaller theoretical bound compared to the traditional MEE algorithm under heavy-tailed and skewed noise conditions. The simulation results revealed the robustness of the IM-QIP estimates and verified the theoretical expectations and superior performance of the RIM-MEE algorithm.

The minimum error entropy (MEE) criterion in information theoretic learning is an efficient way to deal with non-Gaussian signal processing. And the minimum dispersion (MD) criterion has been widely applied in stable signal processing. In this letter, we show that there exists an equivalence between the MD criterion and the MEE criterion where symmetric $\alpha$ -stable $(S\alpha S)$ random variables are considered as the errors of the adaptive signal processing. As an application, we propose an algorithm with the MEE criterion for the time delay estimation (TDE) problem which was solved by the MD criterion.

This paper presents the results of a comparative study on the impact of various error functions in multilayer feedforward neural networks used for classification problems. The objective is the comparison of the properties of the error functions, both in terms of the training speed and the generalization ability. In an effort to avoid complexities introduced by more advanced learning algorithms, the simple backpropagation with momentum algorithm has been employed. A member of classification problems were solved with neural networks that have been trained with the usual mean square error function, the mean absolute error function, the cross-entropy or maximum likelihood function, the Kalman-Kwasny error function (1991), as well as a novel error function designed by the authors. The results indicate that, in most problems examined, an error function other than the usual mean square gives a better performance, both in terms of the number of epochs needed for training, as well as the obtained generalization ability of the trained network.

Novel robust minimum error entropy wasserstein distribution kalman filter under model uncertainty and non-gaussian noise

Recently, we have proposed the minimum error entropy (MEE) criterion as an information theoretic alternative to the widely used mean square error criterion in supervised adaptive system training. For this purpose, we have formulated a nonparametric estimator for Renyi's entropy that employs Parzen windowing. Mathematical investigation of the proposed entropy estimator revealed interesting insights about the process of information theoretical learning. This new estimator and the associated criteria have been applied to the supervised and unsupervised training of adaptive systems in a wide range of problems successfully. In this paper, we analyze the structure of the MEE performance surface around the optimal solution, and we derive the upper bound for the step size in adaptive linear neuron (ADALINE) training with the steepest descent algorithm using MEE. In addition, the effects of the entropy order and the kernel size in Parzen windowing on the shape of the performance surface and the eigenvalues of the Hessian at and around the optimal solution are investigated. Conclusions from the theoretical analyses are illustrated through numerical examples.

Minimization of the error entropy (MEE) cost function was introduced for nonlinear and non-Gaussian signal processing. In this paper, we show that this cost function has a close relation to a introduced correntropy criterion and M-estimation, thus it also theoretically explains the robustness of MEE to outliers. Based on this understanding, we propose a modification to the MEE cost function named minimization of error entropy with fiducial points, which sets the bias for MEE in an elegant and robust way. The performance of this new criterion is compared with the original MEE and the mean square error criterion (MSE) in robust regression and short-term prediction of a chaotic time series.