Efficient Entropy Estimation for Mutual Information Analysis Using B-Splines

Estimation of complete mutual information exploiting nonlinear magnitude-phase dependence: Application to spatial FNC for complex-valued fMRI data

Side-channel attacks have recently become one of primary threats that target modern computing devices. Several techniques have been proposed to improve the strength of side-channel attacks. Preprocessing is one of those techniques that improves the side-channel attacks by processing the measured leakage traces before performing the statistical analysis. The advantages of preprocessing techniques can potentially increase the performance and precision of side-channel attacks. Preprocessing of the measured leakage typically leads to faster side-channel attacks that require less number of measurements. A novel combined attack which utilizes the preprocessing techniques effectively is proposed in this work. Mutual information analysis (MIA) and correlation power analysis (CPA) are implemented for the preprocessing and attack steps interchangeably. For the first time, a combined attack is performed on a lightweight Prince cipher. The proposed preprocessing technique is demonstrated to reduce the required number of measurements to disclose a secret key by up to 25%.

Maximum Relevance Minimum Redundancy (mRMR) is a widely used feature selection method that is applied in a wide range of applications in various fields. mRMR adds to the optimal subset the features that have high relevance to the target variable while having minimum redundancy with each other. Mutual information is a key component of mRMR as it measures the degree of dependence between two variables. However, the real value of mutual information is not known and needs to be estimated. The aim of this study is to examine whether the choice of mutual information estimator affects the performance of mRMR. To this end, three variations of mRMR are compared. The first one uses Parzen window estimation to assess mutual information between continuous variables. The second is based on equidistant partitioning using the cells method, while the third incorporates a bias-corrected version of the same estimator. All methods are tested with and without a regularization term in the mRMR denominator, introduced to improve numerical stability. The evaluation is conducted on synthetic datasets where the target variable is defined as a combination of continuous features, simulating both linear and nonlinear dependencies. To demonstrate the applicability of the proposed methods, we also include a case study in real-world classification tasks. The study carried out showed that the choice of mutual information estimator can affect the performance of mRMR and it must be carefully selected depending on the dataset and the parameters of the examined problem. The application of the corrected mutual information estimator improves the performance of mRMR in the examined setup.

Side Channel Attacks are a popular modern cryptanalysis technique used by adversaries in embedded devices to break the security key. In these types of attacks, the attackers are keen on identifying the weaknesses of the physical implementation of the cryptosystem and utilize such vulnerabilities to extract the key. Power Analysis Attack is a form of Side Channel Attack in which, the adversary exploits power consumed by a cryptographic device during encryption to obtain the key. Mutual Information Analysis (MIA) is a concept introduced in information theory that measures the dependence between two random variables. In MIA based Power Analysis Attack, mutual information between two random variables is taken as the side channel distinguisher. Here, the two variables are physical leakages of the device and the power model based on key estimates. Since this method has more advantages to attackers compared to other methods, it is vital for cryptanalysts to find better countermeasures against this. But, due to the lack of efficient implementations it is hard for cryptanalysts to do that kind of research. In this paper, we present a methodology to accelerate MIA based Power Analysis Attacks using a GPU (Graphical Processor Unit) like NVIDIA Compute Unified Device Architecture (CUDA). Our proposed method promises to better utilize the capabilities of NVIDIA CUDA and obtain a speedup of more than 100 times compared to its sequential version.

Mutual information (MI) is a common criterion in independent component analysis (ICA) optimization. MI is derived from probability density functions (PDF). There are scenarios in which assuming a parametric form for the PDF leads to poor performance. Therefore, the need arises for non-parametric PDF and MI estimation. Existing non-parametric algorithms suffer from high complexity, particularly in high dimensions. To counter this obstacle, we present an ICA algorithm based on accelerated kernel entropy estimation. It achieves both high separation performance and low computational complexity. For K sources with N samples, our ICA algorithm has an iteration complexity of at most \(\mathcal{O}(KN {\rm log})N + K^{2}N\).

Estimating mutual information (MI) between two continuous random variables X and Y allows to capture non-linear dependencies between them, non-parametrically. As such, MI estimation lies at the core of many data science applications. Yet, robustly estimating MI for high-dimensional X and Y is still an open research question. In this paper, we formulate this problem through the lens of manifold learning. That is, we leverage the common assumption that the information of X and Y is captured by a low-dimensional manifold embedded in the observed high-dimensional space and transfer it to MI estimation. As an extension to state-of-the-art kNN estimators, we propose to determine the k-nearest neighbors via geodesic distances on this manifold rather than from the ambient space, which allows us to estimate MI even in the high-dimensional setting. An empirical evaluation of our method, G-KSG, against the state-of-the-art shows that it yields good estimations of MI in classical benchmark and manifold tasks, even for high dimensional datasets, which none of the existing methods can provide.

: Calculating the amount of information about a random function contained in another random function has important uses in communication theory. An expression for the mutual information for continuous time random processes has been given by Gelfand and Yaglom, Chiang, and Perez by generalizing Shannon's result in a natural way. Under a condition of absolute continuity of measures the continuous time expression has the same form as Shannon's result. For two Gaussian processes Gelfand and Yaglom express the mutual information in terms of a mean square estimation error. We generalize this result to diffusion processes and express the solution in a different form which is more naturally related to a corresponding filtering problem. We also use these results to calculate some information rates.

Mutual Information (MI) is a useful Information Theory tool for the recognition of mutual dependence between data sets. Several methods have been developed fore estimation of MI when both data sets are of the discrete type or when both are of the continuous type. However, MI estimation between a discrete range data set and a continuous range data set has not received so much attention. We therefore present here a method for the estimation of MI for this case, based on the kernel density approximation. This calculation may be of interest in diverse contexts. Since MI is closely related to the Jensen Shannon divergence, the method developed here is of particular interest in the problems of sequence segmentation and set comparisons.

A large variety of side channel analyses performed on embedded devices involve the linear correlation coefficient as wrong-key distinguisher. This coefficient is actually a sound statistical tool to quantify linear dependencies between univariate variables. At CHES 2008, Gierlichs et al. proposed to use the mutual information measure as an alternative to the correlation coefficient since it detects any kind of statistical dependency. Substituting it for the correlation coefficient may indeed be considered as a natural extension of the existing attacks. Nevertheless, the first published applications have raised several open issues. In this paper, we conduct a theoretical analysis of MIA in the Gaussian leakage model to explore the reasons why and when it is a sound key recovery attack. Also, we generalise MIA to higher-orders (i.e., against masked implementations). Secondly, we address the main practical issue of MIA: the mutual information estimation which itself relies on the estimation of statistical distributions. We describe three classical estimation methods and we apply them in the context of MIA. Eventually, we present various attack simulations and practical attack experiments that allow us to check the efficiency of MIA in practice and to compare it to classical correlation-based attacks.

Diversified text-to-image generation via deep mutual information estimation

Medical image registration using Edgeworth-based approximation of Mutual Information

In deep clustering frameworks, autoencoder (AE)- or variational AE-based clustering approaches are the most popular and competitive ones that encourage the model to obtain suitable representations and avoid the tendency for degenerate solutions simultaneously. However, for the clustering task, the decoder for reconstructing the original input is usually useless when the model is finished training. The encoder-decoder architecture limits the depth of the encoder so that the learning capacity is reduced severely. In this article, we propose a decoder-free variational deep embedding for unsupervised clustering (DFVC). It is well known that minimizing reconstruction error amounts to maximizing a lower bound on the mutual information (MI) between the input and its representation. That provides a theoretical guarantee for us to discard the bloated decoder. Inspired by contrastive self-supervised learning, we can directly calculate or estimate the MI of the continuous variables. Specifically, we investigate unsupervised representation learning by simultaneously considering the MI estimation of continuous representations and the MI computation of categorical representations. By introducing the data augmentation technique, we incorporate the original input, the augmented input, and their high-level representations into the MI estimation framework to learn more discriminative representations. Instead of matching to a simple standard normal distribution adversarially, we use end-to-end learning to constrain the latent space to be cluster-friendly by applying the Gaussian mixture distribution as the prior. Extensive experiments on challenging data sets show that our model achieves higher performance over a wide range of state-of-the-art clustering approaches.

A class of implementations of mutual information (MI) based image registration estimate MI from the joint histogram of the overlap of two images. The consequence of this approach is that the MI estimate thus obtained is not overlap invariant: its value tends to increase when the overlapped region is getting smaller. When the two images are very noisy or are so different that the correct MI peak is very weak, it may lead to incorrect registration results using the maximization of mutual information (MMI) criterion. In this paper, we present a new joint histogram estimation scheme for overlap invariant MI estimation. The idea is to keep it a constant the number of samples used for joint histogram estimation. When one image is completely within another, this condition is automatically satisfied. When one image (floating image) partially overlaps another image (reference image) after applying a certain geometric transformation, it is possible that, for a pixel from the floating image, there is no corresponding point in the reference image. In this case, we generate its corresponding point by assuming that its value is a random variable following the distribution of the reference image. In this way, the number of samples utilized for joint histogram estimation is always the same as that of the floating image. The efficacy of this joint histogram estimation scheme is demonstrated by using several pairs of remote sensing images. Our results show that the proposed method is able to produce a mutual information measure that is less sensitive to the size of overlap and the peak found is more reliable for image registration.

Mutual information (MI) is quite popular as a cost function for intensity based registration of images due to its ability to handle highly non-linear relationships between intensities of the two images. More recently, quadratic mutual information (QMI) has been proposed as an alternative measure that computes Euclidean distance instead of KL divergence between the joint and the product of the marginal densities of pixel intensities. In this paper, we examine the conditions under which QMI is advantageous over the classical MI measure, for the image registration problem. We show that QMI is a better cost function to use for optimization methods such as stochastic gradient descent. We show that the QMI cost function remains much smoother than the classical MI measure on stochastic subsampling of the image data. As a consequence, QMI has a higher probability of convergence, even for larger degrees of initial misalignment of the images.

Recently, a generic DPA attack using the mutual information index as the side channel distinguisher has been introduced. Mutual Information Analysis’s (MIA) main interest is its claimed genericity. However, it requires the estimation of various probability density functions (PDF), which is a task that involves the complicated problem of selecting tuning parameters. This problem could be the cause of the lower efficiency of MIA that has been reported. In this paper, we introduce an approach that selects the tuning parameters with the goal of optimizing the performance of MIA. Our approach differs from previous works in that it maximizes the ability of MIA to discriminate one key among all guesses rather than optimizing the accuracy of PDF estimates. Application of this approach to various leakage traces confirms the soundness of our proposal.