Medical image registration using Edgeworth-based approximation of Mutual Information

: Calculating the amount of information about a random function contained in another random function has important uses in communication theory. An expression for the mutual information for continuous time random processes has been given by Gelfand and Yaglom, Chiang, and Perez by generalizing Shannon's result in a natural way. Under a condition of absolute continuity of measures the continuous time expression has the same form as Shannon's result. For two Gaussian processes Gelfand and Yaglom express the mutual information in terms of a mean square estimation error. We generalize this result to diffusion processes and express the solution in a different form which is more naturally related to a corresponding filtering problem. We also use these results to calculate some information rates.

Mutual Information (MI) has been extensively used as a similarity measure in image registration and motion estimation, and it is particularly robust for 3D multimodal medical image registration. However, MI estimators are known i) to have a high variance and ii) to be computationally costly. In order to overcome these drawbacks, we propose a new similarity measure based on an Edgeworth-based third order expansion of MI and named 3-EMI in the following. This kind of approximation is well known in signal processing, and especially in Independent Components Analysis (ICA), but its computation is easier since data can be prewhitened contrary to images in registration. The performance of affine and non-rigid registrations based on the 3-EMI metric is studied through computer results in the context of cardiac multislice computed tomography. In fact, an estimate of the 3-EMI metric using sample statistics is compared to a histogram-based estimate of the standard normalized MI, showing a better robustness of the 3-EMI measure with respect to the range of the searched deformation. In addition, in practice, one part of the floating image may be missing regarding the reference image. Computer results show that our approach is less sensitive to such a practical problem.

A class of implementations of mutual information (MI) based image registration estimate MI from the joint histogram of the overlap of two images. The consequence of this approach is that the MI estimate thus obtained is not overlap invariant: its value tends to increase when the overlapped region is getting smaller. When the two images are very noisy or are so different that the correct MI peak is very weak, it may lead to incorrect registration results using the maximization of mutual information (MMI) criterion. In this paper, we present a new joint histogram estimation scheme for overlap invariant MI estimation. The idea is to keep it a constant the number of samples used for joint histogram estimation. When one image is completely within another, this condition is automatically satisfied. When one image (floating image) partially overlaps another image (reference image) after applying a certain geometric transformation, it is possible that, for a pixel from the floating image, there is no corresponding point in the reference image. In this case, we generate its corresponding point by assuming that its value is a random variable following the distribution of the reference image. In this way, the number of samples utilized for joint histogram estimation is always the same as that of the floating image. The efficacy of this joint histogram estimation scheme is demonstrated by using several pairs of remote sensing images. Our results show that the proposed method is able to produce a mutual information measure that is less sensitive to the size of overlap and the peak found is more reliable for image registration.

Estimation of complete mutual information exploiting nonlinear magnitude-phase dependence: Application to spatial FNC for complex-valued fMRI data

Diversified text-to-image generation via deep mutual information estimation

Mutual Information (MI) is a useful Information Theory tool for the recognition of mutual dependence between data sets. Several methods have been developed fore estimation of MI when both data sets are of the discrete type or when both are of the continuous type. However, MI estimation between a discrete range data set and a continuous range data set has not received so much attention. We therefore present here a method for the estimation of MI for this case, based on the kernel density approximation. This calculation may be of interest in diverse contexts. Since MI is closely related to the Jensen Shannon divergence, the method developed here is of particular interest in the problems of sequence segmentation and set comparisons.

Estimating mutual information (MI) between two continuous random variables X and Y allows to capture non-linear dependencies between them, non-parametrically. As such, MI estimation lies at the core of many data science applications. Yet, robustly estimating MI for high-dimensional X and Y is still an open research question. In this paper, we formulate this problem through the lens of manifold learning. That is, we leverage the common assumption that the information of X and Y is captured by a low-dimensional manifold embedded in the observed high-dimensional space and transfer it to MI estimation. As an extension to state-of-the-art kNN estimators, we propose to determine the k-nearest neighbors via geodesic distances on this manifold rather than from the ambient space, which allows us to estimate MI even in the high-dimensional setting. An empirical evaluation of our method, G-KSG, against the state-of-the-art shows that it yields good estimations of MI in classical benchmark and manifold tasks, even for high dimensional datasets, which none of the existing methods can provide.

In deep clustering frameworks, autoencoder (AE)- or variational AE-based clustering approaches are the most popular and competitive ones that encourage the model to obtain suitable representations and avoid the tendency for degenerate solutions simultaneously. However, for the clustering task, the decoder for reconstructing the original input is usually useless when the model is finished training. The encoder-decoder architecture limits the depth of the encoder so that the learning capacity is reduced severely. In this article, we propose a decoder-free variational deep embedding for unsupervised clustering (DFVC). It is well known that minimizing reconstruction error amounts to maximizing a lower bound on the mutual information (MI) between the input and its representation. That provides a theoretical guarantee for us to discard the bloated decoder. Inspired by contrastive self-supervised learning, we can directly calculate or estimate the MI of the continuous variables. Specifically, we investigate unsupervised representation learning by simultaneously considering the MI estimation of continuous representations and the MI computation of categorical representations. By introducing the data augmentation technique, we incorporate the original input, the augmented input, and their high-level representations into the MI estimation framework to learn more discriminative representations. Instead of matching to a simple standard normal distribution adversarially, we use end-to-end learning to constrain the latent space to be cluster-friendly by applying the Gaussian mixture distribution as the prior. Extensive experiments on challenging data sets show that our model achieves higher performance over a wide range of state-of-the-art clustering approaches.

The Correlation Power Analysis (CPA) is probably the most used side-channel attack because it seems to fit the power model of most standard CMOS devices and is very efficiently computed. However, the Pearson correlation coefficient used in the CPA measures only linear statistical dependences where the Mutual Information (MI) takes into account both linear and nonlinear dependences. Even if there can be simultaneously large correlation coefficients quantified by the correlation coefficient and weak dependences quantified by the MI, we can expect to get a more profound understanding about interactions from an MI Analysis (MIA). We study methods that improve the non-parametric Probability Density Functions (PDF) in the estimation of the entropies and, in particular, the use of B-spline basis functions as pdf estimators. Our results indicate an improvement of two fold in the number of required samples compared to a classic MI estimation. The B-spline smoothing technique can also be applied to the rencently introduced Cramer-von-Mises test.

The training dynamics of hidden layers in deep learning are poorly understood in theory. Recently, the Information Plane (IP) was proposed to analyze them, which is based on the information-theoretic concept of mutual information (MI). The Information Bottleneck (IB) theory predicts that layers maximize relevant information and compress irrelevant information. Due to the limitations in MI estimation from samples, there is an ongoing debate about the properties of the IP for the supervised learning case. In this work, we derive a theoretical convergence for the IP of autoencoders. The theory predicts that ideal autoencoders with a large bottleneck layer size do not compress input information, whereas a small size causes compression only in the encoder layers. For the experiments, we use a Gram-matrix based MI estimator recently proposed in the literature. We propose a new rule to adjust its parameters that compensates scale and dimensionality effects. Using our proposed rule, we obtain experimental IPs closer to the theory. Our theoretical IP for autoencoders could be used as a benchmark to validate new methods to estimate MI in neural networks. In this way, experimental limitations could be recognized and corrected, helping with the ongoing debate on the supervised learning case.

A new mutual information (MI)-based feature-selection method to solve the so-called large p and small n problem experienced in a microarray gene expression-based data is presented. First, a grid-based feature clustering algorithm is introduced to eliminate redundant features. A huge gene set is then greatly reduced in a very efficient way. As a result, the computational efficiency of the whole feature-selection process is substantially enhanced. Second, MI is directly estimated using quadratic MI together with Parzen window density estimators. This approach is able to deliver reliable results even when only a small pattern set is available. Also, a new MI-based criterion is proposed to avoid the highly redundant selection results in a systematic way. At last, attributed to the direct estimation of MI, the appropriate selected feature subsets can be reasonably determined.

Medical image registration is one basic task in medical image processing. It can align multi images coming from different modes or time, and then it offer guarantee to images post processing. In this article we mainly research the mutual information matching method, and using image pixel grey to compute mutual information. We put forward improved registration algorithm. We compare improved mutual information registration method and classical first grade mutual information and second grade mutual information registration method. During aligning, by extracting image feature signs or edge information, using them as reference feature value of two images aligning information and improving multi parameters optimization algorithm, we get satisfactory experimental results. Regarding to the two-dimensional medical image registration, we utilized the improved method combining gradient information and mutual information, has solved the question of partial extreme value without spatial information when pure use the mutual information. The feasibility of this algorithm is proved by corresponding experiments.

This paper proposes a method to maximize the rate of reliable communication for polar codes operating on channels with memory. The channel is learned implicitly from data by optimizing a neural polar decoder (NPD). This approach enables simultaneous optimization of the code rate over the input distribution and the design of a practical coding scheme within the framework of polar codes. The proposed approach applies to scenarios where the channel model is unknown and treated as a black-box that produces output samples from input samples. We use NPDs to estimate the mutual information (MI) between the channel inputs and outputs, and optimize a parametric model of the input distribution. The methodology involves a two-phase process: a training phase and an inference phase. In the training phase, two steps are repeated iteratively. The first step optimizes the NPD to estimate the MI of the channel inputs and outputs. The second step improves the input distribution parameters by maximizing the MI estimate obtained with the NPD. In the inference phase, the optimized model is used to construct polar codes. This approach uses the Honda-Yamamoto (HY) scheme, which implements polar codes with optimized input distributions, together with list decoding. Experimental results on memoryless and finite state channels (FSCs) demonstrate the effectiveness of this approach, particularly in cases where the channel’s capacity-achieving input distribution is non-uniform. For these cases, significant improvements in MI and bit error rates (BERs) are shown over those achieved by uniform and independent and identically distributed (i.i.d.) input distributions, validating our method for block lengths up to 1024. This data-driven approach can be utilized in real-world communication systems, bridging theoretical capacity estimation and practical coding performance.

Reconstructing the incomplete point cloud into the complete and uniform one is a fundamental task in 3D point cloud processing. Some existing studies have explored the use of deep learning networks and representation learning to implement point cloud completion, but the completed shapes still appear unrealistic and non-uniform. To address this issue, we propose the idea of simulating the process of point cloud completion with local-to-global reasoning (LGR). Motivated to achieve the fine LGR, a novel mutual information (MI) maximization-based similarity operation is proposed, which realizes the reconstruction from incomplete point cloud to complete one by maximizing MI between global features and the prior features from the same 3D object. We adopt the Jenson-Shannon MI estimator to maximize the MI between global features and the priors in specific dimensions, which effectively increases the similarity of global and prior features. Compared with other existing works and similarity operations, the superior results indicate the efficacy of the proposed method and its advantages over existing ones in both the synthetic and real-world datasets. Our source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/wendydidi/MISO-PCN</uri> .

Maximum Relevance Minimum Redundancy (mRMR) is a widely used feature selection method that is applied in a wide range of applications in various fields. mRMR adds to the optimal subset the features that have high relevance to the target variable while having minimum redundancy with each other. Mutual information is a key component of mRMR as it measures the degree of dependence between two variables. However, the real value of mutual information is not known and needs to be estimated. The aim of this study is to examine whether the choice of mutual information estimator affects the performance of mRMR. To this end, three variations of mRMR are compared. The first one uses Parzen window estimation to assess mutual information between continuous variables. The second is based on equidistant partitioning using the cells method, while the third incorporates a bias-corrected version of the same estimator. All methods are tested with and without a regularization term in the mRMR denominator, introduced to improve numerical stability. The evaluation is conducted on synthetic datasets where the target variable is defined as a combination of continuous features, simulating both linear and nonlinear dependencies. To demonstrate the applicability of the proposed methods, we also include a case study in real-world classification tasks. The study carried out showed that the choice of mutual information estimator can affect the performance of mRMR and it must be carefully selected depending on the dataset and the parameters of the examined problem. The application of the corrected mutual information estimator improves the performance of mRMR in the examined setup.