On variational inference and maximum likelihood estimation with the $ \lambda $-exponential family

The Maximum Likelihood (ML) and Best Linear Unbiased (BLU) estimators of the location and scale parameters of an extreme value distribution (Lawless [1982]) are compared under conditions of small sample sizes and Type I censorship. The comparisons were made in terms of the mean square error criterion. According to this criterion, the ML estimator of σ in the case of very small sample sizes (n < 10) and heavy censorship (low censoring time) proved to be more efficient than the corresponding BLU estimator. However, the BLU estimator for σ attains parity with the corresponding ML estimator when the censoring time increases even for sample sizes as low as 10. The BLU estimator of σ attains equivalence with the ML estimator when the sample size increases above 10, particularly when the censoring time is also increased. The situation is reversed when it came to estimating the location parameter μ, as the BLU estimator was found to be consistently more efficient than the ML estimator despite the improved performance of the ML estimator when the sample size increases. However, computational ease and convenience favor the ML estimators. Keywords: maximum likelihood estimatorsbest linear unbiased estimatorslife-testingreliability

For a balanced two‐way mixed model, the maximum likelihood (ML) and restricted ML (REML) estimators of the variance components were obtained and compared under the non‐negativity requirements of the variance components by Lee and Kapadia (1984). In this note, for a mixed (random blocks) incomplete block model, explicit forms for the REML estimators of variance components are obtained. They are always non‐negative and have smaller mean squared error (MSE) than the analysis of variance (AOV) estimators. The asymptotic sampling variances of the maximum likelihood (ML) estimators and the REML estimators are compared and the balanced incomplete block design (BIBD) is considered as a special case. The ML estimators are shown to have smaller asymptotic variances than the REML estimators, but a numerical result in the randomized complete block design (RCBD) demonstrated that the performances of the REML and ML estimators are not much different in the MSE sense.

Over-the-air computation (OAC) is a promising technique to realize fast model aggregation in the uplink of federated edge learning (FEEL). OAC, however, hinges on accurate channel-gain precoding and strict synchronization among edge devices, which are challenging in practice. As such, how to design the maximum likelihood (ML) estimator in the presence of residual channel-gain mismatch and asynchronies is an open problem. To fill this gap, this paper formulates the problem of misaligned OAC for FEEL and puts forth a whitened matched filtering and sampling scheme to obtain oversampled, but independent samples from the misaligned and overlapped signals. Given the whitened samples, a sum-product ML (SP-ML) estimator and an aligned-sample estimator are devised to estimate the arithmetic sum of the transmitted symbols. In particular, the computational complexity of our SP-ML estimator is linear in the packet length, and hence is significantly lower than the conventional ML estimator. Extensive simulations on the test accuracy versus the average received energy per symbol to noise power spectral density ratio (EsN0) yield two main results: 1) In the low EsN0 regime, the aligned-sample estimator can achieve superior test accuracy provided that the phase misalignment is not severe. In contrast, the ML estimator does not work well due to the error propagation and noise enhancement in the estimation process. 2) In the high EsN0 regime, the ML estimator attains the optimal learning performance regardless of the severity of phase misalignment. On the other hand, the aligned-sample estimator suffers from a test-accuracy loss caused by phase misalignment.

The Multiple Comparison Procedures with Modeling Techniques (MCP-Mod) framework has been recently approved by the U.S. Food, Administration, and European Medicines Agency as fit-for-purpose for phase II studies. Nonetheless, this approach relies on the asymptotic properties of Maximum Likelihood (ML) estimators, which might not be reasonable for small sample sizes. In this paper, we derived improved ML estimators and correction for their covariance matrices in the censored Weibull regression model based on the corrective and preventive approaches. We performed two simulation studies to evaluate ML and improved ML estimators with their covariance matrices in (i) a regression framework (ii) the Multiple Comparison Procedures with Modeling Techniques framework. We have shown that improved ML estimators are less biased than ML estimators yielding Wald-type statistics that controls type I error without loss of power in both frameworks. Therefore, we recommend the use of improved ML estimators in the MCP-Mod approach to control type I error at nominal value for sample sizes ranging from 5 to 25 subjects per dose.

When in a full exponential family the maximum likelihood estimate (MLE) does not exist, the MLE may exist in the Barndorff-Nielsen completion of the family. We propose a practical algorithm for finding the MLE in the completion based on repeated linear programming using the R contributed package rcdd and illustrate it with three generalized linear model examples. When the MLE for the null hypothesis lies in the completion, likelihood ratio tests of model comparison are almost unchanged from the usual case. Only the degrees of freedom need to be adjusted. When the MLE lies in the completion, confidence intervals are changed much more from the usual case. The MLE of the natural parameter can be thought of as having gone to infinity in a certain direction, which we call a generic direction of recession. We propose a new one-sided confidence interval which says how close to infinity the natural parameter may be. This maps to one-sided confidence intervals for mean values showing how close to the boundary of their support they may be.

A maximum likelihood (ML) estimator is developed for determining time delay between signals received at two spatially separated sensors in the presence of uncorrelated noise. This ML estimator can be realized as a pair of receiver prefilters followed by a cross correlator. The time argument at which the correlator achieves a maximum is the delay estimate. The ML estimator is compared with several other proposed processors of similar form. Under certain conditions the ML estimator is shown to be identical to one proposed by Hannan and Thomson [10] and MacDonald and Schultheiss [21]. Qualitatively, the role of the prefilters is to accentuate the signal passed to the correlator at frequencies for which the signal-to-noise (S/N) ratio is highest and, simultaneously, to suppress the noise power. The same type of prefiltering is provided by the generalized Eckart filter, which maximizes the S/N ratio of the correlator output. For low S/N ratio, the ML estimator is shown to be equivalent to Eckart prefiltering.

In estimation theory, the asymptotic (in the number of samples) efficiency of the Maximum Likelihood (ML) estimator is a well known result [1]. Nevertheless, in some scenarios, the number of snapshots may be small. We recently investigated the asymptotic behavior of the Stochastic ML (SML) estimator at high Signal to Noise Ratio (SNR) and finite number of samples [2] in the array processing framework: we proved the non-Gaussiannity of the SML estimator and we obtained the analytical expression of the variance for the single source case. In this paper, we generalize these results to multiple sources, and we obtain variance expressions which demonstrate the non-efficiency of SML estimates.

The reverberation power spectral density (PSD) is often required for dereverberation and noise reduction algorithms. In this work, we compare two maximum likelihood (ML) estimators of the reverberation PSD in a noisy environment. In the first estimator, the direct path is first blocked. Then, the ML criterion for estimating the reverberation PSD is stated according to the probability density function of the blocking matrix (BM) outputs. In the second estimator, the speech component is not blocked. Instead, the ML criterion for estimating the speech and reverberation PSD is stated according to the probability density function of the microphone signals. To compare the expected mean square error (MSE) between the two ML estimators of the reverberation PSD, the Cramer–Rao Bounds (CRBs) for the two ML estimators are derived. We show that the CRB for the joint reverberation and speech PSD estimator is lower than the CRB for estimating the reverberation PSD from the BM outputs. Experimental results show that the MSE of the two estimators indeed obeys the CRB curves. Experimental results of multimicrophone dereverberation and noise reduction algorithm show the benefits of using the ML estimators in comparison with another baseline estimators.

Cooperative communication combats the effects offadingwithoutthe need for multiple antennas.The cooperative networks use multiple distributed nodes that virtually behave like multiple antennas each having their own local oscillators. But the oscillators cause multiple Carrier Frequency Offsets (CFO), unknown channel gains and Multiple Timing Offsets (MTO). For a good performance of the system perfect synchronization is needed which is achieved by estimation and compensation of multiple offsets. Also, the coverage and capacity can be improved by employing intermediate nodes called relays within cooperative networks. These cooperative relay networks can perform in a better manner even when considering MCFOs, MTOs and unknown channel gains by having more computationally efficient decoders that can use Maximum Likelihood (ML) estimation subject to the Cramer-Rao Lower Bound (CRLB) on the variance of any impartial estimate. The simulation results show that frequency offset still exist after Amplify and Forward (AF) relaying. The performance of a conventional cooperative network seems to have improved with respect to Bit Error Rate of the system when an AF relay is hosted. The estimate of the frequency offset is chosen such that it gives the signal, a high probability of occurrence using ML estimator. In ML estimate, since the bias and variance is difficult to compute, a lower bound called the CRLB on the variance of any unbiased estimate is derived which is satisfied by the ML estimator.Keywords: AF, Cooperative Communication, CFO, ML Estimation and Compensation, OFDM

Integrated information and variational inference provide influential mathematical frameworks in neuroscience. Yet, the understanding of the connection between the two is limited. Here, we study a minimal model to show how variational inference displays large integrated information for highly correlated target distributions, in contrast with alternative inference approaches like maximum likelihood estimation.

Laser pointing systems for small targets are mainly confronted with two pointing errors, jitter and boresight, arising due to atmospheric turbulence, mechanical vibrations and errors in optical alignment. Knowledge of these parameters provides information about the quality of the pointing and tracking system. In the past, several techniques have been investigated to estimate these parameters from returned laser signals. These include the key ratio technique, the chi-squared method and the maximum likelihood (ML) estimation technique. These techniques have been studied in the literature for Gaussian irradiance profiles. In particular, the ML estimation technique has been found to provide excellent results. In this paper, we extend the ML estimation technique from Gaussian profiles to near-Gaussian irradiance profiles. Our results show that the modified estimator performs much better than a direct application of the original ML estimator.

The EM algorithm is a widely used technique for finding maximum likelihood (ML) estimates when the data are not fully observed. Despite its popularity for computing ML estimates in unrestricted problems and the need for simplified computations for problems with equality and inequality restrictions, there have been few applications of the algorithm to restricted ML estimation. The EM algorithm is presented for restricted ML estimation and provides its applications to the probit model under equality and inequality restrictions using two small data sets.

Coherent processing of data from distributed HF receivers requires the receivers to be synchronized to an accuracy within fractions of a wavelength. The HF band extends from 3 MHz to 30 MHz and has wavelengths corresponding to time delays ranging from approximately 33 ns to 333 ns. Thus the receivers must be synchronized to within a few nanoseconds. This is typically beyond the capabilities of even GPS-disciplined oscillators. Traditional clock synchronization methods require distributing a common reference signal to all the receivers, either by cables or by using a local reference transmitter. Both these approaches introduce extra cost and complexity. In this paper, we present an approach for clock synchronization which uses the coherent component of wideband ambient HF background noise to estimate time delays between receivers. The approach exploits the flexibility of inexpensive broadband direct-conversion digital receivers in the 3 - 30 MHz band. Two techniques are presented for measuring time delays between receivers: 1) Time-domain Green's Function estimator based on an approach recently used in underwater acoustics, and 2) an optimal Maximum Likelihood (ML) estimator. Both estimators are shown to choose the maximum of particular generalized cross correlations. The Camer-Rao Lower Bound (CRLB) for time delay estimation from ambient noise is also derived and simulation results are presented that demonstrate the performance of the estimators. For low signal to noise ratio (SNR) or low time-bandwidth product (TBP), the estimators do not achieve the CRLB due to peak ambiguities in the generalized cross correlations. However, for sufficiently large SNR and TBP, we demonstrate that the ML estimator is able to overcome the peak ambiguities and achieve the CRLB. In this case, the ML estimator is able to achieve sub-nanosecond RMS delay errors using a 2.5 s observation of 2 MHz bandwidth isotropic HF ambient noise. Thus the ML estimator is the preferred method for synchronizing the distributed receivers.

Examining the robustness properties of maximum likelihood (ML) estimators of parameters in exponential power and generalized t distributions has been considered together. The well-known asymptotic properties of ML estimators of location, scale and added skewness parameters in these distributions are studied. The ML estimators for location, scale and scale variant (skewness) parameters are represented as an iterative reweighting algorithm (IRA) to compute the estimates of these parameters simultaneously. The artificial data are generated to examine performance of IRA for ML estimators of parameters simultaneously. We make a comparison between these two distributions to test the fitting performance on real data sets. The goodness of fit test and information criteria approve that robustness and fitting performance should be considered together as a key for modeling issue to have the best information from real data sets.

Node self-localization is a key research topic for wireless sensor networks (WSNs). There are two main algorithms, the triangulation method and the maximum likelihood (ML) estimator, for angle of arrival (AOA) based self-localization. The ML estimator requires a good initialization close to the true location to avoid divergence, while the triangulation method cannot obtain the closed-form solution with high efficiency. Here, we develop a set of efficient closed-form AOA based self-localization algorithms using auxiliary variables based methods. First, we formulate the self-localization problem as a linear least squares problem using auxiliary variables. Based on its closed-form solution, a new auxiliary variables based pseudo-linear estimator (AVPLE) is developed. By analyzing its estimation error, we present a bias compensated AVPLE (BCAVPLE) to reduce the estimation error. Then we develop a novel BCAVPLE based weighted instrumental variable (BCAVPLE-WIV) estimator to achieve asymptotically unbiased estimation of locations and orientations of unknown nodes based on prior knowledge of the AOA noise variance. In the case that the AOA noise variance is unknown, a new AVPLE based WIV (AVPLE-WIV) estimator is developed to localize the unknown nodes. Also, we develop an autonomous coordinate rotation method to overcome the tangent instability of the proposed algorithms when the orientation of the unknown node is near <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\pi/2$</tex> </formula> . We also derive the Cramér-Rao lower bound of the ML estimator. Extensive simulations demonstrate the new algorithms achieve much higher localization accuracy than the triangulation method and avoid local minima and divergence in iterative ML estimators.