Active-LATHE: An Active Learning Algorithm for Boosting the Error Exponent for Learning Homogeneous Ising Trees

Suppose Q is a family of discrete memoryless channels. An unknown member of Q will be available, with perfect, causal output feedback for communication. Is there a coding scheme (possibly with variable transmission time) that can achieve the Burnashev error exponent uniformly over Q ? For two families of channels we show that the answer is yes. Furthermore, for each of these two classes, in addition to achieve the maximum error exponent, it is possible to uniformly attain any given fraction of the channel capacity. Therefore, in terms of achievable rates and delay, there are situations in which the knowledge of the channel becomes irrelevant. In the second part of the thesis, we show that for arbitrary sets of channels the Burnashev error exponent cannot in general be uniformly achieved. In particular we give a sufficient condition for a pair of channels so that no coding strategy reaches Burnashev's exponent simultaneously on both channels. As a third part we study a scenario where communication is carried by first testing the channel by means of a training sequence, then coding according to the channel estimate. We provide an upper bound on the maximum achievable error exponent of such coding schemes. This bound is typically much lower than the maximum achievable error exponent over a channel with feedback. For example in the case of binary symmetric channels this bound has a slope that vanishes at capacity. This result suggests that in terms of error exponent, a good universal feedback scheme combines channel estimation with information delivery, rather than separating them. In the final chapter, we address the question of communicating quickly and reliably. We consider a simple situation of two message communication over a known channel with feedback. We propose a simple decoding rule, and show that it minimizes a weighted combination of the probability of error and decoding delay for a certain range of crossover probabilities and combination weights.

Along with the channel capacity, the error exponent is one of the most important information-theoretic measures of reliability, as it sets ultimate bounds on the performance of communication systems employing codes of finite complexity. In this paper, we derive an exact analytical expression for the random coding error exponent, which provides significant insight regarding the ultimate limits to communications through Nakagami-m fading channels. An important fact about this error exponent is that it determines the behavior of error probability in terms of the transmission rate and the code length that reflects the coding complexity required to achieve a certain level of reliability. Moreover, from the derived analytical expression, we can easily compute the necessary codeword length without extensive Monte-Carlo simulation to achieve a predefined upper bound for error probability at a rate below the channel capacity. We also improve the random coding bound by expurgating bad codewords from the code ensemble, since random coding error exponent is determined by selecting codewords independently according to the input distribution where good and bad codewords contribute equally to the overall average error probability. Finally, we derive exact analytical expressions for the cutoff rate, critical rate, and expurgation rate and verify the analytical expressions via Monte-Carlo simulation.

A single-sensor two-detectors system is considered where the sensor communicates with both detectors and Detector 1 communicates with Detector 2, all over noise-free rate-limited links. The sensor and both detectors observe discrete memoryless source sequences whose joint probability mass function depends on a binary hypothesis. The goal at each detector is to guess the binary hypothesis in a way that, for increasing observation lengths, the probability of error under one of the hypotheses decays to zero with largest possible exponential decay, whereas the probability of error under the other hypothesis can decay to zero or to a small positive number arbitrarily slow. For the setting with positive communication rates from the sensor to the detectors and when both detectors are interested in maximizing the error exponent under the same hypothesis, we characterize the set of all possible exponents in a special case of testing against independence. In this case the cooperation link allows Detector 2 to increase its Type-II error exponent by an amount that is equal to the exponent attained at Detector 1. We also provide a general inner bound on the set of achievable error exponents that shows a tradeoff between the exponents at the two detectors in most cases. When the two detectors aim at maximizing the error exponent under different hypotheses and the distribution at the Sensor is different under the two hypotheses, then we show that such a tradeoff does not exist. We propose a general scheme that allows each detector to attain the same exponent as if it was the only detector in the system. For the setting with zero-rate communication on both links, we exactly characterize the set of possible exponents and the gain brought up by cooperation, in function of the number of bits that are sent over the two links. Notice that, for this setting, tradeoffs between the exponents achieved at the two detectors arise only in few particular cases. In all other cases, each detector achieves the same performance as if it were the only detector in the system.

A central challenge in hypothesis testing (HT) lies in determining the optimal balance between Type I (false positive) and Type II (non-detection or false negative) error probabilities. Analyzing these errors' exponential rate of convergence, known as error exponents, provides crucial insights into system performance. Error exponents offer a lens through which we can understand how operational restrictions, such as resource constraints and impairments in communications, affect the accuracy of distributed inference in networked systems. This survey presents a comprehensive review of key results in HT, from the foundational Stein's Lemma to recent advancements in distributed HT, all unified through the framework of error exponents. We explore asymptotic and non-asymptotic results, highlighting their implications for designing robust and efficient networked systems, such as event detection through lossy wireless sensor monitoring networks, collective perception-based object detection in vehicular environments, and clock synchronization in distributed environments, among others. We show that understanding the role of error exponents provides a valuable tool for optimizing decision-making and improving the reliability of networked systems.

In this work we show how an improved lower bound to the error exponent of the memoryless multiple-access (MAC) channel is attained via the use of linear codes, thus demonstrating that structure can be beneficial even in cases where there is no capacity gain. We show that if the MAC channel is modulo-additive, then any error probability, and hence any error exponent, achievable by a linear code for the corresponding single-user channel, is also achievable for the MAC channel. Specifically, for an alphabet of prime cardinality, where linear codes achieve the best known exponents in the single-user setting and the optimal exponent above the critical rate, this performance carries over to the MAC setting. At least at low rates, where expurgation is needed, our approach strictly improves performance over previous results, where expurgation was used at most for one of the users. Even when the MAC channel is not additive, it may be transformed into such a channel. While the transformation is lossy, we show that the distributed structure gain in some "nearly additive" cases outweighs the loss, and thus the error exponent can improve upon the best known error exponent for these cases as well. Finally we apply a similar approach to the Gaussian MAC channel. We obtain an improvement over the best known achievable exponent, given by Gallager, for certain rate pairs, using lattice codes which satisfy a nesting condition.

This paper studies clustering of data samples generated from composite distributions using the Kolmogorov-Smirnov (KS) based K-means algorithm. All data sequences are assumed to be generated from unknown continuous distributions. The maximum intra-cluster KS distance of each distribution cluster is assumed to be smaller than the minimum inter-cluster KS distance of different clusters. The analysis of convergence and upper bounds on the error probability are provided for both cases with known and unknown number of clusters. Furthermore, it is shown that the probability of error decays exponentially as the number of samples in each data sequence goes to infinity, and the error exponent is only a function of the difference of the inter-cluster and intra-cluster KS distances. The analysis is validated by simulation results.

This work studies error exponent limits in hypothesis testing (HT) in a distributed scenario with partial communication constraints. We derive general conditions on the Type I error restriction under which the error exponent of the optimal Type II error has a closed-form characterization for the task of testing against independence. We show that the error exponent is preserved for a family of decreasing Type I error restrictions. Complementing this analysis, new expressions are derived to bound the optimal Type II error probability for a finite number of observations. These bounds shed light about the velocity at which error exponent limits are attained with the number of samples.

This paper studies distributed binary test of statistical independence under communication (information bits) constraints. While testing independence is very relevant in various applications, distributed independence test is particularly useful for event detection in sensor networks where data correlation often occurs among observations of devices in the presence of a signal of interest. By focusing on the case of two devices because of their tractability, we begin by investigating conditions on Type I error probability restrictions under which the minimum Type II error admits an exponential behavior with the sample size. Then, we study the finite sample-size regime of this problem. We derive new upper and lower bounds for the gap between the minimum Type II error and its exponential approximation under different setups, including restrictions imposed on the vanishing Type I error probability. Our theoretical results shed light on the sample-size regimes at which approximations of the Type II error probability via error exponents became informative enough in the sense of predicting well the actual error probability. We finally discuss an application of our results where the gap is evaluated numerically, and we show that exponential approximations are not only tractable but also a valuable proxy for the Type II probability of error in the finite-length regime.

In this paper, we investigate the performance of the Viterbi decoding algorithm with/without Automatic Repeat reQuest (ARQ) over a Rician flat fading channel with unlimited interleaving. We show that the decay rate of the average bit error probability with respect to the bit energy to noise ratio is at least equal to d <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</sub> at high-bit energy to noise ratio for both cases (with ARQ and without ARQ), where d <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</sub> is the free distance of the convolutional code. The Yamamoto-Itoh flag helps to reduce the average bit error probability by a factor of 4(d) <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</sub> with a negligible retransmission rate. We also prove an interesting result that the average bit error probability decays exponentially fast with respect to the Rician factor for any fixed bit energy per noise ratio. In addition, the average bit error exponent with respect to the Rician factor is shown to be d <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</sub> .

In light of recent results by Verdú and Han on channel capacity, we examine three problems: the strong converse condition to the channel coding.theorem, the capacity of arbitrary channels with feedback and the Neyman‐Pearson hypothesis testing type‐II error exponent. It is first remarked that the strong converse condition holds if and only if the sequence of normalized channel information densities converges in probability to a constant. Examples illustrating this condition are provided. A general formula for the capacity of arbitrary channels with output feedback is then obtained. Finally, a general expression for the Neyman‐Pearson type‐II error exponent based on arbitrary observations subject to a constant bound on the type‐I error probability is derived.

We consider the error exponent of the memoryless multiple-access (MAC) channel. We show that if the MAC channel is modulo-additive, then any error probability, and hence any error exponent, achievable by a linear code for the corresponding single-user channel, is also achievable for the MAC channel. Specifically, for an alphabet of prime cardinality, where linear codes achieve the best known exponents in the single-user setting (and the optimal exponent above the critical rate), this performance carries over to the MAC setting. At least at low rates, where expurgation is needed, our approach strictly improves performance over previous results, where expurgation was used at most for one of the users. Even when the MAC channel is not additive, it may be transformed into such a channel. While the transformation is lossy, we show that the distributed structure gain in some “nearly additive” cases outweighs the loss, and thus we can improve upon the best known exponent for these cases as well. This approach is related to that previously proposed for the Gaussian MAC channel, and is based on “distributed structure”.

We provide finite sample guarantees for the classical Chow-Liu algorithm (IEEE Trans. Inform. Theory, 1968) to learn a tree-structured graphical model of a distribution. For a distribution P on Σn and a tree T on n nodes, we say T is an e-approximate tree for P if there is a T-structured distribution Q such that D(P || Q) is at most e more than the best possible tree-structured distribution for P. We show that if P itself is tree-structured, then the Chow-Liu algorithm with the plug-in estimator for mutual information with O(|Σ|3 ne−1) i.i.d. samples outputs an e-approximate tree for P with constant probability. In contrast, for a general P (which may not be tree-structured), Ω(n2e−2) samples are necessary to find an e-approximate tree. Our upper bound is based on a new conditional independence tester that addresses an open problem posed by Canonne, Diakonikolas, Kane, and Stewart (STOC, 2018): we prove that for three random variables X,Y,Z each over Σ, testing if I(X; Y ∣ Z) is 0 or ≥ e is possible with O(|Σ|3/e) samples. Finally, we show that for a specific tree T, with O(|Σ|2ne−1) samples from a distribution P over Σn, one can efficiently learn the closest T-structured distribution in KL divergence by applying the add-1 estimator at each node.

This paper looks at graphical models and discusses the quality of tree approximations by formulating the problem as a detection problem. One of the widely used algorithms for tree-structured approximation and modeling is the Chow-Liu algorithm. While this algorithm is optimal for Gaussian distributions in the sense of the Kullback-Leibler (KL) divergence, it is not optimal when compared with other information divergences and criteria such as Area Under the Curve (AUC). In this paper, we discuss the quality of tree approximation using the AUC. We define the correlation approximation matrix (CAM) and show that the KL divergence and AUC depend on the eigenvalues of the CAM. Examples show that the quality of tree approximations is in general not good for both information divergences and AUC when the number of nodes is large.

Given side information that an Ising tree-structured graphical model is homogeneous and has no external field, we derive the exact asymptotics of learning its structure from independently drawn samples. Our results, which leverage the use of probabilistic tools from the theory of strong large deviations, refine the large deviation (error exponents) results of Tan et al. (2011) and strictly improve those of Bresler and Karzand (2020). In addition, we extend our results to the scenario in which the samples are observed in random noise. In this case, we show that they strictly improve on the recent results of Nikolakakis et al. (2019). Our theoretical results demonstrate keen agreement with experimental results for sample sizes as small as that in the hundreds.

A generalized concept of interference suppression is introduced for multiuser systems with a known memoryless transmit non-linearity, such as in the downlink amplifier of a satellite communication system. By considering the operation of commonly used multiuser detection techniques from a viewpoint of constellation structure, we extend these notions to nonlinear multiuser systems. We also analyze the performance of such multiuser detectors in terms of their asymptotic multiuser error exponents, which reduce to the asymptotic multiuser effective energies for the respective detectors in the absence of nonlinearity. The asymptotic conditional error exponent is introduced as a quantitative measure for evaluating nonlinear multiuser detectors based on the conditional probability of error at high signal-to-noise ratio (SNR). The optimal affine detector that maximizes the asymptotic error exponent for a given user is derived along with a successive interference annulment (SIA) scheme that maximizes the asymptotic conditional error exponent at each stage of successive detection. We have shown that the optimal affine detector (and hence the SIA detector) is a generalization of existing linear (and successive interference canceller) detectors to include the possibility of nonlinearity and therefore, nonadditive multiuser interference. We introduce the concept of joint successive interference annulment (JSIA) for nonlinear systems, which uses the joint structural relationship between two or more users in an ordered set to achieve higher performance gains in terms of the asymptotic conditional error exponent. We generalize the joint successive interference annulment detector to include more than two users at each stage of JSIA detection to scale the tradeoff between detection complexity and performance gain. We also introduce the decorrelating detector for nonlinear systems which essentially employs a successive interference canceller with a decorrelation kernel in an equivalent but higher-order linear system. Finally, we analyze the optimal affine detector and hence, the family of proposed joint successive interference annulment detectors when the multiuser system is perturbed with noise inside the nonlinearity. Simulation results demonstrate that the proposed detectors based on successive interference annulment consistently outperform conventional multiuser detectors based on successive interference cancellation.