A Survey on Error Exponents in Distributed Hypothesis Testing: Connections with Information Theory, Interpretations, and Applications.

A single-sensor two-detectors system is considered where the sensor communicates with both detectors and Detector 1 communicates with Detector 2, all over noise-free rate-limited links. The sensor and both detectors observe discrete memoryless source sequences whose joint probability mass function depends on a binary hypothesis. The goal at each detector is to guess the binary hypothesis in a way that, for increasing observation lengths, the probability of error under one of the hypotheses decays to zero with largest possible exponential decay, whereas the probability of error under the other hypothesis can decay to zero or to a small positive number arbitrarily slow. For the setting with positive communication rates from the sensor to the detectors and when both detectors are interested in maximizing the error exponent under the same hypothesis, we characterize the set of all possible exponents in a special case of testing against independence. In this case the cooperation link allows Detector 2 to increase its Type-II error exponent by an amount that is equal to the exponent attained at Detector 1. We also provide a general inner bound on the set of achievable error exponents that shows a tradeoff between the exponents at the two detectors in most cases. When the two detectors aim at maximizing the error exponent under different hypotheses and the distribution at the Sensor is different under the two hypotheses, then we show that such a tradeoff does not exist. We propose a general scheme that allows each detector to attain the same exponent as if it was the only detector in the system. For the setting with zero-rate communication on both links, we exactly characterize the set of possible exponents and the gain brought up by cooperation, in function of the number of bits that are sent over the two links. Notice that, for this setting, tradeoffs between the exponents achieved at the two detectors arise only in few particular cases. In all other cases, each detector achieves the same performance as if it were the only detector in the system.

This paper studies binary hypothesis testing with a single sensor that communicates with two decision centers over a memoryless broadcast channel. The main focus lies on the tradeoff between the two type-II error exponents achievable at the two decision centers. In our proposed scheme, we can partially mitigate this tradeoff when the transmitter has a probability larger than 1/2 to distinguish the alternate hypotheses at the decision centers, i.e., the hypotheses under which the decision centers wish to maximize their error exponents. In the cases where these hypotheses cannot be distinguished at the transmitter (because both decision centers have the same alternative hypothesis or because the transmitter’s observations have the same marginal distribution under both hypotheses), our scheme shows an important tradeoff between the two exponents. The results in this paper thus reinforce the previous conclusions drawn for a setup where communication is over a common noiseless link. Compared to such a noiseless scenario, here, however, we observe that even when the transmitter can distinguish the two hypotheses, a small exponent tradeoff can persist, simply because the noise in the channel prevents the transmitter to perfectly describe its guess of the hypothesis to the two decision centers.

Information theory was developed to determine the fundamental limits on the performance of communication systems. Detection theory involves an application of statistical decision theory to the problem of determining the presence or absence of signals in noise. Both of these theories deal with the communication problem. The relationship between information theory and conventional (centralized) detection theory has been discussed in the literature. Kullback [Ku159] discussed the use of discrimination to study hypothesis testing problems. Middleton [Mid60] employed cost functions based on information theory for optimizing signal detection systems. Csiszar et al. [CsL71] and Blahut [Bla74] have formulated the detection problem as a coding problem and have carried out an asymptotic analysis of detection systems based on the error exponent function. In this chapter, we briefly present some extensions of the above work for the distributed detection problem. In Section 7.2, we discuss the design of distributed detection systems based on an information theoretic cost function. In Section 7.3, we present a brief summary of some asymptototic results on the performance of distributed detection systems.

A distributed binary hypothesis testing (HT) problem involving two parties, one referred to as the observer and the other as the detector is studied. The observer observes a discrete memoryless source (DMS) and communicates its observations to the detector over a discrete memoryless channel (DMC). The detector observes another DMS correlated with that at the observer, and performs a binary hypothesis test on the joint distribution of the two DMS's using its own observed data and the information received from the observer. The trade-off between the type I error probability and the type II error-exponent of the HT is explored. Single-letter lower bounds on the optimal type II error-exponent are obtained by using two different coding schemes, a separate HT and channel coding scheme and a joint HT and channel coding scheme based on hybrid coding for the matched bandwidth case. Exact single-letter characterization of the same is established for the special case of testing against conditional independence, and it is shown to be achieved by the separate HT and channel coding scheme. An example is provided where the joint scheme achieves a strictly better performance than the separation based scheme.

The classical result of Blahut, which characterizes achievable error exponents in binary hypothesis testing, is generalized in two different directions. First, in M-ary hypothesis testing, the tradeoff of all M(M-1) types of error exponents and corresponding optimal decision schemes are explored. Then, motivated by a power-constrained distributed detection scenario, binary hypothesis testing is revisited, and the tradeoff of power consumption versus error exponents is fully characterized. In the latter scenario, sensors are allowed to make random decisions as to whether they should remain silent and save power, or transmit and improve detection quality. It is then shown by an example that optimal sensor decisions may indeed be random

In light of recent results by Verdú and Han on channel capacity, we examine three problems: the strong converse condition to the channel coding.theorem, the capacity of arbitrary channels with feedback and the Neyman‐Pearson hypothesis testing type‐II error exponent. It is first remarked that the strong converse condition holds if and only if the sequence of normalized channel information densities converges in probability to a constant. Examples illustrating this condition are provided. A general formula for the capacity of arbitrary channels with output feedback is then obtained. Finally, a general expression for the Neyman‐Pearson type‐II error exponent based on arbitrary observations subject to a constant bound on the type‐I error probability is derived.

Suppose Q is a family of discrete memoryless channels. An unknown member of Q will be available, with perfect, causal output feedback for communication. Is there a coding scheme (possibly with variable transmission time) that can achieve the Burnashev error exponent uniformly over Q ? For two families of channels we show that the answer is yes. Furthermore, for each of these two classes, in addition to achieve the maximum error exponent, it is possible to uniformly attain any given fraction of the channel capacity. Therefore, in terms of achievable rates and delay, there are situations in which the knowledge of the channel becomes irrelevant. In the second part of the thesis, we show that for arbitrary sets of channels the Burnashev error exponent cannot in general be uniformly achieved. In particular we give a sufficient condition for a pair of channels so that no coding strategy reaches Burnashev's exponent simultaneously on both channels. As a third part we study a scenario where communication is carried by first testing the channel by means of a training sequence, then coding according to the channel estimate. We provide an upper bound on the maximum achievable error exponent of such coding schemes. This bound is typically much lower than the maximum achievable error exponent over a channel with feedback. For example in the case of binary symmetric channels this bound has a slope that vanishes at capacity. This result suggests that in terms of error exponent, a good universal feedback scheme combines channel estimation with information delivery, rather than separating them. In the final chapter, we address the question of communicating quickly and reliably. We consider a simple situation of two message communication over a known channel with feedback. We propose a simple decoding rule, and show that it minimizes a weighted combination of the probability of error and decoding delay for a certain range of crossover probabilities and combination weights.

Along with the channel capacity, the error exponent is one of the most important information-theoretic measures of reliability, as it sets ultimate bounds on the performance of communication systems employing codes of finite complexity. In this paper, we derive an exact analytical expression for the random coding error exponent, which provides significant insight regarding the ultimate limits to communications through Nakagami-m fading channels. An important fact about this error exponent is that it determines the behavior of error probability in terms of the transmission rate and the code length that reflects the coding complexity required to achieve a certain level of reliability. Moreover, from the derived analytical expression, we can easily compute the necessary codeword length without extensive Monte-Carlo simulation to achieve a predefined upper bound for error probability at a rate below the channel capacity. We also improve the random coding bound by expurgating bad codewords from the code ensemble, since random coding error exponent is determined by selecting codewords independently according to the input distribution where good and bad codewords contribute equally to the overall average error probability. Finally, we derive exact analytical expressions for the cutoff rate, critical rate, and expurgation rate and verify the analytical expressions via Monte-Carlo simulation.

In this work we show how an improved lower bound to the error exponent of the memoryless multiple-access (MAC) channel is attained via the use of linear codes, thus demonstrating that structure can be beneficial even in cases where there is no capacity gain. We show that if the MAC channel is modulo-additive, then any error probability, and hence any error exponent, achievable by a linear code for the corresponding single-user channel, is also achievable for the MAC channel. Specifically, for an alphabet of prime cardinality, where linear codes achieve the best known exponents in the single-user setting and the optimal exponent above the critical rate, this performance carries over to the MAC setting. At least at low rates, where expurgation is needed, our approach strictly improves performance over previous results, where expurgation was used at most for one of the users. Even when the MAC channel is not additive, it may be transformed into such a channel. While the transformation is lossy, we show that the distributed structure gain in some "nearly additive" cases outweighs the loss, and thus the error exponent can improve upon the best known error exponent for these cases as well. Finally we apply a similar approach to the Gaussian MAC channel. We obtain an improvement over the best known achievable exponent, given by Gallager, for certain rate pairs, using lattice codes which satisfy a nesting condition.

This work studies error exponent limits in hypothesis testing (HT) in a distributed scenario with partial communication constraints. We derive general conditions on the Type I error restriction under which the error exponent of the optimal Type II error has a closed-form characterization for the task of testing against independence. We show that the error exponent is preserved for a family of decreasing Type I error restrictions. Complementing this analysis, new expressions are derived to bound the optimal Type II error probability for a finite number of observations. These bounds shed light about the velocity at which error exponent limits are attained with the number of samples.

In this paper, we study upper and lower bounds on the error exponents for joint source-channel coding with decoder side-information. The results in the paper are nontrivial extensions of Csiszar's classical paper “Joint Source-Channel Error Exponent”, Problems of Control and Information Theory, 1980. Unlike the joint source-channel coding result in Csiszar's paper, it is not obvious whether the lower bound and the upper bound are equivalent even if the channel coding error exponent is known. For a class of channels, including symmetric channels, we apply a game-theoretic result to establish the existence of a saddle point and, hence, prove that the lower and upper bounds are the same if the channel coding error exponent is known. More interestingly, we show that encoder side-information does not increase the error exponents in this case.

This paper studies distributed binary test of statistical independence under communication (information bits) constraints. While testing independence is very relevant in various applications, distributed independence test is particularly useful for event detection in sensor networks where data correlation often occurs among observations of devices in the presence of a signal of interest. By focusing on the case of two devices because of their tractability, we begin by investigating conditions on Type I error probability restrictions under which the minimum Type II error admits an exponential behavior with the sample size. Then, we study the finite sample-size regime of this problem. We derive new upper and lower bounds for the gap between the minimum Type II error and its exponential approximation under different setups, including restrictions imposed on the vanishing Type I error probability. Our theoretical results shed light on the sample-size regimes at which approximations of the Type II error probability via error exponents became informative enough in the sense of predicting well the actual error probability. We finally discuss an application of our results where the gap is evaluated numerically, and we show that exponential approximations are not only tractable but also a valuable proxy for the Type II probability of error in the finite-length regime.

The problem of multiple hypothesis testing (HT) for arbitrarily varying sources (AVS) is considered. The achievable error probability exponents (reliabilities) region is derived, optimal decision schemes are described. The result extends the known ones by Fu and Shen and by Tuncel. The Chernoff bounds for AVS binary and M-ary HT are specified via indication of a Sanov theorem for those sources.

The Chow-Liu algorithm has been a mainstay for the learning of tree-structured graphical models from i.i.d. sampled data vectors. Its theoretical properties have been well-studied and are well-understood. In this paper, we focus on the class of trees that are arguably even more fundamental, namely homogeneous trees in which each pair of nodes that forms an edge has the same correlation ρ. We ask whether we are able to further reduce the error probability of learning the structure of the homogeneous tree model when active learning is allowed. Our figure of merit is the error exponent, which quantifies the exponential rate of decay of the error probability with an increasing number of data samples. We design and analyze an algorithm Active Learning Algorithm for Trees with Homogeneous Edges (ACTIVE-LATHE), which surprisingly boosts (increases) the error exponent. Our analysis hinges on judiciously exploiting the minute but detectable statistical variation of the samples to allocate more data to parts of the graph in which we are less confident of being correct.

A distributed hypothesis testing (HT) problem is considered, comprising two nodes and a unidirectional communication link. The receiving node is required to make a decision as to the probability distribution in effect. A binning process is used in order to minimize the probability of error, resulting in a new achievable error-exponent. A sub-class of HT problems with general hypotheses is defined, which contains many interesting and relevant problems. The advantage of the binning strategy in comparison to the non-binning approach is demonstrated by means of a binary symmetric example.