Feedback communication over unknown channels

Communication over unknown discrete memoryless channels with instantaneous and perfect feedback is considered. For a given set of channels we define a notion of optimal coding schemes in terms of achievable rate and error exponent, and prove the existence of such coding schemes for two families of channels

Along with the channel capacity, the error exponent is one of the most important information-theoretic measures of reliability, as it sets ultimate bounds on the performance of communication systems employing codes of finite complexity. In this paper, we derive an exact analytical expression for the random coding error exponent, which provides significant insight regarding the ultimate limits to communications through Nakagami-m fading channels. An important fact about this error exponent is that it determines the behavior of error probability in terms of the transmission rate and the code length that reflects the coding complexity required to achieve a certain level of reliability. Moreover, from the derived analytical expression, we can easily compute the necessary codeword length without extensive Monte-Carlo simulation to achieve a predefined upper bound for error probability at a rate below the channel capacity. We also improve the random coding bound by expurgating bad codewords from the code ensemble, since random coding error exponent is determined by selecting codewords independently according to the input distribution where good and bad codewords contribute equally to the overall average error probability. Finally, we derive exact analytical expressions for the cutoff rate, critical rate, and expurgation rate and verify the analytical expressions via Monte-Carlo simulation.

Universally achievable error exponents pertaining to certain families of channels (most notably, discrete memoryless channels (DMCs) and various ensembles of random codes, are studied by combining the competitive minimax approach, proposed by Feder and Merhav, with Chernoff bound and Gallager's techniques for the analysis of error exponents. In particular, we derive a single-letter expression for the largest, universally achievable fraction xi of the optimum error exponent pertaining to the optimum maximum-likelihood (ML) decoding. Moreover, a simpler single-letter expression for a lower bound to xi is presented. To demonstrate the tightness of this lower bound, we use it to show that xi = 1, for the binary symmetric channel (BSC), when the random coding distribution is uniform over: i) all codes (of a given rate), and ii) all linear codes, in agreement with well-known results. We also show that xi = 1 for the uniform ensemble of systematic linear codes, and for that of time-varying convolutional codes in the bit-error-rate sense. For the latter case, we also derive the corresponding universal decoder explicitly and show how it can be efficiently implemented using a slightly modified version of the Viterbi algorithm which employs two trellises.

We investigate a nonparametric hypothesis testing problem, in which we assume a testing data stream is generated by one of a set of distributions (hypotheses), and the goal is to test which one of the multiple distributions generates the testing data stream, i.e., which hypothesis occurs. We assume that some distributions in the set are unknown with only training sequences generated by the corresponding distributions are given. We construct the generalized likelihood (GL) test, and characterize the error exponent of the maximum error probability. We show that the error exponent is captured by the Chernoff distance between each pair of distributions as well as the KL divergence between the approximated distributions (via training sequences) and the true distributions. We also show that the ratio between the lengths of training and testing sequences plays an important role in determining the error decay behavior.

Capacity formulas and random-coding and sphere-packing exponents are derived for a generalized family of Gel'fand-Pinsker coding problems. Information is to be reliably transmitted through a noisy channel with random state sequence. Partial information about the state sequence is available to the encoder and decoder. Two families of channels are considered: (1) compound discrete memoryless channels (C-DMC); and (2) channels with arbitrary memory, subject to an additive cost constraint, or more generally to a constraint on the conditional type of the channel output given the input. Both problems are closely connected. For the C-DMC case, our random-coding and sphere-packing exponents coincide at high rates, thereby determining the reliability function of the channel family. The random-coding exponent is achieved using a 3D binning scheme and a maximum penalized mutual information decoder. In the case of arbitrary channels, with memory, a larger random-coding error exponent than in the C-DMC case is obtained. Applications of this study include watermarking, data hiding, communication in presence of partially known interferers, and problems such as broadcast channels, all of which involve the fundamental idea of binning.

In light of recent results by Verdú and Han on channel capacity, we examine three problems: the strong converse condition to the channel coding.theorem, the capacity of arbitrary channels with feedback and the Neyman‐Pearson hypothesis testing type‐II error exponent. It is first remarked that the strong converse condition holds if and only if the sequence of normalized channel information densities converges in probability to a constant. Examples illustrating this condition are provided. A general formula for the capacity of arbitrary channels with output feedback is then obtained. Finally, a general expression for the Neyman‐Pearson type‐II error exponent based on arbitrary observations subject to a constant bound on the type‐I error probability is derived.

A single-sensor two-detectors system is considered where the sensor communicates with both detectors and Detector 1 communicates with Detector 2, all over noise-free rate-limited links. The sensor and both detectors observe discrete memoryless source sequences whose joint probability mass function depends on a binary hypothesis. The goal at each detector is to guess the binary hypothesis in a way that, for increasing observation lengths, the probability of error under one of the hypotheses decays to zero with largest possible exponential decay, whereas the probability of error under the other hypothesis can decay to zero or to a small positive number arbitrarily slow. For the setting with positive communication rates from the sensor to the detectors and when both detectors are interested in maximizing the error exponent under the same hypothesis, we characterize the set of all possible exponents in a special case of testing against independence. In this case the cooperation link allows Detector 2 to increase its Type-II error exponent by an amount that is equal to the exponent attained at Detector 1. We also provide a general inner bound on the set of achievable error exponents that shows a tradeoff between the exponents at the two detectors in most cases. When the two detectors aim at maximizing the error exponent under different hypotheses and the distribution at the Sensor is different under the two hypotheses, then we show that such a tradeoff does not exist. We propose a general scheme that allows each detector to attain the same exponent as if it was the only detector in the system. For the setting with zero-rate communication on both links, we exactly characterize the set of possible exponents and the gain brought up by cooperation, in function of the number of bits that are sent over the two links. Notice that, for this setting, tradeoffs between the exponents achieved at the two detectors arise only in few particular cases. In all other cases, each detector achieves the same performance as if it were the only detector in the system.

A central challenge in hypothesis testing (HT) lies in determining the optimal balance between Type I (false positive) and Type II (non-detection or false negative) error probabilities. Analyzing these errors' exponential rate of convergence, known as error exponents, provides crucial insights into system performance. Error exponents offer a lens through which we can understand how operational restrictions, such as resource constraints and impairments in communications, affect the accuracy of distributed inference in networked systems. This survey presents a comprehensive review of key results in HT, from the foundational Stein's Lemma to recent advancements in distributed HT, all unified through the framework of error exponents. We explore asymptotic and non-asymptotic results, highlighting their implications for designing robust and efficient networked systems, such as event detection through lossy wireless sensor monitoring networks, collective perception-based object detection in vehicular environments, and clock synchronization in distributed environments, among others. We show that understanding the role of error exponents provides a valuable tool for optimizing decision-making and improving the reliability of networked systems.

In this work we show how an improved lower bound to the error exponent of the memoryless multiple-access (MAC) channel is attained via the use of linear codes, thus demonstrating that structure can be beneficial even in cases where there is no capacity gain. We show that if the MAC channel is modulo-additive, then any error probability, and hence any error exponent, achievable by a linear code for the corresponding single-user channel, is also achievable for the MAC channel. Specifically, for an alphabet of prime cardinality, where linear codes achieve the best known exponents in the single-user setting and the optimal exponent above the critical rate, this performance carries over to the MAC setting. At least at low rates, where expurgation is needed, our approach strictly improves performance over previous results, where expurgation was used at most for one of the users. Even when the MAC channel is not additive, it may be transformed into such a channel. While the transformation is lossy, we show that the distributed structure gain in some "nearly additive" cases outweighs the loss, and thus the error exponent can improve upon the best known error exponent for these cases as well. Finally we apply a similar approach to the Gaussian MAC channel. We obtain an improvement over the best known achievable exponent, given by Gallager, for certain rate pairs, using lattice codes which satisfy a nesting condition.

We show how to exploit a noisy feedback link to implement high-reliability communication. We specify a variable-length coding strategy that achieves the error exponent (in delay) of erasure decoding using any noisy feedback channel which has a positive zero-rate random coding error exponent. Building on this result, we give a second approach that, depending only on the capacity of the feedback link, achieves an error exponent up to half of the Burnashev exponent - the maximum exponent that can be achieved with a noiseless feedback link. The resulting exponent can be far larger than the exponent of erasure decoding, particularly at rates close to capacity

In this paper, we consider a distributed detection scenario where a number of remote sensors is linked to a decision fusion center by a fading multiaccess channel. The communication is assumed to be noncoherent meaning that channel gains are unknown at both sensors and the fusion center. Each sensor makes a binary local decision and communicates it to the fusion center simultaneously. We investigate the detection performance of the system in terms of error probability and error exponent under both Rayleigh and Rician fading scenarios. We reveal that on-off keying is the most energy efficient modulation scheme when the channel is subject to Rayleigh fading and that optimizing the modulation scheme can lead to a gain in error exponent under Rician fading scenario. Under both fading scenarios, optimal decision fusion rules can be reduced to simple threshold tests.

The Chow-Liu algorithm has been a mainstay for the learning of tree-structured graphical models from i.i.d. sampled data vectors. Its theoretical properties have been well-studied and are well-understood. In this paper, we focus on the class of trees that are arguably even more fundamental, namely homogeneous trees in which each pair of nodes that forms an edge has the same correlation ρ. We ask whether we are able to further reduce the error probability of learning the structure of the homogeneous tree model when active learning is allowed. Our figure of merit is the error exponent, which quantifies the exponential rate of decay of the error probability with an increasing number of data samples. We design and analyze an algorithm Active Learning Algorithm for Trees with Homogeneous Edges (ACTIVE-LATHE), which surprisingly boosts (increases) the error exponent. Our analysis hinges on judiciously exploiting the minute but detectable statistical variation of the samples to allocate more data to parts of the graph in which we are less confident of being correct.

We consider multiple repetition strategies with fixed delay decoding for discrete memoryless channels with noiseless feedback. Existing binary schemes by Schalkwijk and Zigangirov are analyzed and their results are extended. The general error exponents are computed and presented by elegant expressions in the strictly symmetric case. An important class of precoded sequences, so-called flip sequences, is found and their degrading effect on the error exponent is investigated. This effect is shown negligible when the repetition parameters are chosen such that the transmission rate is maximized. Even when signalling at channel capacity, the error exponent is shown to be strictly positive.

This paper studies distributed binary test of statistical independence under communication (information bits) constraints. While testing independence is very relevant in various applications, distributed independence test is particularly useful for event detection in sensor networks where data correlation often occurs among observations of devices in the presence of a signal of interest. By focusing on the case of two devices because of their tractability, we begin by investigating conditions on Type I error probability restrictions under which the minimum Type II error admits an exponential behavior with the sample size. Then, we study the finite sample-size regime of this problem. We derive new upper and lower bounds for the gap between the minimum Type II error and its exponential approximation under different setups, including restrictions imposed on the vanishing Type I error probability. Our theoretical results shed light on the sample-size regimes at which approximations of the Type II error probability via error exponents became informative enough in the sense of predicting well the actual error probability. We finally discuss an application of our results where the gap is evaluated numerically, and we show that exponential approximations are not only tractable but also a valuable proxy for the Type II probability of error in the finite-length regime.

In this paper, we investigate the performance of the Viterbi decoding algorithm with/without Automatic Repeat reQuest (ARQ) over a Rician flat fading channel with unlimited interleaving. We show that the decay rate of the average bit error probability with respect to the bit energy to noise ratio is at least equal to d <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</sub> at high-bit energy to noise ratio for both cases (with ARQ and without ARQ), where d <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</sub> is the free distance of the convolutional code. The Yamamoto-Itoh flag helps to reduce the average bit error probability by a factor of 4(d) <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</sub> with a negligible retransmission rate. We also prove an interesting result that the average bit error probability decays exponentially fast with respect to the Rician factor for any fixed bit energy per noise ratio. In addition, the average bit error exponent with respect to the Rician factor is shown to be d <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</sub> .