Representation of Quad Tree Decomposition and Various Segmentation Algorithms in Information Bottleneck Method

The rapid growth of encrypted network traffic poses significant challenges for network monitoring and security systems, as traditional methods often fail to accurately classify encrypted services. This paper proposes a novel approach to achieve robust classification of encrypted network services using Convolutional Neural Networks (CNNs) optimized by the Information Bottleneck (IB) method (IB-CNN). The IB method is employed to reduce the complexity of the model while retaining the most relevant information for classification, thereby improving its performance and generalization capabilities. We design and implement a CNN architecture tailored to the characteristics of encrypted network traffic, and apply the IB method to optimize the intermediate representations. Extensive experiments on a diverse dataset of encrypted network services demonstrate that our approach significantly outperforms existing methods in terms of accuracy and robustness, even in the presence of varying network conditions and adversarial attacks. Additionally, we analyze the impact of different IB parameters on the classification performance and provide insights into the optimal configuration for practical deployment. This study contributes to advancing the state-of-the-art in encrypted network service classification and offers a promising direction for enhancing the security and reliability of network monitoring systems.

It is well-known that the information bottleneck method and rate distortion theory are related. Here it is described how the information bottleneck can be considered as rate distortion theory for a family of probability measures where information divergence is used as distortion measure. It is shown that the information bottleneck method has some properties that are not shared with rate distortion theory based on any other divergence measure. In this sense the information bottleneck method is unique.

The compression step of the Cloud Radio Access Networks (C-RAN) is studied in this paper. In order to maximize the achievable sum rate in the uplink of C-RAN, constrained by the finite-capacity fronthaul links to the Central Processor (CP) in the cloud, an optimization algorithm of quantizers used for compression is proposed. In C-RAN, there are multiple Radio Units (RUs) compressing and forwarding the correlated received signals simultaneously. Due to the correlations, Wyner-Ziv coding is used. Hence a joint optimization of the quantizers is required at the CP. In this paper we extend the Information Bottleneck (IB) method for this joint optimization. It is a combination of the Alternating Information Bottleneck method (AIB) and the Alternating Bi-Section method, which are both proposed in this paper. Then we use the proposed algorithms to study some key features of C-RAN.

Semi-supervised cross-entropy clustering with information bottleneck constraint

The new 5G communications standard increases data rates and supports low-latency communication that places constraints on the computational complexity of channel decoders. 5G low-density parity-check (LDPC) codes have the so-called protograph-based raptor-like (PBRL) structure which offers inherent rate-compatibility and excellent performance. Practical LDPC decoder implementations use message-passing decoding with finite precision, which becomes coarse as complexity is more severely constrained. Performance degrades as the precision becomes more coarse. Recently, the information bottleneck (IB) method was used to design mutual-information-maximizing lookup tables that replace conventional finite-precision node computations. The IB approach exchanges messages represented by integers with very small bit width. This paper extends the IB principle to the flexible class of PBRL LDPC codes as standardized in 5G. The extensions include puncturing and rate-compatible IB decoder design. As an example of the new approach, a 4-bit information bottleneck decoder is evaluated for PBRL LDPC codes over a typical range of rates. Frame error rate simulations show that the proposed scheme outperforms offset min-sum decoding algorithms and operates very close to double-precision sum-product belief propagation decoding.

Recent works describe the design of discrete decoders for low-density parity-check codes by application of mutual information maximizing clustering algorithms in discrete density evolution. In the resulting discrete message passing decoders only integers are exchanged and node operations become simple lookup operations. Earlier works only describe discrete decoders for binary modulation schemes. This paper presents a new technique called message alignment which enables to design discrete decoders for higher-order modulation schemes. First, we design a channel output quantizer for quadrature amplitude modulation with the Information Bottleneck method. The quantizer attempts to preserve the relevant information on the modulation symbols. Afterwards, we illustrate that the assignment of several bits to one modulation symbol does not allow straightforward decoder design with the available discrete density evolution technique. The proposed message alignment solves this problem. The resulting discrete decoder is compared with state-of-the-art decoders using bit error rate simulations.

This paper synthesizes a recent line of work on automated predictive model making inspired by Rate-Distortion theory, in particular by the Information Bottleneck method. Predictive inference is interpreted as a strategy for efficient communication. The relationship to thermodynamic efficiency is discussed. The overall aim of this paper is to explain how this information theoretic approach provides an intuitive, overarching framework for predictive inference.

A network with two sources, one relay, and one destination is considered. Under the assumption of noisy source- relay links causing the relay to be unable to decode without error, we propose a quantizer design framework where the quantizer jointly compresses the soft information available for both sources at the relay. The quantizer design is based on the information bottleneck method using the notion of relevant information as an optimization criterion.

Seeking goals carried out by agents with a level of competency requires an “understanding” of the structure of their world. While abstract formal descriptions of a world structure in terms of geometric axioms can be formulated in principle, it is not likely that this is the representation that is actually employed by biological organisms or that should be used by biologically plausible models. Instead, we operate by the assumption that biological organisms are constrained in their information processing capacities, which in the past has led to a number of insightful hypotheses and models for biologically plausible behaviour generation. Here we use this approach to study various types of spatial categorizations that emerge through such informational constraints imposed on embodied agents. We will see that geometrically-rich spatial representations emerge when agents employ a trade-off between the minimisation of the Shannon information used to describe locations within the environment and the reduction of the location error generated by the resulting approximate spatial description. In addition, agents do not always need to construct these representations from the ground up, but they can obtain them by refining less precise spatial descriptions constructed previously. Importantly, we find that these can be optimal at both steps of refinement, as guaranteed by the successive refinement principle from information theory. Finally, clusters induced by these spatial representations via the information bottleneck method are able to reflect the environment’s topology without relying on an explicit geometric description of the environment’s structure. Our findings suggest that the fundamental geometric notions possessed by natural agents do not need to be part of their a priori knowledge but could emerge as a byproduct of the pressure to process information parsimoniously.

An image can be described in terms of appearance frequency of visual words. This representation is implemented in bag-of-visual-words (BoVW)-based loop closure detection for its efficiency and effectiveness. However, traditional BoVW-based approaches are strongly affected by false positive loops due to scene ambiguity caused by redundant words in the vocabulary and fail to detect bidirectional loops in monocular mode. Aiming at overcoming these problems, we propose a novel vocabulary construction algorithm named hierarchical sequential information bottleneck (HsIB) by leveraging the maximization of mutual information (MMI) mechanism. First, feature descriptors are extracted from training images for visual vocabulary construction. Second, HsIB extracts discriminative yet informative visual words through the MMI mechanism in vocabulary construction, which treats feature descriptors clustering as a process of data compression. Finally, the clustering process reaches a tradeoff between compactness and discrimination and improves the performance of traditional BoVW-based loop closure detection. The proposed method is compared with state-of-the-art methods on publicly available datasets. We also create a challenging dataset to further evaluate the performance of HsIB on bidirectional loops. To the best of our knowledge, we are the first to implement information bottleneck (IB) method in visual-SLAM (vSLAM) loop closure detection, and we obtain impressive results.

In this paper, we revisit the information bottleneck problem whose formulation and solution are of great importance in both information theory and statistical learning applications. We go into details as to why the problem was first introduced and how the algorithm proposed using Lagrangian method to solve such problems fell short of an exact solution. We then revisit the limitations of such Lagrangian methods, and propose to adopt a more systematic method, namely, Alternate Direction Method of Multipliers (ADMM) to develop a more efficient ADMM algorithm with randomized permutation orders to solve such problems. More importantly, we mathematically demonstrate how our suggested method outperforms the original Information Bottleneck (IB) method. At the end, we provide numerical results to demonstrate the notable advantages our algorithm attains as compared with the well-known IB approach in terms of both attained objective function values and the resulting constraints. We further inspect the concepts of accuracy and convergence and the trade-off between them in our method.

We consider the cooperative detection for distributed sensing via low Earth orbit constellations. The source data gathered by the ground sensors are non-orthogonally sent to satellite access points, and then aggregated at a central satellite via inter-satellite links (ISLs). To overcome the intrinsic ISL bandwidth limitation, we consider the deep auto-encoding paradigm to jointly design the ISL transceivers among satellites, and propose a novel deep variational information bottleneck (DVIB) method which maximizes the end-to-end sensing accuracy under bandwidth constraints. Specifically, the mathematically untractable ISL bandwidth constraint is first transformed into an entropy-based format. Then a customized batch-norm layer is introduced, where the messages on ISLs are considered as latent variables and are regularized with entropy-constrained posterior for efficient compression. Compared to the benchmark, the proposed DVIB method is shown to simultaneously reduce the bandwidth overhead by 30% and enhance the sensing accuracy by 2–5 dB, validating the significance of relevant information extraction on ISLs.

Multiview clustering (MVC) is a popular research topic in the fields of data mining and pattern recognition, which focuses on fully exploring and employing the correlations between views to jointly discover a consistent cluster structure across data. Typically, weighted MVC is a common clustering method aimed at learning the importance or weights of each view and applying them to explore the complementary information between views. However, current weighted MVCs primarily focus on the quality of each view while overlooking the crucial role of pseudo-label-based self-supervision in weight learning. In addition, most weighted MVCs only use a weighting mechanism to utilize complementary features without sufficiently considering the consistency relationship between the clustering results of individual views and the final clustering result. Aiming to solve the above problems, this article proposes a structure-enhanced self-supervised weighted information bottleneck (S2WIB) method for MVC. Specifically, the S2WIB method establishes a view-weight learning mechanism that leverages both the view-contained information and the self-supervised information to learn view weights, and then integrates the weighted information from different views. Meanwhile, based on the information bottleneck (IB) theory, it explores view correlations from two perspectives, namely complementary information and consistent view cluster structure information, thereby fully exploiting the potential information contained in multiview data. Experiments on various multiview text datasets, multifeature image datasets, multiangle video datasets, multimodal text-image datasets, as well as large-scale datasets and biological multiomics datasets, demonstrate that the S2WIB method performs effectively and exhibits superiority across various fields.

We develop an algorithm, the joint time-frequency segmentation algorithm, where the wavelet packet coefficients of the analyzed speech signal are represented as tiles of a time-frequency representation adapted to the characteristics of the signal itself. Further, our algorithm enables the decomposition of the speech signal into transient and non-transient components, respectively. Any block of wavelet packet coefficients, whose tiling height is larger than or equal to the tiling width belongs to the transient component and vice versa for the non-transient component. The transient component is selectively amplified and recombined with the original speech to generate the modified speech with energy adjusted to be equal to the original speech. The intelligibility of the original and modified speech is evaluated by 16 human listeners. Word recognition rate results show that the modified speech significantly improves speech intelligibility in background noise, i.e., by 10% absolute at 0 dB to 27% absolute at -30 dB.

Heart sound segmentation is the process of identifying the various events like S1, S2, S3, S4, and murmurs present in the phonocardiogram (PCG) signal. In this paper, a heart sound segmentation algorithm for segmenting S1 and S2 heart sound which corresponds to the systolic and diastolic activity of the heart is presented. The proposed method is based on variational mode decomposition (VMD) algorithm. The recorded PCG is decomposed into various modes using VMD. A novel criterion is proposed to select an appropriate mode which is based on center frequency and relative energy. Then the Shannon entropy envelope of the selected mode is computed. S1 and S2 heart sound is segmented by applying dynamic threshold on the Shannon entropy envelope. The segmentation algorithm is tested on both available standard database (Physionet, Pascal, Michigan, and eGeneralMedical) and recorded signal. The simulation results demonstrate that the proposed method achieved the sensitivity, positive predictivity, and the accuracy of above 95% with detection error rate of below 6%.