MAJSCC: Mamba-based adaptive joint source channel coding for wireless image transmission

As one of the key techniques to realize semantic communications, end-to-end optimized neural joint source-channel coding (JSCC) has made great progress over the past few years. A general trend in many recent works pushing the model adaptability or the application diversity of neural JSCC is based on the convolutional neural network (CNN) backbone, whose model capacity is yet limited, inherently leading to inferior system coding gain against traditional coded transmission systems. In this paper, we establish a new neural JSCC backbone that can also adapt flexibly to diverse channel conditions and transmission rates within a single model, our open-source project aims to promote the research in this field. Specifically, we show that with elaborate design, neural JSCC codec built on the emerging Swin Transformer backbone achieves superior performance than conventional neural JSCC codecs built upon CNN, while also requiring lower end-to-end processing latency. Paired with two spatial modulation modules that scale latent representations based on the channel state information and target transmission rate, our baseline SwinJSCC can further upgrade to a versatile version, which increases its capability to adapt to diverse channel conditions and rate configurations. Extensive experimental results show that our SwinJSCC achieves better or comparable performance versus the state-of-the-art engineered BPG + 5G LDPC coded transmission system with much faster end-to-end coding speed, especially for high-resolution images, in which case traditional CNN-based JSCC yet falls behind due to its limited model capacity.

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> In this paper, we present a cross-layer approach for video transmission in wireless LANs that employs joint source and application-layer channel coding, together with rate adaptation at the wireless physical layer (PHY). While the purpose of adopting PHY rate adaptation in modern wireless LANs like the IEEE 802.11a/b is to maximize the throughput, in this paper we exploit this feature to increase the robustness of wireless video. More specifically, we investigate the impact of adapting the PHY transmission rate, thus changing the throughput and packet loss channel characteristics, on the rate-distortion performance of a transmitted video sequence. To evaluate the video quality at the decoder, we develop a cross-layer modeling framework that considers jointly the effect of application-layer joint source-channel coding (JSCC), error concealment, and the PHY transmission rate. The resulting models are used by an optimization algorithm that calculates the optimal JSCC allocation for each video frame, and PHY transmission rate for each outgoing transport packet. The comprehensive simulation results obtained with the H.264/AVC codec demonstrate considerable increase in the PSNR of the decoded video when compared with a system that employs separately JSCC and PHY rate adaptation. Furthermore, our performance analysis indicates that the optimal PHY transmission rate calculated by the proposed algorithm, can be significantly different when compared with rate adaptation algorithms that target throughput improvement. </para>

In this paper, we address the problem of QoS (Quality of Service) provisioning for multi-user video streaming over multiple heterogeneous wireless networks based on the distributed, cross-layer design framework. By jointly considering the rate allocation and the Joint Source-Channel Coding (JSCC), our proposal aims at maximizing the QoS provisioning under the given resource constraint. At first, we develop and evaluate a framework for optimal video rate allocation over multiple networks based on the observed Available Bit Rate (ABR) and the Round Trip Time (RTT) over each access network, as well as the video rate-distortion characteristics. The rate allocation is formulated as a convex optimization problem that minimizes the sum of all video streams expected distortion. Then, we propose an analytical JSCC scheme for error-resilient scalable encoded video, and integrate the JSCC with the specific rate allocation algorithm to improve the constructed video quality by optimally applying the appropriate channel coding rate given the constraints imposed by the transmission rate and the prevailing channel conditions. Objective and subjective simulation results are provided which demonstrate the effectiveness of our proposed joint scheme.

Recent research on joint source channel coding (JSCC) for wireless communications has achieved great success owing to the employment of deep learning (DL). However, the existing work on DL based JSCC usually trains the designed network to operate under a specific signal-to-noise ratio (SNR) regime, without taking into account that the SNR level during the deployment stage may differ from that during the training stage. A number of networks are required to cover the scenario with a broad range of SNRs, which is computational inefficiency (in the training stage) and requires large storage. To overcome these drawbacks our paper proposes a novel method called Attention DL based JSCC (ADJSCC) that can successfully operate with different SNR levels during transmission. This design is inspired by the resource assignment strategy in traditional JSCC, which dynamically adjusts the compression ratio in source coding and the channel coding rate according to the channel SNR. This is achieved by resorting to attention mechanisms because these are able to allocate computing resources to more critical tasks. Instead of applying the resource allocation strategy in traditional JSCC, the ADJSCC uses the channel-wise soft attention to scaling features according to SNR conditions. We compare the ADJSCC method with the state-of-the-art DL based JSCC method through extensive experiments to demonstrate its adaptability, robustness and versatility. Compared with the existing methods, the proposed method takes less storage and is more robust in the presence of channel mismatch.

We propose an adaptive lossy joint source-channel coding (JSCC) scheme for sending correlated sources over two-terminal discrete-memoryless two-way channels (DM-TWCs). The main idea is to couple the independent operations of the terminals via an adaptive coding mechanism, which can mitigate cross-interference resulting from simultaneous channel transmissions and concurrently exploit the sources' correlation to reduce the end-to-end reconstruction distortions. Our adaptive JSCC scheme not only subsumes existing lossy coding methods for two-way simultaneous communication but also improves their performance. Furthermore, we derive outer bounds for our two-way lossy transmission problem and establish complete JSCC theorems in some special settings. In these special cases, a non-adaptive separate source-channel coding (SSCC) scheme achieves the optimal performance, thus simplifying the design of the source-channel communication system.

We investigate joint source channel coding (JSCC) for wireless image transmission over multipath fading channels. Inspired by recent works on deep learning based JSCC and model-based learning methods, we combine an autoencoder with orthogonal frequency division multiplexing (OFDM) to cope with multipath fading. The proposed encoder and decoder use convolutional neural networks (CNNs) and directly map the source images to complex-valued baseband samples for OFDM transmission. The multipath channel and OFDM are represented by non-trainable (deterministic) but differentiable layers so that the system can be trained end-to-end. Furthermore, our JSCC decoder further incorporates explicit channel estimation, equalization, and additional subnets to enhance the performance. The proposed method exhibits 2.5 – 4 dB SNR gain for the equivalent image quality compared to conventional schemes that employ state-of-the-art but separate source and channel coding such as Better Portable Graphics (BPG) and Low-Density Parity-Check (LDPC) schemes. The performance further improves when the system incorporates the channel state information (CSI) feedback. The proposed scheme is robust against OFDM signal clipping and parameter mismatch for the channel model used in training and evaluation.

The architecture of the latest Graphic Processing Unit (GPU) has surpassed the previous application-specific stream architecture. This has led to an architecture consisting of a number of uniform programmable units integrated on the same chip which facilitate the general-purpose computing beyond the graphic processing. With the multiple programmable units executing in parallel, the latest GPU shows superior performance. Furthermore, programmers can have a direct control on the GPU pipeline using easy-to-use parallel programming environments, whereas they had to rely on specific graphics API's in the past. These advances in hardware and software make General-Purpose GPU (GPGPU) computing widespread. In this paper, using the latest GPU and its software environment, we parallelize a computationally demanding financial application and optimize its performance. We also analyze the performance results compared with those obtained using CPU only. Experimental results show that GPU can achieve a superior performance, greater than 190x, compared with the CPU-only case.

Layered Wyner–Ziv video coding for transmission over unreliable channels

Developing channel-adaptive deep joint source-channel coding (JSCC) systems is a critical challenge in wireless image transmission. While recent advancements have been made, most existing approaches are designed for static channel environments, limiting their ability to capture the dynamics of channel environments. As a result, their performance may degrade significantly in practical systems. In this paper, we consider time-varying block fading channels, where the transmission of a single image can experience multiple fading events. We propose a novel coarse-to-fine channel-adaptive JSCC framework (CFA-JSCC) that is designed to handle both significant fluctuations and rapid changes in wireless channels. Specifically, in the coarse-grained phase, CFA-JSCC utilizes the average signal-to-noise ratio (SNR) to adjust the encoding strategy, providing a preliminary adaptation to the prevailing channel conditions. Subsequently, in the fine-grained phase, CFA-JSCC leverages instantaneous SNR to dynamically refine the encoding strategy. This refinement is achieved by re-encoding the remaining channel symbols whenever the channel conditions change. Additionally, to reduce the overhead for SNR feedback, we utilize a limited set of channel quality indicators (CQIs) to represent the channel SNR and further propose a reinforcement learning (RL)-based CQI selection strategy to learn this mapping. This strategy incorporates a novel reward shaping scheme that provides intermediate rewards to facilitate the training process. Experimental results demonstrate that our CFA-JSCC provides enhanced flexibility in capturing channel variations and improved robustness in time-varying channel environments.

Deep learning based joint source-channel coding (DJSCC) has recently made significant progress and emerged as a potential solution for future wireless communication. However, there are still several crucial issues that necessitate further in-depth exploration to enhance the efficiency of DJSCC, such as channel quality adaptability, bandwidth adaptability, and the delicate balance between efficiency and complexity. This work proposes an adaptive deep joint source-channel coding scheme tailored for one-to-many wireless image transmission scenarios (ADMIT). First, to effectively improve transmission performance, neighboring attention is introduced as the backbone for the proposed ADMIT method. Second, a channel quality adaptive module (CQAM) is designed based on multi-scale feature fusion, which seamlessly adapts to fluctuating channel conditions across a wide range of channel signal-to-noise ratios (CSNRs). Third, to be precisely tailored to different bandwidth resources, the channel gained adaptive module (CGAM) dynamically adjusts the significance of individual channels within the latent space, which ensures seamless varying bandwidth accommodation with a single model through bandwidth adaptation and symbol completion. Additionally, to mitigate the imbalance of loss across multiple bandwidth ratios during the training process, the gradient normalization (GradNorm) based training strategy is leveraged to ensure adaptive balancing of loss decreasing. The extensive experimental results demonstrate that the proposed method significantly enhances transmission performance while maintaining relatively low computational complexity. The source codes are available at: https://github.com/llsurreal919/ADMIT.

Purpose: To develop an adaptive radiotherapy (ART) strategy based on in vivo dose verification using volumetric imaging, real‐time physiologically constrained deformable image registration (PC‐DIR) and high performance computation (HPC). Method and Materials: Comparison of delivered and planned dose distributions for patients undergoing radiation therapy, especially IMRT, can be used to estimate treatment outcome, and develop a successful ART strategy that accounts for patient positioning errors, organ deformation, physiological changes (i.e. tumor shrinkage, edema), and incorporating this back into the treatment planning. Using 4DCT imaging, we have developed a ART methodology based on 1) PC‐DIR for auto‐contouring; 2) automated internal target volume generation; 3) probability density function (PDF) for specific structures; and 4) graphics processor unit (GPU) based HPC for real‐time computation using NVIDIA 8800GTX. Results: Compared with the same algorithm without the PC constraint, PC‐DIR resulted in superior performance with up to 91% reduction in inverse consistency errors. The computer generated ITV yielded contours typically within 3mm of physician drawn contours, and also served as a check of physician drawn ITVs. The PDF was constructed for the ITV to determine the dose distribution based on spatial probability using time averaging over a respiratory cycle. GPU based registrations of 256×256×64 imaging volumes required 3.6 sec for 100 iterations, a 56X speedup compared with conventional CPU computing (i.e. dual core 2.4GHz CPU). Dose remapping and dose volume histogram computations were near real‐time (⩽3sec). Conclusion: We have developed key elements of a clinically implementable ART strategy, which include real‐time computations using GPU, and accurate and robust registrations based on PC‐DIR. These elements incorporate 4DCT imaging coupled with GPU based PC‐DIR and frequent conebeam CT (CBCT) verification. This methodology can exploit ongoing improvement in GPU hardware performance and CBCT image quality.

Graphics processing units (GPU) are considered to have superior performance over the central processing units (CPU) in performing common scientific computations. Number of factors that can seriously change this conception are usually overlooked. In this paper, some of these factors are taken into account and their impact is measured, analysed and discussed. Matrix multiplication and Shell sorting examples are used to show that GPU superiority is not always clear.

We consider the problem of efficient image transmission over noisy time-varying channels subject to a low transmission energy constraint (and fixed bandwidth/delay constraints). We examine the limits of desirability of a highly representation using a joint source-channel coding (JSCC) framework. Specifically, invoking as a platform a state-of-the-art wavelet image coder, we demonstrate how the resulting highly digital stream, appropriately protected against channel noise, is not always the best solution. We show how a hybrid scheme based on simple partitioning of the wavelet image representation into compressed and uncompressed components can lead to significantly improved performance (of the order of 3 dB in PSNR for Rayleigh channels) over popular JSCC schemes which are based on compressed, entropy-coded, and appropriately unequal error-protected (UEP) source representations.

We propose an iterative design algorithm for jointly optimizing source and channel codes. The joint design combines channel-optimized vector quantization (COVQ) for the source code with rate-compatible punctured convolutional (RCPC) coding for the channel code. Our objective is to minimize the average end-to-end distortion. For a given channel SNR and transmission rate, our joint source and channel code design achieves an optimal allocation of bits between the source and channel coders. This optimal allocation can reduce distortion by up to 6 dB over suboptimal allocations for the source data set considered. We also compare the distortion of our joint iterative design with that of two suboptimal design techniques: COVQ optimized for a given channel bit-error-probability, and RCPC channel coding optimized for a given vector quantizer. We conclude by relaxing the fixed transmission rate constraint and jointly optimizing the transmission rate, source code, and channel code.

In this paper, we investigate the impact of channel coding on two kinds of non-orthogonal multiple access (NOMA) schemes, either with a successive interference cancellation (SIC) receiver or with a joint decoding (JD) receiver. Firstly, the information-theoretic individual achievable signal-to-noise ratio (SNR) regions of SIC and JD are analyzed. Then single-user-oriented (SU) code and multi-users-oriented (MU) code are considered for SIC and JD, and the practical individual achievable SNR regions of SIC and JD with these two kinds of channel codes are obtained, which provides an intuitive way to evaluate system performance under different channel conditions and transmission rates. Simulation results show that, the practical boundaries of individual achievable SNR region of SIC generally lie inside the boundaries of JD, as predicted by the theoretical analysis. Moreover, SU code, which is capacity-approaching over point-to-point channel, performs better than MU code for SIC, while the choice of channel codes for JD depends on transmission rates and channel conditions.