Utilizing Information Bottleneck to Evaluate the Capability of Deep Neural Networks for Image Classification †

Recent major developments in semantic communication systems stem from integration of deep learning (DL) techniques. Following the discovery of capacity achieving codes, the primary motivation for adopting the semantic approach, which retrieves meaning without requiring an exact reconstruction, is its potential to further conserve resources such as bandwidth and power. In this paper, we propose a novel semantic communication framework for textual data over additive white Gaussian noise (AWGN) channels via DL. Our framework leverages the information bottleneck (IB) principle to balance minimizing bit transmission under wireless channel rate constraints with maximizing semantic information retention. Unlike previous works, we integrate the bilingual evaluation understudy (BLEU) sentence similarity score into the training objective to enhance model performance. In particular, inspired by knowledge distillation, we utilize large language models (LLMs) during training to transfer their knowledge of text semantics into our model. Using IB principle, we train a neural semantic encoder at the transmitter and a neural semantic decoder at the receiver that incorporates into its objective function the rate constraint together with the BLEU score and the knowledge encoded in the soft probabilities produced by the LLM. Through extensive experiments, our proposed framework demonstrates a notable improvement of up to 45% in text semantic similarity compared to state-of-the-art benchmarks operating at the same channel capacity, significantly outperforming traditional communication systems. Moreover, it exhibits robustness to variations in signal-to-noise ratio (SNR) and achieves significant gains across both low and medium SNR regimes.

Neural ordinary differential equations (neural ODEs) have emerged as a novel network architecture that bridges dynamical systems and deep learning. However, the gradient obtained with the continuous adjoint method in the vanilla neural ODE is not reverse-accurate. Other approaches suffer either from an excessive memory requirement due to deep computational graphs or from limited choices for the time integration scheme, hampering their application to large-scale complex dynamical systems. To achieve accurate gradients without compromising memory efficiency and flexibility, we present a new neural ODE framework, PNODE, based on high-level discrete adjoint algorithmic differentiation. By leveraging discrete adjoint time integrators and advanced checkpointing strategies tailored for these integrators, PNODE can provide a balance between memory and computational costs, while computing the gradients consistently and accurately. We provide an open-source implementation based on PyTorch and PETSc, one of the most commonly used portable, scalable scientific computing libraries. We demonstrate the performance through extensive numerical experiments on image classification and continuous normalizing flow problems. We show that PNODE achieves the highest memory efficiency when compared with other reverse-accurate methods. On the image classification problems, PNODE is up to two times faster than the vanilla neural ODE and up to 2.3 times faster than the best existing reverse-accurate method. We also show that PNODE enables the use of the implicit time integration methods that are needed for stiff dynamical systems.

In this work, we examine the approximation capabilities of deep neural networks utilizing the Rectified Quadratic Unit (ReQU) activation function, defined as max(0,x)2, for approximating Hölder-regular functions with respect to the uniform norm. We constructively prove that deep neural networks with ReQU activation can approximate any function within the R-ball of r-Hölder-regular functions (Hr,R([-1,1]d)) up to any accuracy ϵ with at most Oϵ-d/2r neurons and fixed number of layers. This result highlights that the effectiveness of the approximation depends significantly on the smoothness of the target function and the characteristics of the ReQU activation function. Our proof is based on approximating local Taylor expansions with deep ReQU neural networks, demonstrating their ability to capture the behavior of Hölder-regular functions effectively. Furthermore, the results can be straightforwardly generalized to any Rectified Power Unit (RePU) activation function of the form max(0,x)p for p≥2, indicating the broader applicability of our findings within this family of activations.

Cryo Electron Microscopy (Cryo-EM) is currently one of the main tools to reveal the structural information of biological specimens at high resolution. Despite the great development of the techniques involved to solve the biological structures with Cryo-EM in the last years, the reconstructed 3D maps can present lower resolution due to errors committed while processing the information acquired by the microscope. One of the main problems comes from the 3D alignment step, which is an error-prone part of the reconstruction workflow due to the very low signal-to-noise ratio (SNR) common in Cryo-EM imaging. In fact, as we will show in this work, it is not unusual to find a disagreement in the alignment parameters in approximately 20–40% of the processed images, when outputs of different alignment algorithms are compared.In this work, we present a novel method to align sets of single particle images in the 3D space, called DeepAlign. Our proposal is based on deep learning networks that have been successfully used in plenty of problems in image classification. Specifically, we propose to design several deep neural networks on a regionalized basis to classify the particle images in sub-regions and, then, make a refinement of the 3D alignment parameters only inside that sub-region. We show that this method results in accurately aligned images, improving the Fourier shell correlation (FSC) resolution obtained with other state-of-the-art methods while decreasing computational time.

Feature mapping using far-field microphones for distant speech recognition

The article focuses on studying the effectiveness of two different Hybrid Neural Networks (HNNs) architectures for solving real-world image classification problems. The first approach investigated in the research is a hybridization technique that allows creation of HNN based on a classical neural network by replacing a number of hidden layers of the neural network with a variational quantum circuit, which allows to reduce the complexity of the classical part of the neural network and move part of computations to a quantum device. The second approach is a hybridization technique based on utilizing quanvolutional operations for image processing as the first quantum convolutional layer of the hybrid neural network, thus building a Quanvolutional Neural Network (QNN). QNN leverages quantum phenomena to facilitate feature extraction, enabling the model to achieve higher accuracy metrics than its classical counterpart. The effectiveness of both architectures was tested on several image classification problems. The first one is a classical image classification problem of CIFAR10 images classification, widely used as a benchmark for various imagery-related tasks. Another problem used for the effectiveness study is the problem of geospatial data analysis. The second problem represents a real-world use case where quantum computing utilization can be very fruitful in the future. For studying the effectiveness, several models were assembled: HNN with a quantum device that replaces one of the hidden layers of the neural network, QNN based on quanvolutional operation and utilizes VGG-16 architecture as a classical part of the model, and also an unmodified VGG-16 was used as a reference model. Experiments were conducted to measure the models' key efficiency metrics: maximal accuracy, complexity of a quantum part of the model and complexity of a classical part of the model. The results of the research indicated the feasibility of both approaches for solving both proposed image classification problems. Results were analyzed to outline the advantages and disadvantages of every approach in terms of selected key metrics. Experiments showed that QNN architectures proved to be a feasible and effective solution for critical practical tasks requiring higher levels of model prediction accuracy and, simultaneously, can tolerate higher processing time and significantly increased costs due to a high number of quantum operations required. Also, the results of the experiments indicated that HNN architectures proved to be a feasible solution for time-critical practical tasks that require higher processing speed and can tolerate slightly decreased accuracy of model predictions.

Tampered images pose a serious predicament since digitized media is a ubiquitous part of our lives. These are facilitated by the availability of image editing software and recent advances in deep Generative Adversarial Networks (GANs). We propose an innovative method to formulate the problem of 10-calizing manipulated regions in fake images as a deep representation learning problem using the Information Bottleneck (IB) principle. We devise a convolutional neural net-based architecture, InfoPrint (IP), that uses variational inference to approximate the IB formulation. Testing on three standard datasets, we demonstrate that InfoPrint outperforms the state-of-the-art by 3% points or more. Additionally, we demonstrate that it has the ability to to detect alterations made by inpainting GANs.

Deep Convolution Neural Network is one of the most powerful tools to solve complex problems of image classification, image recognition, financial analysis, medical diagnosis and many similar problems. A dental panoramic image consists of collection of teeth of both upper jaw and lower jaw. Automatic classification of dental panoramic images into various tooth types such as canines, incisors, premolars and molars has been a challenging task and involves crucial role of an experienced dentist. In this paper, we propose a technique for numbering and classification of the panoramic dental images. The proposed algorithm consists of four stages namely pre-processing, segmentation, numbering and classification. The pre-processed panoramic dental images are segmented using fuzzy c-mean clustering and subjected to vertical integral projection to extract a single tooth. The image dataset consists of 400 dental panoramic images collected from various dental clinics. The 400 dental images are divided into 240 training samples and 160 testing samples. The image data set is augmented by applying various transformations. Panoramic dental images are further numbered using a universal dental numbering system. Finally, the classification is done with the help of 6-layer deep convolution neural network (DCNN) consisting of 3 convolutional neural network and 3 fully connected network. The tooth is classified as canine, incisor, molar and premolar. An accuracy of 95% has been achieved for augmented database and 92% for original dataset with the proposed algorithm. The proposed numbering and classification of dental panoramic images is useful in biomedical application and postmortem recording of dental records. In case of big calamity, the system can also assist the dentist in recording post mortem dental record that is a very lengthy and arduous task.

Automated image classification is an essential task of the computer vision field. The tagging of images into a set of predefined groups is referred to as image classification. The implementation of computer vision to automate image classification would be beneficial because manual image evaluation and identification can be time-consuming, particularly when there are many images of different classes. Deep learning approaches are proven to overperform existing machine learning techniques in many fields in recent years, and computer vision is one of the most notable examples. The very deep neural network (VDCNN) is a powerful deep learning model for image classification, and this paper examines it briefly using MNIST handwritten digit dataset. This dataset is used to prove the efficacy of very deep neural networks over other deep learning models. The proposed study aims to comprehend the very deep neural network architecture used to accomplish a handwritten digit recognition task. The feasibility of the proposed model is evaluated using mean accuracy, validation accuracy, and standard deviation. The study results of the very deep neural network model are compared to a convolutional neural network and convolutional neural network with batch normalization. According to the results of the comparison study, very deep neural networks achieve high accuracy of 99.1% for a handwritten dataset. The outcome of the proposed work is used to interpret how well a very deep neural network performs when compared to the other two models of deep neural networks. This proposed architecture may be used to automate the classification of handwritten digits dataset.

In this study, the authors address a challenging problem of aesthetic image classification, which is to label an input image as high‐ or low‐aesthetic quality. We take both the local and global features of images into consideration. A novel deep convolutional neural network named ILGNet is proposed, which combines both the inception modules and a connected layer of both local and global features. The ILGnet is based on GoogLeNet. Thus, it is easy to use a pre‐trained GoogLeNet for large‐scale image classification problem and fine tune their connected layers on a large‐scale database of aesthetic‐related images: AVA, i.e. domain adaptation . The experiments reveal that their model achieves the state of the arts in AVA database. Both the training and testing speeds of their model are higher than those of the original GoogLeNet.

The generalization capability of deep neural networks has been substantially improved by applying a wide spectrum of regularization methods, e.g., restricting function space, injecting randomness during training, augmenting data, etc. In this work, we propose a simple yet effective regularization method named progressive self-knowledge distillation (PS-KD), which progressively distills a model's own knowledge to soften hard targets (i.e., one-hot vectors) during training. Hence, it can be interpreted within a framework of knowledge distillation as a student becomes a teacher itself. Specifically, targets are adjusted adaptively by combining the ground-truth and past predictions from the model itself. We show that PS-KD provides an effect of hard example mining by rescaling gradients according to difficulty in classifying examples. The proposed method is applicable to any supervised learning tasks with hard targets and can be easily combined with existing regularization methods to further enhance the generalization performance. Furthermore, it is confirmed that PS-KD achieves not only better accuracy, but also provides high quality of confidence estimates in terms of calibration as well as ordinal ranking. Extensive experimental results on three different tasks, image classification, object detection, and machine translation, demonstrate that our method consistently improves the performance of the state-of-the-art baselines. The code is available at https://github.com/lgcnsai/PS-KD-Pytorch.

As the architectures and capabilities of deep neural networks evolve, they become more sophisticated to train and use. Deep Learning Recommendation Model (DLRM), a new neural network for recommendation systems, introduces challenging requirements for deep neural network training and inference. The size of the DLRM model is typically large and not able to fit on a single GPU memory. Unlike other deep neural networks, DLRM requires both model-parallel and data-parallel for the bottom part and top part of the model when running on multiple GPUs. Due to the hybrid-parallel model, the all-to-all communication is used for welding the top and bottom parts together. We have observed that the all-to-all communication is costly and is a bottleneck in the DLRM training/inference. In this paper, we propose a novel approach to reduce the communication volume by using DLRM’s properties to compress the transferred data without information loss. We demonstrate benefits of our method by training DLRM MLPerf on eight AMD Instinc$\mathrm{t}^{\mathrm{T}\mathrm{M}}$ MI100 accelerators. The experimental results show 59% and 38% improvement in the time-to-solution of the DLRM MLPerf training for FP32 and mixed-precision, respectively.

This paper investigates task-oriented communication for multi-device cooperative edge inference, where a group of distributed low-end edge devices transmit the extracted features of local samples to a powerful edge server for inference. While cooperative edge inference can overcome the limited sensing capability of a single device, it substantially increases the communication overhead and may incur excessive latency. To enable low-latency cooperative inference, we propose a learning-based communication scheme that optimizes local feature extraction and distributed feature encoding in a task-oriented manner, i.e., to remove data redundancy and transmit information that is essential for the downstream inference task rather than reconstructing the data samples at the edge server. Specifically, we leverage Tishby’s information bottleneck (IB) principle (Tishby et al., 2000) to extract the task-relevant feature at each edge device, and adopt the distributed information bottleneck (DIB) framework of Aguerri and Zaidi, 2021, to formalize a single-letter characterization of the optimal rate-relevance tradeoff for distributed feature encoding. To admit flexible control of the communication overhead, we extend the DIB framework to a distributed deterministic information bottleneck (DDIB) objective that explicitly incorporates the representational costs of the encoded features. As the IB-based objectives are computationally prohibitive for high-dimensional data, we adopt variational approximations to make the optimization problems tractable. To compensate for the potential performance loss due to the variational approximations, we also develop a selective retransmission (SR) mechanism to identify the redundancy in the encoded features among multiple edge devices to attain additional communication overhead reduction. Extensive experiments on multi-view image classification and multi-view object recognition tasks evidence that the proposed task-oriented communication scheme achieves a better rate-relevance tradeoff than existing methods.

Genetic programming (GP) is a well-known evolutionary computation technique, which has been successfully used to solve various problems, such as optimization, image analysis, and classification. Transfer learning is a type of machine learning approach that can be used to solve complex tasks. Transfer learning has been introduced to GP to solve complex Boolean and symbolic regression problems with some promise. However, the use of transfer learning with GP has not been investigated to address complex image classification tasks with noise and rotations, where GP cannot achieve satisfactory performance, but GP with transfer learning may improve the performance. In this paper, we propose a novel approach based on transfer learning and GP to solve complex image classification problems by extracting and reusing blocks of knowledge/information, which are automatically discovered from similar as well as different image classification tasks during the evolutionary process. The proposed approach is evaluated on three texture data sets and three office data sets of image classification benchmarks, and achieves better classification performance than the state-of-the-art image classification algorithm. Further analysis on the evolved solutions/trees shows that the proposed approach with transfer learning can successfully discover and reuse knowledge/information extracted from similar or different problems to improve its performance on complex image classification problems.

Nowadays, deep and transfer learning technologies and its algorithms have become more popular in research area of computer vision field for image recognition and classification tasks. There are various kinds of neural networks broadly used in research area of image recognition and classification problems such as CNN, DNN, etc. In this research work, the researcher proposed and developed two customized modules. The first module is used for authentication of students in the online examination system. The second module is about to find the behavior analysis and facial expression recognition of students during online examination. The proposed authentication and facial expression analysis module is constructed by combining features of CNN and transfer learning approach with pre-trained architectures. To develop proposed system the models used are trained and validated by using various datasets. It also calculates the performance of students during the online examination period. The accuracy of these deep learning models highly based on data quantity used for training and testing model. Training and testing datasets depend on types of problems considered for which models are to be built.