Lightweight Network for Vietnamese Landmark Recognition based on Knowledge Distillation

Advances in smart home technology and IoT devices had made us capable of monitoring human activities in a non-intrusive way. This data, in turn, enables us to predict the health status and energy consumption patterns of residents of these smart homes. Machine learning has been an excellent tool for the prediction of human activities from raw sensor data of a single resident. However, Multi Resident activity recognition is still a challenging task, as there is no correlation between sensor values and resident activities. In this paper, we have applied deep learning algorithms on the real world ARAS Multi Resident data set, which consists of data from two houses, each with two residents. We have used different variations of Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), and their combination on the data set and kept the labels separate for both residents. We have evaluated the performance of models based on several metrics.

The deep convolutional neural network has made significant progress in cloud detection. However, the compromise between having a compact model and high accuracy has always been a challenging task in cloud detection for large-scale remote sensing imagery. A promising method to tackle this problem is knowledge distillation, which usually lets the compact model mimic the cumbersome model’s output to get better generalization. However, vanilla knowledge distillation methods cannot properly distill the characteristics of clouds in remote sensing images. In this paper, we propose a novel self-attention knowledge distillation approach for compact and accurate cloud detection, named Bidirectional Self-Attention Distillation (Bi-SAD). Bi-SAD lets a model learn from itself without adding additional parameters or supervision. With bidirectional layer-wise features learning, the model can get a better representation of the cloud’s textural information and semantic information, so that the cloud’s boundaries become more detailed and the predictions become more reliable. Experiments on a dataset acquired by GaoFen-1 satellite show that our Bi-SAD has a great balance between compactness and accuracy, and outperforms vanilla distillation methods. Compared with state-of-the-art cloud detection models, the parameter size and FLOPs are reduced by 100 times and 400 times, respectively, with a small drop in accuracy.

Deep learning models have gained prominence in AI research due to their exceptional feature extraction capabilities. In fingerprint-based indoor localization, 1D and 2D convolutional neural networks (CNNs) are frequently utilized for their superior performance and minimal hardware requirements. However, these models primarily extract spatial features and often exhibit degraded real-time performance due to signal inconsistencies caused by shadowing, fading, and multipath effects. To address this, incorporating temporal variations through 3D CNNs can enhance outcomes, albeit with increased computational costs. This study proposes a knowledge distillation (KD) approach using 3D CNNs for indoor localization, to bridge the computational and performance gaps. The proposed KD framework employs a complex teacher model (TM) to train two simpler student models A (SM-A) and B (SM-B). We employ both response-based and feature-based KD methods to optimize the student models’ weights. The proposed techniques are evaluated using the UJILIB dataset collected from two floors of the University of Jaume I library. To enhance model performance, data augmentation is applied, yielding substantial improvements in test accuracy of 10.56% for TM, 16.19% for SM-A, and 16.93% for SM-B. The application of KD results in accuracy improvements of 5.45% for SM-A and 3.35% for SM-B, consequently reducing their average positioning errors (APEs) from 1.070 m and 1.030 m to 0.840 m and 0.944 m, respectively. Compared to KD-based SM-A, TM shows only 3.30% higher positioning accuracy, but with a 65.14% increase in computational complexity. Notably, the KD-enhanced SM-A also achieves a significantly lower APE of 0.84 m compared to the 2D CNN of 1.72 m.

paper, we focus on exploring the relationship between weave patterns and their mechanical properties in woven fiber composites through Machine Learning. Specifically, we explore the interactions between woven architectures and in-plane stiffness properties through Deep Convolutional Neural Network (DCNN) and Generative Adversarial Network (GAN). Our research is important for exploring how woven composite’s pattern is related to its mechanical properties and accelerating woven composite design as well as optimization. We focus on two tasks: (1) Stiffness prediction: Predicting in-plane stiffness properties for given weave patterns. Our DCNN extracts high-level features through several convolutional and fully connected layers to determine the final predictions. (2) Weave pattern prediction: Predicting weave patterns for target stiffness properties, which can be treated as the reverse task of the first one. Due to many-to-one mapping between weave patterns and the composite properties, we utilize a Decoder Neural Network as our baseline model and compare its performance with GAN and Genetic Algorithm. We represent the weave patterns as 2D checkerboard models and use finite element analysis (FEA) to determine in-plane stiffness properties, which serve as input data for our ML framework. We show that: (1) for stiffness prediction, DCNN can predict stiffness values for a given weave pattern with relatively high accuracy (above 93%); (2) for weave pattern prediction, the GAN model gives the best prediction accuracy (above 92%) while Decoder Neural Network has the best time efficiency. HAOTIAN FENG

The well-being of residents is a top priority for megacities, which is why urban design and sustainable development are crucial topics. Quality of Life (QoL) is used as an effective key performance index (KPI) to measure the efficiency of a city plan’s quantity and quality factors. For city dwellers, QoL for pedestrians is also significant. The walkability concept evaluates and analyzes the QoL in a walking scene. However, the traditional questionnaire survey approach is costly, time-consuming, and limited in its evaluation area. To overcome these limitations, the paper proposes using artificial intelligence (AI) technology to evaluate walkability data collected through a questionnaire survey using virtual reality (VR) tools. The proposed method involves knowledge extraction using deep convolutional neural networks (DCNNs) for information extraction and deep learning (DL) models to infer QoL scores. Knowledge distillation (KD) is also applied to reduce the model size and improve real-time performance. The experiment results demonstrate that the proposed approach is practical and can be considered an alternative method for acquiring QoL.

Rolling bearings are often exposed to high speeds and pressures, leading to the symmetry in their rotating structure being disrupted, which can lead to serious failures. Intelligent rolling bearing fault diagnosis is a critical part of ensuring operation of machinery, and it has been facilitated by the growing popularity of convolutional neural networks (CNNs). The outstanding performance of fault diagnosis CNNs results from complex and redundant network structures and parameters, resulting in huge storage and computational requirements, which makes it challenging to implement these models in resource-limited industrial devices. This study aims to address this problem by proposing a comprehensive compression method for CNNs that is applied to intelligent fault diagnosis. It involves several different compression methods, including tensor train decomposition, parameter quantization, and knowledge distillation for deep network compression. This results in a significant decrease in redundancy and speeding up the training of CNN models. Firstly, tensor train decomposition is applied to reduce redundant connections in both convolutional and fully connected layers. The next step is to perform parameter quantization to minimize the bits needed for parameter representation and storage. Finally, knowledge distillation is used to restore accuracy to the compressed model. The effectiveness of the proposed approach is confirmed by an experiment and ablation study with different models on several datasets. The results show that it can significantly reduce redundant information and floating-point operations with little degradation in accuracy. Notably, on the CWRU dataset, with about 60% parameter reduction, there is no degradation in our model’s accuracy. The proposed approach is a new attempt at the intelligent fault diagnosis of rolling bearings in industrial equipment.

FitCNN: A cloud-assisted and low-cost framework for updating CNNs on IoT devices

The groundbreaking performance of transformers in Natural Language Processing (NLP) tasks has led to their replacement of traditional Convolutional Neural Networks (CNNs), owing to the efficiency and accuracy achieved through the self‐attention mechanism. This success has inspired researchers to explore the use of transformers in computer vision tasks to attain enhanced long‐term semantic awareness. Vision transformers (ViTs) have excelled in various computer vision tasks due to their superior ability to capture long‐distance dependencies using the self‐attention mechanism. Contemporary ViTs like Data Efficient Transformers (DeiT) can effectively learn both global semantic information and local texture information from images, achieving performance comparable to traditional CNNs. However, their impressive performance comes with a high computational cost due to a very large number of parameters, hindering their deployment on devices with limited resources like smartphones, cameras, drones, etc. Additionally, ViTs require a large amount of data for training to achieve performance comparable to benchmark CNN models. Therefore, we identified two key challenges in deploying ViTs on smaller form factor devices: the high computational requirements of large models and the need for extensive training data. As a solution to these challenges, we propose compressing large ViT models using Knowledge Distillation (KD), which is implemented data‐free to circumvent limitations related to data availability. Additionally, we conducted experiments on object detection within the same environment in addition to classification tasks. Based on our analysis, we found that data‐free knowledge distillation is an effective method to overcome both issues, enabling the deployment of ViTs on less resource‐constrained devices.

A review on neural network techniques for the prediction of road traffic accident severity

Fingerprint-based indoor positioning systems (F-IPS) may provide inexpensive solutions to GPS-denied environments. Most F-IPSs adopt traditional machine learning for position prediction, resulting in low accuracy. Deep neural networks (DNN) were recently employed for F-IPSs to minimize positioning errors. Nevertheless, a DNN-IPS fails to guarantee high accuracy in dynamic environments as it is sensitive to changes in the input data. A convolutional neural network (CNN) is recommended to replace DNN due to its capability to learn the overall topology of fingerprinting images and capture highly abstract features. Due to the convolution process and image representation, CNN-IPS incurs prohibitive storage and computational requirement for implementation on resource-limited devices. This paper incorporates knowledge distillation (KD) into CNN-IPS to distil knowledge from large deep CNNs into small CNNs. The pre-trained teacher network uses the soft probability output where the score vector from the trained network is converted into a probability distribution, which is softened by the temperature hyperparameter, leading to a more simplified model. Based on the numerical results, KD-CNN-IPS manifests better localization performance where 79.84% of the positioning errors are within 2 meters while its testing time is only 79.68% of that of the teacher model. Compared to the CNN-IPS, KD-CNN-IPS with precisely the same architecture and size could achieve a performance improvement of 13.65% in terms of the average positioning error.

Remote sensing image scene classification is to label remote sensing images as a specific scene category by understanding the semantic information of the images. It is an essential link in remote sensing image analysis and interpretation and has important research value. Convolutional neural networks (CNNs) have been dominant in remote sensing image scene classification due to their powerful feature extraction capabilities. The general trend has been to make deeper and wider CNN architectures to achieve higher classification accuracy. However, these advances to improve accuracy enlarge the network, creating too many parameters and high computational costs. Large models are difficult to deploy on resource-constrained edge devices for practical applications. Furthermore, CNNs can effectively capture local information but are weak in extracting global features. To overcome these drawbacks, we propose a novel knowledge distillation (KD) based method by employing Swin Transformer as a teacher network for guiding MobileNetV2 with Parameter-Free Attention (MobileNetV2-PFA). First, we modify MobileNetV2 by introducing PFA into the inverted bottleneck block; this improvement helps the model learn more latent and robust features without extra parameters. Second, Swin Transformer is an excellent architecture for capturing long-range dependencies via shifted window-based attention. So, we utilize the long-range dependency information from the Swin Transformer to assist MobileNetV2-PFA training through KD. Experimental results on the challenging NWPU-RESISC45 dataset show that the proposed method outperforms the original MobileNetV2 in classification accuracy with low computational consumption.

Using low-rank adaptation to fine-tune pre-trained neural network models has attracted widespread attention due to its advantages of low resource requirements, high precision, and no additional inference delay. However, most existing methods are designed for large language models based on the Transformer structure and lack adaptation to convolutional neural networks (CNN) widely used on Internet of Things (IoT) devices. In addition, IoT devices are usually deployed outdoors and collect large amounts of data that are affected by environmental conditions. Providing a highly robust CNN model is a prerequisite for providing high-quality services. To this end, this paper proposes a highly robust seed-filter adaptation method (SeFA) for pre-trained CNNs. SeFA introduces an adaptation branch with the same structure as the backbone network. In the adaptation branch, some filters are first designated seed filters and grouped. Then, additional filters are generated from the grouped seed filters and nonlinear transformation functions (NLFs) with different hyperparameters. The seed filters' parameters are updated during model training, and the NLF hyperparameters are randomly initialized and frozen. Both grouping seed filters and configuring NLFs with non-learnable hyperparameters can improve the model's robustness. This is because grouping seed filters can generate diverse filters on demand without increasing the model's complexity, and the NLFs' rules can regularize the model. It is worth mentioning that the number of fine-tuning parameters of the pre-trained model that can adapt to downstream tasks can be flexibly controlled by specifying the number of seed filters. The key idea of this paper is to propose SeFA with flexible, controllable, learnable parameters, high robustness, and adaptability to pretrained CNN models, thereby facilitating fine-tuning on resource-constrained IoT devices and providing highly robust visual services. Experimental results on the CIFAR-10, CIFAR-10-C, CIFAR-100, CIFAR-100-C, and Icons-50 datasets demonstrate that the proposed SeFA outperforms other state-of-the-art methods. Specifically, on the ResNet-152 model and the CIFAR-10-C dataset, the accuracy of our SeFA is about 7% higher than that of the full fine-tuning method. In addition, we verify the efficiency of SeFA-based models for fine-tuning and inference on the device.

The study presents the results of the development and testing of deep learning neural network architectures, which demonstrate high accuracy rates in classifying neurophysiological data, in particular hemodynamic brain activation patterns obtained by functional near-infrared spectroscopy, during solving mathematical problems on spatial-numerical associations. The analyzed signal represents a multidimensional time series of oxyhemoglobin and deoxyhemoglobin dynamics. Taking the specificity of the fNIRS signal into account, a comparative analysis of 2 types of neural network architectures was carried out: (1) architectures based on recurrent neural networks: recurrent neural network with long short-term memory, recurrent neural network with long short-term memory with fully connected layers, bidirectional recurrent neural network with long short-term memory, convolutional recurrent neural network with long short-term memory; (2) architectures based on convolutional neural networks with 1D convolutions: convolutional neural network, fully convolutional neural network, residual neural network. Trained long short-term memory recurrent neural network architectures showed worse results in accuracy in comparison with 1D convolutional neural network architectures. Residual neural network (model_Resnet) demonstrated the highest accuracy rates in three experimental conditions more than 88% in detecting age-related differences in brain activation during spatial-numerical association tasks considering the individual characteristics of the respondents’ signal.

Deep convolutional neural networks (DNNs) have achieved outstanding performance in this field. Meanwhile, handwritten Chinese character recognition (HCCR) is a challenging area of research in the field of computer vision. DNNs require a large number of parameters and high memory consumption. To address these issues, this paper proposes an approach based on an attention mechanism and knowledge distillation. The attention mechanism improves the feature extraction and the knowledge distillation reduces the number of parameters. The experimental results show that ResNet18 achieves a recognition accuracy of 97.63% on the HCCR dataset with 11.25 million parameters. Compared with other methods, this study improves the performance for HCCR.

Identifying lethal vector and non-vector mosquitoes can become difficult for a layperson and sometimes even for experts, considering their visual similarities. Recently, deep learning (DL) became a solution to assist in differentiating the two mosquito types to reduce infections and enhance actions against them. However, the existing methods employed to develop a DL model for such a task tend to require massive amounts of computing resources and steps, making them impractical. Based on existing methods, most researchers rely on training pre-trained state-of-the-art (SOTA) deep convolutional neural networks (DCNN), which usually require about a million parameters to train. Hence, this method proposes an approach to craft a model with a far lower computing cost while attaining similar or even significantly better performance than pre-existing models in automating the taxonomy of several mosquitoes. This method combines the approach of layer-wise compression and feature fusion with enhanced residual learning that consists of a self-normalizing activation and depthwise convolutions.•The proposed method yielded a model that outperformed the most recent and classic state-of-the-art deep convolutional neural network models.•With the help of the modified residual block and knowledge distillation, the proposed method significantly reduced a fused model's cost while maintaining competitive performance.•Unlike other methods, the proposed method had the best performance-to-cost ratio.