Towards Efficient Forward Propagation on Resource-Constrained Systems

Deep Convolutional Neural Networks (CNNs) perform billions of operations for classifying a single input. To reduce these computations, this paper offers a solution that leverages a combination of runtime information and the algorithmic structure of CNNs. Specifically, in numerous modern CNNs, the outputs of compute-heavy convolution operations are fed to activation units that output zero if their input is negative. By exploiting this unique algorithmic property, we propose a predictive early activation technique, dubbed SnaPEA. This technique cuts the computation of convolution operations short if it determines that the output will be negative. SnaPEA can operate in two distinct modes, exact and predictive. In the exact mode, with no loss in classification accuracy, SnaPEA statically re-orders the weights based on their signs and periodically performs a single-bit sign check on the partial sum. Once the partial sum drops below zero, the rest of computations can simply be ignored, since the output value will be zero in any case. In the predictive mode, which trades the classification accuracy for larger savings, SnaPEA speculatively cuts the computation short even earlier than the exact mode. To control the accuracy, we develop a multi-variable optimization algorithm that thresholds the degree of speculation. As such, the proposed algorithm exposes a knob to gracefully navigate the trade-offs between the classification accuracy and computation reduction. Compared to a state-of-the-art CNN accelerator, SnaPEA in the exact mode, yields, on average, 28% speedup and 16% energy reduction in various modern CNNs without affecting their classification accuracy. With 3% loss in classification accuracy, on average, 67.8% of the convolutional layers can operate in the predictive mode. The average speedup and energy saving of these layers are 2.02x and 1.89x, respectively. The benefits grow to a maximum of 3.59x speedup and 3.14x energy reduction. Compared to static pruning approaches, which are complimentary to the dynamic approach of SnaPEA, our proposed technique offers up to 63% speedup and 49% energy reduction across the convolution layers with no loss in classification accuracy.

New instruments currently in design for Earth observational remote sensing from space offer promising advancements in the use of remote sensing for soil studies. Due to the enormous quantities of data generated by these high spectral resolution sensors, efficient use will only be feasible if new algorithms are developed to compress the data while maintaining information content. A spectral band selection algorithm is proposed to identify the important spectral bands for compression and classification of soil reflectance data. This algorithm was developed from the shape dominancy concept of Karhunen‐Loeve based optimal features. The high dimensional raw data were first transformed into much lower dimensionality by the derived spectral features. Canonical analysis was then used to transform the data under the maximal separability criterion into their final signal space where the data classification was performed. Soil data sets with and without stratification by soil order and climatic moisture zone were used to test the algorithm. The probabilities of correct classification of soil organic matter content using Landsat Multispectral Scanner (MSS) bands, Thematic Mapper (TM) bands, and the bands identified using the new band selection algorithm, were calculated and compared. The algorithm was successful in finding important spectral bands for soil organic matter content classification. The probabilities of correct classification obtained by using these bands were 0.850 and 0.883 for unstratified data sets, as compared to 0.600 to 0.640 for MSS bands and 0.640 to 0.653 for TM bands. Probabilities of correct classification for climate‐stratified data ranged from 0.910 to 0.980 using the calculated bands. The overall data compression ratio achieved using the algorithm was greater than 10 with no loss in classification accuracy.

Deep learning techniques have attracted much attention in the radar automatic target recognition. In this paper, we investigate an acceleration method of the convolutional neural network (CNN) on the field-programmable gate array (FPGA) for the embedded application of the millimeter-wave (mmW) radar-based human activity classification. Considering the micro-Doppler effect caused by a person's body movements, the spectrogram of mmW radar echoes is adopted as the CNN input. After that, according to the CNN architecture and the properties of the FPGA implementation, several parallel processing strategies are designed as well as data quantization and optimization of classification decision to accelerate the CNN execution. Finally, comparative experiments and discussions are carried out based on a measured dataset of nine individuals with four different actions by using a 77-GHz mmW radar. The results show that the proposed method not only maintains the high classification accuracy but also improves its execution speed, memory requirement, and power consumption. Specifically, compared with the implementation of the same network model on a graphics processing unit, it could achieve the speedup of about 30.42% at the cost of the classification accuracy loss of only 0.27%.

The global rise in disasters—driven by climate change and increasing terrorist threats—has made rapid identification and recovery of trapped victims critical. Structural collapses pose unique challenges for search and rescue (SAR) operations, particularly due to non-line-of-sight (NLOS) environments that obscure signal clarity and human presence detection. To address this, machine learning (ML) approaches must be enhanced to effectively process complex NLOS signals. This study explores the classification potential of ML models for ultrawideband line-of-sight and NLOS signal datasets, employing Independent Component Analysis (ICA) for feature extraction. ICA isolates latent components from high-dimensional data, enhancing the interpretability and performance of classification. A Bagged Ensemble classifier is then applied to the extracted features. Experimental results on an NLOS signal dataset show that down sampled data can be efficiently processed without loss of classification accuracy or feature quality, achieving an 86.53% accuracy rate. Furthermore, the proposed method reduces computational load while accelerating classification speed. These findings underscore the transformative potential of ensemble-based ML approaches like Bagged Ensemble in improving the accuracy, speed, and efficiency of SAR missions—thereby strengthening disaster response and saving more lives in real-world collapse scenarios.

Training Deep Neural Networks (DNNs) places immense compute requirements on the underlying hardware platforms, expending large amounts of time and energy. An important factor contributing to the long training times is the increasing dataset complexity required to reach state-of-the-art performance in real-world applications. To address this challenge, we explore the use of input mixing, where multiple inputs are combined into a single composite input with an associated composite label for training. The goal is for training on the mixed input to achieve a similar effect as training separately on each the constituent inputs that it represents. This results in a lower number of inputs (or mini-batches) to be processed in each epoch, proportionally reducing training time. We find that naive input mixing leads to a considerable drop in learning performance and model accuracy due to interference between the forward/backward propagation of the mixed inputs. We propose two strategies to address this challenge and realize training speedups from input mixing with minimal impact on accuracy. First, we reduce the impact of inter-input interference by exploiting the spatial separation between the features of the constituent inputs in the network's intermediate representations. We also adaptively vary the mixing ratio of constituent inputs based on their loss in previous epochs. Second, we propose heuristics to automatically identify the subset of the training dataset that is subject to mixing in each epoch. Across ResNets of varying depth, MobileNetV2 and two Vision Transformer networks, we obtain upto 1.6 × and 1.8 × speedups in training for the ImageNet and Cifar10 datasets, respectively, on an Nvidia RTX 2080Ti GPU, with negligible loss in classification accuracy.

Hardware accelerators are being increasingly deployed to boost the performance and energy efficiency of deep neural network (DNN) inference. In this paper we propose Thundervolt, a new framework that enables aggressive voltage underscaling of high-performance DNN accelerators without compromising classification accuracy even in the presence of high timing error rates. Using post-synthesis timing simulations of a DNN accelerator modeled on the Google TPU, we show that Thundervolt enables between 34%-57% energy savings on state-of-the-art speech and image recognition benchmarks with less than 1% loss in classification accuracy and no performance loss. Further, we show that Thundervolt is synergistic with and can further increase the energy efficiency of commonly used run-time DNN pruning techniques like Zero-Skip.

Pattern recognition involves identifying a new instance as a member of a class based on the features extracted from problem space. Not all features contribute to increase in accuracy of classification. Wrapper based methodology selects an optimal subset of reduced features with no much loss in classification accuracy. Inspired by the bio evolutionary behaviour of flowers in optimal reproduction, a wrapper based optimal feature selection methodology is devised. In this paper, a Binary Flower Pollination Algorithm (BFPA) is proposed to address the combinatorial problem of feature selection. Support Vector machine (SVM) is used as a subset evaluator to calculate the fitness function to be optimized. Proposed algorithm is evaluated using 5 datasets from UCI repository. Switching probability is varied to achieve a fair tradeoff between exploration and exploitation. Feature subset chosen by the proposed method serves as input to 6 traditional classifiers to analyse the success of the algorithm. Experimental results are also compared with existing literature and observed that the proposed methodology identifies feature subset with reduced features and improved accuracy.

In analog neuromorphic chips, designers can embed computing primitives in the intrinsic physical properties of devices and circuits, heavily reducing device count and energy consumption, and enabling high parallelism, because all devices are computing simultaneously. Neural network parameters can be stored in local analog non-volatile memories (NVMs), saving the energy required to move data between memory and logic. However, the main drawback of analog sub-threshold electronic circuits is their dramatic temperature sensitivity. In this paper, we demonstrate that a temperature compensation mechanism can be devised to solve this problem. We have designed and fabricated a chip implementing a two-layer analog neural network trained to classify low-resolution images of handwritten digits with a low-cost single-poly complementary metal-oxide-semiconductor (CMOS) process, using unconventional analog NVMs for weight storage. We demonstrate a temperature-resilient analog neuromorphic chip for image recognition operating between 10°C and 60°C without loss of classification accuracy, within 2% of the corresponding software-based neural network in the whole temperature range.

paper, we focus on exploring the relationship between weave patterns and their mechanical properties in woven fiber composites through Machine Learning. Specifically, we explore the interactions between woven architectures and in-plane stiffness properties through Deep Convolutional Neural Network (DCNN) and Generative Adversarial Network (GAN). Our research is important for exploring how woven composite’s pattern is related to its mechanical properties and accelerating woven composite design as well as optimization. We focus on two tasks: (1) Stiffness prediction: Predicting in-plane stiffness properties for given weave patterns. Our DCNN extracts high-level features through several convolutional and fully connected layers to determine the final predictions. (2) Weave pattern prediction: Predicting weave patterns for target stiffness properties, which can be treated as the reverse task of the first one. Due to many-to-one mapping between weave patterns and the composite properties, we utilize a Decoder Neural Network as our baseline model and compare its performance with GAN and Genetic Algorithm. We represent the weave patterns as 2D checkerboard models and use finite element analysis (FEA) to determine in-plane stiffness properties, which serve as input data for our ML framework. We show that: (1) for stiffness prediction, DCNN can predict stiffness values for a given weave pattern with relatively high accuracy (above 93%); (2) for weave pattern prediction, the GAN model gives the best prediction accuracy (above 92%) while Decoder Neural Network has the best time efficiency. HAOTIAN FENG

This paper introduces an automated method for estimating sex from the lower and upper limbs based on diaphyseal CSG properties. The proposed method was developed and evaluated using 389 femurs, 412 tibias, and 404 humeri of adult individuals from a modern Greek reference sample, the Athens Collection. The skeletal properties, which were extracted with the CSG-Toolkit, were analyzed with step-wise DFA (evaluated with LOOCV) and subsequently with RBF kernel SVM supervised learning. SVM cross-validation was based on a 20-fold stratified random sample splitting as well as a chronological split based on year of birth to further assess the effect of secular change in sex estimation capacity. Maximum cross-validated classification accuracy from step-wise DFA reached 94.8% for the femur, 94.7% for the tibia, and 97.3% for the humerus, whereas SVM cross-validated results were similar although slightly lower, mainly due to the more strict cross-validation scheme. Our results suggest that the proposed sex estimation method is reasonably robust to secular change, since there was limited loss in classification accuracy between different chronological groups, despite the presence of secular change in stature of the Greek population during the examined period. The proposed method has been implemented as a function for the GNU Octave environment, named estimate_sex, which comprises a self-intuitive graphical user interface for facilitating sex estimation and is freely available under a suitable license.

We propose a hierarchical stimulus presentation paradigm for a P300-based Hangul (Korean script) input system. A P300-based input system is one of the most promising noninvasive brain-computer interface (BCI) applications based on its direct applicability in many computer programs. While the previous row/column stimulus presentation paradigm has been well-suited to the English input, it may not be optimal for a Hangul input because Hangul has a distinct hierarchical structure. To overcome the limitation of the previous paradigms, we developed a new P300-based Hangul input system by taking the unique hierarchical structure of Hangul into account for creating a hierarchical stimulus presentation paradigm. By using the hierarchical structure, we can effectively reduce the window size of the interface without loss of classification accuracy. A performance comparison shows that the hierarchical paradigm exhibits higher classification accuracy than the row/column paradigm even with a smaller window size. Thus the proposed hierarchical paradigm is more efficient to spell Hangul and will be more useful for BCI-based Hangul input for a text messenger, e-mail program, word processor, and so on.

Deep learning (DL) has shown outstanding performances in many fields, including remote sensing (RS). DL is turning into an essential tool for the RS research community. Recently, many cloud platforms have been developed to provide access to large-scale computing capacity, consequently permitting the usage of DL architectures as a service. However, this opened the door to new challenges associated with the privacy and security of data. The RS data used to train the DL algorithms have several privacy requirements. Some of them need a high level of confidentiality, such as satellite images related to public security with high spatial resolutions. Moreover, satellite images are usually protected by copyright, and the owner may strictly refuse to share them. Therefore, privacy-preserving deep learning (PPDL) techniques are a possible solution to this problem. PPDL enables training DL on encrypted data without revealing the original plaintext. This study proposes a hybrid PPDL approach for object classification for very-high-resolution satellite images. The proposed encryption scheme combines Paillier homomorphic encryption (PHE) and somewhat homomorphic encryption (SHE). This combination aims to enhance the encryption of satellite images while ensuring a good runtime and high object classification accuracy. The method proposed to encrypt images is maintained through the public keys of PHE and SHE. Experiments were conducted on real-world high-resolution satellite images acquired using the SPOT6 and SPOT7 satellites. Four different CNN architectures were considered, namely ResNet50, InceptionV3, DenseNet169, and MobileNetV2. The results showed that the loss in classification accuracy after applying the proposed encryption algorithm ranges from 2% to 3.5%, with the best validation accuracy on the encrypted dataset reaching 92%.

In the context of support vector machines (SVM), high dimensional input vectors often reduce the computational efficiency and significantly slow down the classification process. In this paper, we propose a strategy to rank individual components according to their influence on the class assignments. This ranking is used to select an appropriate subset of the features. It replaces the original feature set without significant loss in classification accuracy. Often, the generalization ability of the classifier even increases due to the implicit regularization achieved by feature pruning.

Traffic classification using statistical characteristics (or fingerprints) of IP flows such as packet size and packet inter-arrival time has showed its preliminary success in sense of accuracy and simplicity. However, the need of large memory to store the fingerprints makes it impractical to deploy such method on backbone networks where a high-speed memory is needed to catch up with the high packet rate, and a large high-speed memory is always expensive. In this paper we apply the Distributional Clustering (DC) algorithm proposed by the pattern recognition community to compress the protocol fingerprints. We also presented a new algorithm named Distributional Quantification (DQ) that has lower overhead than DC. We evaluated the accuracy of classification using compact protocol fingerprints under various compression ratios through experiments. Our results show that DC outperforms DQ in terms of classification accuracy. The experimental results indicate that a compression ratio of 9 can be achieved with no more than 10% loss in classification accuracy.

Adaboost has been found by many researchers to be an extremely successful classification algorithm. However, it is often stated that Adaboost takes a significant amount of time to learn a good classification. This paper aims to show that the Adaboost algorithm is not the cause of the slow learning. By altering how Adaboost is used, a significant increase in learning speed, over 14 times faster, can be achieved at the cost of a small loss in classification accuracy.